Archive for the 'Requirement 11 – Regularly test security systems and processes' Category


Information Supplements Versus The PCI DSS

At various times over the years, the Council has repeatedly told QSAs, Participating Organizations (PO) and anyone else that has asked questions about statements in the Information Supplements the following.

“Information Supplements only offer guidance to organizations and do not replace or supplant anything stated in the PCI DSS.”

So what are the point then of the Information Supplements?

Boy is that a good question. As a QSA, I often ask myself that very question after some of the inane conversations with clients and prospective clients regarding Information Supplements and their supposed “guidance”.

The first thing everyone should remember about Information Supplements is that they are developed and written by a committee at the suggestion of the Council, POs or as part of special interest work groups. These committees are made up of personnel from interested POs, QSAs, ISAs, vendors and anyone else willing to participate in their development. They are edited by a representative from the Council and reviewed by the Committee and are then submitted to all POs, QSAs and ISAs for review and comment. Similar in concept to the development and review of RFCs by the IETF.

The other key point about Information Supplements are that they are developed to give QSAs, ISAs and organizations ideas and guidance on how best to appropriately meet the requirements of the PCI DSS and the Reporting Template testing. Again, as the Council has repeatedly stated, the Information Supplements do not replace the explicit guidance and testing requirements in the PCI DSS and the Reporting Template. They are merely suggests on an approach.

Yet time and again, QSAs and ISAs get these priceless documents tossed in our faces and are told we do not know what we are talking about. “The Information Supplement says …” is always put out there as the justification as to why an organization is doing something it should not be doing or as the rationale for why the organization is not in compliance with the PCI DSS. And we again are forced to explain that the Council never has said that an Information Supplement replaces the guidance and testing in the PCI DSS or the Reporting Template.

The first question anyone, and I do mean anyone, should ask about any statement in an Information Supplement is, “Does the PCI DSS and/or the Reporting Template explicitly say the same thing?” Those are the only two documents that matter and the only documents that your organization will be assessed against. If it is not explicitly called out in either of those documents, then it is not accurate and does not reflect the compliance requirements.

As an example. I was on a conference call recently regarding the Council’s Information Supplement on penetration testing. This supplement was issued in March, 2015 and is possibly one of the most confusing and contradictory pieces of “guidance” we have ever encountered. In fact, it has created more confusion than it has actually clarified. In my very humble opinion, the Council would be better off taking it out of circulation because of all of the trouble it creates for QSAs, penetration testers, ASVs and clients. It is possibly one of the worst written of the Information Supplements and, while people both on the Committee that developed it and externally supplied the Council with numerous suggestions for changes, those changes were not incorporated into the document. Why those changes were not incorporated is anyone’s guess. But we in the PCI community ended up with possibly the worst expressed and misunderstood guidance available.

As usual, the client was arguing over the scope of their penetration testing. I get the fact that organizations want to minimize costs and scope as much as possible. However when you listen to some security professionals arguments on this topic, you just wonder how they got to their positions as they argue over not testing systems and devices that are painfully obvious to be in scope.

And as also is usual, the first piece of confusion regarding scope is in Section 2, page 5, first paragraph after the bullets and states the following.

“It is not a requirement to test from within the CDE to the servers inside the CDE; and testing exclusively from within the CDE perimeter will not satisfy the requirement. However, when access to the CDE is obtained as a result of the testing, the penetration tester may elect to continue exploring inside the network and further the attack against other systems within the CDE, and may also include testing any data-exfiltration prevention (data-loss prevention) controls that are in place.”

One would think that to any reasonably intelligent information security professional, the first part of the sentence, “It is not a requirement to test from within the CDE to the servers inside the CDE;” would be considered a pure line of garbage. Never mind that none of the recognized penetration testing methodologies ever suggest such an approach. But people arguing never consider that fact. Nope. The people arguing are so focused on cutting their PCI compliance bill that it does not matter that the statement is pure and unsupported garbage. It is considered the gospel truth. Otherwise, why would the Council allow such a statement? Good question. We have asked the Council that question and the answer back is? You guessed it.

“Information Supplements only offer guidance to organizations and do not replace or supplant anything stated in the PCI DSS.”

Again, never mind it is in no way supported by the guidance provided by the PCI DSS for requirement 11.3 which says:

“The intent of a penetration test is to simulate a real-world attack situation with a goal of identifying how far an attacker would be able to penetrate into an environment. This allows an entity to gain a better understanding of their potential exposure and develop a strategy to defend against attacks.”

But argue that point they do even when you point out that arguing this point is basically arguing that any attacker would stop at the perimeter of the CDE and would go no further.

Seriously? If you believe that fact, you must also believe in Santa Claus, the Easter Bunny, the Tooth Fairy and any other of the multitude of mythical fictional creatures. Or you are just lying to yourself and are in serious denial about your organization’s security posture. But argue on they do.

Then you pair that to the second part of that first sentence of this paragraph that says, “… and testing exclusively from within the CDE perimeter will not satisfy the requirement.” Just adds to the out of scope argument.

As I point out when bitch slapped with this terrible writing, if you go back and carefully re-read the second part of the first sentence, what it points out is that penetration testing from only inside the CDE is not sufficient to meet the penetration testing requirements of the PCI DSS requirement 11.3. In no way does that sentence say or even further imply that the CDE is out of scope. It is actually saying that penetration testing should be done from within the CDE, but that penetration testing only inside the CDE does not meet 11.3. But people will still argue that the CDE is out of scope.

That the CDE is in scope is further supported by the definitions of “critical systems” from section 2.2.1 of the document which defines that not only are systems within the CDE in scope, but also those that are outside the CDE but could affect the security of those systems inside the CDE (i.e., what the Council and the Open PCI DSS Scoping Toolkit refer to as “connected to” systems). However, people arguing over scope rarely, if ever, tie these two section together and then argue that because they are in separate sections they cannot be possibly together even though the entire document is about only one subject, penetration testing and requirements in 11.3 of the PCI DSS.

So before you go off telling your QSA or ISA that the Information Supplement says something. Think about what the information supplement says. Is the guidance from the Information Supplement even implied in the PCI DSS? Read the guidance in the PCI DSS and the testing procedures from the Reporting Template. If the PCI DSS or the Reporting Template do not explicitly have the same language in them that the Information Supplement has, then the Information Supplement is merely a suggestion.

And if the guidance from the Information Supplement does not make sense, pull your head out of your posterior and use some God given common sense. Ask your QSA or ISA to explain it, before going off halfcocked and thinking that someone could actually think such things made sense.

But again, why would the Council allow such statements? Good question. We have asked the Council that question and the answer back is? You guessed it.

“Information Supplements only offer guidance to organizations and do not replace or supplant anything stated in the PCI DSS.”

Clear as mud? You bet.

But what did you expect? It is PCI.

For all of you in the United States, have a happy and safe Thanksgiving holiday.


Compensating Control Refresher

From time to time, organizations find themselves in the predicament of not being able to meet a PCI DSS requirement due to business or technical constraints. To address that situation, the PCI SSC has provided the compensating control worksheet (CCW) as a way to work around those requirements that cannot be met directly as stated in the PCI DSS. When the CCW was updated back in 2010 for v1.2, I wrote about those changes and how to write a CCW. However, here we are at v3.1, five years down the road and I still see a lot of poorly and improperly written CCWs. As a result, I think it is time to take people through a refresher on the CCW.

First and foremost, the writing of any CCW is your organization’s responsibility. Your QSA can provide input and guidance, but the origination of the CCW is up to the organization. Once developed, your QSA can review it and make suggestions to enhance and improve the CCW. Once that has been completed, you will then want your acquiring bank to review it to ensure that they will accept it as part of your self-assessment questionnaire (SAQ) or Report On Compliance (ROC) filing.

Secondly, the format of the CCW is dictated by the Council and that format is provided in Appendix B of the SAQ D or in Appendix C of the ROC. Failure to use the proper format will create issues with your QSA, your bank and with the Council, particularly if you are doing a ROC. So please use the Council supplied format and not develop something on your own.

Finally, the PCI SSC has stated that any requirement can have a CCW. In the past, the Council instructed QSAs and ISAs that requirement 3.2 [Do not store sensitive authentication data after authorization (even if encrypted). If sensitive authentication data is received, render all data unrecoverable upon completion of the authorization process] was not allowed to have a CCW. At the 2014 Community Meeting, the Council backed away from that restriction and said that any requirement can have a CCW with no restrictions. However, as a QSA I would have a serious problem accepting a CCW for requirement 3.2 because storing sensitive authentication data (SAD) is the whole reason why the PCI DSS was created to stop.

To remind everyone, the CCW is broken into seven sections.

  • Identification of the PCI DSS requirement(s) being compensated.
  • The constraint or business justification for needing the CCW.
  • The original objective of the requirement(s) being compensated.
  • Identification of any additional risks because of the CCW
  • The compensating controls.
  • The procedures your QSA/ISA followed to confirm that the compensating controls are in place and functioning.
  • The procedures followed by your organization to maintain the compensating controls.

While the Council tells everyone to have an individual compensating control for each requirement, there are some places where a compensating control is the same for a number of requirements. This most often occurs for requirements in section 8 around the various user management requirements or 6.1, 2.2, 11.2 and the processes of vulnerability management. I would highly recommend using one CCW per requirement, but I can understand why you might combine some. Just be judicial in combining them. Also, list not only the number of the requirement(s), but also the text of the requirement from the Requirements column in the PCI DSS. While your QSA might have memorized the PCI DSS requirements, bankers and others that will read the CCW have typically not committed to that level of detail and it will help them with the context of the CCW.

The business justification needs to be more than just “we don’t want to” or “it was too hard”. Believe it or not, I have had a lot of organizations provide just such simplistic and silly reasons for justifying a CCW. Proper justifications can involve budgetary constraints, timing (e.g., not enough time to complete remediation by the end of the assessment period), application requirements (e.g., the application requires XP to run) and/or vendor requirements (e.g., the vendor requires a hardware upgrade to correct the issue). If you do have a target date for addressing the CCW, this is where you want to provide that information so that readers know that the CCW has some time limit.

The original objective is the easiest part of the CCW to develop. The Council has provided the “Guidance” column in the PCI DSS for each requirement and it is the verbiage in that Guidance column that you should use to explain the original objective of the requirement. If you are using the CCW for multiple requirements, this section can get rather lengthy and I would recommend identifying the Guidance information with its requirement to help understanding of the information.

The next section can sometimes be the toughest to develop and that is identification of any additional risks because you are using a CCW. In some cases, there may actually be no additional risk perceived by using a CCW. One such example is when organizations have a separate system management VLAN where network and system administrators can use telnet, SNMPv2 and other “unsecure” protocols in addition to SSH, RDP and other secure protocols to manage devices/systems. These system management VLANs typically require the use of an out of band (OOB) to gain access, administrator credentials different from the administrator’s user credentials and two factor authentication to name just a few of the controls you see in this example. These management/administrative VLANs are no more risky than using only secure protocols.

However, if you are compensating for having to keep Windows XP running, that will likely be a very different story and depending on the compensating controls put in place, the risk could be moderately higher than not have XP around. The key here is that it is that the risk should be assessed and then honestly discussed in the CCW. If you think you are going to say that having XP does not increase risk to your cardholder data environment (CDE), I would seriously think again regardless of your compensating controls in place because any outdated Windows implementation is a security problem waiting to happen regardless of how you think you have mitigated the risk.

The compensating controls section is where the rubber finally meets the road. It is here that you document each individual control that compensates for your organization’s inability to meet the requirement(s) in question. I recommend that people either bullet point or number list each individual control. The reason is that in the next two sections, you need to tie the validation and maintenance items to the controls in this section and doing some sort of list makes it easy for people to ensure they have covered all controls in each section.

The most common mistake made in this section is organizations state that they have a project to remediate the issue(s). Sorry, but this is NOT a control. It is nice information, but it is not a control that can be relied upon. QSAs never want to ever see such statements made about future projects ever in this section. This section is all about what you are doing from a controls perspective to manage the fact that you cannot meet the requirement(s).

Valid controls in this section must also go “above and beyond” what is required by the PCI DSS. Examples of “above and beyond” include:

  • Reviewing log data in real time for a particular condition that would indicate an out of compliance condition on a control. This is above and beyond because log data only needs to be reviewed daily for such conditions.
  • Using whitelisting to identify applications that do not belong on a PC and generating an alert in real time if such applications are found. Such technology is above and beyond because it is not currently required by the PCI DSS.
  • Using critical file monitoring to identify rogue applications that do not belong on a PC and generating alerts in real time if found. Critical file monitoring is a PCI requirement, but this goes above and beyond because monitoring is only required on a weekly basis.

The list here can go on and on, but hopefully I have given you some ideas of how to create compensating controls that can actually compensate for your inability to comply with the requirement(s).

One key point though is that you cannot use a requirement in the same requirement group to compensate for a different requirement in the same group. For example, requirement 6.4 has bunches of sub-requirements under it. You cannot write a compensating control for one sub-requirement in 6.4 and then use a different sub-requirement under 6.4 as one of your compensating controls regardless if it is above and beyond.

The next section will list how the controls were assessed by your QSA/ISA to prove they have been implemented. So using our previous bullet list, here is what the control validation bullets would look like.

  • Observed the system information event management (SIEM) solution and verified that alerts are generated in near real time for [control failure condition] and that the alert is followed up by the security analyst to determine if the alert is valid. If valid, the security analyst opens a service ticket and assigns that ticket to the appropriate area for further investigation.
  • Observed the [whitelisting solution name] and verified that if rogue applications are loaded on a workstation a near real time alert is generated back to the [whitelisting solution name] master console and that the alert is followed up by the security analyst to determine if the alert is valid. If valid, the security analyst opens a service ticket and assigns that ticket to the appropriate area for further investigation.
  • Observed the [critical file monitoring solution name] and verified that if rogue applications are loaded on a workstation a near real time alert is generated back to the [critical file monitoring solution name] master console and that the alert is followed up by the security analyst to determine if the alert is valid. If valid, the security analyst opens a service ticket and assigns that ticket to the appropriate area for further investigation.

Finally, you need to document what your organization will do to ensure that the controls remain implemented and effective. This is where most compensating controls fall apart. The organization gets through their assessment and then neglects to keep the compensating controls working. Using our list from the compensating controls section, the maintenance controls would look something like this.

  • [Organization name] reviews on a [weekly/monthly/quarterly] basis the SIEM and test that the alerts for the [control failure condition] are still functioning as designed.
  • [Organization name] reviews on a [weekly/monthly/quarterly] basis the [whitelisting solution name] and test that the alerts for rogue applications are still functioning as designed.
  • [Organization name] reviews on a [weekly/monthly/quarterly] basis the [critical file monitoring solution name] and test that the alerts for rogue applications are still functioning as designed.

A good idea in the maintenance section is to set timeframes for remediating any control testing failures.

One other important item of note about the controls, validation and maintenance lists. Notice that there are no “forward looking” statements made such as someone “will” perform or “will” review. CCWs must be shown to be in place and operating. A promise of implementing a control is NOT a control either. The control must be shown to be operating and maintained. That is an important point a lot of organization miss. It means that CCWs cannot be created at the last minute and then be operational past the filing of your SAQ or ROC. If you are going to have to use a CCW, that means you will need to identify the situation early and then get the compensating controls implemented, validated and through at least one maintenance cycle before it can be accepted.

CCWs can buy organizations time while they address issues that will take longer to address than their PCI assessment period. Unfortunately, there are organizations that see the CCW as a way to be judged PCI compliant without addressing their serious security shortcomings. It is not unusual for large organizations to have a number of CCWs particularly if they have legacy applications and hardware. However, I would highly recommend that all organizations only rely on CCWs if there are no other options to achieving PCI compliance.


PCI Issues Penetration Testing Information Supplement

On Thursday, March 26, the PCI SSC issued their latest information supplement. This one is focused on the topic of penetration testing. Not surprisingly, I have a number of comments based on what I have read. There are some good things that have come out of this effort. But in my opinion, the committee missed the boat in a few areas and those will lead to even more confusion between penetration testers, QSAs and clients.


Talk about making a mess of a mess, I believe the writers accomplished that amazingly well in their discussion on scoping. This is probably the area that creates the most confusion and will likely create the most problems.

On page 5 and the first bullet, talk about confusing. It took me a number of readings to even get what I think they are trying to communicate.

“The scope of the internal penetration test is the internal perimeter of the CDE from the perspective of any out-of-scope LAN segment that has access to a unique type of attack on the CDE perimeter. Critical systems or those systems that may impact the security of the CDE should also be included in the scope. Testing must include both application-layer and network-layer assessments.”

Out of scope LAN segments are out of scope because there are no rules that allow them access to the CDE. I agree that you need to test that these LAN segments are truly out of scope, but if there is a unique type of attack available from such a LAN segment into the CDE, that would seem to imply that the LAN segment was not out of scope. Confusing and likely to cause a lot of pen testers to force a lot of unnecessary testing and fishing for bizarre issues on out of scope network segments.

The next sentence paired with the first seems to just add to the confusion because it implies again that the out of scope LANs are not out of scope. These should have been two separate bullet points in my opinion.

In the end, what I think the writers are saying is that even if deemed out of scope, all LANs and VLANs need to be confirmed to be out of scope. However, how they have said that in my opinion leads to a lot of confusion.

If the last one did not cause your head to spin around, this one probably will. It is the third bullet at the top of page 5.

“To be considered out of scope for PCI DSS, a system component must be isolated (segmented) from the CDE, such that even if the out-of-scope system component was compromised it could not impact the security of the CDE. Therefore, the penetration test may include systems not directly related to the processing, transmission or storage of cardholder data to ensure these assets, if compromised, could not impact the security of the CDE.”

Correct me if I’m wrong, but this seems to imply that everything is in scope for penetration testing. I know the writers of this supplement have used the word ‘MAY’, but we also know how ‘MAY’ tends to be interpreted by the Council (i.e., MUST) and how it will be interpreted by a lot of our clients (i.e., do not do it). Until a few penetration testing reports are reviewed by the Council, the verdict will be out on the correct approach. However, coupled with the theme created by these bullets, I am guessing that the committee could not agree on how to express how to scope the penetration test.

But if those bullets were not confusing enough on scoping, the writers add insult to injury. After the bulleted list at the top of page 5, the first paragraph states the following.

“It is not a requirement to test from within the CDE to the servers inside the CDE; and testing exclusively from within the CDE perimeter will not satisfy the requirement. However, when access to the CDE is obtained as a result of the testing, the penetration tester may elect to continue exploring inside the network and further the attack against other systems within the CDE, and may also include testing any data-exfiltration prevention (data-loss prevention) controls that are in place.”

An internal pen test does not require testing from within the cardholder data environment (CDE)? This sounds like the “Three Hop Rule” all over again. But then we twist it further somehow justifying the first sentence.

All this diatribe does is just further obfuscates scoping. What simply should have been said is that everything in-scope for PCI compliance is to be penetration tested. Instead, we got a lot of words that, in my very humble opinion, do nothing to clarify scoping.


On page 6 under 2.3.2 PA-DSS Compliant Applications we get the following guidance.

“If a payment application has been PA-DSS validated, the application’s functionality does not need to be tested as part of the entity’s PCI DSS compliance validation. However, the implementation of the application does need to be tested. This includes both the operating system and any exposed services, but not the payment application’s functionality (e.g., authentication, key management, transaction processing, etc.) since this was validated as part of the PA-DSS application validation.”

Under the PCI DSS, even PA-DSS validated applications still must be tested to prove that their implementations were completed per the vendor’s PA-DSS implementation guide. Yet for penetration testing, PA-DSS validated applications, their operating systems and exposed services are exempt?

I hope the acquiring banks are geared up for the fights this is going to cause between the penetration testers and their clients. For example, in the restaurant and hospitality industries, I can see the situation where penetration testing only occurs on firewalls, routers and switches because everything else is PA-DSS validated.

I do not believe this is what the Council wanted, but that is what this supplement is stating. If we are truly talking about security of our networks and systems, this is not how you achieve it.

Social Engineering

I was pleasantly surprised that this was included in this supplement. However, as I read through this section, I got more concerned that it was even included.

For section 2.5 Social Engineering on page 7, we encounter the following.

“Social engineering is the attempt to gain information, access, or introduce unauthorized software into the environment through the manipulation of end users. PCI DSS v3.0 reconfirms testing by requiring industry accepted penetration testing approaches (many of which include social engineering as part of their approach) and to have an approach to penetration testing that “considers the threats and vulnerabilities experienced by merchants in the last 12 months.” This may include social-engineering attacks as a method used for introducing malware into the environment.”

This makes social engineering attacks legitimate if the client has been breached by such methods in the past year. But does this include physical as well as logical social engineering? Since it is not explicitly called out, I would assume so, but it should have been defined. Regardless of that being explicitly called out, this is a great thing to see in this document.

On page 8, the last paragraph of the discussion on social engineering states:

“Social-engineering testing may not be appropriate or provide a meaningful result for all organizations. Although social-engineering testing is not a requirement of PCI DSS, an organization may consider documenting the reason(s) for foregoing social-engineering testing and include applicable documentation with the internal and external penetration test reports, particularly if social-engineering attacks were encountered in the last 12 months.”

What started with such promise ends on a very sour note. This last paragraph gives organizations an out. So I guess we can look forward to a lot of organizations issuing inventive explanations to avoid social engineering testing.

Some Miscellaneous Issues

On page 2, the seventh bullet down in the Terminology we have the following.

National Vulnerability Database (NVD): The U.S. government repository of standards based vulnerability management data. This data enables automation of vulnerability management, security measurement, and compliance (e.g., FISMA).”

I know they just grabbed the description from the National Vulnerability Database (NVD) Web site, but that does not mean that it is appropriate for use when discussing penetration testing. The Common Vulnerabilities and Exposures (CVE) database maintained by Mitre Corporation would have been the better reference since the NVD uses the CVE as the catalog of vulnerabilities.

I have no idea how the Federal Information Systems Management Act (FISMA) ties into the NVD. If they had truly pulled the content exactly from the NVD Web site they would have grabbed the fact that it ties to the Security Content Automation Protocol (SCAP) not FISMA.

My last comment on this topic is regarding the fact that the NVD (or the CVE) enable automation of vulnerability management. Had they brought in the reference to SCAP, I might have had an easier time going along with things. The NVD/CVE do not contain the actual exploits, they are merely a catalog of the vulnerabilities. As a result, I question the value of either the NVD or the CVE as to it automating vulnerability management. Yes, the CVE catalogs all of the vulnerabilities, but that is the value. In my opinion, the use of either of these in regards to the automation of the vulnerability management process is a stretch. They are a foundation that will allow for automation, but they do not in and of themselves actually enable automation.

Under 2.3.3 Web Applications on page 6, we have the following.

“It is common for an environment to host a web application that was not specifically coded for the organization such as commercial, off-the-shelf web-mail interfaces, document-sharing tools, file-transfer services, network-device administrative interfaces, etc. In these instances, the web application does not typically need an application-layer penetration test as the entity is not responsible for the source code of this type of software. Instead, the tester should perform a network-layer test and ensure the software was implemented, configured, and is currently being maintained in a secure manner (disabling or uninstalling unused services, blocking unused ports, applying current updates, etc.).”

I can see it now, organizations arguing over the pen testing of IBM Websphere, Oracle eCommerce or similar Web application frameworks because they are a package. It is a framework that you then must write an application for, but penetration testers will get into arguments over that aspect.

A Few Good Things

Yes, I have complaints, but there are some good things that are contained in this supplement.

Probably the best thing about this information supplement is the checklist at the end of the document for evaluating penetration testing reports. The Council should have a similar item for vulnerability scanning reports as well.

The discussion of the penetration testing methodology in section 4 of the document is a good read for both penetration testers and the people that contract for penetration testing services. But keep in mind it is a framework and not the entire rule/procedure set for conducting a proper penetration test. Yes, there are some good details in areas, but it is not the detailed roadmap that some people will portray it.

Section 3 has a good discussion on how to evaluate the qualifications of a penetration tester. It was nice to see that while they document the common certifications a penetration tester may hold, they caution readers that certifications alone does not make a penetration tester qualified.

The case studies in section 6 probably do more to clarify than the previous five sections. They are good examples of properly conducted penetration tests.

Finally, there is a good discussion on the definition of a “significant change”. However, I think this discussion as well as a discussion about the words “periodic” and “periodically” should be put into a discussion about risk management and assessment, rather than brought up here. I wrote a post on this topic a while back to provide some guidance to people and it was nice to see that my thoughts jive with what they wrote in the supplement.

What Was Missed

The biggest item I think that was missed was how to handle the penetration testing of huge environments. Case study 6.3 discusses the concept of sampling somewhat, but it deals with a retailer with only six locations, not thousands of locations. There are also veiled references to sampling in section 2.3.4 in discussing a separate testing environment. But there is no deliberate and complete discussion on this very important topic.

Retailers such as Wal-Mart, Target and Home Depot have more than a thousand locations with 15+ checkout lanes meaning there are, at a minimum, 15K devices out in the field that technically need to be penetration tested. How does an organization penetration test such a large number of devices annually? The answer is that they do not test them all. It is physically impossible. However, this information supplement does not address that situation and provide guidance as to what the Council deems as acceptable.

While I use some of the largest retailers in the world as my example, it is not just them as there are mid-sized retailers as well as hospitality organizations that face this same issue. I had hoped that how to approach these situations would have been explicitly addressed in greater detail but it was not.

The writers also missed a huge topic regarding social engineering testing. Unlike vulnerability scanning and penetration testing, in my experience, it is virtually impossible for any organization to “pass” social engineering testing. So what would be considered “passing” in a social engineering test? That is not discussed but should have been. Since this document broaches the subject, it is probably a good topic to propose for a SIG for this year’s SIG recommendations so we get social engineering covered in an information supplement rather than as a sidebar to penetration testing.

It is not that this information supplement is an entirely bad document. But the advice it provides needs to be taken with a bit of caution. As usual, the shortcomings will be leveraged by all of the “check boxers” to do as little as possible so that you get that box checked for 11.3.


The Three Hop Rule

At the 2014 Community Meeting, the PCI SSC responded to a question about network segmentation with what has come to be termed the “Three Hop Rule”.  The statement was made that if a device/system was “three hops or more” away from the cardholder data environment (CDE), then it was out of scope.  A lot of us in the room were taken aback by this statement.  And based on some questions of late regarding this subject, there is a lot of confusion out there regarding what the Council was trying to say.

First, the term “hop” is not a network security term nor does it even have any security implications.  The term “hop” is defined as:

“Data packets pass through routers and gateways on the way.  Each time packets are passed to the next device, a hop occurs.”

The count of three therefore is the number of hops or “hop count” between devices.  Hop count is defined as:

“Each router along the data path constitutes a hop, as the data is moved from one Layer 3 network to another.  Hop count is therefore a basic measurement of distance in a network.”

Nowhere in these definitions is there any statement about hops, the number of hops between devices and any correlation of hops and hop count as some form of security.  Hence why a lot of us were really concerned about this statement and likely why there is so much confusion and discussion resulting from the comment.

What we believe the Council was getting at was the number of network segments there are between a device/system and the CDE.  However, having three network layers between the CDE and devices/systems is also no guarantee of security.

What provides security at Layer 3 are the access control lists (ACL) or rules that allow or deny packets to traverse particular paths of the network.  ACLs can be implemented to control what devices and/or ports and services can communicate between various networks.  But just because there are ACLs implemented at each hop is also no guarantee that the number of hops between devices also secure the devices.

This is why the requirements in requirement 1 of the PCI DSS require that the QSA review all relevant ACLs to ensure that the network is truly segmented.  It is also why in v3, requirement 11.3 requires that the penetration testing also prove that the network is truly segmented.  As a result, the number of hops between the CDE and a device should not be considered a guarantee and never will be a guarantee that a device is out of scope.

The bottom line is that, in order to be truly out of scope, there needs to be ZERO hops between a device and the CDE.


Forensic Examinations And Facts

I am watching the news reports on the Sony breach and laughing at all of the “facts” that are being bandied about.  I want to use the Sony breach as a teachable moment and explain that the “facts” may not be as factual as represented by the media, forensic examiners or even the FBI.  I have done a number of forensic investigations and from my own experience there is a lot of effort required to prove conclusively that a particular device or actor is the actual attacker.

So let us take a look at the “evidence” we have at this point and see if the conclusions drawn should be treated as facts.

My first issue is how quickly the FBI and Mandiant have come out with the “fact” that North Korea is behind the attack.  According to the timelines I have seen, it was on November 21. 2014 when Sony was told by the attackers, GOP, that Sony had been hacked.  So in around three weeks of time the FBI and Mandiant have figured out, definitively, it was North Korea that was behind the attack.  Granted, Mandiant and the Bureau could have been investigating this long before, but given the way the news reports were written, I have to believe that Sony had no idea anything was wrong until November 21.

Why do I find this timeline spurious?  It took Mandiant over three years to trace things back to the Chinese for their report, APT1, last year and we are to believe that the FBI has the skill and manpower to trace a “sophisticated attack” (Kevin Mandia’s words to Sony) back to North Korea?  I find that hard to believe.  Not because the Bureau and Mandiant are not skilled, but that it is just impossible to cram a year’s worth of investigation into a few weeks, regardless of the manpower tossed at the investigation.

In my own experience, I typically had ideas as to what and how things happened within a few weeks, but now the difficult work of determining exactly how things went down began.  It can take months or even years to figure out an attack if it is ever figured out.  It is why NTSB investigations of airplane crashes take at least a year to have a report issued.  Any attack may not be as simple or uncomplicated as you initially think.

“Technical analysis of the data deletion malware used in this attack revealed links to other malware that the FBI knows North Korean actors previously developed. For example, there were similarities in specific lines of code, encryption algorithms, data deletion methods, and compromised networks.”

We do know for a fact that hackers reuse other attackers’ code.  Why reinvent the wheel if you do not need to?  Hence the variants of all of the attack code to not only evade anti-virus but to also enhance or improve techniques and methods.  Just because there are similarities in some lines of code, algorithms, methods, etc., does not mean that it was the North Koreans that were the actual actors.  It just means that the attackers used code attributed to North Korea.  Key word, “attributed”.  To me, a far better piece of evidence would have been if the code had been written in Korean or a North Korean dialect.

“The FBI also observed significant overlap between the infrastructure used in this attack and other malicious cyber activity the U.S. government has previously linked directly to North Korea. For example, the FBI discovered that several Internet protocol (IP) addresses associated with known North Korean infrastructure communicated with IP addresses that were hardcoded into the data deletion malware used in this attack.”

Hard coded IP addresses are evidence?  So does that mean that everyone is guilty if I write their telephone number on a napkin and that turns up as evidence?  No.  A better piece of evidence would have been log data that actually can tie those IP addresses to the data that was exfiltrated out of Sony.  Just because IP addresses are hardcoded in an application does not necessarily imply that the IP end point was in fact the actual endpoint.  Hackers regularly own other organizations’ and governments’ servers to obfuscate their actual location.  Just because there’s a hardcoded IP address in a piece of code does not necessarily mean that is the endpoint.  It just means that a device could be involved.

“Separately, the tools used in the SPE attack have similarities to a cyber attack in March of last year against South Korean banks and media outlets, which was carried out by North Korea.”

The attack on certain South Korean banks and TV stations in 2013 was never definitively pinned on North Korea, it was just suspected.  The prime piece of evidence was a Chinese IP address that was assumed to implicate North Korea.  So using the South Korean attack as though it was definitively proved to be done by North Korea is not a fact.

While I had some issues with the Mandiant report on China and their investigation methods, the information being offered as “facts” that North Korea is behind the Sony breach are positively appalling.  People want an answer immediately and so one is given regardless of accuracy or even believability.  However, this is a technology issue and so it is easy to feed the public supposed “facts” since only the true technology people in the world will know the difference.

Unfortunately a breach such as the one at Sony will take time, probably a lot of time.  I would not be surprised if we end up with a lot of “suspicions” and “assumptions” when a final analysis is done and released, if we ever get a definitive answer.  The reason I believe that is that I do not think Sony had the kind of security implemented and working given the amount of information that has been supposedly gathered by the attackers.  The other clue in this is that it was November 21 when Sony was notified by the attackers they had been breached.

The key take away here is that forensic examinations very rarely prove WHO the bad actor was that caused the breach.  This is particularly true when the attacker is outside the organization.  There are just too many ways that an attacker can obfuscate their actual identity/location.

What forensic examinations do provide is a road map of improvements and enhancements in an organization’s security measures and procedures to minimize future attacks.  Note that I did not say “prevent” future attacks.  I use minimize because security is never an absolute.  Anyone with an extreme desire to attack an organization will do so regardless of how well your security program is constructed and executed.

Bruce Schneier points out this very fact about determined attackers in his post on the Sony breach.  I have always referred to this as the ‘98-2 Rule’.  Properly implemented and managed information security keeps 98% of attackers out.  However it is the remaining 2% that are determined enough to figure out how to work around even the best security.  All any organizations can do about that remaining 2% is to put controls in place so that when the 2% get through, they are detected as soon as possible and their impact minimized.  This is why security frameworks are so important because they provide organizations with guidance as to what it does take to only have the 2% to worry about.

Given the limited evidence provided thus far, could it be that this is all a sophisticated marketing ruse that went sideways?  Would it not be apropos if Seth Rogen and his production company did the attack as a promotional stunt and the attackers they hired found out that Sony was ripe for such an attack and then went further than what they were supposed to?

Something to think about.


The ASV Process Is Broken – Part 3

So what are my ideas on fixing the ASV process?

Modify The ASV Program

The conditions that drove the ASV process originally made sense.  Vulnerability scanning tools were predominately open source and anyone could do scanning and just about anyone was doing vulnerability scanning.  The results produced out of the open source tools could be highly questionable at best and the reporting was haphazard and about as trustworthy at times as a three dollar bill.  Even in large organizations, the people doing the vulnerability scanning did not necessarily have networking, security or even IT backgrounds.  Then there was a tremendously high false positive rate out of the open source tools.  As a result, most organizations ignored the results they received because they found that they could not be trusted.

The purpose of the ASV program was to bring some sanity and professionalism to the vulnerability scanning process.  MasterCard invented the ASV program (it was not called ASV then) back in 2005.  A test network was built and prospective ASVs were required to run their vulnerability scanners against this network and produce results which were then reviewed by MasterCard.  It was a much a test of the vulnerability scanning tool as it was of the person running the tool.  When the program transitioned to the PCI SSC, the Council added a multiple choice test to the process, but the virtual network testing and report review is still part of the process.

The trouble with this process is that the vulnerability scanning tool is no longer the problem.  Every ASV uses a commercial vulnerability scanning tool from either Tenable, Qualys, Saint, Tripwire or similar commercial tool vendor these days because they cannot afford to do otherwise.  Since these tool vendors are also ASVs, requiring a vulnerability scan for ASV certification has become a truly pointless exercise.  Other than the possibility of not properly entering the IP addresses to be scanned and running the wrong scanning policy, there really is very little that someone can screw up with a scanning tool.

The skill in vulnerability scanning today is reviewing the results, dealing with false positive results, working to address results with compensating controls and, with the Councils new edict on combining reports, working to get passing quarterly scans.

Therefore, in my opinion, training and testing of ASVs should be focused on the following.

  •  Determining the scope of vulnerability scanning.
  • Vulnerability scanning methodology.
  • Interpreting vulnerability scanning reports to confirm knowledge of the process and the meaning of the results.
  • What constitutes a false positive result and how to document a false positive result.
  • Development and documentation of an appropriate compensating control for a vulnerability.
  • Process for how to produce an acceptable passing scanning report from multiple reports.

And let us not limit ASV certification to just independent consulting firms.  As with the internal security assessor (ISA) program, open the ASV program to internal personnel as well.  Most large companies have independent vulnerability scanning teams that are as capable to more than capable than their ASV brethren.  There is no longer any reason that these internal people cannot do the ASV scans particularly if they meet the same standards and qualifications.

Approved Vulnerability Scanning Tools

I am not suggesting that the Council needs to develop a certification process for these tools as there are already plenty of sources that assess such tools.

The Council would publish a list based on the criteria developed by one or more independent tool assessment sources.  This list would define those tools acceptable to use for ASV vulnerability scanning.  The PCI DSS should then require that the QSA confirm that the vulnerability scanner used by the ASV is on the list in addition to confirming scope and the scanning policy used.

Require A Vulnerability Scanning Methodology

With the PCI DSS v3, the Council now requires penetration testers to use a documented and industry accepted penetration testing methodology.  Yet, there is no such requirement for vulnerability testing.

Most vulnerability scanning is done using what I call the “toss it against the wall and see what sticks” approach.  Basically, every possible vulnerability is run against every device.  Most commercial vulnerability scanners interpret banners, signatures and other markers to trim the list of vulnerabilities to be tested based on what they believe the target to be.  However, when you are scanning an external network blind, scanners cannot always properly interpret what an IP address resolves to as a device because of the mix of responses that they receive.  As a result, scanners do not necessary trim tests increasing false positive results or they trim them too much and the test is not complete.

Then there is the automated nature of today’s vulnerability scanning.  While I understand the desire to reduce costs of vulnerability scanning, the “point and click” nature of today’s ASV scanning has made it flawed.  And it gets worse as organizations get passing scans.  As a QSA, I cannot tell you how many passing scans I have reviewed where an organization could be hacked six ways to Sunday with the remaining vulnerabilities.  As a security professional, it scares me to death.  But as a QSA, while I can bring these up, they get no play because they do not have a CVSS of 4.0 or greater.  You hope that these vulnerabilities get picked up in an organization’s penetration test.

But there is no guarantee of that happening because the penetration tester’s vulnerability scanner may or may not pick up the same vulnerabilities.  As a result, part of the penetration testing methodology should include a review of all vulnerabilities found since the last penetration test and those should be tested for in the current penetration test to ensure they have been addressed.

Obviously, I have a preference to the methodology I discussed back in Part 2.  But there are a number of methodologies posted out on the Internet from a variety of good sources.  All I ask is that the vulnerability scanning methodology be integrated with the penetration testing methodology so that there are not gaps in coverage.

Require Monthly External Vulnerability Scanning

Before everyone panics, I am not asking that ASV scans be run monthly.  Although if the ASV program is modified, for organizations with internal ASVs that is a possibility.  I would still require the quarterly ASV scan, but I would add in monthly scans run by anyone deemed qualified as is allowed for internal vulnerability scans.

My primary rationale for this recommendation is driven by this simple fact.  When the dominant solution vendor releases patches on the second Tuesday of every month and the vast majority of those fixes have a CVSS score of 4.0 or greater, anyone that thinks quarterly scanning keeps them secure is seriously kidding themselves.  Not that a lot of security professionals bought into the quarterly vulnerability scanning requirement, even as a bare minimum.  But without the standard requiring it, a QSA has no leg to stand on other than to intimidate and shame people into doing monthly scanning.

Even if you are not Microsoft centric in your external environment, with the breaches that have occurred and the revelations of Shellshock and Poodle, it is painfully obvious that the quarterly requirement is not going to keep organizations secure.  I got a lot of calls after both of these vulnerabilities were announced with clients asking if their passing scans were no longer valid.  I was a bit schizophrenic in my thoughts.  On the one hand, I was glad they were at least thinking about the security implications of these vulnerabilities.  But their concern about their passing scans just highlights the importance of meeting a PCI requirement and passing their PCI assessment versus being secure.  Because, while I only got a few calls, you know that there are too many people that are congratulating themselves on dodging the bullets of Shellshock and Poodle because of the fortuitous timing of their  quarterly scans and that they got an additional 30, 60 or even 90 days to address them.

Then there are those organizations that run solutions such as IBM’s Websphere or Oracle’s eCommerce suites.  Both of these vendors not only patch their own application frameworks, but they also release those patches to the underlying operating systems that are compatible with their application frameworks.  But worse, these vendors do not release monthly patch releases, they do patch releases on quarterly, semi-annual or even annual bases.  As a result, there is a high likelihood that some operating system patches could be left out of these releases due to compatibility or timing issues.  The work around is to mitigate any remaining vulnerabilities through additional logging, additional monitoring, changes in firewall rules, changes in IDS/IPS rules, etc.  The additional vulnerability scanning could help organizations identify these issues and address them quicker than quarterly.

A side benefit of monthly scanning will be improving the ability of organizations and their QSAs to determine if an organization’s patching and mitigation processes are working according to requirement 6.1.  Quarterly scans typically document a lot of vulnerabilities, mostly those under a CVSS of 4.0.  As a result, whether or not an organization is properly managing their environment can be very difficult and time consuming leading to missing items that should be addressed.  Having reports more often can facilitate getting these issues addressed sooner rather than later and keeping the volume lower and less daunting.

The bottom line in all of this is that monthly scanning is required to even have a chance at being secure these days.  Yet the vast majority of organizations are only doing quarterly scans and thinking they are secure.  That practice must change.

So there we have it.  My thoughts on the ASV process and how I would go about fixing it.


The ASV Process Is Broken – Part 2

The next reason I believe the process is broken is with the automated scanning processes.  They do not seem to be accurately assessing the security of Web servers, firewalls, routers and other externally facing devices that form an organization’s perimeter.  In my opinion, it seems that in our drive to bring down the cost, we have created a false sense of security by having the concept of a “passing” scan and only requiring scanning on a quarterly basis.

It is hard to believe that external vulnerability scanning came into vogue more than a decade ago.  There were no automated solutions.  No popping out to a Web site, entering IP addresses and scheduling a scan.  It was all manual.  A person was required to configure the scanning solution and then launch the scanner.

There was a proven methodology used for approaching vulnerability scanning.  That methodology involved approaching the organization as an attacker would. .  The security professional would go out to domain registrars and other official sources to obtain potential IP address ranges, registered domains and other public information.  Then the professional would develop a profile of the organization’s external security posture using Nmap and similar tools as well as running non-intrusive scans to identify systems and potential entry points.  But even more insidious, the security professional would do what became to be known as “Google Hacking” to find out what, if any, information was available on the Internet that might facilitate the attack.  Log data, firewall configurations and even authentication credentials could be obtained through Google Hacking.  Even today, while Google does its best to minimize the information they collect, it is amazing the amount of information that can still be obtained with specially crafted Google searches.

Once the preliminaries were done, it was time to actually scan.  The most accepted approach used at that time was to run two scans, one fully external with security measures enabled and one with testing done inside the DMZ.  The scan done with the security measures enabled was to confirm that an organization’s firewalls, load balancers or other front ends were functioning as configured.  Essentially this scan showed what an attacker would discover in their scanning of the organization’s external presence.  There were two purposes of the second scan.  The first was to provide an inventory of vulnerabilities that were being obfuscated by the security measures so that they could be addressed or mitigated.

The second purpose of the DMZ scan was for what some would call today a form of “threat intelligence”.  The second scan provided a view of what an attacker would encounter if they were able to circumvent the organization’s external security.  It answered the question of how large the attack surface was in the DMZ should it be compromised?  What else could be subverted and used to further an attacker’s goal of getting even deeper into a network?  The point of this part of the exercise was to provide the organization with an idea of what to look for should anomalies begin to appear in log data.

Throughout this scanning process, separate scans were run for firewalls, routers, load balancers, Windows devices, Linux devices, etc.  The rationale for that was to obtain clear results based on the type of device.  No reason to toss a bunch of Windows vulnerabilities against a Cisco firewall or F5 load balancer and then have to sift through the results to remove the false positive entries generated by the Windows devices they protect.

All of this information would be collated, analyzed and an actionable report produced for management and security professionals to absorb and understand the organization’s true external security posture.  Once the draft report was delivered there would be one or more calls or meetings with the organization’s security personnel to tweak the message of the results, develop action plans to address the results and then a final meeting with management and security personnel to deliver the final report.  At the end of this process, management and security personnel knew exactly where they stood from a security perspective and what needed to be done to ensure that they remained secure going forward.

Obviously, such an approach is fairly comprehensive and that costs money to produce.  And it was the cost that drove the automation of the process so that it was less labor intensive.

In the mad dash to the lowest possible cost, the following is what typically happens today.

Organizations subscribe to an ASV vulnerability scanning service.  If the organization is lucky, they go through a scoping call with a human being at the ASV service to confirm the IP addresses that are in-scope for PCI compliance.  Having been on a number of these calls with ASVs, it is more of a questionnaire approach and there is very little done by the ASV to actually confirm the scope of the scanning.  It is more or less a conversation that explains how to enter the IP addresses into the scanning tool’s Web interface and to schedule a scan, not a diligent conversation on whether or not a device is in scope or not.  There is typically a brief discussion on how to use the Web site to dispute scanning results that caused a non-passing scan.  In total, the call takes 20 to 30 minutes at which point customers are essentially on their own.

Scans are run on a quarterly time period.  Scans may be rerun if vulnerabilities are identified that result in a non-passing scan.  Once a passing scan is obtained, a passing certificate is generated and the organization moves on to the next quarter.  Every now and then a result may be disputed, but those situations are typically rare.  The bottom line is that the ASV process is fairly automated with very little, if any, human intervention.

From a tool perspective, the vulnerability scanner is tossing every vulnerability at every IP address they are given.  Firewalls, load balancers, switches and routers are tested for Windows and Linux vulnerabilities.  False positive results can be rampant but most scanners weed out the obvious false positive results based on OS and device signatures.  But even if a few false positives remain, as long as they have a CVSS score of less than 4.0, no one cares.  And the scanner is configured to run as quickly as possible because it needs to be available for the next customer so that the ASV does not have to have an individual scanner for every customer.

All of this is driven by customers’ desire to minimize cost as much as possible.  Thus, in order to stay in business, the ASV scanning service providers have done an admirable job of driving down cost.  But at what other costs?

While I think we can all agree that the current approach is flawed, the Council would point to the fact that the quarterly ASV scanning requirement is only a minimum and that organizations should be doing their own external vulnerability scanning more often than quarterly.  And that is what is missed by a lot of organizations is that they need to do more than what the PCI DSS requires to be secure.  And no matter how loud that message is repeated, it seems to get missed over and over again.

However, a lot of organizations do not do anything more than the PCI DSS requires.  We have this “bare minimum” mentality because, if it is not mandated in the standard, then we must be secure if we do only what is required.  This results in the false sense of security that organizations have when they only do what the PCI DSS or any other security framework for that matter mandates.

In addition, I cannot tell you how many organizations totally write off vulnerabilities that have a CVSS of less than 4.0 as “trivial” representing no threat.  There is this mistaken belief that, because the CVSS score does not warrant immediate patching, it is not a threat.  However, when you have them in relative abundance and of certain types, those “trivial” vulnerabilities can be put together in a particular sequence and used to compromise a network.  To the shock and horror of a lot of organizations, their security is breached in moments during their penetration test using those “trivial” vulnerabilities.

Then we have the limitations of the tools themselves.  Their vulnerabilities are constructed for the general audience, not a specific network.  As a result, depending on security measures and configurations, a particular vulnerability for say HTTP or FTP might not register due to a load balancer or other devices in the communication path.  Yet with a minor tweak, the same vulnerability could actually work.  And that was another point of doing that second scan inside the DMZ back in the “old” days.  Taking the two scans allowed you to correlate that while HTTP or FTP was open; it was vulnerable because that vulnerability was shown on the internal DMZ scan.  As a result, you could take additional steps to ensure that the vulnerability could not be exploited from the Internet.

The bottom line in all of this is that ASV scans have become a pointless piece of the compliance assessment process.  It has not been because the ASVs are bad, it is that the process they are forced to follow is not doing what it once did at the beginning.  It does not help that merchants and service providers treat ASV scans as a “check the box” activity and that because they do something quarterly they are secure.

If the process is broken, what should be done to fix it?  That is the subject of my next post.


Optiv Security is looking for experienced QSAs for their PCI practice. If you are an experienced QSA and are looking for a change, go to the Web site (, and click on the 'Check Out Our Openings' button and apply.

If you are posting a comment, be patient, as the comments will not be published until they are approved.

If your organization has a PCI opportunity, is in need of assistance with a PCI issue or if you would like the PCI Guru to speak at your meeting, you can contact the PCI Guru at pciguru AT gmail DOT com.

I do allow vendors to post potential solutions in response to issues that I bring up in posts. However, the PCI Guru does not endorse any specific products, so "Caveat Emptor" - let the buyer beware. Also, if I feel that the response is too "sales-ee", I reserve the right to edit or not even authorize the response.


November 2015
« Oct    

Enter your email address to subscribe to the PCI Guru blog and receive notifications of new posts by email.

Join 1,399 other followers


Get every new post delivered to your Inbox.

Join 1,399 other followers