Information Supplements Versus The PCI DSS

At various times over the years, the Council has repeatedly told QSAs, Participating Organizations (PO) and anyone else that has asked questions about statements in the Information Supplements the following.

“Information Supplements only offer guidance to organizations and do not replace or supplant anything stated in the PCI DSS.”

So what are the point then of the Information Supplements?

Boy is that a good question. As a QSA, I often ask myself that very question after some of the inane conversations with clients and prospective clients regarding Information Supplements and their supposed “guidance”.

The first thing everyone should remember about Information Supplements is that they are developed and written by a committee at the suggestion of the Council, POs or as part of special interest work groups. These committees are made up of personnel from interested POs, QSAs, ISAs, vendors and anyone else willing to participate in their development. They are edited by a representative from the Council and reviewed by the Committee and are then submitted to all POs, QSAs and ISAs for review and comment. Similar in concept to the development and review of RFCs by the IETF.

The other key point about Information Supplements are that they are developed to give QSAs, ISAs and organizations ideas and guidance on how best to appropriately meet the requirements of the PCI DSS and the Reporting Template testing. Again, as the Council has repeatedly stated, the Information Supplements do not replace the explicit guidance and testing requirements in the PCI DSS and the Reporting Template. They are merely suggests on an approach.

Yet time and again, QSAs and ISAs get these priceless documents tossed in our faces and are told we do not know what we are talking about. “The Information Supplement says …” is always put out there as the justification as to why an organization is doing something it should not be doing or as the rationale for why the organization is not in compliance with the PCI DSS. And we again are forced to explain that the Council never has said that an Information Supplement replaces the guidance and testing in the PCI DSS or the Reporting Template.

The first question anyone, and I do mean anyone, should ask about any statement in an Information Supplement is, “Does the PCI DSS and/or the Reporting Template explicitly say the same thing?” Those are the only two documents that matter and the only documents that your organization will be assessed against. If it is not explicitly called out in either of those documents, then it is not accurate and does not reflect the compliance requirements.

As an example. I was on a conference call recently regarding the Council’s Information Supplement on penetration testing. This supplement was issued in March, 2015 and is possibly one of the most confusing and contradictory pieces of “guidance” we have ever encountered. In fact, it has created more confusion than it has actually clarified. In my very humble opinion, the Council would be better off taking it out of circulation because of all of the trouble it creates for QSAs, penetration testers, ASVs and clients. It is possibly one of the worst written of the Information Supplements and, while people both on the Committee that developed it and externally supplied the Council with numerous suggestions for changes, those changes were not incorporated into the document. Why those changes were not incorporated is anyone’s guess. But we in the PCI community ended up with possibly the worst expressed and misunderstood guidance available.

As usual, the client was arguing over the scope of their penetration testing. I get the fact that organizations want to minimize costs and scope as much as possible. However when you listen to some security professionals arguments on this topic, you just wonder how they got to their positions as they argue over not testing systems and devices that are painfully obvious to be in scope.

And as also is usual, the first piece of confusion regarding scope is in Section 2, page 5, first paragraph after the bullets and states the following.

“It is not a requirement to test from within the CDE to the servers inside the CDE; and testing exclusively from within the CDE perimeter will not satisfy the requirement. However, when access to the CDE is obtained as a result of the testing, the penetration tester may elect to continue exploring inside the network and further the attack against other systems within the CDE, and may also include testing any data-exfiltration prevention (data-loss prevention) controls that are in place.”

One would think that to any reasonably intelligent information security professional, the first part of the sentence, “It is not a requirement to test from within the CDE to the servers inside the CDE;” would be considered a pure line of garbage. Never mind that none of the recognized penetration testing methodologies ever suggest such an approach. But people arguing never consider that fact. Nope. The people arguing are so focused on cutting their PCI compliance bill that it does not matter that the statement is pure and unsupported garbage. It is considered the gospel truth. Otherwise, why would the Council allow such a statement? Good question. We have asked the Council that question and the answer back is? You guessed it.

“Information Supplements only offer guidance to organizations and do not replace or supplant anything stated in the PCI DSS.”

Again, never mind it is in no way supported by the guidance provided by the PCI DSS for requirement 11.3 which says:

“The intent of a penetration test is to simulate a real-world attack situation with a goal of identifying how far an attacker would be able to penetrate into an environment. This allows an entity to gain a better understanding of their potential exposure and develop a strategy to defend against attacks.”

But argue that point they do even when you point out that arguing this point is basically arguing that any attacker would stop at the perimeter of the CDE and would go no further.

Seriously? If you believe that fact, you must also believe in Santa Claus, the Easter Bunny, the Tooth Fairy and any other of the multitude of mythical fictional creatures. Or you are just lying to yourself and are in serious denial about your organization’s security posture. But argue on they do.

Then you pair that to the second part of that first sentence of this paragraph that says, “… and testing exclusively from within the CDE perimeter will not satisfy the requirement.” Just adds to the out of scope argument.

As I point out when bitch slapped with this terrible writing, if you go back and carefully re-read the second part of the first sentence, what it points out is that penetration testing from only inside the CDE is not sufficient to meet the penetration testing requirements of the PCI DSS requirement 11.3. In no way does that sentence say or even further imply that the CDE is out of scope. It is actually saying that penetration testing should be done from within the CDE, but that penetration testing only inside the CDE does not meet 11.3. But people will still argue that the CDE is out of scope.

That the CDE is in scope is further supported by the definitions of “critical systems” from section 2.2.1 of the document which defines that not only are systems within the CDE in scope, but also those that are outside the CDE but could affect the security of those systems inside the CDE (i.e., what the Council and the Open PCI DSS Scoping Toolkit refer to as “connected to” systems). However, people arguing over scope rarely, if ever, tie these two section together and then argue that because they are in separate sections they cannot be possibly together even though the entire document is about only one subject, penetration testing and requirements in 11.3 of the PCI DSS.

So before you go off telling your QSA or ISA that the Information Supplement says something. Think about what the information supplement says. Is the guidance from the Information Supplement even implied in the PCI DSS? Read the guidance in the PCI DSS and the testing procedures from the Reporting Template. If the PCI DSS or the Reporting Template do not explicitly have the same language in them that the Information Supplement has, then the Information Supplement is merely a suggestion.

And if the guidance from the Information Supplement does not make sense, pull your head out of your posterior and use some God given common sense. Ask your QSA or ISA to explain it, before going off halfcocked and thinking that someone could actually think such things made sense.

But again, why would the Council allow such statements? Good question. We have asked the Council that question and the answer back is? You guessed it.

“Information Supplements only offer guidance to organizations and do not replace or supplant anything stated in the PCI DSS.”

Clear as mud? You bet.

But what did you expect? It is PCI.

For all of you in the United States, have a happy and safe Thanksgiving holiday.

Advertisement

PCI Issues Penetration Testing Information Supplement

On Thursday, March 26, the PCI SSC issued their latest information supplement. This one is focused on the topic of penetration testing. Not surprisingly, I have a number of comments based on what I have read. There are some good things that have come out of this effort. But in my opinion, the committee missed the boat in a few areas and those will lead to even more confusion between penetration testers, QSAs and clients.

Scoping

Talk about making a mess of a mess, I believe the writers accomplished that amazingly well in their discussion on scoping. This is probably the area that creates the most confusion and will likely create the most problems.

On page 5 and the first bullet, talk about confusing. It took me a number of readings to even get what I think they are trying to communicate.

“The scope of the internal penetration test is the internal perimeter of the CDE from the perspective of any out-of-scope LAN segment that has access to a unique type of attack on the CDE perimeter. Critical systems or those systems that may impact the security of the CDE should also be included in the scope. Testing must include both application-layer and network-layer assessments.”

Out of scope LAN segments are out of scope because there are no rules that allow them access to the CDE. I agree that you need to test that these LAN segments are truly out of scope, but if there is a unique type of attack available from such a LAN segment into the CDE, that would seem to imply that the LAN segment was not out of scope. Confusing and likely to cause a lot of pen testers to force a lot of unnecessary testing and fishing for bizarre issues on out of scope network segments.

The next sentence paired with the first seems to just add to the confusion because it implies again that the out of scope LANs are not out of scope. These should have been two separate bullet points in my opinion.

In the end, what I think the writers are saying is that even if deemed out of scope, all LANs and VLANs need to be confirmed to be out of scope. However, how they have said that in my opinion leads to a lot of confusion.

If the last one did not cause your head to spin around, this one probably will. It is the third bullet at the top of page 5.

“To be considered out of scope for PCI DSS, a system component must be isolated (segmented) from the CDE, such that even if the out-of-scope system component was compromised it could not impact the security of the CDE. Therefore, the penetration test may include systems not directly related to the processing, transmission or storage of cardholder data to ensure these assets, if compromised, could not impact the security of the CDE.”

Correct me if I’m wrong, but this seems to imply that everything is in scope for penetration testing. I know the writers of this supplement have used the word ‘MAY’, but we also know how ‘MAY’ tends to be interpreted by the Council (i.e., MUST) and how it will be interpreted by a lot of our clients (i.e., do not do it). Until a few penetration testing reports are reviewed by the Council, the verdict will be out on the correct approach. However, coupled with the theme created by these bullets, I am guessing that the committee could not agree on how to express how to scope the penetration test.

But if those bullets were not confusing enough on scoping, the writers add insult to injury. After the bulleted list at the top of page 5, the first paragraph states the following.

“It is not a requirement to test from within the CDE to the servers inside the CDE; and testing exclusively from within the CDE perimeter will not satisfy the requirement. However, when access to the CDE is obtained as a result of the testing, the penetration tester may elect to continue exploring inside the network and further the attack against other systems within the CDE, and may also include testing any data-exfiltration prevention (data-loss prevention) controls that are in place.”

An internal pen test does not require testing from within the cardholder data environment (CDE)? This sounds like the “Three Hop Rule” all over again. But then we twist it further somehow justifying the first sentence.

All this diatribe does is just further obfuscates scoping. What simply should have been said is that everything in-scope for PCI compliance is to be penetration tested. Instead, we got a lot of words that, in my very humble opinion, do nothing to clarify scoping.

PA-DSS

On page 6 under 2.3.2 PA-DSS Compliant Applications we get the following guidance.

“If a payment application has been PA-DSS validated, the application’s functionality does not need to be tested as part of the entity’s PCI DSS compliance validation. However, the implementation of the application does need to be tested. This includes both the operating system and any exposed services, but not the payment application’s functionality (e.g., authentication, key management, transaction processing, etc.) since this was validated as part of the PA-DSS application validation.”

Under the PCI DSS, even PA-DSS validated applications still must be tested to prove that their implementations were completed per the vendor’s PA-DSS implementation guide. Yet for penetration testing, PA-DSS validated applications, their operating systems and exposed services are exempt?

I hope the acquiring banks are geared up for the fights this is going to cause between the penetration testers and their clients. For example, in the restaurant and hospitality industries, I can see the situation where penetration testing only occurs on firewalls, routers and switches because everything else is PA-DSS validated.

I do not believe this is what the Council wanted, but that is what this supplement is stating. If we are truly talking about security of our networks and systems, this is not how you achieve it.

Social Engineering

I was pleasantly surprised that this was included in this supplement. However, as I read through this section, I got more concerned that it was even included.

For section 2.5 Social Engineering on page 7, we encounter the following.

“Social engineering is the attempt to gain information, access, or introduce unauthorized software into the environment through the manipulation of end users. PCI DSS v3.0 reconfirms testing by requiring industry accepted penetration testing approaches (many of which include social engineering as part of their approach) and to have an approach to penetration testing that “considers the threats and vulnerabilities experienced by merchants in the last 12 months.” This may include social-engineering attacks as a method used for introducing malware into the environment.”

This makes social engineering attacks legitimate if the client has been breached by such methods in the past year. But does this include physical as well as logical social engineering? Since it is not explicitly called out, I would assume so, but it should have been defined. Regardless of that being explicitly called out, this is a great thing to see in this document.

On page 8, the last paragraph of the discussion on social engineering states:

“Social-engineering testing may not be appropriate or provide a meaningful result for all organizations. Although social-engineering testing is not a requirement of PCI DSS, an organization may consider documenting the reason(s) for foregoing social-engineering testing and include applicable documentation with the internal and external penetration test reports, particularly if social-engineering attacks were encountered in the last 12 months.”

What started with such promise ends on a very sour note. This last paragraph gives organizations an out. So I guess we can look forward to a lot of organizations issuing inventive explanations to avoid social engineering testing.

Some Miscellaneous Issues

On page 2, the seventh bullet down in the Terminology we have the following.

“National Vulnerability Database (NVD): The U.S. government repository of standards based vulnerability management data. This data enables automation of vulnerability management, security measurement, and compliance (e.g., FISMA).”

I know they just grabbed the description from the National Vulnerability Database (NVD) Web site, but that does not mean that it is appropriate for use when discussing penetration testing. The Common Vulnerabilities and Exposures (CVE) database maintained by Mitre Corporation would have been the better reference since the NVD uses the CVE as the catalog of vulnerabilities.

I have no idea how the Federal Information Systems Management Act (FISMA) ties into the NVD. If they had truly pulled the content exactly from the NVD Web site they would have grabbed the fact that it ties to the Security Content Automation Protocol (SCAP) not FISMA.

My last comment on this topic is regarding the fact that the NVD (or the CVE) enable automation of vulnerability management. Had they brought in the reference to SCAP, I might have had an easier time going along with things. The NVD/CVE do not contain the actual exploits, they are merely a catalog of the vulnerabilities. As a result, I question the value of either the NVD or the CVE as to it automating vulnerability management. Yes, the CVE catalogs all of the vulnerabilities, but that is the value. In my opinion, the use of either of these in regards to the automation of the vulnerability management process is a stretch. They are a foundation that will allow for automation, but they do not in and of themselves actually enable automation.

Under 2.3.3 Web Applications on page 6, we have the following.

“It is common for an environment to host a web application that was not specifically coded for the organization such as commercial, off-the-shelf web-mail interfaces, document-sharing tools, file-transfer services, network-device administrative interfaces, etc. In these instances, the web application does not typically need an application-layer penetration test as the entity is not responsible for the source code of this type of software. Instead, the tester should perform a network-layer test and ensure the software was implemented, configured, and is currently being maintained in a secure manner (disabling or uninstalling unused services, blocking unused ports, applying current updates, etc.).”

I can see it now, organizations arguing over the pen testing of IBM Websphere, Oracle eCommerce or similar Web application frameworks because they are a package. It is a framework that you then must write an application for, but penetration testers will get into arguments over that aspect.

A Few Good Things

Yes, I have complaints, but there are some good things that are contained in this supplement.

Probably the best thing about this information supplement is the checklist at the end of the document for evaluating penetration testing reports. The Council should have a similar item for vulnerability scanning reports as well.

The discussion of the penetration testing methodology in section 4 of the document is a good read for both penetration testers and the people that contract for penetration testing services. But keep in mind it is a framework and not the entire rule/procedure set for conducting a proper penetration test. Yes, there are some good details in areas, but it is not the detailed roadmap that some people will portray it.

Section 3 has a good discussion on how to evaluate the qualifications of a penetration tester. It was nice to see that while they document the common certifications a penetration tester may hold, they caution readers that certifications alone does not make a penetration tester qualified.

The case studies in section 6 probably do more to clarify than the previous five sections. They are good examples of properly conducted penetration tests.

Finally, there is a good discussion on the definition of a “significant change”. However, I think this discussion as well as a discussion about the words “periodic” and “periodically” should be put into a discussion about risk management and assessment, rather than brought up here. I wrote a post on this topic a while back to provide some guidance to people and it was nice to see that my thoughts jive with what they wrote in the supplement.

What Was Missed

The biggest item I think that was missed was how to handle the penetration testing of huge environments. Case study 6.3 discusses the concept of sampling somewhat, but it deals with a retailer with only six locations, not thousands of locations. There are also veiled references to sampling in section 2.3.4 in discussing a separate testing environment. But there is no deliberate and complete discussion on this very important topic.

Retailers such as Wal-Mart, Target and Home Depot have more than a thousand locations with 15+ checkout lanes meaning there are, at a minimum, 15K devices out in the field that technically need to be penetration tested. How does an organization penetration test such a large number of devices annually? The answer is that they do not test them all. It is physically impossible. However, this information supplement does not address that situation and provide guidance as to what the Council deems as acceptable.

While I use some of the largest retailers in the world as my example, it is not just them as there are mid-sized retailers as well as hospitality organizations that face this same issue. I had hoped that how to approach these situations would have been explicitly addressed in greater detail but it was not.

The writers also missed a huge topic regarding social engineering testing. Unlike vulnerability scanning and penetration testing, in my experience, it is virtually impossible for any organization to “pass” social engineering testing. So what would be considered “passing” in a social engineering test? That is not discussed but should have been. Since this document broaches the subject, it is probably a good topic to propose for a SIG for this year’s SIG recommendations so we get social engineering covered in an information supplement rather than as a sidebar to penetration testing.

It is not that this information supplement is an entirely bad document. But the advice it provides needs to be taken with a bit of caution. As usual, the shortcomings will be leveraged by all of the “check boxers” to do as little as possible so that you get that box checked for 11.3.