In the November 2016 Assessor Newsletter from the PCI SSC, there is a clarification on what ‘Not Tested’ actually means and implies. I am sure this will really get some service providers whipped up as it will create some issues with work they perform on behalf of their customers.
The following is taken directly from that newsletter.
“Recently, AQM has received some questions about the impact of using “Not Tested” as a response within a completed ROC. This article is intended to address a few points briefly, with published documentation to follow.
- Due to an oversight, the option for “Not Tested” was not included in the summary findings table within the summary overview when that table was introduced with the ROC Reporting Template for use with PCI DSS v3.2. We will publish an errata for the ROC Reporting Template shortly.
- Some have asked whether one can have a compliant AOC in instances where “Not Tested” was used. While PCI SSC is not able to comment on matters of compliance, we would direct you to read the verbiage at Part 3 PCI DSS Validation of the Attestation of Compliance below:
How to achieve “all questions answered affirmatively” is the question. PCI SSC does not consider “Not Tested” to be an affirmative statement. The difference between “Not Tested” and “Not Applicable” is that no testing at all is performed for “Not Tested” whereas for “Not Applicable” some testing is performed to confirm a given control is truly not applicable. As such, between “Not Tested” and “Not Applicable,” only “Not Applicable” can be considered an affirmative response.
The intent in introducing “Not Tested” was to achieve a better level of transparency as to the level of compliance and this clarification supports that intent. If you have questions or suggestions, please reach out to the QSA Program Manager.”
It is that second to the last paragraph that will likely send most people off of the deep end. Their comment that the “PCI SSC does not consider “Not Tested” to be an affirmative statement” really got me going. What exactly then was the point of using ‘Not Tested’ if you did not consider it an affirmative statement? Which by the way, when using affirmative as an adjective, means “asserting the truth, validity, or fact of something.” Last I checked, ‘Not Tested’ would be considered a truth or fact.
There are a number of options for the Council to take here.
- Change the wording in the ‘Compliant’ box in Part 3 to reflect that an entity is compliant with all of the requirements tested.
- Give us a box in Part 3 that says ‘Compliant with Exceptions’ or something of that ilk which would allow those entities not testing certain requirements to still be judged compliant with what was tested.
- Tell QSAs that an AOC cannot be filled out for assessments that mark any requirements as ‘Not Tested’ because an AOC is not relevant.
I remember at a number of past Community Meetings various Council representatives repeatedly and emphatically told those of us from the Accounting community that PCI assessments were not SAS 70 (now SSAE 16) engagements when we would invoke SAS 70 like rules for sampling, testing and the like. Well, I hate to say it, but the Council is sure turning them into one with all of these pronouncements.
UPDATE: On the Council’s Webinar on Thursday, December 15, it was announced that the Council will be making changes to the AOC and will issue new guidance on this topic sometime in the first Quarter of 2017. So stay tuned for an update.
UPDATE: From the January 2017 Assessor Newsletter.
“Update on use of “Not Tested” with PCI DSS
In the November 2016 Assessor Newsletter, PCI SSC addressed the use of “Not Tested” in PCI DSS and the resulting impact on the Attestation on Compliance (AoC). The article included no change in intent, but rather acknowledgment of existing confusion. PCI SSC recognizes the challenges that the clarification on “NotTested” highlighted and is working internally on providing further documentation to support assessors in addressing such cases.
PCI SSC cannot comment on matters of compliance, as those decisions – as always – are business decisions that should be made by the assessor with the payment card brands and/or acquirers. The article ‘Clarification – “Not Tested” use in PCI DSS’ was never intended to change this, and assessors should continue to work with these entities as they have since the “Not Tested” designation was introduced.
Please note that FAQ 1331 “Can SAQ eligibility criteria be used for determining applicability of PCI DSS requirements for onsite assessments?” has been updated this month to better support the intended use of the “Not Tested” and “Not Applicable” findings.”
The bottom line is that “Not Tested” cannot be a response to any requirement and then expect to have a compliant ROC/SAQ and resulting AOC.
PCI Guru 
It presents a problem for merchants who have more than one entirely separate channel each of which may qualify for one of the smaller SAQs. If you complete separate SAQs for each channel then fine, as the Not Tested requirements do not appear in the SAQ. But if, to simplify reporting, you combine them both into a single SAQ D (which used to be the only way you could report it) then you have to mark the irrelevant requirements Not Tested and the merchant cannot be compliant.
A lot of QSAs were using the Not Tested option for co-location data centers where they are only providing HVAC, electrical and racks for customers. Why have to mark everything NA when it’s not in-scope in the first place?
I think the original intention of Not Tested was for partial assessments, e.g. only testing a specific security control. In that case, it makes sense that the entity is not considered fully compliant.
However, the uses for Not Tested has since then been expanded. Merchants assessed using the method described in FAQ 1331 (basing the scope on the appropriate SAQ) are just as Compliant as Merchants assessed with the traditional method.
Unfortunately, the Council has failed to keep up with this development (even though it’s their own invention) and is now repeating the obsolete original intention of Not Tested.
If we have to check “Non-Compliant” when some requirements are Not Tested, then the whole concept of Not Tested is effectively useless.
For example, if we follow the guidance in PCI SSC FAQ 1331 and base the scope of an assessment on an SAQ, we are instructed to use Not Tested for requirements not included in the SAQ. It’s a great way to simplify an assessment. Now we can’t use it anymore, because it would result in a “Non-Compliant” AoC.
I can’t see a single scenario where Not Tested can be used anymore. No Merchant or Service Provider would ever accept a “Non-Compliant” AoC if they have done everything that is required of them.
PCI SSC seems to be split internally with one side focusing on security and common sense, and the other focusing on making PCI DSS assessment as complex and expensive as possible. The latter seems to have the upper hand at the moment.
The major issue is that this is not how PCI is currently viewed within the community. You are “Compliant” or “Not Compliant” currently, no one is used to being provided with a ROC with partially tested requirements just to check the box for their clients, they are used to getting an AOC that says yes or no. Also, these facilities could not say they are “PCI Compliant” on their website or marketing material anymore either.
What makes this a little odd in my book, is when you utilize the PCI SSC FAQ 1331: “Can SAQ eligibility criteria be used for determining applicability of PCI DSS requirements for onsite assessments?” to perform a ROC, or maybe an SAQ-D for a service provider (who utilizes only a hosted payment page for example). They would now not receive a compliant AOC unless all requirements were tested and proven to be “Not Applicable”.
I do agree with the PCI SSCs POV, as if every requirement has not been assessed (or determined to be not applicable) an entity is not “PCI Compliant”. However, as I said before, this isn’t the mind set that the card brands, acquirers, or the rest of the world has. They want someone to be compliant or not on paper. Period.
Thoughts?
The problem though is with service providers that are providing rack space for clients that MIGHT have to be PCI compliant. The only controls that apply to them are physical security and policies. The rest of the DSS is pointless. When the ‘Not Tested’ label was introduced, QSAs were told to use for these sorts of instances rather than marking all the stuff ‘Not Applicable’ with the same reason for NA for every box. Now you cannot do that. So what the is the point of Not Tested if you essentially not able to use it? IN my opinion, there is no point.