PCI’s “Scarlet Letter”

I will forewarn you all now.  This is a rant.  I want my soapbox and my minute to get this out of my system.

Imagine you are a student and at the beginning of the semester, that for this class you will be required to write a paper on a topic of your choosing but you must cover a series of 100 points and those points are given to you by the teacher.  Therefore, you go and do research, you write a paper that covers the 100 points and you hand in your paper for grading.  You get your paper back and find that you failed because you followed an old version of the 100 points, your topic was not an approved topic, and you were supposed to write your paper in the “Queen’s” English.  Sound farfetched?  Well that is exactly what the PCI SSC is doing to the QSACs in the name of quality assurance.  All of those QSACs you see with a status of “In Remediation”, in my very humble opinion, got the short shrift in being assessed in the PCI SSC’s QA program.

Now do not get me wrong.  I am all for quality assurance processes and making sure those organizations are following the stated rules.  However, the key phrase is “stated rules” and that is where things have gone terribly wrong.  Back in March or April of this year, the PCI SSC sent out to the QSACs the ‘grading scale’ that would be used to assess reports issued by the QSACs under the PCI SSC’s new quality assurance program.  This was the first time anyone outside of the PCI SSC had seen this document.  At that time, we were told that our reports would be assessed against these rules for the QA program.  That is fine except, the reports that were being assessed were generated prior to the issuance of the grading scale.  Not only that, the grading scale was developed against the v1.2 standard of the PCI DSS and a lot of the reports being assessed were v1.1 or even v1.0.  So, what do you think the chances are of passing?  I would say slim to none.  Based on an informal poll of those that have been through this year’s QA process, not one QSAC has passed.  Boy, there is a shock, it was impossible to pass.

I understand the PCI SSC wants to ensure that its QSACs are doing their work and that the work is supported by documentation.  Any QSAC that goes through the QA process after the grading scale was published will have the benefit of knowing against what they will be assessed.  Those organizations in the first wave were blind to the assessment rules.  The only way to make this fair would be to assess all QSACs’ reports for the time period prior to the issuance of the rules, but the chances of that happening are none.  And if that was not bad enough, they are only reviewing the reports, not the QSAC’s work papers that support the work.  In another change this year, the PCI SSC has forced all QSACs to put legal language in their proposals so that the PCI SSC can have access to a QSAC’s work papers.  Prior to this, it seems that the PCI SSC does not have the legal right to see the work papers.  I am not sure how you assess something without one of the key components, but apparently, it makes sense in their world.

A lot of QSACs’ reports have been reviewed by the card brands themselves and have been accepted.  Now, all of a sudden, there is a new Sheriff in town, and they are determined to make things “right.”  However, let us make the process fair.  Review QSAC reports and make comments regarding how we can make things better, but give people 12 months after you released your grading scale before you start holding people accountable.  It is hard to justify penalizing someone for a grading scale you just created and distributed, for a standard that may or may not apply to the reports you are reviewing, and then apply that grading scale to those documents that were created well before the grading scale you are now gauging them against.

So, for those QSACs that are in the first wave of the QA program, my sympathies.  Your entry on the QSA list will be flagged in RED, the PCI SSC’s version of the ‘Scarlet Letter’.  It will be interesting to see those QSACs that will be spared this indignity by having their QA review in later years so that they have time to address the grading scale.  If you do not believe there are favorites, look again.  Time and again, certain QSACs seem to dodge bullets that catch others.  It is not what you know, it is whom you know.

Oh, and to add insult to injury.  At our recertification training in April, we were told by one of the PCI SSC’s QA people that we should develop a report template around the grading scale so that we were always certain to pass the QA process.  While such an approach makes it easy for the graders to assess reports, it kind of defeats the purpose of the QA program in my book.

Reasonable Assurance

Robert Carr, Chairman and CEO of Heartland, tossed his QSA under the bus this week saying that it was the QSA’s fault that Heartland suffered a data breach last year.  Rich Mogull then wrote a great rebuttal piece on why Mr. Carr had his head up his posterior.  However, while they both point out some valid issues with the PCI assessment process, they also show a lack of understanding from the QSA’s point of view as to why the process works the way it does.  It is this misunderstanding that I want to address.
Assessments whether they are audits involving attestation or not are conducted under the concept of ‘reasonable assurance’.  Reasonable assurance is defined by the audit community as an “acknowledgment that it is not possible to assert absolutely and certainly that an event will (or will not) occur.”  Reasonable assurance, in my opinion, means that a reasonable person would consider that the testing conducted would lead them to believe that an event would be discovered.  Why is it not possible to assert absolutely?  Because, no organization is going to pay to have a QSA be on site 24×7 to review every PCI requirement that may be impacted by any operational change.  It is not cost effective and it is not efficient.
To get to reasonable assurance, an assessor develops tests and sample sizes such that the potential that any event will be likely detected.  For the PCI DSS, what needs to be tested is well documented by the requirements.  It is how that testing should best be conducted to give the QSA reasonable assurance is where things go awry.  The problem we are seeing when talking to prospects is that not all QSAs are conducting rigorous testing.  In fact, we have heard of instances where the QSA is only doing a day or two of on-site fieldwork even for the very largest of merchants or service providers.  While a lot of work can be done off-site reviewing documentation and conducting other tests, there is still a significant amount of work that needs to be done on-site.  We have been able to significantly reduce on-site fieldwork even for our largest clients, but we are still doing anywhere from a week to six weeks of on-site fieldwork.
Then there is sampling.  Sampling is the key to reasonable assurance.  We welcomed the PCI SSC’s requirement to document and justify the QSA’s sampling methodology in the Report On Compliance.  Because, at the end of the day, if the samples are not appropriately picked and of the right size, then reasonable assurance cannot be achieved.  However, we continue to hear from clients and prospects that their previous or current QSA is only sampling a small portion of locations or servers and why do we need to sample so much more?  In one extreme case, a chain with more than 1,000 outlets, each of which is storing encrypted cardholder data, let us know that their current QSA was only testing two of those locations.  Based on our sampling methodology, we had determined that we would have to test at least 90 locations annually to get reasonable assurance based on the number of POS versions and network configurations that are involved.
Is reasonable assurance perfect?  No.  Is there a better way to conduct PCI assessments? Probably not if you want a cost effective and efficient process.  QSAs need to step up and implement proper testing and sampling processes to better ensure that any gaps are going to be found.  The PCI SSC is trying to do this by requiring this information be documented in the ROC as well as with their QA review process that they just started this past year.  Are all gaps going to be found even with these enhanced assessment processes?  No.  Assessors are human and are fallible.  However, I would like to think that the number of gaps that remain open will be few and small enough that they will not cause a huge compromise.