Reasonable Assurance

Robert Carr, Chairman and CEO of Heartland, tossed his QSA under the bus this week saying that it was the QSA’s fault that Heartland suffered a data breach last year.  Rich Mogull then wrote a great rebuttal piece on why Mr. Carr had his head up his posterior.  However, while they both point out some valid issues with the PCI assessment process, they also show a lack of understanding from the QSA’s point of view as to why the process works the way it does.  It is this misunderstanding that I want to address.
Assessments whether they are audits involving attestation or not are conducted under the concept of ‘reasonable assurance’.  Reasonable assurance is defined by the audit community as an “acknowledgment that it is not possible to assert absolutely and certainly that an event will (or will not) occur.”  Reasonable assurance, in my opinion, means that a reasonable person would consider that the testing conducted would lead them to believe that an event would be discovered.  Why is it not possible to assert absolutely?  Because, no organization is going to pay to have a QSA be on site 24×7 to review every PCI requirement that may be impacted by any operational change.  It is not cost effective and it is not efficient.
To get to reasonable assurance, an assessor develops tests and sample sizes such that the potential that any event will be likely detected.  For the PCI DSS, what needs to be tested is well documented by the requirements.  It is how that testing should best be conducted to give the QSA reasonable assurance is where things go awry.  The problem we are seeing when talking to prospects is that not all QSAs are conducting rigorous testing.  In fact, we have heard of instances where the QSA is only doing a day or two of on-site fieldwork even for the very largest of merchants or service providers.  While a lot of work can be done off-site reviewing documentation and conducting other tests, there is still a significant amount of work that needs to be done on-site.  We have been able to significantly reduce on-site fieldwork even for our largest clients, but we are still doing anywhere from a week to six weeks of on-site fieldwork.
Then there is sampling.  Sampling is the key to reasonable assurance.  We welcomed the PCI SSC’s requirement to document and justify the QSA’s sampling methodology in the Report On Compliance.  Because, at the end of the day, if the samples are not appropriately picked and of the right size, then reasonable assurance cannot be achieved.  However, we continue to hear from clients and prospects that their previous or current QSA is only sampling a small portion of locations or servers and why do we need to sample so much more?  In one extreme case, a chain with more than 1,000 outlets, each of which is storing encrypted cardholder data, let us know that their current QSA was only testing two of those locations.  Based on our sampling methodology, we had determined that we would have to test at least 90 locations annually to get reasonable assurance based on the number of POS versions and network configurations that are involved.
Is reasonable assurance perfect?  No.  Is there a better way to conduct PCI assessments? Probably not if you want a cost effective and efficient process.  QSAs need to step up and implement proper testing and sampling processes to better ensure that any gaps are going to be found.  The PCI SSC is trying to do this by requiring this information be documented in the ROC as well as with their QA review process that they just started this past year.  Are all gaps going to be found even with these enhanced assessment processes?  No.  Assessors are human and are fallible.  However, I would like to think that the number of gaps that remain open will be few and small enough that they will not cause a huge compromise.

Advertisement