16
Aug
09

Reasonable Assurance

Robert Carr, Chairman and CEO of Heartland, tossed his QSA under the bus this week saying that it was the QSA’s fault that Heartland suffered a data breach last year.  Rich Mogull then wrote a great rebuttal piece on why Mr. Carr had his head up his posterior.  However, while they both point out some valid issues with the PCI assessment process, they also show a lack of understanding from the QSA’s point of view as to why the process works the way it does.  It is this misunderstanding that I want to address.
Assessments whether they are audits involving attestation or not are conducted under the concept of ‘reasonable assurance’.  Reasonable assurance is defined by the audit community as an “acknowledgment that it is not possible to assert absolutely and certainly that an event will (or will not) occur.”  Reasonable assurance, in my opinion, means that a reasonable person would consider that the testing conducted would lead them to believe that an event would be discovered.  Why is it not possible to assert absolutely?  Because, no organization is going to pay to have a QSA be on site 24×7 to review every PCI requirement that may be impacted by any operational change.  It is not cost effective and it is not efficient.
To get to reasonable assurance, an assessor develops tests and sample sizes such that the potential that any event will be likely detected.  For the PCI DSS, what needs to be tested is well documented by the requirements.  It is how that testing should best be conducted to give the QSA reasonable assurance is where things go awry.  The problem we are seeing when talking to prospects is that not all QSAs are conducting rigorous testing.  In fact, we have heard of instances where the QSA is only doing a day or two of on-site fieldwork even for the very largest of merchants or service providers.  While a lot of work can be done off-site reviewing documentation and conducting other tests, there is still a significant amount of work that needs to be done on-site.  We have been able to significantly reduce on-site fieldwork even for our largest clients, but we are still doing anywhere from a week to six weeks of on-site fieldwork.
Then there is sampling.  Sampling is the key to reasonable assurance.  We welcomed the PCI SSC’s requirement to document and justify the QSA’s sampling methodology in the Report On Compliance.  Because, at the end of the day, if the samples are not appropriately picked and of the right size, then reasonable assurance cannot be achieved.  However, we continue to hear from clients and prospects that their previous or current QSA is only sampling a small portion of locations or servers and why do we need to sample so much more?  In one extreme case, a chain with more than 1,000 outlets, each of which is storing encrypted cardholder data, let us know that their current QSA was only testing two of those locations.  Based on our sampling methodology, we had determined that we would have to test at least 90 locations annually to get reasonable assurance based on the number of POS versions and network configurations that are involved.
Is reasonable assurance perfect?  No.  Is there a better way to conduct PCI assessments? Probably not if you want a cost effective and efficient process.  QSAs need to step up and implement proper testing and sampling processes to better ensure that any gaps are going to be found.  The PCI SSC is trying to do this by requiring this information be documented in the ROC as well as with their QA review process that they just started this past year.  Are all gaps going to be found even with these enhanced assessment processes?  No.  Assessors are human and are fallible.  However, I would like to think that the number of gaps that remain open will be few and small enough that they will not cause a huge compromise.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Announcements

If you are posting a comment, be patient, as the comments will not be published until they are approved.

If your organization has a PCI opportunity, is in need of assistance with a PCI issue or if you would like the PCI Guru to speak at your meeting, you can contact the PCI Guru at pciguru AT gmail DOT com.

I do allow vendors to post potential solutions in response to issues that I bring up in posts. However, the PCI Guru does not endorse any specific products, so "Caveat Emptor" - let the buyer beware. Also, if I feel that the response is too "sales-ee", I reserve the right to edit or not even authorize the response.

Calendar

August 2009
M T W T F S S
« Jul   Sep »
 12
3456789
10111213141516
17181920212223
24252627282930
31  

Enter your email address to subscribe to the PCI Guru blog and receive notifications of new posts by email.

Join 1,868 other followers


%d bloggers like this: