PCI’s “Scarlet Letter”

I will forewarn you all now.  This is a rant.  I want my soapbox and my minute to get this out of my system.

Imagine you are a student and at the beginning of the semester, that for this class you will be required to write a paper on a topic of your choosing but you must cover a series of 100 points and those points are given to you by the teacher.  Therefore, you go and do research, you write a paper that covers the 100 points and you hand in your paper for grading.  You get your paper back and find that you failed because you followed an old version of the 100 points, your topic was not an approved topic, and you were supposed to write your paper in the “Queen’s” English.  Sound farfetched?  Well that is exactly what the PCI SSC is doing to the QSACs in the name of quality assurance.  All of those QSACs you see with a status of “In Remediation”, in my very humble opinion, got the short shrift in being assessed in the PCI SSC’s QA program.

Now do not get me wrong.  I am all for quality assurance processes and making sure those organizations are following the stated rules.  However, the key phrase is “stated rules” and that is where things have gone terribly wrong.  Back in March or April of this year, the PCI SSC sent out to the QSACs the ‘grading scale’ that would be used to assess reports issued by the QSACs under the PCI SSC’s new quality assurance program.  This was the first time anyone outside of the PCI SSC had seen this document.  At that time, we were told that our reports would be assessed against these rules for the QA program.  That is fine except, the reports that were being assessed were generated prior to the issuance of the grading scale.  Not only that, the grading scale was developed against the v1.2 standard of the PCI DSS and a lot of the reports being assessed were v1.1 or even v1.0.  So, what do you think the chances are of passing?  I would say slim to none.  Based on an informal poll of those that have been through this year’s QA process, not one QSAC has passed.  Boy, there is a shock, it was impossible to pass.

I understand the PCI SSC wants to ensure that its QSACs are doing their work and that the work is supported by documentation.  Any QSAC that goes through the QA process after the grading scale was published will have the benefit of knowing against what they will be assessed.  Those organizations in the first wave were blind to the assessment rules.  The only way to make this fair would be to assess all QSACs’ reports for the time period prior to the issuance of the rules, but the chances of that happening are none.  And if that was not bad enough, they are only reviewing the reports, not the QSAC’s work papers that support the work.  In another change this year, the PCI SSC has forced all QSACs to put legal language in their proposals so that the PCI SSC can have access to a QSAC’s work papers.  Prior to this, it seems that the PCI SSC does not have the legal right to see the work papers.  I am not sure how you assess something without one of the key components, but apparently, it makes sense in their world.

A lot of QSACs’ reports have been reviewed by the card brands themselves and have been accepted.  Now, all of a sudden, there is a new Sheriff in town, and they are determined to make things “right.”  However, let us make the process fair.  Review QSAC reports and make comments regarding how we can make things better, but give people 12 months after you released your grading scale before you start holding people accountable.  It is hard to justify penalizing someone for a grading scale you just created and distributed, for a standard that may or may not apply to the reports you are reviewing, and then apply that grading scale to those documents that were created well before the grading scale you are now gauging them against.

So, for those QSACs that are in the first wave of the QA program, my sympathies.  Your entry on the QSA list will be flagged in RED, the PCI SSC’s version of the ‘Scarlet Letter’.  It will be interesting to see those QSACs that will be spared this indignity by having their QA review in later years so that they have time to address the grading scale.  If you do not believe there are favorites, look again.  Time and again, certain QSACs seem to dodge bullets that catch others.  It is not what you know, it is whom you know.

Oh, and to add insult to injury.  At our recertification training in April, we were told by one of the PCI SSC’s QA people that we should develop a report template around the grading scale so that we were always certain to pass the QA process.  While such an approach makes it easy for the graders to assess reports, it kind of defeats the purpose of the QA program in my book.


4 Responses to “PCI’s “Scarlet Letter””

  1. 1 Bryce
    September 22, 2009 at 7:27 AM

    IMHO – I would also say that there is a strong inference in the scoring scale that 75% is the “pass mark” using the scoring template provided.

    Does this not mean that an organisation could actually be non-compliant in a few non-critical areas and still pass – thus negating that they have to be compliant in all areas of the PCI-DSS.

    This could result in organisations now being selective in their efforts to close gaps, leaving out certain areas, and still expecting to pass once they reach the stated 75%

    • September 26, 2009 at 6:55 PM

      Here is what is happening. Good QSAs are being dinged on the lack of documentation in their reports. No one ever informed them that such a level of minutia was required or even desired. Because of the lack of guidance, until very recently, they prepared their reports based on their best understanding and the practices that are followed by the financial auditing profession. In the auditing profession, the detail is only provided in a report when there is a problem with compliance. All other detail is contained in an auditor’s work papers.

      Unfortunately, the PCI SSC is not run by auditors and the program did not have access to work papers until very recently. As a result, they expect to have a level of detail in the report beyond what a normal auditor would provide. The problem with this approach is that we have been told by the PCI SSC to provide enough detail to prove we did the work, but not too much detail. For some of the SAP responses, there doesn’t seem to be a good balance between too much and too little. You either provide all the detail or none at all.

  2. 3 lyalc
    September 20, 2009 at 7:18 PM

    I’m a QSA, and not using my real email details for obvious reasons.
    I’m engaged at a client site where for the past 2 years, they have been marked as complaint by a global, high profile QSA company.
    This year, the client get a list of 120 pages of gaps from their (now ex, I understand) QSA company.

    This included obvious things like log monitoring failures, IDS not up to date etc.
    The list of gaps still missed things like
    – unprotected, unencrypted SQL servers with up to 3 years of records (several dozen times the Level 1 transaction threshold) on the general office LAN, where they had been located (and populated with current data) for about 8 years.
    – No database hardening, or password policies.
    – SQL access to the core databases (containing PAN) by numerous staff.
    – No secure development methodology of policy

    While such apparent incompetence is to be vigorously deplored, the client had been assuming they were compliant for 3 years. These sorts of outcomes must be weeded out for PCI DSS to have any credibility, in my view.


    • September 21, 2009 at 5:14 AM

      I was not implying that there were not QSAs out there that did not deserve getting beat up in the QA process.

      I just wanted to point out that, at least for this year, there is no way for a QSA to pass given that the report assessment criteria was just provided in April 2009 (August 2009 for v1.1), but the reports being reviewed were completed in 2008.

      The problem is that QSAs that deserved being dinged are lumped in with QSAs that got dinged for not be omnipotent and all knowing about the grading scale. As a result, the merchant or service provider has no idea whether the ‘red’ is for incompetence or just poor report writing.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s


If you are posting a comment, be patient, as the comments will not be published until they are approved.

If your organization has a PCI opportunity, is in need of assistance with a PCI issue or if you would like the PCI Guru to speak at your meeting, you can contact the PCI Guru at pciguru AT gmail DOT com.

I do allow vendors to post potential solutions in response to issues that I bring up in posts. However, the PCI Guru does not endorse any specific products, so "Caveat Emptor" - let the buyer beware. Also, if I feel that the response is too "sales-ee", I reserve the right to edit or not even authorize the response.


August 2009
« Jul   Sep »

Enter your email address to subscribe to the PCI Guru blog and receive notifications of new posts by email.

Join 1,981 other followers


%d bloggers like this: