I will forewarn you all now. This is a rant. I want my soapbox and my minute to get this out of my system.
Imagine you are a student and at the beginning of the semester, that for this class you will be required to write a paper on a topic of your choosing but you must cover a series of 100 points and those points are given to you by the teacher. Therefore, you go and do research, you write a paper that covers the 100 points and you hand in your paper for grading. You get your paper back and find that you failed because you followed an old version of the 100 points, your topic was not an approved topic, and you were supposed to write your paper in the “Queen’s” English. Sound farfetched? Well that is exactly what the PCI SSC is doing to the QSACs in the name of quality assurance. All of those QSACs you see with a status of “In Remediation”, in my very humble opinion, got the short shrift in being assessed in the PCI SSC’s QA program.
Now do not get me wrong. I am all for quality assurance processes and making sure those organizations are following the stated rules. However, the key phrase is “stated rules” and that is where things have gone terribly wrong. Back in March or April of this year, the PCI SSC sent out to the QSACs the ‘grading scale’ that would be used to assess reports issued by the QSACs under the PCI SSC’s new quality assurance program. This was the first time anyone outside of the PCI SSC had seen this document. At that time, we were told that our reports would be assessed against these rules for the QA program. That is fine except, the reports that were being assessed were generated prior to the issuance of the grading scale. Not only that, the grading scale was developed against the v1.2 standard of the PCI DSS and a lot of the reports being assessed were v1.1 or even v1.0. So, what do you think the chances are of passing? I would say slim to none. Based on an informal poll of those that have been through this year’s QA process, not one QSAC has passed. Boy, there is a shock, it was impossible to pass.
I understand the PCI SSC wants to ensure that its QSACs are doing their work and that the work is supported by documentation. Any QSAC that goes through the QA process after the grading scale was published will have the benefit of knowing against what they will be assessed. Those organizations in the first wave were blind to the assessment rules. The only way to make this fair would be to assess all QSACs’ reports for the time period prior to the issuance of the rules, but the chances of that happening are none. And if that was not bad enough, they are only reviewing the reports, not the QSAC’s work papers that support the work. In another change this year, the PCI SSC has forced all QSACs to put legal language in their proposals so that the PCI SSC can have access to a QSAC’s work papers. Prior to this, it seems that the PCI SSC does not have the legal right to see the work papers. I am not sure how you assess something without one of the key components, but apparently, it makes sense in their world.
A lot of QSACs’ reports have been reviewed by the card brands themselves and have been accepted. Now, all of a sudden, there is a new Sheriff in town, and they are determined to make things “right.” However, let us make the process fair. Review QSAC reports and make comments regarding how we can make things better, but give people 12 months after you released your grading scale before you start holding people accountable. It is hard to justify penalizing someone for a grading scale you just created and distributed, for a standard that may or may not apply to the reports you are reviewing, and then apply that grading scale to those documents that were created well before the grading scale you are now gauging them against.
So, for those QSACs that are in the first wave of the QA program, my sympathies. Your entry on the QSA list will be flagged in RED, the PCI SSC’s version of the ‘Scarlet Letter’. It will be interesting to see those QSACs that will be spared this indignity by having their QA review in later years so that they have time to address the grading scale. If you do not believe there are favorites, look again. Time and again, certain QSACs seem to dodge bullets that catch others. It is not what you know, it is whom you know.
Oh, and to add insult to injury. At our recertification training in April, we were told by one of the PCI SSC’s QA people that we should develop a report template around the grading scale so that we were always certain to pass the QA process. While such an approach makes it easy for the graders to assess reports, it kind of defeats the purpose of the QA program in my book.