It is QSA quality assurance assessment season at work. I found out through our QSAC key contact person that we are being assessed again by the PCI SSC to see if our Reports On Compliance (ROCs) are written correctly. This is a rather timely topic given the recent news that the PCI SSC revoked the QSAC and PA-QSAC status of an organization.
If the PCI compliance program has a flaw, this is the spot. In the immortal words of Billy Crystal from his Saturday Night Live skit ‘Fernando’s Hide Away’, “It is better to look good, than to feel good.” And that is exactly what the Scorecard, now known as the Reporting Instructions basically promote. I have written about this topic before, but it is time to remind people of how ridiculous this process is to PCI compliance.
To any QSAs that have been through the QA process, it all comes down to having used the correct language in responding to the requirements of the ROC, rather than whether or not you actually assessed the right things. And to add insult to injury, the PCI SSC advises QSACs to develop a template for the ROC with all the correct language written and proofed to ensure that ROCs are written to the standard the PCI SSC requires. Technically, this allows a QSA to just fill in the blanks so that the ROC can be correctly filled out.
Ironically, on August 3, 2011, this may be exactly what happened to Chief Security Officers (CSO) and why they were stripped of their QSAC and PA-QSAC statuses. CSO may have had the greatest templates for Reports On Compliance (ROC) and Reports On Validation (ROV), but without the supporting documentation, they could have been just filling in the blanks with the right type of information without actually ensuring that the information supported the conclusions of the report. While the FAQ issued by the PCI SSC does not explicitly state the reason for CSO’s QSAC and PA-QSAC status revocation, it does imply that this was likely the case when it says, “CSO’s status as a QSA and PA-QSA was revoked only after careful review of reports and evidence submitted as part of the quality assurance program …”
It is not like the PCI SSC cannot determine this fact; it is just that they likely do not have the resources to go through a proper assessment of a QSAC or PA-QSAC. We have been repeatedly told over the years that the whole reason that all of that verbiage is required in the ROCs and ROVs was that the PCI SSC and the acquiring banks only had that language to give them an idea of what was performed, how it was performed and what the results were. However, the PCI SSC has had the right to review work papers as well as the ROCs and ROVs for over two years now. And what, exactly, the acquiring banks gleaned from the verbiage in the ROCs are debatable as I rarely ever hear back from any institutions regarding questions. As a result, in my humble opinion, there is no good reason for all of the verbiage in the ROCs/ROVs. As long as the PCI SSC has access to any project’s work papers as evidence, there is no reason to document all of the fieldwork in the ROC or ROV. And to take things to their logical conclusion, unless there are compliance issues for a particular requirement, is there really a need for acquiring banks to get anything more than the AOC?
In the past when I have brought this up, it has been rebuffed by the PCI SSC, card brands and processors because they point out that the ROC and ROV are the only pieces of documentation that proves the QSA or PA-QSA did their job. Really? Telling your assessors to have a template and fill in the blanks is better? Seriously? This all comes down to an ability to trust the assessors are doing their job. And if you cannot trust your assessors, then what is the point? Coupling the QA program with an independent assessment of a QSAC’s/PA-QSAC’s work papers should be more than adequate to determine if the evidence exists and the appropriate work is being performed.
Reviewing work papers is a tough process. In the public accounting world, we have internal and external reviews of our work. Internal reviews are typically referred to as inter-office inspections as senior personnel from one office examine another office’s work papers for a sample of engagements to confirm that the work papers support our conclusions and opinions. External reviews are conducted in a similar fashion, but by another accounting firm. Inter-office reviews can occur as often as necessary. External reviews typically occur every three to five years. While all of this can appear a bit self serving, I can tell you from going through numerous inter-office and external inspections that they are anything but easy and typically bring out a number of areas that require improvement or changes in procedures. I would highly recommend to the PCI SSC that they consider the self assessment and independent assessment approach for QSACs and PA-QSACs to supplement the existing PCI SSC QA process.
There would be all sorts of winners if we brought sanity to the ROC and ROV. The first would be the organizations being assessed as they would likely see lower costs for their assessments. I believe this because in my limited analysis of engagement costs, 30% to 50% of the cost of an assessment seems to be attributable to writing the report to meet the requirements documented in the Reporting Instructions. QSAs would be able to create ROCs and ROVs much faster as the only times that would require detailed documentation would be any items that are Not In Place. QSAs would win because they would not have to put forth an inordinate amount of effort generating 200+ page tomes. Acquiring banks and processors would win because they would not have to read through those 200+ pages figure out if there are issues and where they exist.
I intend to bring this topic up again at the PCI Community Meeting in September. Hopefully we can fix this problem and bring some rationality to the PCI compliance reporting process.