It is that time of the year. Time for another get together with the PCI SSC, the card brands, participating organizations and QSAs. This year’s meetings are in Orlando and Barcelona. Unfortunately, I am not going to be in attendance due to scheduling conflicts. Since I will not be able to attend, I thought I would provide a topic for discussion. I want to get the PCI SSC to repeal their inane Report On Compliance (ROC) report writing standard. This standard has become onerous and, in the end, has become “make do” work.
To understand this situation, you need a bit of history. Until last year, the only proof that the PCI SSC and the acquiring banks had to prove a QSA had done their job properly was to read the ROC. The ROC was required to contain references to all of the documentation, interviews and procedures they had observed to ensure that an organization was complying with the PCI DSS. As a result, this caused the PCI SSC to develop an extensive grading and scoring spreadsheet that is used to determine if a ROC covers everything it is required to cover. Each test may have any of the following components.
- Observation;
- Interview;
- Documentation;
- Process/action/state; and
- Network monitoring.
Each of these components may be assessed one to four scoring points depending on the number of occurrences that may be contained in the given test. A ROC must score better than 75% in possible points to avoid remediation. But the PCI SSC expects that a ROC should score no lower than 95% of possible points.
The PCI SSC has instructed QSAs that each test in the ROC must be able to stand on its own. This means that one test is not allowed to reference another test. As a result, QSAs must replicate of a lot of information throughout the report. This obviously introduces the potential for errors and omissions in the report as well as making the report unnecessarily long.
To ensure the report writing process is truly questionable, the PCI SSC recommends that QSACs develop pre-written templates so that all of the components get covered for each test. While a template speeds the report writing process, I would still estimate that the report writing process consumes at least one-third to one-half of a PCI assessment’s budget. Not only does it take time to write, but it takes a lot of time to proof and to review.
As I stated earlier, last year, the PCI SSC began requiring language in all QSA contracts that grants the PCI SSC the right to examine any QSA’s work papers. AS a result, one would think that this report writing standard would no longer be needed, but it is still in place.
Because a lot of our clients use hosting services, we get to see a lot of ROCs that have been prepared by other QSAs. You can really tell those QSAs that have not been through the PCI SSC QA process by the fact that their ROCs are very short and lack detail. But for those QSAs that have been through the QA process, based on my review of their ROCs, the grading scale seems to have caused QSAs to worry more about how the ROC is written and not necessarily on the actual assessment of the security practices of their client. A lot of the writing is more about meeting the scoring template and not about the controls. In some cases, the writing starts you to wonder if the control is really in place.
ROCs can become inordinately long because of the replication of the same information over and over. During our QA remediation, we were told that the average ROC ran around 180 to 200 pages however I have yet to see one produced by us that is under 250 pages and we seem to average around 350 to 400 pages. I have heard from some reviewers at acquiring banks that the only worthwhile information in these tomes is anything that is not in place and any compensating controls. If that is all that is getting read, then what is the point of all of this other information that is being ignored? The point is that it remains the way that the PCI SSC ensures that QSAs are doing their job. And as I stated earlier if the writing makes you question if the control is in place, then what is the quality of all of this writing?
Now that the PCI SSC has the right to review a QSA’s work papers, there is no reason to require all of this pointless verbiage in the ROC. QSAs should be able to have one column for each requirement in the report labeled ‘Status’ and the entry in the column is either ‘In Place’ or ‘Not In Place’. If something is not in place, then the column next to it, labeled ‘Comments’, should document what is being done to bring a requirement into compliance and when that will occur.
If the PCI SSC is not comfortable with this approach, then maybe they have the wrong organizations as QSACs and they need to get rid of those that cannot conduct the work to professional standards. This approach works for financial auditors, there is no reason it cannot work here.
Have a good time in Orlando or Barcelona.
PCI Guru 