It has been a long wait, but the PCI SSC has finally given us a look at the new “scorecard” for v2.0 of the PCI DSS. For those of you that never knew about the “scorecard,” it was given to QSAs to assess Reports On Compliance (ROCs) to ensure that QSAs have properly conducted PCI assessments. I have not had a chance to get through all 112 pages of this document, but I have gotten through the first part of it and I wanted to share my thoughts.
The first change to the “scorecard” is its name. It is no longer the “scorecard,” it is titled ‘ROC Reporting Instructions for PCI DSS v2.0’. The naming seems to indicate that once the QSA review period is over it will be posted to the PCI SSC’s Web site in the Documents Library.
Overall, the document is similar to the scorecard for v1.2.1, but no longer documents the scores that the PCI SSC QA team will use to assess QSAs. However, from the way it is written, I would assume that if a requirement in the ROC does not contain everything documented in the Reporting Instructions, that it is considered to have not met the QA requirement.
Another general comment I have is that it is woefully lacking in examples. While there seems to be a significant amount of guidance provided for what to write in the ROC, there are also ambiguous or unclear references that could be explained if the PCI SSC provided relevant examples of what they desires the QSAs to write.
The biggest change I have found thus far is the removal of the requirement to observe network traffic as the Network Monitoring column is gone from the Reporting Instructions. Prior to this point, QSAs were required to obtain network traffic via WireShark or similar tool to prove that network traffic is encrypted. I reviewed requirements 1.2.1.a, 1.2.1.b, 3.2.1, 3.2.2, 3.2.3 and 11.4.a that had the Network Monitoring requirement in the v1.2.1 scorecard. Based on the training for the 2011 QSA recertification, networking monitoring testing is still something needed for confirming compliance with requirement 1, so even though it has been removed as a column, it appears to still be required. However, from the Reporting Instructions, the network monitoring is not explicit, so this is one of those areas where the PCI SSC will definitely need to clarify things.
The section in the Executive Summary at the front of the ROC that discusses how a network is segmented to minimize scope will now require a fairly detailed discussion regarding that segmentation. All network segments need to be described along with their purpose as well as a discussion of how the segments are architected and whether the segments carry cardholder data (CHD). If access is provided to the cardholder data environment (CDE), that access needs to be described and that description needs to document how access is controlled. It is very clear from the write up surrounding this section that QSAs and their clients will have to put much more work into this section to satisfy the PCI SSC.
Another clarification area is with the review of system configurations done as part of requirements 1 and 2. The guidance now given by the PCI SSC is that they no longer want the documentation to be a list of configuration files that were reviewed by the QSA. However, in the next breath, the Reporting Instructions tell the reader that a QSA must provide enough detail to prove that configuration files were reviewed. So what is an acceptable level of detail? Can we say that we reviewed 5 or 25 firewall configuration files? In the past, we were told that this sort of approach was unacceptable. The PCI SSC will need to provide one or more examples of language that they will accept.
Of all of the things I have read thus far, the one that just gets me seething is from the “Dos and Don’ts” page. One of the “Don’ts” is “Don’t copy responses from one Testing Procedure to another.” Further down on the list is “Don’t cross reference between responses”. After going through our QA assessment and remediation, we were told by the QA person that we needed to do a better job of putting all of the information from earlier requirements that was relevant into every requirement as each requirement needs to be able to stand on its own. But now, according to the Reporting Instructions, you cannot bring all of that documentation to the new requirement by using cut and paste. What a bunch of “make do” work.
But this “make do” work is all because the PCI SSC is basically implying that it cannot trust its QSACs to do the work that is required to ensure an organization is complying with the PCI DSS. However, just because a QSA writes something in a ROC does not mean they actually did the work. It just means that the QSA knows how to write what the PCI SSC wants to read. And to make matters worse, the PCI SSC provides the Reporting Instructions to provide guidance on just what to write as well as telling QSACs to develop ROC templates to speed the writing process.
A prime example of this is a new requirement in the section of the ROC where the QSA documents the list of people interviewed and/or observed. The PCI SSC now requires the QSA to document what these people were interviewed about or were observed doing. The purpose of this new requirement is to provide even more “proof” that a QSA did their job. Another minor example of the PCI SSC trying to get “proof” of a QSA’s work effort is the increase in the level of detail being asked to document is the dates that QSAs were on-site for fieldwork. In the past, QSAs were only required to document the period covered by the assessment. However, QSAs are now being required to also document all of the dates of their fieldwork as well as the duration of their fieldwork and review period.
This is one of my biggest issues with the ROC process. The PCI SSC refuses to adopt a more intelligent and cost effective reporting process of documenting requirement exceptions. Instead, the PCI SSC requires QSAs to document their fieldwork process in the report. As a result, an inordinate amount of time, paper and hence money is spent on what is really, in my humble opinion, a totally worthless effort.
I understand why this was required. When the PCI SSC did not have the right to review a QSA’s work papers and other documentation, having such documentation in the ROC was the only way the PCI SSC, card brands and acquiring banks could assess whether or not a QSA had done their job. Now that more than a year has gone by since the PCI SSC required all QSACs to include verbiage that allows the PCI SSC to review a QSAC’s work papers, putting all of this effort into a response writing requirement should no longer be required. QSAs should be able to mark a requirement either ‘In Place’ or ‘Not In Place’. If a requirement is ‘Not In Place’, then the QSA should document why the requirement is not in place and what the organization is doing to remediate the problem and when the remediation will be complete. Such an approach would make the creation of the ROC much faster and would make the ROC much quicker to read and easier to understand. This is the approach used in the accounting industry for their SSAE 16 reports and there is no reason why the PCI SSC could not adopt the same approach.
The PCI SSC continues to cling to this inane reporting requirement because it apparently is relying on the readers of the ROCs to “rat out” those QSACs that are producing inadequate reports. I hate to be the bearer of bad news, but based on my review of ROCs from other QSACs that I have encountered over the last year, the “bad eggs” are not being weeded out. Based on my interaction with acquiring banks and various card brands, there are a lot of ROCs that are not being read in detail. And even those ROCs that are being read, most comments surround anything that is determined to have been ‘Not In Place’. Occasionally we get a question about an ‘In Place’ item. Obviously the current approach is not working and as long as the PCI SSC continues this approach, we are not going to build trust between the PCI SSC and QSACs.
I know that this is a dilemma for the PCI SSC, but it is something that needs to be addressed and soon. Organizations that have to go through the ROC process are pressuring QSAs to reduce costs as much as possible not only due to our current economic conditions but also because of the thin margins retailers live on. In order to keep the PCI compliance process relevant, the PCI SSC needs to get out in front of this issue. The PCI DSS assessment process is very labor intensive, so the only cost savings to be obtained will be in making the process less labor intensive.
UPDATE: On the morning of September 20, 2011, the PCI SSC released the final version of the Reporting Instructions along with an FAQ. These documents can be obtained from the PCI SSC Document Library under the Addition Documents – QSA heading.