Draft PCI DSS v2.0 “Scorecard” Released

It has been a long wait, but the PCI SSC has finally given us a look at the new “scorecard” for v2.0 of the PCI DSS.  For those of you that never knew about the “scorecard,” it was given to QSAs to assess Reports On Compliance (ROCs) to ensure that QSAs have properly conducted PCI assessments.  I have not had a chance to get through all 112 pages of this document, but I have gotten through the first part of it and I wanted to share my thoughts.

The first change to the “scorecard” is its name.  It is no longer the “scorecard,” it is titled ‘ROC Reporting Instructions for PCI DSS v2.0’.  The naming seems to indicate that once the QSA review period is over it will be posted to the PCI SSC’s Web site in the Documents Library.

Overall, the document is similar to the scorecard for v1.2.1, but no longer documents the scores that the PCI SSC QA team will use to assess QSAs.  However, from the way it is written, I would assume that if a requirement in the ROC does not contain everything documented in the Reporting Instructions, that it is considered to have not met the QA requirement.

Another general comment I have is that it is woefully lacking in examples.  While there seems to be a significant amount of guidance provided for what to write in the ROC, there are also ambiguous or unclear references that could be explained if the PCI SSC provided relevant examples of what they desires the QSAs to write.

The biggest change I have found thus far is the removal of the requirement to observe network traffic as the Network Monitoring column is gone from the Reporting Instructions.  Prior to this point, QSAs were required to obtain network traffic via WireShark or similar tool to prove that network traffic is encrypted.  I reviewed requirements 1.2.1.a, 1.2.1.b, 3.2.1, 3.2.2, 3.2.3 and 11.4.a that had the Network Monitoring requirement in the v1.2.1 scorecard.  Based on the training for the 2011 QSA recertification, networking monitoring testing is still something needed for confirming compliance with requirement 1, so even though it has been removed as a column, it appears to still be required.  However, from the Reporting Instructions, the network monitoring is not explicit, so this is one of those areas where the PCI SSC will definitely need to clarify things.

The section in the Executive Summary at the front of the ROC that discusses how a network is segmented to minimize scope will now require a fairly detailed discussion regarding that segmentation.  All network segments need to be described along with their purpose as well as a discussion of how the segments are architected and whether the segments carry cardholder data (CHD).  If access is provided to the cardholder data environment (CDE), that access needs to be described and that description needs to document how access is controlled.  It is very clear from the write up surrounding this section that QSAs and their clients will have to put much more work into this section to satisfy the PCI SSC.

Another clarification area is with the review of system configurations done as part of requirements 1 and 2.  The guidance now given by the PCI SSC is that they no longer want the documentation to be a list of configuration files that were reviewed by the QSA.  However, in the next breath, the Reporting Instructions tell the reader that a QSA must provide enough detail to prove that configuration files were reviewed.  So what is an acceptable level of detail?  Can we say that we reviewed 5 or 25 firewall configuration files?  In the past, we were told that this sort of approach was unacceptable.  The PCI SSC will need to provide one or more examples of language that they will accept.

Of all of the things I have read thus far, the one that just gets me seething is from the “Dos and Don’ts” page.  One of the “Don’ts” is “Don’t copy responses from one Testing Procedure to another.”  Further down on the list is “Don’t cross reference between responses”.  After going through our QA assessment and remediation, we were told by the QA person that we needed to do a better job of putting all of the information from earlier requirements that was relevant into every requirement as each requirement needs to be able to stand on its own.  But now, according to the Reporting Instructions, you cannot bring all of that documentation to the new requirement by using cut and paste.  What a bunch of “make do” work.

But this “make do” work is all because the PCI SSC is basically implying that it cannot trust its QSACs to do the work that is required to ensure an organization is complying with the PCI DSS.  However, just because a QSA writes something in a ROC does not mean they actually did the work.  It just means that the QSA knows how to write what the PCI SSC wants to read.  And to make matters worse, the PCI SSC provides the Reporting Instructions to provide guidance on just what to write as well as telling QSACs to develop ROC templates to speed the writing process.

A prime example of this is a new requirement in the section of the ROC where the QSA documents the list of people interviewed and/or observed.  The PCI SSC now requires the QSA to document what these people were interviewed about or were observed doing.  The purpose of this new requirement is to provide even more “proof” that a QSA did their job.  Another minor example of the PCI SSC trying to get “proof” of a QSA’s work effort is the increase in the level of detail being asked to document is the dates that QSAs were on-site for fieldwork.  In the past, QSAs were only required to document the period covered by the assessment.  However, QSAs are now being required to also document all of the dates of their fieldwork as well as the duration of their fieldwork and review period.

This is one of my biggest issues with the ROC process.  The PCI SSC refuses to adopt a more intelligent and cost effective reporting process of documenting requirement exceptions.  Instead, the PCI SSC requires QSAs to document their fieldwork process in the report.  As a result, an inordinate amount of time, paper and hence money is spent on what is really, in my humble opinion, a totally worthless effort.

I understand why this was required.  When the PCI SSC did not have the right to review a QSA’s work papers and other documentation, having such documentation in the ROC was the only way the PCI SSC, card brands and acquiring banks could assess whether or not a QSA had done their job.  Now that more than a year has gone by since the PCI SSC required all QSACs to include verbiage that allows the PCI SSC to review a QSAC’s work papers, putting all of this effort into a response writing requirement should no longer be required.  QSAs should be able to mark a requirement either ‘In Place’ or ‘Not In Place’.  If a requirement is ‘Not In Place’, then the QSA should document why the requirement is not in place and what the organization is doing to remediate the problem and when the remediation will be complete.  Such an approach would make the creation of the ROC much faster and would make the ROC much quicker to read and easier to understand.  This is the approach used in the accounting industry for their SSAE 16 reports and there is no reason why the PCI SSC could not adopt the same approach.

The PCI SSC continues to cling to this inane reporting requirement because it apparently is relying on the readers of the ROCs to “rat out” those QSACs that are producing inadequate reports.  I hate to be the bearer of bad news, but based on my review of ROCs from other QSACs that I have encountered over the last year, the “bad eggs” are not being weeded out.  Based on my interaction with acquiring banks and various card brands, there are a lot of ROCs that are not being read in detail.  And even those ROCs that are being read, most comments surround anything that is determined to have been ‘Not In Place’.  Occasionally we get a question about an ‘In Place’ item.  Obviously the current approach is not working and as long as the PCI SSC continues this approach, we are not going to build trust between the PCI SSC and QSACs.

I know that this is a dilemma for the PCI SSC, but it is something that needs to be addressed and soon.  Organizations that have to go through the ROC process are pressuring QSAs to reduce costs as much as possible not only due to our current economic conditions but also because of the thin margins retailers live on.  In order to keep the PCI compliance process relevant, the PCI SSC needs to get out in front of this issue.  The PCI DSS assessment process is very labor intensive, so the only cost savings to be obtained will be in making the process less labor intensive.

UPDATE: On the morning of September 20, 2011, the PCI SSC released the final version of the Reporting Instructions along with an FAQ.  These documents can be obtained from the PCI SSC Document Library under the Addition Documents – QSA heading.


11 Responses to “Draft PCI DSS v2.0 “Scorecard” Released”

  1. 1 QSA Andrew
    December 13, 2011 at 1:50 PM

    Is there a method in place to “Rat out” a bad ROC or is the only risk that your organization ends up submitting a bad ROC as part of the PCI QA process?

    I have just been handed a ROC as part of a new client engagement, where I am going to do the QSA assessment this year. The previous ROC was done by a different organization and it isn’t even close to meeting any standards. Most answers are in the form of “I reviewed documentation, performed interviews and reviewed system settings and the findings are appropriate to meet or exceed the requirements of PCI DSS 2.0”.

    Not that I like ratting out fellow QSAs, but it is difficult to come into an assessment and explain to the client that they may not pass this year without some significant additional work (and even changes to their process) only becuase they have now hired a QSA that follows the reporting instructions.

    • December 13, 2011 at 4:46 PM

      This is what the QSA assessment process is supposedly for, but I am not sure it has actually resulted in a QSAC being reviewed. As standard practice, we give our clients the QSA assessment form as we are required, but even with a lot of prodding on our part, I don’t think very many of our clients actually fill them out and submit them.

  2. September 19, 2011 at 10:46 AM

    Is this document still in draft or does someone have a link they could share?

    • September 19, 2011 at 11:12 AM

      As far as I know, it is still in draft. We are not sure if this document will be posted with all the other documents on the PCI SSC’s Web site’s Document Library or if it will only be distributed to QSACs. Hopefully we will find out this week at the PCI Community Meeting if it is finalized and get the final version.

  3. 5 Jay
    July 14, 2011 at 11:08 AM

    Will be discussing this in part at a talk at BSidesLV, on August 4, 2011. PCIGuru, if its ok, may I post a link here to the talk?

    • July 14, 2011 at 4:06 PM

      I have no problem with you promoting your session at BSidesLV.

  4. 7 dhawk
    May 13, 2011 at 7:38 AM

    Imagine what its like for PCI ISA’s that have to self assess and maintain a ROC with zero guidance and examples to work from….great

  5. May 6, 2011 at 2:50 AM


    Did this document “ROC Reporting Instructions for PCI DSS v2.0” vanish somewhere? I cant see it in the Document Library.

    Jake Eliasz

    • May 6, 2011 at 3:56 AM

      It is in DRAFT and has only been released to QSACs and others for review. It has not been made public as far as I know. And I am only guessing that it will be published in the Documents Library on the PCI SSC Web site.

  6. 10 Lola
    May 2, 2011 at 12:25 PM

    Thanks for the useful information,
    I’m not sure of something, maybe you know the answer, as far as I know, word ROC verion 2.0 doesn’t exit currently,
    So how the QSAs suppose to enroll thier PCI DSS 2.0 audits ?

    • May 2, 2011 at 5:48 PM

      If I remember correctly, the QSACs’ Key Points Of Contact, i.e., those people listed on the PCI SSC’s Web site, were sent a Word version of the PCI DSS v2.0 security assessment procedures by the PCI SSC. If that wasn’t the case, then we converted it from a PDF to a Word document using a PDF conversion utility we have and our administrative staff cleaned it up and proofed it against the original.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s


If you are posting a comment, be patient, as the comments will not be published until they are approved.

If your organization has a PCI opportunity, is in need of assistance with a PCI issue or if you would like the PCI Guru to speak at your meeting, you can contact the PCI Guru at pciguru AT gmail DOT com.

I do allow vendors to post potential solutions in response to issues that I bring up in posts. However, the PCI Guru does not endorse any specific products, so "Caveat Emptor" - let the buyer beware. Also, if I feel that the response is too "sales-ee", I reserve the right to edit or not even authorize the response.


May 2011
« Apr   Jun »

Enter your email address to subscribe to the PCI Guru blog and receive notifications of new posts by email.

Join 1,981 other followers


%d bloggers like this: