I want to shed some light on a troubling practice that I think people should be aware. The practice of which I am referring is QSAs that apparently spend little time on-site conducting their Report On Compliance (ROC) fieldwork. We are hearing a little too often from new clients that the amount of on-site fieldwork we are scheduling is significantly more than the time spent on-site by their last QSA. As a result, I wanted to take this opportunity to discuss why a certain amount of on-site work is required if a QSA is to successfully perform their duties. I am hopeful with the advent of the PCI SSC’s QA program that this practice will come to an end, but you never know.
About a year ago, the PCI SSC, as part of their QA program, issued a scorecard for evaluating a PCI Report On Compliance (ROC). The scorecard calls out five verifications that may be required by each individual PCI DSS requirement. Not every requirement requires all five areas be covered but a number of requirements do. Those areas are:
- Verified by review of documentation;
- Verified by interview;
- Verified by observation of system setting or configuration file;
- Verified by process, action or state; and
- Verified by network traffic monitoring.
Verified by review of documentation is just what you think. The QSA is required to obtain and review all relevant documentation related to PCI DSS compliance. Documentation is best reviewed before conducting any interviews and observations. The allows the QSA to minimize ramp up time on the organization’s cardholder processing environment and gives the QSA a better understanding of the environment to minimize clarifications and questions. Just as an FYI, based on my analysis, there are, at a minimum, 256 discrete documents required to complete a ROC.
As with documentation, verified by interview is also straight forward. A QSA is required to interview all relevant personnel that have knowledge required to complete the ROC. According to the scorecard, there are 129 interview topics that are required to be discussed. Interestingly, some PCI DSS requirements that you would think do not have an interview component actually do. This is an area that caught a lot of QSAs when they went through their QA reviews. Interviews can typically be conducted via conference calls and LiveMeetings, so there is no requirement to be on-site for these.
Verified by observation of system, setting or configuration file is defined as the assessor observed the device, component or server configuration file parameters, system setting parameters or any other parameters to prove that these parameters were set to produce the outcome specified To accomplish this, the assessor may use local system administrators, database administrators and application support personnel as needed. Based on the scorecard, there are 124 observations that are required.
Verified by process, action or state is one that trips a lot of us up. What the PCI SSC is requiring here is that the QSA observe a process, an action or the state of a device so that they can have proof that what the documentation and the interviews state is in fact what is actually executed by the personnel involved. There are 301 observations in this category required by the PCI SSC to be performed for a ROC.
Verified by network traffic monitoring is another tough one. It is tough mostly because a lot of organizations do not have a way to monitor network traffic that is acceptable. Yes, they are monitoring their network traffic, but they are not inspecting it with a tool like WireShark which is what is needed here. What the PCI SSC requires is that the QSA observe the network traffic to make sure it is encrypted and that inappropriate cardholder data is not present. There are 9 observations of network traffic required. These can be done remotely in most cases.
The QSA must truly observe all of these items. Given the number of items that must be observed, it just seems unrealistic to get all of it done in one or two days. It is possible to do some of these observations using conference tools such as WebEx or LiveMeeting. But when asked, our new clients tell us that no such meetings occurred for making these observations. So we are stumped as to how the previous QSA got the observations completed in such a short period of time.
Regardless of what the PCI SSC requires, there is a huge amount you can learn about an organization by spending time on site. An auditor can tell a well run operation just be looking around and talking to people. Looking around and talking to people takes time. In a lot of cases, these sorts of observations of an organization’s operations can point you in the direction of potential compliance issues that just require a bit of digging. You cannot get that kind of time if you are doing back to back to back on meetings to complete your observations. As a result, we have some concerns about whether or not these QSAs are identifying potential compliance issues that may exist.
In the end, these QSAs may be saving you money at the expense of your security.
PCI Guru 
Thanks for this post. The meat of what is required should not be news to anyone doing this work, but I do appreciate seeing some validation of what I have been trying to hammer home. One question though, is there a place where customers (e.g. merchants) can get the test matrix? An electronic copy would be helpful for QSAs as well, but I have had customers asking for this as well.