We are starting to hear rumblings within some organizations regarding conducting their own PCI Report On Compliance (ROC) assessment. While as a QSA I have a vested interest in not seeing this become a common practice, I can understand why organizations would want to at least examine this option. Particularly since the latest study from The Ponemon Institute says that the cost of a PCI ROC is between $225,000 and $500,000.
Remember, a ROC is only required when an organization is conducting a minimum of six million Visa, MasterCard or Discover transactions or two and a half million American Express transactions or one million JCB transactions. We are not talking about your Mom & Pop store around the corner or even your local chain. We are talking about organizations such as Wal*Mart, Exxon/Mobil and Amazon.com. The obvious reason organizations want to conduct their own PCI ROC is the belief that in-sourcing the assessment will save them money. However, as I will discuss later, there will likely be few if any cost savings.
The main benefit of conducting your own ROC assessment is that you use your own personnel. While the PCI DSS is silent on this subject, MasterCard and Visa Southeast Asia require the use of internal audit personnel. The use of internal audit personnel would seem to be the obvious choice because of their independence, their familiarity with assessing business processes against a standard and organizing and retaining the necessary documentation. Most major merchants already have an internal audit function, so there is no need to necessarily increase headcount. On the face of things, conducting a ROC internally sounds fairly straight forward.
As usual before you can save any money, there are some issues to conducting your own PCI ROC assessment.
- Most internal audit personnel are financial auditors not IT auditors and, as such, they have limited technology skills. And even with organizations that have an IT audit capability, the level of technical skills required by the PCI ROC is more than the skills available. The PCI ROC requires a significant amount of technological background in order to be conducted. Skills such as analyzing firewall, router and switch configurations, understanding access control mechanisms and conducting vulnerability scans and penetration tests are required for an assessment to be properly performed. As a result, your internal audit group will likely lack the necessary technical skills to conduct the assessment. This means investing in either training your existing audit staff or contracting for the skills necessary, neither of which is cheap.
- Until recently, there was a limited amount of PCI compliance training available for internal audit personnel. With MasterCard’s requirement of training internal audit personnel by July 2011 if an organization desires to conduct their own assessment, the PCI SSC has developed training for non-QSAs and that training will be available in 2010. However, such training is also not inexpensive. In addition, training is required annually to stay on top of the PCI DSS, so it is an ongoing cost.
- Since the ROC assessment process is an annual occurrence, the trained internal audit personnel will likely not retain all of the knowledge necessary to conduct the program from year to year because they do not conduct ROCs all year long like a QSA. This will mean ramp up time as well as the potential that the assessment will miss possible compliance issues or issues may be misinterpreted.
I have personally seen the results of a number of internally conducted PCI ROCs. None of them properly interpreted the PCI DSS requirements. In all cases the internal assessment judged the organization as being in full PCI compliance. However a review of the work showed that none of these organizations were PCI compliant. In some cases, the evidence collected in no way meets the documentation requirements of the PCI SSC let alone documented compliance with the PCI ROC requirement. And the worst problem, these PCI ROCs were all signed off by the organization’s Chief Financial Officer representing that the PCI ROC accurately represented their PCI compliance.
In the end, I am not convinced that organizations that desire to conduct their own PCI ROC will achieve the cost savings they believe they should achieve. And while on the surface it may appear to be cheaper to do the assessment internally, the skills and high level of training required to obtain a proper PCI ROC are likely beyond the investment required to get the job done. Finally, because your internal auditors are not conducting PCI ROCs every day there is a higher risk that your assessment may miss potential threats to your cardholder data environment.
I am not saying that you should not consider conducting your own PCI ROC. I am just pointing out that it is not a simple internal IT audit. It is also not as inexpensive as you might think. So please think about this before you go down the internal assessment road and get your organization in trouble.
UPDATE: The PCI SSC now offers the Internal Security Assessor (ISA) certification program for internal personnel. It is supposedly equivalent to their QSA certification program. Like the QSA program, ISAs are required to re-certifiy annually, so it is not a one shot deal. It is also not cheap at around $2,500 per person not including expenses. For MasterCard Level 2 merchants that want to do their own assessment after June 30, 2011, you will be required to either hire a QSA or have internal assessors attend and pass the ISA certification.
PCI Guru 
Hello, Guru do you have any insights about automatic ROC filling tools? thank you very much!
I have worked with RSA Archer, Sure Cloud and Total Compliance Tracking (TCT) that all have ROC reporting tools. They do not fill in the ROC but they do make managing the effort easier and provide dashboards to show status.
What do you think I should focus if I want to build a tool to help fill the ROC A LITTLE BIT FASTER?
I don’t believe it can be done short of an AI/ML solution that would read the documentation, create the samples, conduct the interviews, review all of the evidence and then put it in all of the correct requirements. Short of that, I don’t see anything that makes it any easier.
I suppose I fall more into working with Level 1 organizations on their PCI report on compliance and in my opinion, it’s very important for those clients to hire outside companies to guide them and ensure compliance.
While I agree with much of your analysis there are a few points to consider:
1) Until this February, the PCI SSC did not offer ANY training to Merchants. The Compliance Test Procedures Score sheet was not available to non-QSAs. A hard copy is provided as part of the Merchant training (not ISA) they started offering in February but still is not publicly available on the website. Having access to the Test Procedures Score Sheet would help merchants better understand whether they actually have the internal capabilities to perform an assessment themselves.
2)Performing assessments internally (ISA) is most likely best suited to organizations that have made serious efforts to isolate, segregate and minimize the footprint of the payment environment. At the other end of the spectrum there are companies I’m aware of that do have the in-house capabilities. IMHO they are a small subset of the Level 1s and 2s out there.
3) Even if an organization has the skillsets and capabilities in-house, there may be issues with independence of the individuals performing the assessment. This is something that can be addressed but becomes more difficult for high volume ecommerce sites with a smaller ratio of employees to transaction volume.
4) IMHO, if an organization (level1/level2) is going to perform an assessment internally then it should be required to prepare a ROC (and submit the score sheet) even if it is only required to submit a SAQ. There are almost certainly going to be some CNN moments if merchants are allowed to self assess based on submitting SAQs.
Obligatory disclaimer: My opinions are my own and do not necessarily represent the view of my employer.
The question remains, if a large organization cannot accurately evaluate it’s PCI Compliance, then how will a mom and pop shop do it?
Large organizations have the problem of complexity and issues with only partially assimilated acquisitions. In addition, in some cases merchants have so decimated their IT staffs that they have lost key people that understood the POS and cardholder data environment.
On the whole, most large merchants have fairly well run IT organizations that can go through a ROC like any other compliance effort such as Sarbanes Oxley. Where the problems come in is with the “Oh, we process cardholder data there?” issues. Because of their size and complexity, a lot of processes end up in the nooks and crannies of the organization and have to be dug out.
In the case of the Mom & Pop, their environments are not complex. If you think about it, they cannot be complex because they would get eaten alive on the expense of having such an environment. In addition, such organizations only have to go through the self-assessment questionnaire (SAQ), not the full blown Report On Compliance (ROC). SAQs A, B and C are straight forward and should not create any issues. However, anyone going through the SAQ D (aka ROC Light), I highly recommend that, at a minimum, a merchant put a QSA on retainer to answer questions as they come up going through the SAQ D.
With all of that said, in the end, all the PCI compliance process is going to do is get organizations focused on security and the fact that security is a never ending process. It is not rocket science, the PCI standards are just best practices that are the minimum any organization should be doing to ensure cardholder data security.
I work with the lowest level of mom & pop shops. These are people who are sole proprietorships with no other employees. Many of them run small e-Commerce websites that only pull in a few hundred dollars a month. Just saying “PCI” has their eyes glazing over. Although, I will admit that I’m moving away from these types of clients. I do recommend to them that they only use hosted payment solutions, and if they are willing to just use a PayPal account and have no merchant account at all.
Some listen, others do not.
I also see the Web development crowd just now finally starting to look at PCI but not really following through. I did a post over at xemion.com about the importance to themselves and to their clients about PCI but I don’t know if it’s actually making a dent.