“Usually when PCI-compliant companies are breached, the real culprit is the assessor, the person who confirmed the company had met the PCI Requirements.” Jeff Multz, Dell SecureWorks
This is a very interesting approach for an employee at a qualified security assessor company (QSAC) to use to drum up business, toss all QSAs, including his own organization’s QSAs, under the bus. I know that is not what he meant to do, but that is certainly what he did with this statement in his posting a few days ago.
I think most QSAs know where Mr. Multz is coming from. He is more than likely venting over losses to QSACs that we all know are more interested in revenue generation than security. They further that goal by incenting their QSAs to do as many PCI assessments as possible in the shortest amount of time as well as identify opportunities for selling the QSAC’s security appliances to solve compliance problems. And to just pile on, they further their revenue generation by being the low cost provider through a focus on volume of work over quality. As Kurt Vonnegut said in Cat’s Cradle, “In this world, you get what you pay for.”
Getting back though to Mr. Multz and his statement that QSAs are responsible for all breaches, let us see how that plays out with a few breaches.
During the Target breach, it was the QSA that was socially engineered and gave away the keys to the kingdom and missed all of the alerts generated by the FireEye software. At Neiman Marcus, it was the QSA that missed the alerts for 60+ days that the malware was reinstalling nightly. It was the QSA that swapped out the points of interaction (POI) at Barnes & Noble for malware infested POI.
Sorry Mr. Multz, but it was employees and/or contractors at all of these organizations, not the QSA that had a part in these breaches and all breaches for that matter. I really do not see how you can hold a QSA responsible for the inaction and errors of employees/contractors. Organizations are not going to pay to have QSAs on site, 24×7, to babysit all of their employees to maintain compliance with PCI or any other compliance program. Not only that, no security framework is ever going to stop breaches, all they do is hopefully minimizing the impact when a breach occurs.
However, Mr. Multz was not done.
“The PCI Requirements were created so that organizations would focus on securing their networks, but many assessors only focus on meeting the requirements rather than security.”
From this statement it is painfully obvious that Mr. Multz does not understand what an assessment is about and how the assessment process works. The job of a QSA is to execute the tests as defined in the PCI DSS Reporting Template and report the results of that testing – nothing more, nothing less. Organizations are judged by a QSA as compliant with the PCI DSS whether they are just squeaking by or if they have a full on security program next to none. Organizations do not get “extra credit” or “atta boys” if they have gone beyond the requirements.
While the original intent of the standards was to focus on securing cardholder data, that got morphed by the wonderfully misdirected marketing job that was done by certain card brands before the PCI standards came together. For those of us around the security industry more than a decade ago, we advised Visa and MasterCard to stop pushing their cardholder information security program (CISP) and site data protection (SDP) standards as “The Way” that was going to stop breaches. We explained that, properly implemented, CISP and SDP should minimize the number of PANs obtained, but it would not completely stop breaches. It was only recently that the card brands started to realize this fact and stop pushing the PCI standards as a panacea of security. If you have noticed with the rollout of EMV, Visa, MasterCard and the PCI SSC have stated that EMV is not a “silver bullet” solution and in other statements stated there are no “silver bullet” solutions. That is a long way from a decade ago when their security standards were sold as the “be all to end all” for stopping breaches. Unfortunately for QSAs everywhere, that message is out there and we have to deal with it every day.
All of this is not to say that QSAs cannot and do not make recommendations to organizations regarding their security programs and how and where it needs to improve. I constantly make suggestions during my PCI assessments on how my client needs to improve their security posture. However, it is ultimately up to the organization to put such changes in place, not the QSA’s responsibility. If an organization chooses inaction, I will bring it up again and again. But as the old proverb states, “you can lead a horse to water, but you cannot make them drink”.
Where the PCI DSS assessment process truly fails is the point in time approach (with the exception of vulnerability scanning and a few other select requirements). To address that shortcoming, the Council has introduced the concept of business as usual (BAU) and it is my guess that we will see that concept placed into the standard in the next version. It will be then that QSAs will have to test PCI compliance over a 12 month period similar to testing procedures financial auditors perform for annual financial audits.
As a result, the inclusion of BAU as part of the PCI DSS will likely be the straw that breaks the camel’s back for a lot of organizations. This is because BAU will require organizations to track their compliance with the PCI DSS 24x7x365 as they should have been doing all along. But from experience, I can tell you that there is no organization I have ever encountered that was compliant with any standard all of the time because people make mistakes. As such, BAU is designed to shed light on those mistakes and require organizations to identify them and remediate them. For organizations just squeaking by, this will probably make PCI compliance truly impossible to achieve. If you are one of those organizations complaining about compliance with the current PCI DSS, just wait until BAU gets added. Organizations that are truly interested in security are already implementing BAU because they see the operational value in integrating security controls with their other business controls. BAU will show the true colors of those organizations that want security versus those that are checking a box.
And that gets me to Mr. Multz’s actual reason for his post, what makes a good QSA? Good QSAs understand that the world is not perfect nor is security. Good QSAs know that compliance with the PCI DSS does not and will not eliminate breaches. Good QSAs know that the goal of PCI compliance is to minimize security control errors, provide an ability to recognize security control errors as soon as possible and then remediate those security control errors such that the security controls are only non-compliant for the shortest possible amount of time.
But just because a company has such errors does not automatically mean that they are not PCI compliant. A good QSA only judges an organization non-compliant when the QSA has evidence that problems are consistently recurring and are not being corrected in a timely manner or corrected at all.
I appreciate Mr. Multz’s frustration but as a QSA I do not appreciate him tossing me under the bus with the QSAs that are doing a disservice to PCI compliance. Like any industry, there are good service providers and there are bad service providers. Those of us in this industry all know who the bad ones are and we hope they will get weeded out. But from my own long experience in consulting, that does not always happen.
So in my very humble opinion, Mr. Multz needs to suck it up and deal with it, but stop tossing QSAs under the bus in the process. QSAs are only the messengers.
PCI Guru 
I read Mr. Multz from a client perspective and he hit the nail on the head – a QSA can be just as guilty of a breach as the customer they “certify.” This is because ,as Mr. Multz points out, companies look at QSAs as judge and jury in their PCI complaince. Once the QSA finds the company compliant, even if they are not, management will proudly wave the certificate around and say “look at the great work we did!!! The QSA has found us PCI compliant!!!” And with that compliance activities go dead for another year with the occasional blip when an ASV scan is run, or when a new vulnerablity is found and needs to be patched. There is no talk of “how can we do PCI better?” the response whenever that question is asked is “We passed last year, we just have to do the same this year.” When the new version of PCI comes out there is the talk of “what do we have to do to maintain our compliance.” followed by the “No, we don’t have to that because that is too big, that is not what the requirement really says.” or “why are we doing this now, it was good enough in the past.”
Add to this that there is no incentive/or reason for a QSA to find a client non-compliant. In fact, it goes against the QSA’s best interest. You have stated in the past, as have many other “guru’s” on PCI – “If you don’t like what your QSA is telling you, find a new one.” My first QSA, quite rightly finds me non-compliant, I am just going to go to the next one on the list until one QSA finds me complaint, and changes are that won’t be long because the QSACs know that if I don’t like what I hear, I am no longer a client, and they lose my annual fee.
Mr. Multz states two key issues being the QSA missing things that they should have seen either because not enough information was given, or the QSA does not ask the right questions (does happen), or that the QSA suggests a technology to “help” with compliance that is not a right fit for the company.
A company can easily expose the gaping holes in the assessment process without any thought. The easiest one is checkbox auditing. Very easy to setup and get away with. This is where I give you “all” of my documentation broken down by requriement number, not by application/system. I can very easily leave out key peices of information, account management, vulnerability management, or log management. Because the QSA has information for all requirements, it is very easy to miss what is missing. Combine this with a short audit period and the problem is harder to spot. I do this trick, know that I am not really compliant, and promise to fix this “next year”.
Remember, I am a company. My primary objective is to make money. WOOHOO I get my peice of paper that says I am compliant, that is like getting an award in school for sitting in the same desk everyday. I may, or may not, display it proudly on my desk, but I know that it is a bogus award.
Your problem is that you are a great QSA and people ask for you by name, I am sure. This is because the companies that you assess WANT to be assessed, and know what they are getting into when you walk through the door. You see only the clients who want to be assessed, and how truly take pride in being PCI compliant. I would think that the the average, or even slightly above average company would shudder if the CEO stated that the “PCI Guru” was coming into audit their compliance efforts. This is because they know with you in the building all of their little secrets, which are very easy to hide from the average assessor, would come out of the woodwork and they would get a big non-compliant stamp.
I am not saying that the companies want to be non-compliant, but most companies can’t be compliant 100% of the time. This comes down to internal politics, funding, resource issues, and competing priorities. In a perfect world, and a perfectly designed network PCI should be easy, and all requirements should be able to be monitored in near real time with one console and a small team of analysts. We don’t live in a perfect world, and the sheer mention of PCI gets analyst’s and management’s backs up with the thought of “Oh here we go again, more work and more spending for no usable outcome!!!”
With this attitude the prevailing thought is to get the first guy who will agree that what we are doing is “good enough” and we will move on. The QSACs that allow this to happen are just as guilty as the company when there is a breach.
If the Council wants to make the system “better” and have a better way to ensure compliance, they should put the onus on the acqurier to engage the QSA on behalf of the entitiy being audited. This will make the QSA accountable to the acquirer and will be more likely to find compliance gaps. It will also force the organization to remediate those gaps instead of going off to find another QSA. The other solution would be for the Council to have a standard “agenda of an audit” which clearly outlines who the QSA will speak with and when. This can save the company from only presenting who they want the QSA to talk to and only presenting the information that they want to see.
You bring up some good points
QSACs are held accountable from a standpoint of gross negligence and malfeasance just as financial auditors can be found guilty. However, until one of the QSACs gets hauled into court over a breach and is successfully prosecuted for that, we’ll unfortunately have the situation we have now. The reason that has not occurred is, unlike a financial audit, the bulk of testing for PCI is at a point in time versus a financial audit which does testing over an entire year. At any given point in time any organization can be judged PCI compliant. It’s those other points in time as you point out where they get into trouble. That is why I think business as usual (BAU) along with testing over a year will change that situation and will put the QSACs willing to accept that risk directly in the line of fire.
That all said, I also think that the merchant side of PCI compliance is rapidly coming to an end as more and more merchants adopt P2PE and E2EE solutions. The only remaining entities that will have to be assessed will be the service providers and banks that still have cardholder data (CHD) and cannot get rid of it.
Good article. I have to say though that I respect some of the points that Mr Multz is making. I do think that QSA’s should be held more accountable if certified organisations are subject to breaches of CHD. Of course, nobody is saying they would be soley responsible – management and the employees are first and foremost responsible for the design and operation the controls – but if I’d certified an organisation as compliant and they were subject to a breach of cardholder data shortly after I would expect to have my assessment scrutinised and questions asked of my work. Taking responsibility for actions is the nature of being a professional – i’d hate to be thought of as “tick box” monkeys
Assessors are held accountable through the PCI SSC’s quality assurance (QA) program which reviews ROCs and their related work paper evidence collected to insure that the review work is being done and that the evidence supports the conclusions in the ROC. Trust me, the last thing a QSAC wants is to show up on the Council’s Web site as being “In Remediation” due to QA issues found during a review.
To hold assessors accountable for breaches is ludicrous. Assessors have no way to control the ineptitude of their clients’ employees and their clients’ actions. It’s no different than me holding you accountable for the actions of your neighbor. As I pointed out, organizations are not going to hire minions of compliance overseers to babysit their personnel comply with PCI or any other compliance program. That is too costly.
To properly work, organizations need to integrate security controls with other controls and monitor all controls. When controls go out of tolerance, the controls should be immediately examined and adjustments made. Someone independent of the controls (internal audit, QSA, ISA, etc.) should then periodically come through and test the controls to ensure that they are in fact working, are monitored and when out of compliance have been corrected or adjusted.
“Organizations do not get ‘extra credit’ or ‘atta boys’ if they have gone beyond the requirements.” Yes, but why not?
If we agree that there is no silver bullet for security, then we must also agree that security cannot be a pass/fail endeavor. Sure, it’s a pipe dream, but what if QSAs gave letter grades based on security posture and compliance? Businesses could be required to display their security grades like restaurants display their health department ratings – allowing consumers to support the businesses that do the most to keep them safe.
We’ve got a mountain of evidence pointing to the need foe security beyond compliance, so why don’t we do something that encourages and/or rewards it?
The only problem is whose risk acceptance model do we use for your grading scale?
Your willingness for risk acceptance could be radically different from mine and radically different from the organization being graded. Does that make any of us wrong? No. Just that some of us are more willing to accept certain risks and not accept other risks. That is just human nature.
This is why the risk assessment process is so important when assessing an organization. That risk assessment should give the assessor an idea as to what the risk acceptance tolerance of the organization is and how the assessor should couch their assessment. Where the assessment program comes into play is giving the assessor the minimum baseline so that if an organization says, for example, that their eCommerce operation is not a risk and does not need protection, the assessor can point to the work program and explain why eCommerce is a risk and what the bare minimum is required to minimize the risk.
Agree completely. Mr. Multz made me feel like he was throwing me under the bus as well. Rather than blame QSA’s, perhaps Mr. Multz should deal with the many clients I have who ask me every year: “Just tell me the MINIMUM I need to do to get through this….” As you so correctly stated: We (QSA’s) are just the messenger.