A lot of people are always calling their PCI assessment an audit. However, certified public accountants (CPA) would tell them that there is a vast difference between the two.
An assessment is defined as:
“… to measure something or calculate a value for it. Although the process of producing an assessment may involve an audit by an independent professional, its purpose is to provide a measurement rather than to express an opinion about the fairness of statements or quality of performance.”
The key point of difference between an audit and an assessment is the “opinion”. While people would argue that a QSA is judging them PCI compliant, judging is not the same as offering an opinion. The reason is that a PCI assessment is done as of a point in time, not over a period of time. Yes there are some tests in the PCI assessment process such as with change management and vulnerability scanning that are tested over a period of time. However the bulk of testing for PCI compliance occurs at a given point in time, most often the time of the assessment. Such limited testing does not provide the basis for opining on any security program.
An audit is defined as:
“Audits provide third party assurance to various stakeholders that the subject matter is free from material misstatement.”
As an example, a financial audit comprises testing and sampling that is performed over the audit period, typically a period of one year. In addition, an auditor must conduct testing such that they can provide reasonable assurance that there are no material misstatements during the audit period.
The first important phrase is “reasonable assurance” and it is defined as:
“Acknowledgment that it is not possible to assert absolutely and certainly that an event will (or will not) occur.”
Going back to our financial audit example, what reasonable assurance points out is that it is impossible for a financial auditor to essentially redo all of the work performed by an organization’s accounting staff to prove that all of the transactions performed over the audit period were processed exactly as they should have been. As a result, an auditor creates tests of processes and controls and then generates sample sizes based on the risk and the number of transactions performed throughout the audit period such that it is likely the procedures will identify any errors or omissions. If the testing of those samples does not result in any errors or omissions being discovered, then the auditor believes that there is reasonable assurance that there are no material misstatements. If errors or omissions are found, then the auditor must increase their sample size to determine if the errors or omissions are systemic in nature (i.e., the process/controls are broken) or if they are true mistakes. The bottom line about reasonable assurance is that everyone (client, auditor, auditor’s certification body) agrees that if processes/controls are broken, the auditor’s procedures for detecting those breakdowns are sufficient to identify them.
And now we get to what we mean by “material”. Materiality is defined as:
“Information is material if its omission or misstatement could influence the economic decision of users taken on the basis of the financial statements. Materiality depends on the size of the item or error judged in the particular circumstances of its omission or misstatement. Thus, materiality provides a threshold or cut-off point rather than being a primary qualitative characteristic which information must have if it is to be useful.”
Materiality is a judgment call by the auditor based on an examination of risk and whether that risk could result in a misstatement of facts in the financial reports. Years ago we were working with a large client. We relied on their external financial auditor and their assessment of the point of sale (POS) systems user management and access controls audit for Sarbanes Oxley (SOX) to satisfy some of the PCI requirements 7 and 8 testing. However, two years in, the external financial auditor deemed that the controls surrounding the POS systems were no longer material to the financial audit and stopped their testing. As a result, we were left with having to assess the user management and access controls ourselves.
At this point, I am sure a lot of you are wondering other than getting you all to stop calling PCI assessments “audits”, what are you saying?
Business as usual (BAU) is going to change how PCI assessments are performed. Since organizations will have been required to embed controls and monitoring into their business processes, the PCI assessment will likely be changed into a true audit. The reason will be that BAU will require record keeping that will allow a QSA to test for exception conditions for PCI requirements and ensure that the exceptions were corrected and how quickly they were corrected.
While I know a lot of organizations will complain about this sort of process, this is how a proper information security program should work in the first place. Information security controls and monitoring should be embedded into all relevant processes in an organization. Business management and information security should be monitoring and measuring the controls and, when an out of compliance condition occurs, the appropriate actions are taken to either bring the controls back into compliance or the controls are updated/changed to reflect changing conditions.
In rare situations, an organization might find that a control is no longer required because changes have made the control obsolete. This is typically the case when an organization introduces new application software or new network architecture and the control environment wholly changes and controls end up as inadequate, monitoring for the wrong condition(s) or in the wrong place.
BAU is not a penalty; it is an approach to keep an organization on its security “game” by embedding controls and monitoring into the relevant business processes. By doing so an organization then has a mechanism in place to maintain its information security compliance as close to 100% as is humanly possible.
But that will be the rub. This approach will likely find a lot of organizations identifying that staying compliant is nearly impossible because of constant out of compliance situations that will be brought to light. The side benefit of BAU will be to demonstrate just how important security training for all personnel is and that security technology is not the biggest cause of security issues, it is human error. BAU statistics will provide the focus for security training of personnel to address shortcomings. In theory, that training should minimize the security issues from human mistakes and make an organization’s security posture all that much better.
Implementing BAU will take time. It is also not a silver bullet. Like its financial audit brethren, errors and omissions can still occur under BAU, but they are more likely to be caught and addressed before they can spin out of control.
PCI Guru 
I think we all can agree to disagree, but I’m not aware of any case law where the “audit attorneys” went knocking on someone’s door about misuse of the word. Please edify me.
For the PCICo it’s simply a different type of audit. Even the QSAs can’t agree here.
Except for pure numbers which must balance, audits and assessments all require a certain degree of opinion by auditors, making them both subjective in nature. Heck, I can’t even get QSAs to agree on a PCI DSS Requirement 11.3.2 approach to my CDE. I call that the “snowflake factor.” There, I said it.
The AICPA doesn’t control words; it controls audit approaches and methodologies in its domains.
Put away AT-101 and come back to reality, Jeff.
After too many years working for large auditing firms, I have apparently drunk too much of their Kool Aid. LOL!
All I can tell you is that CPAs control more than you think because of the Recessions from the late 1870s forward and the Great Depression. The were legitimized by federal and state legislatures in the mistaken belief that their auditing processes would solve the huge swings in economies.
Worldwide the AICPA AT-101 standard and its sister International Federation of Accountants (IFAC) standards are considered the audit methodology “gold” standards. No different than how the world views ISO standards. The purpose of those standards is to guide auditors in how to audit such that they reduce the likelihood for fraud, errors or misstatements of fact to go unnoticed. The idea being that a requisite amount of sampling can identify any errors or misstatements and determine if those errors or misstatements are systemic in nature (indicating that one or more controls are not functioning as designed) or just mistakes.
As I stated earlier, we looked into conducting PCI assessments under the AT-101 standard and our prospective clients found the cost and amount of work so daunting that such an audit never got off the ground. However, if it had, I think a lot of the issues that go unnoticed or not found would disappear because of the amount of testing that would be conducted in all areas of the organization would have identified a lot of the little things that create huge security problems.
I don’t know why the PCICo looked into performing PCI DSS audits under AT-101 when, in fact, the PCI DSS is in itself a globally accepted audit standard with scoping, sampling, testing procedures, et al, built in.
AT-101 is not an audit program in the sense of the PCI DSS. What AT-101 is are the documented rules as to how an audit must be performed to be considered an ‘audit’ and to allow for the generation of an opinion. As such, it discusses things such as how to scope an audit, effective sampling, how to conduct testing, procedures to follow, how to assess internal controls, etc.
In my opinion, having the definition of “audit” baked in laws is irrelevant. If we look at most dictionaries, there’s always more than one definition of “audit”. The first one is always about reviewing financial records which is what CPA’s are tasked to do but there’s always another definition that read like “a careful check or review of something”. The term can be used by a civil engineer to test the structure of a building or by a safety officer to test how safe a situation is. Many health care systems in the world use the term “clinical audit” to assess and improve their services. It’s fundamentally comparing something to a reference.
I agree with Stephen on this one. As QSA’s, we’re expressing opinions on whether or not the assessed entities is compliant with PCI DSS (or other PCI SSC standards) just like doctors express opinions on health issues. Of course, these opinions are not to be considered as an opinion according to AICPA’s standards.
You can think it’s irrelevant until the attorneys show up at your door. That was what the PCI SSC was avoiding because if it had been called an audit, then only CPA firms could have done the work in a lot of States.
CPAs do more than just audit numbers, they audit procedures and compliance with procedures. If that sounds familiar it should because that is what a PCI assessor does. However, CPAs are required to do it with much more rigor because of AT-101 and the fact that they opine on their results. Under AT-101 auditing standards, the vast majority of firms would be paying a small fortune to be told they are non-compliant year after year. Merchants are not willing to pay a small fortune for anything because their margins are too tight, so the card brands had to come up with an approach that would not cost a King’s ransom but would give reasonable insight into their card processing operations. Imperfect I agree, but it was the best that could be done.
I’m sorry you don’t like the fact that CPAs control the terminology, but that is how the world works at the moment.
The council has constructed a ROC process that is in affect an opinion from the QSA. Which is why the U.S. firms of the big 4 accounting firm are reticent to issue ROCs. The council’s reporting standard is simply incongruent with the notion of an assessment. The large acctg firms would need to perform actual audit testing to be able to issue a ROC. Thus providing 3rd party assurance in the form of an opinion. Or the council could redesign the reporting standard to align with an assessment — which is what is actually being done by the QSA community.
A QSA does NOT issue an opinion because an opinion requires a formal audit, which the PCI process does not meet because it does not comply with AICPA AT-101. AT-101 would require an amount of testing that the vast majority of merchants and service providers will not pay to have done. With the coming of business as usual (BAU), the amount of testing to meet AT-101 will be possible and could likely be handled in a reasonable amount of time. However, BAU will cause heartburn in a different way because it will require organizations to keep records year round of what they did and why.
Au contraire, my friend! When the QSA checks that in-place box they are imparting their opinion that the presented evidence or their testing results support their opinion that the control is in place. Just like any other audit. There is no difference.
From the English language dictionary perspective, you are correct. From an audit perspective, you are not. The reason is that the AICPA and state CPA societies “own” the terms audit and opinion because they had their definitions put into most state’s law regarding audits and opinions. As a result, QSAs do not offer an audit opinion. QSAs offer the results of an assessment. I know it’s a semantics and “depends on the definition of what the word ‘is’ is”, but that is how you need to look at it.
It seems to me we’re mixing fruits with our vegetables here, so I agree with #shift4sms. CPAs and QSAs audit against completely different controls.
The term “material” in the context of an audit points to generally accepted accounting principles and accounting key controls. Yes? The PCI DSS audit involves security key controls. While the term “assessment” is kinder and gentler, at the end of the day it’s a PCI ROC audit. The security controls are in place, not in place, or [with a CCW] may be in place based on the QSAs’ varied professional opinions and/or “reasonable assurance” the testing samples are a representation of the entire audit cycle.
I agree with most of your stuff, sir, just not this one.
After 25+ years working for CPA firms, like it or not, CPAs control the definitions. That’s thanks to the fact that they had their definitions and approaches baked into most States’ laws regarding auditing. In almost all states, if you want to conduct a true “audit”, then you must follow the AICPA standards of AT-101 as well as any related standards. In some states, only CPAs are allowed to produce any work product that is called an “audit”. This is what drove the PCI SSC to call their work products “assessments”.
But it is AT-101 that will drive the testing to the point of being extremely painful due to the concepts of reasonable assurance and materiality. Companies that go through SSAE 16 SOC 2/3 audits always complain the first time about the amount and types of testing done by the auditors. That is because most organizations do not keep the records and details necessary to prove they are actually following their information security policies, standards and procedures. IT is typically not as disciplined as the financial side of the house when it comes to such auditable record keeping.
Funny, I wrote on this exact same topic back in 2010 and I used many of your exact same arguments but I came to an almost opposite conclusion. IMHO it’s an audit mislabeled as an assessment.
http://paymenttidbits.blogspot.com/2010/10/is-it-audit-or-assessment.html
It does not offend me calling it either. I just get annoyed when people correct others over the term.
Other than on this topic, keep up the good work. 😉
The issue is that the testing done under the PCI DSS Reporting Template is not sufficient under the AICPA auditing standard AT-101 to opine. Therefore PCI is an assessment, not an audit.
That said, I know when we looked at enforcing AT-101, the required amount of testing would have increased the cost of a PCI assessment by five fold. As a result, there would be no merchants that would be willing to pay that much annually. Which is all the more reason with BAU coming to shrink your cardholder data environment (CDE) as small as possible.