The Problems With Big Data

I developed a presentation on big data for a series of education sessions I am delivering for a financial institution trade association.  As I was putting the presentation together, I realized that this was probably a good topic for the blog as a lot of you are running headlong toward big data either for log data analysis or just as the next “I need to be doing this” technology fad.

Most people do not realize that big data has been around for quite a while, relatively speaking.  Google, Yahoo and similar Web service providers have been dealing with big data for years.  But it was only recently that big data management frameworks such as Apache Hadoop, Google BigQuery and MongoDB became publically available through shareware, commercial solutions and software as a service (SaaS).

So we are all on the same page, let us define “big data.”  The best definition I have seen for big data is from Gartner.

“Big data are high-volume, high-velocity, and/or high-variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization.”

Examples of big data include information such as Web search results, electronic messages (e.g., SMS, email and instant messages), social media postings, pictures, videos and even system log data.  However, it can also include credit/debit card transactions, check images, receipts and other transactional information depending on the source of the information.  As a result, big data can easily end up in-scope for PCI compliance.

The first problem with big data is that organizations are expected to know the data going into their big data repositories.  The reason “know your data” is so important with big data is that it comes from potentially a wide variety of sources such as:

• Social media
• News feeds
• Images
• Streaming media (audio and video)
• Documents
• Messaging systems
• Audit logs
• Transaction logs
• Web sites
• System logs

With this diversity of information sources, it is anyone’s guess as to how much sensitive information could end up in an organization’s big data repositories.  But worse yet, anyone in the big data field will tell you that you need to anticipate all potential sensitive data so that you can secure and protect it appropriately.  Why is this?  Because big data tools allow everything to be searchable: text, images, audio, video, anything.  So if you do not want the information to be searchable, then you need to identify it and either encrypt it, truncate it or remove it so that it cannot be found.  As those of you that are using data loss prevention (DLP) or other tools to find cardholder data (CHD) stored on your systems are well aware, finding CHD is not as easy it would appear.  As a result, finding it in big data could be the ultimate finding the needle in the haystack game.

The next problem with big data is that the security tools for big data are very early in their development and, in some cases, are really bolt on after thoughts that use constantly running queries to find and protect the sensitive data.  While a lot of vendors claim they can secure data at the “field” level, I have spoken to a number of clients going through big data implementations that tell me this is a pipe dream at the moment.  As such, in all cases I am aware; big data protection is currently accomplished through totally encrypting all of the data and very severely restricting access.

Which begs the question, how can big data be PCI compliant?  Well it can be PCI compliant as long as: (1) access to it is extremely limited (only a very few people have access such as two to three), or (2) you are able to truncate, remove or encrypt the CHD contained in your big data hive(s).  Given that accurately locating CHD can be nearly impossible with current techniques, I just do not see option 2 as currently viable, so extremely limited access is your only workable option.  Even then I seriously doubt that big data is a good place for CHD to be stored as severely limiting access is also not viable given why big data is being implemented.  As a result, big data is probably not a good place for sensitive data, cardholder or otherwise, until the security tools catch up.

I think my readers will recognize why their log data would fit into the big data category.  It definitely has high volume as it typically comes from a large pool of devices that are generating potentially hundreds to thousands of entries per second which also satisfies the high velocity requirement as well.  And the high variety aspect is also satisfied as log data from a Cisco or Juniper device is nothing like log data generated from a Windows or Linux server or any other litany of network devices.  However, while the likelihood of CHD should be nonexistent in log data, CHD can still end up in log data due to debugging being performed.  As such, I am not certain I would be comfortable with log data in a big data hive until the maturity of the security tools is better.

The bottom line at the moment is I just do not see big data being ready for PCI compliance at this point.  I am sure that someday big data will be capable of being PCI compliant, but not at this time.  So all of you that need to be PCI compliant running towards the light at the end of the big data tunnel.  That light you see is not then end of the tunnel but an oncoming train about to run you over.

Compliance, Compliance Testing and Security

I was recently on a Webinar presented by a major security vendor and one of their points was that executive management is finally starting to realize that compliance does not equal security.  If you read this blog regularly, you know I really do not like the phrase “compliance does not equal security” and I view it as a convenient dodge by those who use it as a way to weasel out of their responsibilities.

But during this Webinar I had an epiphany regarding this topic.  It is the confusion between security, compliance testing and reporting and the act of compliance by your technology, employees and business partners with your organization’s security policies, standards and procedures that is the problem.

I know I am just asking for flame mail with this post, but I am so tired of people looking to blame everyone but themselves about their inadequacies surrounding information security.  As I have done before, to paraphrase Tom Hank’s character in ‘A League of Their Own’, “There’s a reason security is hard.  If it wasn’t hard, everyone would do it.”

Security is not always easy, particularly when upper management does not have buy in.  But even when upper management supports security efforts, I have seen security personnel not take advantage of that fact and get the job done.  Security does not have to be hard, but it does take more than just slamming some firewalls and intrusion prevention gear down, tossing a SIEM into the mix and thinking you are done.  Security is a never ending journey because someone is always coming up with new ways to attack you.

Anyway, to start off, let us take a look at some definitions first so we are all on the same page.

Compliance is defined as:

“Conformity in fulfilling official requirements.”

“Official requirements?”  Could that possible mean your organization’s security policies, standards and procedures?  You bet.  In this instance, we are talking about those that correspond to the PCI DSS, but this also applies to ISO 27K, FISMA, HIPAA, GLBA or any multitude of frameworks and regulatory requirements.

Conformity is defined as:

“Compliance with standards, rules, or laws.”

Based on these definitions, security is all predicated on complying with what are deemed an adequate set of security policies, standards and procedures.  Conversely, if you are not complying with an adequate set of security policies, standards and procedures, then your organization cannot be as secure as it could be.  As a result, compliance has to equal security as long as the security policies, standards and procedures are considered adequate.  Therefore security professionals that quote the mantra, “compliance does not equal security” either have a problem with the compliance side of the equation (most likely) or with the standards/frameworks (the dodge).

Over the years there have been a lot of discussions about the PCI DSS, ISO 27K, FISMA and other security frameworks and whether or not they are adequate.  The important thing to remember is that all of these standards or frameworks are merely ante into the information security game.  They are the bare minimum or a baseline to get to a basic level of security.  Should you being doing more?  Definitely, but what those efforts beyond the standard/framework are depends on what you are trying to secure, your network and application architectures and a multitude of other factors related to your computing environment and how it is used.  Those are factors that cannot be taken into account by any standard/framework because they would start to become impossible for others to follow and implement.  The bottom line here is that if you want someone to tell you exactly what to do to secure your networks and applications, go hire a consultant you trust and they will tell you everything you want to know.

The rub in all of this is that, based on the breach reports from Verizon Business Services, Trustwave, et.al. as well as compliance testing reports I have reviewed, none of you out there are 100% compliant to begin with, let alone even close.  Every organization I am aware has problems complying with the basics, let alone with any advanced security requirements in the published standards/frameworks.  So if you cannot comply with what you already have, explain to me how a different framework is going to change that fact unless it is less stringent than the framework you are already trying to use?  And if that other framework is less stringent, while that may solve the compliance issue (which I seriously doubt), exactly how is a less stringent framework going to make you secure?  The answer is that it will not make you secure.

What security professionals struggle with is that compliance is a never ending, 24x7x365 effort.  Drop your guard for an instant and it can be game over.  But provided your security policies, standards and procedures are appropriate and detailed (the reason why you want to use an appropriate standard/framework), your organization is not as secure as it can be unless your personnel and devices comply 100% of the time with every defined security policy, standard and procedure.  If you want confirmation of these facts, again, just look at the breach analysis reports year after year.  The reason there are breaches is because of non-compliance with one, but usually more, of an organization’s security policies, standards and/or procedures.

This brings me to the rumblings of late regarding a rethinking of defense in depth.  Defense in depth is predicated on using layers of security devices and controls to minimize the risk that a security incident occurs not to completely prevent an incident although you might get lucky.  For example, firewalls are the sledge hammer of security tools.  However, because we need to have ports open for outsiders to access applications, we follow our firewalls with intrusion detection/prevention devices to ensure that no one abuses the protocols used by the ports.  We follow that up with monitoring of log data from the firewalls, IDS/IPS, routers, switches and servers to identify any “sneaky” attacks using the protocols we allow.  The layers are there to cover the various holes we need to have in order to make our networks and applications function.  The tighter and smaller we can make those holes, the more secure we will be, but there will still be some amount of risk.  So we bring in more layers to cover those risks until it is more expensive to address the risk than to accept the risk.  That remaining risk is the residual risk that we therefore manage and control through detection and correction.

The other thing defense in depth relies on is the control triad.  The idea being that, because you cannot entirely prevent every security incident, you need a way to detect the incident so that you can take action to stop or minimize the impact of the incident.  You follow that up with periodic assessments of your control environment to identify and correct any deficiencies or improve your program based on new information regarding security.  The follow up assessments can be activities such as a root cause analysis (RCA) of an incident, an internal audit of user accounts and user rights or brining in a network security team to assess your security architecture and controls.  All of these activities will result in findings and recommendations to make your security systems and controls better.

And that brings us full circle to the PCI assessment.  It is merely a tool used by the acquiring banks, card brands, processors and others to obtain reasonable assurance that your organization is doing what it can to minimize the possibility of a breach of cardholder data.  It is not meant to be, nor could it ever be, an absolute complete assessment of an organization’s security posture and therefore provide absolute assurance that a breach will not occur (even though the PCI SSC and card brands tend to imply that fact).  Compliance assessments are only a snapshot of personnel and device compliance at the time the reports were written.  This is no different than going to the doctor for your annual physical which results in a snapshot of your health at that point in time.  It is not that those compliance reports are worthless; they just need to be referenced and used properly based on the fact that they are a snapshot.  Just as your doctor will tell you to lose weight or stop smoking, compliance reports provide recommendations on where you can make improvements or adjustments in your policies, standards and procedures based on what compliance evidence was found, or not found, during the assessment.

So, what are the lessons to be learned?

• Security is not and never will be perfect; there will always be residual risk that must be managed and controlled.
• Compliance does equal security, at least as best as your preferred standard or framework defines it plus whatever enhancements you have made.
• Compliance assessments and reports point out where your organization was not compliant and needs to do better, not to prove your organization is secure.

Use the tools at your disposal correctly, stay current on threats and monitor your security posture and you will likely live a long, prosperous and secure life.

Keep hiding behind “compliance does not equal security” and you will forever be living off of your “luck” until it runs out (usually sooner rather than later).

Security And Compliance

I have written a lot about this topic over the years and was recently reviewing my Compliance Is Not Security – Busted! post and the comments that came in regarding it.

A theme of a number of the comments was that compliance does not equal security.  DUH!  I have never once said or even implied that compliance equaled security as – yes, here it comes – security is not perfect!  However, if you are complying with any security program/framework such as the PCI DSS, ISO 27K, etc., then you are likely more secure than those who are not.

Security technology such as firewalls, routers, servers, applications, etc. can all be set up with rules that are complied with 100% of the time, day in and day out, no exceptions.  The problem comes down to people who are fallible.  Their compliance is never 100% and you are probably lucky to have anyone above 90%, no matter how much security awareness training you do.  As a result, in organizations that are truly complying with the PCI standards, this is where the security breach starts, with people for one reason or another.

No, I am not necessarily talking about social engineering, although social engineering is growing because of the fact that organizations have invested a lot in security technologies yet people are fallible.  People can be the root cause because of any or all of the following.

• How dare you do that to me!  This is the most obvious of the people issues that comes to mind.  Face it, when backed into a corner, people lash out just like a trapped animal.  The supposedly wronged party wants their proverbial “pound of flesh.”  They get that pound of flesh by hurting the organization that has just hurt them.  This can be as minimal as taking office supplies to downloading databases to a USB drive as they empty their desk.  Obviously, a database, network or system administrator’s access is much different than a clerk’s.  However, if your security is minimal on the inside as it is in most organizations, the clerk may actually have better access than the administrators when it comes to sensitive information.  Such a situation may not be the fault of the administrators, that old version of POS or ERP may not have the ability to be more granular regarding access to information.
• Over inundated with alerts and cannot identify real alerts from false positives.  This typically occurs when an automated tool is implemented but never tuned to the organization’s environment.  In this sort of an environment, finding real alerts can be like finding a needle in a haystack when there are thousands of alerts an hour scrolling by on the screen.  This usually makes management wonder why the tool was needed in the first place.
• Saw an alert and ignored it.  We see this most often coupled with the aforementioned inundation issue.  The other most common version of this issue is with internally used SSL certificates that were generated incorrectly or use a default certificate supplied by the application.  Users then see the “There is a problem with this Website’s security certificate” or similar error message in their browser whenever these flawed certificates are encountered and become conditioned to ignore the error message.  Over time, they become conditioned to ignore all of these sorts of messages, including those for malware infected Web sites and, surprise, you have been compromised.  I have lost count how many people have said to me, “We just ignore those alerts because we know they are false positives.”
• Saw the alert but got side tracked and never came back to it.  This is a problem we see all of the time.  For example, the person that monitors the network is also the person that manages the network and configures the network.  An alert comes in and the person begins a root cause analysis (RCA) only to get pulled away because a remote facility is offline.  The offline issue gets resolved, but other issues come up as well as meetings and telephone calls and the person never gets back to the RCA for the alert because there is no “tickler” to remind them to go back and complete the RCA.  In the meantime, the attacker has gained their beachhead and is probing the network for whatever value it may contain.
• Just did not put together all of the pieces to know they were compromised.  Like the reasons 9/11 occurred, most organizations do not correlate all of the potential incidents occurring in their networks and therefore do not understand that there is an active effort to compromise their network or that they have already been compromised until well after the incident has caused damage.  The reason this is important is that once an attacker is inside your organization’s security perimeter, it is typically game over because there are few controls to prevent access and identify that data is being taken.

If you have read the Verizon Business Services Data Breach Investigations Reports (DBIR) over the years you know how the bulk of attacks get inside, they are the result of people.  For the last two years, the DBIR has used the VERIS Event Threat Grid to show how breaches occur.  Across the top of the grid are the categories; Malware, Hacking, Social, Misuse, Physical, Error and Environmental.  The Social, Misuse and Error categories imply mistakes or deliberate acts of people.  If you read the definitions on the VERIS Web site, Malware is also very people centric as is hacking.  Surprisingly to some will be that the Physical and Environmental categories also have a good number of people errors.  Based on just a quick read, it looks to be that about 60% to even 70% of all of the incidents categorized by VERIS has some form of people error component.

Since we are not going to get rid of people in our organizations any time soon, what are you to do?

• Admit that people are the problem and focus your security measures accordingly.  Every 12 step program says the first step is to admit the problem which, in this case, is that people are fallible.  As a result, we need to construct our security measures such that this fallibility is minimized as much as possible.  One of the best solutions is to integrate alerts into your help desk or change management system so that a ticket is generated.  Those tickets need to have an escalation process behind them so that if they are not investigated within a period of time, they are bumped up to the next higher rung of management and that escalation continues until the tickets are finally addressed.  This way there is visibility for the alerts should they slip through the cracks.  As a side benefit of this approach, you gain statistics to reinforce why you need more staff and/or more/better tools.
• Strengthen your internal security measures.  As things stand, once inside most organization’s security perimeter, there is very little that stands in the way of an experienced attacker getting the data they desire.  Regardless of whether it is an insider attack or an attacker has managed to get inside, there is already justification for organizations to beef up their internal security measures.  To address this problem, I would recommend the security architectures as documented in my Fort Knox approach, Forrester’s Zero Trust Model or McGladrey’s Ultra Secure Network.  But most organizations do not have the infrastructure architecture, the application architecture or even the will to take such approaches.  But that does not excuse an organization from just saying they cannot do anything.  If anything, most organizations could vastly improve the monitoring they do on their internal networks.  Monitoring needs to be coupled with reducing the total number of ports that are open between network segments.  Most internal networks do a terrible job of this because of a variety of factors including applications people that cannot tell what ports need to be open to avoiding operational issues by just leaving things open.  Another area of improvement is reviewing user access rights on all systems and applications, not just those in-scope for PCI compliance.
• Constantly tune your alerting system(s).  Just as attack methods are not static, neither are networks, systems and applications.  Changes are occurring all of the time in an organization’s IT environment, yet if you ask the people running the SIEM about changes, nine times out of ten, nothing seems to be changing other than requests to look for a new signature or anomaly.  There is a belief in the SIEM user community that a SIEM’s update process is making the necessary changes in the policies that ship with the SIEM.  To a certain extent SIEM solutions are similar to anti-virus and malware solutions.  However, because a SIEM monitors log data and the log data provided varies greatly from organization to organization, each organization needs to periodically review and adjust their alerting criteria to make sure that it reflects the organization’s operating environment and not just some template from the SIEM vendor.  If an organization is not reviewing its SIEM alerting rules based on the changes made, at least quarterly, then it is highly likely that the SIEM is not alerting properly.
• Establish separate consoles from your SIEM for network, system, security and application administrators.  What a network administrator is looking for is vastly different from what an application administrator is looking for and what any particular group might be looking for to generate an alert.  As a result, to have only one console is really silly and non-productive.  Yet time and again, we see SIEM implementations with just that, one console and everyone being driven by email or SMS alerts.  The people alerted then have to get to the SIEM to find out what exactly triggered the alert and then determine what to do about it.  Having your own console view simplified things by only listing that viewer’s alerts and no one else’s alerts.  This allows people to focus on their problems and not the whole organizations problems.  The idea behind the single console is that if everyone knows what is going on overall, then correlation would occur because everyone sees everything.  While you would think that would be the case, in reality, people just want to fix their problem and move on, not the entire organization.  Which leads to my last point.
• Watch the overall alerting picture so that correlations can be made.  According to most sources, today’s attacks are becoming more sophisticated and multi-pronged in their approach.  For example, while most DDoS attacks are just to be a pain in the posterior to the target and disrupt access to the target’s Web site, there are those DDoS attacks that are used as cover so that people inside are blinded to the real attack(s).  Whether or not the DDoS was a decoy depends on what other events or incidents occurred during the DDoS attack, if your alerting system did its work.  Higher end SIEM solutions can provide basic correlation rules, but most SIEM solutions require the end user to develop those correlation rules.  It is these correlation rules that help organization identify these more sophisticated attacks.  That said, these correlation rules do not have to be very sophisticated.  For example, during a DDoS attack, you really only need to look for malware attacks, failed authentication attempts and other anomalies that would be likely indicators of the DDoS attack being used to mask the real attack.

Is all of this going to address your security issues?  Sorry, not a chance.  None of the above stops all breaches, it merely minimizes the possibility that a breach goes on for months or years.  Hopefully it minimizes a breach down to weeks, days, maybe even hours in some cases but it will never totally eliminate them.  Security is not perfect.

There is a side benefit to all of this and that is it will assist you in doing RCA.  RCA is very effective in getting rid of those nagging operation issues that occur from time to time and mess up the delivery of your organization’s goods and services.  All of the information you collect for security purposes can also be used to find the needle in the haystack that is causing a database to corrupt, a network connection to drop or a server to fail because now you have information as to what was going on that led up to the problem.

The reason an organization is not secure is that there are so many areas of improvement needed that the full control triad is no longer functioning and holes exist that will allow an attacker to operate without the knowledge of the organization.  Until the controls are implemented and operating properly, it will be impossible to determine if they are secure or not.  The recommendations I have made will hopefully give you a better picture of what you face and reacting to issues that need attention before your organization is the next one to be breached.

How The PCI Standards Will Really Die

Welcome to the new year.  I hope the holidays have been treating you well and the coming year is good as well.

There have been a number of articles written about why and how the PCI compliance process will die.  It is not that I look forward to the PCI standards dying as they have brought a needed visibility to information security and privacy as well as the fact that PCI keeps me gainfully employed.  However if things stay on their current trajectory, the PCI standards will eventually die, but not for the reasons being quoted in today’s articles.  The real killers of the PCI compliance process will be the card brands and the PCI Security Standards Council.  Yes, the very folks that brought us the PCI standards will bring the ultimate demise of their precious set of standards.

The first death knell I see is that it is very easy to issue edicts from on high when you do not have to implement them.  Over the years, clarifications have been issued, quality assurance reviews performed, forensic examinations conducted and a host of other activities have resulted in “enhancements” to how the PCI standards are assessed and enforced.  Do not get me wrong, a lot of what has been done was needed and appreciated.

However, by the same token, some of what has come down has been a nightmare to implement.  Any QSAC not using some sort of automated system to conduct their PCI assessments will find it impossible to meet the current and any future documentation and tracking standards now required by the PCI SSC’s QA process.  Under the current standards, QSACs need to document who they interviewed and what the persons were interviewed about as well as tying documentation and observations to the tests performed.  Without some sort of automated process, these requirements are just too intensive to perform manually.

Documentation received and reviewed needs to have its file name, date of issue and a description of its purpose in the PCI assessment process documented.  The basic PCI DSS has a minimum of around 200 discrete documents that are required for the PCI assessment process.  The average we see for most of our engagements is over 600 documents which also include not only policies, standards and procedures, but configuration files, interview notes and observations such as screen shots, log files and file dumps.  You really have to question any QSAC that tells you they manually manage the process.  They either have an amazing and magically efficient project management process, they have very, very inexpensive staff (i.e., overseas labor) or they are short cutting the processes and producing a work product that does not comply with the PCI SSC QA program and have yet to be assessed by the PCI SSC (the most likely scenario).

Even using simple SharePoint or Lotus Notes solutions are not cheap when you consider the cost of the server(s) and the storage of all of documentation collected, which can be around 5 to 10GB per project, as well as all of the requisite system maintenance.  Servers and storage may be cheap, but it all adds up, the more clients you assess.  And speaking of the storage of documentation, the PCI SSC requires that documentation related to PCI assessments be stored for at least three years.  For those of us with electronic work paper management systems, this is not a problem.  However, given the amount of paper generated by these projects, those QSACs using the traditional paper filing methods will find a lot of shelf space taken up by their PCI engagements if they are truly following the procedures required by the PCI SSC.

All of this drives up the cost of a proper PCI assessment, more than I think the card brands and the PCI SSC are willing to admit.  It is not that I think the card brands and PCI SSC do not care about this situation, but more related to they do not have an understanding of the operational ramifications of their edicts.  The card brands and PCI SSC tread a very fine line here and to this point they have been heavy handed in the issuing of their edicts.  Going forward, the PCI SSC needs to ask the QSACs, Participating Organizations and ASVs to assess the cost and time impacts of these edicts so that they can be weighed against their benefits versus what is done now which is more of a procedural and proofing review.  If this is not done, there will soon come a point where merchants and service providers will push back hard and refuse to go through the process due to the cost and the amount of time involved to be assessed.

The next death knell is the inane process that is called the PCI Report On Compliance (ROC).  When the PCI SSC did not have access to the QSACs’ work papers, the current ROC writing process made some sense as there was no other way for the PCI SSC or the processors and acquiring banks to know if the QSACs had really done the work they were saying they had done.  However, all of that changed a number of years ago when the PCI SSC required QSACs to add a disclaimer to their contracts stating that the PCI SSC had the right to review all work products.  Yet even with this change, we continue to have to write an insanely detailed ROC, typically numbering in a minimum of 300+ pages for even the most basic of ROCs.

Unfortunately, there are QSACs out there that apparently have not been through the PCI SSC QA process and that dreaded of all states – Remediation.  As a result, they have much lower costs because they are not documenting their assessment work as completely as they need to and are not sampling, observing or interviewing like QSACs that have been through the QA process.  In addition, based on some work products we have seen, they also do not care about the quality of the resulting ROC as it looks entirely like a ‘find and replace’ of a template and makes no sense when you read it.  In talking to other large QSACs that have been through the QA process multiple times, the PCI SSC has indicated that they are monitoring the large QSACs more than the little QSACs because there is more risk with the large QSACs.  While true to an extent, we have encountered a number of smaller QSACs that perform assessments for large clients due to their much lower cost structure and their willingness to ‘overlook’ compliance issues.  If the PCI SSC does not go after these QSACs soon, there will likely be a number of breaches that occur due to the QSACs’ lack of diligence in performing their assessments.

I know of a number of QSACs that would like to see Bob Russo and the representatives of the various card brands to actually work as staff on a few PCI assessment engagements so that they can better appreciate the inordinate amount of work involved in generating a ROC.  I think they would be shocked at the amount of work effort they have driven into a process that is already too complicated and prone for error.

As it stands today, the ROC writing, review and proofing process is probably 50% to 60% of a good QSAC’s project costs.  To address this, the PCI SSC QA group tells QSACs to develop one or more templates for writing the ROC which, from what we have seen from some other QSACs, means a lot of mass ‘find and replace’ to speed the ROC writing process.  For the last few years, a number of QSACs have brought the ROC writing process up at the Community Meetings.  However the card brands continue to shoot down any sort of changes to the process.  As a result, the cost of producing a ROC is driven by the size and complexity of the merchants’ or service providers’ cardholder data environment (CDE).  These costs will only continue to rise as long as the PCI SSC does not allow QSACs to mark items as ‘In Place’ with only a check box and rely on the QSAC’s work papers versus the verbosity required now.  If this sort of process can work for financial auditors, it can work here as well.

A third death knell is the PCI SSC and card brands continuing to quote that the majority of breaches are the result of organizations not complying with the PCI DSS.  In discussions with a number of the PCI forensic examination companies, I am hearing that the card brands cannot believe the fact that more and more organizations were PCI compliant at the time of their breach.  The PCI SSC and card brands have apparently convinced themselves that the PCI standards are “perfect” and they cannot imagine that an organization could be breached unless that organization was not complying with the PCI standards.  There is no security standard that I am aware that totally prevent breaches.  So while the PCI standards are good baseline security standards, the card brands and PCI SSC seem to have forgotten that security is not perfect and that any security standard only minimizes the damage done when a breach occurs if the standard is truly followed.

And as organizations have gotten the PCI “religion,” the effort required to compromise them from the outside via traditional attacks has increased significantly.  As a result, successful attackers have changed strategy and work on social engineering their way past the bulk of an organization’s security measures.  The PCI DSS only has a little bit on social engineering in requirement 12.6 regarding security awareness training.  And even those organizations with the most robust of security awareness programs will tell you that, even after extensive security awareness training, human beings are still fallible and that some people still do very questionable things that continue to put organizations at risk, sometimes significant risk.  Even when you have the most diligent of employees, they still make mistakes in judgment from time to time.

Until the human element can be totally removed, there will always be a certain amount of risk that will never go away.  Again, the PCI SSC and card brands seem to not want to acknowledge the failings of the human element and appear to believe that technology is the savior based on the focus of the PCI standards.  However time and again, every security professional has seen very sophisticated security technologies circumvented by human error or just plain apathy towards security (i.e., “it always happens to someone else, not my organization” or “we’re too small to be a target”).

Until the PCI SSC and the card brands drop the “holier than thou” attitude toward the PCI standards and stop the public pillory of organizations that have been breached, there will continue to be editorial commentary regarding the pointlessness of the standards and ever more serious push back to complying with the standards.

These are the reasons why the PCI SSC and the card brands will be the ones that will kill the PCI standards.  At the moment, they are so far removed from the process; they do not understand how complicated and expensive the process has become which is why merchants and service providers are complaining about the ever increasing costs and effort related to the PCI assessment process.

The PCI SSC and card brands also seem to have forgotten that QSACs have to make money doing these assessments and, when you pile on clarifications and edicts that do nothing to streamline and simplify the process; you are only driving the costs of the process higher.  And higher costs only make merchants and service providers, who are on thin margins to being with, even more incentivized to use the much lower cost QSACs, driving the diligent QSACs out of the market, thus increasing the likelihood of breaches.

Again, it is not that I want the PCI standards to go away as I think they have brought a real benefit.  However, if these issues are not addressed, the PCI standards will end up going away.  I fear that, with them gone, there will be no carrot to ensure the security of cardholder information and we will end up back where we were before the PCI standards existed.

What To Focus On In 2013

It is the end of the year and, like all other pundits, here is another idea on what 2013 will bring in the way of security issues.  After reading a lot of the other predictions out there, I tend to agree with those from Verizon Business Services’ Data Breach Investigation Report researchers.  While everyone else is predicting cyber-Armageddon as the biggest threat, the researchers at Verizon Business Services see a lot more of the same for 2013.

The biggest threat Verizon identifies is more attacks on authentication systems.  This is most likely because your vendor or your developers talked you into storing authentication information in your database that is Internet facing.  We see this all of the time with eCommerce and Internet banking solutions.  The external user credentials end up being stored in the database along with order entry, inventory and pricing data.  This is typically done because using a directory system for such purposes is difficult and, at times, not as functional as when authentication data is stored in and used from a database.  Given the prevalence of SQL attacks, all of that information results in being available for the taking through a SQL injection attack.  As a result, the attackers compromise the authentication system, gain access to everyone’s credentials, including administrators, and it is likely ‘game over’ regarding the rest of your security measures.

I want to touch next on social engineering because it is typically directly related to compromising authentication systems even though the second place attack method most concerning Verizon researchers is application attacks.  Social engineering is all about tricking your end users into giving up key information so that an attacker can compromise your environment.  The most common piece of information an attacker tries to obtain is an end user’s credentials for logging onto the network.  Hence the reason why I wanted to discuss this after the authentication system attacks.  Social engineering is the most insidious of attack methods because it does not involve any of an organization’s security technology.  And worst of all, if social engineering is successful, all or most of your organization’s security technology is effectively neutralized as a result.  That is because most organizations have little or no security once someone is on the inside.

Now let us look at the second most concerning attack to Verizon which is application attacks.  Verizon is saying that application attacks are more of a threat to governments and large applications.  Regardless of the target, any organization with an application presence on the Internet is a potential target such as with eCommerce or Internet banking.  A lot of these applications are based on on-line frameworks such as IBM’s Websphere or Oracle’s Application Framework.  It is not that these frameworks are insecure, it is that they still require development effort and it is those custom development efforts that do not guarantee a secure application.  The problem comes from the fact that a lot of people believe that using a framework means that little to no security testing even though the amount of custom development done in these frameworks can be more extensive than starting from scratch.  As a result, we see a lot of organizations tossing Internet applications into production with little or no security testing and then ending up with breaches as a result.

In addition, there are third party applications served up by application service providers (ASP).  A lot of small and mid-sized businesses (SMB) use these sorts of applications to have an online presence.  As a result, a lot of SMBs believe that these solutions do not require any security testing because the vendor and the ASP do that for them.  However, we are encountering more and more attacks on SMBs, particularly those that have wealthy clientele such as country clubs and exclusive financial institutions because their applications are notorious for not being secure.  SMBs are constantly amazed that; (1) they were targeted and, (2) the application was not bettered secured.  Yet attackers know that while the take out of SMBs could be significantly less than a large organization, an SMB is usually easier to compromise because they do not have the security and monitoring resources of a large organization.  As a result, SMBs are becoming a larger and larger target for attackers.

Finally, something that concerns me as the previously discussed threats is mobile devices and devices not under the organization’s control, also known as ‘bring your own device’ or BYOD.  I think these devices will surpass the other three threats over the next few years because most organizations have difficulty maintaining security on their servers, desktops and notebooks, let alone something like an iPhone or an Android tablet.  The worst thing about mobile devices is that they are so easily lost and it fascinates me how many people lose their mobile devices.  The bottom line about mobile devices and BYOD is that you must be very, very careful as to what you allow these devices to access and how you grant that access.  And you must make sure that these devices are not allowed to download information, even if encrypted, as that information is highly likely to be lost.

So what should you be doing regarding these threats?  Here are my top things organizations should be doing to minimize the risks presented by these threats.

• Trust no one.  This is particularly true of mobile devices or BYODs, but it also applies to your own internal systems.  Forrester has promoted this in their ‘Zero Trust Model’ as well as me in the ‘Fort Knox Approach’.  This is not as easy as one might think, but the approach makes sense in these days of attacks on authentication systems and social engineering approaches.
• Classify your data.  This is usually a difficult project, but they pay dividends at the end because everyone understands why certain information cannot be allowed out of the control of the organization.  It also allows people to justify to others why data cannot be allowed to be accessed by people that do not need access as well as via mobile or BYOD.
• Require encryption on mobile devices and BYOD.  Even if you do not allow data to end up on these devices, you do not even want the memory or other information that might be inadvertently stored on these devices out of your control.  As a result, if you encrypt these devices, there is a high likelihood that, if they are lost, the person finding them will just wipe them and start over.
• When possible, use a directory system for authentication.  This is always painful for systems that operate outside of the traditional control environment of internal users.  Directory systems are usually designed to be more secure than any sort of database authentication system because they are assumed to be at risk from the start.  However, just because they are designed to be secure does not mean they cannot be implemented in an insecure manner.  Windows Active Directory takes a lot of heat for being insecure; however a lot of that heat is due to silly implementations to support insecure authentication methods for compatibility.
• Conduct security awareness training.  The only thing that minimizes social engineering is consistent and regular security awareness training.  However, do not kid yourself or management.  Everyone has their ‘moments’ and does something they should not.  That said there are always those that just never seem to get it which is why you need other controls and monitoring to ensure you maintain your security.  However, to just throw up your hands and say it is pointless is also not a position to take.
• Secure your applications.  This means conducting application code reviews and testing applications before they are put into production not after the fact.  Unlike networks where you need to put them into production before testing them, applications can be tested before going into production.  It amazes me how many organizations put their applications into production and, by the time they finally get around to testing them, they have already been compromised.  And while automated application code testing solutions are all of the rage, we still find that the best results come from the more traditional human code review not automated tools.
• Monitor your network and applications.  This is a double edge sword.  You know what to look for, however, you have so many ports open that it is near to impossible to recognize bad traffic from good traffic.  And it is not necessarily the fault of your IT department as most packaged applications require an inordinate amount of ports open to function properly.  However, the key thing to monitor, more than anything, is any traffic going outside of your network to an unknown location.  When you see traffic going to Eastern Europe, China or any unexpected IP address, your monitoring system should generate an alert as that is typically a key indicator that you have been compromised.

Have a happy holiday season and I will “see” you all next year.

Pre-Authorization Data – The Card Brands Weigh In

I just wanted to pass along what I have heard thus far on this topic from the various card brands.  I’ll keep updating this post as I get more responses.

American Express (aka Trustwave) came back with the following.

“Per your inquiry regarding the storage of pre-authorized data, from what I can tell the American Express DSOP program currently does not address this topic. However, both the PCI SSC and the card brands (including American Express) have made it abundantly clear that pre-authorization data is to be protected with the same zeal as post-authorization data. That means encrypting it and restricting access to it. The reason the PCI SSC has not issued any directives regarding pre-authorization data yet is that it is a complicated environment and cannot be dealt with in a simple manner with the same approach working for all occurrences.

So, while pre-authorization data seemingly is not covered by the PCI DSS/DSOP programs at this time, you must do everything you can to protect it.

I hope this somewhat answers your question. I plan on doing more research to find any official documentation regarding the subject.”

MasterCard International followed up with their response.

“Merchants should talk with their acquirers to determine the pre-authorization rules particular to their vertical market and region.”

Visa quoted their Visa International Operating Regulations manual; section Account and Transaction Information Security Requirements VIOR 2.1.E (Updated):

“Ensure that all agents and Merchants do not store any of the following, subsequent to Authorization:
–    Full contents of any data taken from the Magnetic Stripe (on a Card, in a Chip, or elsewhere)
–    Card Verification Value 2 used to verify Card-Absent Transactions
–    PIN or the encrypted PIN block”

Discover has come back with the following.

“Currently the following restrictions apply:

(a) Sensitive Authentication Data obtained in connection with a Card Transaction must not be retained after receipt of an Authorization Response.
(b) Sensitive Authentication Data must not be recorded on Transaction Documentation or any other records or evidence of Card Transaction.
(c) Any processing, transmission, or storage of Cardholder Data must be conducted in accordance with PCI DSS requirements.”

JCB is still to be heard from.

I was deeply disappointed with MasterCard’s response.  Acquiring banks, for the most part, cannot answer basic questions about the PCI DSS, so we are supposed to believe that they are experts on retention of pre-authorization data based on a company’s vertical market and region?  Talk about passing the buck.

I am also surprised that I have yet to see any of the supposedly “very specific rules that prohibit any storage of sensitive authentication data (SAD) and do not make any exceptions.”  Not that I was expecting any as the Council has a tendency of making a mountain out of a molehill at times.

Based on the responses, I would highly recommend that merchants take American Express’ advice and protect pre-authorization data with the same voracity as they do with post-authorization data.

PA-DSS Validation Clarification

On July 23, 2012 we received the following communication from James Barrow, Director of AQM Programs, with the PCI Security Standards Council.  I found it worthy of posting so that everyone understands the procedures their QSA needs to follow regarding applications that are supposedly PA-DSS validated.

The Council has recently received inquiries related to the Validation of Payment Applications process and there seems to be some confusion related to the PCI SSC listing of validated applications.  The Council’s website is the authoritative listing of applications that have been accepted by the Council.  This is the listing that should be checked by the assessors during each engagement with a merchant.  If the merchant’s application (both name and version number) are not on this list, it cannot be considered validated.

There are some instances where a merchant might provide you with a document (not issued by the Council) stating that the application has undergone some type of review and has been deemed compliant.  However, if the payment application is not listed on the PCI SSC website, it cannot be considered validated.  If such an instance of this arises during one of your engagements, you as the assessor must perform your due diligence in determining if the application is capable of meeting all of the DSS requirements.

For the PA-DSS community we realize that some applications are not applicable to the PA-DSS program.  The eligibility for the program is contained in the document entitled “Which Applications are Eligible for PA-DSS Validation?  A Guiding Checklist” available at the Council’s document library.  If an application is not eligible for the program, it does not preclude you from performing an assessment.  However, at the end of the assessment you cannot communicate to your client that per the assessment the application has been “validated”, nor can the client (vendor) expect the assessment to have any bearing on a merchant’s ability to achieve DSS compliance.

Following the above guidance should help to remove any miscommunication or misunderstanding in the payment ecosystem as to what applications are considered validated, and the steps that need to be taken should a non-validated application be identified in the field.

The key here is that if the version of your payment application is not on the PCI SSC’s PA-DSS list, it is not considered PA-DSS validated and your QSA must assess it accordingly.  I cannot tell you how many merchants we encounter where they have a different version of the application, yet the merchant insists that we treat it the same as the version that is PA-DSS validated.

We also run into software vendors that insist that the version the merchant is running is not significantly different from the version that is PA-DSS validated.  While this could be an accurate statement, the vendor needs to have submitted the version to a PA-QSA for validation of that fact.  The PA-DSS has a procedure that the PA-QSA can follow to determine that version changes have not affected cardholder data processing and the application’s PA-DSS validation.  Without that validation, as a QSA, our hands are tied and we must conduct a full assessment of the application under the PCI DSS.

Much to the chagrin of a lot of merchants, a PA-DSS validation does not imply that they are PCI DSS compliant.  There is also this mistaken belief by merchants that a PA-DSS validation implies that the QSA does not have to assess the application.  Under the PCI DSS, a QSA still must assess the application’s implementation and ensure that it was implemented per the vendor’s instructions to maintain its PA-DSS validation.  The trouble is that this implementation assessment may not save much, if any, time for the QSA.

A Reason Why The PCI Standards Get No Respect

Call it the “Rodney Dangerfield Effect.”  Conflicts of interest seem to pervade the PCI compliance process and it is something the PCI SSC and the card brands need to clear up before their precious standards get even more bloodied in the media.

I have run across another processor that dictates the use of a particular QSAC.  Now do not get me wrong, I am a capitalist and interested in making money just like the other guy.  But I have to say that I am not a shark like some of my competitors.  I know this post will sound like someone bemoaning sour grapes but, in my opinion, this situation just makes the whole PCI compliance process look like a worthless sham.

What prompts this post is a call with one of our clients that we have performed PCI assessments for years, even before the PCI SSC existed.  They are implementing a new point of sale (POS) terminal that requires them to use a different credit card transaction processor because their existing processor is not yet certified to process transactions from this new terminal.  Fair enough.

The new terminal is a test installation to see if the service should be expanded to all of our client’s locations.  Since the terminal will only generate a couple of thousand transactions in the coming year, the new processor has identified our client as a Level 4 merchant and is treating them accordingly.  In reviewing the processor’s contract, our client found that the contract dictates that they use a specific QSAC to “assist” them in filling out their PCI Self-Assessment Questionnaire (SAQ) A.  Knowing the SAQ A, our client cannot figure out what a QSAC would do to assist, but it is in the processor’s contract.

Our client’s first question was, “Since when does a processor have the right to force us to use a particular QSA?”  We explained that we have been told that while the PCI SSC and the card brands allow processors to have rules that go above and beyond the PCI SSC’s and card brand’s requirements.  While I understand that the processor is likely trying to ensure that their Level 4 merchants are not just checking the ‘Yes’ box on their SAQs, forcing the use of a particular QSAC seems a bit questionable.  Particularly when we have been told that some QSACs are giving processors payments for all of the customers they refer.

I have written about this issue before with processors charging fees to merchants for the filing of their SAQs.  There is also the scam of forcing merchants to use a specific PCI Approved Scanning Vendor (ASV) to scan the merchant’s networks even when the merchant does not have an ecommerce presence or outsources their ecommerce to a third party that already provides their ecommerce customers ASV reports.  This is just one more questionable requirement that processors demand that makes merchants and the media think PCI is a scam.

Their next question was, “Since you already do our ROC, can’t we just submit that to our new processor?”  You can do that, but you need the new processor’s approval as they do not have to accept our work.  What is the likelihood that the new processor will accept our client’s ROC?  No idea and I am anxious to hear what our client tells us in that regard.

The problem here is that the processor in question and the QSAC have numerous connections that give a distinct impression of conflicts of interest.  First, the QSAC in question runs the processor’s Level 4 merchant compliance program.  That program dictates that the QSAC perform some sort of assessment process in order for any of the processor’s Level 4 merchants to create and submit their SAQ.

The justification the processor gave our client was that the PCI SSC requires this action.  Last I checked, the PCI SSC and the card brands did not require a QSA to fill out an SAQ.  MasterCard has a deadline of June 30, 2012 for Level 2 merchants to have either an ISA fill out their SAQ or have a QSA conduct a PCI ROC.  Until October 2010, Visa Canada required that a QSA sign off on all SAQs.  But those are the only SAQ rules involving QSAs that I am aware.

Next, the QSAC and the processor have swapped various personnel over the years.  As a result, there is an appearance that the two are essentially one in the same given that the QSAC runs the processor’s compliance program and the processor dictates that their merchants use the QSAC for PCI assessments.  I know that people move between organizations in the same industry all the time, but the number of people that have gone between these two would seem to be higher than expected.

I guess since I am an employee of a public accounting firm in the United States, I have greater sensitivity to conflicts of interest than most.  The American Institute of Certified Public Accountants (AICPA) has very specific rules in regards to conflicts of interest.  We have an entire department dedicated to ensuring that we avoid conflicts of interest.  As a result, we regularly look at the services provided to our clients and ensure that we are not in conflict or even give an appearance of a conflict of interest.

Now I am not suggesting that the PCI SSC and card brands go to the levels that the AICPA requires.  But let us face it, it is the Wild West out there and some of the QSACs do not care what conflicts they may have and how it might hurt the PCI compliance processes.  The PCI SSC only requires its assessors document the services they provide to the organizations they assess in their assessment reports.  While that offers a certain amount of transparency, when you read some of these ROCs, it becomes painfully obvious that some QSACs are assessing their own security services.  In some cases, the organization being assessed has outsourced almost everything related to their PCI compliance to the QSAC doing their assessment.  What do you think the likelihood is that those services will be assessed as not compliant if there are compliance issues?  One would assume it will be very unlikely.

But it can get even worse.  A certain QSAC operates one of the card brand’s merchant PCI compliance program.  Merchants submit their Attestations of Compliance (AOC) and Reports On Compliance (ROC) to this card brand through the QSAC which manages the process.  Does this QSAC inform their clients that accept this card brand’s cards of this fact?  Not that I have ever been told by any prospects.  Does the QSAC list the management of the card brand’s merchant program on their ROCs?  Not that I have seen and I have seen a number of their ROCs for merchants that accept the card brand’s cards and I have never seen the program listed.  Does the QSAC submit their ROCs to the program that they manage?  They must as the ROCs I have seen are from merchants that accept the card brand’s cards.  Is this a conflict of interest?  One would think, but this is how things operate today.

The bottom line is that in this age of openness and transparency, it is these sorts of relationships and actions that give a very bad impression to the outside world.  The PCI SSC and the card brands need to enhance their rules for QSACs that define conflicts of interest.  Until this is done, the PCI standards will continue to be ridiculed and viewed as pointless.

Is It ‘WHO’ Or ‘WHAT’ That Is Important?

There is a very active discussion going on in security circles about understanding adversaries and how that impacts security strategy.  I have taken a contrarian position in this argument and have stated that, in the scheme of things, I do not believe that you need to waste time understanding your enemy.  What I think matters most is what needs to be secured and how it needs to be secured.  This post is to discuss my rationale for this approach and relies on my prior post regarding the Fort Knox approach to security.

Sun Tzu famously said it was important to, “Keep your friends close and your enemies closer.”  The biggest difference with cyber-attacks is that the enemy are true mercenaries in that they come together because of an interest in a target, an interest in achieving their own particular goal, such as proving they are the best hacker or social engineer, or just because.  As a result, when your enemies can number in the hundreds or even thousands and have their own potentially unique motives for why they are attacking, it is near to impossible to do an analysis of the enemy, such as Sun Tzu suggests, that provides you with any sort of significant defensive advantage.

But what about advanced persistent threat (APT) attacks?  There is usually a common actor in APT, either a competitor, organized crime or a government.  However these sponsors usually hire the technical “muscle” for the actual attack.  The backer of the APT attack provides these mercenaries with a list of information they wish to be retrieved from the target organization(s).  So while APT can provide you with a traditional enemy, that enemy is obscured by the mercenaries actually conducting the attack.  Again, an analysis of the enemy provides limited to no advantage in your defense because you only see the mercenaries, not the sponsor.

But I think the biggest nail in the coffin for enemy analysis is related to attack strategies.  When reports from Verizon, Trustwave and other forensic examination firms consistently report that the same basic attack strategies are successful, it does not matter who the enemy is and why they are attacking when anyone from a neophyte to expert can break into your systems because of the same stupid mistakes or human errors.  By the time you have the enemy analysis done, your organization’s information is long gone.

In my opinion, ‘WHAT’ is more important in that organizations understand ‘WHAT’ information they need to protect and then go about appropriately protecting it.  If that sounds familiar, it should because that was the basis of my Fort Knox post.  If you think about it, a Fort Knox strategy does not worry about ‘WHO’ is trying to get the gold, it is all about protecting the gold regardless of ‘WHO’.

The bottom line is that in a cyber-attack, ‘WHO’ is attacking you is irrelevant.  You do not need to waste your time figuring out ‘WHO’ the attacker is and what are their motives.  It is all about your information that they wish to obtain.  So stop wasting time on enemy analysis and start properly protecting your organization’s critical, sensitive information.  I think you will find that the Fort Knox strategy will make your security efforts much more easy to implement and maintain.

UPDATE: In a brief moment of clarity on my part, I realized after making this post that the Fort Knox security approach is just another way of looking at the ‘Zero Trust’ security model that was proposed by John Kindervag of Forrester a while back.  See my earlier posts on the Zero Trust security approach for more information.

Zero Trust Security – The Cultural Discussion

Zero Trust Security – The Technical Discussion

Another Year, Another QSA Re-Certification

It is that time of the year when I have to go through the PCI SSC’s Qualified Security Assessor (QSA) re-certification process.

To add to the re-certification process this year, I have been sick for the last two months with a cold that turned into a nasty case of bronchitis along with laryngitis that then caused a severe case of sinusitis.  I just could not catch a break this Spring.  The good news is that I am finally on the mend and should be back to normal in another couple of weeks.

However, even illness does not get you out of the QSA re-certification process.  So, I put it off as long as I could and took the examination this morning.

As I expected, there was not a lot of new material in this year’s QSA update.  The biggest focus of this year’s training seemed to be:

• The interrelationship of the various PCI standards;
• Roles and responsibilities of QSAs, ASVs, merchants, service providers, acquirers, PCI SSC and the card brands;
• Scoping of the cardholder data environment and cardholder data discovery; and
• The integration of the PA-DSS with the PCI DSS.

Other than that, it was for the most part a reinforcement of the changes in the PCI DSS v2.0 to make sure that QSAs really understand the standard.

There is an interesting section on what not to write in the In Place column.  The unfortunate aspect about this section of the training was that the examples that were presented were straight out of ROCs that the PCI SSC QA program had reviewed.  Some of those responses were very difficult to read they were so bad.

There is also a discussion on network segmentation.  Unfortunately, the examples were very simple.  I wish our clients had such simplified networks.  However, because this discussion is in this year’s presentation materials indicates there are apparently still a lot of QSAs that do not understand the concept of network segmentation and what constitutes good segmentation from poor segmentation.

As I am finishing this post, I have been told I passed the QSA re-certification examination.  So I am a QSA for another year.