Vulnerability Management

On July 1, 2012, requirement 6.2.a went from a “best practice” to an official requirement.  Since v2.0 of the PCI DSS was issued, there has been a very active discussion regarding what the PCI SSC was trying to get at with this revision.  But it is not just requirement 6.2 that is involved in this process, requirements 2.2, 6.5.6, 10.4, 11.2.1 and 11.2.3 also include references to 6.2 and, with requirement 6.1, comprise the PCI DSS vulnerability management program.

To refresh people, requirement 6.2 states:

“Establish a process to identify and assign a risk ranking to newly discovered security vulnerabilities.”

There is a note that is included with requirement 6.2.  That note is a clarification regarding how risk rankings should be set.

“Risk rankings should be based on industry best practices. For example, criteria for ranking “High” risk vulnerabilities may include a CVSS base score of 4.0 or above, and/or a vendor-supplied patch classified by the vendor as “critical,” and/or a vulnerability affecting a critical system component.”

As simple as this requirement is, it is amazing how complicated organizations attempt to make it.  All the PCI DSS desires from this requirement is that there is a documented process that identifies vulnerabilities and assigns a risk ranking to them, if the organization desires to change that ranking.

The purpose of the revision to requirement 6.2 is to provide organizations with the ability of determine what they patch, when they patch, based on the risk presented by the vulnerability in the organization’s environment.  The Participating Organizations (PO) had pushed for this change in the PCI DSS v2.0 because, according to the POs, QSAs were demanding patching of software that either did not exist in the environment or existed on systems out of scope.

I can appreciate the POs situation as I had encountered numerous horror stories regarding patching over the years.  In a number of cases, POs that were running Apache for Windows were being told to patch Microsoft Internet Information Server (IIS) even though it was not installed because versions of vulnerability scanners and patching programs incorrectly reported that IIS was not patched.  In another situation, a QSA was requiring IBM iSeries systems patched for Web server patches that were unrelated to IBM Websphere.

The key clarification is that the PCI SSC does not require an organization to reinvent the wheel and create a unique ranking system.  If an organization wants to use the CVSS ranking approach, that is perfectly fine.  What the revision to requirement 6.2 allows is that if the CVSS for a vulnerability that has a CVSS of 4.2 for example, the organization can revise it to a value below 4.0 as long as they document the process for that downgrading of the CVSS.

To perform this calculation, I recommend using the National Institute of Standards and Technology’s (NIST) Common Vulnerability Scoring System (CVSS) calculator.  The key metric to adjust to your environment when you do not have systems that run software with the vulnerability is under Environmental Score Metrics, the ‘Percentage of vulnerable systems (TargetDistribution)’ value.  By adjusting this value to ‘None’ or ‘Low’, the overall CVSS score will drop well below 4.0.

The next piece of formal documentation you need to have is the explanation for why you changed the CVSS score.  If you do not have this documentation, then you are not allowed to change the CVSS value.  This documentation does not need to be extensive, just the explanation that justifies the changes you made to the variables used to compute the CVSS.

The other stumbling block for organizations is requirement 6.1 which states:

“Ensure that all system components and software are protected from known vulnerabilities by having the latest vendor-supplied security patches installed. Install critical security patches within one month of release. “

It is the final sentence in requirement 6.1 that creates the most consternation, “Install critical security patches within one month of release.”  But that is not the only deadline.  The note for 6.1 also has a deadline.

“An organization may consider applying a risk-based approach to prioritize their patch installations. For example, by prioritizing critical infrastructure (for example, public-facing devices and systems, databases) higher than less-critical internal devices, to ensure high-priority systems and devices are addressed within one month, and addressing less critical devices and systems within three months.”

If a risk-based approach is followed and systems are prioritized, then critical infrastructure needs to be patched within one month and the remaining systems patched within three months.  While this gives a bit more leeway, organizations still can have issues getting patches implemented within a month and even within three months.  For merchants with simple environments and low device counts, complying with requirement 6.1 is annoying but can be accomplished.

But for organizations with large device counts and documented change control processes, getting patches done in a 30 day cycle is typically not possible.  Why you might ask?  Because by the time the patch is released by the vendor, the organization obtains the patch, tests the patch in their various applications’ environments, puts the patch through their quality assurance and regression testing processes for those environments, and then implements the patch in production.  The quickest some of these organizations can get a patch from release to production is 45 days, but more likely 60 days.  Because of this long patch cycle, these organizations scan more often, monitor in real-time and implement other mitigating controls to manage the risks related to the vulnerabilities they cannot patch in the 30 day window.

Another problem outside of organizations’ control is application vendors.  A lot of point-of-sale (POS) and e-Commerce software vendors issue updates on quarter, semi-annual or even annual bases.  These vendors explicitly document in their contracts that organizations are not allowed to independently patch their systems as that will void the vendor’s support agreement.  As a result, an organization can be technically out of compliance with requirement 6.1 for months and mitigations can only do so much.

This has been discussed at length during QSA and open sessions at the PCI Community Meetings.  POs and QSAs argued that the 30 day deadline was not realistic and explained why.  Finally, QSAs were told to use their judgment and evaluate the organization’s vulnerability management process and determine if vulnerabilities could “fall through the cracks.”  If the vulnerability process is considered rigorous and vulnerabilities were believed to be processed without being lost, then the vulnerability management process could be allowed to meet the requirements of 6.1 regardless of the timeline in which vulnerabilities were actually patched.

So what are the lessons to be learned?

• The 30 and 90 day patch timeframes are goals to shoot for, and you should always try to meet those deadlines.  But as long as your organization can prove it has a rigorous vulnerability management program and have documentation that it works and works reliably, it is in compliance with requirement 6.1.
• You do not need to reinvent the wheel and come up with a new vulnerability ranking system.  Use the CVSS and modify the inputs as necessary to reflect your organization’s particular environment.
• Any changes you make to a CVSS for a vulnerability need to be justified and documented.
• Document your vulnerability management policies, standards and procedures and live by those documents.  If you cannot prove your process works, then you are not in compliance with the PCI DSS.

Service Provider PCI Compliance Process

As a recent question pointed out to me, while service providers now seem to understand they need to be PCI compliant, they do not seem to understand the process under which they assess their PCI compliance.  As a result, I thought I would use this space to clarify this process.

When Do I Have To Be PCI Compliant?

The PCI SSC defines a Service Provider in the PCI DSS Glossary as:

“Business entity that is not a payment brand, directly involved in the processing, storage, or transmission of cardholder data.  This also includes companies that provide services that control or could impact the security of cardholder data.  Examples include managed service providers that provide managed firewalls, IDS and other services as well as hosting providers and other entities.  Entities such as telecommunications companies that only provide communication links without access to the application layer of the communication link are excluded.”

On the straight forward side of this definition, if your organization processes, stores or transmits cardholder data on behalf of one or more other organizations, your organization is considered a service provider and needs to be PCI compliant.  That point seems to be clear and most organizations that are processing cardholder data for others understand this fact.

It is organization that do not directly process, store or transmit cardholder data but could impact the security of cardholder data that get tripped up.  The bottom line is, any third party that has access to an organization’s cardholder data environment (CDE) needs to be PCI compliant for those services offered that come into contact with the CDE.

Based on this definition, the following are examples of services would need to be PCI compliant.  This is not a complete list but something to assist you in understanding why your organization might need to be PCI compliant.

• Management of firewalls that are considered part of an organization’s CDE.
• Management of networks that carry cardholder data that is not encrypted.
• Management of encrypted networks that carry cardholder data and the MSP manages the encryption keys for encrypting the network.
• Management of servers that process, store or transmit cardholder data.
• Management of PCs that have access to cardholder data.
• Management of applications that process, store or transmit cardholder data.

The first question service providers typically ask is, “How am I supposed to know if my customer needs our services to be PCI compliant?”  My rather indignant response is, “What, you cannot ask them?”

If you are a service provider and you are providing any value added services, you need to be asking every customer about their regulatory or legal requirements when going through the sales and contract processes.  It is not just PCI these days, but in the US there is HIPAA, GLBA, SOX and there all sorts of similar requirements in various parts of the world that you should be finding out whether your prospects need to comply.  If you are not, then you are sitting on a potential legal time bomb should something happen and your value add services are deemed the culprit.

SAQ D or ROC?

Service providers are broken into two levels, Level 1 service providers and Level 2 service providers.  Level 1 service providers are those that process more than 300,000 Visa, MasterCard or Discover transactions annually and are required to perform a full PCI assessment using a qualified security assessor (QSA) that results in a Report On Compliance (ROC).  Level 2 service providers are those that process 300,000 annual transactions or less and they can conduct their own assessment using the SAQ D.  SAQ D is the only SAQ that service providers are allowed to use.

The first issue most MSPs face is determining what parts of the ROC or SAQ are relevant.  There are no easy answers in this area as it all comes down to those value added services you are providing and how you provide those services.  However, before you can determine what parts of the ROC or SAQ are relevant, you need to determine the services that are to be assessed.

Where MSPs can miss the boat is taking the lowest common denominator approach and only assessing those value added services that all customers need to be PCI compliant.  Typically this results in an AOC that discusses physical security (requirement 9) and security policies (requirement 12).  If you take the lowest common denominator approach, then do not complain when each of your customers’ QSA have to traipse through your facilities and disrupt operations over the value added services you did not cover in your AOC.

The best approach is to assess all value added services you provide so that you do not have to worry about whether or not a given service is PCI compliant.  The reason this is important is that customers are required to make sure that all service providers that need to be PCI compliant are PCI compliant.  It is not that customers cannot work with non-PCI compliant service providers; it is just a lot easier to work with PCI compliant service providers.  The reason is that non-PCI compliant service providers have to be annually assessed as part of the customer’s PCI assessment as well as the customer is supposed to be monitoring the service provider’s PCI compliant efforts.  As a result, you will have an easier time and be easier to deal with if all of your value added services are PCI compliant.

Now back to what parts of the PCI DSS are relevant.  There are too many permutations of services and PCI requirements to go into this for every possible situation, so I am just going to provide some examples.

If you are providing firewall management services, then you are going to have to comply with requirements 1, 2, 4 (if your organization is managing any transport encryption), 5 (the PCs/servers used to manage the firewalls), 6 (change control for the PCs, servers and firewalls), 7, 8, 9, 10 (log management and analysis for the PCs, servers and firewalls), 11 (PCs, servers and firewalls) and 12.  Some tests in these sections may be Not Applicable for your organization, but you will need to go through all of the tests in the sections.

If you are managing applications such as in a software as a service (SaaS) environment, you are going to have to comply with requirements 1 (if you manage any firewalls or routers that could be in-scope), 2 (servers, firewalls, routers, switches), 3 (if your application encrypts the cardholder data), 4 (if your network encrypts cardholder data), 5 (servers running the application as well as any support PCs and servers), 6 (change management as well as any software development), 7, 8, 9, 10, 11 and 12.  Some tests in these sections may be Not Applicable for your organization, but you will need to go through all of the tests in the sections.

As you can see, even a “minor” service such as managing firewalls can require a significant amount of effort to be PCI compliant.

Do I Need A QSA?

A Qualified Security Assessor (QSA) is only required if: (1) your organization is considered a Level 1 service provider, or (2) you wish to have your organization listed on Visa’s Global Registry of Service Providers or with the MasterCard Registration Program (MRP).  In both of these cases, you will need to hire a QSA and produce a Report On Compliance (ROC) and Attestation Of Compliance (AOC).  If you can answer ‘No’ to both of these, then you can do your own assessment using the SAQ D.

Scope Of Assessment

At the 2012 PCI Community Meetings, the PCI SSC clarified scoping for PCI assessments.  The PCI SSC stated that:

“To be considered entirely out of scope, a network or system must be isolated from, with no connectivity into, the CDE (such that even if the system is “owned”, it cannot impact the security of the CDE).
If connections are limited to specific ports or services, those systems are included in PCI DSS scope to verify applicable controls are in place.
Applicable PCI DSS controls may vary for in-scope systems depending on the function of the system and presence of other controls.”

As a result, any of your systems that have access into a customer’s CDE are in-scope for compliance as they could be used to access the CDE.  This does not mean that these systems need to meet all of the requirements of the PCI DSS.  But it does mean that they need to be evaluated to determine those portions of the PCI DSS that do apply.  All of this is dependent on the potential to access cardholder data inside the CDE.

To minimize the impact of this clarification, we are seeing a lot of MSPs using “jump boxes,” PCs that do nothing but provide access into the systems to be monitored or managed.  This provides an air gap between the MSPs internal systems and the customer’s environment, including the CDE.  As a result, the jump boxes need to be PCI compliant and the internal systems at the MSP just need to have current anti-virus and anti-malware and be properly authenticated.

Hopefully this post helps all of you managed service providers that are facing PCI compliance.

This post was revised after a review by Walt Conway at 403Labs pointed out some inaccuracies.  A big thank you to Walt for his diligent eye.

What To Focus On In 2013

It is the end of the year and, like all other pundits, here is another idea on what 2013 will bring in the way of security issues.  After reading a lot of the other predictions out there, I tend to agree with those from Verizon Business Services’ Data Breach Investigation Report researchers.  While everyone else is predicting cyber-Armageddon as the biggest threat, the researchers at Verizon Business Services see a lot more of the same for 2013.

The biggest threat Verizon identifies is more attacks on authentication systems.  This is most likely because your vendor or your developers talked you into storing authentication information in your database that is Internet facing.  We see this all of the time with eCommerce and Internet banking solutions.  The external user credentials end up being stored in the database along with order entry, inventory and pricing data.  This is typically done because using a directory system for such purposes is difficult and, at times, not as functional as when authentication data is stored in and used from a database.  Given the prevalence of SQL attacks, all of that information results in being available for the taking through a SQL injection attack.  As a result, the attackers compromise the authentication system, gain access to everyone’s credentials, including administrators, and it is likely ‘game over’ regarding the rest of your security measures.

I want to touch next on social engineering because it is typically directly related to compromising authentication systems even though the second place attack method most concerning Verizon researchers is application attacks.  Social engineering is all about tricking your end users into giving up key information so that an attacker can compromise your environment.  The most common piece of information an attacker tries to obtain is an end user’s credentials for logging onto the network.  Hence the reason why I wanted to discuss this after the authentication system attacks.  Social engineering is the most insidious of attack methods because it does not involve any of an organization’s security technology.  And worst of all, if social engineering is successful, all or most of your organization’s security technology is effectively neutralized as a result.  That is because most organizations have little or no security once someone is on the inside.

Now let us look at the second most concerning attack to Verizon which is application attacks.  Verizon is saying that application attacks are more of a threat to governments and large applications.  Regardless of the target, any organization with an application presence on the Internet is a potential target such as with eCommerce or Internet banking.  A lot of these applications are based on on-line frameworks such as IBM’s Websphere or Oracle’s Application Framework.  It is not that these frameworks are insecure, it is that they still require development effort and it is those custom development efforts that do not guarantee a secure application.  The problem comes from the fact that a lot of people believe that using a framework means that little to no security testing even though the amount of custom development done in these frameworks can be more extensive than starting from scratch.  As a result, we see a lot of organizations tossing Internet applications into production with little or no security testing and then ending up with breaches as a result.

In addition, there are third party applications served up by application service providers (ASP).  A lot of small and mid-sized businesses (SMB) use these sorts of applications to have an online presence.  As a result, a lot of SMBs believe that these solutions do not require any security testing because the vendor and the ASP do that for them.  However, we are encountering more and more attacks on SMBs, particularly those that have wealthy clientele such as country clubs and exclusive financial institutions because their applications are notorious for not being secure.  SMBs are constantly amazed that; (1) they were targeted and, (2) the application was not bettered secured.  Yet attackers know that while the take out of SMBs could be significantly less than a large organization, an SMB is usually easier to compromise because they do not have the security and monitoring resources of a large organization.  As a result, SMBs are becoming a larger and larger target for attackers.

Finally, something that concerns me as the previously discussed threats is mobile devices and devices not under the organization’s control, also known as ‘bring your own device’ or BYOD.  I think these devices will surpass the other three threats over the next few years because most organizations have difficulty maintaining security on their servers, desktops and notebooks, let alone something like an iPhone or an Android tablet.  The worst thing about mobile devices is that they are so easily lost and it fascinates me how many people lose their mobile devices.  The bottom line about mobile devices and BYOD is that you must be very, very careful as to what you allow these devices to access and how you grant that access.  And you must make sure that these devices are not allowed to download information, even if encrypted, as that information is highly likely to be lost.

So what should you be doing regarding these threats?  Here are my top things organizations should be doing to minimize the risks presented by these threats.

• Trust no one.  This is particularly true of mobile devices or BYODs, but it also applies to your own internal systems.  Forrester has promoted this in their ‘Zero Trust Model’ as well as me in the ‘Fort Knox Approach’.  This is not as easy as one might think, but the approach makes sense in these days of attacks on authentication systems and social engineering approaches.
• Classify your data.  This is usually a difficult project, but they pay dividends at the end because everyone understands why certain information cannot be allowed out of the control of the organization.  It also allows people to justify to others why data cannot be allowed to be accessed by people that do not need access as well as via mobile or BYOD.
• Require encryption on mobile devices and BYOD.  Even if you do not allow data to end up on these devices, you do not even want the memory or other information that might be inadvertently stored on these devices out of your control.  As a result, if you encrypt these devices, there is a high likelihood that, if they are lost, the person finding them will just wipe them and start over.
• When possible, use a directory system for authentication.  This is always painful for systems that operate outside of the traditional control environment of internal users.  Directory systems are usually designed to be more secure than any sort of database authentication system because they are assumed to be at risk from the start.  However, just because they are designed to be secure does not mean they cannot be implemented in an insecure manner.  Windows Active Directory takes a lot of heat for being insecure; however a lot of that heat is due to silly implementations to support insecure authentication methods for compatibility.
• Conduct security awareness training.  The only thing that minimizes social engineering is consistent and regular security awareness training.  However, do not kid yourself or management.  Everyone has their ‘moments’ and does something they should not.  That said there are always those that just never seem to get it which is why you need other controls and monitoring to ensure you maintain your security.  However, to just throw up your hands and say it is pointless is also not a position to take.
• Secure your applications.  This means conducting application code reviews and testing applications before they are put into production not after the fact.  Unlike networks where you need to put them into production before testing them, applications can be tested before going into production.  It amazes me how many organizations put their applications into production and, by the time they finally get around to testing them, they have already been compromised.  And while automated application code testing solutions are all of the rage, we still find that the best results come from the more traditional human code review not automated tools.
• Monitor your network and applications.  This is a double edge sword.  You know what to look for, however, you have so many ports open that it is near to impossible to recognize bad traffic from good traffic.  And it is not necessarily the fault of your IT department as most packaged applications require an inordinate amount of ports open to function properly.  However, the key thing to monitor, more than anything, is any traffic going outside of your network to an unknown location.  When you see traffic going to Eastern Europe, China or any unexpected IP address, your monitoring system should generate an alert as that is typically a key indicator that you have been compromised.

Have a happy holiday season and I will “see” you all next year.

The Barnes & Noble Breach Take Aways

On October 24, 2012 it was announced that Barnes & Noble had a credit card breach that was the result of tampered credit card terminals.  As a result of the breach, Barnes & Noble pulled all of the credit card terminals out of their stores so that they can be examined.  The story published in the New York Times has some points that should be interesting to other large merchants.

“We have acted at the direction of the U.S. government and they have specifically told us not to disclose it, and there we have complied.”

This is probably the most important take away you should have because a lot of incident response plans miss this point.  While the credit card companies want to the notified immediately of a breach, law enforcement should be the first outside entity notified and then the card companies, if approved by law enforcement.  The reason is that law enforcement may want the breach to continue in an effort to more easily identify and apprehend the perpetrators and that may include allowing the perpetrators to use the stolen cards for purchases.

But the next question that typically comes up is who in law enforcement should be notified?  If you are not a large or regional entity, then you should notify your local police department or county sheriff.  If you are a regional or large sized merchant in the United States, you should contact the United States Secret Service and/or the Federal Bureau of Investigation.  In either case, whatever law enforcement entity you contact should be consulted with before notifying anyone else outside the organization and that includes notifying the card brands.

“The company determined that only one keypad in each of the 63 stores had been hacked.  Nevertheless, the company has not reinstalled the devices.”

The 63 stores involved were all across the country from San Diego, Miami, Chicago, New York and other locations in between.  This implies either a very organized criminal group that operates in a lot of locations or to a localized group that was able to infiltrate the operation that configures and ships out the terminals for Barnes & Noble.  Based on investigations similar to this, it is most likely that a criminal operation infiltrated a centralized location that is responsible for the configuration, repair and replacement of credit card terminals for Barnes & Noble.

So what can a merchant do to minimize this sort of attack?  Here are some actions to consider.

• Contract with only a reliable terminal supplier.  In this age of lowest cost providers, there is a big temptation to use anyone as a supplier, particularly if their costs are the lowest.  However, the old adage of “you get what you pay for” is very relevant in these situations.  As part of your vendor selection process, you should be asking a supplier of terminals what they do to ensure that terminals do not get tampered with.  If you cannot get an answer or the answer you get is “trust us,” then you should probably not consider them as a vendor.  At a minimum, vendors should put their employees through periodic background checks (at least every three to five years), track which employees work on what units, do random physical internal inspections of units and random testing of units to ensure that they are not tampered with before they are sent out.  If you are doing this activity in-house, you should also be following this process.
• Lock down your terminals.  Anyone that has been into a Barnes & Noble might recall that terminals just sat on the counter.  As a result, they were easy to quickly swap out with a doctored unit.  I have been involved in a number of situations where merchants had terminals doctored because they were easy to swap out.  If terminals are locked in a cradle and only the manager on duty has the key, anyone trying to swap terminals is going to have to have a key to free the device.  This prevents swaps that occur after hours when only the cleaning people are present.  In addition, the keys to these terminal cradles needs to be different for each location so that one key does not open every cradle at every location.  The common key is a lesson the gas station industry has only recently addressed.
• Use tamper-proof serialized security tape or stickers over the seams of the terminal and check them daily.  This is a trick that has been used for quite a while with gas pumps.  The key is to at least daily (I recommend at each manager shift change); have the stickers checked to make sure that they are still in place and log that activity.  If they have been tampered with or are missing, the lane should be immediately taken out of service and your loss prevention unit contacted.
• Confirm a terminal swap.  A lot of merchants are very lax in their terminal swap procedures.  If a terminal turns up with instructions to swap it with another or a technician appears at the location with a new terminal, the store personnel do it, no questions asked.  That is wrong.  At a minimum, a good terminal swap procedure should involve the generation of a trouble ticket in a help desk system or similar and having the store manager confirm the swap with the help desk or POS support.  No ticket, no swap, no exceptions.
• Put video monitoring on all your POS locations.  This does not stop such a swap from occurring, but it does at least record such an event if it does occur.  This is particularly important in situations where the customer also acts as cashier as with any self checkout situation.
• Use MAC address filtering on your store location networks.  If a device is unplugged and a new device is plugged in with a different MAC address it will not work.  Yes, I know for some of you this creates a bad situation.  But I always ask people in response, “Why should store personnel be swapping equipment in the first place?”
• Monitor your sensitive devices.  If a credit card terminal or POS gets unplugged from your network, you should generate an alert.  That alert should then be correlated to a help desk ticket.  If there is no ticket, then someone should immediately notify loss prevention and also follow up with store management to find out why the device was unplugged.
• Monitor your network.  Terminals or POS should only be communicating with your service provider for transaction authorization and your routers(s) and/or firewall(s) should be configured accordingly.  If a terminal or POS attempts to communicate with any other external IP address, that should generate an alert to corporate IT and security that should then be investigated immediately.  This will catch those devices that are tampered with and then transfer data to a server outside of your network.  It is highly likely that the communication will be encrypted, but the traffic will be directed to an external IP address that should be blocked if your firewall(s) or router(s) are configured properly.

UPDATE – October 24, 2012: I am quoted on the Financial Times Web site regarding the B&N breach. http://www.ft.com/intl/cms/s/0/919e3292-1dd7-11e2-8e1d-00144feabdc0.html#axzz2BjZ8C9WJ

UPDATE – November 1, 2012: I got to be a guest blogger on CNBC because of this blog entry. http://www.cnbc.com/id/49624444/How_Retailers_Can_Avoid_a_Credit_Card_Breach

UPDATE – November 5, 2012: I am interviewed by a USA Today reporter. http://www.usatoday.com/story/tech/2012/11/05/debit-card-account-theft-qa/1677537/

UPDATE – November 6, 2012: I am quoted on the front page of USA Today. http://www.usatoday.com/story/tech/personal/2012/11/05/debit-card–numbers-pins-stolen-at-pos-terminals/1675795/

UPDATE – November 8, 2012: I am interviewed on WCCO 830 Radio in Minneapolis.

The Amazon Cloud And PCI Compliance

If there ever was a hot topic these days it would be “The Cloud” and, in particular, the Amazon cloud.  And that discussion inevitably leads to how are the Amazon cloud offerings are PCI compliant?  A lot of this discussion has to do with the very limited amount of information regarding the Amazon service offerings.  For some very bizarre reason, Amazon puts organizations interested in their PCI compliant services in a Catch-22 situation.  Unless you sign up for one or more of the services, you cannot obtain the information on how the Amazon service offerings are PCI compliant.  As a result, there is a lot of mis-information running around regarding the Amazon cloud.  So to debunk all of the myths running around, I thought I would explain what the Amazon cloud is and is not and how it ends up PCI compliant and what you need to understand when deciding to use the Amazon cloud.

And before I get calls from someone at AWS about the fact that I am somehow singling them out or I am being unfair.  I do not have a problem with AWs or anyone organizations’ cloud service offerings.  What I have an issue with is how some service providers use obfuscation and confusion about their services in ways that make customers unsure of whether they are getting something that is PCI compliant.  As I see it, the AWS service offerings seem to be PCI compliant, but there are things that possibly should be further explained so that everyone understands how that compliance is achieved.

The first part of the mythology revolves around what PCI compliant services Amazon Web Services, LLC (AWS) is actually providing.  According to AWS’s Attestation Of Compliance (AOC), AWS is a Hosting Provider for Web and Hardware.  The AOC calls out that the following services have been assessed PCI compliant.

• Amazon Elastic Compute Cloud (EC2);
• Amazon Virtual Private Cloud (VPC);
• Amazon Simple Storage Services (S3);
• Amazon Elastic Block Store (EBS);
• Amazon Relational Database Service (RDS);
• Amazon Elastic Load Balancing (ELB); and
• Amazon Identity and Access Management (IAM).

The AOC lists nothing for software provided through any of their services.  As a result, a big myth that gets busted right off the bat is that AWS is providing software.  At the end of the day, all AWS’s services are offering is Infrastructure as a Service (IaaS).  As a result, how AWS is PCI compliant is fairly easy to figure out.  They have totally minimized their responsibility on the PCI compliance front.

In addition to the AOC, AWS provides customers with a document entitled “AWS PCI DSS Controls Responsibility Summary” (CRS).  This document explains the various services and the responsibilities a customer organization has when using these services.

The first piece of infrastructure used by AWS is virtualization in the form of Xen as their hypervisor.  Because of the way AWS has implemented Xen, every virtual instances created by EC2 acts like an individual physical server in that there are no connections to any other server unless the organization defines such connections.  This is referred to in the CRS as instance isolation.  Finally comes the firewall.  EC2 includes a firewall that is managed by the customer.  Access to the firewall is controlled by an X.509 certification and access credentials provided through IAM.  In addition to utilities to manage the cloud environment, AWS provides various application programming interfaces (API) to manage the AWS cloud environment.

The bottom line is that, at a minimum, an organization needs to subscribe to EC2, VPC and S3 in order to build a basic platform capable of computing (i.e., server, connectivity and storage).  The need for other services outside of these will depend on what the organization is attempting to accomplish, whether or not they need the flexibility and scalability provided by AWS and other business factors.

From a PCI compliance perspective, the CRS categorizes the 12 PCI requirements into those that are AWS’s responsibility, shared responsibility between AWS and their customer and those requirements that are solely the customer’s responsibility.

In the AWS is responsible category falls requirement 9 or physical security and environment controls.  Since AWS is providing the facilities to operating the underlying physical hardware, it is solely responsible for this requirement.

In the shared responsibility category falls requirements 1, 10 and 11.  For requirement 1, AWS acknowledges that this is a shared compliance responsibility between AWS and their customer.  However, AWS’s responsibility is only to provide a firewall and ensure that it segregates their customers from one another.  The remainder of the responsibility for complying with requirement 1 is left to the customer.

For requirement 10, AWS indicates that they are responsible for:

• Maintaining log files for EC2 and S3 customer management operations (e.g. creation, modifications and deletion of these environments) for at least a year.
• Maintaining logs for the underlying software that provides the various services for at least a year.

This log information is monitored at least daily and is available to customers for their particular environment should it be necessary.  All other parts of requirement 10 are the responsibility of the customer.

For requirement 11, AWS indicates that they are responsible for ensuring the security of their environment including ensuring wireless security.  Customers are responsible for ensuring the security of the environments they construct using AWS’s services.

All of the remaining requirements, 2, 3, 4, 5, 6, 7, 8 and 12 are solely the responsibility of the customer.

So after all of this rigmarole, what is the advantage to be gained?  Not much near as I can tell.  The bulk of responsibility for PCI compliance still falls on the organization using the AWS services.  So organizations looking to offload as much of their PCI compliance responsibilities as they can to AWS are looking in the wrong place.

But it does not end there.  We are seeing more and more startup service providers that are using AWS services to avoid the capital costs of hardware and software of a 24/7/365 operation.  Where this becomes tricky is when you have a service provider providing PCI compliant services effectively using AWS for their “data center.”  In some cases, these service providers are trading on the fact that because AWS is PCI compliant, then their services must also be compliant.  However, what these service providers forget is that once they start going beyond the IaaS model and offer services in the Platform as a Service (PaaS) and Software as a Service (SaaS) realm, they are now responsible for portions of PCI compliant that Amazon is not.  As a result, organizations need to conduct due diligence on vendors using other cloud providers to provide their services to ensure that everyone is PCI compliant.

So do I think your organization should rush right out and sign up for AWS?  Maybe if you have the right business case.  But I do have some concerns regarding AWS’s service offerings and the statements surrounding how they are PCI compliant.

My first concern is in regards to requirement 1.2.3.  This requirement is one of the few that is not allowed to be marked ‘Not Applicable’.  As such, the QSA is required to document what procedures they conducted to ensure that any existing wireless is either not in-scope or that there is wireless in-scope and how it is secured.  To document this, AWS’s QSA has written:

“[AWS] maintain this control for all internal and external services that it provides. In EC2 and VPC environments, this includes the network at the hardware and management level networks, which are not exposed to customers.”

This statement says nothing of what procedures were conducted to ensure that wireless was not visible to customers as well as the controls AWS maintains to ensure wireless stays out of scope.  Essentially, we are asked to trust AWS that wireless is not on any customer networks.  Now, to be fair, AWS is operating secured data centers comprised with racks of hardware all virtualized, so the likelihood that wireless would exist in such an environment on any one customer’s network is remote at best. .  However, the PCI assessment process is all about verifying such statements, not just accepting them at face value as fact.  As a result, I am concerned that what is supplied as evidence for complying with this test leaves much to be desired.  What should be documented here are the procedures the QSA used to confirm that the controls AWS has in place are adequate to ensure that rogue wireless does not end up in their data centers.

Related to requirement 1.2.3 is requirement 11.1.  As with 1.2.3, 11.1 is also not allowed to be marked as ‘Not Applicable’ regardless of whether wireless is implemented or not.  For all of the tests under 11.1, the following statement is made.

“[AWS] maintain[s] this control internally.”

So what exactly does AWS do to ensure that their data centers remain wireless free or that wireless does not end up on the customer side of the network?  No idea.  I would like to assume that AWS is doing the right things in this regard, but, again, the PCI assessment process does not allow for assumptions, they require proof and this just does not pass muster.  At a minimum, there should be a discussion of the procedures used by AWS to ensure wireless is not an issue.

While we are discussing requirement 11, we should cover vulnerability scanning, penetration testing, intrusion detection and critical file monitoring.  All of which are the customer’s responsibility, not AWS’s.  Again, AWS is providing IaaS and nothing else, so any such controls will need to be provided by the customer.

When reviewing the detailed responses in requirement 9, it was interesting to see that AWS is responsible for ensuring that for the portion of any customer’s cardholder data environment (CDE) that exists in AWS, AWS ensures that destruction of hardcopy materials are properly destroyed so to be unrecoverable.  This begs the question, “Why would AWS have any hardcopy to destroy in the first place if they do not have access to customers’ environments?”  No further explanation is given, but one would guess it was their lawyer’s idea just in case AWS might somehow come into contact with CHD on hardcopy.

The next area I have issue with is not related to the service, but related to how an organization contracts for the service.  In an effort to fully automate things, unless you are a Fortune 50 looking to put your entire computing environment in AWS’s data centers, you can forget about negotiating a contract.  When you sign up for any AWS service, you either accept their contractual terms and conditions by checking the ‘Accept’ box and clicking Okay, or you don’t get to use AWS.  I know of a number of organizations that had real issues with that approach and, as a result, backed away from a more aggressive use of the AWS environment or decided they just could not accept the terms and did not go to the cloud at all.  While the AWS contract does cover PCI compliance, but it essentially makes the customer the one legally responsible for compliance with AWS providing support when necessary.

So that is AWS in a nutshell.  Not a bad thing, but something an organization needs to go into with their eyes wide open and understanding that they still have significant responsibilities even though they are now in “The Cloud.”

Third Party Service Providers And PCI Compliance

There seems to be a lot of confusion regarding third parties that provide networking or hosting services and their obligations regarding PCI compliance.  This confusion is not uncommon as merchants and their service providers have not necessarily been provided enough guidance to understand their obligations.  I hope this post will clarify those obligations for all involved.

If you learn nothing else from this post, if a third party is providing your organization a service that has access to your cardholder data environment (CDE) or the third party could come into contact with your cardholder data (CHD), then that third party must ensure that the service complies with all relevant PCI requirements.  As a result, the third party needs to either allow you or your QSA to assess the services that they are providing or provide you with an Attestation Of Compliance (AOC) that documents that those services have been assessed and they are PCI compliant.

In the past, I have stated that third parties could also submit a letter signed by an officer of the third party stating that all of the services provided to their customer are PCI compliant.  Now that v2.0 of the PCI DSS has a separate AOC and the PCI SAQs have the AOC built into the SAQ, there should be no reason to need such a letter or to ask for one.  If a letter is what your third party is offering, it is better than nothing, but you should be pushing them hard for an AOC.  If they are reluctant to get you an AOC, as part of your vendor management process, you should take that into account and probably begin looking for a new vendor that will provide an AOC for their services.

The most common issue we run into with third parties is that their AOC or other representations of PCI compliance do not cover all of the services provided to the customer.  In case after case, we see the AOC covering requirements 9 and 12 and nothing else even though the services provided may require compliance with some or all of PCI requirements 1, 2, 3, 4, 5, 6, 7, 8, 10 and 11.

In a lot of cases, it is not that the third party does not want to comply with PCI; it is they are taking the lowest common denominator approach and only picked those services where all customers requiring PCI compliance are asking for an AOC.  That way they have reduced their costs of a QSA to assess their environment.  These third parties are accepting the fact that any customer that needs more services assessed will have to do it themselves.

Related to this issue is the third party that offers their SSAE 16 Service Organization Control (SOC) 1 report has proof of PCI compliance.  While a SOC 1 report can cover a few PCI requirements, people must remember that the SOC 1 report is structured specifically for financial auditors to ensure that the controls at a third party are properly constructed to support financial reporting at the customers.  As a result, a SOC 1 report is not going to be a substitute for an AOC that covers all services.  There is an alternative to this and that is to have the third party go through a SSAE SOC 2 report that focuses on the security controls of the PCI in-scope services provided.  We are hearing from third parties inquiring into the SOC 2 report, but cost and a lack of customers requesting such a report are driving why we do not see more SOC 2 reports available.

Another common issue we encounter is the refusal of the third party to cooperate in assessing the services provided to ensure they are PCI compliant.  There are still third parties that argue their services are not in-scope for PCI compliance even when it is painfully obvious that the third party’s personnel have access to their customer’s CDE and/or CHD.

The most common third party relationship we encounter is the management of routers or other layer 3 devices.  Where we encounter the most confusion in this relationship is in regards to the use of encryption to keep the network services organization out of scope for PCI compliance.  The key here is if the network services organization manages the encryption of the network, then they are in-scope for PCI compliance.  The reason is that the employees of the network services organization have access to the encryption keys and therefore could decrypt the communications and gain access to CHD transmitted over the network.  As a result, at a minimum, the network services organization is responsible for complying with some or all of requirements 1, 2, 4, 6, 7, 8, 9, 10 and 12.  If you receive such services and are not getting an AOC that covers these requirements, then you should be doing more work on your own as well as asking the third party why they are not covering more of the necessary PCI requirements.

The next most common service we encounter is the network services firm that is managing or monitoring an organization’s firewalls, remote access or intrusion detection/prevention.  Such services always put the third party in-scope for PCI compliance.  Some or all of requirements 1, 2, 6, 7, 8, 9 and 12 will need to be assessed for compliance with the PCI DSS.  The log capture and analysis requirements in requirement 10 may also be complied with if your organization is not capturing and analyzing the log data from these devices.

Another group of third parties we encounter a lot are records retention vendors.  Organizations like Iron Mountain have conducted their own PCI compliance project and readily hand out their AOC to customers.  However, where we see issues is with such vendors that provide their own tape library for their customers to use for backup.  We have encountered a number of third party’s doing the encryption at their library which puts them in-scope for PCI compliance, at a minimum, for requirements 3, 4, 6, 7, 8, 9, 10, 11 and 12.

We encounter outsourcing the data center a lot with large organizations, but small and mid-sized organizations are also hopping on the data center outsourcing bandwagon.  Where this puts the third party in-scope for PCI compliance is when the third party is responsible for maintaining the environment such as applying patches, managing servers or any other activities that would allow the third party’s personnel to potentially have access to CHD.  In such situations, at a minimum, the third party is responsible for complying with some or all of requirements 2, 5, 6, 7, 8, 9, 10 and 12.  Compliance with some or all of requirement 1 may be applicable if the third party is managing your firewalls or routers.  Compliance with some or all of requirements 3 and 4 may also be applicable if the third party is responsible for managing encryption keys for encrypting CHD or encrypting communications.

Where the most confusion regarding third party responsibilities occurs is in regards to “The Cloud.”  The most common reason for this is that every vendor seems to have a different definition for what “The Cloud” is, based on their particular services.  Using the definitions provided by the National Institute of Standards and Technology (NIST) in their publication SP800-145, ‘The NIST Definition Of Cloud Computing’, I can provide the following guidance.

If your organization is purchasing Infrastructure as a Service (IaaS), then the third party providing these services will typically be out of scope for PCI compliance except for requirements 9 and 12.  There are some instances where IaaS implementations may require compliance with the PCI DSS if the third party is managing network infrastructure that comes into contact with CHD as is usually the case with private cloud environments.

For Platform as a Service (PaaS) and Software as a Service (SaaS), the third party will have to provide PCI compliance for the services they are providing to your organization.  That is because with either of these service offerings, the third party must have access to the CDE and will have the potential of coming into contact with CHD.

The problem with the majority of PaaS and SaaS vendors is that they only deal with your organization through a Web-based interface, i.e., everything is automated – contracts, support, etc.  As a result, the contract is a “take it or leave it” situation that does not usually cover everything needed for PCI compliance, there is no way to independently verify the representations made by the third party as well as the fact that the AOC provided by the third party typically only covers only the physical security requirements in requirement 9 and possibly some of requirements 11 and 12 and nothing related to the other requirements, even though the third party may have responsibilities for PCI compliance outside of what is represented in their AOC.

If this is the case, there is little you or any QSA can do to properly assess the environment to ensure it is truly PCI compliant.  As a result, we have a lot of organizations that try to develop compensating controls for these cloud implementations.  These organizations very quickly and frustratingly find out that there are very few, if any, controls on their side of the equation that can get them to “above and beyond” the original requirement.

I know there are a lot of other examples of services being provided to merchants.  But, hopefully these examples can assist you in clarifying what you need or do not need from your third parties when it comes to PCI compliance.

Defense In Depth

I have a slide in my security presentation deck that discusses the concept of defense in depth and how when you start opening ports or start using encrypted data streams how you are punching holes into one or more of your security layers.  It amazes me how many people still do not understand how defense in depth works and how much security standards such as the PCI DSS rely on this concept.

So let us take a look at the various elements of security and the requirements of the PCI DSS and see how they bring defense in depth to bear.  Keep in mind this is an example and does not encompass everything an organization could do to increase defense in depth.

For most organizations, the first level of defense is at their firewall.  Requirements 1 and 2 talk to how you should use a firewall and secure it.  The biggest mistake that organizations make is not configuring their firewall properly.  And by configuration, I am not just talking about the configuration of the firewall’s software; I am also talking about where and how the firewall is used in the network.

The next level of defense for most networks is usually some form of intrusion detection/ prevention system.  Some of the requirements in 10 and 11 talk to intrusion detection/prevention.  IDS/IPS capability may be provided in a separate appliance or may be part of an organization’s firewall.  The key to using an IDS/IPS is ensuring that it is kept current with its attack signatures, monitoring its log data and/or console and ensuring that it is not be overwhelmed by network traffic.

One thing that continues to amaze me is how many implementations of IDS/IPS I encounter where the IDS/IPS are in the middle of encrypted data streams.  IDS/IPS systems cannot examine encrypted data streams unless they have the decryption keys which they typically do not have access.  As a result, encrypted data streams are not examined and therefore sensitive data and or attacks could be going right past the IDS/IPS.

How users authenticate to your network and devices is also a level of defense.  Requirements 7 and 8 of the PCI DSS talk to this point.  And it is not just authentication to applications that process, store or transmit cardholder data, it is also authentication to infrastructure devices and to databases.

It has been more than five years since the “sa” default password debacle and yet you still encounter applications that use service accounts to access their database and those service accounts have no password.  The rationale?  “We did not want to code the password into the application,” is the common reply.

The other big area of authentication issues that you encounter is with firewalls, routers switches and other network infrastructure.  The problem is that the network administrators all use the same account and password.  You can understand their rationale, particularly those networks where you are administering thousands and thousands of devices.

There are a number of ways to address this situation, but these are my favorite two.  The first is to implement 802.1X authentication using a RADIUS server.  Under this scenario, every network administrator has their own unique account and password to access the network devices.  Those unique accounts should be different from the network administrator’s account they use to get email and network access like every other user.  A lot of organizations already have the RADIUS server implemented for remote access, so adding in network administration access control is relatively easy.

The second way to address network administration access is to use a “jump box.”  In a “jump box” implementation, two or more “jump boxes” are placed at strategic points on the network and all network administration access is conducted through a “jump box.”  The “jump box” is fully instrumented in that all keystrokes, applications, etc. are logged and those logs are reviewed at least daily to ensure that network administrators are not changing things they should not be changing.  That means comparing service tickets for the network against the logs from the “jump boxes” and ensuring that only what was required to be changed was changed.  “Jump boxes” can also be used to control access for server administration.

A level of defense that usually gets little recognition is operating system (OS) hardening.  What some people seem to forget is that any computerized device has an OS whether it is a firewall, router, switch or server.  Requirement 2 talks not only to the hardening of wireless, but also firewalls, switches, routers and servers.  Every vendor publishes a guide that explains how to securely implement their OS.  Where things can get sticky is with third parties that argue that their product or software will not function if you follow the vendor’s OS hardening recommendations.  In my experience, testing a vendor’s product or software in a hardened environment typically does not have an adverse result.  However, the key is to conduct a test.

Another level of defense is anti-virus and anti-malware software.  This solution also usually includes a personal firewall on mobile devices such as notebooks, netbooks and smartphones.  Requirement 5 of the PCI DSS talks to anti-virus and anti-malware while a requirement in 1 talks to personal firewalls.  Nothing gets some people wound up more than anti-virus software.  The requirements in 5 can have compensating controls, but implementing those compensating controls consistently on mobile devices is usually just about impossible.  So while you may not have anti-virus/malware on your e-Commerce servers, you should have it on all of your desktops, notebooks, netbooks and other systems.

A level of defense that most organizations poorly manage is their collection and analysis of log data from their network devices and servers.  Requirement 10 speaks to the importance of log data.  As I have written before, log data is IT’s version of commercial aircraft’s flight data recorder.  If you want to why a problem occurred, log data from your devices can usually point you to the reason.  The problem most IT professionals have with log data is that they do not want to log everything because that generates too much data in their opinion.  However, until you have an incident, you do not know what log data will be important in identifying why the incident occurred therefore you need all of it.  The last thing you want to have happen is to tell management that you could not determine the cause of an incident because you did not record the critical information required in identifying the incident.

The final defense most people think of is application development which is covered by requirement 6.  If you are going to get push back, this is the most likely and consistent place you will get push back to the PCI DSS or any other security program.  Application developers are very protective of their environments, so when you start infringing in their area, they can get rather upset.  As a result, you hear the typical lament from developers that security, “restricts their creativity.”

In today’s rush to get things done, application developers usually do not have security at the front of their minds.  As a result, by the time anyone knows that there is a security issue, it is too late for it to be fixed and the application goes into production with the fix to be part of version 2.  That was the whole point the PCI DSS addresses in requirements 6.4, 6.5 and 6.6; avoid putting susceptible software into production.  The whole point of these requirements is to build a certain amount of security into the development process to minimize the amount of security issues that end up in production.

The real final defense is an organization’s policies, standards and procedures.  Yes, that paperwork that everyone thinks is “make do” work, really does have a purpose.  An organization’s policies, standards and procedures are the rules that everyone is to follow to ensure security.  Those rules also provide a way to measure people’s compliance so that, in the event of an incident, those people that did not follow the policies, standards or procedures can be shown their mistakes so that they can correct their actions in the future.  These rules also provide an organization’s framework for explaining to personnel as to how the organization is protecting their information assets and defining those assets.

There are a lot more options for defense in depth, but I think you get the idea.  Now that you understand how defense in depth works, you should now also understand what happens when security personnel are asked to open ports for an application or change configurations that reduce the number of levels in an organization’s defenses.  The fewer levels involved the higher the likelihood that a lapse in control can result in a breach, particularly when a number of lapses in controls are occurring simultaneously.  This is how supposedly PCI compliant organizations end up breached.

2011 Verizon Breach Report

At this year’s Community Meeting, Christopher Novak of Verizon Business Services made a presentation regarding the 2011 Data Breach Investigations Report.  A lot of people blow off these sorts of presentations as being vendor focused or just a bunch of meaningless statistics.  However, Verizon provided a very good presentation on the nature of today’s threats and what organizations should be doing to protect their networks and systems.  Verizon’s Data Breach reports are a wealth of information; however I wanted to highlight just a couple of things that all organizations should take away from the report.

The first take away from the Verizon statistics is that while the media focus on the threat of insiders, it is the threat presented by external parties that is accounting for 92% of breaches, up 22% over 2010.  Not that insiders are not a threat, but only 17% of breaches involved insiders, down 31% over 2010.  Threats from business partners were less than 1% and only 9% involved multiple parties.  Yes, all of that adds to more than 100%, but as Mr. Novak reminded us, there are numerous instances while external actors ultimately breached the organization they were assisted by one or more insiders or a business partner.

The next important set of statistics is an analysis of breaches against the PCI DSS and its requirements for organizations that were in-scope for PCI compliance.  The first statistic that every organization should be concerned about is that Verizon found that 89% of all PCI in-scope organizations breached had been assessed by a QSA or themselves as not compliant with the PCI DSS.  This statistic should be a wake-up call to all organizations that complying with the PCI DSS is important.

In reviewing this demographic mix and the associated lack of compliance, we believe that the data reinforces an assertion we’ve been making for the past three years: to reduce risk, organizations of all sizes need to implement the basic tenets of an information risk management program and maintain this initial investment over time.  This includes network and data defense technology basics (firewalls, anti-virus, identity and access management), as well as the non-technical aspects of security and risk management (policy and process development).
Verizon Business Services 2011 Data Breach Investigations Report – page 62

PCI DSS Compliance and Breach Correlation

But the really important statistics were the analysis of compliance against each of the 12 PCI DSS requirements when a breach occurred.  What this analysis shows is that there are a lot of areas for improvement.

For building and maintaining a secure network, less than 50% of breached organizations were compliant with requirements 1 and 2 of the PCI DSS.  In protecting cardholder data, while breached organizations (requirements 3 and 4) were better at protecting cardholder data when it was transmitted (63%), they were horrible at protecting cardholder data when it is stored (43%).

It appears that organizations were complying with requirement 5 at 70% compliance.  However, given other survey results that claim over 95% of organizations have anti-virus and anti-malware, this is a troubling statistic as it points to incompetence in the installation and operation of these tools.

As I would expect, less than 50% of organizations were in compliance with requirement 6.  Whether that is related to patching or poor development practices, this part of the PCI DSS is always a bone of contention.

While overall, the requirements to implement strong access control measures, requirement 8 lags physical security and having access control systems implemented.  What this says is that while organizations have the tools in place, they are doing a poor job of managing them.

Regularly monitoring and testing networks is probably the area that needs the most improvement but explains another statistic in the report.  Compliance with requirement 10 and 11 were only 39% and 38% respectively when a breach occurred.  Earlier in the report, Verizon has a bubble chart that shows how long it took to breach an organization, how long it took the breached organization to acknowledge they were breached and then the time it took to address the breach.  It took attackers hours to days to breach an organization.  Unfortunately, it took weeks or months for an organization to know they were breached.  And to add insult to injury, it took additional weeks to contain the breach.

And finally, compliance with requirement 12 was only 44%.  A lot of organizations blow off documentation.  But it is the foundation of an organization’s security posture because it explains and describes why the other 11 requirements are important.  Without that foundation, is it any wonder that the other requirements are not in compliance.

One of the lingering questions from our discussions around PCI in this report is always that of relevancy. It’s all well and good to validate compliance with the PCI DSS, but does it actually help reduce risk? Insofar as that translates to a sincere security program—one that seeks to maintain validation on an ongoing basis—the data strongly suggests the answer is “yes.”
Verizon Business Services 2011 Data Breach Investigations Report – page 64

So what should an organization focus on to improve their security posture?  Here are my thoughts on a shortlist of issues that, if dealt with, would go a long way in securing organizations.

1. Really monitor your networks and systems.  No, I am not suggesting that everyone needs to go out and invest in a security incident and event manager (SIEM) or similar log management and analysis solution, but some organizations have no other choices because of their size and situation.  Regardless of how you choose to approach this problem, the key is that you must monitor your networks and systems.  And you need to monitor them regularly (daily) if not in real time.  The best organizations are doing real time monitoring and go through the PCI DSS tests and develop rules to monitor compliance with those requirements.  Attackers know that they can get away with their breaches because organizations are lax in their monitoring.  Just tightening monitoring up would go a long way in reducing the amount of data breached if not reducing the number of breaches.
2. Manage your access control systems.  No, it is not sexy to review user lists on a quarterly basis and disable, remove or reclassify users.  But here is a prime example of where organizations have a tool and are not using it correctly to improve their security.  Based on the report, it turns out that a lot of breaches were the result of poor user management practices and lousy passwords.  If this area got addressed, it would also go a long way in cleaning up the problem.
3. Engineer your networks and systems to be secure.  It fascinates me how many security professionals do things in the name of expediency that open huge holes in an organization’s security posture.  Another fascinating thing is how many security professionals have little knowledge of operating systems and applications.  Most security professionals seem to come out of the network administration teams as they were maintaining firewalls and routers when they went into security.  As a result, while they are very good at securing the network, server and application security lag because they just do not understand it nor do some find it all that important.
4. Test your security.  The only reason I can figure that vulnerability scanning and penetration testing is not being performed is that it will be used against security personnel to show that they are not doing their job.  While there are those security personnel that talk the talk but do not walk the walk, the vast majority of security professionals do an admirable job with few resources and even fewer plaudits.  However, given the rate of vulnerability production by attackers and researchers, if an organization is not conducting quarterly or even monthly vulnerability testing, there is no way an organization is going to protect itself.  And while the value of penetration testing will probably always be argued, I can tell you that it does have its place and can be invaluable if properly conducted.  This is particularly true for all of those organizations that think that because their vulnerabilities are all less than CVSS 4 that they are secure.
5. It is time to get control of application patching.  For small organizations, patching is only an issue due to a lack of focus on it as an issue.  For large organizations, it is the sheer volume of systems and devices that need patching.  There are numerous solutions for large organizations to deal with this situation, so find the tool that best fits your organization and get it implemented.  Oh, and get off the 30 day issue, that was clarified years ago.  You only need to have a reliable process and demonstrate that your patching process works on your time frame and patches do not fall between the cracks.

If all organizations could just focus on these five items and execute them at least 90% or better all of the time, a lot of the current breach issues would go away.  But that is the rub is it not?  A lot of organizations have issues maintaining that level of execution compliance.  Breaches occur because organizations get sloppy and, even with defense in depth in their security controls, there are too many controls where execution consistency has dropped leaving gaping holes in the various levels of security.

And as a reminder, these items address today’s problems.  However, once addressed, attackers will find other ways in, so the improvement process needs to be continuous.  And while organizations address the latest security concerns, they must be vigilant to cover the prior concerns because old vulnerabilities and exploits have a way of coming back.

Draft PCI DSS v2.0 “Scorecard” Released

It has been a long wait, but the PCI SSC has finally given us a look at the new “scorecard” for v2.0 of the PCI DSS.  For those of you that never knew about the “scorecard,” it was given to QSAs to assess Reports On Compliance (ROCs) to ensure that QSAs have properly conducted PCI assessments.  I have not had a chance to get through all 112 pages of this document, but I have gotten through the first part of it and I wanted to share my thoughts.

The first change to the “scorecard” is its name.  It is no longer the “scorecard,” it is titled ‘ROC Reporting Instructions for PCI DSS v2.0’.  The naming seems to indicate that once the QSA review period is over it will be posted to the PCI SSC’s Web site in the Documents Library.

Overall, the document is similar to the scorecard for v1.2.1, but no longer documents the scores that the PCI SSC QA team will use to assess QSAs.  However, from the way it is written, I would assume that if a requirement in the ROC does not contain everything documented in the Reporting Instructions, that it is considered to have not met the QA requirement.

Another general comment I have is that it is woefully lacking in examples.  While there seems to be a significant amount of guidance provided for what to write in the ROC, there are also ambiguous or unclear references that could be explained if the PCI SSC provided relevant examples of what they desires the QSAs to write.

The biggest change I have found thus far is the removal of the requirement to observe network traffic as the Network Monitoring column is gone from the Reporting Instructions.  Prior to this point, QSAs were required to obtain network traffic via WireShark or similar tool to prove that network traffic is encrypted.  I reviewed requirements 1.2.1.a, 1.2.1.b, 3.2.1, 3.2.2, 3.2.3 and 11.4.a that had the Network Monitoring requirement in the v1.2.1 scorecard.  Based on the training for the 2011 QSA recertification, networking monitoring testing is still something needed for confirming compliance with requirement 1, so even though it has been removed as a column, it appears to still be required.  However, from the Reporting Instructions, the network monitoring is not explicit, so this is one of those areas where the PCI SSC will definitely need to clarify things.

The section in the Executive Summary at the front of the ROC that discusses how a network is segmented to minimize scope will now require a fairly detailed discussion regarding that segmentation.  All network segments need to be described along with their purpose as well as a discussion of how the segments are architected and whether the segments carry cardholder data (CHD).  If access is provided to the cardholder data environment (CDE), that access needs to be described and that description needs to document how access is controlled.  It is very clear from the write up surrounding this section that QSAs and their clients will have to put much more work into this section to satisfy the PCI SSC.

Another clarification area is with the review of system configurations done as part of requirements 1 and 2.  The guidance now given by the PCI SSC is that they no longer want the documentation to be a list of configuration files that were reviewed by the QSA.  However, in the next breath, the Reporting Instructions tell the reader that a QSA must provide enough detail to prove that configuration files were reviewed.  So what is an acceptable level of detail?  Can we say that we reviewed 5 or 25 firewall configuration files?  In the past, we were told that this sort of approach was unacceptable.  The PCI SSC will need to provide one or more examples of language that they will accept.

Of all of the things I have read thus far, the one that just gets me seething is from the “Dos and Don’ts” page.  One of the “Don’ts” is “Don’t copy responses from one Testing Procedure to another.”  Further down on the list is “Don’t cross reference between responses”.  After going through our QA assessment and remediation, we were told by the QA person that we needed to do a better job of putting all of the information from earlier requirements that was relevant into every requirement as each requirement needs to be able to stand on its own.  But now, according to the Reporting Instructions, you cannot bring all of that documentation to the new requirement by using cut and paste.  What a bunch of “make do” work.

But this “make do” work is all because the PCI SSC is basically implying that it cannot trust its QSACs to do the work that is required to ensure an organization is complying with the PCI DSS.  However, just because a QSA writes something in a ROC does not mean they actually did the work.  It just means that the QSA knows how to write what the PCI SSC wants to read.  And to make matters worse, the PCI SSC provides the Reporting Instructions to provide guidance on just what to write as well as telling QSACs to develop ROC templates to speed the writing process.

A prime example of this is a new requirement in the section of the ROC where the QSA documents the list of people interviewed and/or observed.  The PCI SSC now requires the QSA to document what these people were interviewed about or were observed doing.  The purpose of this new requirement is to provide even more “proof” that a QSA did their job.  Another minor example of the PCI SSC trying to get “proof” of a QSA’s work effort is the increase in the level of detail being asked to document is the dates that QSAs were on-site for fieldwork.  In the past, QSAs were only required to document the period covered by the assessment.  However, QSAs are now being required to also document all of the dates of their fieldwork as well as the duration of their fieldwork and review period.

This is one of my biggest issues with the ROC process.  The PCI SSC refuses to adopt a more intelligent and cost effective reporting process of documenting requirement exceptions.  Instead, the PCI SSC requires QSAs to document their fieldwork process in the report.  As a result, an inordinate amount of time, paper and hence money is spent on what is really, in my humble opinion, a totally worthless effort.

I understand why this was required.  When the PCI SSC did not have the right to review a QSA’s work papers and other documentation, having such documentation in the ROC was the only way the PCI SSC, card brands and acquiring banks could assess whether or not a QSA had done their job.  Now that more than a year has gone by since the PCI SSC required all QSACs to include verbiage that allows the PCI SSC to review a QSAC’s work papers, putting all of this effort into a response writing requirement should no longer be required.  QSAs should be able to mark a requirement either ‘In Place’ or ‘Not In Place’.  If a requirement is ‘Not In Place’, then the QSA should document why the requirement is not in place and what the organization is doing to remediate the problem and when the remediation will be complete.  Such an approach would make the creation of the ROC much faster and would make the ROC much quicker to read and easier to understand.  This is the approach used in the accounting industry for their SSAE 16 reports and there is no reason why the PCI SSC could not adopt the same approach.

The PCI SSC continues to cling to this inane reporting requirement because it apparently is relying on the readers of the ROCs to “rat out” those QSACs that are producing inadequate reports.  I hate to be the bearer of bad news, but based on my review of ROCs from other QSACs that I have encountered over the last year, the “bad eggs” are not being weeded out.  Based on my interaction with acquiring banks and various card brands, there are a lot of ROCs that are not being read in detail.  And even those ROCs that are being read, most comments surround anything that is determined to have been ‘Not In Place’.  Occasionally we get a question about an ‘In Place’ item.  Obviously the current approach is not working and as long as the PCI SSC continues this approach, we are not going to build trust between the PCI SSC and QSACs.

I know that this is a dilemma for the PCI SSC, but it is something that needs to be addressed and soon.  Organizations that have to go through the ROC process are pressuring QSAs to reduce costs as much as possible not only due to our current economic conditions but also because of the thin margins retailers live on.  In order to keep the PCI compliance process relevant, the PCI SSC needs to get out in front of this issue.  The PCI DSS assessment process is very labor intensive, so the only cost savings to be obtained will be in making the process less labor intensive.

UPDATE: On the morning of September 20, 2011, the PCI SSC released the final version of the Reporting Instructions along with an FAQ.  These documents can be obtained from the PCI SSC Document Library under the Addition Documents – QSA heading.

The “Magic” Vulnerability – Revised

What started this post is that I have recently received a number of calls and messages from clients and colleagues.  The conversations have all gone basically the same.  They were calling me and telling me that their ASV had failed their vulnerability scan because the OS detected was unsupported and they are wondering whether or not I have encountered this before.

My first question usually was along the lines of; “So, what vulnerabilities did they detect?”

“None,” was the confused answer at the other end of the line.

“What?  They must have detected at least one high, severe or critical vulnerability?  That is the only way you can fail,” I would ask, now also confused.

“Nope.  Nothing.  Just the fact that the OS is unsupported,” I was told.

Do not get me wrong.  I am not advocating the use of unsupported operating systems, particularly unsupported versions of Windows.  The risk of course is that one or more vulnerabilities show up that the vendor will not fix because the OS is no longer supported.  So there is good reason to avoid this situation.  However, there are also situations when you just get no other choice either due to your own organization’s issues and politics or software vendor issues.

This situation got me thinking and doing some research since I did not remember ever seeing or being told that an unsupported OS was an automatic vulnerability scan failure.  I no longer do external vulnerability scanning, so my recollections of training and working on the ASV side of our business is a bit fuzzy and very rusty.  However, I had never failed a client for an unsupported OS.  So when this issue came up, my only action was to determine what had changed.

The first thing I did was review the latest version of the PCI ASV Scanning Procedures, v1.1.  I searched for terms such as ‘old’, ‘unsupported’, ‘out of date’, ‘OS’ and ‘operating system’.  No matches.  So there is nothing in the ASV scanning procedures that fail an organization for running an unsupported OS.  Even the PCI DSS does not call out unsupported software, so procedurally; I am thinking there is nothing explicit regarding unsupported OSes causing a failed vulnerability scan.

So when I made the original posting, I got a comment from one of my readers pointing me to the ASV Program Guide.  Low and behold on the top of page 16 is the following:

“The ASV scan solution must be able to verify that the operating system is patched for these known exploits. The ASV scanning solution must also be able to determine the version of the operating system and whether it is an older version no longer supported by the vendor, in which case it must be marked as an automatic failure by the ASV. “

So there is no “magic” vulnerability I was missing as the PCI SSC does specify that a scan automatically fails if the OS is unsupported.

But that is not the entire story.  The key to this whole process is that the vulnerability scanner used must be able to verify the operating system.  While all vulnerability scanners attempt to identify the operating system, the reliability of this identification process is suspect at best.  I am not aware of any vendor of security testing tools that makes a claim that they will identify an operating system 100% of the time.  This is because of the fact that there are many, many things that can influence the OS signature that the tools cannot control and therefore can greatly affect the ability of the tool to identify the OS, particularly when talking about external scanning.  And if an organization follows the OS security hardening guidelines, a lot of unsupported OSes will not be properly or reliably identified by vulnerability scanners.  As a result, I find it hard to believe that the PCI SSC intended to have ASVs only rely on the results of a vulnerability scanner, but that seems to be the case.

So with this clarification, I contacted our ASV personnel and they have told me that they too have been failing vulnerability scans if they run across unsupported operating systems.  I ask if the OS signature is inconclusive, then there is not a failure?  Yes, if the scan comes back and does not identify the OS, then they have nothing to go on to fail the scan and the scan passes.  Given the difficulties vulnerability scanners can have identifying the target operating systems such as when scanning through network firewalls, Web application firewalls, load balancers and the like, I now ask if they feel that these identifications are reliable enough to fail a scan.  I am told this is why they confirm the information with the client before issuing the report so that the report is accurate.  So if a client is not honest, they could influence the results of their scan?  I am reluctantly told that is probably true.

Then there is the issue that not all operating systems are created equal.  Operating systems such as MVS, VMS and MCP are nowhere as risky, if they are even risky to begin with, as Windows and Linux.  A lot of ASVs would argue that they never come across these operating systems running Web services.  However, all of them have the capability of running Web services and I personally know of a number of organizations that run their Web services from such environments.  Organizations are running these older versions of operating systems mostly because of the financial considerations of migrating to something else.  However, I can guarantee that none of the dozens of vulnerability scanners that I have used in the last 10 years would accurately identify any of these operating systems, let alone tell you the version unless some service message header information was retrieved by these tools.  And even then, most tools do not parse the header to determine the OS so it would take human intervention to make that determination.

Regardless of the failure, most ASVs do have a review or appeal process that allows organizations to dispute findings and to submit compensating controls to address any failures.  So for organizations that cannot get rid of unsupported OSes, they can use a compensating control.  Like compensating controls for the PCI DSS, the organization is responsible for writing the compensating control and the ASV needs to assess the compensating control to ensure that it will address the issues identified by the vulnerability scan.

So, if you can fail an organization over an unsupported OS, why is it that you do not automatically fail on unsupported application software?  I went through the Program Guide and there are all sorts of other criteria for applications but nothing regarding the fact of what to do if they too are unsupported.  Applications such as IBM Websphere and Oracle Commerce can become unsupported just as easily as their OS brethren.  And in my experience, use of unsupported application software is even more prevalent than unsupported OSes under the idea that if it is not broken and does not have vulnerabilities, why upgrade?  When I asked our ASV group if they fail organizations on unsupported applications I got silence and then the response that they will fail an application if the vulnerability scanner provides a high, severe or critical vulnerability.  To tell you the truth, while vulnerability scanners regularly return text header information for a lot of applications, I would be hard pressed without doing a lot of research to find out if the version being reported was unsupported.  However, scanners could provide this feedback if they were programmed to provide it.

Then there are all of the conspiracy theories out there that say the PCI SSC and technology companies are working together to drive software sales by forcing organizations to upgrade and there would appear to be a lot of anecdotal evidence that would seem to support this argument.  In reality it is not that the software companies are working together with regulators such as the PCI SSC so much as software companies operate this way in order to focus development and support resources on fewer, more current versions.  As a result, it is just happenstance that regulations cause organizations to have to update their software.

The bottom line in all of this is that you have options to avoid a failing vulnerability scan because of an unsupported OS.  The best method, and the one I most recommend, is do not use unsupported operating systems in the first place.  However, as a former CIO, I do understand the real world and the issues IT departments face.  As a result, I recommend all of the following which may or may not require you to develop a compensating control.

• Implement not only a network firewall, but also a Web application firewall (WAF) and make sure that the rules are extremely restrictive for servers running unsupported operating systems.
• Configure your firewalls to block the broadcasting of any OS signature information.  Masking the OS signature will provide the benefit of not advertising to the world that the OS running whatever application is unsupported.  This is not a perfect solution as, 9 times out of 10, the application itself will likely advertise the fact that the underlying OS is unsupported.  It is very important to note that this is only a stop gap measure and you should still be actively in the process of migrating to a supported OS.
• Implement real-time monitoring of firewalls, servers and applications.  Define very specific alerting criteria to ensure that any suspicious activity is immediately reported and operations personnel immediately follow up on any alerts to determine whether they are a false positive.
• Implement a host-based intrusion detection/prevention solution on any servers that run the unsupported OS.  If using a HIPS solution, you may also want to consider using its preventative capabilities for certain critical incidents.
• Implement real-time log analysis for firewall, servers and applications.  Define very specific alerting criteria to ensure that any suspicious activity is immediately reported and operations personnel immediately follow up on any alerts to determine whether they are a false positive.
• Actively use your incident response procedures to address any incidents that are identified with any unsupported OS.