Service Provider PCI Compliance Process

As a recent question pointed out to me, while service providers now seem to understand they need to be PCI compliant, they do not seem to understand the process under which they assess their PCI compliance.  As a result, I thought I would use this space to clarify this process.

When Do I Have To Be PCI Compliant?

The PCI SSC defines a Service Provider in the PCI DSS Glossary as:

“Business entity that is not a payment brand, directly involved in the processing, storage, or transmission of cardholder data.  This also includes companies that provide services that control or could impact the security of cardholder data.  Examples include managed service providers that provide managed firewalls, IDS and other services as well as hosting providers and other entities.  Entities such as telecommunications companies that only provide communication links without access to the application layer of the communication link are excluded.”

On the straight forward side of this definition, if your organization processes, stores or transmits cardholder data on behalf of one or more other organizations, your organization is considered a service provider and needs to be PCI compliant.  That point seems to be clear and most organizations that are processing cardholder data for others understand this fact.

It is organization that do not directly process, store or transmit cardholder data but could impact the security of cardholder data that get tripped up.  The bottom line is, any third party that has access to an organization’s cardholder data environment (CDE) needs to be PCI compliant for those services offered that come into contact with the CDE.

Based on this definition, the following are examples of services would need to be PCI compliant.  This is not a complete list but something to assist you in understanding why your organization might need to be PCI compliant.

• Management of firewalls that are considered part of an organization’s CDE.
• Management of networks that carry cardholder data that is not encrypted.
• Management of encrypted networks that carry cardholder data and the MSP manages the encryption keys for encrypting the network.
• Management of servers that process, store or transmit cardholder data.
• Management of PCs that have access to cardholder data.
• Management of applications that process, store or transmit cardholder data.

The first question service providers typically ask is, “How am I supposed to know if my customer needs our services to be PCI compliant?”  My rather indignant response is, “What, you cannot ask them?”

If you are a service provider and you are providing any value added services, you need to be asking every customer about their regulatory or legal requirements when going through the sales and contract processes.  It is not just PCI these days, but in the US there is HIPAA, GLBA, SOX and there all sorts of similar requirements in various parts of the world that you should be finding out whether your prospects need to comply.  If you are not, then you are sitting on a potential legal time bomb should something happen and your value add services are deemed the culprit.

SAQ D or ROC?

Service providers are broken into two levels, Level 1 service providers and Level 2 service providers.  Level 1 service providers are those that process more than 300,000 Visa, MasterCard or Discover transactions annually and are required to perform a full PCI assessment using a qualified security assessor (QSA) that results in a Report On Compliance (ROC).  Level 2 service providers are those that process 300,000 annual transactions or less and they can conduct their own assessment using the SAQ D.  SAQ D is the only SAQ that service providers are allowed to use.

The first issue most MSPs face is determining what parts of the ROC or SAQ are relevant.  There are no easy answers in this area as it all comes down to those value added services you are providing and how you provide those services.  However, before you can determine what parts of the ROC or SAQ are relevant, you need to determine the services that are to be assessed.

Where MSPs can miss the boat is taking the lowest common denominator approach and only assessing those value added services that all customers need to be PCI compliant.  Typically this results in an AOC that discusses physical security (requirement 9) and security policies (requirement 12).  If you take the lowest common denominator approach, then do not complain when each of your customers’ QSA have to traipse through your facilities and disrupt operations over the value added services you did not cover in your AOC.

The best approach is to assess all value added services you provide so that you do not have to worry about whether or not a given service is PCI compliant.  The reason this is important is that customers are required to make sure that all service providers that need to be PCI compliant are PCI compliant.  It is not that customers cannot work with non-PCI compliant service providers; it is just a lot easier to work with PCI compliant service providers.  The reason is that non-PCI compliant service providers have to be annually assessed as part of the customer’s PCI assessment as well as the customer is supposed to be monitoring the service provider’s PCI compliant efforts.  As a result, you will have an easier time and be easier to deal with if all of your value added services are PCI compliant.

Now back to what parts of the PCI DSS are relevant.  There are too many permutations of services and PCI requirements to go into this for every possible situation, so I am just going to provide some examples.

If you are providing firewall management services, then you are going to have to comply with requirements 1, 2, 4 (if your organization is managing any transport encryption), 5 (the PCs/servers used to manage the firewalls), 6 (change control for the PCs, servers and firewalls), 7, 8, 9, 10 (log management and analysis for the PCs, servers and firewalls), 11 (PCs, servers and firewalls) and 12.  Some tests in these sections may be Not Applicable for your organization, but you will need to go through all of the tests in the sections.

If you are managing applications such as in a software as a service (SaaS) environment, you are going to have to comply with requirements 1 (if you manage any firewalls or routers that could be in-scope), 2 (servers, firewalls, routers, switches), 3 (if your application encrypts the cardholder data), 4 (if your network encrypts cardholder data), 5 (servers running the application as well as any support PCs and servers), 6 (change management as well as any software development), 7, 8, 9, 10, 11 and 12.  Some tests in these sections may be Not Applicable for your organization, but you will need to go through all of the tests in the sections.

As you can see, even a “minor” service such as managing firewalls can require a significant amount of effort to be PCI compliant.

Do I Need A QSA?

A Qualified Security Assessor (QSA) is only required if: (1) your organization is considered a Level 1 service provider, or (2) you wish to have your organization listed on Visa’s Global Registry of Service Providers or with the MasterCard Registration Program (MRP).  In both of these cases, you will need to hire a QSA and produce a Report On Compliance (ROC) and Attestation Of Compliance (AOC).  If you can answer ‘No’ to both of these, then you can do your own assessment using the SAQ D.

Scope Of Assessment

At the 2012 PCI Community Meetings, the PCI SSC clarified scoping for PCI assessments.  The PCI SSC stated that:

“To be considered entirely out of scope, a network or system must be isolated from, with no connectivity into, the CDE (such that even if the system is “owned”, it cannot impact the security of the CDE).
If connections are limited to specific ports or services, those systems are included in PCI DSS scope to verify applicable controls are in place.
Applicable PCI DSS controls may vary for in-scope systems depending on the function of the system and presence of other controls.”

As a result, any of your systems that have access into a customer’s CDE are in-scope for compliance as they could be used to access the CDE.  This does not mean that these systems need to meet all of the requirements of the PCI DSS.  But it does mean that they need to be evaluated to determine those portions of the PCI DSS that do apply.  All of this is dependent on the potential to access cardholder data inside the CDE.

To minimize the impact of this clarification, we are seeing a lot of MSPs using “jump boxes,” PCs that do nothing but provide access into the systems to be monitored or managed.  This provides an air gap between the MSPs internal systems and the customer’s environment, including the CDE.  As a result, the jump boxes need to be PCI compliant and the internal systems at the MSP just need to have current anti-virus and anti-malware and be properly authenticated.

Hopefully this post helps all of you managed service providers that are facing PCI compliance.

This post was revised after a review by Walt Conway at 403Labs pointed out some inaccuracies.  A big thank you to Walt for his diligent eye.

The Barnes & Noble Breach Take Aways

On October 24, 2012 it was announced that Barnes & Noble had a credit card breach that was the result of tampered credit card terminals.  As a result of the breach, Barnes & Noble pulled all of the credit card terminals out of their stores so that they can be examined.  The story published in the New York Times has some points that should be interesting to other large merchants.

“We have acted at the direction of the U.S. government and they have specifically told us not to disclose it, and there we have complied.”

This is probably the most important take away you should have because a lot of incident response plans miss this point.  While the credit card companies want to the notified immediately of a breach, law enforcement should be the first outside entity notified and then the card companies, if approved by law enforcement.  The reason is that law enforcement may want the breach to continue in an effort to more easily identify and apprehend the perpetrators and that may include allowing the perpetrators to use the stolen cards for purchases.

But the next question that typically comes up is who in law enforcement should be notified?  If you are not a large or regional entity, then you should notify your local police department or county sheriff.  If you are a regional or large sized merchant in the United States, you should contact the United States Secret Service and/or the Federal Bureau of Investigation.  In either case, whatever law enforcement entity you contact should be consulted with before notifying anyone else outside the organization and that includes notifying the card brands.

“The company determined that only one keypad in each of the 63 stores had been hacked.  Nevertheless, the company has not reinstalled the devices.”

The 63 stores involved were all across the country from San Diego, Miami, Chicago, New York and other locations in between.  This implies either a very organized criminal group that operates in a lot of locations or to a localized group that was able to infiltrate the operation that configures and ships out the terminals for Barnes & Noble.  Based on investigations similar to this, it is most likely that a criminal operation infiltrated a centralized location that is responsible for the configuration, repair and replacement of credit card terminals for Barnes & Noble.

So what can a merchant do to minimize this sort of attack?  Here are some actions to consider.

• Contract with only a reliable terminal supplier.  In this age of lowest cost providers, there is a big temptation to use anyone as a supplier, particularly if their costs are the lowest.  However, the old adage of “you get what you pay for” is very relevant in these situations.  As part of your vendor selection process, you should be asking a supplier of terminals what they do to ensure that terminals do not get tampered with.  If you cannot get an answer or the answer you get is “trust us,” then you should probably not consider them as a vendor.  At a minimum, vendors should put their employees through periodic background checks (at least every three to five years), track which employees work on what units, do random physical internal inspections of units and random testing of units to ensure that they are not tampered with before they are sent out.  If you are doing this activity in-house, you should also be following this process.
• Lock down your terminals.  Anyone that has been into a Barnes & Noble might recall that terminals just sat on the counter.  As a result, they were easy to quickly swap out with a doctored unit.  I have been involved in a number of situations where merchants had terminals doctored because they were easy to swap out.  If terminals are locked in a cradle and only the manager on duty has the key, anyone trying to swap terminals is going to have to have a key to free the device.  This prevents swaps that occur after hours when only the cleaning people are present.  In addition, the keys to these terminal cradles needs to be different for each location so that one key does not open every cradle at every location.  The common key is a lesson the gas station industry has only recently addressed.
• Use tamper-proof serialized security tape or stickers over the seams of the terminal and check them daily.  This is a trick that has been used for quite a while with gas pumps.  The key is to at least daily (I recommend at each manager shift change); have the stickers checked to make sure that they are still in place and log that activity.  If they have been tampered with or are missing, the lane should be immediately taken out of service and your loss prevention unit contacted.
• Confirm a terminal swap.  A lot of merchants are very lax in their terminal swap procedures.  If a terminal turns up with instructions to swap it with another or a technician appears at the location with a new terminal, the store personnel do it, no questions asked.  That is wrong.  At a minimum, a good terminal swap procedure should involve the generation of a trouble ticket in a help desk system or similar and having the store manager confirm the swap with the help desk or POS support.  No ticket, no swap, no exceptions.
• Put video monitoring on all your POS locations.  This does not stop such a swap from occurring, but it does at least record such an event if it does occur.  This is particularly important in situations where the customer also acts as cashier as with any self checkout situation.
• Use MAC address filtering on your store location networks.  If a device is unplugged and a new device is plugged in with a different MAC address it will not work.  Yes, I know for some of you this creates a bad situation.  But I always ask people in response, “Why should store personnel be swapping equipment in the first place?”
• Monitor your sensitive devices.  If a credit card terminal or POS gets unplugged from your network, you should generate an alert.  That alert should then be correlated to a help desk ticket.  If there is no ticket, then someone should immediately notify loss prevention and also follow up with store management to find out why the device was unplugged.
• Monitor your network.  Terminals or POS should only be communicating with your service provider for transaction authorization and your routers(s) and/or firewall(s) should be configured accordingly.  If a terminal or POS attempts to communicate with any other external IP address, that should generate an alert to corporate IT and security that should then be investigated immediately.  This will catch those devices that are tampered with and then transfer data to a server outside of your network.  It is highly likely that the communication will be encrypted, but the traffic will be directed to an external IP address that should be blocked if your firewall(s) or router(s) are configured properly.

UPDATE – October 24, 2012: I am quoted on the Financial Times Web site regarding the B&N breach. http://www.ft.com/intl/cms/s/0/919e3292-1dd7-11e2-8e1d-00144feabdc0.html#axzz2BjZ8C9WJ

UPDATE – November 1, 2012: I got to be a guest blogger on CNBC because of this blog entry. http://www.cnbc.com/id/49624444/How_Retailers_Can_Avoid_a_Credit_Card_Breach

UPDATE – November 5, 2012: I am interviewed by a USA Today reporter. http://www.usatoday.com/story/tech/2012/11/05/debit-card-account-theft-qa/1677537/

UPDATE – November 6, 2012: I am quoted on the front page of USA Today. http://www.usatoday.com/story/tech/personal/2012/11/05/debit-card–numbers-pins-stolen-at-pos-terminals/1675795/

UPDATE – November 8, 2012: I am interviewed on WCCO 830 Radio in Minneapolis.

The Amazon Cloud And PCI Compliance

If there ever was a hot topic these days it would be “The Cloud” and, in particular, the Amazon cloud.  And that discussion inevitably leads to how are the Amazon cloud offerings are PCI compliant?  A lot of this discussion has to do with the very limited amount of information regarding the Amazon service offerings.  For some very bizarre reason, Amazon puts organizations interested in their PCI compliant services in a Catch-22 situation.  Unless you sign up for one or more of the services, you cannot obtain the information on how the Amazon service offerings are PCI compliant.  As a result, there is a lot of mis-information running around regarding the Amazon cloud.  So to debunk all of the myths running around, I thought I would explain what the Amazon cloud is and is not and how it ends up PCI compliant and what you need to understand when deciding to use the Amazon cloud.

And before I get calls from someone at AWS about the fact that I am somehow singling them out or I am being unfair.  I do not have a problem with AWs or anyone organizations’ cloud service offerings.  What I have an issue with is how some service providers use obfuscation and confusion about their services in ways that make customers unsure of whether they are getting something that is PCI compliant.  As I see it, the AWS service offerings seem to be PCI compliant, but there are things that possibly should be further explained so that everyone understands how that compliance is achieved.

The first part of the mythology revolves around what PCI compliant services Amazon Web Services, LLC (AWS) is actually providing.  According to AWS’s Attestation Of Compliance (AOC), AWS is a Hosting Provider for Web and Hardware.  The AOC calls out that the following services have been assessed PCI compliant.

• Amazon Elastic Compute Cloud (EC2);
• Amazon Virtual Private Cloud (VPC);
• Amazon Simple Storage Services (S3);
• Amazon Elastic Block Store (EBS);
• Amazon Relational Database Service (RDS);
• Amazon Elastic Load Balancing (ELB); and
• Amazon Identity and Access Management (IAM).

The AOC lists nothing for software provided through any of their services.  As a result, a big myth that gets busted right off the bat is that AWS is providing software.  At the end of the day, all AWS’s services are offering is Infrastructure as a Service (IaaS).  As a result, how AWS is PCI compliant is fairly easy to figure out.  They have totally minimized their responsibility on the PCI compliance front.

In addition to the AOC, AWS provides customers with a document entitled “AWS PCI DSS Controls Responsibility Summary” (CRS).  This document explains the various services and the responsibilities a customer organization has when using these services.

The first piece of infrastructure used by AWS is virtualization in the form of Xen as their hypervisor.  Because of the way AWS has implemented Xen, every virtual instances created by EC2 acts like an individual physical server in that there are no connections to any other server unless the organization defines such connections.  This is referred to in the CRS as instance isolation.  Finally comes the firewall.  EC2 includes a firewall that is managed by the customer.  Access to the firewall is controlled by an X.509 certification and access credentials provided through IAM.  In addition to utilities to manage the cloud environment, AWS provides various application programming interfaces (API) to manage the AWS cloud environment.

The bottom line is that, at a minimum, an organization needs to subscribe to EC2, VPC and S3 in order to build a basic platform capable of computing (i.e., server, connectivity and storage).  The need for other services outside of these will depend on what the organization is attempting to accomplish, whether or not they need the flexibility and scalability provided by AWS and other business factors.

From a PCI compliance perspective, the CRS categorizes the 12 PCI requirements into those that are AWS’s responsibility, shared responsibility between AWS and their customer and those requirements that are solely the customer’s responsibility.

In the AWS is responsible category falls requirement 9 or physical security and environment controls.  Since AWS is providing the facilities to operating the underlying physical hardware, it is solely responsible for this requirement.

In the shared responsibility category falls requirements 1, 10 and 11.  For requirement 1, AWS acknowledges that this is a shared compliance responsibility between AWS and their customer.  However, AWS’s responsibility is only to provide a firewall and ensure that it segregates their customers from one another.  The remainder of the responsibility for complying with requirement 1 is left to the customer.

For requirement 10, AWS indicates that they are responsible for:

• Maintaining log files for EC2 and S3 customer management operations (e.g. creation, modifications and deletion of these environments) for at least a year.
• Maintaining logs for the underlying software that provides the various services for at least a year.

This log information is monitored at least daily and is available to customers for their particular environment should it be necessary.  All other parts of requirement 10 are the responsibility of the customer.

For requirement 11, AWS indicates that they are responsible for ensuring the security of their environment including ensuring wireless security.  Customers are responsible for ensuring the security of the environments they construct using AWS’s services.

All of the remaining requirements, 2, 3, 4, 5, 6, 7, 8 and 12 are solely the responsibility of the customer.

So after all of this rigmarole, what is the advantage to be gained?  Not much near as I can tell.  The bulk of responsibility for PCI compliance still falls on the organization using the AWS services.  So organizations looking to offload as much of their PCI compliance responsibilities as they can to AWS are looking in the wrong place.

But it does not end there.  We are seeing more and more startup service providers that are using AWS services to avoid the capital costs of hardware and software of a 24/7/365 operation.  Where this becomes tricky is when you have a service provider providing PCI compliant services effectively using AWS for their “data center.”  In some cases, these service providers are trading on the fact that because AWS is PCI compliant, then their services must also be compliant.  However, what these service providers forget is that once they start going beyond the IaaS model and offer services in the Platform as a Service (PaaS) and Software as a Service (SaaS) realm, they are now responsible for portions of PCI compliant that Amazon is not.  As a result, organizations need to conduct due diligence on vendors using other cloud providers to provide their services to ensure that everyone is PCI compliant.

So do I think your organization should rush right out and sign up for AWS?  Maybe if you have the right business case.  But I do have some concerns regarding AWS’s service offerings and the statements surrounding how they are PCI compliant.

My first concern is in regards to requirement 1.2.3.  This requirement is one of the few that is not allowed to be marked ‘Not Applicable’.  As such, the QSA is required to document what procedures they conducted to ensure that any existing wireless is either not in-scope or that there is wireless in-scope and how it is secured.  To document this, AWS’s QSA has written:

“[AWS] maintain this control for all internal and external services that it provides. In EC2 and VPC environments, this includes the network at the hardware and management level networks, which are not exposed to customers.”

This statement says nothing of what procedures were conducted to ensure that wireless was not visible to customers as well as the controls AWS maintains to ensure wireless stays out of scope.  Essentially, we are asked to trust AWS that wireless is not on any customer networks.  Now, to be fair, AWS is operating secured data centers comprised with racks of hardware all virtualized, so the likelihood that wireless would exist in such an environment on any one customer’s network is remote at best. .  However, the PCI assessment process is all about verifying such statements, not just accepting them at face value as fact.  As a result, I am concerned that what is supplied as evidence for complying with this test leaves much to be desired.  What should be documented here are the procedures the QSA used to confirm that the controls AWS has in place are adequate to ensure that rogue wireless does not end up in their data centers.

Related to requirement 1.2.3 is requirement 11.1.  As with 1.2.3, 11.1 is also not allowed to be marked as ‘Not Applicable’ regardless of whether wireless is implemented or not.  For all of the tests under 11.1, the following statement is made.

“[AWS] maintain[s] this control internally.”

So what exactly does AWS do to ensure that their data centers remain wireless free or that wireless does not end up on the customer side of the network?  No idea.  I would like to assume that AWS is doing the right things in this regard, but, again, the PCI assessment process does not allow for assumptions, they require proof and this just does not pass muster.  At a minimum, there should be a discussion of the procedures used by AWS to ensure wireless is not an issue.

While we are discussing requirement 11, we should cover vulnerability scanning, penetration testing, intrusion detection and critical file monitoring.  All of which are the customer’s responsibility, not AWS’s.  Again, AWS is providing IaaS and nothing else, so any such controls will need to be provided by the customer.

When reviewing the detailed responses in requirement 9, it was interesting to see that AWS is responsible for ensuring that for the portion of any customer’s cardholder data environment (CDE) that exists in AWS, AWS ensures that destruction of hardcopy materials are properly destroyed so to be unrecoverable.  This begs the question, “Why would AWS have any hardcopy to destroy in the first place if they do not have access to customers’ environments?”  No further explanation is given, but one would guess it was their lawyer’s idea just in case AWS might somehow come into contact with CHD on hardcopy.

The next area I have issue with is not related to the service, but related to how an organization contracts for the service.  In an effort to fully automate things, unless you are a Fortune 50 looking to put your entire computing environment in AWS’s data centers, you can forget about negotiating a contract.  When you sign up for any AWS service, you either accept their contractual terms and conditions by checking the ‘Accept’ box and clicking Okay, or you don’t get to use AWS.  I know of a number of organizations that had real issues with that approach and, as a result, backed away from a more aggressive use of the AWS environment or decided they just could not accept the terms and did not go to the cloud at all.  While the AWS contract does cover PCI compliance, but it essentially makes the customer the one legally responsible for compliance with AWS providing support when necessary.

So that is AWS in a nutshell.  Not a bad thing, but something an organization needs to go into with their eyes wide open and understanding that they still have significant responsibilities even though they are now in “The Cloud.”

Third Party Service Providers And PCI Compliance

There seems to be a lot of confusion regarding third parties that provide networking or hosting services and their obligations regarding PCI compliance.  This confusion is not uncommon as merchants and their service providers have not necessarily been provided enough guidance to understand their obligations.  I hope this post will clarify those obligations for all involved.

If you learn nothing else from this post, if a third party is providing your organization a service that has access to your cardholder data environment (CDE) or the third party could come into contact with your cardholder data (CHD), then that third party must ensure that the service complies with all relevant PCI requirements.  As a result, the third party needs to either allow you or your QSA to assess the services that they are providing or provide you with an Attestation Of Compliance (AOC) that documents that those services have been assessed and they are PCI compliant.

In the past, I have stated that third parties could also submit a letter signed by an officer of the third party stating that all of the services provided to their customer are PCI compliant.  Now that v2.0 of the PCI DSS has a separate AOC and the PCI SAQs have the AOC built into the SAQ, there should be no reason to need such a letter or to ask for one.  If a letter is what your third party is offering, it is better than nothing, but you should be pushing them hard for an AOC.  If they are reluctant to get you an AOC, as part of your vendor management process, you should take that into account and probably begin looking for a new vendor that will provide an AOC for their services.

The most common issue we run into with third parties is that their AOC or other representations of PCI compliance do not cover all of the services provided to the customer.  In case after case, we see the AOC covering requirements 9 and 12 and nothing else even though the services provided may require compliance with some or all of PCI requirements 1, 2, 3, 4, 5, 6, 7, 8, 10 and 11.

In a lot of cases, it is not that the third party does not want to comply with PCI; it is they are taking the lowest common denominator approach and only picked those services where all customers requiring PCI compliance are asking for an AOC.  That way they have reduced their costs of a QSA to assess their environment.  These third parties are accepting the fact that any customer that needs more services assessed will have to do it themselves.

Related to this issue is the third party that offers their SSAE 16 Service Organization Control (SOC) 1 report has proof of PCI compliance.  While a SOC 1 report can cover a few PCI requirements, people must remember that the SOC 1 report is structured specifically for financial auditors to ensure that the controls at a third party are properly constructed to support financial reporting at the customers.  As a result, a SOC 1 report is not going to be a substitute for an AOC that covers all services.  There is an alternative to this and that is to have the third party go through a SSAE SOC 2 report that focuses on the security controls of the PCI in-scope services provided.  We are hearing from third parties inquiring into the SOC 2 report, but cost and a lack of customers requesting such a report are driving why we do not see more SOC 2 reports available.

Another common issue we encounter is the refusal of the third party to cooperate in assessing the services provided to ensure they are PCI compliant.  There are still third parties that argue their services are not in-scope for PCI compliance even when it is painfully obvious that the third party’s personnel have access to their customer’s CDE and/or CHD.

The most common third party relationship we encounter is the management of routers or other layer 3 devices.  Where we encounter the most confusion in this relationship is in regards to the use of encryption to keep the network services organization out of scope for PCI compliance.  The key here is if the network services organization manages the encryption of the network, then they are in-scope for PCI compliance.  The reason is that the employees of the network services organization have access to the encryption keys and therefore could decrypt the communications and gain access to CHD transmitted over the network.  As a result, at a minimum, the network services organization is responsible for complying with some or all of requirements 1, 2, 4, 6, 7, 8, 9, 10 and 12.  If you receive such services and are not getting an AOC that covers these requirements, then you should be doing more work on your own as well as asking the third party why they are not covering more of the necessary PCI requirements.

The next most common service we encounter is the network services firm that is managing or monitoring an organization’s firewalls, remote access or intrusion detection/prevention.  Such services always put the third party in-scope for PCI compliance.  Some or all of requirements 1, 2, 6, 7, 8, 9 and 12 will need to be assessed for compliance with the PCI DSS.  The log capture and analysis requirements in requirement 10 may also be complied with if your organization is not capturing and analyzing the log data from these devices.

Another group of third parties we encounter a lot are records retention vendors.  Organizations like Iron Mountain have conducted their own PCI compliance project and readily hand out their AOC to customers.  However, where we see issues is with such vendors that provide their own tape library for their customers to use for backup.  We have encountered a number of third party’s doing the encryption at their library which puts them in-scope for PCI compliance, at a minimum, for requirements 3, 4, 6, 7, 8, 9, 10, 11 and 12.

We encounter outsourcing the data center a lot with large organizations, but small and mid-sized organizations are also hopping on the data center outsourcing bandwagon.  Where this puts the third party in-scope for PCI compliance is when the third party is responsible for maintaining the environment such as applying patches, managing servers or any other activities that would allow the third party’s personnel to potentially have access to CHD.  In such situations, at a minimum, the third party is responsible for complying with some or all of requirements 2, 5, 6, 7, 8, 9, 10 and 12.  Compliance with some or all of requirement 1 may be applicable if the third party is managing your firewalls or routers.  Compliance with some or all of requirements 3 and 4 may also be applicable if the third party is responsible for managing encryption keys for encrypting CHD or encrypting communications.

Where the most confusion regarding third party responsibilities occurs is in regards to “The Cloud.”  The most common reason for this is that every vendor seems to have a different definition for what “The Cloud” is, based on their particular services.  Using the definitions provided by the National Institute of Standards and Technology (NIST) in their publication SP800-145, ‘The NIST Definition Of Cloud Computing’, I can provide the following guidance.

If your organization is purchasing Infrastructure as a Service (IaaS), then the third party providing these services will typically be out of scope for PCI compliance except for requirements 9 and 12.  There are some instances where IaaS implementations may require compliance with the PCI DSS if the third party is managing network infrastructure that comes into contact with CHD as is usually the case with private cloud environments.

For Platform as a Service (PaaS) and Software as a Service (SaaS), the third party will have to provide PCI compliance for the services they are providing to your organization.  That is because with either of these service offerings, the third party must have access to the CDE and will have the potential of coming into contact with CHD.

The problem with the majority of PaaS and SaaS vendors is that they only deal with your organization through a Web-based interface, i.e., everything is automated – contracts, support, etc.  As a result, the contract is a “take it or leave it” situation that does not usually cover everything needed for PCI compliance, there is no way to independently verify the representations made by the third party as well as the fact that the AOC provided by the third party typically only covers only the physical security requirements in requirement 9 and possibly some of requirements 11 and 12 and nothing related to the other requirements, even though the third party may have responsibilities for PCI compliance outside of what is represented in their AOC.

If this is the case, there is little you or any QSA can do to properly assess the environment to ensure it is truly PCI compliant.  As a result, we have a lot of organizations that try to develop compensating controls for these cloud implementations.  These organizations very quickly and frustratingly find out that there are very few, if any, controls on their side of the equation that can get them to “above and beyond” the original requirement.

I know there are a lot of other examples of services being provided to merchants.  But, hopefully these examples can assist you in clarifying what you need or do not need from your third parties when it comes to PCI compliance.

Supermarket Chain Notifies Customers Of Self Checkout Skimming

Customers of Lucky Supermarkets, a subsidiary of Save Mart Supermarkets, got an early Thanksgiving gift when they were notified that 20 Lucky Supermarkets and one Save Mart store in California had discovered skimming devices inside those stores’ self checkout systems.  As retailers move to more and more automation surrounding checkout, the more risk they take on unless they put in place controls to minimize those risks.

Grocery stores can learn a lot from gas stations and convenient stores and the controls these organizations have put in place to secure gas pumps.

• Locks are keyed the same.  It turned out that gas pump manufacturers for as far back as they have had locks on pumps use the same keys.  I worked at a gas station when I was in high school a long, long time ago.  Yet, six years ago conducting a security review for a State Transportation Department, I found that the pump keys that I had neglected to return 30 years earlier would open the gas pumps at the maintenance garages.  Whoa!  Turns out self checkout manufacturers have the same problem unless you request them to use a different key and lock combination.  So the first lesson to learn is to explicitly request a unique to your organization lock and key set for your self checkout units.
• Locks are not perfect.  Even if you change out the locks, they can still be picked fairly quickly by someone who knows what they are doing.  So, for a second level of defense, you need to add serialized tamperproof tape to the doors that allow access to the inside of the self checkout unit.  Newer self checkout units only allow ease of access to change out the register tape and nothing else.  Only authorized technicians have the ability to gain access to the rest of the device.  However, best practice is to put serialized tape on all of the doors regardless.
• You may need tamperproof tape inside.  Older (age is all relative) self checkout units allow access to too much of the internal workings and/or do not have tamperproof parts inside the cabinet.  That means that “bad guys” can insert a skimmer without any obvious changes.  To avoid that problem, you can wrap connections with tamperproof tape so that if anyone attempts to take the connectors apart, it will be obvious upon inspection of the tamperproof tape.  Discuss your situation with your self checkout vendor as they can advise you on what should be taped.
• Locally monitor your equipment.  The serial numbers on the tape need to be recorded on a log sheet and the tape (inside and outside) should be checked at every manager shift change.  Any tape that appears to have been tampered with should be investigated and that unit taken out of service until an authorized technician deems it safe to put back in service.  In addition, video monitoring should be in place to monitor each self checkout.  Staff should be present at all time to monitor these devices.  Typically a single person can cover two isles compromised of a total of four self checkout devices.  Any more than that and your staff can be easily distracted and miss someone tampering with a device.
• Remotely monitor your equipment.  Most self checkout devices have the ability to be centrally managed and monitored.  In the event a door is opened, the unit can send an alert to the central monitoring console.  When an alert is generated, operations personnel should immediately contact the retail outlet and have store management immediately investigate the alert and inform the central monitoring station of the devices status.  If the store is not in operation, then the central operations personnel should contact local law enforcement and store management that an emergency exists at the store.

As a friendly reminder, security is not perfect.  These controls have to be executed perfectly every day, every year.  That is where things always go awry.  However, if you do execute these controls consistently, your organization should be very difficult to compromise.  The “bad guys” will know that and will find an easier target.

What To Do About Insiders

The first posting I did on this subject was to provide an understanding that, despite the news stories, the insider threat is a very real threat and needs to be addressed.  However, what is an organization to do?  Employees and others need to have access to certain information in order to get their jobs done.  What steps should an organization take to minimize the insider threat?

First, I need to be very clear about this.  Even when you do all of what I recommend, you are only minimizing the insider threat.  The insider threat can never be totally mitigated.  Insiders must have access to information that the general public or even you business partners do not have access.  As a result, should an employee get sloppy with controls or go “rogue,” you can expect to lose whatever information that person had access.  Remember my mantra – security is not perfect.

I posted some ideas a while back on controls for automation.  Here are my minimum recommendations for manual controls to put into place to minimize the insider threat.

• Management needs to recognize the importance of management controls.  The “tone at the top” really does mean something when it comes to controls.  However, management needs to understand that these sorts of controls are no absolute guarantee of avoiding issues.  Properly implemented, monitored and adjusted as necessary, such a control environment will confirm to the rest of the organization that management believes that controls are important.  If management does not know what to do regarding management controls, then they should consult with a public accounting firm as they are very aware of control environments and can assist in the design of a control environment.

• Preventive controls.  Preventative controls, as their name implies, put in place something to prevent a problem.  A prime example of a manual preventive control is requiring a minimum of two signatures on checks.  The larger the amount on the check, the more people that have to sign off on the check.  Under such an approach multiple people have to collude to defraud the system.  This sort of approach can also be taken for report reviews of inventory, cash on hand and any other metrics that are important to the survival of the organization.  The idea is to ensure that at least two people are involved in these reviews and that they physically sign off on their review and document and start an investigation into any irregularities.
• Detective controls.  As the name implies, detective controls are controls used to detect problems.  Following the example in preventative controls, the other people signing off on a check or reviewing a critical metric report is a detective control.  If the reviewer feels that something is not right with what they are reviewing, they are obligated to notify their immediate supervisor of the issue and ask the submitter to physically document the situation.  Once documented, the reviewer can then either sign off and accept the explanation, or refuse and further investigate.
• Corrective controls.  Corrective controls are those controls used to ensure that the preventative and detective controls are focused on the right problems and are going to be able to be relied upon going forward.  Keeping to the theme, in the event of an irregularity being identified, management should then institute a root cause analysis and determine what caused the situation and make the necessary changes to the preventative and detective controls to ensure that people do not try to circumvent the control environment.
• Hold employees responsible for the control environment.  Management may be responsible for establishing controls, but it is the employees that make the control environment actually work.  Employees should have their key controls evaluated at least annually to reinforce the importance of controls.  In our check example, the people signing off on checks should be evaluated on how many checks with problems are issued by the organization that they were required to sign.
• Solicit control improvement ideas from employees.  The problem most organizations have with management controls is keeping them relevant.  A common example we see is a problem that occurred ten years ago has been addressed by automated controls in a new computer system, yet management continues to require the manual control to be followed.  Most of the time, employees know exactly what needs to be done, but management does not want to recognize that fact.
• Have a third party periodically assess your controls.  In addition to employees providing ideas, organizations should periodically invite a third party, such as their accounting firm, to assess the control environment and recommend changes.  A number of years ago I worked with a large organization where we discovered that the way one of their computer systems had recently been modified, checks could be generated and bypass approvals and oversight.

For those of you that are going to recommend these minimum controls, my heart goes out to you.  The road ahead is likely to be very bumpy and contentious if your organization has a mediocre control environment.

Something to share with management as you push this sort of project is that there are very measureable benefits to implementing controls.  Every organization that I have worked with over the years has found that a byproduct of their controls projects has been fewer customer complaints and fewer employee screw ups.  Avoiding problems or making them smaller and less impactful on customers can add up to serious savings in time and money.

If you have a mature control environment, take a look at how you can make them better, more effective and more relevant.  If you do not have a mature control environment, then take baby steps.  Look to your accounting area as they will likely have the most robust control environment.  Grab one of those accountants and use them to help you look at other areas that may have problems that controls can address.

Best of luck to all of you on your journey.

2011 Verizon Breach Report

At this year’s Community Meeting, Christopher Novak of Verizon Business Services made a presentation regarding the 2011 Data Breach Investigations Report.  A lot of people blow off these sorts of presentations as being vendor focused or just a bunch of meaningless statistics.  However, Verizon provided a very good presentation on the nature of today’s threats and what organizations should be doing to protect their networks and systems.  Verizon’s Data Breach reports are a wealth of information; however I wanted to highlight just a couple of things that all organizations should take away from the report.

The first take away from the Verizon statistics is that while the media focus on the threat of insiders, it is the threat presented by external parties that is accounting for 92% of breaches, up 22% over 2010.  Not that insiders are not a threat, but only 17% of breaches involved insiders, down 31% over 2010.  Threats from business partners were less than 1% and only 9% involved multiple parties.  Yes, all of that adds to more than 100%, but as Mr. Novak reminded us, there are numerous instances while external actors ultimately breached the organization they were assisted by one or more insiders or a business partner.

The next important set of statistics is an analysis of breaches against the PCI DSS and its requirements for organizations that were in-scope for PCI compliance.  The first statistic that every organization should be concerned about is that Verizon found that 89% of all PCI in-scope organizations breached had been assessed by a QSA or themselves as not compliant with the PCI DSS.  This statistic should be a wake-up call to all organizations that complying with the PCI DSS is important.

In reviewing this demographic mix and the associated lack of compliance, we believe that the data reinforces an assertion we’ve been making for the past three years: to reduce risk, organizations of all sizes need to implement the basic tenets of an information risk management program and maintain this initial investment over time.  This includes network and data defense technology basics (firewalls, anti-virus, identity and access management), as well as the non-technical aspects of security and risk management (policy and process development).
Verizon Business Services 2011 Data Breach Investigations Report – page 62

PCI DSS Compliance and Breach Correlation

But the really important statistics were the analysis of compliance against each of the 12 PCI DSS requirements when a breach occurred.  What this analysis shows is that there are a lot of areas for improvement.

For building and maintaining a secure network, less than 50% of breached organizations were compliant with requirements 1 and 2 of the PCI DSS.  In protecting cardholder data, while breached organizations (requirements 3 and 4) were better at protecting cardholder data when it was transmitted (63%), they were horrible at protecting cardholder data when it is stored (43%).

It appears that organizations were complying with requirement 5 at 70% compliance.  However, given other survey results that claim over 95% of organizations have anti-virus and anti-malware, this is a troubling statistic as it points to incompetence in the installation and operation of these tools.

As I would expect, less than 50% of organizations were in compliance with requirement 6.  Whether that is related to patching or poor development practices, this part of the PCI DSS is always a bone of contention.

While overall, the requirements to implement strong access control measures, requirement 8 lags physical security and having access control systems implemented.  What this says is that while organizations have the tools in place, they are doing a poor job of managing them.

Regularly monitoring and testing networks is probably the area that needs the most improvement but explains another statistic in the report.  Compliance with requirement 10 and 11 were only 39% and 38% respectively when a breach occurred.  Earlier in the report, Verizon has a bubble chart that shows how long it took to breach an organization, how long it took the breached organization to acknowledge they were breached and then the time it took to address the breach.  It took attackers hours to days to breach an organization.  Unfortunately, it took weeks or months for an organization to know they were breached.  And to add insult to injury, it took additional weeks to contain the breach.

And finally, compliance with requirement 12 was only 44%.  A lot of organizations blow off documentation.  But it is the foundation of an organization’s security posture because it explains and describes why the other 11 requirements are important.  Without that foundation, is it any wonder that the other requirements are not in compliance.

One of the lingering questions from our discussions around PCI in this report is always that of relevancy. It’s all well and good to validate compliance with the PCI DSS, but does it actually help reduce risk? Insofar as that translates to a sincere security program—one that seeks to maintain validation on an ongoing basis—the data strongly suggests the answer is “yes.”
Verizon Business Services 2011 Data Breach Investigations Report – page 64

So what should an organization focus on to improve their security posture?  Here are my thoughts on a shortlist of issues that, if dealt with, would go a long way in securing organizations.

1. Really monitor your networks and systems.  No, I am not suggesting that everyone needs to go out and invest in a security incident and event manager (SIEM) or similar log management and analysis solution, but some organizations have no other choices because of their size and situation.  Regardless of how you choose to approach this problem, the key is that you must monitor your networks and systems.  And you need to monitor them regularly (daily) if not in real time.  The best organizations are doing real time monitoring and go through the PCI DSS tests and develop rules to monitor compliance with those requirements.  Attackers know that they can get away with their breaches because organizations are lax in their monitoring.  Just tightening monitoring up would go a long way in reducing the amount of data breached if not reducing the number of breaches.
2. Manage your access control systems.  No, it is not sexy to review user lists on a quarterly basis and disable, remove or reclassify users.  But here is a prime example of where organizations have a tool and are not using it correctly to improve their security.  Based on the report, it turns out that a lot of breaches were the result of poor user management practices and lousy passwords.  If this area got addressed, it would also go a long way in cleaning up the problem.
3. Engineer your networks and systems to be secure.  It fascinates me how many security professionals do things in the name of expediency that open huge holes in an organization’s security posture.  Another fascinating thing is how many security professionals have little knowledge of operating systems and applications.  Most security professionals seem to come out of the network administration teams as they were maintaining firewalls and routers when they went into security.  As a result, while they are very good at securing the network, server and application security lag because they just do not understand it nor do some find it all that important.
4. Test your security.  The only reason I can figure that vulnerability scanning and penetration testing is not being performed is that it will be used against security personnel to show that they are not doing their job.  While there are those security personnel that talk the talk but do not walk the walk, the vast majority of security professionals do an admirable job with few resources and even fewer plaudits.  However, given the rate of vulnerability production by attackers and researchers, if an organization is not conducting quarterly or even monthly vulnerability testing, there is no way an organization is going to protect itself.  And while the value of penetration testing will probably always be argued, I can tell you that it does have its place and can be invaluable if properly conducted.  This is particularly true for all of those organizations that think that because their vulnerabilities are all less than CVSS 4 that they are secure.
5. It is time to get control of application patching.  For small organizations, patching is only an issue due to a lack of focus on it as an issue.  For large organizations, it is the sheer volume of systems and devices that need patching.  There are numerous solutions for large organizations to deal with this situation, so find the tool that best fits your organization and get it implemented.  Oh, and get off the 30 day issue, that was clarified years ago.  You only need to have a reliable process and demonstrate that your patching process works on your time frame and patches do not fall between the cracks.

If all organizations could just focus on these five items and execute them at least 90% or better all of the time, a lot of the current breach issues would go away.  But that is the rub is it not?  A lot of organizations have issues maintaining that level of execution compliance.  Breaches occur because organizations get sloppy and, even with defense in depth in their security controls, there are too many controls where execution consistency has dropped leaving gaping holes in the various levels of security.

And as a reminder, these items address today’s problems.  However, once addressed, attackers will find other ways in, so the improvement process needs to be continuous.  And while organizations address the latest security concerns, they must be vigilant to cover the prior concerns because old vulnerabilities and exploits have a way of coming back.

VoIP And PCI Compliance

I have had some interesting conversations with people lately regarding voice over IP (VoIP).  It fascinates me as to how little people really know and understand how this technology works.  But what really scares me is how this lack of information is putting organizations at risk.

The most obvious problem with VoIP is segmenting it away from the cardholder data environment (CDE).  I am really disturbed by the number of organizations that have no security around their VoIP.  Yes a lot of organizations have segmented the VoIP from the rest of their network, but there are no controls that stop anyone from getting into that network segment.  As a result, anyone with the right set of tools can gain access to the traffic in the VoIP network segment.

The next thing that scares me is the lack of security surrounding the VoIP servers including any call recording servers.  People treat these VoIP servers just like their traditional PBX.  Unlike their PBX that likely ran a proprietary version of UNIX, VoIP servers are just Windows or Linux servers running a VoIP application.  As a result, they are susceptible to all of the viruses, malware and everything else any other server is susceptible.  However, these servers typically do not run anti-virus (performance issues) or are they hardened to any rational security standard.  When they get infected or hacked, it seems to be a shock to the system administrator.

And what about the call recording technology?  We keep hearing from vendors of call recording solutions that they use proprietary recording methods requiring special CODECs.  While in some instances the proprietary claims are true, what we are finding more and more is that vendors are just manipulating file header information such that Windows Media Player, iTunes and the like do not recognize the file as being in a valid format.  However, tools such as VLC Media Player are able to see past the header changes and recognize these files for what they are, WAV, MP3 and the like.  Thus some proprietary formatting claims are all a bunch of smoke and cannot necessarily be relied upon for security or privacy.  Another tell on the proprietary nature of call recordings is that, when you “convert” a recording, if the conversion software seems to be copying the recording more than actually converting it, it is likely that the header is being fixed to WAV, MP3, etc.  Real audio conversions typically take more time than just copying because the file has to be fully processed.

But the final insult in this whole scenario is the lack of understanding security personnel have regarding the VoIP protocols.  While VoIP call setup and teardown are usually conducted over TCP/IP (a stateful protocol), the actual call itself is conducted via streaming protocols over UDP/IP (a non-stateful protocol).  As a result, when you start talking to security people about VoIP security, their knee-jerk response is to tell you that VoIP is secured by the corporate firewall.  However, given that the VoIP protocols are stateless, even behind a firewall really does not provide any protection.

So you have a VoIP solution in place.  What should you be doing to ensure its security if it is in-scope for PCI compliance?  Here are my thoughts.

• Properly segment your VoIP from the rest of your network.  This means either physically or logically separating the VoIP from the rest of your network.  This also means implementing access control rules so that only those devices and people that need access to the VoIP network have access.  If you are also using your VoIP phones as the network jack for a PC, make sure to VLAN that jack to something other than the VoIP VLAN.
• If you can, implement the VoIP segment without DNS and DHCP and use MAC filtering to avoid the accidental or deliberate plugging in of a PC into a network jack that is VoIP only.  At a minimum, use MAC filtering to at least control what gets plugged into the LAN jack.
• Closely monitor your VoIP segment and generate alerts on any devices that are unplugged or plugged in.  Also monitor for any protocols other than the VoIP protocols that your VoIP system uses.
• Do not use the last octet or any other portion of the phone’s IP address as the extension number.  Yes, I know this is an easy way for the help desk to identify and troubleshoot phones, but it is also easy for an attacker to locate targets of interest, so keep that in mind when you are implementing your VoIP solution.
• Never, ever connect your VoIP network to another VoIP network outside of your explicit control.  Given that VoIP primarily uses UDP/IP, you cannot expect any firewall to protect your VoIP system from anything outside of your control.  Always use plain old telephone system (POTS) circuits to connect to any foreign network.  I know that is not as sexy as VoIP, but how else can you protect your VoIP system from outside influences?
• Work with your VoIP vendor to harden all servers that manage the VoIP system.  These are just Windows, Linux, etc. systems.  Obviously you will need to do some testing of this and you may not be able to use all of the hardening items in your server hardening standard, but you would be surprised at what you can do that the vendor says will not work.  Remember, they are just trying to cover their butts should a problem crop up.
• Be careful implementing VoIP on traditional PBXs.  A lot of these solutions are just PCs or servers and can be easily hacked once on your network just like their full VoIP brethren.
• Get a hold of VLC Media Player or similar tool and see if you can play a recording off of your call recording system.  We are getting about a 25% hit rate using VLC to play recordings.  A lot of the success of this approach depends on the age of the call recording system.  The newer the systems, the more likely it is that you will find that the recordings are just tweaks of existing standards.

An Update On The MPLS Privacy Debate

The MPLS private network discussion continues.

A lot of network administrators and carriers argue that MPLS networks are private because the PCI SSC says they are private.  As more and more organizations migrate from ATM and Frame Relay, this topic keeps coming up again and again lately.  Because of the push back from carriers and network administrators, I went back and re-read FAQ number 8705.

“In general, MPLS networks are considered “private” networks and do not require encryption. This, however, is dependent upon the specific provider and/or configuration. If the IP addresses are public and the MPLS network provides exposure to the Internet either through the LSR or other device (if the edge router has an Internet port) then it should be reviewed carefully as it is likely considered “untrusted”. The QSA should review the implementation and determine whether the IP addresses are public such that the MPLS network provides exposure to the Internet, before concluding that the MPLS network is considered private. If the QSA cannot gain that assurance, then the whole network should be in scope. The PCI SSC is not compiling a list of approved MPLS solutions nor do they have any plans to do so. This requirement for encrypted transmissions is intended to apply to transmissions outside of an internal network to an external third party, going over an open, public network; this requirement does not apply to transmissions over an internal network protected by external facing firewalls, since that is not considered a public network.”

Apparently, carriers and network administrators only read the first sentence of the FAQ and conveniently forget the next three sentences.  But it is those three sentences that document the criteria to determine whether or not an MPLS network is private.  The criteria a QSA is to use to evaluate an MPLS network’s privacy are:

• How is the MPLS network configured?
• Does the LSR come into direct contact with the Internet?

While these appear to be fairly simple questions to be answered, these questions are anything but simple or even easily answered.

The first question, how is the MPLS network configured is a problem for a lot of QSAs and network administrators as well as carriers.  MPLS is just a specialized IP network, so how the network is engineered drives just how private is private.  The problem with relying on IP addressing as the only criteria of whether or not an MPLS network is private is not proof positive.  I would argue that, even if the IP addressing on the MPLS network is RFC 1918 compliant, if the subnet is not the same as the network connecting to the network, then a QSA should look into the network to confirm that it is private.  This is particularly true if the addressing on the MPLS network is an ARIN registered address block belonging to the carrier.  Yes, such a network would be private for the carrier, but could be anything but private for the carrier’s customers’ traffic.

The second question is also not as straight forward to answer.  Just because private addressing is used on the MPLS network does not mean that it does not come into contact with the Internet or Internet traffic.  Unless you have visibility through the entire network and the rules used for that network, it is anyone’s guess as to whether or not it comes into contact with the Internet.

Of course all of this implies that the carrier is willing to show you their MPLS network configuration and share other information about their MPLS network.  But getting such a candid talk about a carrier’s network is sometimes anything but easy.  I have personally encountered carriers that refused to explain anything about their network and also refused to allow anyone to look at their LSR configurations.  As a result, we had no way to confirm or deny that the network was private.  To add insult to injury, I have been told by carriers that I was wrong in requesting to look into the configuration of their network and that this was not what the PCI SSC intended.  That said, I have also jumped through hoops to work out a way to confirm as best I could that the MPLS network was private.

MPLS is just an IP-based wide area network and because it uses IP, it can have a number of vulnerabilities just like IP networks.  Carriers use human beings to manage these networks and they are fallible just like our own employees.  Therefore, it is highly likely that mistakes will sometimes occur that will affect the privacy of the network.  I am guessing that once we have a breach in the MPLS cloud, MPLS will no longer be automatically considered private and encryption will be required.

And it is not just MPLS networks.  Most ATM and Frame Relay networks are routed over MPLS backbones by the carriers.  So just because you do not use MPLS does not mean that you are immune to the risks of MPLS.

In the end, we will have to rely on the statements and representations of the carrier as to whether or not the network is private.  Is this a good way to secure your organization?  It is as long as your carrier never causes a problem.

PCI And Virtualization

I just received an invitation for a Webinar on Virtualization and PCI compliance.  My friend, John Kindervag is one of the panelists and, no, this is not an unpaid advertisement for anyone to attend even though I have provided the link to register.  For an hour they will be discussing this topic because now the PCI DSS v2.0 references virtualization.  Let us be very clear, while the PCI DSS prior to v2.0 never explicitly discussed virtualization, QSAs were instructed on how to approach virtualization security.  And as you will see, virtualization security is no different than any other operating system security.

In my very humble opinion, virtualization is a one minute security issue, if that long.  Let us cut to the chase, as small an attack vector virtualization can be, it is still a potential attack vector, so you need to secure it.  Is that clear enough?  The real issue is how to secure a virtualized environment.

There are two different forms of virtualization.  There are stand-alone hypervisors (what NIST refers to as “bare metal”) like VMware vSphere, VMware ESXi, Microsoft Hyper-V and Citrix XenServer.  Bare metal hypervisors are what we typically run into the most in our PCI compliance engagements, but not necessarily a guarantee.  There are also VMware Server, VMware Desktop and Microsoft VirtualPC (what NIST refers to as “hosted”) that require a host OS to run on as an application no different than Microsoft Word.  Obviously, the attack vectors are wildly different for each type of virtualization.

For whatever reason, it seems that a lot of IT professionals do not recognize that a hypervisor is an operating system.  Yes it is a very specialized operating system, but it is an operating system just like Linux or Windows.  Most hypervisors are based on Linux or UNIX and have a few security hardening similarities.  But given a hypervisor’s specialization, they have significantly different security hardening requirements from their Linux or UNIX counterparts.  As such, hypervisor vendors typically provide a security hardening standard for each of their hypervisor operating systems.  All you need to do is go to the hypervisor vendor’s Web site and download the security hardening guide for your version of hypervisor.  Which brings up a good point, if your hypervisor vendor does not provide a security hardening guide, then you need to find a different hypervisor.

For bare metal implementations, the only thing you have to secure is the hypervisor itself.  However, with hosted virtualization, you need to secure the host operating system as well as the hypervisor.  In addition to the hypervisor, you will need to follow the host operating system vendor’s security hardening guide to ensure that the host OS is also secure.

But hardening your virtualized operating system is not the end of the job.  You need to properly implement your virtualized environment securely as well and that is more than just hardening the hypervisor.  The most obvious security item that needs to be done is that any guest operating systems implemented need to also be securely hardened.  It still surprises me the number of IT professionals that somehow seem to think that because they are implementing Windows or Linux as a virtual machine that there is something different about security and you can totally skip or skimp on hardening.  Security hardening procedures need to be completely followed regardless of whether the guest OS is stand-alone or in a virtual machine.

The next area that seems to get the short shrift is infrastructure security.  This is particularly true of the management of the hypervisor environment.  Most implementations I have seen do a good job of securely connecting the virtual machines, but the hypervisor infrastructure environment leaves a lot to be desired from a security perspective.  The first mistake I see is that the hypervisor management environment is not segregated from other networks.  In the first scenario I commonly see, the production network and the hypervisor management network are on the same segment.  If an attacker compromises any virtual machine, they gain access to the hypervisor management environment and can therefore gain access to the virtual cardholder data environment.  In the other scenario, the corporate network and hypervisor network are one and the same and therefore everyone that is on the corporate network can also gain access to the hypervisor management network.  The way to fix both of these situations is to put the hypervisor management network on its own network segment.  I also recommend to organizations that they dedicate a NIC to only that segment.  However, if an organization already has an operations management network segment separate from other networks, I have no problem having the hypervisor management network in that segment as well.

The other scenario I frequently see is virtual machines from the cardholder data environment (CDE) intermingled with virtual machines that are not part of the CDE.  The problem here is that in the event of a compromise of a non-CDE virtual machine, CDE virtual machines may be accessible because of the configuration of the virtualization environment.  The best way to use virtualization for PCI compliance is to isolate your CDE virtual machines in a physically separate virtual environment from your non-CDE virtual machines.

For the truly paranoid, you can also fiddle with parameters such as physical/logical NIC assignments as well as SAN configurations.  While these sorts of configuration changes can provide additional security to the equation, I have my doubts as to the significance of these changes from a security perspective.  In my years of dealing with virtualization, these sorts of configuration changes have been more for performance reasons and enhanced security was just a nice byproduct.

Finally, there is the maintenance aspect of virtualization.  I think everyone gets the fact that virtualized or not, the guest operating systems need to be maintained and patched just like their stand-alone brethren.  However, when you ask organizations how often they patch their hypervisor; some will say to you very honestly, “You have to patch it?”  Earlier on I stated that a hypervisor is also an operating system and, as such, it needs to be patched just like any other operating system.  Granted a hypervisor does not usually get patched every month like Windows, but there are patches issued every so often by hypervisor vendors.

Best of luck to John and the round table that are presenting this month on virtualization and PCI compliance.  Hopefully this post will help explain what they will be discussing as well as lead to more insightful questions on the topic.