Why vSkimmer Should Not Matter

It was announced this week by McAfee that a new threat to merchants has been discovered called vSkimmer.  This is a very insidious threat as most merchants will likely not know they have been infected until it is too late.

The net of vSkimmer is that it is malware of the highest order built for the explicit purpose of collecting track 2 data from Windows point of sale (POS) systems.  Worse yet, whoever wrote this little gem of software intends to enhance it in 2013 to include the ability to skim EMV cards’ “track” data as well.

vSkimmer can be deployed like Stuxnet through a USB thumb drive, as malware in an email message or on a Web site or any number of ways.  When installed, vSkimmer determines the operating system and version, hostname, active username and various other operational characteristics of the POS system.  It then inventories running tasks and memory to determine where track 2 data is stored and begins recording that data.

vSkimmer works whether the POS system is connected to the Internet or not.  When the POS is connected to the Internet, it transmits the data obtained to a control server using HTTP.  When the POS is not connected to the Internet, the information is stored until someone connects a USB device labeled ‘KARTOXA007’ and copies all the information it obtained onto the USB device.

As usual, the Internet is abuzz regarding how this will be addressed by the PCI DSS.  Sorry to disappoint, but it is already addressed.  Here are some key requirements in the PCI DSS that should mitigate vSkimmer.

• Requirement 1.2.1.a requires that only that network traffic that is necessary is allowed through the firewall.  Merchants should be only allowing connectivity from the POS or card terminal to their processor and nowhere else.  Any traffic attempting to go anywhere else should be flagged and IT alerted to investigate.
• Requirement 5.1 requires that you have anti-virus and anti-malware software installed on POS devices.  Given this is a Windows specific threat and Windows is highly susceptible to being infected, you should have done this already.  While anti-virus solutions are not perfect in always identifying such malware, since McAfee and other anti-virus solution vendors are the ones that found vSkimmer, I would imagine that they all have or will very soon have signatures for vSkimmer.
• Requirement 6.1 requires that systems are patched current.  The problem with patching POS systems is that a lot of vendors issue POS updates for the OS and their application on a quarterly or even annual basis and do not recommend that merchants patch their POS systems directly from Microsoft because of compatibility issues.
• Requirement 10.2 which requires the logging of events.  In the case of a USB device being plugged into the POS system, at a minimum you should see that the portable device enumerator service going active when a USB device is plugged in and if the device is new, you should see system event log entries regarding the loading of device drivers to support the USB device.  None of these actions should be seen in your log data, so if you monitor for these events, you will know that USB devices are potentially being plugged into your POS systems.
• Requirement 10.5.5 requires the use of file integrity monitoring which would catch the installation of vSkimmer as a foreign piece of software even though it masks itself as ‘svchost.exe’.  This would provide a backup control for requirement 5.1 if vSkimmer changes its approach as to its file name and is not caught by the anti-virus solution.

In addition to the PCI requirements, you can do the following to increase your security in regards to vSkimmer.

• Do not allow USB devices to be connected to your POS systems.  Most card terminals are RS232 devices, but USB is becoming more common.  The Windows Group Policy function can be used to disable USB ports on Windows systems.  There are also third party solutions that will disable USB ports.  A lot of these third party solutions can offer additional granularity in what types of USB devices can be connected.  This can be very advantageous when you are using USB card terminals which still need to connect, but other USB devices should not be allowed.  One of my more imaginative clients hot glues the ports shut on their POS systems.
• Train your staff on the vSkimmer threat.  Explain how it works and what they can do to minimize this threat such as not allowing anyone to manipulate the POS systems other than employees responsible for the care and maintenance of the systems.
• Lock your POS systems in a sealed cabinet or cage and only allow the manager on duty to have the key.  This may also involve additional security on POS servers if those are also used by your POS solution.
• Periodically review video of your POS stations to determine if cashiers or other personnel appear to be manipulating the POS system.

If you adopt all of these measures, you will significantly reduce the threat presented by vSkimmer and will likely never encounter it.

Have You Been Paying Attention?

Mandiant recently released an interesting report on APT1, the supposed Chinese Army hackers that are behind most of the serious digital attacks being conducted these days.  The reason I bring this up is that, even before the Mandiant report, I had been getting questions regarding how the PCI DSS, ISO 27K, HIPAA and other standards could possibly block or even alert an organization that they are under such an attack or have been compromised by advanced persistent threat (APT).  I will use the APT acronym in this post to indicate any sophisticated attacker, not just those that follow the methods used by APT to compromise a network.

It fascinates me how many organizations jump through hoops to tightly control and monitor their inbound network traffic from the Internet, business partners and any other untrusted network.  Yet the same organization does nothing about the outbound side of the equation.  It is the age old belief that all internal traffic can be trusted because it is from the inside of the network.

It is this trust factor that APT relies upon in their ability to compromise your network.  Because you have tightly controlled your inbound traffic, APT relies on social engineering to gain access to the internal network.  Once inside, since most organizations do little to nothing to control or monitor their outbound traffic.  It is therefore a relatively simple task to connect from the inside of a network to any external IP address and set up a communications link.

Even when organizations are controlling and managing their outbound traffic, it is still easy to connect using ports 80 (HTTP) or 443 (HTTPS).  In this day and age, it is virtually impossible to be in business without allowing ports 80 and 443 outbound.  Attackers know this and use this fact against a target in order to gain a way out.

So what can you do to minimize this risk?

• Tighten you inbound traffic from the Internet and business partners.  I consistently find that organizations allow traffic from countries where they have no business relationships.  As a result, these IP addresses can be used to launch denial of service and hack attacks.  If you do not limit where inbound traffic can come from, you are only asking for a denial of service attack or worse.
• Control what external IP addresses and URLs your internal network can connect.  While a lot of organizations today use overseas business partners, is that any reason to let your internal network connect to any IP address?  No.  This is typically set up by a network engineer that does not want to be bothered by a lot of service requests to allow IP addresses or URLs.  However today’s sophisticated network attacks are only stopped if you control what IP addresses or URLs your users can connect.  Yes, this potentially creates a “pain in the rear” issue when someone needs to access something you have blocked.  But would you rather stop a breach or let it go unchecked until you accidentally trip over it?
• Control your outbound ports with the same exuberance that you use on your inbound traffic.  If all you need from your internal network outbound are ports 80 and 443, then those should be the only ports allowed out on your firewall.  Enough said.
• Monitor your outbound network traffic for “anomalies.”  The biggest anomaly you should be looking for is encrypted traffic over any port other than 443 (HTTPS).  The reason is that any application using anything other than 443 for encrypted communications is likely an attacker trying to obfuscate the fact that they have compromised your network.  After all, your firewall and IDS/IPS cannot read the packets encapsulated in the encrypted stream, so the attacker effectively blinds them to what is actually going on.  Encrypted traffic that does use 443 hopefully will get blocked by your white or black list of IP addresses and URLs.
• Analyze the log data generated from your firewalls and IDS/IPS looking for potential APT attacks.  Are there consistent indications that devices are trying to connect to blocked IP addresses or URLs?  Do those incidents occur after hours or when the user is not logged on?  These are examples of conditions that should be flagged and investigated to ensure that something bad is not occurring.

Remember, security is not perfect.  What these controls do is make it more difficult for APT to successfully compromise your network.  However, if APT puts their mind to it, there are likely ways to successfully compromise your network but it will take a significant amount of work.  In the end, I would like to assume that since there are many more targets easier to get at, APT will go after the easy targets and leave your organization alone, at least for the time being.

Service Provider PCI Compliance Process

As a recent question pointed out to me, while service providers now seem to understand they need to be PCI compliant, they do not seem to understand the process under which they assess their PCI compliance.  As a result, I thought I would use this space to clarify this process.

When Do I Have To Be PCI Compliant?

The PCI SSC defines a Service Provider in the PCI DSS Glossary as:

“Business entity that is not a payment brand, directly involved in the processing, storage, or transmission of cardholder data.  This also includes companies that provide services that control or could impact the security of cardholder data.  Examples include managed service providers that provide managed firewalls, IDS and other services as well as hosting providers and other entities.  Entities such as telecommunications companies that only provide communication links without access to the application layer of the communication link are excluded.”

On the straight forward side of this definition, if your organization processes, stores or transmits cardholder data on behalf of one or more other organizations, your organization is considered a service provider and needs to be PCI compliant.  That point seems to be clear and most organizations that are processing cardholder data for others understand this fact.

It is organization that do not directly process, store or transmit cardholder data but could impact the security of cardholder data that get tripped up.  The bottom line is, any third party that has access to an organization’s cardholder data environment (CDE) needs to be PCI compliant for those services offered that come into contact with the CDE.

Based on this definition, the following are examples of services would need to be PCI compliant.  This is not a complete list but something to assist you in understanding why your organization might need to be PCI compliant.

• Management of firewalls that are considered part of an organization’s CDE.
• Management of networks that carry cardholder data that is not encrypted.
• Management of encrypted networks that carry cardholder data and the MSP manages the encryption keys for encrypting the network.
• Management of servers that process, store or transmit cardholder data.
• Management of PCs that have access to cardholder data.
• Management of applications that process, store or transmit cardholder data.

The first question service providers typically ask is, “How am I supposed to know if my customer needs our services to be PCI compliant?”  My rather indignant response is, “What, you cannot ask them?”

If you are a service provider and you are providing any value added services, you need to be asking every customer about their regulatory or legal requirements when going through the sales and contract processes.  It is not just PCI these days, but in the US there is HIPAA, GLBA, SOX and there all sorts of similar requirements in various parts of the world that you should be finding out whether your prospects need to comply.  If you are not, then you are sitting on a potential legal time bomb should something happen and your value add services are deemed the culprit.

SAQ D or ROC?

Service providers are broken into two levels, Level 1 service providers and Level 2 service providers.  Level 1 service providers are those that process more than 300,000 Visa, MasterCard or Discover transactions annually and are required to perform a full PCI assessment using a qualified security assessor (QSA) that results in a Report On Compliance (ROC).  Level 2 service providers are those that process 300,000 annual transactions or less and they can conduct their own assessment using the SAQ D.  SAQ D is the only SAQ that service providers are allowed to use.

The first issue most MSPs face is determining what parts of the ROC or SAQ are relevant.  There are no easy answers in this area as it all comes down to those value added services you are providing and how you provide those services.  However, before you can determine what parts of the ROC or SAQ are relevant, you need to determine the services that are to be assessed.

Where MSPs can miss the boat is taking the lowest common denominator approach and only assessing those value added services that all customers need to be PCI compliant.  Typically this results in an AOC that discusses physical security (requirement 9) and security policies (requirement 12).  If you take the lowest common denominator approach, then do not complain when each of your customers’ QSA have to traipse through your facilities and disrupt operations over the value added services you did not cover in your AOC.

The best approach is to assess all value added services you provide so that you do not have to worry about whether or not a given service is PCI compliant.  The reason this is important is that customers are required to make sure that all service providers that need to be PCI compliant are PCI compliant.  It is not that customers cannot work with non-PCI compliant service providers; it is just a lot easier to work with PCI compliant service providers.  The reason is that non-PCI compliant service providers have to be annually assessed as part of the customer’s PCI assessment as well as the customer is supposed to be monitoring the service provider’s PCI compliant efforts.  As a result, you will have an easier time and be easier to deal with if all of your value added services are PCI compliant.

Now back to what parts of the PCI DSS are relevant.  There are too many permutations of services and PCI requirements to go into this for every possible situation, so I am just going to provide some examples.

If you are providing firewall management services, then you are going to have to comply with requirements 1, 2, 4 (if your organization is managing any transport encryption), 5 (the PCs/servers used to manage the firewalls), 6 (change control for the PCs, servers and firewalls), 7, 8, 9, 10 (log management and analysis for the PCs, servers and firewalls), 11 (PCs, servers and firewalls) and 12.  Some tests in these sections may be Not Applicable for your organization, but you will need to go through all of the tests in the sections.

If you are managing applications such as in a software as a service (SaaS) environment, you are going to have to comply with requirements 1 (if you manage any firewalls or routers that could be in-scope), 2 (servers, firewalls, routers, switches), 3 (if your application encrypts the cardholder data), 4 (if your network encrypts cardholder data), 5 (servers running the application as well as any support PCs and servers), 6 (change management as well as any software development), 7, 8, 9, 10, 11 and 12.  Some tests in these sections may be Not Applicable for your organization, but you will need to go through all of the tests in the sections.

As you can see, even a “minor” service such as managing firewalls can require a significant amount of effort to be PCI compliant.

Do I Need A QSA?

A Qualified Security Assessor (QSA) is only required if: (1) your organization is considered a Level 1 service provider, or (2) you wish to have your organization listed on Visa’s Global Registry of Service Providers or with the MasterCard Registration Program (MRP).  In both of these cases, you will need to hire a QSA and produce a Report On Compliance (ROC) and Attestation Of Compliance (AOC).  If you can answer ‘No’ to both of these, then you can do your own assessment using the SAQ D.

Scope Of Assessment

At the 2012 PCI Community Meetings, the PCI SSC clarified scoping for PCI assessments.  The PCI SSC stated that:

“To be considered entirely out of scope, a network or system must be isolated from, with no connectivity into, the CDE (such that even if the system is “owned”, it cannot impact the security of the CDE).
If connections are limited to specific ports or services, those systems are included in PCI DSS scope to verify applicable controls are in place.
Applicable PCI DSS controls may vary for in-scope systems depending on the function of the system and presence of other controls.”

As a result, any of your systems that have access into a customer’s CDE are in-scope for compliance as they could be used to access the CDE.  This does not mean that these systems need to meet all of the requirements of the PCI DSS.  But it does mean that they need to be evaluated to determine those portions of the PCI DSS that do apply.  All of this is dependent on the potential to access cardholder data inside the CDE.

To minimize the impact of this clarification, we are seeing a lot of MSPs using “jump boxes,” PCs that do nothing but provide access into the systems to be monitored or managed.  This provides an air gap between the MSPs internal systems and the customer’s environment, including the CDE.  As a result, the jump boxes need to be PCI compliant and the internal systems at the MSP just need to have current anti-virus and anti-malware and be properly authenticated.

Hopefully this post helps all of you managed service providers that are facing PCI compliance.

This post was revised after a review by Walt Conway at 403Labs pointed out some inaccuracies.  A big thank you to Walt for his diligent eye.

Bring Your Own Device And PCI Compliance

Bring your own device or BYOD is all the latest rage.  I believe that the reason for that exuberance is the consumerization of technology.  It is that exuberance through BYOD that has made everyone an “IT expert.”  Just ask any user of a smartphone or tablet and they will baffle you with vendor lingo regarding BYOD.  However, regardless of what these people think, there is a big difference from consumer use and enterprise use, the first of which is security.  In this post I am going to look at the minimum PCI DSS requirements BYOD will have to comply in order for your organization to maintain PCI compliance.

From a PCI perspective, requirement 12.3 is very relevant in the BYOD discussion.

“Develop usage policies for critical technologies (for example, remote-access technologies, wireless technologies, removable electronic media, laptops, tablets, personal data/digital assistants (PDAs), e-mail usage and Internet usage) and define proper use of these technologies. “

Under 12.3, there are eight sub-requirements that require:

• Explicit approval by authorized parties.  Just because you have a smartphone or tablet, does not imply you get an automatic pass to connect, you must be approved for that privilege.  In most organizations, that means you need to provide a business need to have such a privilege granted.  While access to email through a smartphone or tablet is one thing, access to cardholder data should be another and be granted very judiciously, if at all.
• Authentication for use of the technology.  If you think you are using a PIN, think again.  You still have to log onto any PCI in-scope system using a password that meets the PCI DSS requirements.  By the way, two-factor authentication is required to access cardholder data remotely, so BYOD does not get you a pass there either.
• A list of all such devices and personnel with access.  For organizations issuing BYOD, this is not a problem.  For organizations allowing any Apple iOS, Android or Windows device to connect is problematic.  While it can be done, you will need to document who was granted access, for what reason/purpose, and what they are using to obtain that access (i.e., make, model, etc.) even if your organization is not providing the devices.
• Labeling of devices to determine owner, contact information and purpose.   I take issue with identifying purpose of the device as that, in my opinion, could just make someone ever more curious as to what is on the device.  However, identifying the owner, giving their general business address and general voice and facsimile telephone numbers for their location is what I would recommend.
• Acceptable uses of the technology.   Email is one thing, doing something that involves cardholder data is another.  I think if companies think this through, there is probably little, if any, reason for BYOD to be even near the cardholder data environment (CDE).  However, if for some bizarre reason you can come up with a valid reason, then all requirements for remote access of cardholder data apply such as personal firewall, no way to disable the firewall, strong passwords, two-factor authentication, encrypted connection, etc.  And remember that these devices have keyboard loggers, so all data input is recorded so keep that fact in mind when designing your information security requirements for BYOD.
• Acceptable network locations for the technologies.  In the case of BYOD, this is anywhere outside of the organization’s network perimeter.
• List of company-approved products.  For those organizations issuing the BYOD, this is not a problem.  For those of you that allow anyone with anything to connect as long as it runs iOS, Android or Windows 8 RT/Pro, your list is going to say “Any iOS, Android or Windows 8 RT/Pro device.”
• Automatic disconnect of sessions for remote-access technologies after a specific period of inactivity.  With BYOD, this can be problematic as most BYOD do not use traditional remote access connectivity and inactivity can all be in the eye of the beholder, so to speak.  As a result, inactivity timeouts can be difficult, if not impossible, to enforce in some instances.  As a result, you may have to be either creative or use a compensating control to comply with this requirement.

Going through the above, I think most organizations would see BYOD for what it is – a fad.  Do not get me wrong, BYOD has its uses in the corporate environment, but they are fairly limited and likely does not include cardholder data.

However, if you have come up with a business justification for processing, storing or transmitting cardholder data using BYOD, there are number of other requirements you are going to have to address.  I know there are other potential requirements that could be involved, but these are the most well known that will need to be complied with under the PCI DSS.

• Requirement 1.4 – Install personal firewall software on any mobile and/or employee-owned computers with direct connectivity to the Internet (for example, laptops used by employees), which are used to access the organization’s network.   While this can be accomplished by a number of security vendors, it is the enterprise management of those solutions that is currently lacking and the ability to globally enforce policies.
• Requirement 3.4 – Render PAN unreadable anywhere it is stored (including on portable digital media, backup media, and in logs).  Most cell phones and tablets do not support device encryption.  As a result, how will you protect any cardholder information stored on the device?  Remember that these devices have keyboard loggers, so any cardholder data input on the device is collected and stored by the device whether you like it or not.  The bottom line is that you will have to restrict BYOD to only those devices that can support whole device encryption.
• Requirement 4.1 – Use strong cryptography and security protocols (for example, SSL/TLS, IPSEC, SSH, etc.) to safeguard sensitive cardholder data during transmission over open, public networks.  You would not think that this would be a significant problem, but it can turn out to be very significant and I will speak about this later on.
• Requirement 8.3 – Incorporate two-factor authentication for remote access (network-level access originating from outside the network) to the network by employees, administrators, and third parties.  This capability is available, but if you are using any of the solutions that run on the BYOD, your users will likely have issues trying to connect and get their token value from that device.
• Requirement 8.5.10 – Require a minimum password length of at least seven characters.  This is likely a deal breaker for users.  Most like their PIN or swipe and will not want to give them up for a seven character, strong password.  In addition, some security solutions may create a situation where the phone cannot be answered without unlocking the device and, a seven character password may cause calls to be missed.
• Requirement 8.5.11 – Use passwords containing both numeric and alphabetic characters.  This can be problematic when some virtual keyboards require flipping between three or more screens to get to certain special characters.  With phones with physical keyboards, there may be limitations to the number of special characters available that could create problems with the password uniqueness requirement in 8.5.12.
• Requirement 8.5.13 – Limit repeated access attempts by locking out the user ID after not more than six attempts.  Some security systems may not be able to enforce this without wiping the device.
• Requirement 8.5.14 – Set the lockout duration to a minimum of 30 minutes or until administrator enables the user ID.  I am not aware of a security solution currently available that can enforce this on a smartphone or tablet.  You will have to meet this with a compensating control.
• Requirement 8.5.15 – If a session has been idle for more than 15 minutes; require the user to re-authenticate to re-activate the terminal or session.  As with 8.5.14, I am not aware of a security solution that can enforce this on a smartphone or tablet.  You will have to meet this with a compensating control.
• Requirement 10.2 – Implement automated audit trails for all system components to reconstruct events.  This is not an issue with Android and may be an issue with Windows 8 RT, but is definitely an issue with Apple iOS.  There is no utility I am aware, outside of forensic utilities, which can meet this requirement in an Apple iOS environment.  In addition, I am not certain how you get this log data back for any sort of analysis without chewing up a tremendous amount of data bandwidth or device memory.  At the end of the day, this will also likely be satisfied by a compensating control, if you can actually come up with enough controls that go above and beyond the PCI requirements.
• Requirement 10.3 – Record audit trail entries for all system components for each event.  If meeting requirement 10.2 is a gymnastic event, then there is the configuration of the log data to ensure that all of the necessary log information is collected.  I am sure that under Android and Windows there is probably some way to ensure that the necessary log data is required.  But with Apple, iOS, Apple will have to be able to provide this capability.  And knowing how stubborn Apple can be about having their hand forced in these matters, getting access to configuration of log data let alone log data will likely be a battle.  Again this will also likely be satisfied by a compensating control, if you can actually come up with enough controls that go above and beyond the PCI requirements.
• Requirement 10.4 – Using time-synchronization technology, synchronize all critical system clocks and times and ensure that the following is implemented for acquiring, distributing, and storing time.  For smartphones, they get their time synchronization from the carrier.  However, that time does not likely correlate to the time used by enterprise systems.  It will be close, but not necessarily the same thus complicating any forensic examination if one is required.  This can obviously dealt with in a compensating control.
• Requirement 10.5 – Secure audit trails so they cannot be altered.  Given that users have administrator access on their BYOD; this could be a problem that cannot be easily solved.  Yes, there are security solutions available that can lock down a device, but they can also lock them down so far that users can begin to wonder what the point of having the device is.  As a result, you will find you have a very fine line to tread in this area.

In the end, you start to understand why BYOD is a difficult thing to justify when you need to comply with all of the aforementioned PCI requirements.  But there are a few other considerations that you will still need to address.

The first situation that should concern any organization considering BYOD is the loss of that BYOD.  It is virtually guaranteed that BYOD will result in lost devices and you will need policies, standards and procedures to address that eventuality.  The way most organizations address this issue is providing a remote device wipe capability that can be invoked whenever a device is reported lost.  Not that such capability ensures that every lost device is wiped, but it is better than nothing, but not by much.  This is usually backed up by the bad password entry policy that wipes the device after six incorrectly entered passwords.  So even if the remote wipe does not destroy the data, the bad logon attempts will.

BYOD brings up an information ownership issue as, according to the latest statistics, 70% of all BYOD are owned by the employee, not the organization.  As a result, you are allowing the organization’s information to be processed or stored on a device not owned or necessarily even controlled by the organization.  While you can have an employee sign an agreement regarding the organization’s ownership of the information and the employee’s responsibilities for protecting that information, the issue of enforcement of such an agreement can be very problematic depending on platforms and other technology issues.  You can have a remote wipe capability, but that brings up the potential legal issue of can you also wipe an employee’s personal information such as contacts, music and pictures as well?  Just remember the old saying, “Possession is nine tenths of the law.”  When you finally get through court proceedings, does it really matter if you won when the data has already likely been disseminated?

Then there is the question about whether your applications actually capable of dealing with BYOD?  Most internally facing applications that employees desire access are not engineered for secure, remote access from BYOD.  As a result, a lot of organizations are turning to VPN technology to solve the remote access issue.  However, organizations using VPN are finding out that, while VPN clients are free for download to the BYOD, licensing for the BYOD VPN client is over and above the licensing the organization has already purchased for notebooks.  In addition, depending on the device, running the VPN client can make devices run very slow to the point of being worthless when connected.

When VPN is not a solution a lot of organizations are using remote desktop (RDP), Citrix virtual desktop or similar remote control environments to provide secure access to internal applications.  Having worked with a few of these in a smartphone and tablet environment, I can tell you their use is haphazard at best due primarily to screen size and lack of a mouse and a real keyboard.  In addition, we also find that some of these secure remote desktop solutions are not using secure communication methods.

In addition to VPN and remote control, a lot of organizations are implementing HTTPS for secure connectivity to their applications.  However, this creates all sorts of new security issues related to authentication and protecting the applications which are typically not engineered to be externally facing.  We are finding that in the rush to enable applications for HTTPS, there are numerous security vulnerabilities that are being introduced.  We also see vulnerabilities as well with applications developed specifically for mobile devices.  In the haste to get BYOD up and running, the security vulnerabilities are not corrected before the applications are put into use (a violation of PCI DSS requirement 6.6) which puts the applications and, potentially, the internal networks at significant risk of compromise.

And finally, data input from smartphone and tablets can be highly erroneous not just because of human typing errors but also because of auto-correction systems that are implemented on these devices.  Anyone considering significant data entry through BYOD is just asking for trouble as this erroneous data input could result in legal issues later on due to mis-spellings and mistakes in addresses and other information.

Most executives do not understand the security and privacy issues of BYOD because they have not encountered them and are not aware of them even when time is taken to educate them.  Unfortunately it usually takes an executive losing their BYOD to help management appreciate the issues with BYOD and to slow down the drive to integrate BYOD until their concerns regarding security and privacy are addressed.

As you can see, using BYOD is not as simple a process as your end users might think.  This is even truer when that BYOD will be part of your cardholder data environment (CDE).  There are a number of innovative solutions for BYOD that are secure, but those solutions are expensive and make the BYOD only a display device.  However, if you want to be able to sleep at night, I would highly recommend looking at those purpose built solutions.

The Barnes & Noble Breach Take Aways

On October 24, 2012 it was announced that Barnes & Noble had a credit card breach that was the result of tampered credit card terminals.  As a result of the breach, Barnes & Noble pulled all of the credit card terminals out of their stores so that they can be examined.  The story published in the New York Times has some points that should be interesting to other large merchants.

“We have acted at the direction of the U.S. government and they have specifically told us not to disclose it, and there we have complied.”

This is probably the most important take away you should have because a lot of incident response plans miss this point.  While the credit card companies want to the notified immediately of a breach, law enforcement should be the first outside entity notified and then the card companies, if approved by law enforcement.  The reason is that law enforcement may want the breach to continue in an effort to more easily identify and apprehend the perpetrators and that may include allowing the perpetrators to use the stolen cards for purchases.

But the next question that typically comes up is who in law enforcement should be notified?  If you are not a large or regional entity, then you should notify your local police department or county sheriff.  If you are a regional or large sized merchant in the United States, you should contact the United States Secret Service and/or the Federal Bureau of Investigation.  In either case, whatever law enforcement entity you contact should be consulted with before notifying anyone else outside the organization and that includes notifying the card brands.

“The company determined that only one keypad in each of the 63 stores had been hacked.  Nevertheless, the company has not reinstalled the devices.”

The 63 stores involved were all across the country from San Diego, Miami, Chicago, New York and other locations in between.  This implies either a very organized criminal group that operates in a lot of locations or to a localized group that was able to infiltrate the operation that configures and ships out the terminals for Barnes & Noble.  Based on investigations similar to this, it is most likely that a criminal operation infiltrated a centralized location that is responsible for the configuration, repair and replacement of credit card terminals for Barnes & Noble.

So what can a merchant do to minimize this sort of attack?  Here are some actions to consider.

• Contract with only a reliable terminal supplier.  In this age of lowest cost providers, there is a big temptation to use anyone as a supplier, particularly if their costs are the lowest.  However, the old adage of “you get what you pay for” is very relevant in these situations.  As part of your vendor selection process, you should be asking a supplier of terminals what they do to ensure that terminals do not get tampered with.  If you cannot get an answer or the answer you get is “trust us,” then you should probably not consider them as a vendor.  At a minimum, vendors should put their employees through periodic background checks (at least every three to five years), track which employees work on what units, do random physical internal inspections of units and random testing of units to ensure that they are not tampered with before they are sent out.  If you are doing this activity in-house, you should also be following this process.
• Lock down your terminals.  Anyone that has been into a Barnes & Noble might recall that terminals just sat on the counter.  As a result, they were easy to quickly swap out with a doctored unit.  I have been involved in a number of situations where merchants had terminals doctored because they were easy to swap out.  If terminals are locked in a cradle and only the manager on duty has the key, anyone trying to swap terminals is going to have to have a key to free the device.  This prevents swaps that occur after hours when only the cleaning people are present.  In addition, the keys to these terminal cradles needs to be different for each location so that one key does not open every cradle at every location.  The common key is a lesson the gas station industry has only recently addressed.
• Use tamper-proof serialized security tape or stickers over the seams of the terminal and check them daily.  This is a trick that has been used for quite a while with gas pumps.  The key is to at least daily (I recommend at each manager shift change); have the stickers checked to make sure that they are still in place and log that activity.  If they have been tampered with or are missing, the lane should be immediately taken out of service and your loss prevention unit contacted.
• Confirm a terminal swap.  A lot of merchants are very lax in their terminal swap procedures.  If a terminal turns up with instructions to swap it with another or a technician appears at the location with a new terminal, the store personnel do it, no questions asked.  That is wrong.  At a minimum, a good terminal swap procedure should involve the generation of a trouble ticket in a help desk system or similar and having the store manager confirm the swap with the help desk or POS support.  No ticket, no swap, no exceptions.
• Put video monitoring on all your POS locations.  This does not stop such a swap from occurring, but it does at least record such an event if it does occur.  This is particularly important in situations where the customer also acts as cashier as with any self checkout situation.
• Use MAC address filtering on your store location networks.  If a device is unplugged and a new device is plugged in with a different MAC address it will not work.  Yes, I know for some of you this creates a bad situation.  But I always ask people in response, “Why should store personnel be swapping equipment in the first place?”
• Monitor your sensitive devices.  If a credit card terminal or POS gets unplugged from your network, you should generate an alert.  That alert should then be correlated to a help desk ticket.  If there is no ticket, then someone should immediately notify loss prevention and also follow up with store management to find out why the device was unplugged.
• Monitor your network.  Terminals or POS should only be communicating with your service provider for transaction authorization and your routers(s) and/or firewall(s) should be configured accordingly.  If a terminal or POS attempts to communicate with any other external IP address, that should generate an alert to corporate IT and security that should then be investigated immediately.  This will catch those devices that are tampered with and then transfer data to a server outside of your network.  It is highly likely that the communication will be encrypted, but the traffic will be directed to an external IP address that should be blocked if your firewall(s) or router(s) are configured properly.

UPDATE – October 24, 2012: I am quoted on the Financial Times Web site regarding the B&N breach. http://www.ft.com/intl/cms/s/0/919e3292-1dd7-11e2-8e1d-00144feabdc0.html#axzz2BjZ8C9WJ

UPDATE – November 1, 2012: I got to be a guest blogger on CNBC because of this blog entry. http://www.cnbc.com/id/49624444/How_Retailers_Can_Avoid_a_Credit_Card_Breach

UPDATE – November 5, 2012: I am interviewed by a USA Today reporter. http://www.usatoday.com/story/tech/2012/11/05/debit-card-account-theft-qa/1677537/

UPDATE – November 6, 2012: I am quoted on the front page of USA Today. http://www.usatoday.com/story/tech/personal/2012/11/05/debit-card–numbers-pins-stolen-at-pos-terminals/1675795/

UPDATE – November 8, 2012: I am interviewed on WCCO 830 Radio in Minneapolis.

The Amazon Cloud And PCI Compliance

If there ever was a hot topic these days it would be “The Cloud” and, in particular, the Amazon cloud.  And that discussion inevitably leads to how are the Amazon cloud offerings are PCI compliant?  A lot of this discussion has to do with the very limited amount of information regarding the Amazon service offerings.  For some very bizarre reason, Amazon puts organizations interested in their PCI compliant services in a Catch-22 situation.  Unless you sign up for one or more of the services, you cannot obtain the information on how the Amazon service offerings are PCI compliant.  As a result, there is a lot of mis-information running around regarding the Amazon cloud.  So to debunk all of the myths running around, I thought I would explain what the Amazon cloud is and is not and how it ends up PCI compliant and what you need to understand when deciding to use the Amazon cloud.

And before I get calls from someone at AWS about the fact that I am somehow singling them out or I am being unfair.  I do not have a problem with AWs or anyone organizations’ cloud service offerings.  What I have an issue with is how some service providers use obfuscation and confusion about their services in ways that make customers unsure of whether they are getting something that is PCI compliant.  As I see it, the AWS service offerings seem to be PCI compliant, but there are things that possibly should be further explained so that everyone understands how that compliance is achieved.

The first part of the mythology revolves around what PCI compliant services Amazon Web Services, LLC (AWS) is actually providing.  According to AWS’s Attestation Of Compliance (AOC), AWS is a Hosting Provider for Web and Hardware.  The AOC calls out that the following services have been assessed PCI compliant.

• Amazon Elastic Compute Cloud (EC2);
• Amazon Virtual Private Cloud (VPC);
• Amazon Simple Storage Services (S3);
• Amazon Elastic Block Store (EBS);
• Amazon Relational Database Service (RDS);
• Amazon Elastic Load Balancing (ELB); and
• Amazon Identity and Access Management (IAM).

The AOC lists nothing for software provided through any of their services.  As a result, a big myth that gets busted right off the bat is that AWS is providing software.  At the end of the day, all AWS’s services are offering is Infrastructure as a Service (IaaS).  As a result, how AWS is PCI compliant is fairly easy to figure out.  They have totally minimized their responsibility on the PCI compliance front.

In addition to the AOC, AWS provides customers with a document entitled “AWS PCI DSS Controls Responsibility Summary” (CRS).  This document explains the various services and the responsibilities a customer organization has when using these services.

The first piece of infrastructure used by AWS is virtualization in the form of Xen as their hypervisor.  Because of the way AWS has implemented Xen, every virtual instances created by EC2 acts like an individual physical server in that there are no connections to any other server unless the organization defines such connections.  This is referred to in the CRS as instance isolation.  Finally comes the firewall.  EC2 includes a firewall that is managed by the customer.  Access to the firewall is controlled by an X.509 certification and access credentials provided through IAM.  In addition to utilities to manage the cloud environment, AWS provides various application programming interfaces (API) to manage the AWS cloud environment.

The bottom line is that, at a minimum, an organization needs to subscribe to EC2, VPC and S3 in order to build a basic platform capable of computing (i.e., server, connectivity and storage).  The need for other services outside of these will depend on what the organization is attempting to accomplish, whether or not they need the flexibility and scalability provided by AWS and other business factors.

From a PCI compliance perspective, the CRS categorizes the 12 PCI requirements into those that are AWS’s responsibility, shared responsibility between AWS and their customer and those requirements that are solely the customer’s responsibility.

In the AWS is responsible category falls requirement 9 or physical security and environment controls.  Since AWS is providing the facilities to operating the underlying physical hardware, it is solely responsible for this requirement.

In the shared responsibility category falls requirements 1, 10 and 11.  For requirement 1, AWS acknowledges that this is a shared compliance responsibility between AWS and their customer.  However, AWS’s responsibility is only to provide a firewall and ensure that it segregates their customers from one another.  The remainder of the responsibility for complying with requirement 1 is left to the customer.

For requirement 10, AWS indicates that they are responsible for:

• Maintaining log files for EC2 and S3 customer management operations (e.g. creation, modifications and deletion of these environments) for at least a year.
• Maintaining logs for the underlying software that provides the various services for at least a year.

This log information is monitored at least daily and is available to customers for their particular environment should it be necessary.  All other parts of requirement 10 are the responsibility of the customer.

For requirement 11, AWS indicates that they are responsible for ensuring the security of their environment including ensuring wireless security.  Customers are responsible for ensuring the security of the environments they construct using AWS’s services.

All of the remaining requirements, 2, 3, 4, 5, 6, 7, 8 and 12 are solely the responsibility of the customer.

So after all of this rigmarole, what is the advantage to be gained?  Not much near as I can tell.  The bulk of responsibility for PCI compliance still falls on the organization using the AWS services.  So organizations looking to offload as much of their PCI compliance responsibilities as they can to AWS are looking in the wrong place.

But it does not end there.  We are seeing more and more startup service providers that are using AWS services to avoid the capital costs of hardware and software of a 24/7/365 operation.  Where this becomes tricky is when you have a service provider providing PCI compliant services effectively using AWS for their “data center.”  In some cases, these service providers are trading on the fact that because AWS is PCI compliant, then their services must also be compliant.  However, what these service providers forget is that once they start going beyond the IaaS model and offer services in the Platform as a Service (PaaS) and Software as a Service (SaaS) realm, they are now responsible for portions of PCI compliant that Amazon is not.  As a result, organizations need to conduct due diligence on vendors using other cloud providers to provide their services to ensure that everyone is PCI compliant.

So do I think your organization should rush right out and sign up for AWS?  Maybe if you have the right business case.  But I do have some concerns regarding AWS’s service offerings and the statements surrounding how they are PCI compliant.

My first concern is in regards to requirement 1.2.3.  This requirement is one of the few that is not allowed to be marked ‘Not Applicable’.  As such, the QSA is required to document what procedures they conducted to ensure that any existing wireless is either not in-scope or that there is wireless in-scope and how it is secured.  To document this, AWS’s QSA has written:

“[AWS] maintain this control for all internal and external services that it provides. In EC2 and VPC environments, this includes the network at the hardware and management level networks, which are not exposed to customers.”

This statement says nothing of what procedures were conducted to ensure that wireless was not visible to customers as well as the controls AWS maintains to ensure wireless stays out of scope.  Essentially, we are asked to trust AWS that wireless is not on any customer networks.  Now, to be fair, AWS is operating secured data centers comprised with racks of hardware all virtualized, so the likelihood that wireless would exist in such an environment on any one customer’s network is remote at best. .  However, the PCI assessment process is all about verifying such statements, not just accepting them at face value as fact.  As a result, I am concerned that what is supplied as evidence for complying with this test leaves much to be desired.  What should be documented here are the procedures the QSA used to confirm that the controls AWS has in place are adequate to ensure that rogue wireless does not end up in their data centers.

Related to requirement 1.2.3 is requirement 11.1.  As with 1.2.3, 11.1 is also not allowed to be marked as ‘Not Applicable’ regardless of whether wireless is implemented or not.  For all of the tests under 11.1, the following statement is made.

“[AWS] maintain[s] this control internally.”

So what exactly does AWS do to ensure that their data centers remain wireless free or that wireless does not end up on the customer side of the network?  No idea.  I would like to assume that AWS is doing the right things in this regard, but, again, the PCI assessment process does not allow for assumptions, they require proof and this just does not pass muster.  At a minimum, there should be a discussion of the procedures used by AWS to ensure wireless is not an issue.

While we are discussing requirement 11, we should cover vulnerability scanning, penetration testing, intrusion detection and critical file monitoring.  All of which are the customer’s responsibility, not AWS’s.  Again, AWS is providing IaaS and nothing else, so any such controls will need to be provided by the customer.

When reviewing the detailed responses in requirement 9, it was interesting to see that AWS is responsible for ensuring that for the portion of any customer’s cardholder data environment (CDE) that exists in AWS, AWS ensures that destruction of hardcopy materials are properly destroyed so to be unrecoverable.  This begs the question, “Why would AWS have any hardcopy to destroy in the first place if they do not have access to customers’ environments?”  No further explanation is given, but one would guess it was their lawyer’s idea just in case AWS might somehow come into contact with CHD on hardcopy.

The next area I have issue with is not related to the service, but related to how an organization contracts for the service.  In an effort to fully automate things, unless you are a Fortune 50 looking to put your entire computing environment in AWS’s data centers, you can forget about negotiating a contract.  When you sign up for any AWS service, you either accept their contractual terms and conditions by checking the ‘Accept’ box and clicking Okay, or you don’t get to use AWS.  I know of a number of organizations that had real issues with that approach and, as a result, backed away from a more aggressive use of the AWS environment or decided they just could not accept the terms and did not go to the cloud at all.  While the AWS contract does cover PCI compliance, but it essentially makes the customer the one legally responsible for compliance with AWS providing support when necessary.

So that is AWS in a nutshell.  Not a bad thing, but something an organization needs to go into with their eyes wide open and understanding that they still have significant responsibilities even though they are now in “The Cloud.”

Third Party Service Providers And PCI Compliance

There seems to be a lot of confusion regarding third parties that provide networking or hosting services and their obligations regarding PCI compliance.  This confusion is not uncommon as merchants and their service providers have not necessarily been provided enough guidance to understand their obligations.  I hope this post will clarify those obligations for all involved.

If you learn nothing else from this post, if a third party is providing your organization a service that has access to your cardholder data environment (CDE) or the third party could come into contact with your cardholder data (CHD), then that third party must ensure that the service complies with all relevant PCI requirements.  As a result, the third party needs to either allow you or your QSA to assess the services that they are providing or provide you with an Attestation Of Compliance (AOC) that documents that those services have been assessed and they are PCI compliant.

In the past, I have stated that third parties could also submit a letter signed by an officer of the third party stating that all of the services provided to their customer are PCI compliant.  Now that v2.0 of the PCI DSS has a separate AOC and the PCI SAQs have the AOC built into the SAQ, there should be no reason to need such a letter or to ask for one.  If a letter is what your third party is offering, it is better than nothing, but you should be pushing them hard for an AOC.  If they are reluctant to get you an AOC, as part of your vendor management process, you should take that into account and probably begin looking for a new vendor that will provide an AOC for their services.

The most common issue we run into with third parties is that their AOC or other representations of PCI compliance do not cover all of the services provided to the customer.  In case after case, we see the AOC covering requirements 9 and 12 and nothing else even though the services provided may require compliance with some or all of PCI requirements 1, 2, 3, 4, 5, 6, 7, 8, 10 and 11.

In a lot of cases, it is not that the third party does not want to comply with PCI; it is they are taking the lowest common denominator approach and only picked those services where all customers requiring PCI compliance are asking for an AOC.  That way they have reduced their costs of a QSA to assess their environment.  These third parties are accepting the fact that any customer that needs more services assessed will have to do it themselves.

Related to this issue is the third party that offers their SSAE 16 Service Organization Control (SOC) 1 report has proof of PCI compliance.  While a SOC 1 report can cover a few PCI requirements, people must remember that the SOC 1 report is structured specifically for financial auditors to ensure that the controls at a third party are properly constructed to support financial reporting at the customers.  As a result, a SOC 1 report is not going to be a substitute for an AOC that covers all services.  There is an alternative to this and that is to have the third party go through a SSAE SOC 2 report that focuses on the security controls of the PCI in-scope services provided.  We are hearing from third parties inquiring into the SOC 2 report, but cost and a lack of customers requesting such a report are driving why we do not see more SOC 2 reports available.

Another common issue we encounter is the refusal of the third party to cooperate in assessing the services provided to ensure they are PCI compliant.  There are still third parties that argue their services are not in-scope for PCI compliance even when it is painfully obvious that the third party’s personnel have access to their customer’s CDE and/or CHD.

The most common third party relationship we encounter is the management of routers or other layer 3 devices.  Where we encounter the most confusion in this relationship is in regards to the use of encryption to keep the network services organization out of scope for PCI compliance.  The key here is if the network services organization manages the encryption of the network, then they are in-scope for PCI compliance.  The reason is that the employees of the network services organization have access to the encryption keys and therefore could decrypt the communications and gain access to CHD transmitted over the network.  As a result, at a minimum, the network services organization is responsible for complying with some or all of requirements 1, 2, 4, 6, 7, 8, 9, 10 and 12.  If you receive such services and are not getting an AOC that covers these requirements, then you should be doing more work on your own as well as asking the third party why they are not covering more of the necessary PCI requirements.

The next most common service we encounter is the network services firm that is managing or monitoring an organization’s firewalls, remote access or intrusion detection/prevention.  Such services always put the third party in-scope for PCI compliance.  Some or all of requirements 1, 2, 6, 7, 8, 9 and 12 will need to be assessed for compliance with the PCI DSS.  The log capture and analysis requirements in requirement 10 may also be complied with if your organization is not capturing and analyzing the log data from these devices.

Another group of third parties we encounter a lot are records retention vendors.  Organizations like Iron Mountain have conducted their own PCI compliance project and readily hand out their AOC to customers.  However, where we see issues is with such vendors that provide their own tape library for their customers to use for backup.  We have encountered a number of third party’s doing the encryption at their library which puts them in-scope for PCI compliance, at a minimum, for requirements 3, 4, 6, 7, 8, 9, 10, 11 and 12.

We encounter outsourcing the data center a lot with large organizations, but small and mid-sized organizations are also hopping on the data center outsourcing bandwagon.  Where this puts the third party in-scope for PCI compliance is when the third party is responsible for maintaining the environment such as applying patches, managing servers or any other activities that would allow the third party’s personnel to potentially have access to CHD.  In such situations, at a minimum, the third party is responsible for complying with some or all of requirements 2, 5, 6, 7, 8, 9, 10 and 12.  Compliance with some or all of requirement 1 may be applicable if the third party is managing your firewalls or routers.  Compliance with some or all of requirements 3 and 4 may also be applicable if the third party is responsible for managing encryption keys for encrypting CHD or encrypting communications.

Where the most confusion regarding third party responsibilities occurs is in regards to “The Cloud.”  The most common reason for this is that every vendor seems to have a different definition for what “The Cloud” is, based on their particular services.  Using the definitions provided by the National Institute of Standards and Technology (NIST) in their publication SP800-145, ‘The NIST Definition Of Cloud Computing’, I can provide the following guidance.

If your organization is purchasing Infrastructure as a Service (IaaS), then the third party providing these services will typically be out of scope for PCI compliance except for requirements 9 and 12.  There are some instances where IaaS implementations may require compliance with the PCI DSS if the third party is managing network infrastructure that comes into contact with CHD as is usually the case with private cloud environments.

For Platform as a Service (PaaS) and Software as a Service (SaaS), the third party will have to provide PCI compliance for the services they are providing to your organization.  That is because with either of these service offerings, the third party must have access to the CDE and will have the potential of coming into contact with CHD.

The problem with the majority of PaaS and SaaS vendors is that they only deal with your organization through a Web-based interface, i.e., everything is automated – contracts, support, etc.  As a result, the contract is a “take it or leave it” situation that does not usually cover everything needed for PCI compliance, there is no way to independently verify the representations made by the third party as well as the fact that the AOC provided by the third party typically only covers only the physical security requirements in requirement 9 and possibly some of requirements 11 and 12 and nothing related to the other requirements, even though the third party may have responsibilities for PCI compliance outside of what is represented in their AOC.

If this is the case, there is little you or any QSA can do to properly assess the environment to ensure it is truly PCI compliant.  As a result, we have a lot of organizations that try to develop compensating controls for these cloud implementations.  These organizations very quickly and frustratingly find out that there are very few, if any, controls on their side of the equation that can get them to “above and beyond” the original requirement.

I know there are a lot of other examples of services being provided to merchants.  But, hopefully these examples can assist you in clarifying what you need or do not need from your third parties when it comes to PCI compliance.

Defense In Depth

I have a slide in my security presentation deck that discusses the concept of defense in depth and how when you start opening ports or start using encrypted data streams how you are punching holes into one or more of your security layers.  It amazes me how many people still do not understand how defense in depth works and how much security standards such as the PCI DSS rely on this concept.

So let us take a look at the various elements of security and the requirements of the PCI DSS and see how they bring defense in depth to bear.  Keep in mind this is an example and does not encompass everything an organization could do to increase defense in depth.

For most organizations, the first level of defense is at their firewall.  Requirements 1 and 2 talk to how you should use a firewall and secure it.  The biggest mistake that organizations make is not configuring their firewall properly.  And by configuration, I am not just talking about the configuration of the firewall’s software; I am also talking about where and how the firewall is used in the network.

The next level of defense for most networks is usually some form of intrusion detection/ prevention system.  Some of the requirements in 10 and 11 talk to intrusion detection/prevention.  IDS/IPS capability may be provided in a separate appliance or may be part of an organization’s firewall.  The key to using an IDS/IPS is ensuring that it is kept current with its attack signatures, monitoring its log data and/or console and ensuring that it is not be overwhelmed by network traffic.

One thing that continues to amaze me is how many implementations of IDS/IPS I encounter where the IDS/IPS are in the middle of encrypted data streams.  IDS/IPS systems cannot examine encrypted data streams unless they have the decryption keys which they typically do not have access.  As a result, encrypted data streams are not examined and therefore sensitive data and or attacks could be going right past the IDS/IPS.

How users authenticate to your network and devices is also a level of defense.  Requirements 7 and 8 of the PCI DSS talk to this point.  And it is not just authentication to applications that process, store or transmit cardholder data, it is also authentication to infrastructure devices and to databases.

It has been more than five years since the “sa” default password debacle and yet you still encounter applications that use service accounts to access their database and those service accounts have no password.  The rationale?  “We did not want to code the password into the application,” is the common reply.

The other big area of authentication issues that you encounter is with firewalls, routers switches and other network infrastructure.  The problem is that the network administrators all use the same account and password.  You can understand their rationale, particularly those networks where you are administering thousands and thousands of devices.

There are a number of ways to address this situation, but these are my favorite two.  The first is to implement 802.1X authentication using a RADIUS server.  Under this scenario, every network administrator has their own unique account and password to access the network devices.  Those unique accounts should be different from the network administrator’s account they use to get email and network access like every other user.  A lot of organizations already have the RADIUS server implemented for remote access, so adding in network administration access control is relatively easy.

The second way to address network administration access is to use a “jump box.”  In a “jump box” implementation, two or more “jump boxes” are placed at strategic points on the network and all network administration access is conducted through a “jump box.”  The “jump box” is fully instrumented in that all keystrokes, applications, etc. are logged and those logs are reviewed at least daily to ensure that network administrators are not changing things they should not be changing.  That means comparing service tickets for the network against the logs from the “jump boxes” and ensuring that only what was required to be changed was changed.  “Jump boxes” can also be used to control access for server administration.

A level of defense that usually gets little recognition is operating system (OS) hardening.  What some people seem to forget is that any computerized device has an OS whether it is a firewall, router, switch or server.  Requirement 2 talks not only to the hardening of wireless, but also firewalls, switches, routers and servers.  Every vendor publishes a guide that explains how to securely implement their OS.  Where things can get sticky is with third parties that argue that their product or software will not function if you follow the vendor’s OS hardening recommendations.  In my experience, testing a vendor’s product or software in a hardened environment typically does not have an adverse result.  However, the key is to conduct a test.

Another level of defense is anti-virus and anti-malware software.  This solution also usually includes a personal firewall on mobile devices such as notebooks, netbooks and smartphones.  Requirement 5 of the PCI DSS talks to anti-virus and anti-malware while a requirement in 1 talks to personal firewalls.  Nothing gets some people wound up more than anti-virus software.  The requirements in 5 can have compensating controls, but implementing those compensating controls consistently on mobile devices is usually just about impossible.  So while you may not have anti-virus/malware on your e-Commerce servers, you should have it on all of your desktops, notebooks, netbooks and other systems.

A level of defense that most organizations poorly manage is their collection and analysis of log data from their network devices and servers.  Requirement 10 speaks to the importance of log data.  As I have written before, log data is IT’s version of commercial aircraft’s flight data recorder.  If you want to why a problem occurred, log data from your devices can usually point you to the reason.  The problem most IT professionals have with log data is that they do not want to log everything because that generates too much data in their opinion.  However, until you have an incident, you do not know what log data will be important in identifying why the incident occurred therefore you need all of it.  The last thing you want to have happen is to tell management that you could not determine the cause of an incident because you did not record the critical information required in identifying the incident.

The final defense most people think of is application development which is covered by requirement 6.  If you are going to get push back, this is the most likely and consistent place you will get push back to the PCI DSS or any other security program.  Application developers are very protective of their environments, so when you start infringing in their area, they can get rather upset.  As a result, you hear the typical lament from developers that security, “restricts their creativity.”

In today’s rush to get things done, application developers usually do not have security at the front of their minds.  As a result, by the time anyone knows that there is a security issue, it is too late for it to be fixed and the application goes into production with the fix to be part of version 2.  That was the whole point the PCI DSS addresses in requirements 6.4, 6.5 and 6.6; avoid putting susceptible software into production.  The whole point of these requirements is to build a certain amount of security into the development process to minimize the amount of security issues that end up in production.

The real final defense is an organization’s policies, standards and procedures.  Yes, that paperwork that everyone thinks is “make do” work, really does have a purpose.  An organization’s policies, standards and procedures are the rules that everyone is to follow to ensure security.  Those rules also provide a way to measure people’s compliance so that, in the event of an incident, those people that did not follow the policies, standards or procedures can be shown their mistakes so that they can correct their actions in the future.  These rules also provide an organization’s framework for explaining to personnel as to how the organization is protecting their information assets and defining those assets.

There are a lot more options for defense in depth, but I think you get the idea.  Now that you understand how defense in depth works, you should now also understand what happens when security personnel are asked to open ports for an application or change configurations that reduce the number of levels in an organization’s defenses.  The fewer levels involved the higher the likelihood that a lapse in control can result in a breach, particularly when a number of lapses in controls are occurring simultaneously.  This is how supposedly PCI compliant organizations end up breached.

2011 Verizon Breach Report

At this year’s Community Meeting, Christopher Novak of Verizon Business Services made a presentation regarding the 2011 Data Breach Investigations Report.  A lot of people blow off these sorts of presentations as being vendor focused or just a bunch of meaningless statistics.  However, Verizon provided a very good presentation on the nature of today’s threats and what organizations should be doing to protect their networks and systems.  Verizon’s Data Breach reports are a wealth of information; however I wanted to highlight just a couple of things that all organizations should take away from the report.

The first take away from the Verizon statistics is that while the media focus on the threat of insiders, it is the threat presented by external parties that is accounting for 92% of breaches, up 22% over 2010.  Not that insiders are not a threat, but only 17% of breaches involved insiders, down 31% over 2010.  Threats from business partners were less than 1% and only 9% involved multiple parties.  Yes, all of that adds to more than 100%, but as Mr. Novak reminded us, there are numerous instances while external actors ultimately breached the organization they were assisted by one or more insiders or a business partner.

The next important set of statistics is an analysis of breaches against the PCI DSS and its requirements for organizations that were in-scope for PCI compliance.  The first statistic that every organization should be concerned about is that Verizon found that 89% of all PCI in-scope organizations breached had been assessed by a QSA or themselves as not compliant with the PCI DSS.  This statistic should be a wake-up call to all organizations that complying with the PCI DSS is important.

In reviewing this demographic mix and the associated lack of compliance, we believe that the data reinforces an assertion we’ve been making for the past three years: to reduce risk, organizations of all sizes need to implement the basic tenets of an information risk management program and maintain this initial investment over time.  This includes network and data defense technology basics (firewalls, anti-virus, identity and access management), as well as the non-technical aspects of security and risk management (policy and process development).
Verizon Business Services 2011 Data Breach Investigations Report – page 62

PCI DSS Compliance and Breach Correlation

But the really important statistics were the analysis of compliance against each of the 12 PCI DSS requirements when a breach occurred.  What this analysis shows is that there are a lot of areas for improvement.

For building and maintaining a secure network, less than 50% of breached organizations were compliant with requirements 1 and 2 of the PCI DSS.  In protecting cardholder data, while breached organizations (requirements 3 and 4) were better at protecting cardholder data when it was transmitted (63%), they were horrible at protecting cardholder data when it is stored (43%).

It appears that organizations were complying with requirement 5 at 70% compliance.  However, given other survey results that claim over 95% of organizations have anti-virus and anti-malware, this is a troubling statistic as it points to incompetence in the installation and operation of these tools.

As I would expect, less than 50% of organizations were in compliance with requirement 6.  Whether that is related to patching or poor development practices, this part of the PCI DSS is always a bone of contention.

While overall, the requirements to implement strong access control measures, requirement 8 lags physical security and having access control systems implemented.  What this says is that while organizations have the tools in place, they are doing a poor job of managing them.

Regularly monitoring and testing networks is probably the area that needs the most improvement but explains another statistic in the report.  Compliance with requirement 10 and 11 were only 39% and 38% respectively when a breach occurred.  Earlier in the report, Verizon has a bubble chart that shows how long it took to breach an organization, how long it took the breached organization to acknowledge they were breached and then the time it took to address the breach.  It took attackers hours to days to breach an organization.  Unfortunately, it took weeks or months for an organization to know they were breached.  And to add insult to injury, it took additional weeks to contain the breach.

And finally, compliance with requirement 12 was only 44%.  A lot of organizations blow off documentation.  But it is the foundation of an organization’s security posture because it explains and describes why the other 11 requirements are important.  Without that foundation, is it any wonder that the other requirements are not in compliance.

One of the lingering questions from our discussions around PCI in this report is always that of relevancy. It’s all well and good to validate compliance with the PCI DSS, but does it actually help reduce risk? Insofar as that translates to a sincere security program—one that seeks to maintain validation on an ongoing basis—the data strongly suggests the answer is “yes.”
Verizon Business Services 2011 Data Breach Investigations Report – page 64

So what should an organization focus on to improve their security posture?  Here are my thoughts on a shortlist of issues that, if dealt with, would go a long way in securing organizations.

1. Really monitor your networks and systems.  No, I am not suggesting that everyone needs to go out and invest in a security incident and event manager (SIEM) or similar log management and analysis solution, but some organizations have no other choices because of their size and situation.  Regardless of how you choose to approach this problem, the key is that you must monitor your networks and systems.  And you need to monitor them regularly (daily) if not in real time.  The best organizations are doing real time monitoring and go through the PCI DSS tests and develop rules to monitor compliance with those requirements.  Attackers know that they can get away with their breaches because organizations are lax in their monitoring.  Just tightening monitoring up would go a long way in reducing the amount of data breached if not reducing the number of breaches.
2. Manage your access control systems.  No, it is not sexy to review user lists on a quarterly basis and disable, remove or reclassify users.  But here is a prime example of where organizations have a tool and are not using it correctly to improve their security.  Based on the report, it turns out that a lot of breaches were the result of poor user management practices and lousy passwords.  If this area got addressed, it would also go a long way in cleaning up the problem.
3. Engineer your networks and systems to be secure.  It fascinates me how many security professionals do things in the name of expediency that open huge holes in an organization’s security posture.  Another fascinating thing is how many security professionals have little knowledge of operating systems and applications.  Most security professionals seem to come out of the network administration teams as they were maintaining firewalls and routers when they went into security.  As a result, while they are very good at securing the network, server and application security lag because they just do not understand it nor do some find it all that important.
4. Test your security.  The only reason I can figure that vulnerability scanning and penetration testing is not being performed is that it will be used against security personnel to show that they are not doing their job.  While there are those security personnel that talk the talk but do not walk the walk, the vast majority of security professionals do an admirable job with few resources and even fewer plaudits.  However, given the rate of vulnerability production by attackers and researchers, if an organization is not conducting quarterly or even monthly vulnerability testing, there is no way an organization is going to protect itself.  And while the value of penetration testing will probably always be argued, I can tell you that it does have its place and can be invaluable if properly conducted.  This is particularly true for all of those organizations that think that because their vulnerabilities are all less than CVSS 4 that they are secure.
5. It is time to get control of application patching.  For small organizations, patching is only an issue due to a lack of focus on it as an issue.  For large organizations, it is the sheer volume of systems and devices that need patching.  There are numerous solutions for large organizations to deal with this situation, so find the tool that best fits your organization and get it implemented.  Oh, and get off the 30 day issue, that was clarified years ago.  You only need to have a reliable process and demonstrate that your patching process works on your time frame and patches do not fall between the cracks.

If all organizations could just focus on these five items and execute them at least 90% or better all of the time, a lot of the current breach issues would go away.  But that is the rub is it not?  A lot of organizations have issues maintaining that level of execution compliance.  Breaches occur because organizations get sloppy and, even with defense in depth in their security controls, there are too many controls where execution consistency has dropped leaving gaping holes in the various levels of security.

And as a reminder, these items address today’s problems.  However, once addressed, attackers will find other ways in, so the improvement process needs to be continuous.  And while organizations address the latest security concerns, they must be vigilant to cover the prior concerns because old vulnerabilities and exploits have a way of coming back.

VoIP And PCI Compliance

I have had some interesting conversations with people lately regarding voice over IP (VoIP).  It fascinates me as to how little people really know and understand how this technology works.  But what really scares me is how this lack of information is putting organizations at risk.

The most obvious problem with VoIP is segmenting it away from the cardholder data environment (CDE).  I am really disturbed by the number of organizations that have no security around their VoIP.  Yes a lot of organizations have segmented the VoIP from the rest of their network, but there are no controls that stop anyone from getting into that network segment.  As a result, anyone with the right set of tools can gain access to the traffic in the VoIP network segment.

The next thing that scares me is the lack of security surrounding the VoIP servers including any call recording servers.  People treat these VoIP servers just like their traditional PBX.  Unlike their PBX that likely ran a proprietary version of UNIX, VoIP servers are just Windows or Linux servers running a VoIP application.  As a result, they are susceptible to all of the viruses, malware and everything else any other server is susceptible.  However, these servers typically do not run anti-virus (performance issues) or are they hardened to any rational security standard.  When they get infected or hacked, it seems to be a shock to the system administrator.

And what about the call recording technology?  We keep hearing from vendors of call recording solutions that they use proprietary recording methods requiring special CODECs.  While in some instances the proprietary claims are true, what we are finding more and more is that vendors are just manipulating file header information such that Windows Media Player, iTunes and the like do not recognize the file as being in a valid format.  However, tools such as VLC Media Player are able to see past the header changes and recognize these files for what they are, WAV, MP3 and the like.  Thus some proprietary formatting claims are all a bunch of smoke and cannot necessarily be relied upon for security or privacy.  Another tell on the proprietary nature of call recordings is that, when you “convert” a recording, if the conversion software seems to be copying the recording more than actually converting it, it is likely that the header is being fixed to WAV, MP3, etc.  Real audio conversions typically take more time than just copying because the file has to be fully processed.

But the final insult in this whole scenario is the lack of understanding security personnel have regarding the VoIP protocols.  While VoIP call setup and teardown are usually conducted over TCP/IP (a stateful protocol), the actual call itself is conducted via streaming protocols over UDP/IP (a non-stateful protocol).  As a result, when you start talking to security people about VoIP security, their knee-jerk response is to tell you that VoIP is secured by the corporate firewall.  However, given that the VoIP protocols are stateless, even behind a firewall really does not provide any protection.

So you have a VoIP solution in place.  What should you be doing to ensure its security if it is in-scope for PCI compliance?  Here are my thoughts.

• Properly segment your VoIP from the rest of your network.  This means either physically or logically separating the VoIP from the rest of your network.  This also means implementing access control rules so that only those devices and people that need access to the VoIP network have access.  If you are also using your VoIP phones as the network jack for a PC, make sure to VLAN that jack to something other than the VoIP VLAN.
• If you can, implement the VoIP segment without DNS and DHCP and use MAC filtering to avoid the accidental or deliberate plugging in of a PC into a network jack that is VoIP only.  At a minimum, use MAC filtering to at least control what gets plugged into the LAN jack.
• Closely monitor your VoIP segment and generate alerts on any devices that are unplugged or plugged in.  Also monitor for any protocols other than the VoIP protocols that your VoIP system uses.
• Do not use the last octet or any other portion of the phone’s IP address as the extension number.  Yes, I know this is an easy way for the help desk to identify and troubleshoot phones, but it is also easy for an attacker to locate targets of interest, so keep that in mind when you are implementing your VoIP solution.
• Never, ever connect your VoIP network to another VoIP network outside of your explicit control.  Given that VoIP primarily uses UDP/IP, you cannot expect any firewall to protect your VoIP system from anything outside of your control.  Always use plain old telephone system (POTS) circuits to connect to any foreign network.  I know that is not as sexy as VoIP, but how else can you protect your VoIP system from outside influences?
• Work with your VoIP vendor to harden all servers that manage the VoIP system.  These are just Windows, Linux, etc. systems.  Obviously you will need to do some testing of this and you may not be able to use all of the hardening items in your server hardening standard, but you would be surprised at what you can do that the vendor says will not work.  Remember, they are just trying to cover their butts should a problem crop up.
• Be careful implementing VoIP on traditional PBXs.  A lot of these solutions are just PCs or servers and can be easily hacked once on your network just like their full VoIP brethren.
• Get a hold of VLC Media Player or similar tool and see if you can play a recording off of your call recording system.  We are getting about a 25% hit rate using VLC to play recordings.  A lot of the success of this approach depends on the age of the call recording system.  The newer the systems, the more likely it is that you will find that the recordings are just tweaks of existing standards.