Archive for June, 2009


Server Virtualization and PCI

Virtualization is all the rage.  And why not?  It has the promise of getting the most out of servers that are in the data center.  However, as with any technology, it comes with its own issues.  I bring up virtualization because I have run into a number of questions about virtualization over the last few weeks, so I thought it would make a good topic.

First, most people do not know or recognize the four types of virtualization in use today.  There is the type for today’s post that is the virtualization of servers.  There is virtualization of the network in the form of virtualized LANs or VLANs.  There is virtualization of storage using storage area networks otherwise know as SANs.  Finally, there is the up and coming arena of virtualized applications.

For virtual servers, we need to discuss the basic virtual server environment.  First, you have the hypervisor.  The first hypervisors were applications that ran on top of a host OS such as Windows or Linux allowing a single machine to run multiple guest OSes.  VMware Workstation, VMware Server and VirtualPC are examples of hypervisor applications.  Then came stand-alone hypervisors that run as their own environment and do not require a host OS.  VMware ESXi, Xen and HyperV are examples of stand-alone hypervisors.  The stand-alone hypervisor is how most organizations implement virtual servers for production.  The key feature of stand-alone hypervisors is their ability to allow guest OSes to execute over a cluster of physical servers allowing a significant amount of processor power to be harnessed for virtual servers.  For disk storage, virtual servers are typically tied to a SAN.  Only the stand-alone hypervisor typically boots from the physical disk(s) in the physical server, but that does not have to be the case.

A lot has been written over the last couple of years regarding virtual machine exploits, in particular Blue Pill and SubVirt.  However, these exploits are only theoretical and require either Intel’s or AMD’s latest processors that incorporate virtualization on the chip.  Experts are divided on whether or not these exploits can even work, let alone go undetected.  The developer of Blue Pill claims to have developed code for the exploit, but the code has never been posted anywhere for independent review.  SubVirt was a strictly theoretical exercise conducted by the University of Michigan and Microsoft software engineers and code was never developed.  Until a stand-alone hypervisor attack exists, the jury will be out on their viability.

So, what are the risks to hypervisors and virtual servers?  The good news for stand-alone hypervisors is that while there are exploits that create various denial of service conditions, none of these exploits will compromise the hypervisor.  You should be aware that there are exploits that target hypervisor applications or their host OS and then compromise the guest OSes.  There are also exploits that compromise guest OSes and then compromise the hypervisor and, potentially, any other guest OSes.  A lot of these exploits require that some form of guest OS to guest OS communication be implemented, so as long as that sort of communication is not configured, the guest  OSes will not cause the virtual environment to be compromised.  The important thing to remember is that there are still risks to virtualized environments even though those risks are relatively low compared to other operating environments.

Now that we have a background in virtualization of servers, what is the impact when we bring PCI into the picture?  For PCI purposes, virtualization just adds a layer of complexity.  Just remember the PCI “golden rule” of processing, storing or transmitting cardholder data and apply it to the added complexity of virtualization.  When virtual servers are processing, storing or transmitting cardholder data, the virtual server, its underlying hypervisor and SAN are all in scope.  That does not mean to say that all virtual servers on the same hypervisor in the same SAN are also in scope.  However, there are conditions that can bring those other environments into scope.

The most obvious way virtual servers are brought into scope is the same for any server.  When the network is not properly segmented, any PCI in scope server that is on a network segment with out of scope servers, those out of scope servers become in scope because they are on the same network segment.

The way SANs are brought into scope is when the PCI virtual server shares storage with other virtual servers that may or may not be in-scope.  This typically happens when the hypervisor boots from a common area in the SAN for the execution of standard server builds.  There is also the possibility that a common SAN area is used by more than one guest OS but, while this is possible, it is typically not a configuration you are going to see.

How are other virtual servers brought into scope for PCI?  This occurs when virtual servers executing in the same cluster are configured with guest communications enabled.  Guest communications allows servers to communicate between them without using traditional network or other server-to-server communications.  When a virtual server that is in scope for PCI has these communications enabled, all other guest systems attached to this virtual server are also in scope.

How do you keep virtual servers out of scope?

  • Make sure that you logically or physically isolate virtual servers that are in scope for PCI from those virtual servers that are out of scope.
  • Make sure that you have formal, documented controls that ensure that in scope and out of scope, virtual servers maintain their separation/segregation.
  • Do not implement any virtual server to virtual server communications in your in scope virtual environment.

In Scope – Example I

I thought I would write about a recent meeting I had with a client regarding whether or not their proposed point-of-sale (POS) solution was going to be in or out of scope.  Obviously, the hope was that it would be out of scope.

I would like to commend this organization for taking the high road to PCI compliance.  I wish there were more organizations like them.  After slugging through a very ugly PCI compliance process last year with a multitude of compensating controls to get them compliant, senior management set a goal of removing all cardholder data (CHD) from their point-of-sale (POS) systems over the next year and a half.  To accomplish this, they are going to implement a tokenization solution on their centralized transaction switch that they are developing in-house.  As part of this effort, the company is purchasing a new POS solution that is PABP compliant with the vendor stating that the next version will be PA-DSS compliant.  The discussion at my meeting was to determine if the POS solution would be in or out of scope.

Prior to the meeting, the client had provided me with some documentation describing their plans for the POS implementation.  I have taken some editorial license to protect the identity of the client, but here is the verbiage that I was provided regarding the configuration of the POS solution.

“The POS PC is connected to a separate credit card terminal by a USB connection.  There is a software client running that communicates with their centralized transaction hub that then submits transactions for authentication as well as handling settlement.  The only interaction between the PC and the terminal is to pass the amount of the transaction from the PC to the terminal and for the terminal to tell the PC whether or not the transaction was approved or declined.”

On the face of things, the solution appears to keep the POS solution out of scope.  However, this is a prime example of why documentation must be complete, clear and accurate.  Interpreting the documentation and lacking a detailed network diagram, I assumed that the credit card terminal and the PC had separate network connections.

As Paul Harvey was so famous for saying, “And now for the rest of the story.”  It turned out that the documentation had a slight flaw.  The person who wrote the majority of the documents had inconsistently used the terms ‘terminal’ and ‘POS’ through out.  A common mistake since most people that develop documentation are usually very close to the implementation process and make the assumption that everyone that reads the document has the same background with the topic in question.  As a result, it was not clear whether the terminal was the PC or the terminal was the credit card terminal.  What the meeting uncovered was that the client software that communicated with the centralized transaction server was in fact running on the PC, not the credit card terminal.  This meant that all communications to and from the credit card terminal were routed through the PC to the centralized transaction server.  That means that the POS solution is in scope.

The moral of the story is that a good QSA never assumes they know everything about a solution and why they seem to ask many questions about something you would think they would know.  The reason is that a good QSA knows that it is your organization’s implementation of the product, not another organization’s implementation and there will be differences.  And those differences could mean the difference between in scope and out of scope.


Wireless Security – Random Thoughts On How To Fix

This has possibly been the hardest post yet to write.  Mainly because I am at a loss for answers.  There just does not seem to be a lot of solutions out there to address real wireless attacks.  So, I have done my best to come up with some thoughts on how to conduct a wireless assessment that will provide some reasonable level of assurance that your network is not compromised.  Note, I said ‘reasonable’ as I do not think there is a way to get absolute assurance that your network cannot be compromised when wireless is involved.

  • Document the business reasons for implementing a wireless network.  Just because you can, does not always mean you should.  In a significant number of situations, you will find that the only reason for implementing wireless is just for the convenience it offers.  Does your organization really need wireless ‘guns’ that update inventory in real time or can you use guns that record inventory and then upload it in batch when the ‘gun’ is placed in a cradle?  In most situations, the cradle works just as well as the wireless solution.  That is not to say that there are not situations that warrant a wireless solution.  I have a number of clients that use wireless terminals and handhelds in innovative ways to improve customer service.  However, until there is a real business purpose with a real return on investment, do what you can to push back and not implement wireless. But be advised, since some vendors are now only producing wireless solutions, finding a hard wired alternative may not be possible.
  • Architect your wireless network to be secure from the start.  There are ways to do this that are not as onerous as you might think.  Primarily, it needs to be isolated away from the rest of your network.  The reason is that no matter the security you implement, wireless uses the public airwaves to transmit, the key word being ‘public’.  As a public network, attackers can eavesdrop on your wireless whenever they want and they can and will make attempts to crack your security all they want and there is nothing you can do to stop it.  Once your wireless network is isolated, treat it as the public network it is and implement firewalls, IDS/IPS and any other security measures on your wireless network segment.  Make sure that you create a consistent configuration so that you minimize the potential for introducing a mistake   One of the best methods is to use those centralized, managed wireless solutions versus individual wireless access points.
  • The PCI SSC needs to change requirement 11.1 to address the realities of the real world.  First, I question the usefulness of wireless scanning in the first place and I would highly recommend that it be dropped.  But assuming it is here to stay, for all but the very smallest of merchants, scanning with a wireless analyzer quarterly is a pipe dream.  I would recommend that quarterly testing is only a requirement when it is possible.  For all other merchants that wish to perform wireless testing with an analyzer I would recommend that requirement 11.1 suggest a sampling approach to ensure that all facilities are tested when significant network changes are implemented at the facility or at least once every three to four years.  Let us face facts here, there is no way Best Buy, Wal*Mart or Target are going to test their hundreds or thousands of stores on a quarterly basis.  It is just physically impossible.  They do not even conduct individual store financial audits that often, so who thought they would get wireless scans done that often?  Next, the PCI SSC has to provide in requirement 11.1 some additional alternative solutions besides an IDS/IPS on the wireless network segment.  Based on my experience, almost all of my clients that are using wireless are creating a compensating control to satisfy requirement 11.1.  It seems to me that if the majority of organizations with wireless are using a compensating control to meet the requirement, then the PCI SSC needs to create a requirement that does not require the majority of organizations to use a compensating control to satisfy the requirement.
  • If your organization has decided to use wireless scanning with an analyzer, admit that wireless scanning requires a technical expertise that your organization likely does not have.  This is a perfect project for a qualified network security consultant to perform.  The costs for such projects are easy to control as they are driven by the location and number of facilities you need scanned.  If your facilities are widely scattered, you may want to go with a consulting firm that better covers your locations so that you can minimize travel costs.  You can also control costs by using a consistent configuration for your wireless.  That way you can use a sample of facilities versus scanning every facility.  However, since building construction usually varies from location to location, that may require making sure that all your facilities are scanned within a one or two year period.
  • Don’t be buffaloed by a consultant’s certifications.  Customers are usually baffled by all the letters following a consultant’s name (even I have a boatload of letters after my name).  While certifications are good, it’s a consultant’s practical experience with security and wireless that counts.  Nine times out of ten, the consultant that meets with you will not be the one that does the work.  So, make sure that you and someone from your technical staff review the biographies of the consultants’ that will actually work on your project and that you personally talk to them either face-to-face or by phone.  Ask them about the wireless assessment engagements they have done.  Have them describe the process and make sure that it matches the process the sales person described.  Ask them about the typical findings that result from such projects and make sure that they can explain their findings to both technical and non-technical personnel.  And of course, make sure that you are not buying the process that I’ve discussed earlier.
  • Don’t buy supposedly sophisticated looking tools.  Regardless of whether you are doing it yourself or getting a consultant to assist, don’t buy based on tools.  A lot of people do good work with NetStumbler/Kismet, and the right wireless card.  Some of these tools are just expensive solutions using the same techniques as the person with shareware tools.  So when evaluating wireless security solutions, ask the vendor tough questions about how their solution discovers rogue access points and get them to address my earlier points on why wireless scanning is flawed.  In most situations, you will find that these vendors are offering a solution no better than the one you can get for free.  When talking to consultants, be wary of the consultant that talks about their tools and does not talk much about their process.  Consultants that talk ad-nauseam about their tools typically do not have the experience to deliver the results that you desire.  They are typically going to be no better than anyone else with a scanner.
  • Get a good understanding of the consultant’s process.  Ask the consultant to describe their wireless security assessment process.  Experienced consultants will have a number of service offerings in this area from basic scanning (essentially what I describe earlier but with a much more robust analysis of the results) to a full out wireless assessment that can resemble something out of a good spy movie.  Obviously, the more sophisticated it gets, the higher the cost.  However, for some clientele such as DoD contractors and the like, a very detailed and sophisticated analysis of all things wireless is what they require in order to satisfy contractual requirements.  For most merchants, what they need is something towards the lower end of the cost scale that will provide them with a reasonable assurance that their network is secure.  For most processors, their wireless assessment will likely be a bit more robust than a merchant’s because of the added risk they have due to the data they retain.

I have taken up a lot of bandwidth on this topic, possibly too much.  However, I think you start to see that wireless is not as simple a technology to secure as some of the security standards portray.    Wireless is not a technology that you just “add on” when you need it.  In the end, the most critical aspect to wireless is that it requires significant forethought before being added to a network.


PABP or PA-DSS Compliance Does Not Imply PCI DSS Compliance

I wrote about this at the ‘old’ Web site but apparently, it bears repeating after dealing with a number of clueless software vendors lately.  PABP or PA-DSS compliance does NOT, repeat, NOT mean that your customers are automatically PCI DSS compliant.  These standards complement one another, but they do not supersede each other.  The PABP and PA-DSS standards just mean that the certified software properly processes, stores or transmits cardholder data.

Some vendors seem to be very myopic regarding their responsibilities regarding their customer’s compliance with the PCI DSS.  They apparently have never read the PCI DSS requirements.  They believe that if they successfully go through the PABP/PA-DSS certification process, their responsibility is done.  All they have to do once their solution is certified is sell their software and say that they have their PABP/PA-DSS certification.  Wrong!

In some cases, I have to wonder how these vendors even got their PABP/PA-DSS compliance in the first place.  The reason I believe this is the difficulty I have had getting a hold of the required implementation guide.  I have run into instances where vendors have refused to provide an implementation guide or have even admitted that such a document does not exist.  Hello!  This is a requirement of the PABP and PA-DSS standard.  See page vii of the PA-DSS Requirements and Security Assessment Procedures, Software Vendors, third bullet point that states, “Creating a PA-DSS Implementation Guide, specific to each payment application, according to the requirements in this document.”  If that is not enough, on page ix under PA-DSS Implementation Guide and Appendix A are dedicated to nothing but what the implementation guide needs to document.  And just to add insult to injury because a lot of you vendors have so irritated me on this subject, PA-DSS requirements 1.1.4, 1.1.5, 2.1, 2.7, 3.1, 3.2, 4.2, 6.1, 6.2, 9.1,10.1, 11.2, 11.3, 12.1, 12.2, 13.1, 14.1 and 14.2 all reference the implementation guide and also reference PCI DSS requirements as well.  Similar language was in Visa USA’s PABP standard, so do not say that you were not aware of this requirement.

If vendors think their customers have no right to the implementation guide, think again.  On page ix of the PA-DSS Requirements and Security Assessment Procedures under Customers, bullet two states, “Configuring the payment application (where configuration options are provided) according to the PA-DSS Implementation Guide provided by the vendor” and reiterates under bullet three. “Configuring the payment application in a PCI DSS-compliant manner.”  Therefore, you vendors out there that believe your implementation guides are state secrets, you had better provide this document to your customers or your customers are going to be judged non-compliant with the PCI DSS because of your bullheadedness.

For all of the vendors that are providing implementation services to customers, on page viii of the PA-DSS Requirements and Security Assessment Procedures states under Resellers and Integrators at bullet two, “Configuring the payment application (where configuration options are provided) according to the PA-DSS Implementation Guide provided by the vendor” and at bullet three, “Configuring the payment application (or instructing the merchant to do so) in a PCI DSS-compliant manner.”  So, vendor’s consultants need to be following the implementation guide as well and they need to leave it behind for the customer to reference.

With that said, there are a number of PCI DSS requirements that your precious PABP/PA-DSS certification does not address.  The first is PCI DSS requirements related to encryption key management.  Yes, the PABP/PA-DSS covers this topic, but in the end, someone still has to mange these keys once the system is implemented at a customer.  In some cases, the customer is responsible for key management.  In other case, we have seen the vendor or another third party managing key management.  PCI DSS requirement 3.6 must be followed by whoever is responsible for encryption key management.  And if this is the responsibility of a third party, you had better expect to have the customer’s QSA investigate and assess how you are performing this requirement.

Then there is backup and recovery.  Requirement 3.4 requires that the PAN be encrypted even on backup media.  Unless your application software is also providing the backup and recovery functionality, I seriously doubt that you have any influence on this procedure.  So, you had better address this in your implementation guide.

And finally, so as not to be piling on the software vendors, there are PCI DSS requirements 6.1 (patching) and 6.2 (vulnerability identification).  For most vendors, they have little control over these, yet a majority of those vendors I have dealt with all claim that any questions regarding section 6 of the PCI DSS have been addressed by their PABP/PA-DSS certification.  Really?  I do not think that you even want to be responsible for these, but if you really do, you are more than welcome to them.  You are really only on the hook for 6.3 through 6.6.  And with requirement 6.6, one could argue responsibility is split between the vendor (code reviews) and customer (application firewall).

Thank you all for the soapbox.  I hope to get back on track soon.


The Shortcomings Of Wireless IDS/IPS

In my first post, I discussed the wireless analyzer approach to complying with requirement 11.1.  I documented where I think the current techniques fall short and give organizations a false sense of security.  In this post, I am going to give you what I think are the shortcomings of wireless intrusion detection/prevention systems.

Wireless IDS/IPS seen to break down into two types, ones that work like the wireless analyzer approach from my previous post on this subject and the ones that work like an IDS/IPS.  Let us discuss the analyzer IDS/IPS first.

From the analyzer IDS/IPS style products, I have had demonstrations of, most of these solutions work essentially the same way as the wireless analyzer methods I discussed in my last post.  These products typically work with wireless sensors connected to your network and a central server that also provides analysis of the wired network when a suspect rogue AP is discovered.  The wireless sensor is used as a wireless spectrum analyzer to locate potential rogue APs.  The idea being that multiple sensors can triangulate on the rogue AP and provide the location.  The ability of these sensors to accurately locate APs outside of a 15-foot radius can make things dicey and potentially expensive.  Therefore, for large facilities, you can expect to have to spend a lot on sensors for full protection.  For example, an average Wal*Mart is around 100,000 square feet.  In order to provide adequate coverage, the average Wal*Mart store would require approximately 445 sensors.

On the wired side of things, these analyzer IDS/IPS solutions along with their exclusively wired solutions are looking for rogue network traffic, ICMP response, MAC address and/or SNMP information that indicates the device is a rogue AP.  In the end, they sound sophisticated, but they still rely on the fact that the rogue access point is configured to be discovered.

Attackers know how these solutions operate and configure their rogue APs to deter or even avoid these identification techniques.  As a result, these more sophisticated solutions are also blind to the truly rogue AP.

In addition to these obvious issues, false positives can be quite a problem for those solutions that conduct monitoring of the wireless spectrum.  This is particularly true in situations where APs are added on a regular basis outside of your facilities.  And with wireless becoming more and more common, that can keep your security team quite occupied while they sort through the false positives to find the real potential threats.

And then there is the whole issue of 802.11 devices being the only source of compromise.  If an attacker is going to go to the length of compromising your network, why would they not use cellular technology and avoid 802.11 all together?  With 3G cellular networking all the rage, the speed of these cellular solutions are no longer a limiting factor.  None of these solutions truly addresses the cellular issue, so there is still a vulnerability.  Unfortunately, the security vendors, PCI SSC and card brands seem to only react to incidents, not think ahead.  So, until a breach occurs involving cellular, we will likely not see anything to address this risk.

And what about other forms of wireless such as Bluetooth and satellite?  Before you write them off as either not having any transmission distance or being too complicated and expensive, it is that short sidedness that will get you in trouble.  Believe it or not, there are Bluetooth USB adapters that have ranges of up to 350’.  In addition, pairing and security codes are well documented by vendors so attaching to any Bluetooth device is an easy proposition.  Bluetooth can be used to load malware on a system and begin the compromise process.  If you think satellite is the last safe wireless solution, at this year’s Black Hat, Adam Laurie discussed not just hacking satellite TV but also data transmissions.

In the end, the important thing to remember is that the public airwaves are just that – ‘public’.  And you must treat them as public or you will get burned.

In a future post, I will discuss my thoughts on how I think the PCI DSS should address these shortcomings.

June 2009