Meaningful Security Awareness Training

Is there really such a thing or am I dreaming?

For any of you that are involved in security awareness efforts, you know what I am talking about.  Security awareness training efforts just do not seem to catch fire or interest anyone, even yourself.  So what is one to do?  The reason I bring this topic up is that I attended an FS-ISAC Webinar where Lance Spitzner of SANS offered a very interesting idea in regards to security awareness training.

Before I get too deep into security awareness training, a bit of background.  Mr. Spitzner pointed out in his presentation that human beings are very bad at judging risk.  Worse yet, since most people have a poor understanding of the Internet and technology, that risk judgment gets even worse as most people, even IT professionals, just do not seem to get the risks presented by the Internet.  He presented numerous examples of just how bad people are at risk evaluation without information or a frame of reference.

If you think this is a fallacy, go and read peoples’ Facebook and LinkedIn pages.  The amount of proprietary and sensitive information that can be gleaned off of social media pages is terrifying.  But it is not just social media that creates problems.  Organizations do it to themselves all of the time with their own Web sites.  In the name of openness and marketing savvy, organizations post amazing amounts of proprietary and sensitive information on their own Web sites.  The ready availability of all of this proprietary and sensitive information lends itself to making a successful social engineering attack a foregone conclusion.

As I like to point out in my social engineering presentations, no one walks down the street indiscriminately handing out their business cards to everyone they encounter.  The looks I get from that statement are truly amazing.  Most of the people in the room do not get it and, unfortunately, probably never will get it until they are breached.  However those that do get it usually turn pale as it is usually the first time they realized the seriousness of the situation and the amount of risk to their organization all of that information presents.  When you are on the Web, organizations and people post their life stories for anyone and everyone to see.  Is it any wonder why the DEF CON “How Strong Is Your Schmooze” social engineering contest is so successful?

I have thought about this situation a lot lately as social engineering becomes more and more one of the tools used to initiate a breach.  I keep coming back to what can we do to minimize this disaster?  Security awareness training just does not seem to be working and then I run across Lance Spitzner’s presentation to FS-ISAC.  Mr. Spitzner states that people are just like any other device in the mix.  He points out that Microsoft patches Windows every month on ‘Patch Tuesday’, so why are we not “patching” people at least monthly.  What an idea.

Mr. Spitzer stated me to thinking about people as running the human operating system, or hOS as I prefer to now call it (my apologies to Apple).  And all of a sudden, viewing people as devices running hOS clarifies security awareness training.  Now, I know there are some that will chastise me for dehumanizing people.  But I justify my doing that in the fact that it needed to happen so that we can better understand and educate people so they are less at risk.

If you accept the premise that people should be treated no differently than any other device, then you know that security awareness training on hiring or an annual basis is just not often enough.  Microsoft issues patches to their software monthly if not more often in emergencies.  Since human beings are terrible at judging risk, then how can only annual security awareness training address the issues in hOS?  And that is just it, annual training cannot.  hOS needs to be regularly “patched” just like every other operating system.  That means at least a monthly patching cycle.  But doing monthly “patching” is not just all you have to do for hOS, you also need to make the “patching” relevant to hOS.

This is where subscribing to news and information security RSS feeds or listservs to track what the hot issues are for hOS.  Just like Microsoft focuses their patching of Windows on the most dangerous vulnerabilities, hOS vulnerabilities also need to be tracked the same as technological vulnerabilities.

By decoupling people from the equation and looking at the security awareness problem as patching another OS, it liberates you to think in new ways.  So, what are the steps in patching an OS?

1. Identify the vulnerability.
2. Determine the risk of the vulnerability.
3. Determine what can be done to mitigate or remediate the vulnerability.
4. Develop a program to mitigate or remediate the vulnerability.
5. Test the mitigation or remediation of the vulnerability.
6. Implement the mitigation or remediation of the vulnerability.

Now apply those principles to hOS on a monthly basis.  Taking the results of your research you can surely come up with a vulnerability or two that has occurred whether it is a suspicious email with a PDF or spreadsheet attachment or a drive-by attack.  Take those as examples, explain their risk, explain how to recognize these attacks, and then put your marketing department to work on developing a message that trains people.  There is a reason I recommend the marketing department and that is because you can share this information with your customers as well as your employees.

Send out these messages on a regular basis, preferably monthly but definitely more often than annually.  Follow up those messages with ‘lunch and learn’ or similar sessions to further discuss those messages and to ask attendees if they have any security questions or have encountered any threats at work or even at home.  These steps should show your customers and fellow employees that security does not require knowledge of technology so much as just using common sense and being skeptical.

Security awareness does not need to be conducted like a TV sitcom, but it can work if you take the approach that you are patching another device and you make that training relevant.

Why vSkimmer Should Not Matter

It was announced this week by McAfee that a new threat to merchants has been discovered called vSkimmer.  This is a very insidious threat as most merchants will likely not know they have been infected until it is too late.

The net of vSkimmer is that it is malware of the highest order built for the explicit purpose of collecting track 2 data from Windows point of sale (POS) systems.  Worse yet, whoever wrote this little gem of software intends to enhance it in 2013 to include the ability to skim EMV cards’ “track” data as well.

vSkimmer can be deployed like Stuxnet through a USB thumb drive, as malware in an email message or on a Web site or any number of ways.  When installed, vSkimmer determines the operating system and version, hostname, active username and various other operational characteristics of the POS system.  It then inventories running tasks and memory to determine where track 2 data is stored and begins recording that data.

vSkimmer works whether the POS system is connected to the Internet or not.  When the POS is connected to the Internet, it transmits the data obtained to a control server using HTTP.  When the POS is not connected to the Internet, the information is stored until someone connects a USB device labeled ‘KARTOXA007’ and copies all the information it obtained onto the USB device.

As usual, the Internet is abuzz regarding how this will be addressed by the PCI DSS.  Sorry to disappoint, but it is already addressed.  Here are some key requirements in the PCI DSS that should mitigate vSkimmer.

• Requirement 1.2.1.a requires that only that network traffic that is necessary is allowed through the firewall.  Merchants should be only allowing connectivity from the POS or card terminal to their processor and nowhere else.  Any traffic attempting to go anywhere else should be flagged and IT alerted to investigate.
• Requirement 5.1 requires that you have anti-virus and anti-malware software installed on POS devices.  Given this is a Windows specific threat and Windows is highly susceptible to being infected, you should have done this already.  While anti-virus solutions are not perfect in always identifying such malware, since McAfee and other anti-virus solution vendors are the ones that found vSkimmer, I would imagine that they all have or will very soon have signatures for vSkimmer.
• Requirement 6.1 requires that systems are patched current.  The problem with patching POS systems is that a lot of vendors issue POS updates for the OS and their application on a quarterly or even annual basis and do not recommend that merchants patch their POS systems directly from Microsoft because of compatibility issues.
• Requirement 10.2 which requires the logging of events.  In the case of a USB device being plugged into the POS system, at a minimum you should see that the portable device enumerator service going active when a USB device is plugged in and if the device is new, you should see system event log entries regarding the loading of device drivers to support the USB device.  None of these actions should be seen in your log data, so if you monitor for these events, you will know that USB devices are potentially being plugged into your POS systems.
• Requirement 10.5.5 requires the use of file integrity monitoring which would catch the installation of vSkimmer as a foreign piece of software even though it masks itself as ‘svchost.exe’.  This would provide a backup control for requirement 5.1 if vSkimmer changes its approach as to its file name and is not caught by the anti-virus solution.

In addition to the PCI requirements, you can do the following to increase your security in regards to vSkimmer.

• Do not allow USB devices to be connected to your POS systems.  Most card terminals are RS232 devices, but USB is becoming more common.  The Windows Group Policy function can be used to disable USB ports on Windows systems.  There are also third party solutions that will disable USB ports.  A lot of these third party solutions can offer additional granularity in what types of USB devices can be connected.  This can be very advantageous when you are using USB card terminals which still need to connect, but other USB devices should not be allowed.  One of my more imaginative clients hot glues the ports shut on their POS systems.
• Train your staff on the vSkimmer threat.  Explain how it works and what they can do to minimize this threat such as not allowing anyone to manipulate the POS systems other than employees responsible for the care and maintenance of the systems.
• Lock your POS systems in a sealed cabinet or cage and only allow the manager on duty to have the key.  This may also involve additional security on POS servers if those are also used by your POS solution.
• Periodically review video of your POS stations to determine if cashiers or other personnel appear to be manipulating the POS system.

If you adopt all of these measures, you will significantly reduce the threat presented by vSkimmer and will likely never encounter it.

Have You Been Paying Attention?

Mandiant recently released an interesting report on APT1, the supposed Chinese Army hackers that are behind most of the serious digital attacks being conducted these days.  The reason I bring this up is that, even before the Mandiant report, I had been getting questions regarding how the PCI DSS, ISO 27K, HIPAA and other standards could possibly block or even alert an organization that they are under such an attack or have been compromised by advanced persistent threat (APT).  I will use the APT acronym in this post to indicate any sophisticated attacker, not just those that follow the methods used by APT to compromise a network.

It fascinates me how many organizations jump through hoops to tightly control and monitor their inbound network traffic from the Internet, business partners and any other untrusted network.  Yet the same organization does nothing about the outbound side of the equation.  It is the age old belief that all internal traffic can be trusted because it is from the inside of the network.

It is this trust factor that APT relies upon in their ability to compromise your network.  Because you have tightly controlled your inbound traffic, APT relies on social engineering to gain access to the internal network.  Once inside, since most organizations do little to nothing to control or monitor their outbound traffic.  It is therefore a relatively simple task to connect from the inside of a network to any external IP address and set up a communications link.

Even when organizations are controlling and managing their outbound traffic, it is still easy to connect using ports 80 (HTTP) or 443 (HTTPS).  In this day and age, it is virtually impossible to be in business without allowing ports 80 and 443 outbound.  Attackers know this and use this fact against a target in order to gain a way out.

So what can you do to minimize this risk?

• Tighten you inbound traffic from the Internet and business partners.  I consistently find that organizations allow traffic from countries where they have no business relationships.  As a result, these IP addresses can be used to launch denial of service and hack attacks.  If you do not limit where inbound traffic can come from, you are only asking for a denial of service attack or worse.
• Control what external IP addresses and URLs your internal network can connect.  While a lot of organizations today use overseas business partners, is that any reason to let your internal network connect to any IP address?  No.  This is typically set up by a network engineer that does not want to be bothered by a lot of service requests to allow IP addresses or URLs.  However today’s sophisticated network attacks are only stopped if you control what IP addresses or URLs your users can connect.  Yes, this potentially creates a “pain in the rear” issue when someone needs to access something you have blocked.  But would you rather stop a breach or let it go unchecked until you accidentally trip over it?
• Control your outbound ports with the same exuberance that you use on your inbound traffic.  If all you need from your internal network outbound are ports 80 and 443, then those should be the only ports allowed out on your firewall.  Enough said.
• Monitor your outbound network traffic for “anomalies.”  The biggest anomaly you should be looking for is encrypted traffic over any port other than 443 (HTTPS).  The reason is that any application using anything other than 443 for encrypted communications is likely an attacker trying to obfuscate the fact that they have compromised your network.  After all, your firewall and IDS/IPS cannot read the packets encapsulated in the encrypted stream, so the attacker effectively blinds them to what is actually going on.  Encrypted traffic that does use 443 hopefully will get blocked by your white or black list of IP addresses and URLs.
• Analyze the log data generated from your firewalls and IDS/IPS looking for potential APT attacks.  Are there consistent indications that devices are trying to connect to blocked IP addresses or URLs?  Do those incidents occur after hours or when the user is not logged on?  These are examples of conditions that should be flagged and investigated to ensure that something bad is not occurring.

Remember, security is not perfect.  What these controls do is make it more difficult for APT to successfully compromise your network.  However, if APT puts their mind to it, there are likely ways to successfully compromise your network but it will take a significant amount of work.  In the end, I would like to assume that since there are many more targets easier to get at, APT will go after the easy targets and leave your organization alone, at least for the time being.

Security And Compliance

I have written a lot about this topic over the years and was recently reviewing my Compliance Is Not Security – Busted! post and the comments that came in regarding it.

A theme of a number of the comments was that compliance does not equal security.  DUH!  I have never once said or even implied that compliance equaled security as – yes, here it comes – security is not perfect!  However, if you are complying with any security program/framework such as the PCI DSS, ISO 27K, etc., then you are likely more secure than those who are not.

Security technology such as firewalls, routers, servers, applications, etc. can all be set up with rules that are complied with 100% of the time, day in and day out, no exceptions.  The problem comes down to people who are fallible.  Their compliance is never 100% and you are probably lucky to have anyone above 90%, no matter how much security awareness training you do.  As a result, in organizations that are truly complying with the PCI standards, this is where the security breach starts, with people for one reason or another.

No, I am not necessarily talking about social engineering, although social engineering is growing because of the fact that organizations have invested a lot in security technologies yet people are fallible.  People can be the root cause because of any or all of the following.

• How dare you do that to me!  This is the most obvious of the people issues that comes to mind.  Face it, when backed into a corner, people lash out just like a trapped animal.  The supposedly wronged party wants their proverbial “pound of flesh.”  They get that pound of flesh by hurting the organization that has just hurt them.  This can be as minimal as taking office supplies to downloading databases to a USB drive as they empty their desk.  Obviously, a database, network or system administrator’s access is much different than a clerk’s.  However, if your security is minimal on the inside as it is in most organizations, the clerk may actually have better access than the administrators when it comes to sensitive information.  Such a situation may not be the fault of the administrators, that old version of POS or ERP may not have the ability to be more granular regarding access to information.
• Over inundated with alerts and cannot identify real alerts from false positives.  This typically occurs when an automated tool is implemented but never tuned to the organization’s environment.  In this sort of an environment, finding real alerts can be like finding a needle in a haystack when there are thousands of alerts an hour scrolling by on the screen.  This usually makes management wonder why the tool was needed in the first place.
• Saw an alert and ignored it.  We see this most often coupled with the aforementioned inundation issue.  The other most common version of this issue is with internally used SSL certificates that were generated incorrectly or use a default certificate supplied by the application.  Users then see the “There is a problem with this Website’s security certificate” or similar error message in their browser whenever these flawed certificates are encountered and become conditioned to ignore the error message.  Over time, they become conditioned to ignore all of these sorts of messages, including those for malware infected Web sites and, surprise, you have been compromised.  I have lost count how many people have said to me, “We just ignore those alerts because we know they are false positives.”
• Saw the alert but got side tracked and never came back to it.  This is a problem we see all of the time.  For example, the person that monitors the network is also the person that manages the network and configures the network.  An alert comes in and the person begins a root cause analysis (RCA) only to get pulled away because a remote facility is offline.  The offline issue gets resolved, but other issues come up as well as meetings and telephone calls and the person never gets back to the RCA for the alert because there is no “tickler” to remind them to go back and complete the RCA.  In the meantime, the attacker has gained their beachhead and is probing the network for whatever value it may contain.
• Just did not put together all of the pieces to know they were compromised.  Like the reasons 9/11 occurred, most organizations do not correlate all of the potential incidents occurring in their networks and therefore do not understand that there is an active effort to compromise their network or that they have already been compromised until well after the incident has caused damage.  The reason this is important is that once an attacker is inside your organization’s security perimeter, it is typically game over because there are few controls to prevent access and identify that data is being taken.

If you have read the Verizon Business Services Data Breach Investigations Reports (DBIR) over the years you know how the bulk of attacks get inside, they are the result of people.  For the last two years, the DBIR has used the VERIS Event Threat Grid to show how breaches occur.  Across the top of the grid are the categories; Malware, Hacking, Social, Misuse, Physical, Error and Environmental.  The Social, Misuse and Error categories imply mistakes or deliberate acts of people.  If you read the definitions on the VERIS Web site, Malware is also very people centric as is hacking.  Surprisingly to some will be that the Physical and Environmental categories also have a good number of people errors.  Based on just a quick read, it looks to be that about 60% to even 70% of all of the incidents categorized by VERIS has some form of people error component.

Since we are not going to get rid of people in our organizations any time soon, what are you to do?

• Admit that people are the problem and focus your security measures accordingly.  Every 12 step program says the first step is to admit the problem which, in this case, is that people are fallible.  As a result, we need to construct our security measures such that this fallibility is minimized as much as possible.  One of the best solutions is to integrate alerts into your help desk or change management system so that a ticket is generated.  Those tickets need to have an escalation process behind them so that if they are not investigated within a period of time, they are bumped up to the next higher rung of management and that escalation continues until the tickets are finally addressed.  This way there is visibility for the alerts should they slip through the cracks.  As a side benefit of this approach, you gain statistics to reinforce why you need more staff and/or more/better tools.
• Strengthen your internal security measures.  As things stand, once inside most organization’s security perimeter, there is very little that stands in the way of an experienced attacker getting the data they desire.  Regardless of whether it is an insider attack or an attacker has managed to get inside, there is already justification for organizations to beef up their internal security measures.  To address this problem, I would recommend the security architectures as documented in my Fort Knox approach, Forrester’s Zero Trust Model or McGladrey’s Ultra Secure Network.  But most organizations do not have the infrastructure architecture, the application architecture or even the will to take such approaches.  But that does not excuse an organization from just saying they cannot do anything.  If anything, most organizations could vastly improve the monitoring they do on their internal networks.  Monitoring needs to be coupled with reducing the total number of ports that are open between network segments.  Most internal networks do a terrible job of this because of a variety of factors including applications people that cannot tell what ports need to be open to avoiding operational issues by just leaving things open.  Another area of improvement is reviewing user access rights on all systems and applications, not just those in-scope for PCI compliance.
• Constantly tune your alerting system(s).  Just as attack methods are not static, neither are networks, systems and applications.  Changes are occurring all of the time in an organization’s IT environment, yet if you ask the people running the SIEM about changes, nine times out of ten, nothing seems to be changing other than requests to look for a new signature or anomaly.  There is a belief in the SIEM user community that a SIEM’s update process is making the necessary changes in the policies that ship with the SIEM.  To a certain extent SIEM solutions are similar to anti-virus and malware solutions.  However, because a SIEM monitors log data and the log data provided varies greatly from organization to organization, each organization needs to periodically review and adjust their alerting criteria to make sure that it reflects the organization’s operating environment and not just some template from the SIEM vendor.  If an organization is not reviewing its SIEM alerting rules based on the changes made, at least quarterly, then it is highly likely that the SIEM is not alerting properly.
• Establish separate consoles from your SIEM for network, system, security and application administrators.  What a network administrator is looking for is vastly different from what an application administrator is looking for and what any particular group might be looking for to generate an alert.  As a result, to have only one console is really silly and non-productive.  Yet time and again, we see SIEM implementations with just that, one console and everyone being driven by email or SMS alerts.  The people alerted then have to get to the SIEM to find out what exactly triggered the alert and then determine what to do about it.  Having your own console view simplified things by only listing that viewer’s alerts and no one else’s alerts.  This allows people to focus on their problems and not the whole organizations problems.  The idea behind the single console is that if everyone knows what is going on overall, then correlation would occur because everyone sees everything.  While you would think that would be the case, in reality, people just want to fix their problem and move on, not the entire organization.  Which leads to my last point.
• Watch the overall alerting picture so that correlations can be made.  According to most sources, today’s attacks are becoming more sophisticated and multi-pronged in their approach.  For example, while most DDoS attacks are just to be a pain in the posterior to the target and disrupt access to the target’s Web site, there are those DDoS attacks that are used as cover so that people inside are blinded to the real attack(s).  Whether or not the DDoS was a decoy depends on what other events or incidents occurred during the DDoS attack, if your alerting system did its work.  Higher end SIEM solutions can provide basic correlation rules, but most SIEM solutions require the end user to develop those correlation rules.  It is these correlation rules that help organization identify these more sophisticated attacks.  That said, these correlation rules do not have to be very sophisticated.  For example, during a DDoS attack, you really only need to look for malware attacks, failed authentication attempts and other anomalies that would be likely indicators of the DDoS attack being used to mask the real attack.

Is all of this going to address your security issues?  Sorry, not a chance.  None of the above stops all breaches, it merely minimizes the possibility that a breach goes on for months or years.  Hopefully it minimizes a breach down to weeks, days, maybe even hours in some cases but it will never totally eliminate them.  Security is not perfect.

There is a side benefit to all of this and that is it will assist you in doing RCA.  RCA is very effective in getting rid of those nagging operation issues that occur from time to time and mess up the delivery of your organization’s goods and services.  All of the information you collect for security purposes can also be used to find the needle in the haystack that is causing a database to corrupt, a network connection to drop or a server to fail because now you have information as to what was going on that led up to the problem.

The reason an organization is not secure is that there are so many areas of improvement needed that the full control triad is no longer functioning and holes exist that will allow an attacker to operate without the knowledge of the organization.  Until the controls are implemented and operating properly, it will be impossible to determine if they are secure or not.  The recommendations I have made will hopefully give you a better picture of what you face and reacting to issues that need attention before your organization is the next one to be breached.

What To Focus On In 2013

It is the end of the year and, like all other pundits, here is another idea on what 2013 will bring in the way of security issues.  After reading a lot of the other predictions out there, I tend to agree with those from Verizon Business Services’ Data Breach Investigation Report researchers.  While everyone else is predicting cyber-Armageddon as the biggest threat, the researchers at Verizon Business Services see a lot more of the same for 2013.

The biggest threat Verizon identifies is more attacks on authentication systems.  This is most likely because your vendor or your developers talked you into storing authentication information in your database that is Internet facing.  We see this all of the time with eCommerce and Internet banking solutions.  The external user credentials end up being stored in the database along with order entry, inventory and pricing data.  This is typically done because using a directory system for such purposes is difficult and, at times, not as functional as when authentication data is stored in and used from a database.  Given the prevalence of SQL attacks, all of that information results in being available for the taking through a SQL injection attack.  As a result, the attackers compromise the authentication system, gain access to everyone’s credentials, including administrators, and it is likely ‘game over’ regarding the rest of your security measures.

I want to touch next on social engineering because it is typically directly related to compromising authentication systems even though the second place attack method most concerning Verizon researchers is application attacks.  Social engineering is all about tricking your end users into giving up key information so that an attacker can compromise your environment.  The most common piece of information an attacker tries to obtain is an end user’s credentials for logging onto the network.  Hence the reason why I wanted to discuss this after the authentication system attacks.  Social engineering is the most insidious of attack methods because it does not involve any of an organization’s security technology.  And worst of all, if social engineering is successful, all or most of your organization’s security technology is effectively neutralized as a result.  That is because most organizations have little or no security once someone is on the inside.

Now let us look at the second most concerning attack to Verizon which is application attacks.  Verizon is saying that application attacks are more of a threat to governments and large applications.  Regardless of the target, any organization with an application presence on the Internet is a potential target such as with eCommerce or Internet banking.  A lot of these applications are based on on-line frameworks such as IBM’s Websphere or Oracle’s Application Framework.  It is not that these frameworks are insecure, it is that they still require development effort and it is those custom development efforts that do not guarantee a secure application.  The problem comes from the fact that a lot of people believe that using a framework means that little to no security testing even though the amount of custom development done in these frameworks can be more extensive than starting from scratch.  As a result, we see a lot of organizations tossing Internet applications into production with little or no security testing and then ending up with breaches as a result.

In addition, there are third party applications served up by application service providers (ASP).  A lot of small and mid-sized businesses (SMB) use these sorts of applications to have an online presence.  As a result, a lot of SMBs believe that these solutions do not require any security testing because the vendor and the ASP do that for them.  However, we are encountering more and more attacks on SMBs, particularly those that have wealthy clientele such as country clubs and exclusive financial institutions because their applications are notorious for not being secure.  SMBs are constantly amazed that; (1) they were targeted and, (2) the application was not bettered secured.  Yet attackers know that while the take out of SMBs could be significantly less than a large organization, an SMB is usually easier to compromise because they do not have the security and monitoring resources of a large organization.  As a result, SMBs are becoming a larger and larger target for attackers.

Finally, something that concerns me as the previously discussed threats is mobile devices and devices not under the organization’s control, also known as ‘bring your own device’ or BYOD.  I think these devices will surpass the other three threats over the next few years because most organizations have difficulty maintaining security on their servers, desktops and notebooks, let alone something like an iPhone or an Android tablet.  The worst thing about mobile devices is that they are so easily lost and it fascinates me how many people lose their mobile devices.  The bottom line about mobile devices and BYOD is that you must be very, very careful as to what you allow these devices to access and how you grant that access.  And you must make sure that these devices are not allowed to download information, even if encrypted, as that information is highly likely to be lost.

So what should you be doing regarding these threats?  Here are my top things organizations should be doing to minimize the risks presented by these threats.

• Trust no one.  This is particularly true of mobile devices or BYODs, but it also applies to your own internal systems.  Forrester has promoted this in their ‘Zero Trust Model’ as well as me in the ‘Fort Knox Approach’.  This is not as easy as one might think, but the approach makes sense in these days of attacks on authentication systems and social engineering approaches.
• Classify your data.  This is usually a difficult project, but they pay dividends at the end because everyone understands why certain information cannot be allowed out of the control of the organization.  It also allows people to justify to others why data cannot be allowed to be accessed by people that do not need access as well as via mobile or BYOD.
• Require encryption on mobile devices and BYOD.  Even if you do not allow data to end up on these devices, you do not even want the memory or other information that might be inadvertently stored on these devices out of your control.  As a result, if you encrypt these devices, there is a high likelihood that, if they are lost, the person finding them will just wipe them and start over.
• When possible, use a directory system for authentication.  This is always painful for systems that operate outside of the traditional control environment of internal users.  Directory systems are usually designed to be more secure than any sort of database authentication system because they are assumed to be at risk from the start.  However, just because they are designed to be secure does not mean they cannot be implemented in an insecure manner.  Windows Active Directory takes a lot of heat for being insecure; however a lot of that heat is due to silly implementations to support insecure authentication methods for compatibility.
• Conduct security awareness training.  The only thing that minimizes social engineering is consistent and regular security awareness training.  However, do not kid yourself or management.  Everyone has their ‘moments’ and does something they should not.  That said there are always those that just never seem to get it which is why you need other controls and monitoring to ensure you maintain your security.  However, to just throw up your hands and say it is pointless is also not a position to take.
• Secure your applications.  This means conducting application code reviews and testing applications before they are put into production not after the fact.  Unlike networks where you need to put them into production before testing them, applications can be tested before going into production.  It amazes me how many organizations put their applications into production and, by the time they finally get around to testing them, they have already been compromised.  And while automated application code testing solutions are all of the rage, we still find that the best results come from the more traditional human code review not automated tools.
• Monitor your network and applications.  This is a double edge sword.  You know what to look for, however, you have so many ports open that it is near to impossible to recognize bad traffic from good traffic.  And it is not necessarily the fault of your IT department as most packaged applications require an inordinate amount of ports open to function properly.  However, the key thing to monitor, more than anything, is any traffic going outside of your network to an unknown location.  When you see traffic going to Eastern Europe, China or any unexpected IP address, your monitoring system should generate an alert as that is typically a key indicator that you have been compromised.

Have a happy holiday season and I will “see” you all next year.

2012 Database Threats

I attended a Webinar recently put on by Application Security Inc. regarding the threats to databases for the coming year.  If you did not attend it, you missed a good session.  But the most disturbing thing brought up was their top 10 list database vulnerabilities and misconfigurations.  Their top 10 list is:

1. Default or weak passwords
2. SQL injection
3. Excessive user and group privileges
4. Unnecessary DBMS features enabled
5. Broken configuration management
6. Buffer overflows
7. Privilege escalation
8. Denial of Service
9. Unpatched RDBMS
10. Unencrypted data

If you look at my post a while back on the 2011 Verizon Business Services’ reasons for why organizations were breached, there is a great correlation between Verizon’s report and what Application Security Inc. is saying.

Their first point about weak or default passwords is very clear and should not need to be discussed.  In this day and age, we should all be ashamed that this is even on the list, let alone the first item on the list.  The bottom line here is that, if you use default or weak passwords, you deserve to be breached.

They brought up and interesting point about SQL injection attacks that a lot of organizations do miss or underestimate and that is the internal SQL injection.  Most organizations are so focused on the external threat that they forget about the threat from the inside.  Worse yet, most security professionals and DBAs are unaware of the threat SQL injection poses even without the Web.  Since most of today’s attacks are perpetrated once past the perimeter, the protection from the inside attack is very relevant and very important.  Because once an attacker is on the inside, it is relatively trivial to use SQL injection or other techniques to obtain data.  More and more organizations are beginning to understand the insider threat and are firewalling all of their database servers away from the general user community as well as minimizing the number of users that have direct SQL access to those servers.

Excessive privileges cannot always be addressed at the DBMS level.  In today’s packaged software world, a lot of the rights are managed and maintained at the application level and that security matrix is maintained in a database table.  The granularity that can be granted is usually where things go awry because the application’s security system only provides an “all or nothing” approach.  Application vendors are getting better with this because of SOX, HIPAA, PCI and the like.  However, organizations typically need to be on the most current releases to have access to such enhanced security granularity.  Unfortunately, there are very few organizations that can afford the most current release or can implement the most current release due to their extensive modifications.  The simplest way to address this issue is the periodic reviews of database privileges and minimizing those users that have excessive privileges.  In the longer term, I expect we’ll see the return of the data dictionary with the addition of user rights and roles that will manage this problem.

Unnecessary features enabled are a vendor and DBA issue.  In some cases, vendors make changing features impossible or near to impossible once the RDBMS is installed.  In some cases, there are physical reasons as to why a feature must be enabled at installation.  However, there are also instances where features could be enabled or even disabled at anytime, but because the vendor only wanted to do that at installation, that is the only time you can deal with the feature.  This results in a lot of DBAs installing the RDBMS with every feature/function available, whether it is needed or not, just in case they might need it later on.  Do not get me wrong as I understand the drivers for this practice.  In today’s “I needed it yesterday” world, it is tough to be responsive when something will require an entire re-install of the RDBMS and migration of existing data in order to get something done.  It is time for IT people as a whole to start explaining to non-IT people that there are just some tasks that take time to do properly no matter how quickly anyone needs them completed.  Our infrastructure has become susceptible to attack in large part because of this rapid response desire.  If we intend to make things secure, we need to stop and think things through before creating even larger issues.

The pervious issue feeds directly into the next; broken configuration management.  Configuration management is broken because the vendors make it virtually impossible not to break it.  And even when configuration and changing configuration is easy and manageable, DBAs are not always as structured as other IT operational disciplines.  As a result, whether talking about the configuration of the RBDMS or the client that loads on the workstation, configurations are too broad because of the “just in case” factor.  I know it is a pain to only enable what needs to be enabled and then six months later have to reinstall everything just for a particular feature, but that is the correct way to do things if you intend to be secure.

Buffer overflows, privilege escalations and denial of service are all common vulnerabilities that organizations will have differing levels of success in mitigating.  I will tackle the easiest to address first, privilege escalation.  If there is any area where security can always be addressed it is with privilege escalation.  The reason privilege escalation exists is because someone, usually a developer, created the issue because they decided to allow users to perform a task that the user should not be allowed to perform.  Because if they were allowed to perform the function, then they would not need their privileges escalated to perform it.  The easiest thing to do is to disable those functions that require privilege escalation.  However, in some cases, that approach will create operational issues that will be unacceptable.  In those cases, monitor the daylights out of things so that you can be sure that the privilege escalation did not result in a different outcome.

In a lot of cases, there can be little done to address a denial of service (DoS) attack short of blocking the offender(s).  Denial of service does not compromise information; it just makes the information stored in the database unavailable.  And for most organizations, that is an important distinction.  If no information has been or can be compromised, then DoS is an annoyance and should be treated as such.  However, some DoS attacks can be used to defeat security measures in the RDBMS by causing the RDBMS to fallback to a basic operational state.  It is in these situations that one has to be careful because information can be lost.  The easy fix is to put a firewall in front of the database and enable DoS attack protections.

Buffer overflows are the most insidious attacks because, in some cases, there is little that can be done to stop them.  A lot of security professionals make the success of buffer overflow attacks sound like they are all the result of sloppy coding practices.  And while there is some truth to that view, the amount of which depends on the skills of your programmers, the success of buffer overflow attacks is also the result of embedding too much flexibility into our applications and leveraging the capabilities of the RDBMS.  In today’s world of open constructs such as SQL and RegEx, we have effectively made everyone a potential database programmer all in the sake of expediency.  Yes, customer service is highly improved, but at what cost?  Web application firewalls can minimize buffer overflows by “learning” how your SQL calls are typically structured, but they are not a complete answer nor do they completely remove the risks.  The way to fix the problem is to reduce functionality and make applications more complicated and difficult to use.  For most organizations that is not an option.  As a result, we must minimize the risks but be willing to accept the risks that remain as a result of our desire for ease of use and flexibility.  Minimizing the risk may mean implementing that Web application firewall internally as well as externally.

While I was glad to see that unpatched RDBMS software low on the top 10 list, I was very disappointed that it was still in the top 10.  One would think with all of the discussions about the importance of patching software, this would not occur in the top 10.  I understand the issues of compatibility and testing that make patching difficult, but really?  Maybe you need to invest in more than one or two instances of the RDBMS.  This is the cost of doing business the correct way.  If you are not doing things the correct way, then do not complain when you have a breach.  So while you saved yourself money on licensing costs on the front end, you likely paid for that cost savings a hundredfold on the back end.

I also understand the issues and fears with encryption.  For a lot of people, encryption is this mystical science that only certain “geeks” practice and practice well.  For others, the problem with encryption is the perceived loss of ready access to their data.  As time goes on, I would say that unencrypted data will rise to the top of the top 10 list.  Why?  Because the information age is all about the control of information.  The more information you control and can use to your advantage, the more power and control.  If your information can be readily obtained through public sources or the lax security surrounding your information systems, then you have little, if any, power or control.  The next 10 years will likely be spent by most organizations figuring out what information is critical to their business model and implementing the necessary protections around that information.  Critical information will be protected like the gold at Fort Know because, to that organization, that is their “gold” and it must be protected accordingly.  And that protection will likely involve encryption for some or all of it.

I know that people have a lot on their plates these days.  However, if you are a security person or a DBA, you need to leverage these surveys to your advantage and address the top 10 issues.  If more companies did this, the less data that would be breached.

Supermarket Chain Notifies Customers Of Self Checkout Skimming

Customers of Lucky Supermarkets, a subsidiary of Save Mart Supermarkets, got an early Thanksgiving gift when they were notified that 20 Lucky Supermarkets and one Save Mart store in California had discovered skimming devices inside those stores’ self checkout systems.  As retailers move to more and more automation surrounding checkout, the more risk they take on unless they put in place controls to minimize those risks.

Grocery stores can learn a lot from gas stations and convenient stores and the controls these organizations have put in place to secure gas pumps.

• Locks are keyed the same.  It turned out that gas pump manufacturers for as far back as they have had locks on pumps use the same keys.  I worked at a gas station when I was in high school a long, long time ago.  Yet, six years ago conducting a security review for a State Transportation Department, I found that the pump keys that I had neglected to return 30 years earlier would open the gas pumps at the maintenance garages.  Whoa!  Turns out self checkout manufacturers have the same problem unless you request them to use a different key and lock combination.  So the first lesson to learn is to explicitly request a unique to your organization lock and key set for your self checkout units.
• Locks are not perfect.  Even if you change out the locks, they can still be picked fairly quickly by someone who knows what they are doing.  So, for a second level of defense, you need to add serialized tamperproof tape to the doors that allow access to the inside of the self checkout unit.  Newer self checkout units only allow ease of access to change out the register tape and nothing else.  Only authorized technicians have the ability to gain access to the rest of the device.  However, best practice is to put serialized tape on all of the doors regardless.
• You may need tamperproof tape inside.  Older (age is all relative) self checkout units allow access to too much of the internal workings and/or do not have tamperproof parts inside the cabinet.  That means that “bad guys” can insert a skimmer without any obvious changes.  To avoid that problem, you can wrap connections with tamperproof tape so that if anyone attempts to take the connectors apart, it will be obvious upon inspection of the tamperproof tape.  Discuss your situation with your self checkout vendor as they can advise you on what should be taped.
• Locally monitor your equipment.  The serial numbers on the tape need to be recorded on a log sheet and the tape (inside and outside) should be checked at every manager shift change.  Any tape that appears to have been tampered with should be investigated and that unit taken out of service until an authorized technician deems it safe to put back in service.  In addition, video monitoring should be in place to monitor each self checkout.  Staff should be present at all time to monitor these devices.  Typically a single person can cover two isles compromised of a total of four self checkout devices.  Any more than that and your staff can be easily distracted and miss someone tampering with a device.
• Remotely monitor your equipment.  Most self checkout devices have the ability to be centrally managed and monitored.  In the event a door is opened, the unit can send an alert to the central monitoring console.  When an alert is generated, operations personnel should immediately contact the retail outlet and have store management immediately investigate the alert and inform the central monitoring station of the devices status.  If the store is not in operation, then the central operations personnel should contact local law enforcement and store management that an emergency exists at the store.

As a friendly reminder, security is not perfect.  These controls have to be executed perfectly every day, every year.  That is where things always go awry.  However, if you do execute these controls consistently, your organization should be very difficult to compromise.  The “bad guys” will know that and will find an easier target.

What To Do About Insiders

The first posting I did on this subject was to provide an understanding that, despite the news stories, the insider threat is a very real threat and needs to be addressed.  However, what is an organization to do?  Employees and others need to have access to certain information in order to get their jobs done.  What steps should an organization take to minimize the insider threat?

First, I need to be very clear about this.  Even when you do all of what I recommend, you are only minimizing the insider threat.  The insider threat can never be totally mitigated.  Insiders must have access to information that the general public or even you business partners do not have access.  As a result, should an employee get sloppy with controls or go “rogue,” you can expect to lose whatever information that person had access.  Remember my mantra – security is not perfect.

I posted some ideas a while back on controls for automation.  Here are my minimum recommendations for manual controls to put into place to minimize the insider threat.

• Management needs to recognize the importance of management controls.  The “tone at the top” really does mean something when it comes to controls.  However, management needs to understand that these sorts of controls are no absolute guarantee of avoiding issues.  Properly implemented, monitored and adjusted as necessary, such a control environment will confirm to the rest of the organization that management believes that controls are important.  If management does not know what to do regarding management controls, then they should consult with a public accounting firm as they are very aware of control environments and can assist in the design of a control environment.

• Preventive controls.  Preventative controls, as their name implies, put in place something to prevent a problem.  A prime example of a manual preventive control is requiring a minimum of two signatures on checks.  The larger the amount on the check, the more people that have to sign off on the check.  Under such an approach multiple people have to collude to defraud the system.  This sort of approach can also be taken for report reviews of inventory, cash on hand and any other metrics that are important to the survival of the organization.  The idea is to ensure that at least two people are involved in these reviews and that they physically sign off on their review and document and start an investigation into any irregularities.
• Detective controls.  As the name implies, detective controls are controls used to detect problems.  Following the example in preventative controls, the other people signing off on a check or reviewing a critical metric report is a detective control.  If the reviewer feels that something is not right with what they are reviewing, they are obligated to notify their immediate supervisor of the issue and ask the submitter to physically document the situation.  Once documented, the reviewer can then either sign off and accept the explanation, or refuse and further investigate.
• Corrective controls.  Corrective controls are those controls used to ensure that the preventative and detective controls are focused on the right problems and are going to be able to be relied upon going forward.  Keeping to the theme, in the event of an irregularity being identified, management should then institute a root cause analysis and determine what caused the situation and make the necessary changes to the preventative and detective controls to ensure that people do not try to circumvent the control environment.
• Hold employees responsible for the control environment.  Management may be responsible for establishing controls, but it is the employees that make the control environment actually work.  Employees should have their key controls evaluated at least annually to reinforce the importance of controls.  In our check example, the people signing off on checks should be evaluated on how many checks with problems are issued by the organization that they were required to sign.
• Solicit control improvement ideas from employees.  The problem most organizations have with management controls is keeping them relevant.  A common example we see is a problem that occurred ten years ago has been addressed by automated controls in a new computer system, yet management continues to require the manual control to be followed.  Most of the time, employees know exactly what needs to be done, but management does not want to recognize that fact.
• Have a third party periodically assess your controls.  In addition to employees providing ideas, organizations should periodically invite a third party, such as their accounting firm, to assess the control environment and recommend changes.  A number of years ago I worked with a large organization where we discovered that the way one of their computer systems had recently been modified, checks could be generated and bypass approvals and oversight.

For those of you that are going to recommend these minimum controls, my heart goes out to you.  The road ahead is likely to be very bumpy and contentious if your organization has a mediocre control environment.

Something to share with management as you push this sort of project is that there are very measureable benefits to implementing controls.  Every organization that I have worked with over the years has found that a byproduct of their controls projects has been fewer customer complaints and fewer employee screw ups.  Avoiding problems or making them smaller and less impactful on customers can add up to serious savings in time and money.

If you have a mature control environment, take a look at how you can make them better, more effective and more relevant.  If you do not have a mature control environment, then take baby steps.  Look to your accounting area as they will likely have the most robust control environment.  Grab one of those accountants and use them to help you look at other areas that may have problems that controls can address.

Best of luck to all of you on your journey.

The Insider Threat

At this year’s PCI Community Meeting there was a great presentation done by Verizon Business Services on their 2011 Data Breach Investigations Report.  However, one of the things that concerned me about their presentation is that they seemed to downplay the threat insiders pose to the breach of information.

So, I went back and reread their report because I did not get that same interpretation when I originally read the report.  On page 42 of the Verizon report, there is a discussion of “Errors.”  Errors are defined “as anything done (or left undone) incorrectly or inadvertently.”  According to this section, there were 219 instances out of the total 761 breaches where insiders contributed to the breach.  That computes to almost 30% of all breaches.  That is almost twice the 17% quoted as a highlight in the front of the report and used to justify the downplaying of the insider threat.  So the insider threat is still substantial and should not be ignored.

The biggest problem with the insider threat is that it does not matter how much technology you have in place to protect your information assets as it only takes one person in the right place to neutralize every last bit of your high-tech security solutions.  Just ask anyone at any of the recently breached organizations how all of their technology functioned when they suffered their breach.  I am sure they will tell you the technology worked just great – not so much for their people.

First, there are the mistakes everyone makes that are done in the name of efficiency or politeness.

• The sharing of a manager password to expedite a process in the name of good customer service.
• The holding open of a door to a secure facility because someone’s arms are full.
• The swiping of your access card and letting others tailgate through a secured door to be polite.

At the end of the day, we all are guilty at one time or another of doing these things as well as many other bad security acts.

That is the real problem we all face and why security standards focus so much on a layered approach also known as defense in depth.  The hope is that with multiple layers in place, even if one or two layers become non-functional due to the people issues, another layer will stop or at least detect the issue and the issue will be averted or minimized.  However, in most cases, if someone can get the right software onto the right user’s computer, it really does not matter what security is in place.

The first example of such a breach was of RSA back in March 2011.  As the story goes, through the use of electronic mail, attackers targeted RSA network and system administrators with messages containing an Excel spreadsheet as an attachment.  The spreadsheet contained backdoor software that was surreptitiously installed on the computer providing the attackers with remote access into RSA’s network.  With remote access, the attackers are free to scope out the RSA network at their leisure.  Over time, the attackers obtain the code for RSA’s SecurID servers and FOBs.  Once discovered, RSA is forced to replace SecurID FOBs for free to their customers.

Right on the heels of the RSA breach was the breach of Epsilon in April 2011.  Epsilon was a quiet firm that did the electronic mail marketing and loyalty programs for such businesses as Best Buy, Kroger, Marriott and LL Bean.  It creates the biggest opportunity for spear fishing ever seen.  Based on news reports, Epsilon was attacked in a similar fashion as RSA although the two breaches have never been linked to the same attackers by authorities.

In May 2011, the apparent fruits of the RSA breach were unleashed on Lockheed Martin and rumored to have also been unleashed on Northrup Grumman and L3 Communications.  Using fake FOBs, the attackers broke into Lockheed Martin and attempted to gain information from Lockheed Martin’s systems.  News reports indicated that the Lockheed Martin attack was eventually repelled and no information was obtained by the attackers.

Citigroup suffered a double whammy of bad news in June 2011.  The first hit was the admission that their online banking site had been compromised and more than 350,000 customer accounts had their information leaked to the attackers.  Then on June 26, a former Citigroup executive was arrested for embezzling $19 million while they worked in the treasury finance department.  The first event was front page news in the papers and on the Internet.  The second event barely made a notice.

The reason I bring these breaches up is that while these computer attacks are big news, they point to the fact that the bigger threat is actually the people you employ.  The reason why attackers targeted these companies’ employees is that insiders have direct access to information and, in most cases therefore, that attackers do not need to hack any computer systems to gain the information.  This is bore out in the fact that security survey after survey keeps confirming that the vast majority of compromises are the result of some amount of insider involvement.

All organizations are at significant risk to the insider threat because most have done little or nothing to prevent it.  Sarbanes Oxley and the like have done no one a favor in propagating this view of controls.  This is why I think a lot of organizations push back on complying with the PCI DSS, they abhor controls and want nothing to do with controls of any type.  You hear arguments regarding the “stifling of creativity” and “make do work” which are nothing more than whining from people that have no clue as to why controls are needed.  I have documented in a previous post the three phases of a well structured control environment, so I will leave readers to review that for reference.  Properly designed and implemented, internal controls make a big difference because if people know that someone is looking over their shoulder to ensure that their job is done properly, that in and of itself goes a long way to keeping everyone honest as well as minimizing operational errors.

However, just because you have controls does not mean that everything will go smoothly.  A lot of organizations have a great control environment on paper, but do very little to ensure that it is executed as written.  We see time and again organizations that talk a good game, but when you start looking at their operations, the control environment is a paper tiger because no one is enforcing those controls and the controls are being followed haphazardly, if at all.

Then there are the organizations that have never reviewed and streamlined their controls since the founding of the organization.  These sorts of companies have created a controls monster.  What has happened is that as the business encountered a new issue, a new control was placed on top of other controls to address the new threat.  Over the years, all of these controls now keep a huge bureaucracy busy doing nothing but making sure that controls are controlled.

While some of this problem can be laid at the feet of management, it is not entirely all their fault.  Custom and packaged application developers have only recently started implementing security and controls into their software that allow the level of granularity necessary to meet SOX, PCI and other security requirements.  In most cases, security has always existed in applications, just not at the level necessary to properly ensure the security of sensitive information in today’s environment.

The lesson here is to ensure you have controls in place to ensure the security of your sensitive information.  If your applications that process, store or transmit sensitive information do not have the capability to properly protect your sensitive information, then you need to create manual controls to fill in any gaps.  Those manual controls need to have the ability to provide feedback so that if they begin to fail, someone is alerted to that fact and can step in and get the controls functioning again.

Defense In Depth

I have a slide in my security presentation deck that discusses the concept of defense in depth and how when you start opening ports or start using encrypted data streams how you are punching holes into one or more of your security layers.  It amazes me how many people still do not understand how defense in depth works and how much security standards such as the PCI DSS rely on this concept.

So let us take a look at the various elements of security and the requirements of the PCI DSS and see how they bring defense in depth to bear.  Keep in mind this is an example and does not encompass everything an organization could do to increase defense in depth.

For most organizations, the first level of defense is at their firewall.  Requirements 1 and 2 talk to how you should use a firewall and secure it.  The biggest mistake that organizations make is not configuring their firewall properly.  And by configuration, I am not just talking about the configuration of the firewall’s software; I am also talking about where and how the firewall is used in the network.

The next level of defense for most networks is usually some form of intrusion detection/ prevention system.  Some of the requirements in 10 and 11 talk to intrusion detection/prevention.  IDS/IPS capability may be provided in a separate appliance or may be part of an organization’s firewall.  The key to using an IDS/IPS is ensuring that it is kept current with its attack signatures, monitoring its log data and/or console and ensuring that it is not be overwhelmed by network traffic.

One thing that continues to amaze me is how many implementations of IDS/IPS I encounter where the IDS/IPS are in the middle of encrypted data streams.  IDS/IPS systems cannot examine encrypted data streams unless they have the decryption keys which they typically do not have access.  As a result, encrypted data streams are not examined and therefore sensitive data and or attacks could be going right past the IDS/IPS.

How users authenticate to your network and devices is also a level of defense.  Requirements 7 and 8 of the PCI DSS talk to this point.  And it is not just authentication to applications that process, store or transmit cardholder data, it is also authentication to infrastructure devices and to databases.

It has been more than five years since the “sa” default password debacle and yet you still encounter applications that use service accounts to access their database and those service accounts have no password.  The rationale?  “We did not want to code the password into the application,” is the common reply.

The other big area of authentication issues that you encounter is with firewalls, routers switches and other network infrastructure.  The problem is that the network administrators all use the same account and password.  You can understand their rationale, particularly those networks where you are administering thousands and thousands of devices.

There are a number of ways to address this situation, but these are my favorite two.  The first is to implement 802.1X authentication using a RADIUS server.  Under this scenario, every network administrator has their own unique account and password to access the network devices.  Those unique accounts should be different from the network administrator’s account they use to get email and network access like every other user.  A lot of organizations already have the RADIUS server implemented for remote access, so adding in network administration access control is relatively easy.

The second way to address network administration access is to use a “jump box.”  In a “jump box” implementation, two or more “jump boxes” are placed at strategic points on the network and all network administration access is conducted through a “jump box.”  The “jump box” is fully instrumented in that all keystrokes, applications, etc. are logged and those logs are reviewed at least daily to ensure that network administrators are not changing things they should not be changing.  That means comparing service tickets for the network against the logs from the “jump boxes” and ensuring that only what was required to be changed was changed.  “Jump boxes” can also be used to control access for server administration.

A level of defense that usually gets little recognition is operating system (OS) hardening.  What some people seem to forget is that any computerized device has an OS whether it is a firewall, router, switch or server.  Requirement 2 talks not only to the hardening of wireless, but also firewalls, switches, routers and servers.  Every vendor publishes a guide that explains how to securely implement their OS.  Where things can get sticky is with third parties that argue that their product or software will not function if you follow the vendor’s OS hardening recommendations.  In my experience, testing a vendor’s product or software in a hardened environment typically does not have an adverse result.  However, the key is to conduct a test.

Another level of defense is anti-virus and anti-malware software.  This solution also usually includes a personal firewall on mobile devices such as notebooks, netbooks and smartphones.  Requirement 5 of the PCI DSS talks to anti-virus and anti-malware while a requirement in 1 talks to personal firewalls.  Nothing gets some people wound up more than anti-virus software.  The requirements in 5 can have compensating controls, but implementing those compensating controls consistently on mobile devices is usually just about impossible.  So while you may not have anti-virus/malware on your e-Commerce servers, you should have it on all of your desktops, notebooks, netbooks and other systems.

A level of defense that most organizations poorly manage is their collection and analysis of log data from their network devices and servers.  Requirement 10 speaks to the importance of log data.  As I have written before, log data is IT’s version of commercial aircraft’s flight data recorder.  If you want to why a problem occurred, log data from your devices can usually point you to the reason.  The problem most IT professionals have with log data is that they do not want to log everything because that generates too much data in their opinion.  However, until you have an incident, you do not know what log data will be important in identifying why the incident occurred therefore you need all of it.  The last thing you want to have happen is to tell management that you could not determine the cause of an incident because you did not record the critical information required in identifying the incident.

The final defense most people think of is application development which is covered by requirement 6.  If you are going to get push back, this is the most likely and consistent place you will get push back to the PCI DSS or any other security program.  Application developers are very protective of their environments, so when you start infringing in their area, they can get rather upset.  As a result, you hear the typical lament from developers that security, “restricts their creativity.”

In today’s rush to get things done, application developers usually do not have security at the front of their minds.  As a result, by the time anyone knows that there is a security issue, it is too late for it to be fixed and the application goes into production with the fix to be part of version 2.  That was the whole point the PCI DSS addresses in requirements 6.4, 6.5 and 6.6; avoid putting susceptible software into production.  The whole point of these requirements is to build a certain amount of security into the development process to minimize the amount of security issues that end up in production.

The real final defense is an organization’s policies, standards and procedures.  Yes, that paperwork that everyone thinks is “make do” work, really does have a purpose.  An organization’s policies, standards and procedures are the rules that everyone is to follow to ensure security.  Those rules also provide a way to measure people’s compliance so that, in the event of an incident, those people that did not follow the policies, standards or procedures can be shown their mistakes so that they can correct their actions in the future.  These rules also provide an organization’s framework for explaining to personnel as to how the organization is protecting their information assets and defining those assets.

There are a lot more options for defense in depth, but I think you get the idea.  Now that you understand how defense in depth works, you should now also understand what happens when security personnel are asked to open ports for an application or change configurations that reduce the number of levels in an organization’s defenses.  The fewer levels involved the higher the likelihood that a lapse in control can result in a breach, particularly when a number of lapses in controls are occurring simultaneously.  This is how supposedly PCI compliant organizations end up breached.