I am arguing with another client about the retention of log data. They are pushing back because they generate around 190GB of log data daily from their Web sites. That works out to around a whopping 70TB of log data annually. The problem they face is one of: (1) not enough space in their data center for another SAN and, (2) the budget constraints we all face in a down economy. They are not unusual. We get a lot of push back on log data.
The problem is that most security and operations personnel live in the moment or at least the ‘near’ moment. These people are researching a problem that happened a couple of minutes or maybe a day or two ago. As a result, they have a tough time justifying in their own minds why retaining a year’s worth of log data is important. Then there is management’s view on this subject. Management typically does not understand the reasoning for keeping a year’s worth of log data either, let alone what log data really is. Forensic examiners, for the most part, live in the past. They want to know everything that is relevant up to the point where the target determined that an incident had occurred. The problem is that without complete records from every device involved in the incident, the tracing of an incident back to its origination is likely impossible.
And this is typically the first point where most clients push back. “Why do I have to log everything?” is the most common refrain. I like to use an analogy here. I explain that system logs and their relevant management systems are the IT industry’s version of commercial aviation’s flight data recorder. If you want to be able to reconstruct where things went wrong and the incident occurred so that you can address the situation, you need this information. Unfortunately, unlike an airplane, an incident can take months to even be recognized let alone the number of devices that might be involved, so there needs to be a much larger pool of data collected and made available for forensic analysis. And until an incident is recognized, you will not know what information will be germane to the forensic investigation and what is chaff. As a result, you need to be logging as much as you can so that you have as complete a record of your operations as possible.
The second thing we hear most often is, “We only look at log data when we have a problem.” Typically what this means is that the problem has now grown to the size of a whale and has become noticeable by end users who are complaining. Unfortunately, some of those end users are also likely senior management, so it has become a large priority to fix the problem. As a result, you really do not have a problem, you have a crisis. It will now likely take a lot of time, effort and possibly outside consultants to fix the problem at a great expense. If the organization had been analyzing its log data on a daily basis, the problem likely would have been identified long before it became a crisis and could have been taken care of very easily with the existing staff and possibly a bit of overtime.
The third lament is, “Why do I have to retain a year’s worth of log data?” As I stated earlier, an incident may not be recognized until a number of months have passed. The reason this can occur is because vulnerabilities can sometimes be unrecognized for months, even years. Statistics say that, on average, it takes a little over 30 days from the time a vulnerability is recognized until a patch is available. But that is the rub, it is not until the vendor or a researcher identifies the vulnerability and makes it public that you will finally know that you have a vulnerability. And that confirmation by the vendor may occur months after the vulnerability was found by the attackers. In the meantime, the attackers have been using the vulnerability to their advantage without anyone’s knowledge. As a result, your protection systems do not have a signature to recognize this new vulnerability until it is identified months later. So you do not know if you have been compromised until you are able to pass that new signature by all of your systems and log data. If you only have 30 days worth of log data, you may never see that you have been compromised by the new vulnerability. This is where what you do not know really hurts you.
The final common complaint I hear is, “Why do I have to have log data from my routers, switches and other infrastructure?” I always respond back with my own question, “What? Your infrastructure is not involved in an incident?” Your network infrastructure of firewalls, intrusion detection/prevention, routers, switches, load balancers and the like may be the first opportunity you have to gain information on how an incident was engineered. All of these devices are capable of generating log information of their own. It is that information that will be invaluable in determining the initial point(s) of entry into your network. For it is always the infrastructure logs that will tell you whether the incident was caused from the outside or if it was caused from the inside.
The latest breach study from Verizon Business Services indicates that 70% of all breaches occur from the outside of an organization and that 65% of breaches are the result of human error. The CSI Computer Crime Survey indicates that every business will suffer an incident on an average of every two and a half years. Basically this all says that you will have an incident at some point in the near future and it will likely be the result of human error. If you do not have the information to identify and rectify the problem, how will you address the problem? And that is just it; you will not address the problem because likely, you will not even know the problem exists. And this is where organizations get into trouble. Because they do not know they have a problem, they believe all is well until the problem finally becomes so big that it cannot be ignored. Then the organization blames everyone else because they should have been telling them the problem could have existed.
So keep and analyze that log data. Analyze your log data at least daily, but preferably more often. The job and business you save may be your own.
PCI Guru 