09
Dec
14

Significant Change And Periodic

UPDATED: Changed comments on requirement 10.6.2 to reflect the correct interpretation of that requirement.

No words or phrases in the PCI standards elicit more comments and questions than “significant change”, “periodic” and “periodically”.

So what do these mean?  Whatever you want to define them to mean as it is up to each organization to come up with formal definitions.  Those definitions should be based on your organization’s risk assessment.

Here are some suggestions as to appropriate definitions.

Significant Change

Significant changes are those changes that could impact or affect the security of your cardholder data environment (CDE).  Examples of significant changes are:

  • Changing devices such as firewalls, routers, switches and servers. Going from Cisco to Checkpoint firewalls for example is typically understood as a significant change.  However, people always question this concept particularly when going from a Cisco ASA 5505 firewall to an ASA 5520 or moving a virtual machine from one cluster to another.  The problem is that these moves can potentially introduce new vulnerabilities, network paths or even errors that would go unknown until the next vulnerability scan and penetration test.  And your luck would be that those tests are months away, not just a few days.
  • Changes to payment applications. This should be obvious, but I cannot tell you how many people argue the point on changes to applications.  Yet, application changes are possibly the biggest changes that can affect security.  Not only should applications be vulnerability scanned and penetration tested before being put into production, but code review and/or automated code scanning should be performed as well.  If any vulnerabilities are found, they must be corrected or mitigated before the application goes into production.
  • Upgrades or changes in operating systems. Upgrades and changes in operating systems should also be obvious as significant changes.  However, I have run into network and system administrators that want to split hairs over the impact of OS changes.  In my opinion, going from one version of an OS to another is just as significant as changing OSes.
  • Patching of operating systems or applications. While I do not think that patching necessarily results in a significant change, there are some patches such as updates to critical services such as .NET or the IP stack that should be considered significant.  If you are properly working through requirement 6.1 (6.2 in PCI DSS v2) for patch management, you should take this into consideration and indicate if vulnerability scanning and penetration testing are required after any particular patch cycle because of the nature of any of the patches being applied.
  • Network changes. Any time you change the network you should consider that a significant change regardless of how “minor” the change might appear.  Networks can be like puzzles and the movement of devices or wires can result in unintended paths being opened as a result.

I have a lot of clients that have an indicator in their change management system or enter “Significant Change” in the change comments for flagging significant changes.  That way they can try and coordinate significant changes with their scheduled vulnerability scanning and penetration testing.  It does not always work out, but they are trying to make an attempt at minimizing the number of special scans and tests that are performed.  But such an approach also has a side benefit when it comes time to do their PCI assessment as they can call up all significant changes and those can be tied to the vulnerability scans and penetration tests.

I would see this list as the bare minimum of significant changes.  As I stated earlier, it is up to your organization to develop your own definition of what constitutes a significant change.

Periodic and Periodically

Branden Williams was on a Podcast shortly after the PCI DSS v3 was released and made a comment that he felt that the number of occurrences for the words “periodic” or “periodically” were higher in the new version of the PCI DSS than in the previous version.  That got me thinking so I went and checked it out.  Based on my analysis, these words occur a total of 20 times in the PCI DSS v3 with 17 of those occurrences in the requirements/tests.  That is a 150% total increase over v2 and an increase of 113% in the requirements/tests.

First off, just to shatter some people’s perception of the word, “periodic” does not equate to “annual”.  Yes, there may be instances where an activity can occur annually and still meet PCI DSS compliance.  But that is likely a rare occurrence for all but the smallest organizations and is definitely not how the Council has defined it.

The Council uses the words “periodic” and “periodically” to reflect that an organization should be following the results of their risk assessment to determine how often or “periodically” they should perform a certain activity.  For some organizations, that might happen to work out to be annually.  But for most organizations it will work out to be something more often than annually.

So what requirements specific a periodic time period?  Here are some of the more notable occurrences.

  • 5.1.2 For systems considered to be not commonly affected by malicious software, perform periodic evaluations to identify and evaluate evolving malware threats in order to confirm whether such systems continue to not require anti-virus software.Typically this would be done annually, but forensic analysis of breaches has indicated that it needs to be done more often, particularly with Linux and other Unix derivatives. Based on threats semi-annual or even quarterly reviews may be needed for systems you believe to not warrant an anti-virus solution.
  • 5.2 Ensure that all anti-virus mechanisms are maintained as follows: Are kept current, Perform periodic scans, Generate audit logs which are retained per PCI DSS Requirement 10.7.Periodic scanning is always an issue with servers but, surprisingly, even more so with workstations. In my opinion, at a minimum, scans for viruses and malware should be done at least weekly.  This might need to be done daily if the systems are particularly at risk such as in call centers where the workstations my go to the Internet to be able to access competitor sales offerings.
  • 8.2.4.b Additional testing procedure for service providers: Review internal processes and customer/user documentation to verify that: Non-consumer user passwords are required to change periodically; and Non-consumer users are given guidance as to when, and under what circumstances, passwords must change.This requirement pairs with 8.6.2 which requires service providers with remote access to customers’ systems to not use the same credentials for each customer. A number of recent breaches have pointed out the issue such a practice can lead.  Not only are different credentials needed by the password for those credentials needs to change periodically, typically every 90 days.  This will likely spur the sales of enterprise credential vaults and similar solutions in the service provider ranks.But it is not just service provider’s credentials; it is also their customers’ credentials.  Service providers need to advise their customers to change their passwords periodically as well.  And that should also be at 90 day intervals at a minimum.
  • 9.7 Obtain and examine the policy for controlling storage and maintenance of all media and verify that the policy requires periodic media inventories.For this requirement, the PCI DSS already provides a required timeframe of at least annually.
  • 9.8 Examine the periodic media destruction policy and verify that it covers all media and defines requirements for the following:Periodic here typically means quarterly or even monthly if you have the volume of media to be destroyed. The key though is to secure the media until it is destroyed.
  • 9.9 Examine documented policies and procedures to verify they include: Maintaining a list of devices, Periodically inspecting devices to look for tampering or substitution, Training personnel to be aware of suspicious behavior and to report tampering or substitution of devices.Here periodic means at least daily, if not more often. I have clients that examine their points of interaction (POI) at every management shift change which works out to three or four times a day.  Given the POI is becoming the primary target of attacks, this will only become more important as time goes on given the current paradigm.
  • 9.9.2 Periodically inspect device surfaces to detect tampering (for example, addition of card skimmers to devices), or substitution (for example, by checking the serial number or other device characteristics to verify it has not been swapped with a fraudulent device).Again, periodic means at least daily, if not more often. I have clients that examine their points of interaction (POI) at every management shift change which works out to three or four times a day.  Given the POI is becoming the primary target of attacks, this will only become more important as time goes on given the current paradigm.
  • 10.6.2 Review logs of all other system components periodically based on the organization’s policies and risk management strategy, as determined by the organization’s annual risk assessment.This requirement allows systems to be ranked using an organization’s risk assessment to drive how often log data from systems have to be reviewed.  While systems that directly process, store or transmitcardholder data (CHD) must have their log data reviewed at least daily, other systems that are in-scope can have their log data reviewed less often based on the risk they present to the CDE systems.  Based on assessing the risk to these “connected to” systems, you might be able to justify weekly or even monthly review of log data. I doubt this will have a significant impact because most organizations have implemented internal or outsourced system information and event management (SIEM) solutions and are monitoring all in-scope systems in near real time.  But for those few organizations that are struggling with log reviews without a SIEM, this will afford them a bit of breathing space.
  • 12.10.4 Verify through observation, review of policies, and interviews of responsible personnel that staff with responsibilities for security breach response are periodically trained.It amazes me the number of organizations that claim to not have had an incident in the last year, even a virus or malware outbreak. Either they were totally dealt with by their anti-virus solution (hard to believe) or I am not talking to thepeople that deal with these issues (probably more likely).  As a result, testing (which can satisfy this training requirement) is only being done annually just like business continuity plan testing.Given the ever increasing amount of threats, this sort of training needs to be done more often than just annually.  Organizations should be at least testing their incident response plan on a quarterly basis so that people keep their skills up as well we just exercising the plan and finding any gaps or processes that need adjustment.

Hopefully we are now all on the same page with these terms.

04
Dec
14

It Is The QSA’s Fault

“Usually when PCI-compliant companies are breached, the real culprit is the assessor, the person who confirmed the company had met the PCI Requirements.” Jeff Multz, Dell SecureWorks

This is a very interesting approach for an employee at a qualified security assessor company (QSAC) to use to drum up business, toss all QSAs, including his own organization’s QSAs, under the bus.  I know that is not what he meant to do, but that is certainly what he did with this statement in his posting a few days ago.

I think most QSAs know where Mr. Multz is coming from.  He is more than likely venting over losses to QSACs that we all know are more interested in revenue generation than security.  They further that goal by incenting their QSAs to do as many PCI assessments as possible in the shortest amount of time as well as identify opportunities for selling the QSAC’s security appliances to solve compliance problems.  And to just pile on, they further their revenue generation by being the low cost provider through a focus on volume of work over quality.  As Kurt Vonnegut said in Cat’s Cradle, “In this world, you get what you pay for.”

Getting back though to Mr. Multz and his statement that QSAs are responsible for all breaches, let us see how that plays out with a few breaches.

During the Target breach, it was the QSA that was socially engineered and gave away the keys to the kingdom and missed all of the alerts generated by the FireEye software.  At Neiman Marcus, it was the QSA that missed the alerts for 60+ days that the malware was reinstalling nightly.  It was the QSA that swapped out the points of interaction (POI) at Barnes & Noble for malware infested POI.

Sorry Mr. Multz, but it was employees and/or contractors at all of these organizations, not the QSA that had a part in these breaches and all breaches for that matter.  I really do not see how you can hold a QSA responsible for the inaction and errors of employees/contractors.  Organizations are not going to pay to have QSAs on site, 24×7, to babysit all of their employees to maintain compliance with PCI or any other compliance program.  Not only that, no security framework is ever going to stop breaches, all they do is hopefully minimizing the impact when a breach occurs.

However, Mr. Multz was not done.

“The PCI Requirements were created so that organizations would focus on securing their networks, but many assessors only focus on meeting the requirements rather than security.”

From this statement it is painfully obvious that Mr. Multz does not understand what an assessment is about and how the assessment process works.  The job of a QSA is to execute the tests as defined in the PCI DSS Reporting Template and report the results of that testing – nothing more, nothing less.  Organizations are judged by a QSA as compliant with the PCI DSS whether they are just squeaking by or if they have a full on security program next to none.  Organizations do not get “extra credit” or “atta boys” if they have gone beyond the requirements.

While the original intent of the standards was to focus on securing cardholder data, that got morphed by the wonderfully misdirected marketing job that was done by certain card brands before the PCI standards came together.  For those of us around the security industry more than a decade ago, we advised Visa and MasterCard to stop pushing their cardholder information security program (CISP) and site data protection (SDP) standards as “The Way” that was going to stop breaches.  We explained that, properly implemented, CISP and SDP should minimize the number of PANs obtained, but it would not completely stop breaches.  It was only recently that the card brands started to realize this fact and stop pushing the PCI standards as a panacea of security.  If you have noticed with the rollout of EMV, Visa, MasterCard and the PCI SSC have stated that EMV is not a “silver bullet” solution and in other statements stated there are no “silver bullet” solutions.  That is a long way from a decade ago when their security standards were sold as the “be all to end all” for stopping breaches.  Unfortunately for QSAs everywhere, that message is out there and we have to deal with it every day.

All of this is not to say that QSAs cannot and do not make recommendations to organizations regarding their security programs and how and where it needs to improve.  I constantly make suggestions during my PCI assessments on how my client needs to improve their security posture.  However, it is ultimately up to the organization to put such changes in place, not the QSA’s responsibility.  If an organization chooses inaction, I will bring it up again and again.  But as the old proverb states, “you can lead a horse to water, but you cannot make them drink”.

Where the PCI DSS assessment process truly fails is the point in time approach (with the exception of vulnerability scanning and a few other select requirements).  To address that shortcoming, the Council has introduced the concept of business as usual (BAU) and it is my guess that we will see that concept placed into the standard in the next version.  It will be then that QSAs will have to test PCI compliance over a 12 month period similar to testing procedures financial auditors perform for annual financial audits.

As a result, the inclusion of BAU as part of the PCI DSS will likely be the straw that breaks the camel’s back for a lot of organizations.  This is because BAU will require organizations to track their compliance with the PCI DSS 24x7x365 as they should have been doing all along.  But from experience, I can tell you that there is no organization I have ever encountered that was compliant with any standard all of the time because people make mistakes.  As such, BAU is designed to shed light on those mistakes and require organizations to identify them and remediate them.  For organizations just squeaking by, this will probably make PCI compliance truly impossible to achieve.  If you are one of those organizations complaining about compliance with the current PCI DSS, just wait until BAU gets added.  Organizations that are truly interested in security are already implementing BAU because they see the operational value in integrating security controls with their other business controls.  BAU will show the true colors of those organizations that want security versus those that are checking a box.

And that gets me to Mr. Multz’s actual reason for his post, what makes a good QSA?  Good QSAs understand that the world is not perfect nor is security.  Good QSAs know that compliance with the PCI DSS does not and will not eliminate breaches.  Good QSAs know that the goal of PCI compliance is to minimize security control errors, provide an ability to recognize security control errors as soon as possible and then remediate those security control errors such that the security controls are only non-compliant for the shortest possible amount of time.

But just because a company has such errors does not automatically mean that they are not PCI compliant.  A good QSA only judges an organization non-compliant when the QSA has evidence that problems are consistently recurring and are not being corrected in a timely manner or corrected at all.

I appreciate Mr. Multz’s frustration but as a QSA I do not appreciate him tossing me under the bus with the QSAs that are doing a disservice to PCI compliance.  Like any industry, there are good service providers and there are bad service providers.  Those of us in this industry all know who the bad ones are and we hope they will get weeded out.  But from my own long experience in consulting, that does not always happen.

So in my very humble opinion, Mr. Multz needs to suck it up and deal with it, but stop tossing QSAs under the bus in the process.  QSAs are only the messengers.

23
Nov
14

Face It, You Are A Poor Judge Of Risk

“The oldest and strongest emotion of mankind is fear, and the oldest and strongest kind of fear is fear of the unknown.” HP Lovecraft

We have a pop quiz today.

  1. Are you more likely to die from an alligator attack or a shark attack?
  2. Are you more likely to win the PowerBall lottery jackpot or become a movie star?
  3. Are you more likely to die in a vending machine accident or from a lightning strike?
  4. Are you more likely to be elected President of the United States or to date a supermodel?
  5. Are you more likely to die from influenza or from drowning?
  6. Are you more likely to catch influenza or Ebola?

The purpose of this pop quiz is to demonstrate how poorly we humans evaluate and understand risks. I have to admit I got caught on a couple of these as I did the research.

If anything, the Ebola discussion has brought this issue of risk judgment to the forefront given the unfounded fear people have of Ebola. As a mathematician by schooling it has fascinated me as I watch the media reports and government officials cave into the spread of fear over something very highly unlikely to occur to anyone in the general population.

Do not get me wrong. If I were a health care worker anywhere in the world, I would have concerns about my risk of catching Ebola. After all, they are on the front line and Ebola has around a 50% fatality rate. Add into that the informative, but frightening, video that Dr. Sanjay Gupta of CNN did on the difficulty of removing a containment suit without potentially infecting yourself, and it confirms the threat a health care worker should be feeling if confronted with a potential Ebola patient that is symptomatic.

But for anyone outside of health care, there should be little if any reason to be concerned. Yet a good percentage of the public is irrational when it comes to Ebola regardless of the fact that it requires contact with a symptomatic person’s bodily fluids in order to be infected. But unlike a person with influenza, an Ebola infected person that is contagious does not have the mobility required to have contact with people unless those people come to them. As a result, all of these mental gymnastics that people go through about the possibility that an infection could occur on a bus or the subway are silly because the person with Ebola when they are contagious would look worse than a zombie off of ‘The Walking Dead’, assuming they could even walk at that point.

I am sure you are all saying that this is all good and well, but what is the point here in regards to PCI?

Glad you asked. I bring this up because the PCI DSS is heading more and more to be driven by risk and the assessment of that risk. Yet as I have hopefully shown by my quiz questions, people and their organizations are poor at understanding and determining risks. So organizations need to get much better at performing risk assessments (if they are performed at all) so that they can truly understand and manage risks. That said, a risk assessment does not have to be, nor should it be, a huge “death march” of a project. A proper risk assessment should answer the following questions.

  • What are the risks to the organization? This does not have to be an exhaustive, all inclusive list as you find in the various risk assessment methodology frameworks. But should include all of the most likely risks. For PCI compliance, this risk assessment only needs to address the risks to those things that are in-scope for the assessment. However, most organizations need the risk assessment for other reasons, so it often contains all risks, not just PCI risks. If it does contain risks outside of PCI, you should add columns for your other requirements so you can filter out just the PCI, HIPAA, GLBA, FISMA and any other risk frameworks.
  • What is the likelihood of the risk occurring? Typically, I use a scale of 1 to 5 where 1 is it occurs infrequently and 5 represents that it occurs often. If something never occurs, then it should be removed from the list.
  • If the risk occurs, what is the impact on the organization? Here I use a scale of 1 to 3 where 1 is low, 2 is moderate and 3 is high.
  • Multiply the likelihood with the impact and you get the risk rating.
  • Sort the risk ratings from highest to lowest and you have your risk assessment rating completed.

But hold on, you are not done just yet. Now you need to set your organization’s risk threshold. This will likely be a very contentious discussion as you will find that people within the organization have widely differing views on the level of risk they are willing to accept. However, it is important to capture the highlights of this discussion so that you have documentation for future discussions as you discuss future risk assessment results and reset the organization’s risk threshold.

Risks that fall below a certain risk rating are accepted and management formally agrees to accept them. Those above that level you develop methods of mitigating and managing those risks. Under my rating system, the lowest score that can be achieved is 1 and the highest score is 15. A lot of organizations might say that a total score of below 4 is to be accepted. For some organizations a better approach to accepting risk is sometimes to only accept those risks that have an impact of ‘Low’ (i.e., equal to 1). Therefore, all moderate and high impact risks are mitigated and managed.

Once you have your analysis done you will have a list of risks that require mitigation and management through monitoring and other methods.

Answers

  1. According to the Florida Museum of Natural History, between 1948 and 2005 there were 391 alligator attacks resulting in 18 fatalities whereas there were 592 shark attacks with 9 fatalities. That makes the alligator fatality rate almost three times as high as the shark fatality rate.
  2. The odds of winning the PowerBall are around one in 175M. While still incredibly long, the odds of becoming a movie star are significantly lower at one in 1.5M.
  3. Lightning is more deadly but do not underestimate that vending machine. According to the US National Oceanic and Atmospheric Administration (NOAA), the odds of being hit by lightning in the US are one in 1.9M. According to the US National Safety Council, there is a one in 112M chance of dying in a vending machine accident.
  4. The odds are in your favor if you are interested in dating a supermodel. Even better than becoming a movie star. You have a one in 88K chance of dating a supermodel according to Ask the Odds. The odds of being elected President are slim at one in 10M.
  5. The US Centers for Disease Control (CDC) estimate that the odds of drowning are one in 31.4. The CDC estimates that the odds of dying from influenza are around one in 345K.
  6. The CDC estimates that one in eight people will catch the flu in any given year and as seen in a previous answer, there is a one in 345K chance that a person will die as a result. Given the population of the US is around 315M and only four people have actually caught the Ebola virus in the US, there is around a one in 78M chance of catching Ebola in the US but that could change slightly if more infected people enter the US.
15
Nov
14

Security Or Checking A Box?

“Better to remain silent and be thought a fool than to speak out and remove all doubt.” Abraham Lincoln

What is your organization interested in?  Security or checking a box?

Not surprisingly, most people answer “security” and then go on to prove with their actions and words that they are only interested in checking a box.

For all of you out there that argue ad nausea about the meaning of PCI DSS testing requirements and the requisite documentation are interested in one thing and one thing only; checking a box.  I am not talking about the few that have honest differences of opinion on a few of the requirements and how a QSA is interpreting them and assessing them.  I am talking about those of you that fight constantly with your QSA or acquiring bank on the process as a whole.

If you were to step back and listen to your arguments, you would hear someone that is splitting hairs in a vain attempt to avoid having to do something that would improve your organization’s security posture.  In essence, you want to only be judged PCI compliant, not actually be secure.

To add insult to injury, these are also typically the people that argue the most vehemently over the fact that the PCI DSS is worthless because it does not make an organization secure.  Wow!  Want to have your cake and eat it too!  Sorry, but you cannot have it both ways.

Everyone, including the Council, has been very clear that the PCI DSS is a bare minimum for security, not the “be all to end all” for securing an organization.  Organizations must go beyond the PCI DSS to actually be secure.  This where these people and their organizations get stumped because they cannot think beyond the standard.  Without a detailed road map, they are totally and utterly lost.  And heaven forbid they should pay a consultant for help.

But I am encountering a more insidious side to all of this.  As you listen to the arguments, a lot of you arguing about PCI compliance appear to have no interest in breaking a sweat and doing the actual work that is required.  More and more I find only partially implemented security tools, only partially implemented monitoring and only partially implemented controls.  And when you dig into it as we must do with the PCI assessment process, it becomes painfully obvious that when it got hard is when the progress stopped.

“It’s supposed to be hard. If it wasn’t hard, everyone would do it.” Jimmy Duggan – A League Of Their Own

Security guru Bruce Schneier was speaking at a local ISSA meeting recently and when asked about why security is not being addressed better he stated that one of the big reasons is that it is hard and complex at times to secure our technology.  And he is right, security is hard.  It is hard because of our poor planning, lack of inclusion, pick the reason and I am sure there is some truth to it.  But he went on to say that it is not going to get any easier any time soon.  Yes, we will get better tools, but the nature of what we have built and implemented will still make security hard.  We need to admit it will be hard and not sugar coat that fact to management.

Management also needs to clearly understand as well that security is not perfect.  The analogy I like to use is banks.  I point out to people the security around banks.  They have one or more vaults with time locks.  They have video cameras.  They have dye packs in teller drawers.  Yet, banks still get robbed.  But, the banks only stock their teller drawers with a minimal amount of money so the robber can only get a few thousand dollars in one robbery.  Therefore to be successful, a robber has to rob many banks to make a living which increases the likelihood they will get caught.  We need to do the same thing with information security and recognize that breaches will still occur, but because we have controls in place that minimizes the amount or type of information they can obtain.

“There’s a sucker born every minute.” David Hannum

Finally, there is the neglected human element.  It is most often neglected because security people are not people, people.  A lot of people went into information security so that they did not have to interact a lot with people – they wanted to play with the cool tools.  Read the Verizon, Trustwave, etc. breach analysis reports and time and again, the root cause of a breach comes down to human error, not a flaw in one of our cool tools.  Yet what do we do about human error?  Little to nothing.  The reason being that supposedly security awareness training does not work.  Security awareness training does not work because we try to achieve success only doing it once per year not continuously.

To prove a point, I often ask people how long it took them to get their spouse, partner or friend to change a bad habit of say putting the toilet seat down or not using a particular word or phrase.  Never in my life have I ever gotten a response of “immediately”, “days” or “months”, it has always been measured in “years”.  And you always get comments about the arguments over the constant harping about changing the habit.  So why would any rational person think that a single annual security awareness event is going to be successful in changing any human habits?  It is the continuous discussion of security awareness that results in changes in people’s habits.

Not that you have to harp or drone on the topic, but you must keep it in the forefront of people’s mind.  The discussion must be relevant and explain why a particular issue is occurring, what the threat is trying to accomplish and then what the individual needs to do to avoid becoming a victim.  If your organization operates retail outlets, explaining a banking scam to your clerks is pointless.  However, explaining that there is now a flood of fraudulent coupons being generated and how to recognize phony coupons is a skill that all retail clerks need to know.

  • Why are fraudulent coupons flooding the marketplace? Because people need to reduce expenses and they are using creative ways to accomplish that including fraudulent ways.
  • What do the fraudulent coupons do to our company? People using fraudulent coupons are stealing from our company.  When we submit fraudulent coupons to our suppliers for reimbursement, they reject them and we are forced to absorb that as a loss.
  • What can you do to minimize our losses? Here are the ways to identify a fraudulent coupon.  [Describe the characteristics of a fraudulent coupon]  When in doubt, call the store manager for assistance.

Every organization I know has more than enough issues that make writing these sorts of messages easy to come up with a topic at least once a week.  Information security personnel need to work with their organization’s Loss Prevention personnel to identify those issues and then write them up so that all employees can act to prevent becoming victims.

Those of you closet box checkers need to give it up.  You are doing your organizations a huge disservice because you are not advancing information security; you are advancing a check in a box.

06
Nov
14

The ASV Process Is Broken – Part 3

So what are my ideas on fixing the ASV process?

Modify The ASV Program

The conditions that drove the ASV process originally made sense.  Vulnerability scanning tools were predominately open source and anyone could do scanning and just about anyone was doing vulnerability scanning.  The results produced out of the open source tools could be highly questionable at best and the reporting was haphazard and about as trustworthy at times as a three dollar bill.  Even in large organizations, the people doing the vulnerability scanning did not necessarily have networking, security or even IT backgrounds.  Then there was a tremendously high false positive rate out of the open source tools.  As a result, most organizations ignored the results they received because they found that they could not be trusted.

The purpose of the ASV program was to bring some sanity and professionalism to the vulnerability scanning process.  MasterCard invented the ASV program (it was not called ASV then) back in 2005.  A test network was built and prospective ASVs were required to run their vulnerability scanners against this network and produce results which were then reviewed by MasterCard.  It was a much a test of the vulnerability scanning tool as it was of the person running the tool.  When the program transitioned to the PCI SSC, the Council added a multiple choice test to the process, but the virtual network testing and report review is still part of the process.

The trouble with this process is that the vulnerability scanning tool is no longer the problem.  Every ASV uses a commercial vulnerability scanning tool from either Tenable, Qualys, Saint, Tripwire or similar commercial tool vendor these days because they cannot afford to do otherwise.  Since these tool vendors are also ASVs, requiring a vulnerability scan for ASV certification has become a truly pointless exercise.  Other than the possibility of not properly entering the IP addresses to be scanned and running the wrong scanning policy, there really is very little that someone can screw up with a scanning tool.

The skill in vulnerability scanning today is reviewing the results, dealing with false positive results, working to address results with compensating controls and, with the Councils new edict on combining reports, working to get passing quarterly scans.

Therefore, in my opinion, training and testing of ASVs should be focused on the following.

  •  Determining the scope of vulnerability scanning.
  • Vulnerability scanning methodology.
  • Interpreting vulnerability scanning reports to confirm knowledge of the process and the meaning of the results.
  • What constitutes a false positive result and how to document a false positive result.
  • Development and documentation of an appropriate compensating control for a vulnerability.
  • Process for how to produce an acceptable passing scanning report from multiple reports.

And let us not limit ASV certification to just independent consulting firms.  As with the internal security assessor (ISA) program, open the ASV program to internal personnel as well.  Most large companies have independent vulnerability scanning teams that are as capable to more than capable than their ASV brethren.  There is no longer any reason that these internal people cannot do the ASV scans particularly if they meet the same standards and qualifications.

Approved Vulnerability Scanning Tools

I am not suggesting that the Council needs to develop a certification process for these tools as there are already plenty of sources that assess such tools.

The Council would publish a list based on the criteria developed by one or more independent tool assessment sources.  This list would define those tools acceptable to use for ASV vulnerability scanning.  The PCI DSS should then require that the QSA confirm that the vulnerability scanner used by the ASV is on the list in addition to confirming scope and the scanning policy used.

Require A Vulnerability Scanning Methodology

With the PCI DSS v3, the Council now requires penetration testers to use a documented and industry accepted penetration testing methodology.  Yet, there is no such requirement for vulnerability testing.

Most vulnerability scanning is done using what I call the “toss it against the wall and see what sticks” approach.  Basically, every possible vulnerability is run against every device.  Most commercial vulnerability scanners interpret banners, signatures and other markers to trim the list of vulnerabilities to be tested based on what they believe the target to be.  However, when you are scanning an external network blind, scanners cannot always properly interpret what an IP address resolves to as a device because of the mix of responses that they receive.  As a result, scanners do not necessary trim tests increasing false positive results or they trim them too much and the test is not complete.

Then there is the automated nature of today’s vulnerability scanning.  While I understand the desire to reduce costs of vulnerability scanning, the “point and click” nature of today’s ASV scanning has made it flawed.  And it gets worse as organizations get passing scans.  As a QSA, I cannot tell you how many passing scans I have reviewed where an organization could be hacked six ways to Sunday with the remaining vulnerabilities.  As a security professional, it scares me to death.  But as a QSA, while I can bring these up, they get no play because they do not have a CVSS of 4.0 or greater.  You hope that these vulnerabilities get picked up in an organization’s penetration test.

But there is no guarantee of that happening because the penetration tester’s vulnerability scanner may or may not pick up the same vulnerabilities.  As a result, part of the penetration testing methodology should include a review of all vulnerabilities found since the last penetration test and those should be tested for in the current penetration test to ensure they have been addressed.

Obviously, I have a preference to the methodology I discussed back in Part 2.  But there are a number of methodologies posted out on the Internet from a variety of good sources.  All I ask is that the vulnerability scanning methodology be integrated with the penetration testing methodology so that there are not gaps in coverage.

Require Monthly External Vulnerability Scanning

Before everyone panics, I am not asking that ASV scans be run monthly.  Although if the ASV program is modified, for organizations with internal ASVs that is a possibility.  I would still require the quarterly ASV scan, but I would add in monthly scans run by anyone deemed qualified as is allowed for internal vulnerability scans.

My primary rationale for this recommendation is driven by this simple fact.  When the dominant solution vendor releases patches on the second Tuesday of every month and the vast majority of those fixes have a CVSS score of 4.0 or greater, anyone that thinks quarterly scanning keeps them secure is seriously kidding themselves.  Not that a lot of security professionals bought into the quarterly vulnerability scanning requirement, even as a bare minimum.  But without the standard requiring it, a QSA has no leg to stand on other than to intimidate and shame people into doing monthly scanning.

Even if you are not Microsoft centric in your external environment, with the breaches that have occurred and the revelations of Shellshock and Poodle, it is painfully obvious that the quarterly requirement is not going to keep organizations secure.  I got a lot of calls after both of these vulnerabilities were announced with clients asking if their passing scans were no longer valid.  I was a bit schizophrenic in my thoughts.  On the one hand, I was glad they were at least thinking about the security implications of these vulnerabilities.  But their concern about their passing scans just highlights the importance of meeting a PCI requirement and passing their PCI assessment versus being secure.  Because, while I only got a few calls, you know that there are too many people that are congratulating themselves on dodging the bullets of Shellshock and Poodle because of the fortuitous timing of their  quarterly scans and that they got an additional 30, 60 or even 90 days to address them.

Then there are those organizations that run solutions such as IBM’s Websphere or Oracle’s eCommerce suites.  Both of these vendors not only patch their own application frameworks, but they also release those patches to the underlying operating systems that are compatible with their application frameworks.  But worse, these vendors do not release monthly patch releases, they do patch releases on quarterly, semi-annual or even annual bases.  As a result, there is a high likelihood that some operating system patches could be left out of these releases due to compatibility or timing issues.  The work around is to mitigate any remaining vulnerabilities through additional logging, additional monitoring, changes in firewall rules, changes in IDS/IPS rules, etc.  The additional vulnerability scanning could help organizations identify these issues and address them quicker than quarterly.

A side benefit of monthly scanning will be improving the ability of organizations and their QSAs to determine if an organization’s patching and mitigation processes are working according to requirement 6.1.  Quarterly scans typically document a lot of vulnerabilities, mostly those under a CVSS of 4.0.  As a result, whether or not an organization is properly managing their environment can be very difficult and time consuming leading to missing items that should be addressed.  Having reports more often can facilitate getting these issues addressed sooner rather than later and keeping the volume lower and less daunting.

The bottom line in all of this is that monthly scanning is required to even have a chance at being secure these days.  Yet the vast majority of organizations are only doing quarterly scans and thinking they are secure.  That practice must change.

So there we have it.  My thoughts on the ASV process and how I would go about fixing it.

01
Nov
14

The ASV Process Is Broken – Part 2

The next reason I believe the process is broken is with the automated scanning processes.  They do not seem to be accurately assessing the security of Web servers, firewalls, routers and other externally facing devices that form an organization’s perimeter.  In my opinion, it seems that in our drive to bring down the cost, we have created a false sense of security by having the concept of a “passing” scan and only requiring scanning on a quarterly basis.

It is hard to believe that external vulnerability scanning came into vogue more than a decade ago.  There were no automated solutions.  No popping out to a Web site, entering IP addresses and scheduling a scan.  It was all manual.  A person was required to configure the scanning solution and then launch the scanner.

There was a proven methodology used for approaching vulnerability scanning.  That methodology involved approaching the organization as an attacker would. .  The security professional would go out to domain registrars and other official sources to obtain potential IP address ranges, registered domains and other public information.  Then the professional would develop a profile of the organization’s external security posture using Nmap and similar tools as well as running non-intrusive scans to identify systems and potential entry points.  But even more insidious, the security professional would do what became to be known as “Google Hacking” to find out what, if any, information was available on the Internet that might facilitate the attack.  Log data, firewall configurations and even authentication credentials could be obtained through Google Hacking.  Even today, while Google does its best to minimize the information they collect, it is amazing the amount of information that can still be obtained with specially crafted Google searches.

Once the preliminaries were done, it was time to actually scan.  The most accepted approach used at that time was to run two scans, one fully external with security measures enabled and one with testing done inside the DMZ.  The scan done with the security measures enabled was to confirm that an organization’s firewalls, load balancers or other front ends were functioning as configured.  Essentially this scan showed what an attacker would discover in their scanning of the organization’s external presence.  There were two purposes of the second scan.  The first was to provide an inventory of vulnerabilities that were being obfuscated by the security measures so that they could be addressed or mitigated.

The second purpose of the DMZ scan was for what some would call today a form of “threat intelligence”.  The second scan provided a view of what an attacker would encounter if they were able to circumvent the organization’s external security.  It answered the question of how large the attack surface was in the DMZ should it be compromised?  What else could be subverted and used to further an attacker’s goal of getting even deeper into a network?  The point of this part of the exercise was to provide the organization with an idea of what to look for should anomalies begin to appear in log data.

Throughout this scanning process, separate scans were run for firewalls, routers, load balancers, Windows devices, Linux devices, etc.  The rationale for that was to obtain clear results based on the type of device.  No reason to toss a bunch of Windows vulnerabilities against a Cisco firewall or F5 load balancer and then have to sift through the results to remove the false positive entries generated by the Windows devices they protect.

All of this information would be collated, analyzed and an actionable report produced for management and security professionals to absorb and understand the organization’s true external security posture.  Once the draft report was delivered there would be one or more calls or meetings with the organization’s security personnel to tweak the message of the results, develop action plans to address the results and then a final meeting with management and security personnel to deliver the final report.  At the end of this process, management and security personnel knew exactly where they stood from a security perspective and what needed to be done to ensure that they remained secure going forward.

Obviously, such an approach is fairly comprehensive and that costs money to produce.  And it was the cost that drove the automation of the process so that it was less labor intensive.

In the mad dash to the lowest possible cost, the following is what typically happens today.

Organizations subscribe to an ASV vulnerability scanning service.  If the organization is lucky, they go through a scoping call with a human being at the ASV service to confirm the IP addresses that are in-scope for PCI compliance.  Having been on a number of these calls with ASVs, it is more of a questionnaire approach and there is very little done by the ASV to actually confirm the scope of the scanning.  It is more or less a conversation that explains how to enter the IP addresses into the scanning tool’s Web interface and to schedule a scan, not a diligent conversation on whether or not a device is in scope or not.  There is typically a brief discussion on how to use the Web site to dispute scanning results that caused a non-passing scan.  In total, the call takes 20 to 30 minutes at which point customers are essentially on their own.

Scans are run on a quarterly time period.  Scans may be rerun if vulnerabilities are identified that result in a non-passing scan.  Once a passing scan is obtained, a passing certificate is generated and the organization moves on to the next quarter.  Every now and then a result may be disputed, but those situations are typically rare.  The bottom line is that the ASV process is fairly automated with very little, if any, human intervention.

From a tool perspective, the vulnerability scanner is tossing every vulnerability at every IP address they are given.  Firewalls, load balancers, switches and routers are tested for Windows and Linux vulnerabilities.  False positive results can be rampant but most scanners weed out the obvious false positive results based on OS and device signatures.  But even if a few false positives remain, as long as they have a CVSS score of less than 4.0, no one cares.  And the scanner is configured to run as quickly as possible because it needs to be available for the next customer so that the ASV does not have to have an individual scanner for every customer.

All of this is driven by customers’ desire to minimize cost as much as possible.  Thus, in order to stay in business, the ASV scanning service providers have done an admirable job of driving down cost.  But at what other costs?

While I think we can all agree that the current approach is flawed, the Council would point to the fact that the quarterly ASV scanning requirement is only a minimum and that organizations should be doing their own external vulnerability scanning more often than quarterly.  And that is what is missed by a lot of organizations is that they need to do more than what the PCI DSS requires to be secure.  And no matter how loud that message is repeated, it seems to get missed over and over again.

However, a lot of organizations do not do anything more than the PCI DSS requires.  We have this “bare minimum” mentality because, if it is not mandated in the standard, then we must be secure if we do only what is required.  This results in the false sense of security that organizations have when they only do what the PCI DSS or any other security framework for that matter mandates.

In addition, I cannot tell you how many organizations totally write off vulnerabilities that have a CVSS of less than 4.0 as “trivial” representing no threat.  There is this mistaken belief that, because the CVSS score does not warrant immediate patching, it is not a threat.  However, when you have them in relative abundance and of certain types, those “trivial” vulnerabilities can be put together in a particular sequence and used to compromise a network.  To the shock and horror of a lot of organizations, their security is breached in moments during their penetration test using those “trivial” vulnerabilities.

Then we have the limitations of the tools themselves.  Their vulnerabilities are constructed for the general audience, not a specific network.  As a result, depending on security measures and configurations, a particular vulnerability for say HTTP or FTP might not register due to a load balancer or other devices in the communication path.  Yet with a minor tweak, the same vulnerability could actually work.  And that was another point of doing that second scan inside the DMZ back in the “old” days.  Taking the two scans allowed you to correlate that while HTTP or FTP was open; it was vulnerable because that vulnerability was shown on the internal DMZ scan.  As a result, you could take additional steps to ensure that the vulnerability could not be exploited from the Internet.

The bottom line in all of this is that ASV scans have become a pointless piece of the compliance assessment process.  It has not been because the ASVs are bad, it is that the process they are forced to follow is not doing what it once did at the beginning.  It does not help that merchants and service providers treat ASV scans as a “check the box” activity and that because they do something quarterly they are secure.

If the process is broken, what should be done to fix it?  That is the subject of my next post.

19
Oct
14

The ASV Process Is Broken – Part 1

The topic of ASV scanning came up as usual at the 2014 PCI Community Meeting.  The questions all seemed to revolve around how to obtain a passing scan.  What the Council representatives suggested is that multiple scans can be put together to create a passing scan.  Unfortunately, what the Council keeps suggesting as the solution is impossible to implement and here is why.

In a typical environment, an ASV customer logs onto their account with the ASV and schedules their ASV scans of their PCI in-scope assets.  The customer may also add or subtract the number of IP addresses that are scanned as the scope of their external environment may change.  Depending on a number of factors, there may be one scan or multiple scans.  The vulnerability scans are executed on the schedule and the results are returned to the customer.

If there are false positive results or results the customer does not agree, they can apply back to the ASV to have those results removed.  If there are actual vulnerabilities, the customer can contact the ASV with how they have mitigated the vulnerabilities and the ASV can either accept those mitigates and give the customer a passing scan or allow the results to stand.

So where are the problems?

Whether or not the Council acted on facts that cheating was occurring or anecdotal evidence is unknown.  But because of the potential for cheating by customers, the Council mandated a number of years ago that ASVs lock down their scanning solutions so that customers cannot modify anything regarding testing other than the IP addresses involved.  The ASV Program Guide v2.0 on page 11, states:

“However, only an authorized ASV employee is permitted to configure any settings (for example, modify or disable any vulnerability checks, assign severity levels, alter scan parameters, etc), or modify the output of the scan.  Additionally, the ASV scan solution must not provide the ability for anyone other than an authorized ASV employee to alter or edit any reports, or reinterpret any results.”

So right off the bat, the Council’s recommendation of “putting together multiple reports” is not as easily accomplished based on their earlier directives.  That is because it will require the ASV’s customer to get the ASV to agree to put together multiple reports so that they can achieve a passing scan.  That implies that the ASV’s solution will even accommodate that request, but then the ASV needs to be agreeable to even do that task.  Based on the Council’s concerns regarding manipulation of scanning results and the threat of the Council putting ASVs in remediation, I do not believe the ASVs will be agreeable to combining reports as that would clearly be manipulating results to achieve a passing scan.

But it gets worse.  As a lot of people have experienced, they can scan one day and get a passing scan and then scan a day or even hours later and get a failing scan.  The reason this happens is that the vulnerability scanning vendors are adding vulnerabilities to their signature sets as soon as they can, sometimes even before vendors have a patch.  As a result, it is very easy to encounter different results from scan to scan including failing due to a vulnerability that does not yet have a solution or the vendor only just provided a patch.

But if that is not enough, it gets even worse.  Statistically, the odds of getting a passing scan are nearly impossible and gets even worse if you are only doing quarterly scanning.  A review of the National Vulnerability Database (NVD) shows that 94% of vulnerabilities from 2002 to 2014 have a common vulnerability scoring system (CVSS) score of 4.0 or greater.  That means that it is almost impossible to obtain a passing vulnerability scan, particularly if you are only scanning quarterly, when vulnerabilities are announced almost daily and vendors such as Microsoft are coming out monthly with patches.  Those of you scanning monthly can attest that even on a 30 day schedule, a passing scan is nearly impossible to get.

For an organization that has only one Web site, this situation is likely not a problem.  But when organizations have multiple Web sites which a lot of organizations large and small have, you are really struggling in some cases to get passing scans.

But let us add insult to injury.  A lot of organizations have their eCommerce environments running on multiple platforms such as Oracle eCommerce or IBM Websphere.  In those examples, this situation becomes a nightmare.

Platforms such as those from Oracle and IBM may run on Windows or Linux, but Oracle and IBM do not allow the customer to patch those underlying OSes as they choose.  These vendors ship quarterly, semi-annually or on some other schedule, a full update that patches not only their eCommerce frameworks, but also the underlying OS.  The vendors test the full compatibility of their updates to ensure that the update will not break their frameworks.  In today’s 24x7x365 world, these vendors can run into serious issues if eCommerce sites begin to not function due to an update.  However, that also means there is the possibility that critical patches may be left out of an update due to compatibility and stability reasons.  As a result, it is not surprising that in some updates, vulnerabilities may still be present both those that are new and those that have been around for a while.

But if Oracle and IBM are not patching on 30 day schedules, that means there is a high likelihood that the scans will not be passing.  This means that the customer must go to their ASV with compensating controls (CCW) to mitigate these vulnerabilities to obtain passing scans.

The bottom line is that the deck is stacked against an organization obtaining a passing scan.  While the Council and the card brands do not recognize this, the rest of the world sure has come to that determination.

In Part 2, I will discuss the whole ASV approach and how I believe the drive to be the cheapest has turned the ASV process into a mess.




Announcements

FishNet Security is looking for experienced QSAs for their PCI practice. If you are an experienced QSA and are looking for a change, go to the Web site (http://www.fishnetsecurity.com/company/careers), search for 'PCI' and apply.

If you are posting a comment, be patient, as the comments will not be published until they are approved.

If your organization has a PCI opportunity, is in need of assistance with a PCI issue or if you would like the PCI Guru to speak at your meeting, you can contact the PCI Guru at pciguru AT gmail DOT com.

I do allow vendors to post potential solutions in response to issues that I bring up in posts. However, the PCI Guru does not endorse any specific products, so "Caveat Emptor" - let the buyer beware. Also, if I feel that the response is too "sales-ee", I reserve the right to edit or not even authorize the response.

Calendar

December 2014
M T W T F S S
« Nov    
1234567
891011121314
15161718192021
22232425262728
293031  

Enter your email address to subscribe to the PCI Guru blog and receive notifications of new posts by email.

Join 1,030 other followers


Follow

Get every new post delivered to your Inbox.

Join 1,030 other followers