Author Archive for PCI Guru

07
Jul
20

The Security/Compliance Disconnect

I was speaking with someone recently and they tossed out one of the most despised phrases I know:

“Compliance is NOT security!”

I told them to stop right there and take it back or our discussion was over.  Since they really wanted my opinion on the actual topic at hand, we continued.  But I felt the need to explain why I find this statement so repulsive.  Which, by the way, has nothing to do with being an auditor.

The first point I make when discussing this phrase is about security frameworks and that they are merely the foundation for a good security program, not the whole enchilada.  They are only the starting point and that great security programs must go well beyond these frameworks.  The bottom line is that achieving compliance with any security framework means your organization can execute the basics consistently.

The next important point I like to make to people who spew this trope is that if they read any of the data breach or security reports from the likes of Verizon, Trustwave, Security Metrics or any other recognized security company, what do you see?  That the organizations breached could not comply with any of the recognized security frameworks be it PCI DSS, CoBIT, NIST, HIPAA, pick your poison.  Unfortunately, as these reports point out in annoying detail, organizations rarely execute the basics consistently because if they did, they would likely not have been breached.  Which really punches a huge hole in the whole compliance does not equal security argument.

Another point about this statement is that organizations high five over being compliant with a security framework when it really means that they are mediocre at best.  Yet time and again I hear back after PCI assessments that management is so proud that they were assessed compliant.  “Yea, we achieved mediocrity!”

Finally, there is how do you measure how well your security program is operating?  You must have a “yardstick” of some sort and to do that, so you need one of the security frameworks as your yardstick.  Given that these frameworks are only the basics, you need to add in all the additional controls your organization has in place that go beyond the framework.  That activity typically identifies a huge gap in the security program – there are few if any additional controls.  So, there you sit with say the PCI DSS as your “yardstick” and your organization cannot consistently execute the basic security controls in that framework.

Yeah, that is it!  It is the yardstick’s fault!

26
Jun
20

The 2020 PCI Community Meetings Go Virtual

A lot of us remember when the 2017 NACM in Orlando was cancelled due to Hurricane Irma.

Troy Leach announced on Twitter yesterday that the 2020 NACM would be virtual as would all of the other Community Meetings.

It will not be the same, but at least we will be virtually together.

20
May
20

DevOps And PCI – Part 2

In the first post on this topic we discussed the terminology of DevOps and how segregation of duties can get complicated with DevOps.  In this post we will continue to investigate DevOps and discuss the issues you can encounter with change control, documentation and PCI scope.

Change Control

These days it is not unusual to hear DevOps people be proud of hundreds or even thousands of implementations or deployments per day.  That is until someone like a PCI assessor starts inquiring about what, if anything, is done to formally approve all those deployments?  The conversation with developers typically begins to deteriorate as you discuss requirement 6.4.5.2 which states:

“Documented change approval by authorized parties.”

The normal response is that the approval is provided in Jira, ServiceNow or whatever change management tool is being used.  That leads to a discussion of the guidance for requirement 6.4.5.2 which states:

“Approval by authorized parties indicates that the change is a legitimate and approved change sanctioned by the organization.”

With the rapidity and the volume of changes, then next question asked is how can an authorized party assess a change is legitimate and sanctioned if they never actually physically see and review the change that is deployed?

This leads to a discussion of how Jira, Jenkins, Puppet, whatever CI/CD toolsets work along with the automation involved in the change process as well as the “controls” embedded in the workflow of those tools.  The bottom line of which is usually that the only potential human intervention in the process might occur if the code needs a manual code review.

What requirement 6.4.5.2 is about is ensuring that the change process involves human intervention by getting management to approve what is being put into production and that segregation of duties has been ensured and no fraud or other illegal activity has been introduced into the process.  The reason is that we are talking about code that is processing, storing or transmitting sensitive authentication data (SAD) or cardholder data (CHD).  The potential for implementing code that skims that information or does other nefarious actions is too great to just trust a fully automated process with no human intervention.  This risk can all be driven home with a discussion of the 2013 Target breach where the CI/CD process was compromised to repeatedly push malware to thousands of point of sale devices.

While I am only talking PCI in-scope code in this case, fair warning, HIPAA, SOX, GDPR and other regulations are going to require a similar control.  Sensitive information is worth too much today and there is just too much risk that people will take any opportunity to siphon off sensitive data any way they can if appropriate controls are not in place.  If your process is totally automated and cannot detect such fraudulent activities, the ability to put who knows what into the code is too easy.  The last thing any organization wants is to be breached and then try to defend itself when they had poor or no internal controls to prevent the breach.

The bottom line here is that in our haste to push out software we have compromised the controls in that process.  Those controls need to be put back into place to minimize the risk presented by pushing malicious software into applications without a thorough vetting by management.

Documentation

Another area where compliance falters with DevOps is documentation.  Confluence, SharePoint Wiki or a similar tool will be used for documentation and that is where most assessors/auditors will be pointed for their requests for formal documentation.

The first problem with this is when you are an outside assessor/auditor, because you do not have access to the internally used tool.  That can be remedied several ways, but it is always a hurdle because insiders are so used to the fact that everyone they typically work with has access.

Once the assessor/auditor has access, the next problem for all assessors/auditors is finding what they need for their assessment/audit.  Regardless of whether the assessor/auditor gets PDFs or has online access, the most common reason for this issue is terminology.  A lot of times what an assessor/auditor is trying to find will be referred to by the organization in terms that are not consistent with industry or technology accepted terminology.  While all of these documentation tools have search capabilities, searching the document trove for what an assessor/auditor needs for evidence can be highly problematic.  Never mind the fact that clients get frustrated as well because the evidence exists, but the assessor/auditor cannot find it.

Related to these documentation systems is the fact that it can be difficult, if not impossible, for the assessor/auditor to get hardcopy or even usable PDFs of the documentation.  Let us face it, screen shots while readable can miss sentences at the end of a screen and therefore be missed altogether.  As a result, obtaining usable and legible evidence for the assessor’s/auditor’s work papers is not readily possible let alone having it searchable.  The fix to this is to use a browser extension or addon that will create a PDF or image of an entire page.  But that too can run into issues if the organization has locked down their browsers and do not allow such installations.

Regardless of Agile/Scrum or Waterfall, the next problem with documentation is the fact that the documentation is limited or simply does not exist.  I have encountered more and more organizations that again point to The Agile Manifesto, Scrum and the like and state that none of these approaches specify that documentation is required.  It seems that the age-old adage of “if it was hard to develop, it should be hard to understand” is back in vogue.  Never mind the fact that with hundreds or thousands of deployments a day, keeping up with documentation outside of can be impossible.

Consistent use of a change management ticketing system such as Jira or ServiceNow also can be an issue.  It seems that some organizations have exceptions that do not require every change to their environment be entered into their change management solution.  Worse, the criteria used to determine what is and what is not entered is not consistently applied because the criteria were never officially documented nor formally approved.  As a result, there is no way to rely upon the information contained in the change management system to determine that change management is performed as required.

As a result, I am never surprised to have organizations scrambling to develop even business and IT policies, standards and procedures in addition to network diagrams, data flow diagrams, application documentation, database schemas, operations documentation and a whole host of other missing or incomplete documentation.

PCI Scope Implications

Lastly, there is the scoping issue related to the DevOps infrastructure.  Not all of it is usually in scope, but that all depends on how it has been implemented.

At the very least, the Jenkins, Puppet or Ansible portion of the infrastructure are going to be in-scope for PCI.  The reason is that those components feed the application updates into the cardholder data environment (CDE).  So those are considered “Connected To” systems and must be properly configured and secured to be PCI compliant.

Because these CI/CD solutions are “Connected To”, this can become problematic because of who has access to Jenkins, Puppet, et. al.  As I spoke of earlier, because of poor segregation of roles in the Active Directory system, it can turn out that developers have access to these systems and therefore come into scope for PCI compliance as well.  As a result, the whole concept of development separate from production for requirement 6.4.1 does not exist.

Obviously, this segregation of development and production problem only gets worse if you drag even more of the development infrastructure into scope.  Therefore, you want to ensure that only the Jenkins, Puppet, Ansible portion of CI/CD is in scope.

This will mean moving Jenkins, Puppet, Ansible, etc. into your “Connected To” or “Shared Services” network segment.  This can create some issues with the rest of the development environment because of firewall rules and access to it through a Jump Server.  So simply moving that solution into the new network segment may not be as simple as it appears.

Development Metadata

Before we go, there is one more topic that needs to be discussed and that is the metadata in all these development solutions.

We have touched on the controls surrounding the development toolsets, but we have not discussed securing these toolsets and the risks they present.  This may seem a bit odd because since when have we worried about the security of Visual Studio or other integrated development environments (IDE).  However, with the implementation of CI/CD solutions, all these tools become interlinked and integrated.  Essentially, all these tools make up an automated assembly line for building applications.

But even more importantly, for these tools to work together seamlessly, they need to share metadata about what they are doing.  This metadata might seem like it is benign information, but it is particularly important and controls how the applications are built, tested and deployed.  Essentially the metadata is the “secret sauce” that makes your application work as an application within your organization.

We have already discussed the security controls that will be required around the deployment toolset.  But the rest of the development toolset is also going to require security and controls to be implemented to ensure that your software factory’s assembly line does not become a huge risk for attacks.  Attacks that could maliciously modify your applications to stopping the assembly line altogether.

Before you think that this is unrealistic, I would again point you to the infamous 2013 Target breach.  I wrote about the breach at the time and walked people through how what was known about the breach at that time would have made the breach possible.  The success of that attack was to compromise the CI/CD process to implement their malware that skimmed cards in the point of sale system.  So, the idea that development environments are not a viable attack point is not out of the realm of possibility.  And it gets even worse when you add in the use of contract workers to the development process.

So, what should an organization do to address these risks?  I would recommend securing the entire application development environment to PCI configuration standards so that security monitoring of the entire environment can be performed.  That does not mean that all the environment needs to reside in your ‘Connected To” or “Shared Services” DMZ with the CI/CD solution.  But I would create another DMZ to contain the rest of the toolset that feeds the CI/CD solution.  Servers should be properly security hardened and monitored as though they are in-scope for PCI compliance even though they are not in-scope.

There you have it.  The basics of how Agile and PCI can coexist.

17
May
20

DevOps And PCI – Part 1

DevOps are all the rage in organizations that develop applications.  The move to become “Agile” through the implementation of methodologies such as Scrum to replace the traditional waterfall SDLC is ongoing in most organizations.  But these changes can create compliance issues with the PCI standards regarding software development.

Understanding The Terminology

First and foremost, we need to address the terminology surrounding DevOps.

But before we talk about those specific terms, we need to address the elephant in the room which is “Agile”.  The Agile approach to development traces its history back to early 2001 when a group of developers met at a Utah ski resort.  The result of that meeting was ‘The Agile Manifesto’.  However, the roots for Agile were sown even earlier as application development became unable to keep pace with business changes starting in the late 1980s.

The important thing to remember about Agile is that it is not a methodology.  It is merely a set of values (4) and principles (12) related to the development of software.  The Agile Manifesto never describes a roadmap or steps to follow as to how those values and principles should be used.  So, to refer to Agile as a methodology is a misnomer but you will constantly encounter it being referred to as though it were a methodology.

Interestingly enough, the methodologies used with the Agile approach were actually developed before Agile.  Of the number of them that sprang up in the 1990s, Scrum seems to have won out when it comes to a methodology.  Scrum was one of many methodologies such as Kanban, Crystal Clear, Extreme Programming, Feature Driven Development and Dynamic Systems Development Method that came out at that time to address the delivery of software solutions in a timelier manner.  But while Scrum is the most followed methodology, it can also include some of these other methodologies such as Extreme Programming (XP) for example when used.

Scrum involves three types of roles.

  • Product Owner: The Product Owner needs to be a person with vision, authority, and availability because they are responsible for continuously communicating the vision and priorities to the development team.
  • Scrum Master: The Scrum Master is not a project manager.  The Scrum Master’s primary responsibility is to remove any impediments that are obstructing the team from achieving its sprint goals.  The Scrum Master also is the primary contact with the Product Owner.
  • Team: The Scrum team is responsible for completing the work.  For application development, a Scrum team can contain anywhere from three to nine members.  For software projects, a typical team includes a mix of software engineers, architects, information security personnel, programmers, analysts, QA experts, testers, and UI designers.  The team is responsible for determining how it will accomplish the work to be completed.

The final term from Scrum that needs to be defined is Sprint.  A Sprint is a one month or less in duration project that will result in a releasable increment of a product, in this case, an application or application enhancements.  When a Sprint’s horizon is too long the definition of what is being built may change, complexity may change, and risk may change.  The concepts of Sprints are to enable predictability by ensuring inspection and adaptation of progress toward a Sprint Goal at least every calendar month. Another benefit is that Sprints limit risk to one calendar month of cost.

Once defined, some of the key characteristics of Sprints are:

  • No changes are made that would endanger the Sprint Goal;
  • Quality goals do not decrease; and,
  • Scope may be clarified and re-negotiated between the “Product Owner” and “Team” as more is learned.

With these behind us, let us now turn to the terms DevOps.

DevOps is a merging of development and operations staff to work together to develop and implement solutions that will essentially run 24x7x365 with (hopefully) minimal operational interaction.  DevSecOps merely formally adds in the collaboration of information security into that mix even though information security should be included in DevOps as well.

The final topic in our discussion of terminology regards the tools used by DevOps.  While there are a number of vendors in this space, the “Big Dog” at the moment in DevOps is Atlassian with their tools Confluence and Jira.  Another “Big Dog” is Microsoft’s GitHub and the other “Big Dog” in the DevOps tool world is Jenkins which is open source from CloudBees.

  • Confluence is used as a documentation repository for such items as policies, standards and procedures as well as business, application, network and other important documentation.
  • Jira is used as a project and change management ticketing system.
  • GitHub is used to manage the versions of applications.
  • Jenkins is used for automating the build, testing and deployment of applications into production.

All of these tools have competitors from vendors such as ServiceNow, Puppet, Ansible, Chef, Google, and other commercial and open source development and operations tool vendors.  Regardless of vendor, all solutions seem to have these three basic components of documentation repository, project/change management and deployment automation.  It is also not unusual to find multiple tools in place particularly with Jenkins, Ansible, Puppet and Chef.

Segregation Of Duties

The first and most contentious issue that comes up with DevOps is the segregation of duties.  This is typically one of the biggest discussions/arguments an assessor/auditor will get into regarding DevOps is when Agile fans argue that segregation of duties is inconsistent with Scrum, Agile and DevOps.  Their primary reason will be to point to the fact that nowhere in any of the documentation regarding these topics is the term ‘segregation of duties’ or the requirement to ensure segregation of duties.  They would be correct in that regard.

Unfortunately, corporate life is not driven by Scrum, Agile or DevOps in a vacuum.  Corporations are still required to comply with laws, contracts and regulations promulgated on them by government entities, business partners, financial institutions and other parties regardless of what is in their methodologies and approaches.  So, while the argument can be made that the methods and approaches do not state anything on the subject, there are other documents, contracts and requirements that do state it is required.

Whether we are discussing PCI, NIST, SOC, COBIT or any other recognized audit or compliance programs, segregation of duties between roles is and always has been required.  It is one of the key principles to ensure that people do not have the ability to corrupt a process because they have too much control over that process.  The reason it is, that time and again one of, if not the primary root cause of such illicit activities, is the failure to segregate duties and roles thus allowing one person too much control over a process.  The concept of segregation of duties being that the more individuals involved in a process the less likely a process can or will be abused.

In DevOps, the issue of segregation of duties gets complicated because it gets extended into the tools used in the process.  The concept of continuous implementation (CI)/continuous deployment (CD) relies heavily on the use of tools such as Jira and Jenkins to enable such an approach.  This means that the assessor/auditor needs to look into who has access to these tools and what rights they have to influence the workflows that exist in these tools.

This gets even more complicated by the fact that this requires analysis of user and access control information from tools such as Active Directory, RADIUS and even the tools themselves.  In my experience, it is not unusual to peel the onion on these access controls to reveal the fact that segregation of duties really does not exist as thought because all roles are granted to everyone in DevOps and the organization is relying on individuals’ honesty to ensure compliance.

DevOps also can suffer from segregation between production, quality assurance (QA), test and development environments.  This is because a lot of organizations that move to DevOps have the mistaken belief that the “Operations” and “Security” components become part of the development group.  The argument will be made that Agile is all about “breaking down silos”.  While that is true, the mistake they make is that Agile and Scrum were not a call to abrogate the knowledge and controls that all of the players involved bring to the table as separate disciplines.  The goal is to make the disciplines better work together to achieve a common goal in a Sprint.

Where this manifests itself most often is that developers have unfettered access to the production environment.  In a DevOps environment, it is not unusual to find developers scattered all throughout the environment.  They are developing code, they are operating production, they are diagnosing bugs, they are everywhere with no delineation of roles and responsibilities.  It is essentially a free for all.  Everyone pitches in where they need to be involved.

This organized chaos is supposedly “controlled” by Jira through its ticketing.  Agile advocates will claim that since everything has a ticket (not always a true statement) that they maintain segregation through Jira.  They will show the tickets to the assessor/auditor and display that there are different names on the ticket for the developer, the QA person, the people who approved promotion, etc.  While this is true, as I described earlier, the access controls will show that virtually everyone they gave as evidence of segregation can fulfill any of those roles whenever they so choose.  By definition, that is not segregation of duties because there are no actual controls in place to stop someone from running the whole process.

The bottom line in this discussion is that the segregation of duty controls in an Agile environment is usually illusory.  As such, it is management’s responsibility to periodically ensure that segregation of duty controls are truly implemented and testable.

In the next post we will discuss documentation, change control and PCI scope in an Agile environment.

30
Apr
20

The Last (Hopefully) Scoping Discussion

Back in May 2017, the Council finally issued their long awaited Information Supplement on Scoping and Network Segmentation.  Based on some questions I have received since then, there are apparently a lot of people that still have not read the official information supplement.

So, I am invoking “RTFM” which means the first order of business is to get everyone to read the information supplement before asking questions.  The second order of business is to forget everything that was discussed in the Open PCI Scoping Toolkit as the Council will tell you it does not apply and never did apply.  Even though they never offered any alternative until the publication of the aforementioned information supplement.  So, throw away all your copies of the Open PCI Scoping Toolkit as it is not usable anymore.

With the Council’s information supplement, there was a change in terminology in how we refer to the various network segments and what is in scope.  As you will see, the Council’s approach has simplified the scoping classifications.  Because of the pervasiveness of the Open PCI Scoping Toolkit, I have included some references to the categories used in the Toolkit to clarify the Council’s terminology.

  • Cardholder Data Environment (CDE) Systems – These systems are always in scope for PCI compliance. These are systems that are either: (1) a system that directly processes, stores or transmits cardholder data (CHD) or sensitive authentication data (SAD), OR (2) a system or component that is on the same network segment (i.e., same network subnet or VLAN) as a system component that directly processes, stores or transmits CHD/SAD.  With the Open PCI Scoping Toolkit, these were considered ‘Category 1A/1B’ systems.
  • “Connected To” or “Security-Impacting Systems” – These systems are also always in scope for PCI compliance. These systems are basically those that directly connect to systems in the CDE or could influence the security of the systems or data in the CDE.  In the Open PCI Scoping Toolkit, these were the ‘Category 2A/2B/2C/2D’ systems.  Unlike in the Open PCI Scoping Toolkit, the Council chose to simplify things and have only one category versus the “shades of gray” approach.  That said, there are more detailed criteria defined on page 10 of the information supplement that define these systems.  Examples include, but are not limited to, Active Directory (AD) servers, RADIUS servers, TACACS+ servers, Security Information and Event Management (SIEM) solutions, Network Time Protocol (NTP) servers, Domain Name System (DNS) servers and Domain Host Control Protocol (DHCP) servers.  These systems and devices can also be considered as “Shared Services” because they provide service not only to the CDE but also to out of scope systems.
  • Out of Scope Systems – There are four criteria for these systems: (1) The system must NOT process, store or transmit CHD/SAD AND  (2) the system cannot be on the same network segment or subnet as the CDE. AND  (3) the system cannot directly connect to any other system or component in the CDE  AND  (4) The system does not meet ANY of the criteria described for “Connected To” systems.  If all of these criteria are met, then the system is out of scope.  In the Open PCI Open Scoping Toolkit these were the ‘Category 3’ systems.

As we have found out at the Community Meetings since the publication of the information supplement, the Council will demand you use their scoping terminology.  If you use the Open PCI Scoping Toolkit scoping categories, you will be asked to restate your questions or comments using their terminology.  So please from here on out use the Council’s terminology whenever discussing scoping categories.

Why Is Scoping A Problem?

Scoping is a problem because organizations think it is the QSA’s problem.  However, the PCI DSS states on page 10:

“At least annually and prior to the annual assessment, the assessed entity should confirm the accuracy of their PCI DSS scope by identifying all locations and flows of cardholder data, and identify all systems that are connected to or, if compromised, could impact the CDE (for example, authentication servers) to ensure they are included in the PCI DSS scope.”

Eight times out of ten, it falls into the QSA’s lap to determine and confirm PCI scope even though it is the assessed entity’s responsibility to define scope and the QSA’s role is to confirm that analysis.  This is why arguments over scope happen.  QSAs get into trouble because they follow the processes defined below and determine that the scope is not correct.  Had the assessed entity done their work, the argument likely would not have happened or at least would not have been as big as it became.

The purpose of this post is to explain what your QSA is doing when they asked for all the documentation and what they are doing that your organization should be doing before the QSA even shows up.  For QSAs, this is what you should be doing to ensure that the scope of your engagement is correct.

Follow The Data

The first thing that people seem to get wrong about scope is fixating on the storage of CHD and ignoring the processing and transmitting of CHD/SAD.  This is a big reason why voice over IP (VoIP) gets missed.  VoIP typically never stores CHD/SAD.  But when customers are making payments over the telephone, CHD/SAD is being discussed and that is what makes the telephone system a CDE and therefore in scope for the PCI assessment.

The key to resolving this is to follow the CHD/SAD through your networks.  When he was a Council trainer, Art (“Coop”) Cooper was famous for constantly telling his classes to, “Follow the data.”  Therefore, the data flow diagrams overlaid on your network diagrams are so especially important in determining PCI scope.  Done properly, these diagrams allow you to understand where the CHD/SAD flows through your organization (i.e., transmission), where it is processed, as well as where it ends up stored.

From that analysis, you can then document where, if anywhere, the CHD/SAD is encrypted and who manages the encryption keys.  If your organization manages the encryption keys, then you will need to prove and document that those intermediate devices between the encryption endpoints cannot decrypt the CHD/SAD in order to keep them out of scope.  If an outside third party manages the keys, then scope is reduced to where the encryption endpoint is in your environment.  For more about encryption and scope, see my Encryption Series of posts listed on the Post Series References page.

Once you have completed this activity, you have defined your CDE, likely many of them.  It is not unusual for organizations to have their VoIP network and solution as one CDE and then another for their eCommerce or brick & mortar retail.  But there could be even more CDEs depending on your environment.

One other caveat on scoping CDE.  Devices that are in the CDE that do not process, store or transmit CHD/SAD are in scope for PCI compliance.  These include devices and systems such as jump servers, switches, routers, Active Directory domain controllers, DHCP servers, DNS servers and firewalls.

And that is the rub in this process.  It is not unusual to have a client determine that their CDE is larger than they originally believed.  This is particularly true in environments that are rapidly changing.  The reason is that changes occur that involve the processing or transmission of CHD/SAD and people forget that those are also in scope because of their fixation on storage of CHD.  So do not be surprised to be surprised when this analysis turns up with in scope devices that were not believed to be in scope.

Connected To Systems

With the CDE(s) defined, we now we need to define all the systems that connect to the CDE(s), hence the “Connected To” designation by the Council.  The reason Connected To systems are in scope is because they can influence the security of the systems and devices inside the CDE.  The term you will hear some people use is that Connect To systems can be ‘infectious” to systems in the CDE.

The first place to start is by reviewing the firewall rules or access control lists (ACL) that segment your CDEs from the rest of your network segments.  You will likely find specific IP addresses for devices such as Active Directory domain controllers, security incident and event manager (SIEM), FTP, DNS, DHCP, RADIUS, TACACS+ and similar services.  It is not unusual to see application and database servers in a complete network subnet.

The second place to investigate are the organization’s most recent penetration testing results for network segmentation.  It still amazes me how even with a detailed examination of the firewall rules and ACLs that there are still devices that end up with connectivity into the CDE because of human error examining the rules and ACLs.  So use the network segmentation testing to double check your review of the firewall rules and ACLs.

Once you have identified all these networks you then need to make sure that you have an accurate inventory of all the systems and devices on these networks.  I typically ask for Nmap scans of the network subnets to make sure the inventory is complete.  I take the Nmap results and compare those to the organization’s configuration management database (CMDB) or whatever they use to track their system/device inventory.

I also make sure that all the devices and systems found in this process are contained in their internal vulnerability scanning.  Again, it is not unusual to find out that devices and systems are not being scanned quarterly for PCI which is why this check is important.

Now We Have PCI Scope

With all of this done, we now know the scope of the environment and what must be assessed.  But, remember, while you are done for the current assessment, this all needs to be performed again next year.

23
Apr
20

Upcoming PCI Dream Team Events

On Tuesday, May 5, at 11AM ET (3AM UTC) the Dream Team will be doing a virtual session for Secure360. Go here to register for the Secure360 Conference.

Then, on Wednesday, May 13, we are holding a GDPR Birthday Party on BrightTalk to celebrate the second birthday of GDPR.  While we will be taking PCI questions, we will also be entertaining questions on GDPR as well.  To register for the BrightTalk session, go here.

We look forward to your attendance at both of these events.  As always, if you cannot attend either of these sessions, you are more than welcome to submit questions at pcidreamteam AT gmail DOT com.

05
Apr
20

The Joke That Is SAQ A

This week another outbreak of Magecart was detected in at least 19 eCommerce sites.  It is using a new way to obfuscate and gather cardholder data (CHD).  As I read through the latest description, it brought to mind SAQ A.

But before I launch into that diatribe, first a little bit of history so that everyone understands why SAQ A even exists.

In the early wild, wild west days of payment card security on the internet, enterprising solution providers were pandering “outsourced” solutions that would “avoid” compliance with the then Visa Cardholder Information Security Program (CISP) and MasterCard Site Data Protection (SDP) compliance efforts.  What they were selling was a solution that used a variety of Web site techniques to keep the CHD away from the merchant’s Web site.  These solutions sold themselves because they took the merchant out of scope from the very onerous Visa and MasterCard security programs.

Then along came the PCI DSS and the self-assessment questionnaires (SAQ).  As part of that process, the Council and the Brands realized that these so-called out of scope solutions were not really “out of scope”.  The result was SAQ A which covers these outsourced solutions.  For years they had kept their solutions out of the card brands’ compliance programs and now they were included.  SAQ A was good news, bad news moment for the solution providers.  The bad news was that there was no escaping the fact that their customers were now in scope for PCI compliance.  However, the good news was that to placate these solution providers who were lobbying loudly for no scope, the Council and Brands minimized the number of requirements in SAQ A to a very, very bare minimum so that these outsourced solutions would not scare their customer bases off due to PCI compliance.

Just for the record.  SAQ A is the absolute bare minimum number of requirements any merchant can comply with and be considered PCI compliant.  There is nothing less.

And Now The Jokes – Bad As They Are

The first joke is that SAQ A is the absolute prime example of compliance does not equal security, bar none.

Anyone that thinks compliance with SAQ A keeps their customer payments secure is seriously lying to themselves.  Magecart in all of its forms is exhibit number 1 as to why SAQ A is a joke and should be retired.

I have told my clients since SAQ A was published that if they thought compliance with SAQ A would keep them out of trouble to think again.  Yes, SAQ A keeps the processors, banks and brands happy, but it does nothing to manage the risk presented by any web site.  That is because if the code/executable/script on their server that invokes the redirect or iFrame is ever tampered with (as with Magecart), it will not be the processor or bank held legally responsible, it will be the merchant that operates that web site that is legally on the hook.

That is the second joke of SAQ A.  Merchants think they have pushed the payment card processing risk of their eCommerce operation off to a service provider and they have not.  Unknowingly, they still have a lot of skin in the game.  More than they realize or want to realize.

Yet time and again, I encounter merchants following SAQ A that blindly go about life without regularly patching, maintaining or monitoring their web site because “SAQ A says I do not need to do that”.  All of this under the mistaken belief that SAQ A’s requirements create security for that web site which they do not.  Sadly, I have also encountered a number of merchants over the years that have been caught in the SAQ A trap and found out the hard way the monetary and business costs of their beliefs in SAQ A protecting them from bad actors.

SAQ A Is Compliance Not Security

In the last update of the SAQs in 2018, the Council did address a minor shortcoming in SAQ A.  That addition was to require organizations to ensure that their Web server was patched current for critical vulnerabilities.  However, from a risk perspective for an internet-facing system, that did very little to ensure the security of merchant Web sites used for directing payment processing.

Notably, SAQ A does not require at least any of the following:

  • Only one major service running, i.e., Web server with eCommerce application.
  • External and internal vulnerability scanning.
  • External and internal penetration testing.
  • Critical file monitoring to identify if the redirect or iFrame invocation method has been tampered with.
  • Logging and monitoring of the Web server and Web applications.

Most information security professionals would still likely consider even these aforementioned requirements inadequate.  These are all items I have told my clients I recommend, but even these absolute bare minimum steps for securing a Web server are not required for SAQ A compliance.

As a result, is it any surprise that most information security professionals and most QSAs consider SAQ A worthless for anything other than PCI compliance?  Organizations that truly understand information security also realize that SAQ A is not security and follow SAQ A-EP for ensuring the security of their out of scope Web servers.

The bottom line is that we in the payment security industry need to lobby the PCI SSC, banks and card brands to get rid of SAQ A before even more organizations get hurt.

31
Mar
20

Surprise! PCI Largely Ignores Disaster Recovery

As a lot of people are finding out when they rolled out work from home (WFH) for COVID-19, it turned out that compliance with the PCI DSS was not in place and in some cases, not even possible.  If the PCI Dream Team session last week is any example, I think a lot of QSAs got calls and emails from their clients asking what the Hell had happened?

What we are all experiencing is a shortcoming of the PCI DSS and a point that has been discussed off and on since it came into existence.  The problem is that the PCI DSS does not worry about business continuity (BCP) or disaster recovery (DR) unless there is cardholder data (CHD) or sensitive authentication data (SAD) involved which typically only occurs when that data exists at hot/warm recovery sites.  If the CHD/SAD is only there when recovery is occurring, then those locations are out of scope.

Unfortunately, the COVID-19 event is not your normal “disaster”.  With the invocation of WFH, production data centers and offices were still fully operational which is typically the concern of BCP/DR.  What changed was that the government enacted stay at home or shelter in place orders and the recovery site became an employee’s home, not your expected recovery location.  Worse, since WFH would not involve CHD/SAD until it was invoked, QSAs had no reason to assess any plans for such recovery because they were not in-scope until activated.

That said, a lot of QSAs (myself included) usually did have a discussion with clients that, while their BCP/DR plans were not in-scope, clients should occasionally assess those plans to make sure that, when invoked, the plan maintained PCI compliance.  Because when the BCP/DR was invoked, whatever was done had to be PCI compliant the moment it was used.

And there is the rub in all of this.  There is no grace period with PCI compliance because of the invocation of BCP/DR.  You are expected to be 100% PCI compliant regardless.

There are a number of lessons learned due to the COVID-19 disaster.

  • BCP/DR will need to be updated for pandemic incidents including WFH capabilities. According to the World Health Organization (WHO) and the Centers for Disease Control (CDC), pandemics are not going to go away, so the ability to continue operations remotely is going to have to be part of an organization’s recovery options.
  • WFH capabilities will have to be incorporated into normal production capabilities. This effort will need to address PCI compliance as well as HIPAA, CCPA, GDPR and any other relevant security and privacy programs your organization may have to comply with.  I have had a number of organizations enact such capabilities, policies and procedures for various parts of their operations over the years due to changing work requirements of their personnel as WFH offers a number of flexible advantages for retaining employees.
  • Implement virtual desktop infrastructure (VDI) to provide office as well as remote working capabilities. This can also potentially allow WFH to use an employee’s BYOD as long as they are not required to be PCI compliant.  Use of VDI allows the use of thin clients and Chromebooks to be used as workstations making security of those workstations a bit easier as well as reducing the cost.
  • Implement P2PE or end-to-end encryption (E2EE) solutions for the entry of CHD/SAD into applications. There are a number of USB and Bluetooth options available from the various point of interaction (POI) vendors such as Verifone and Ingenico as well as other third-party application vendors.
  • Softphones create a larger scope by bringing the workstation they connect to into full scope for PCI compliance. The number of requirements that need to be assessed can be reduced by using VDI and connecting the softphone to the VDI through the workstation.  Making such a connection though is not “plug and play”, so be prepared to have to work a lot with the VDI vendor to make that connection work.  But do not be surprised if it does not provide a reliable and/or clear connection, so make sure you are prepared to have to place the full workstation into scope to have an acceptable working solution.
  • If you are expecting to use The Cloud for your VDI or application, make sure that you conduct appropriate capacity planning so that you are not caught without the ability to expand that solution due to WFH. A lot of organizations have found with the COVID-19 event that their Cloud implementation did not scale as they thought it would.  It turned out that Cloud providers have capacity limitations just like in-house data centers.  While you are not using that capacity all of the time, you need to reserve it for those instances when you need it.  While not free, that excess capacity can be reserved for such events as COVID-19 so that when you need it, it is available.
23
Mar
20

Work From Home PCI Considerations

The PCI Guru got a question regarding PCI compliance for service providers in today’s emergency work from home (WFH) environment from a blog reader and it got The PCI Dream Team thinking about how does that work?  So, thanks to David Mundhenk of Herjavec Group, Art “Coop” Cooper of NuArx, Ben Rothke of Tapad and Jeff Hall of Online Business Systems for contributing to this list.

Ben & David wrote a piece on the topic last week, and the Dream Team has a webinar on Dealing with PCI DSS Compliance During the COVID-19 Crisis on March 25.

Thanks to the Coronavirus crisis, organizations are now scrambling to get their employees working from home.  This is presenting a whole new series of challenges to their compliance, technology and information security teams as these employees are now operating in a potentially less secure and definitely less private environment.

Home networks are going to be less controlled and secure.  Making matters worse is that most home networks today are Wi-Fi based, not wired, so data is flowing over untrusted networks because everyone in the house knows the Wi-Fi password (assuming there is one and it is not the default).

Bring Your Own Device (BYOD)

The biggest issue we are encountering is those organizations that need to rely on workstations owned by employees because they do not have company-owned and configured equipment to provide.  I have seen many a Tweet and LinkedIn post discussing the shortages of equipment for work from home and what options do they have.  The problem stems from most business continuity plans focusing on events that affect a business location or a community, not the entire country.  As a result, the idea of a pandemic forcing people to work from home was not thought of as a realistic threat.

As a result, bring your own device (BYOD) is the only answer in the near term to getting people working from home.  In discussions not only amongst the Dream Team but with other QSAs, there just do not seem to be any good answers for using BYOD and maintaining PCI compliance.  None of us can come up with ways to maintain compliance with BYOD because there are just too many factors involved from anti-virus (many varieties), limited or non-existent central monitoring and management, vulnerability scanning, penetration testing, patching, differing hardware, differing operating systems and a host of other issues that make it impossible to verify compliance let alone maintain compliance.

One potential option to reduce risk and gain better control with BYOD is using virtual desktop infrastructure (VDI) solutions such as Citrix Workspace, VMware Horizon or Windows Remote Desktop Services.  If you have that infrastructure in place, then we would recommend expanding it for WFH remote access.  If you do not have that infrastructure, you may be able to use Amazon Web Services (AWS), Microsoft Azure, Google Cloud or similar cloud environments to stand it up quickly.  That would allow you to reduce the risk of the BYOD being used but there still would be the risk of memory scraping and keyboard logging on the BYOD that must be managed.

That is not to say that you should not use BYOD as you need to keep your business running.  What it does mean is that you need to have a serious talk to your acquirer to determine how to handle this situation, what the risks are to your solution and then communicate the results of that discussion formally to your QSA.  You may even want to have your QSA on those calls with your acquirer to assist.  In these desperate times, I doubt that any bank is going to say you cannot stay in business as long as you can provide some controls and do your best to manage the risk.

We view BYOD though as a short term solution and that a longer-term solution needs to be developed as current estimates seem to indicate that the crisis will likely extend past the original estimate of four weeks.  That longer-term solution would involve acquiring the necessary hardware and software to implement a managed, secured and controlled environment that can be tested to ensure PCI compliance.

Company Provided Hardware and Software

For those companies that do have the equipment to send home with their employees, this is not a complete list, but a set of bullet points of ideas for how to address PCI compliance in our “new normal”.

  • The easiest topic is remote access. The PCI DSS explicitly calls out that a secure VPN (i.e., encrypted) with multi-factor authentication (MFA) for users to obtain access to the service provider network (8.3.2.a).  But where things can go sideways is complying with 8.3.1.a which requires MFA for non-console access to systems/devices in the cardholder data environment (CDE).  It goes awry because people think that the first MFA covers the MFA into the CDE and it does not.  The reason is that 8.3.1.a was designed to stop the phishing of Administrators to gain access to the cardholder data environment (CDE).  To stop that, you need additional MFA to access the CDE.  That does not mean a separate MFA solution (which would be ideal), but it does mean enforcing a delay in the single MFA so that the same MFA code cannot be used to access the internal network and then also the CDE.  A lot of organizations implement the delay in their remote logon script by invoking a timer delay that expires after the longest time a code can be active (usually 30 seconds).
  • A secure VPN is necessary to remove the home network from scope. Ideally, the VPN should be required to always be in use so that the workstation cannot get to the internet other than over the VPN.  For those that do allow the home network and internet to be accessible, you will need to ensure that the firewall on the workstation appropriately protects the workstation as well as implementing a host intrusion detection solution (HIDS).
  • VDI is also a solution because it allows for the use of thin-client and devices such as Chromebooks to be used to connect. Most VDI solutions also embed a secure remote connection via HTTPS or similar secure connectivity solutions.  If not then you will need to use a secure VPN as documented above.  However, even a thin client runs the risk of memory scraping and keyboard logging so you need to manage those risks.
  • Review all automated workflows to make sure that they are producing the necessary audit trails that will provide evidence for you PCI assessment of what is happening. Where this becomes problematic is when the workflows are developed only for PCI compliance and with the changes for remote operations, those workflows are not picking up new users, devices and other changes that were made to allow for remote work.
  • People that typically work together but now are remote will start using Microsoft Teams, Slack, Skype and other collaboration platforms to communicate and that may include sharing cardholder data (CHD) or sensitive authentication data (SAD) at times. You will need to train and quickly remediate situations if CHD/SAD enters these applications as well as periodically reminding these people that the use of these communication systems for transmitting SAD/CHD is not allowed.  If possible, enable data loss prevention (DLP) or similar capabilities to identify and then redact SAD/CHD in any of these communications.
  • If you are pushing out call center operations, remember that softphones will bring the workstation they connect into scope for PCI compliance because the workstation is now directly connected to the CDE which is, of course, the VoIP telephone system. That means an increase in scope and that those workstations need to be hardened, managed, logged and controlled for PCI compliance.  Call center operations may also require additional network segmentation to be put in place to ensure the size of your CDE does not exponentially grow.
  • While not entirely PCI related but needs to be noted are some other remote call center operation issues to consider that could make compliance with contractual obligations regarding privacy and confidentiality of data discussed or processed by the operator. You may need to supply operators with shredders, printers, additional monitors and other equipment to ensure privacy and productivity.  You may also have to instruct people to locate their work area to a bedroom or other room where a door can isolate the operator while they work so that family members do not come into contact with information or documents they should not view.
  • Ensure that you have ways to document changes happening, their review and approval. A lot of organizations have paper forms, Excel spreadsheets, email forms, etc. they use that sometimes can get lost in people’s inboxes, archives and folders or just lost, period.  You need to make sure that the change management system will work in the remote mode and that change evidence, reviews and approvals are maintained.
  • Logging should not be an issue unless your organization was not logging the VPN or other devices because they were not in scope for PCI compliance but now are in scope. So, you need to review your new workflows to ensure all devices and systems are logging to your SIEM or logging solution so that you comply with PCI requirement 10.
  • Encryption key management could become an issue particularly if your process does not support remote management. This can happen with some hardware security modules (HSM) and systems that require that the key custodians physically input their seed values into the device’s console.  So, going on-site may be required for encryption key changes and that may require formal approval from local authorities to occur.

These are the top of mind ideas that we were able to come up with for this discussion.  However, every environment is different so not everything discussed may be possible for your organization to use and maintain compliance.  We would recommend you work with a QSA to make sure that what you are attempting to do is not creating risks you are unwilling to accept or if you cannot manage appropriately.

We wish all of you the best of luck during this crisis.  We will get through this, but it will likely take some ingenuity in how that happens.

Also, be aware that the Council and the Card Brands are working on this topic as well and I expect more from them in the coming weeks.

Stay safe and healthy.

 

Other WFH resources:

CISA Coronavirus Guidance – https://www.cisa.gov/coronavirus

NIST Teleworking Guidance – https://csrc.nist.gov/publications/detail/sp/800-46/rev-2/final

SANS Work From Home Webcast – https://www.sans.org/webcasts/archive/2020

20
Mar
20

Special PCI Dream Team COVID-19 Session

On Wednesday, March 25, at 1800 UTC (2PM ET) the PCI Dream Team of Ben Rothke, David Mundhenk, Art Cooper and Jeff Hall will be on BrightTalk to discuss the impact on PCI compliance of the current COVID-19 crisis.

This should be a tough session as a lot of organizations are facing difficult decisions as they try to implement work from home (WFH) as well as maintain PCI compliance.  Thanks to the crisis, there are constraints that ordinarily would not be a problem such as a difficulty in obtaining hardware and remotely connecting a large workforce.

To register for the session, go here.

As usual, if you are unable to attend, please send your questions ahead of the session to pcidreamteam AT gmail DOT com.

We look forward to your questions and hope we can provide some help.

Thank you to all of you that attended. We had a lot of great questions from the attendees. We apologize for not being able to get to all of the questions, but I intend to follow up on a few of those in future blog posts.




Announcements

If you are posting a comment, be patient, as the comments will not be published until they are approved.

If your organization has a PCI opportunity, is in need of assistance with a PCI issue or if you would like the PCI Guru to speak at your meeting, you can contact the PCI Guru at pciguru AT gmail DOT com.

I do allow vendors to post potential solutions in response to issues that I bring up in posts. However, the PCI Guru does not endorse any specific products, so "Caveat Emptor" - let the buyer beware. Also, if I feel that the response is too "sales-ee", I reserve the right to edit or not even authorize the response.

Calendar

July 2020
M T W T F S S
 12345
6789101112
13141516171819
20212223242526
2728293031  

Enter your email address to subscribe to the PCI Guru blog and receive notifications of new posts by email.

Join 2,264 other followers