Tales from the Trenches: Vol 2 — They have access to WHAT?!

/by

Since 2004, RedSeal has helped our customers See and Secure their entire complex network. And while those customers may have understood the value of understanding their environment, how it was connected and see what’s at risk, there is often an “Aha” moment when the true significance is clear. The stories of these moments are lore within the walls of RedSeal. But these tales so clearly illustrate the value of RedSeal beyond just theory that we think they’re worth sharing. In the words of our team in the field, the ones working directly with our customers, this blog series will share the moments where it all gets real.

In this edition of the series, Nate Cash, Senior Director, Federal Professional Services/ Director of Information Security at RedSeal looks at a unique application for RedSeal, eliminating potential threats before they could happen.

They have access to WHAT?!

I’m always surprised at the new use-cases we come up with on site with RedSeal. There is a lot of information about a customer’s environment that allows us to answer questions pretty easily, if you know where to look. One Monday morning as I showed up to the office, before I was able to grab coffee, a SOC analyst stopped me at the door to ask me a very simple question, “We have a bunch of site-to-site VPNs with a few business partners, what can they access?”

On the surface this seems like a simple request, “How quickly do you need this information?” I responded. “Last Saturday.” Suddenly the caffeine rush I needed seemed like a long ways off. Turns out, one of the business partners of this customer was breached over the weekend and they notified this customer of the potential. The SOC had been manually mapping out the configs and drawing the paths on the white board, when I came in.

Staring at the board I thought, “There has to be a better way.” When the eureka moment happened. I fired up RedSeal and found the devices which lead to the business partners. In order to map across the VPN tunnels, we needed the config files from the other end. Of course, our business partners were not going to give that up, so I took the configs from our end, and reversed the ACLs changed the IPs based on the tunnel configs and reimported them into RedSeal.

10 min later we were able to answer the question, “what does this business partner have access to?” the answer, 1 server in the DC on 3 ports. But that one server had access to over 20 other servers which increased the downstream risk. This was a fun exercise where we got to see the power of RedSeal and how it can be used to quantify the real risk to the organization and reduce it by putting controls into place.

Once the incident was contained, we decided to go through, and hand jam the rest of the business partners VPN configurations into the model so the SOC would have this information in the event another partner was compromised. After writing up the configs and placing them into the model we found a couple of business partners with configurations that allowed any traffic on any port across.

Firewall engineers know, that sometimes during troubleshooting they’ll configure a device to allow anything across and verify that the firewall is or is not blocking an application. My theory is that these two partners were having issues with traffic going across or getting the VPN tunnel up, and put these rules in as placeholders, probably around two or three A.M. and when things “worked” they were going to come back later to fix them, and never got the time.

But this exercise allowed us to find a major security misconfiguration. If one of those two business partners with the ‘allow any traffic’ were the ones compromised over the weekend this story would have had a much different outcome. In security having a complete picture of your environment is key to being secure. What you don’t know can hurt you.

Interested in how RedSeal can help your team? Click here to set up a demo or an introductory call.  

Tales from the Trenches: Vol 1 — In security, consistency is key.

/by

Since 2004, RedSeal has helped our customers See and Secure their entire complex network. And while those customers may have understood the value of understanding their environment, how it was connected and see what’s at risk, there is often an “Aha” moment when the true significance is clear. The stories of these moments are lore within the walls of RedSeal. But these tales so clearly illustrate the value of RedSeal beyond just theory that we think they’re worth sharing. In the words of our team in the field, the ones working directly with our customers, this blog series will share the moments where it all gets real.

The first in this series is by Nate Cash, Senior Director, Federal Professional Services/ Director of Information Security at RedSeal.

In security, consistency is key.

Once at a customer site while going through the install of RedSeal, we were going over the hardening standards. I clicked on a couple of configurations to start showing how we could go about setting up best practice checks. I had inadvertently pulled up a device which has not been updated in over five years. The customer was shocked, this is one of the many times where I have had to stop mid-sentence while the person I worked with, reached out to someone to “fix the problem.”

The problem is not the fact the device is had not been updated, but somehow their process missed it. This device was just one of many. The first thing we did with RedSeal was develop a set of custom checks to see how many devices passed or failed the latest hardening standard. Once set we started data collections. In 15 min we saw 30% of the devices were not running their latest hardening standards. 

30% of their network devices were not using the latest encryption, had management ACLs set to an old subnet which was now known as their guest subnet, inconsistent SSH versions, telnet still enabled, and some devices pointed to old radius servers, falling back to local accounts. Luckily their firewall blocked the guest subnet from anything internal but, their network management tool still couldn’t access 100% of the devices.

Fixing the underlying problem, not the result of the problem.

With the latest hardening standards in hand, the network engineers got an emergency change request and started logging in to update their configurations. Each device required firmware upgrades, followed by configuration changes. While this was a big undertaking, each day that passed we could grab the numbers and see trending on remediation and report to management.

Once we got the model complete, we found a couple of firewalls that blocked their network management solution from accessing 100% of their network devices. Once the customer opened their firewall, the networking team could start pushing the config to all of the devices. 

As luck would have it, the customer was currently undergoing a review of a new hardening standard. We took the new standard, and I showed them how to create checks for each of the configuration points. At the end of the day, the customer had 75 individual checks for each of the network devices. Upon data collection RedSeal will run those checks against each of the configurations automatically and we could ensure that all network devices passed all of checks required for their baseline configuration. 

This customer had a unique process where devices about to be deployed were plugged into the network and received a static DHCP address. Their network management tool would push the baseline to the config, then the engineers would login the next day to assign the interface Ips per the documentation. The rest of the hardened config was automagically configured by the tool. With this in hand, we were able to add the ‘staging area’ to RedSeal. With every data collection we noticed random inconsistencies where some device would get the whole config and others did not, using those same checks. 

Using RedSeal and custom checks this customer is able to push configs, then double check the config took properly before deploying out into their network. They had better visibility ensuring that all of their devices were hardened, were more confident that automating the work were consistently driving the results they wanted, had a double check with the automation, and essentially reduced their risk significantly, just by checking the configs of their network devices.

Interested in how RedSeal can help your team? Click here to set up a demo or an introductory call.  

InfoSeCon Roundup: OT Top of Mind with Many

/by

The RedSeal team recently attended the sold-out ISSA Triangle InfoSeCon in Raleigh. It was energizing to see and talk to so many people in person. People were excited to be at in-person events again (as were we!) and we had some great discussions at our booth on how RedSeal can help customers understand their environment and stay one step ahead of threat actors looking to exploit existing vulnerabilities. 

Visibility was a top topic at the booth, but I want to focus this blog on our panel discussion, titled ”OT: Still a Security Blindspot”, which, of course, has visibility as a core need. While I was expecting that this topic would be of interest to quite a few attendees, I was not expecting the great reception we received, nor the number of people that stayed behind to talk to us. We had so many great questions, that we have decided to host a webinar with the same title, where we will share some of the same information, and expand it to address some additional topics based on attendee interest. 

I want to highlight some key points we shared during the session:

We started the session talking about OT networks – what they are, and how they exist across all verticals in some way, shape or form (it is not just manufacturers!).  We did share a life sciences customer example to close the session (think about all of the devices in a hospital connected to the Internet – and what could happen if they were hacked!).

Then we got into the “risk” part of the discussion. We shared how OT networks are targeted via vulnerable devices and highlighted actual consequences from some real-life examples. We focused on the Mirai botnet and potential motives from threat actors:  

  • Botnets can simply to be annoying or attention getting, interrupting business 
  • Ransomware has traditionally been for cash or equivalent, but now can also be for deliberate intentional damage
  • Nation State – with espionage as a goal – to disrupt or compromise organizations or governments

Our panelists from Medigate by Claroty provided their perspective based on discussions with their customers. They explained that for many organizations OT was an afterthought when accessing security risks. The audience appeared to agree with this assessment. Unfortunately, this meant that OT has often been the quickest way to cause serious damage at an organization. They have seen this within the 1500+ hospitals they work with. They shared a couple of examples, which clearly demonstrated what can happen if you have vulnerable OT devices:

  • Turning off the air conditioning at a hospital in Phoenix AZ during the summer shut EVERYTHING down
  • Operating rooms must be kept at very specific temps and humidity levels. A customer in CA whose dedicated O.R. HVAC was impacted ended up losing $1M in revenue PER DAY while it was down

One of the reasons that the security risks for OT devices has not been addressed as well as they should is that OT devices have typically been managed by the Facilities organizations, who do not have the training and expertise needed for this task. We did spend some time talking about who “owns” managing and securing these OT devices. Luckily, there is growing awareness of the need for visibility into both OT and IT devices as part of an overall security strategy, and there are emerging solutions to address this need. 

We also spent time discussing how complex managing OT/IT becomes when companies have distributed sites or complex supply chains. The Medigate panelists shared how some of their Life Sciences customers have sites with different OT network topologies, and some even have a mismatch between the topology of the individual site and its production logic. This means there are usually multiple redundant, unmonitored connections at each site, which provides threat actors with numerous opportunities to penetrate the OT network and, once inside, to move laterally within it. 

This led to a conversation among the panelist on how to address the IT/OT visibility needs:

  1. First step: gain visibility into where OT is and then integrate it with existing IT security infrastructure 
  2. Ongoing: alignment and collaboration between and across IT and OT security, as well as with a range of third-party vendors, technicians, and contractors
  3. End goal: enable all teams (facilities, IT, security, networking, etc.) speak the same language from the same source of truth

The above is just a brief review of some of the topics covered during the panel discussion. If you are interested in hearing more about how to address IT/OT visibility needs and hear about how customers are addressing these needs or find out more about RedSeal, please visit our website or contact us today!

IT/OT Convergence

/by

Operational Technology (OT) systems have decades of planning and experience to combat threats like natural disasters – forces of nature that can overwhelm the under-prepared, but which can be countered in advance using well thought out contingency plans. Converging IT with OT brings great efficiencies, but it also sets up a collision between the OT world and the ever-changing threats that are commonplace in the world of Information Technology. 

A Changing Threat Landscape 

The security, reliability, and integrity of the OT systems face a very different kind of threat now – not necessarily more devastating than, say, a flood along the Mississippi, or a hurricane along the coast – but more intelligent and malicious. Bad actors connected over IT infrastructure can start with moves like disabling your backup systems – something a natural disaster wouldn’t set out to do. Bad actors are not more powerful than Mother Nature, but they certainly are more cunning, and constantly create new attack techniques to get around all carefully planned defenses. This is why the traditional strategies have to change; the threat model is different, and the definition of what makes a system “reliable” has changed. 

In the OT world, you used to get the highest reliability using the oldest, most mature equipment that could stay the same, year after year, decade after decade. In the IT world, this is the worst possible situation – out of date electronics are the easiest targets to attack, with the most known vulnerabilities accumulated over time. In the IT world of the device where you are reading this, we have built up an impressive and agile security stack in response to these rapidly evolving threats, but it all depends on being able to install and patch whatever software changes we need as new Tactics, Techniques and Procedures (TTP’s) are invented. That is, in the IT world, rapid change and flexible software is essential to the security paradigm. 

Does this security paradigm translate well to the OT world?

Not really. It creates a perfect storm for those concerned with defending manufacturing, energy, chemical and related OT infrastructure. On the one hand, the OT machinery is built for stability and cannot deliver the “five nines” reliability it was designed for if components are constantly being changed. On the other hand, we have IT threats which can now reach into OT fabric as all the networks blend, but our defense mechanisms against such threats require exactly this rapid pace of updating to block the latest TTP’s! It’s a Catch-22 situation. 

The old answer to this was the air gap – keep OT networks away from IT, and you can evade much of the problem. (Stuxnet showed even this isn’t perfect protection – humans can still propagate a threat across an air gap if you trick them, and it turns out that this isn’t all that hard to do.) Today, the air gap is gone, due to the great economic efficiencies that come from adding modern digital communication pathways to everything we might need to manage remotely – the Internet of Things (IoT).

How do we solve this Catch-22 situation?

So, what can replace the old air gap? In a word, segmentation – it’s possible, even in complex, blended, IT/OT networks to keep data pathways separate, just as it’s essential for the same reason that we keep water pipes and sewer pipes separate when we build houses. The goal is to separate vulnerable and critical OT systems so that they can talk to each other and be managed remotely, but to open only these pathways, and not fall back to “open everything so that we can get the critical traffic through”. Thankfully, this goal is achievable, but the bad news is it’s error prone. Human operators are not good at maintaining complex firewall rules. When mistakes inevitably happen, they fall into two groups:

  1. errors that block something that is needed
  2. errors that leave something open

The first kind of error is immediately noticed, but sadly, the second kind is silent, and, unless you are doing something to automatically detect these errors and gaps, they will accumulate, making your critical OT fabric more and more fragile over time. 

One way to combat this problem is to have a second set of humans – the auditors – review the segmentation regularly. Experience shows, though, that this just propagates the problem – no human beings are good at understanding network interactions and reasoning about complex systems. This is, however, a great job for computers – given stated goals, computers can check all the interactions and complex rules in a converged, multi-vendor, multi-language infrastructure, and make sure only intended communication is allowed, no more and no less.

In summary, IT/OT convergence is inevitable, given the economic benefits, but it creates an ugly Catch-22 scenario for those responsible for security and reliability – it’s not possible to be both super-stable and agile at the same time. The answer is network segmentation, not the old air gapped approach. The trouble with segmentation is it’s hard for humans to manage, maintain and audit without gaps creeping in. Finally, the solution to resolve this Catch-22 is to apply automation – using software such as from RedSeal to automatically verify your segmentation and prevent the inevitable drift, so that OT networks are as prepared for a hacker assault as they are for a natural disaster.