Tales from the Trenches: Vol 9 — The Law of Unintended Consequences, OR Some Doors Swing Both Ways

/by

Since 2004, RedSeal has helped our customers See and Secure their entire complex network. And while those customers may have understood the value of understanding their environment, how it was connected and see what’s at risk, there is often an “Aha” moment when the true significance is clear. The stories of these moments are lore within the walls of RedSeal. But these tales so clearly illustrate the value of RedSeal beyond just theory that we think they’re worth sharing. In the words of our team in the field, the ones working directly with our customers, this blog series will share the moments where it all gets real.

In this edition of the series Bill Burge, RedSeal Professional Services, explains how RedSeal can show you ALL the access from a network change, not just the one access you are expecting.

The Law of Unintended Consequences, OR Some Doors Swing Both Ways

“The law of unintended consequences” states that the more complex the system, the greater the chance that there is no such thing as a small change.

While working with a customer in the early days of my RedSeal Professional Services tenure, I looked for an opportunity to prove the capability of Zones & Policies. In an unfamiliar environment, the easy starting point is creating a policy that examines the access from “Internet to all internal subnets.”

It is easy to setup and easy to discuss the results, UNLESS the results say that most of the Internet can get to most of the internal network.

I thought “I MUST have done something wrong!” I got the impression that the customer felt the same thing, even though neither of us came right out and said it. So, I tore into it.

Using some ad hoc access queries and Detailed Path queries, we figured out the problem and why.

After looking into it, thinking something was amiss, it turned out that RedSeal was RIGHT. It seems there had been a pair of firewall rules for DNS requests:
SRC: inside, SRC PORT: any, DST: outside, DST PORT: 53, PROTOCOL: UDP
(and for the responses)
SRC: outside, SRC PORT: 53, DST: inside, DST PORT: any, PROTOCOL: UDP

At some point, because DNS resolutions got large enough that the responses did not fit in a single UDP packet, DNS needed to include TCP. So, someone simply made a small change and added TCP to each of these rules.

The unintended consequence was that you could reach just about any internal system from the Internet IF you initiated your request from port 53.

After this was verified by the firewall and networking teams, I might have well gone home. Everybody disappeared into meetings to discuss how to fix it, whether it could be done immediately or later that night, etc.

A little time later, I ALMOST felt guilty to point out that they had done pretty much the same thing with NTP, on port 123. (Almost…)

Interested in how RedSeal can help your team? Click here to set up a demo or an introductory call.