Building Stronger Utilities Through Shared Lessons
Water and wastewater utilities have quietly become one of the most targeted sectors in critical infrastructure, and recent incidents across multiple states are making that reality impossible to ignore. These systems process over 32 billion gallons of wastewater every day. An additional 27 billion gallons are withdrawn and delivered from surface water and groundwater sources per day specifically for residential use like providing drinking water, doing laundry, and watering lawns. The public servants operating these critical utilities are keeping our communities safe and functional with little fanfare and, often, with extremely limited resources.
Aging infrastructure, expanding internet connectivity, and a rapidly evolving threat landscape are colliding in ways that demand attention. Rather than dissect any single incident, the pattern of attacks offers something more useful: a set of hard-won lessons about what resilience truly requires in operational environments where a misconfigured control device can have consequences that go well beyond a data breach. Perhaps more than ever, our adversaries are showing us that cyber resilience and operational resilience are no longer separable.
Lesson 1: Visibility Remains a Foundational Challenge
Organizations cannot protect systems they don’t know about. This is one of the key reasons the CIS Critical Security Controls (CIS Controls) begin with inventorying everything you own, software and hardware.
Attackers scan for, build inventories of, and specifically target internet-accessible operational technology (OT). Unfortunately, many of the owners, operators, and security teams responsible for configuring and monitoring those same OT assets are not fully aware of what they own or what is internet-facing. There are many reasons why, but none of them matter during an incident. What matters is that we know, or can know very quickly, everything we need to know about our systems in a crisis.
Visibility is critical to incident response. On the one hand, many of the impacted entities were able to respond extremely quickly and restore authorized access to affected systems before any public safety impacts occurred. On the other, the discovery of these attacks was due to operational impact, even if minor, and not from security alerts, alarms, or other visibility-related detections on the cybersecurity side. Had the adversary been more advanced, the lack of visibility could have caused much greater impacts, including public health concerns.
Takeaway: Essential OT security practices require that we establish and maintain complete asset visibility across both IT and OT environments.
Reference: CIS Control 1 (Inventory and Control of Enterprise Assets)
Lesson 2: Connectivity Creates Both Opportunity and Risk
Many critical infrastructure attacks begin with systems that were never intended to be publicly accessible. For years prior to COVID, the utility community had been on a path of IT/OT convergence. The pandemic simply drove a rapid and unplanned expansion of remote access to OT environments in a very short period. Unfortunately, adequate security didn’t keep pace.
The same remote access tools that let an operator efficiently monitor and manage plants from anywhere without rolling a truck are the capabilities that threat actors exploit. The operational efficiency gained is valuable, especially in a sector with chronically strained resources. The flip side is that many Programmable Logic Controllers (PLCs) and other OT devices are accessible from the open internet, despite basic security controls being widely available and cost-effective.
Every utility must understand which OT assets can be reached from the internet and whether that exposure is truly necessary. PLCs are designed for reliability: they run industrial protocols that were never intended to be exposed to the internet. Whether by design, misconfiguration, or gradual network sprawl, internet-accessible OT becomes visible to anyone running a basic scan. A quick scan for a single network port used by one specific type of PLC revealed nearly 3,000 exposed devices with over 1,800 being in the United States.
Takeaway: Every remote access path into an OT environment increases the attack surface. Connectivity decisions require security review, and existing remote access should be inventoried and validated alongside the assets themselves.
Reference: CIS Control 12 (Network Infrastructure Management), NIST SP 800-82
Lesson 3: Operational Resilience Matters as Much as Prevention
Even strong security programs must prepare for disruption. Preventing every cyber attack is not a realistic goal. What is realistic is that proactively building operational resilience can dramatically limit the blast radius when there is an incident. Emergency protocols, including isolating key systems and switching to manual control in near real time, preserve public safety even as a full investigation begins.
Response to the current wave of attacks on water systems across multiple states reinforced the fact that the ability to switch to manual operation is a key feature of industrial systems. Utilities that maintain documented manual procedures, keep operators trained, and regularly test their emergency response plans fare significantly better than those that wait to figure it out in the middle of a crisis. The EPA recommends regular tabletop exercises to test Emergency Response Plans and train staff on response protocols.
Operational resilience is often heavily influenced by resourcing. An incident that would be easily contained in a highly resourced utility can cause prolonged operational impacts in organizations with limited monitoring, limited segmentation, and no dedicated cybersecurity personnel. To boost resilience, organizations large and small can take advantage of community resources through the MS-ISAC, WaterISAC, and OT-CERT as well as federal resources.
Takeaway: Business continuity, emergency response, and crisis management plans aren’t nice to have items; they are operational requirements. And they require regular updates and regular exercise. Every operator must know their manual fallback procedures before a crisis.
Reference: CIS Control 17 (Incident Response Management); AWIA Emergency Response Plan requirements; CISA Water Sector Incident Response Guide
Lesson 4: Third-Party Risk Extends Into OT Environments
The water sector runs on vendor relationships. Instrumentation vendors, supervisory control and data acquisition (SCADA) integrators, chemical suppliers, managed service providers, and equipment manufacturers all have varying degrees of access to utility systems. And as we learned above, that access is not always well-governed. From underlying operating systems lacking reliable patch management to unsafe connection protocols to an overall misunderstanding of security best practices, the risk provided by third parties creates a complex and challenging web of interdependency.
Third-party risk in OT is different from third-party risk in IT. A vendor remoting into a billing system is a governance and data protection concern. But a vendor remoting into an industrial system is an operational safety concern. The security expectations need to reflect that distinction but often don’t. In recent weeks, we’ve learned that many PLCs are not tucked behind VPNs, rogue wireless access points are common in water districts nationwide, and sometimes vendor security advisories give adversaries the very information they need to remotely reset your devices. While security advisories contain valuable patch information for defenders, they can often act as a tutorial for those with ill intent.
To truly build the operational resilience described above, utilities must be able to both trust their third-party vendors to bake security into their products and processes and hold them accountable when they don’t. This isn’t an overnight solution, but it is required to reduce community-wide risk.
Takeaway: Third-party access to OT systems requires, at a minimum, the same rigor as direct operator access. Every vendor connection to OT should be inventoried, time-limited, monitored, and tied to a specific authorization.
Reference: CIS Control 15 (Service Provider Management); Joint Guidance on OT Remote Access
Lesson 5: Cybersecurity is Not Just a Business Issue, It’s a Public Service Issue
Water utilities don’t make headlines when they work. They make headlines when they don’t. A pressure loss, a contamination scare, a boil-water notice, and other downstream consequences of a successful cyber attack on water infrastructure aren’t abstract to the public. The impacts are felt in hospitals, schools, restaurants, businesses, and homes, and it reframes the issue of cybersecurity investment from a business cost to a public service obligation.
For water utilities, and many other public services, the responsibility for reasonable security can’t rest solely on the IT department. Technology managers in water districts large and small are asked to solve a problem that Fortune 500 companies struggle with, on a fraction of the budget, with a fraction of the staff, and with infrastructure that often predates the internet. Yet in our rapidly evolving technology environment, cyber and physical security decisions increasingly affect service reliability, community trust, regulatory obligations, and operational continuity.
From incident protection to real-time detection, from proactive emergency planning to clear communication pathways in a crisis, and from preventative maintenance to manual operation during an incident or outage, these are public safety decisions supported by and supporting the technical staff that make it all possible. Anticipating emergencies, careful planning, and conducting drills before an incident reduces confusion, builds confidence, and supports faster decision-making to ensure public health and safety is not at risk.
Takeaway: Cybersecurity in the water sector is a resource and prioritization challenge as much as a technical one. Leadership at every level, from lawmakers and regulators down to the local district, has a role in closing it.
Reference: CIS Controls; EPA Cybersecurity Grant Program; WaterISAC Resource Center; Security Strategy for Critical Infrastructure
Where Utilities Go From Here
The past several weeks have been a stress test the sector didn’t ask for. But it reminds us of an idiom that has long been attributed to Churchill: we should never let a good crisis go to waste. We can learn from this event since the pattern is clear: exposed devices, remote reset weaknesses, flat networks, and limited visibility. None of those are unsolvable problems nor do they require a seven-figure budget to address. Some security prioritization and leadership focus today gets us closer to operational readiness tomorrow.
Run through the basics. Complete your AWIA Risk and Resilience Assessment if you haven’t already. Segment your OT network from your IT network. Establish visibility into what you own and disconnect anything openly internet-facing or put it behind proper security tools. Placing internet-facing OT assets behind a VPN is one of the highest-impact, lowest-cost steps you can take almost immediately. Build and exercise your incident response plan. Seek out the free resources that exist specifically for this sector.
Even sophisticated threat actors will go after low-hanging fruit first, so your focus should be to put as many obstacles in the way as possible.
One thing you can do today: Search your organization on Shodan. It’s free, it takes five minutes, and it will show you exactly what your utility looks like to an adversary scanning the internet. If you find OT exposed that shouldn’t be, then you know where to start!
Need help getting started? Join the Multi-State Information Sharing and Analysis Center® (MS-ISAC®) and take advantage of cybersecurity resources, threat intelligence, and incident response support specifically designed to help state, local, tribal, and territorial organizations strengthen resilience.
About the Author
Randy Rose
Senior Vice President, Security Operations and Intelligence
Randy has over two decades of experience across state, local, and federal government. He currently leads Security Operations at the Center for Internet Security where he is privileged to oversee the execution of the Security Operations Center (SOC), Cyber Threat Intelligence (CTI), Multidimensional Threat Intelligence (MDT), Cyber Incident Response Team (CIRT), and Red Team missions in direct support of the Multi-State ISAC. Previously, he led the largest Joint Military Security Operations Center in Europe and the Intelligence Department for the Navy’s Global Cyber Operations Task Force.
Early in his career, he served in the U.S. Air Force and later was New York State’s first local government-focused penetration tester. He moonlights as an adjunct instructor at Siena University and the State University of NY at Albany. He sits on the Board of Directors for two local non-profits helping those in need of housing, food, and trauma services. He loves rock music and is willing to have deep conversations with anyone over his favorite bands.
