Lessons Learned from Creating an OT Cybersecurity Program in a Large, Slow-Moving Government Agency

There are as many ways for an organization to approach managing its OT cybersecurity risks as there are critical infrastructure organizations that need to do so. Thankfully, we're in a time when C-suite risk managers are technically able to understand the basics of their technology risk profile or the potential business impacts to their operating environment from a cybersecurity-related event. 

The language of industrial cyber risk has also matured: many vendors produce webinars, podcasts, brochures, conference booths, etc. warning of the inherent risks of OT cybersecurity. Few still dwell on overly simplistic messaging such as: “There’s this thing called OT cyber risk; you should consider it.” This has given way to developing much more creative, yet succinct notes on detailed subjects ranging from OT architecture to asset protection, system resilience, and much more. Still, the language and logic can be overwhelming even to the most experienced subject matter experts.

Risk managers—especially C-suite and boards of directors—looking to establish a culture of OT cyber awareness and risk remediation proactivity in their organizations need to establish a formal and measurable program approach. The program must provide a shared vision and a practical roadmap that makes sense because it relates to many of the unique characteristics of their organization, OT environment, business model, and risk appetite. While there are resources available that define a common program theme, platform or even just an approach, they are often obscured in a quagmire of vendor messaging slanted to a product or sales strategy.

An OT Cybersecurity Journey Begins

I came to a large critical infrastructure organization—a government agency teeming with its bureaucratic challenges and rigid limitations—with a vision of how an OT cybersecurity program could address business and associated operational risks. Prior to my arrival, the agency had conducted several OT cybersecurity assessments, all highlighting a vital need for an OT cybersecurity program, some even identified specific impacts and their probability of occurrence. The agency was unsure how to proceed, whether to outsource an OT cyber program, build one internally, or to transfer risk to the individual line departments (business units) and make them accountable at the C-suite level. There were more questions than answers.

The communications challenge would be daunting. I needed to build “mental momentum” by learning what the critical organizational drivers and business processes were: 

• How have the existing risk registers been compiled and administered? 

• How accurate were the prior assessments and how did they create any traction for resolving some of the identified risks? 

• Who cared about each risk-and why? 

• How did the IT security function co-exist in an OT-intensive environment?  

This list of questions ultimately went from a couple of dozen to a couple of hundred as I dug deeper trying to find answers.

Each meeting with high-level stakeholders (and their teams) was an opportunity to “sell a few points.”  I had to craft message selection carefully—varying levels of technical comprehension, or interpretations of risk probability, and cultural resistance to any major change made these interactions dynamic. After each meeting, I could be embraced as either a savior or a fool, or even a threat to their autonomy within the agency.

When describing OT security program design and implementation for each self-empowered organizational stakeholder, I avoided many landmines: 

• There would be no elements of surprise, what you see is what you get, 

• They won’t have to revise their current year organizational objectives,

• Initial budget implications would not hit their books,

• Risk tradeoffs would be a collaborative decision between themselves and management.

That toughest category I faced was gaining acceptance of risk treatments and tradeoffs. For example, when we connect a critical OT system to a network monitoring tool (to manage asset inventory and vulnerability management), we would significantly reduce tons of risk and open opportunities to advance other security initiatives and compensating controls within a program framework. 

Many operating units still construed certain risks would far outweigh the benefits of monitoring, no matter how many balancing controls I put in place (such as, in this example, a data diode between the OT system switch and the monitoring device). This trend of distrust was a legacy of the kneejerk OT security program from the past and became a cycle as I plodded through the hundreds of managed OT systems in an agency this size.

This challenge demands action and intention, even while waiting for results. I needed to exercise “active patience” some of my homework included: 

• How does each department handle timing, change, evolution? 

• Who are (or have been) change agents in that organization and how can I recruit them to my “cause?” 

• The ultimate measure of how fast I’d be able to roll out a comprehensive program:  Do I sacrifice program speed for risk reduction certainty? 

Ultimately, I settled on a multi-prong approach that gained traction by understanding the dynamics of the organization at all these levels. Key ingredients included: using the NIST CSF-based risk assessments and needed compensating controls as an “independent arbiter” of what is needed. I had to have champions; I needed a “triggering event” to spur stakeholder investment.

The Catalyzing Incident Creates Urgency

The Colonial Pipeline cybersecurity incident directly impacted jet fuel deliveries to our airports and airline partners. My program may finally receive the senior management attention after laying the groundwork for nearly two-and-a-half years. Soon, department leaders were jumping onboard our program, not due to a vendor webcast or an executive handshake but because of the economics of the agency and our risk profile would be plainly and visibly impacted.

The new momentum brought more resources, more money, a new CISO, a re-alignment of cybersecurity personnel to create a centralized approach with de-centralized elements in the departments. We even conducted an internal branding exercise for the program; using the agency’s leader as a supporter of our program, who began requiring weekly updates on our progress. 

Currently we are focusing on multiple initiatives to manage the 108 controls associated with NIST CSF (1), and soon 2.0. We have commissioned robust R&D projects to supplement our monitoring-supported risk management; built an OT component in our SOC; and are developing a comprehensive, OT-system-specific disaster recovery management program, among other initiatives.

Whenever I attend conferences, I hear questions about HOW we gained traction for an OT cybersecurity program in a backdrop of doomsday PowerPoints and “demos.” The answer: really learn your organization at a level that you have never dreamed of. Build bridges and coalitions to other internal functions that stand to gain from your program. 

Learn your organization’s approach to discussing risk, business drivers, revenue streams, management processes, and more. Study the history of successful past projects and how they unfolded. Understand the political dynamics among the top 10% of managers in the organization.  Lastly, never hesitate to know your doomsday scenarios and other motivational messaging to supplement your “logic” as you build your program. 

The successes required to create, seed, nurture, harvest, and repeat are all within reach. Patience and determination will pay off and your progress will become your momentum until you have created a full foundational platform to adapt to an ever-changing threat landscape and the unique risk impacts on your organization.  Good luck!