‘Dangerous Stuff’
Pinellas County Sheriff Bob Gualtieri speaks at a press conference Monday, along with Oldsmar Mayor Eric Seidel, middle, and City Manager Al Braithwaite, left. On Friday, Gualtieri said, someone remotely accessed the computer system for the city’s water treatment plant and tried to add a large amount of lye to the city’s water supply. [ Pinellas County Sheriff’s Office ]
February 8, 2021
The attack in Oldsmar, a city of 15,000 people in the Tampa Bay area, was caught before it could inflict harm, Sheriff Bob Gualtieri of Pinellas County said at a news conference on Monday. He said the dosing amount of sodium hydroxide — the main ingredient in drain cleaner — was changed from 100 parts per million to 11,100 parts per million, dangerous levels that could have badly sickened residents if it had reached their homes.
Sodium hydroxide is caustic and the main ingredient in drain cleaner. However, the chemical is used in small amounts in the water treatment process to balance the pH level (acidity/alkalinity) and remove metals from drinking water.
In 2007, the water of a town in Massachusetts was accidentally treated with too much lye, causing burns and skin irritation among people who showered with it.
“At no time was there a significant adverse effect on the water being treated,” the sheriff said. “Importantly, the public was never in danger.”
Even if the operator hadn’t caught it, he said, it would have taken more than a day for the water to enter the water supply.
“The protocols that we have in place, monitoring protocols, they work — that’s the good news,” said Oldsmar Mayor Eric Seidel. “Even had they not caught them, there’s redundancies in the system that would have caught the change in the pH level.
The SCADA system’s software would have detected the manipulation and alarmed due to the unauthorized change.
Treated water goes to holding tanks, and before the water is pumped out and to customers, it undergoes a secondary chemical and water quality check, including manual monitoring.
Oldsmar city officials also stressed that it would have taken 24 to 36 hours for water with dangerous amounts of the caustic substance to enter the town’s supply. And in that time, a number of alarms would have sounded.
The lye never would have made it into anyone’s tap, Mayor Eric Seidel said.
“The important thing is to put everyone on notice,” he said. “There’s a bad actor out there.”
Jake Williams, CEO of the cybersecurity firm Rendition Infosec, said engineers have been creating safeguards “since before remote control via cyber was a thing,” making it highly unlikely the breach could have led to “a cascade of failures” tainting Oldsmar’s water.
The unidentified hacker accessed the water treatment plant’s SCADA controls via remote access software, TeamViewer, which was installed on one of several computers the water treatment plant personnel used to conduct system status checks and to respond to alarms or any other issues that arose during the water treatment process.
All computers used by water plant personnel were connected to the SCADA system and used the 32-bit version of the Windows 7 operating system. Further, all computers shared the same password for remote access and appeared to be connected directly to the Internet without any type of firewall protection installed.
Sheriff Gualtieri said in an interview. Though the utility had switched to a different tool six months ago, he said, the TeamViewer program remained in place but unused, providing the door through which the intruder entered and gained full access to the system.
The hacker who’d taken control of the system, clicked open one software function after another until it finally landed on the controls to the water’s levels of sodium hydroxide, also known as lye.
Then, the hacker raised the levels of sodium hydroxide by more than 100 fold — a hazardous level that could sicken residents and corroded pipes.
A supervisor said they’ve disabled the remote-access system used in the attack.
In a statement to The Washington Post, TeamViewer spokesman Patrick Pickhan said the company was aware of reports of the hack, are “monitoring the situation” and condemn “any malicious behavior” on its software.
“We don’t have any indication that our software or platform has been compromised,” Pickhan said. “TeamViewer stands ready to support relevant authorities in their investigation of the technical details such as how the cyber criminals potentially obtained login credentials, which are set and encrypted solely on the device.”
But the near miss incident was the latest alarming sign that critical infrastructure in the United States is vulnerable to cyberattacks. In July, the Cybersecurity and Infrastructure Security Agency warned that infrastructure like water and power plants, emergency services and transportation systems make “attractive targets for foreign powers attempting to do harm to U.S. interests or retaliate for perceived U.S. aggression.”
It’s a hard problem, but one that we need to start addressing,” said Joe Slowik, senior security researcher at DomainTools. He said the hack illustrates “a systemic weakness in this sector.”
Robert M. Lee, CEO of Dragos Security, and a specialist in industrial control system vulnerabilities, said remote access to industrial control systems such as those running water treatment plants has become increasingly common.
“As industries become more digitally connected we will continue to see more states and criminals target these sites for the impact they have on society,” Lee said.
Chris Sistrunk, a technical manager at leading cybersecurity firm FireEye’s Mandiant division, said cybersecurity issues are relatively new for U.S. water utilities, whose biggest problems are pipes freezing and bursting in winter or getting clogged with disposable wipes. The Oldsmar hack highlights the need for more training and basic security protocols, but not drastic measures like sweeping new regulations.
Since last year, there has been “an increase in cyber incidents perpetrated by low sophisticated actors seeking to access and learn about remotely accessible industrial systems,” said Daniel Kapellmann Zafra, manager of analysis at Mandiant Threat Intelligence. “None of these cases has resulted in damage to people or infrastructure.” He added that the incident “highlights the need to strengthen the cybersecurity capabilities across the water and wastewater industry similarly to other critical infrastructure sectors.”
“While the (Oldsmar) incident does not appear to be particularly complex, it highlights the need to strengthen the cybersecurity capabilities across the water and wastewater industry similarly to other critical infrastructure sectors,” he said.
“Many of the victims appear to have been selected arbitrarily,” he said, “such as small critical infrastructure asset owners and operators who serve small populations.”
What concerns experts most is the potential for state-backed hackers intent on doing serious harm targeting water supplies, power grids and other vital services.
“In the industry, we were all expecting this to happen. We have known for a long time that municipal water utilities are extremely underfunded and under-resourced, and that makes them a soft target for cyber attacks,” said Lesley Carhart, principal incident responder at Dragos Security, which specializes in industrial control systems.
“I deal with a lot of municipal water utilities for small, medium and large-sized cities. And in a lot of case, all of them have a very small IT staff. Some of them have no dedicated security staff at all,” she said.
“These are the targets we worry about,” said Eric Chien, a security researcher at Symantec. “This is a small municipality that is likely small-budgeted and under-resourced, which purposely set up remote access so employees and outside contractors can remote in.”
That, Mr. Chien said, makes them a ripe target.
Tarah Wheeler, a Harvard Cybersecurity Fellow, said communities should take every precaution possible when using remote access technology on something as critical as a water supply.
“The systems administrators in charge of major civilian infrastructure like a water treatment facility should be securing that plant like they’re securing the water in their own kitchens,” Wheeler told the Associated Press via email. “Sometimes when people set up local networks, they don’t understand the danger of an improperly configured and secured series of internet-connected devices.”
The nation’s 151,000 public water systems lack the financial fortification of the corporate owners of nuclear power plants and electrical utilities. They are a heterogenous patchwork, less uniform in technology and security measures than in other rich countries.
A 2020 paper in the Journal of Environmental Engineers found that water utilities have been hacked by a variety of actors, including amateurs just poking around, disgruntled former employees, cybercriminals looking to profit and state-sponsored hackers. Although such incidents have been relatively few that does not mean the risk is low and that most water systems are secure. This is because so-called “air gaps” between internet-connected networks and the systems that directly manage pumps and other plant components are becoming less common.
“The reality is that many cybersecurity incidents either go undetected, and consequently unreported or are not disclosed because doing so may jeopardize the victims reputation, customers trust, and, consequently, revenues,” the paper says.
Cybersecurity is an important part of America’s Water Infrastructure Act (AWIA) of 2018, which requires community water systems serving a population of 3,300 or more to assess cybersecurity threats as part of a risk and resilience assessment and incorporate strategies that address cybersecurity in emergency response plans.
So far this year, the Department of Homeland Security has issued 25 advisories listing various industrial control systems that could be vulnerable to hacking. Affected products range from 3D rendering software to security cameras to insulin pumps.
The Biden administration has already signaled its intention of beefing up cybersecurity, a sector its predecessor was roundly accused of not taking seriously enough.
Investigators don’t know whether the attack originated within or outside Pinellas County, Florida or the United States. If the attacker is apprehended, he said, they’ll face state felony charges and possibly federal charges.
In a tweet, Senator Marco Rubio, Republican of Florida, said the attempt to poison the water supply should be treated as a “matter of national security.”
Gualtieri said, they do not know why Oldsmar was targeted. He added that other area municipalities have been alerted to the attack and encouraged to inspect the safeguards to their water treatment systems and other infrastructure.
One thing Gualtieri is certain of is that the hacker knew what he was doing.
“In order to get into the system, somebody had to use some pretty sophisticated ways of doing it,” he said.
The Pinellas County Sheriff’s Office and a digital forensics unit is investigating, along with the FBI and the Secret Service, Gualtieri said.
The Federal Bureau of Investigation (FBI), Cybersecurity and Infrastructure Security Agency (CISA), U.S. Secret Service and the U.S. Environmental Protection Agency (EPA) issued a joint statement about the cyberattack that happened in Oldsmar, Fla., on Feb. 5.
Compromise of U.S. Water Treatment Facility
SUMMARY
On February 5, 2021, unidentified cyber actors obtained unauthorized access to the supervisory control and data acquisition (SCADA) system at a U.S. drinking water treatment plant. The unidentified actors used the SCADA system’s software to increase the amount of sodium hydroxide, also known as lye, a caustic chemical, as part of the water treatment process. Water treatment plant personnel immediately noticed the change in dosing amounts and corrected the issue before the SCADA system’s software detected the manipulation and alarmed due to the unauthorized change. As a result, the water treatment process remained unaffected and continued to operate as normal. The cyber actors likely accessed the system by exploiting cyber-security weaknesses, including poor password security, and an outdated operating system. Early information indicates it is possible that a desktop sharing software, such as TeamViewer, may have been used to gain unauthorized access to the system. Onsite response to the incident included Pinellas County Sheriff Office (PCSO), U.S. Secret Service (USSS), and the Federal Bureau of Investigation (FBI).
The FBI, the Cybersecurity and Infrastructure Security Agency (CISA), the Environmental Protection Agency (EPA), and the Multi-State Information Sharing and Analysis Center (MS-ISAC) have observed cyber criminals targeting and exploiting desktop sharing software and computer networks running operating systems with end of life status to gain unauthorized access to systems. Desktop sharing software, which has multiple legitimate uses—such as enabling telework, remote technical support, and file transfers—can also be exploited through malicious actors’ use of social engineering tactics and other illicit measures. Windows 7 will become more susceptible to exploitation due to lack of security updates and the discovery of new vulnerabilities. Microsoft and other industry professionals strongly recommend upgrading computer systems to an actively supported operating system. Continuing to use any operating system within an enterprise beyond the end of life status may provide cyber criminals access into computer systems.
To report suspicious or criminal activity related to information found in this Joint Cybersecurity Advisory, contact your local FBI field office at http://www.fbi.gov/contact-us/field-offices, or the FBI’s 24/7 Cyber Watch (CyWatch) at (855) 292-3937 or by e-mail at CyWatch@fbi.gov or your local WMD Coordinator. When available, please include the following information regarding the incident: date, time, and location of the incident; type of activity; number of people affected; type of equipment used for the activity; the name of the submitting company or organization; and a designated point of contact. To request incident response resources or technical assistance related to these threats, contact CISA at Central@cisa.gov.
This product is marked TLP:WHITE. Subject to standard copyright rules, TLP:WHITE information may be distributed without restriction.. For more information about TLP, see: https://www.us-cert.gov/tlp
THREAT OVERVIEW
Desktop Sharing Software
The FBI, CISA, EPA, and MS-ISAC have observed corrupt insiders and outside cyber actors using desktop sharing software to victimize targets in a range of organizations, including those in the critical infrastructure sectors. In addition to adjusting system operations, cyber actors also use the following techniques:
• Use access granted by desktop sharing software to perform fraudulent wire transfers.
•Inject malicious code that allows the cyber actors to
Hide desktop sharing software windows;
Protect malicious files from being detected; and,
Control desktop sharing software startup parameters to obfuscate their activity.
• Move laterally across a network to increase the scope of activity
TeamViewer, a desktop sharing software, is a legitimate popular tool that has been exploited by cyber actors engaged in targeted social engineering attacks, as well as large scale, indiscriminate phishing campaigns. Desktop sharing software can also be used by employees with vindictive and/or larcenous motivations against employers. Beyond its legitimate uses, TeamViewer allows cyber actors to exercise remote control over computer systems and drop files onto victim computers, making it functionally similar to Remote Access Trojans (RATs). TeamViewer’s legitimate use, however, makes anomalous activity less suspicious to end users and system administrators compared to RATs.
Windows 7 End of Life
On January 14, 2020, Microsoft ended support for the Windows 7 operating system, which includes security updates and technical support unless certain customers purchased an Extended Security Update (ESU) plan. The ESU plan is paid per-device and available for Windows 7 Professional and Enterprise versions, with an increasing price the longer a customer continues use. Microsoft will only offer the ESU plan until January 2023. Continued use of Windows 7 increases the risk of cyber actor exploitation of a computer system.
Cyber actors continue to find entry points into legacy Windows operating systems and leverage Remote Desktop Protocol (RDP) exploits. Microsoft released an emergency patch for its older operating systems, including Windows 7, after an information security researcher discovered an RDP vulnerability in May 2019. Since the end of July 2019, malicious RDP activity has increased with the development of a working commercial exploit for the vulnerability. Cyber actors often use misconfigured or improperly secured RDP access controls to conduct cyber-attacks. The xDedic Marketplace, taken down by law enforcement in 2019, flourished by compromising RDP vulnerabilities around the world.
MITIGATIONS
General Recommendations
The following cyber hygiene measures may help protect against the aforementioned scheme:
• Update to the latest version of the operating system (e.g. Windows 10).
• Use multiple-factor authentication.
• Use strong passwords to protect Remote Desktop Protocol (RDP) credentials.
• Ensure anti-virus, spam filters, and firewalls are up to date, properly configured and secure.
• Audit network configurations and isolate computer systems that cannot be updated.
• Audit your network for systems using RDP, closing unused RDP ports, applying multiple-factor authentication wherever possible, and logging RDP login attempts.
• Audit logs for all remote connection protocols.
• Train users to identify and report attempts at social engineering.
• Identify and suspend access of users exhibiting unusual activity.
• Examples of cyber-physical safety system controls include:
o Size of the chemical pump
o Size of the chemical reservoir
o Gearing on valves
o Pressure switches, etc.
Water and Wastewater Systems Security Recommendations
The following physical security measures serve as additional protective measures:
Install independent cyber-physical safety systems. These are systems that physically prevent dangerous conditions from occurring if the control system is compromised by a threat actor.
• Examples of cyber-physical safety system controls include:
o Size of the chemical pump
o Size of the chemical reservoir
o Gearing on valves
o Pressure switches, etc.
The benefit of these types of controls in the water sector is that smaller systems, with limited cybersecurity capability, can assess their system from a worst-case scenario. The operators can take physical steps to limit the damage. If, for example, cyber actors gain control of a sodium hydroxide pump, they will be unable to raise the pH to dangerous levels.
TeamViewer Software Recommendations
For a more secured implementation of TeamViewer software:
• Do not use unattended access features, such as “Start TeamViewer with Windows and “Grant easy access.”
• Configure TeamViewer service to “manual start,” so that the application and associated background services are stopped when not in use.
• Set random passwords to generate 10-character alphanumeric passwords.
• If using personal passwords, utilize complex rotating passwords of varying lengths. Note: TeamViewer allows users to change connection passwords for each new session. If an end user chooses this option, never save connection passwords as an option as they can be leveraged for persistence.
• When configuring access control for a host, utilize custom settings to tier the access a remote party may attempt to acquire.
• Require remote party to receive confirmation from the host to gain any access other than “view only.” Doing so will ensure that, if an unauthorized party is able to connect via TeamViewer, they will only see a locked screen and will not have keyboard control.
• Utilize the ‘Block and Allow’ list which enables a user to control which other organizational users of TeamViewer may request access to the system. This list can also be used to block users suspected of unauthorized access
The information in this site is intended solely for the personal non-commercial use of the user who accepts full responsibility for its use. While we have taken every precaution to ensure that the content of this site is both current and accurate, errors can occur.