By Evan Sparks
It was dawn on a cloudy, mild Saturday in fall, and John Smith* was about ready to go to bed. As the head of IT for a community bank in the Southeast, he’d been up all night installing a new release from Any Bank’s∗ core provider. He was finally done at 4:30 in the morning.
Everything was going well, until 6:15, when he got a call from his systems operator. “John, I can’t get to the email server.”
It was Any Bank’s turn to face one of the most pernicious cyber threats banks face today.
While ransomware attacks and payments dipped in 2022, they remained at elevated levels from prior years, with the average ransom demand $4.2 million in 2022, according to cybersecurity firm CrowdStrike. The Wall Street Journal reported that improved email security practices have made ransomware attacks harder for criminals—but it still remains a potent threat vector.
For obvious reasons—ranging from customer perception to threat mitigation—banks can be reluctant to talk about the details of any cybersecurity attack. However, given the growth, complexity and sophistication of ransomware attacks, the CEO of Any Bank agreed to share his bank’s behind-the-scenes story. The story covers the immediate aftermath, recovery and response efforts, and what the bank has learned about how to prevent future attacks.
Due to the sensitivity of the topic, I agreed to quote him and his IT team anonymously, and I’ve changed a few identifying details here and there, including the bank’s name. Otherwise, this is a complete and accurate look inside the anatomy of a ransomware hack at a community bank.
The first few hours
The first step for Smith’s IT team was to physically disconnect the bank’s network from the outside world—while this may have prevented additional damage, Smith knew most of the damage was likely already done.
Smith immediately began his long commute back to the office. Despite the COVID-19 pandemic, which was still raging at the time, the team would need to deal with this in-person. From the car, he called his boss, the bank president, who immediately began a six-hour drive home from a family vacation at the beach. His assistant called in the rest of the IT team and senior leadership.
As he drove in, Smith asked himself, “‘Was this going to be my last day at the bank? What was going to happen?’ You can’t help it; your thoughts go there in that hour-and-a-half drive I had to make to get there.”
By 9 a.m., when Smith arrived at the bank headquarters, he got the full bad news from his systems administrator and the outside vendor Any Bank uses for third-tier tech support. Eighty-two servers were down and fully encrypted. The on-site backups were also completely encrypted. Any Bank hosted its own disaster recovery backups, but the hackers had found the off-site backups segmented on another part of the network—and deleted those too. The phone system was also completely down and encrypted.
“In summary, we had no way to communicate to our internal organization electronically on our system, and furthermore, we had no way for our customers to contact us,” Smith says. “It’s nine o’clock by now; we open at seven. So we’ve already been two hours without any kind of connectivity.”
Patching together a customer solution
Susan Jones*, Any Bank’s head of retail banking and marketing, was texting with division presidents. She jury-rigged a new contact center by having the bank’s communications broker forward calls to a bank of employees’ personal cell phones. (Employees were later compensated for the use of their phones).
Jones also began deploying the bank’s playbook for customer communication, enlisting an outside PR firm to help and setting up a webpage on the bank’s site. The first message was just the basics, avoiding language that would cause undue alarm, describing it as an “an IT security incident disrupting certain business operations.”
The message emphasized that call centers had limited account information but that debit cards would continue to process. (The bank’s AS/400 IBM mainframe that ran the debit card processing was unaffected by the hack.) And Smith’s team was able to find a different path to core system access, so the bank could still provide customer balances.
Calling in the cavalry
Meanwhile, Smith and his team continued to assess the situation and plan a response. The bank’s cyber attack playbook laid out the next steps clearly. They notified law enforcement and the bank’s cyber insurance carrier. They got the bank’s counsel on-site. Any Bank’s IT vendor contacted a remediation company that specializes in ransomware, and the company quickly figured out that the hackers were a group from Russia.
However, the insurer’s preferred remediation vendor was unavailable—the bank was part of the largest cyber attack weekend in recent history—so the insurer approved the other vendor, and they got to work.
By this point, it was 2 p.m. on Saturday.
The challenge ahead was daunting. The bank could limp through the weekend on cell phones, but it needed a functional call center by Monday. And more importantly, it needed to have transaction channels back up in time to process dailies for Monday morning.
Taking back control
While Smith and his team were working to understand the scale of the problem and set up temporary workarounds, the CEO, Dan Miller*, was hard at work with the insurer, remediation firm and bank counsel to identify a path forward.
Not all banks pay a ransom, notes Sarah Cleves, a senior attorney at ABA Insurance Services, which ABA endorses for several bank insurance products. If the bank has a viable backup not touched by the threat actor, it might not need to pay. However, Cleves notes that sometimes insureds pay the ransom even if they have a backup, on the theory that paying the ransom might stop the hacked information from being put on the dark web and thus limit potential liability down the road. In 2021, according to a Delinea survey, 83 percent of companies experiencing a ransomware attack ultimately paid the ransom.
The engagement Miller and his vendors had with the hackers was nothing like a Hollywood conception of a cyber hacker. “It was like you’re dealing with another corporation,” he says. Ransomware has become highly professionalized—in fact, there are criminal enterprises that provide “ransomware as a service” to other criminal enterprises. As Bloomberg columnist Matt Levine puts it, “ransomware is an obviously businesslike crime—you need sales and customer service and compliance and a bunch of developers sitting at a computer—and it turns out to be a surprisingly normal business. Ransomware companies are pretty much normal tech companies whose product is crime.”
Ultimately, there was no way to get systems back online in time without dealing with the ransom hackers, Smith says. With the back and forth over the ransom, it wasn’t until the early hours on Sunday that the bank got the keys to start the decryption process.
But it wasn’t straightforward at all. “As we decrypted on Sunday, we would notice they would immediately re-encrypt,” Smith says. “How are they doing that? There’s no access from the outside world. How are they re-encrypting our hardware, our servers? We learned they had used our own hardware against us.” Any Bank’s 270 PCs in the field had been compromised with malware that would reinfect the servers—and with more than 300 encryption keys coming from any number of the 270 PCs, the scale of the re-encryption was exponential, Smith explains. It took the remediation company writing a script and “slamming it against every file at a very high rate of speed until it found the key that belonged to it, decrypted it and moved to the next file.” It took hours for the larger file servers.
In the end, the bank opened on Monday and never missed a processing day, Miller says. “We never had to tell a customer, ‘We don’t know what your balance is.’”
Systems were 95 percent restored by Wednesday—at least to whatever the new normal was going to be—says Smith. It was a herculean lift, compressing weeks of work into just five days. Unsurprisingly, fatigue was a big challenge. Smith says he took a quick nap on Saturday night during negotiations but after that couple hours of sleep, he was awake 35 hours straight from Sunday to Tuesday. “You’ve got to have depth within your team, because the fatigue factor sets in.”
Smith adds: “If you give me a choice between having my operations center burn to the ground and going through this again, I’d say give me the match.”
Dynamics of ransomware insurance
Any Bank’s insurance carrier was a focal point of coordination in the response, so I asked ABA Insurance Services to elaborate on ransomware insurance. Today, 90 percent of cyber claims are covered under a specific cyber insurance rider or policy, which will include several requirements for how banks must prepare and configure.
While cyber insurance premiums shot up dramatically from 2015 to 2020 (nearly tripling, according to Fitch Ratings), “that’s not happening anymore,” notes Lisa Micciche, AVP for the professional lines program at ABAIS. “We’re talking more of a traditional insurance rate increase of five or 10 percent.”
When a bank experiences a cyber incident, it should contact its insurer within the first couple of hours of discovery, Micciche says. ABAIS has a recommended breach team in place via a 24/7 hotline. Legal advisers help identify whether the exposure or loss of data triggers state and federal notification requirements, adds Sarah Cleves, a senior attorney at ABAIS.
The aftermath
Unsurprisingly, the hackers got in via a phishing attack. An employee clicked on a compromised link in an email the Monday before the ransom demand that inserted a bot into the network. “They moved through the network laterally until they got elevated privileges,” Smith notes.
Any Bank thought it had been doing enough on email security, but the attack jolted Smith. Now the bank has doubled password length requirements, added a forced password change every 45 days and required multi-factor authentication. “If the employees out in the field are not complaining about your security, you’re probably not doing enough,” he says. The bank also blocks embedded links, strips attachments if they don’t meet security parameters and checks IP addresses and domain registration from senders. The bank now tests employees to keep them on their toes. If a person routinely fails these phishing tests, the company policy specifies discipline—with the costs of a breach very much in mind.
On the IT side, Any Bank added a true air-gap backup in the cloud since the disaster recovery backup wasn’t safe enough from the core network.
Any Bank notified its regulator early on, but its incident happened before the interagency notification rule for computer-security incidents was finalized, so banks need to incorporate those requirements into their plan. The bank also notified local law enforcement and the FBI early in the process.
For CEOs, Miller emphasizes the importance of tabletop exercises—so that everyone knows what to do in a crisis moment—and in-depth knowledge of insurance coverage. “A lot of community bank CEOs don’t really understand cyber insurance coverage—and it is complicated, without getting deep into the weeds on it. Our policy actually covered a lot of this area on the bond issue, whereas now most of the coverages are for this type of thing are under the specific cyber portion of your coverage.”
Vendor selection is critical—for IT, remediation, insurance and more. “Make sure you do business with people who care as much about your business as you do,” Miller says. Any Bank speaks highly of its insurer, which covered everything up to the policy limits after the deductible. Lisa Micciche of ABA Insurance Services notes that generally in the industry, in cases where a ransom is paid, about two-thirds goes to the ransom and a third goes to expenses associated with responding.
Any Bank also revisited its data retention policies. “You really need to look at how much your data you’re retaining on your network,” Smith says. The bank is imposing much more stringent retention rules to ensure it keeps only data essential for operations and regulatory compliance.
“What are they after?” he points out. “They’re after your data. If they have your data and they have access to your data, they can get whatever else they want from you. That’s the lifeblood of your bank, and they know that.”
*The names used in this article are pseudonyms.
