Author Archives: Lav Patel

Solarwinds hack explained.

2020-2021 Solarwinds hack explained in details by a software engineer.

According to The Hill, President Biden said Thursday, February 4, 2021, that his administration is launching an “urgent initiative” to improve the nation’s cybersecurity, pointing to concerns around malign efforts by Russia and China. On December 13, 2020, Reuters broke the story that the Treasury Department has been compromised by sophisticated hackers, and afterwords numerous organize confirm the story. Washington Post attributed hack to cozy bear (Russian SVR). To understand Solarwinds hack, first need to understand what Solarwinds does.

What Solarwinds does?

Solarwinds is a software company which has numerous software offering for systems managements. These software programs are primarily managed by IT professionals at large institutions.

What Solarwinds Oroin NMS does?

Orion NMS (Network Management Systems) is the most popular software of Solarwinds and has capabilities of managing and monitoring a wide variety of devices such as servers, desktop, laptop, network devices etc.

Who uses Solarwinds?

The Company has 300,000+ customers worldwide, including 425 of US Fortune 500 companies.

How hacker compromises Solarwinds systems? And Why Solarwinds hack spread for so long without detection?

Since Solarwinds NMS has access to so many devices in an institution and so many institutions use Solarwinds NMS, it becomes a prime target of hackers.

NMS are run on two modes. First is monitoring and second is monitoring/managing. Many of the systems are configured to do both. So if NMS compromised, attacker can do what NMS can do.

According to SEC filling by Solarwinds, hackers infiltrated as early as March 2020, and It took almost 9 months to detect the first instance of hack, so why it took so long to figure out even tough 425 out of Fortunes 500 companies were using it.

Hacker compromise build server of Solarwinds and added malware(named sunburst) to existing software during build process and signed with Solarwinds digital signature. As once software signed by official signature, many of the anti-malware and anti-virus software accept that software behavior as normal, As anti-malware and anti-virus companies don’t know Solarwinds signature has been compromised. This is still evolving situation and some argue only build process was compromise. (But anyhow digital signature became useless).

Sophistication does not stop here. Malware was written in away, so it will stay dormant for first 15 days or so anti-malware and anti-virus can’t associate changes in network pattern to change in Solarwinds software version.

Not just that, Hackers make sure that sunburst maleware’s outgoing traffic look very similar to typical Solarwinds traffic. It seems like the hackers’ goal was to stay in the infected systems as long as possible.

The story gets even crazy, before executing sunburst malware, hacker will make sure it is not private IP or Microsoft IP ranges as it seems like hackers were aware that Microsoft has capabilities to capture this kind of activity, so hackers make sure malware does not connect to Microsoft IP. Kudos to Microsoft.

What can you do to protect your institute against Solarwinds hack?

The situation is still evolving so the current best solution is to work with Solarwinds help desk to resolve the threat. You can get the latest information through twitter hashtag:

  1. #SolarWinds
  2. #SolarWindsOrion
  3. #UNC2542
  4. #APT29
  5. #CozyBear
  6. #SunBurst

DHS Guidelines against Solarwinds hack

The Department of Homeland Security (DHS) has also issued guidelines. Here is a summary of what DHS guidelines

  1. Make image of memory and OS of all instance which are hosting Solarwinds Orion NMS
  2. Analyze unusual activity like new users, service/app account and privileges.
  3. Analyze unusual traffic like new external DNS and other indications of compromise.
  4. Disconnect or power down SolarWinds Orion products, versions 2019.4 through 2020.2.1 HF1, from their network. Future communication from CISA will clarify which version to install and how to obtain securely.
  5. Block all traffic from and to hosts, external to enterprise where Solarwinds Orion was installed.
  6. Identify and remove all threat actor-controlled accounts
  7. After (and only after) all threat actor-controlled accounts and identified persistence mechanisms have been removed:
    1. Treat all hosts monitored by the SolarWinds Orion monitoring software as compromised by threat actors and assume that further persistence mechanisms have been deployed.
    2. Rebuild hosts monitored by the SolarWinds Orion monitoring software using trusted sources.
    3. Reset all credentials used by or stored in SolarWinds software. Such credentials should be considered compromised.
    4. Take actions to remediate kerberoasting, including, as necessary or appropriate, engaging with a 3rd party with experience eradicating APTs from enterprise networks.

My 2cents

  1. Assume all Solarwinds Orion is compromise (even releases before March), so contentiously monitor till proven otherwise.
  2. Block Internet traffic to as many servers as you can.
  3. If you are paranoid, and you have the right to be paranoid even after applying Solarwinds patch. There is possibility that hackers could have created an additional backdoor, so the nuclear option is to rebuild your most critical servers from scratch using NIST-850 and CIS benchmark in a separate network.

What can you do to protect YOURSELF against Solarwinds hack?

This particular security issue is affecting more to the large institution than an individual, but see “General Security Recommendation” section of the articles.

General Security Recommendation.

  1. Don’t give any user or application more access than it needs.
  2. Always use SSL or tunnel even if you are inside secure VPN to avoid man in middle attack.
  3. Always use DNS over SSL to avoid man in middle attack.
  4. Know what is the most important data/intellectual properties (IP) you have. Have it in a separate network than the rest of the network
  5. Only install what you really need.
  6. Have outbound firewall as well inbound firewall especially for your critical data/intellectual properties (IP).
  7. Have additional layers of firewall for your important data/ intellectual properties (IP) which is not controlled by the same user or app.
  8. Don’t use same password again
  9. Have 2-factor authentication.
  10. Use CIS Benchmark
  11. Use NIST-850 or alike policy
  12. At some level CyberSecurity is a cat and mouse game, so subscribe to a security threat mailing list like CISA

This list no way complete, but just the beginning.

About Me

I am a Lead Software Engineer at University of Kansas Medical Center, you can read more bout at my homepage.

Subscribe Now!

                                                                                                                            

Further reading for Solarwinds hack

  • The Massive SolarWinds Hack Explained in Context

Disclosures: This article is my own opinion for entertainment and educational purposes only, and it could be inaccurate as any other human can have. This articles ideas are not represented by my current or past employers.