msp

The ripple effect of the Microsoft outage on MSPs

Team SuperOps

Illustration: Suman

The recent Microsoft outage sent shockwaves across industries, causing disruptions that ranged from flight delays to halted business operations. In a world where technology is the backbone of almost every operation, the outage has served as a stark reminder of how interdependent our IT infrastructure truly is. . For MSPs, this outage was particularly challenging as it highlighted the critical role they play in maintaining business continuity and managing IT services for their clients.

The importance of IT interdependence

From cloud services and virtual machines to on-premise servers and applications, a disruption in one part of the system can have cascading effects. This interdependence requires MSPs to always be prepared to address issues quickly and efficiently, ensuring minimal downtime for their clients. When a major provider like Microsoft experiences an outage, the ripple effects are felt globally, affecting everything from financial transactions to healthcare services.

At SuperOps, we understand the pressures MSPs face during such critical times. Our team is committed to supporting our partners in any way we can. To save you valuable time and effort, here are some necessary steps to remediate issues caused by the Microsoft outage.

Steps to remediate issues

Boot Windows into Safe Mode or the Windows Recovery Environment
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291.sys” and delete it*
Boot the host normally

If you are able to boot an impacted computer into Safe Mode, you can use tools like Screen Connect or Splashtop to connect remotely to the asset. If your RMM is configured to run in Safe Mode, you can script the removal of the “C-00000291*.sys” file.

Emerging threats and precautions

We are noticing new threats, including individuals impersonating CrowdStrike or taking advantage of the situation in other malicious ways. It’s essential to be vigilant when using third-party services, particularly those you haven’t worked with before. Always verify the credibility of any third-party support before proceeding.

Steps for virtual machines in Azure

If you are running a virtual machine in Azure, Microsoft has released steps to repair your OS disk offline:

Attach an unmanaged disk to a VM for offline repair
If your disk is encrypted, follow additional steps to unlock it:
- Unlocking an encrypted disk for offline repair
Once you have accessed the disk, follow the original steps to delete the “C-00000291.sys” file.*

Alternative recovery solutions

If you are unable to boot the machine into Safe Mode, we suggest using your BCDR solution to virtualize the system in the cloud or on a local appliance. Alternatively, perform a full Bare Metal Restore (BMR). Be sure to choose a recovery point from before 19:00 UTC on the 18th of July.