As I’m sure many of your are already aware, yesterday’s extreme weather in Denver (the ‘Bomb Cyclone’ as the meteorologists are calling it) caused a brief system outage for all of RMMI’s hosted services. All of our client-facing services have been fully operational since late yesterday afternoon, and all of our internal and backup systems have now been fully restored. Short of an IT Director who’s running purely on caffeine at this point, there shouldn’t be any further adverse effects.

Now that things are back to normal, we wanted to take a minute to explain the series of events that led to the outage and the steps we take to make sure that your data is safe.

All of RMMI’s hosted services are housed in a Tier 3 Gold Co-location Data Center here in Denver, CO. For those of you unfamiliar with the terminology, a Co-location Data Center (or Colo) is a shared data center that leases rack-space, power, cooling, and data connections to individual companies like us. Sharing the extreme costs involved in building a reliable data center across many clients is the most cost effective way for a company of our size to provide the services we offer. Prior to yesterday’s events, our Colo provider had maintained more that 15 years of up-time without an outage.

Around 10:45 AM yesterday morning, our Colo lost power from both of it’s supplying power grids within minutes of each other. There are battery systems in place that supply temporary power in the event of an outage and give the generators time to come online; however, the battery systems that supply our equipment failed – this resulted in a complete loss of power to all of our equipment.

By 12:00 PM the failed power system had been partially bypassed and a backup generator had been brought online. We were partially able to power up our equipment and had restored access to the Hosted PaperVision system by 12:15 PM

We received a notification from our Colo provider that full power had been restored at 3:45 PM. We were able to bring the rest of our systems online and all client-facing services were fully operational by 4:15 PM.

We know this outage has caused some inconveniences for many of you, and we hope you understand the extreme circumstances we were up against. Our Colo provider has an exemplary track-record, and we have the fullest confidence that despite yesterday’s events, they are still the best provider in the industry.

If you have any questions or concerns stemming from yesterday’s events, please contact me directly at rick@rmm-i.com