Dear Clients,

We'd like to extend our apologies for the service interruption that occurred Friday June 18th. You rely on us to provide solid consistent service to your websites, dedicated servers affected we are sorry.

Incident report that lays out as follow the cause of the disruption, actions we're taking to immediately remedy the situation and long term plans to increase overall service quality.

On June 18th, 2010 service was degraded at our Dearborn, Michigan Data Center (DTW) for roughly 1.5 hours between approximately 18:35 EDT and 20:05 EDT due to a UPS malfunction at the carrier hotel (CH) in Southfield, Michigan which we use for primary Internet access. Power was not affected in our Dearborn facility but connectivity was interrupted for many clients due to this failure.
Incident Time-line (CH)

* 17:21 EDT – A single phase of the input power supply to the UPS powering the CH’s facilities had a voltage surge causing their UPS to discharge until battery power was exhausted. Since this was not a utility power outage their on-site generator did not activate. CH’s on-site staff was alerted immediately by this and engaged electricians to diagnose and solve the problem. Our internal staff was also alerted via our external monitoring and arrived on-site at CH starting at 18:00 EDT.
* 18:35 EDT – On-site electricians brought down AC power to initiate UPS repairs. This caused a loss of power to our core routers which caused degradation in service for many of our clients.
* 20:05 EDT – Technicians completed repairs and returned power to the UPS.

Internal Changes (CH)

Given the above issues we have further diversified our power requirements from CH to include a DC power feed to our core routing gear along with the AC power that was already being supplied. We are also working with the carriers themselves to do the same. Our routers are already equipped with dual power supplies but the CH has no ability to provide true “A/B” power via AC so an AC/DC “A/B” setup will supply the level of redundancy required and alleviate future AC interruptions.

Long term we are announcing this week the acquisition of our second Southeastern Michigan Data Center which will alleviate our reliance on the carrier hotel as a primary point of Internet connectivity.
Internal Changes (DTW)

Weather Statement

The Southfield and Dearborn, Michigan areas were under a Severe Thunderstorm Warning from 21:29 EDT to 22:45 EDT on June 18th, 2010. While this is notable it had no affect on either the CH issue or on the power systems in our Dearborn facility. The power systems under our control worked as planned and while our UPS systems did see some aberrations in power quality our generator was never needed to power the building and the UPS systems performed as planned.

Sincerely,