Since Monday night, the flooding and power outages caused by Hurricane Sandy have severely affected our availability. Harvest’s primary datacenter is located in the Chelsea neighborhood of New York City. The building is home to Google’s NY operations and a major datacenter in the north eastern U.S. While flooding has not affected our datacenter, the power outages in New York City, despite backup generators, have caused repeated issues.

Our team has been working around the clock to keep Harvest available. This post is to keep you informed of our current status and share the details of this event with you thus far.

Current Status

Saturday, Nov 3rd 7:00am ET – At approximately 5:52PM ET on Friday Nov 2nd, Harvest became unavailable due to problems related to the restoration of commercial power in the datacenter. The Harvest team worked through the night to safely restore service after the transition back to commercial power. By 7:00am ET, Harvest service is once again accessible to all customers. We thank you for the extra patience during this weekend evening which allowed us to restore our service properly. With commercial power in place at the data center, we take a big step towards stable conditions.

Thursday, Nov 1st 1:45pm ET – We continue to operate normally on backup generators with plenty of fuel at the datacenter. Based on ConEd’s latest estimate, they expect power to be restored in lower Manhattan by Saturday.

Wednesday, Oct 31st 7:30pm ET – Harvest experienced intermittent network issues from approximately 6:17pm ET till about 7:00pm ET. During this time, some customers were not able to access Harvest. The network issues were resolved at the datacenter and service should be 100% accessible at this time.

Wednesday, Oct 31st 12:30pm ET – Harvest is available and operating normally. Due to the power outage in lower Manhattan, all services are still powered by backup generators. Our datacenter has ample fuel and the ability to refuel as needed. To minimize downtime for our customers, we will continue to operate as-is and monitor the situation closely. Based on a statement from Con Edison (the power company in New York City), it will take an estimated 3 more days before commercial power is restored in lower Manhattan. We will provide updates if and when estimates change.

In the meantime, we have a backup plan in place to migrate our services should we need to (more details on that below). Note that all data continues to be safely backed up on-site and off-site in several locations during this time.

Please also note that you can always check our status at http://harveststatus.com

What’s Been Happening

Around 11:35pm ET on Monday, Oct 29th, Harvest went offline due to a power outage in our datacenter. During this time, datacenter staff worked to restore power and network connectivity. By 6:25am ET on Tuesday, Oct 30th, service had been restored. Harvest continued to operate on backup power generators during this time. Harvest operated normally for the rest of Oct 30th.

At 5:48am ET on Tuesday, Oct 31st, Harvest experienced a second outage window due to power failures in neighboring datacenters. This caused network connection problems which made Harvest unreachable by customers. By 7:00am ET, network paths had been re-routed and made Harvest accessible to most customers again. By 9:00am ET, all known network issues were resolved and Harvest became accessible for all customers.

Contingency Plan

While our core datacenter issues were being addressed, our team has been working tirelessly to ensure that we can reliably serve Harvest from a different datacenter if necessary. After around the clock efforts led by Harvester Warwick Poole to deploy and test the setup in a new datacenter, with support provided by our entire team, we are ready to take this step if necessary.

In the event of another extensive outage, we will immediately begin to switch service to a new datacenter located in Dallas, Texas. We expect this move to take approximately 2 hours, during which time Harvest will not be available. Should we need make the switch, we will alert all account owners via email and provide information through Twitter and harveststatus.com.

Thank You

Thanks to all our customers for the understanding and overwhelming support sent our way during this crisis. We couldn’t be more proud to have you good people as our customers. We will continue to do our best to provide service to you during this challenging time. If you have any concerns, please do not hesitate to contact us at support@harvestapp.com.