Thread: Heart servers
View Single Post
  #8  
Old 11-02-2016, 06:00 PM
Steve_PPP's Avatar
Steve_PPP Steve_PPP is offline
Admin, Meets/Events Organiser
 
Join Date: Sep 2008
Location: Burgess Hill, Sussex
Posts: 13,292
Default

Quote:
Originally Posted by iPond View Post
Did anyone else have issues today? Was getting 503 errors most of the day and unable to access?
Yes, the hosts have had more problems today. Their latest update:

Current Summary Of Issues (11/02/16 - 17:30)

Webmail - The webmail service appears to be back up and running again now, and you should be able to access your mailboxes through this correctly.

VPS & Hybrid Servers - We currently have two VPS 'kvmhosts' down - these are "kvmhost248" and "kvmhost249". Sadly it will likely take most of the morning to resolve this. All other kvmhosts for VPS and Hybrid Servers are online with all servers started. However if your VPS or Hybrid Server is definitely still down and a reboot does not fix this, please contact support.

Linux Shared Hosting - An issue still exists on the following web servers unfortunately; web75, web76, web77 and web78. It did appear that this was resolved, but further investigation has shown that there is another issue here. A separate issue has also been identified on web servers 175-183, and is under investigation now.

Premium Hosting - The Premium Hosting service appears to be back up and running again now, and the issue with MySQL database creation has also been addressed and resolved.


SES runs on server web183. I wouldn't be surprised if we see a few more hiccups over the next 24hrs before it settles down but hopefully not.

As an IT infrastructure guy, I have a lot of sympathy for them - it sounds like they had electrical engineers working in their datacenter on *supposedly* non-disruptive jobs but they screwed something up and dropped power to (or spiked) a bunch of racks, leaving their IT guys to pick up the mess. They posted an update earlier today:

Quote:
We are still working our very hardest to resolve the remaining kvmhost and webserver (web75, web76, web77 and web78) issues, many of our staff have been in the office since 8am yesterday and are still here now.
Looks like they were pulling an all-nighter to try and sort it out. Been there, done that. It's a right pain in the ass.

Quote:
Originally Posted by Rdlangy1 View Post
Nice to see we all survived ha ha
Yeah, you spoke too soon mate
Reply With Quote