Did everyone else with a colo-mojo account just have all database servers vanish, or am I lucky?

Bill · 2013-10-05, 08:16 AM

Well, it's been an insane night for me. I have one site that I do for a local organization that is having it's big yearly fundraiser today, and the site is dead dead dead. I havre told godaddy to move the nameservers another server I have, but godaddy is being pokey about updating the namservers, they have not resolved to the temp emergency page yet.

(removed the url - not smart to leave a nonadult url up on an adult board. I always intended to delete it just as soon as a few had seen it. especially you Cleo, I know you are interested in these things,a nd up in the bleak quiet mornings, lol.)

Most of my wordpress has come back, with quite a few errors and plugins failed still, but all my drupal sites are absolutely trashed, like that one whose link I pasted.

I explained to a few other customers who I am working on things for that the serves are down, luckily their sites are not mission critical right now, they can handle it.

I have a mix of good backups and shitty backups, I have never had trouble like this before, even at the worst hosts I ever had I could always get them to restore from a few days back, I guess that isn't the case now with mojo-colo? I just don't know.

Chris just fucking spoiled me. I have to remember that the relationship between host and builder is more adversarial than I have gotten used to.

Bill · 2013-10-05, 08:21 AM

What is truly weird is that about three hours into the database server crash, the ftp servers died too. So I could not even ftp into the server and throw up an emergency html page with event info and maps onto the domain.

How the fuck do ftp servers die three hours after the database servers?

The techs tried to answer at first, and did restore part of the database servers, but they are half corrupted. Then silence for hours. A short while ago a replay saying sorry, they are stuck, and they have to go into the datacenter and fix hardware.

Bill · 2013-10-05, 08:28 AM

Also, to be fair, I manage a certain number of sites that are on ordinary commercial hosts. I use midphase a lot for this kind of thing, just because I have been with them a while.

At midphase, the only techs left on the weekends, from Friday about 3 to Monday about 8am, are lower level with low power accounts. they just can't do all that much for real problems.

One Friday a module corrupted the database server for a site with about 9000 pages. the backups I had would have sufficed, but, I would have lost 20 or so pages. BUT, the low level techs could not, for some reason, restart the crashed database, so my backups were useless anyway.

Finally on sunday morning a tech came in with the power to give me a database on another server, and I pointed the drupal at the new sever with the old backup and had the site alive.

the original database server and their backups tho took a further two days to reactivate, and then walk back thru two days of corrupted restores to hit one that was sound.

That was a weird experience. This is heaven compared to that. Consumer accounts - they are fine until they SUCK, lol.

2013-10-05, 08:16 AM	#1
Bill Selling porn allows me to stay in a constant state of Bliss - ain't that a trip! Join Date: Apr 2003 Posts: 3,914	Well, it's been an insane night for me. I have one site that I do for a local organization that is having it's big yearly fundraiser today, and the site is dead dead dead. I havre told godaddy to move the nameservers another server I have, but godaddy is being pokey about updating the namservers, they have not resolved to the temp emergency page yet. (removed the url - not smart to leave a nonadult url up on an adult board. I always intended to delete it just as soon as a few had seen it. especially you Cleo, I know you are interested in these things,a nd up in the bleak quiet mornings, lol.) Most of my wordpress has come back, with quite a few errors and plugins failed still, but all my drupal sites are absolutely trashed, like that one whose link I pasted. I explained to a few other customers who I am working on things for that the serves are down, luckily their sites are not mission critical right now, they can handle it. I have a mix of good backups and shitty backups, I have never had trouble like this before, even at the worst hosts I ever had I could always get them to restore from a few days back, I guess that isn't the case now with mojo-colo? I just don't know. Chris just fucking spoiled me. I have to remember that the relationship between host and builder is more adversarial than I have gotten used to. Last edited by Bill; 2013-10-05 at 08:50 AM..

2013-10-05, 08:21 AM	#2
Bill Selling porn allows me to stay in a constant state of Bliss - ain't that a trip! Join Date: Apr 2003 Posts: 3,914	What is truly weird is that about three hours into the database server crash, the ftp servers died too. So I could not even ftp into the server and throw up an emergency html page with event info and maps onto the domain. How the fuck do ftp servers die three hours after the database servers? The techs tried to answer at first, and did restore part of the database servers, but they are half corrupted. Then silence for hours. A short while ago a replay saying sorry, they are stuck, and they have to go into the datacenter and fix hardware.

2013-10-05, 08:28 AM	#3
Bill Selling porn allows me to stay in a constant state of Bliss - ain't that a trip! Join Date: Apr 2003 Posts: 3,914	Also, to be fair, I manage a certain number of sites that are on ordinary commercial hosts. I use midphase a lot for this kind of thing, just because I have been with them a while. At midphase, the only techs left on the weekends, from Friday about 3 to Monday about 8am, are lower level with low power accounts. they just can't do all that much for real problems. One Friday a module corrupted the database server for a site with about 9000 pages. the backups I had would have sufficed, but, I would have lost 20 or so pages. BUT, the low level techs could not, for some reason, restart the crashed database, so my backups were useless anyway. Finally on sunday morning a tech came in with the power to give me a database on another server, and I pointed the drupal at the new sever with the old backup and had the site alive. the original database server and their backups tho took a further two days to reactivate, and then walk back thru two days of corrupted restores to hit one that was sound. That was a weird experience. This is heaven compared to that. Consumer accounts - they are fine until they SUCK, lol.