Daniel

Hello,

I apologize for the long delay in getting everyone updated; we've been working on this server and other issues non-stop throughout today, and must have lost track of time.

The current situation with SH110 :

As Kayla mentioned in the previous post - the restore is moving along, although the whole process is really slow. As noted, both of the servers are hooked up via a gigabit link, yet the transfer is still really slow, so we are thinking some sort of a bottleneck is occurring somewhere else. We've done everything in our power to optimize the speed on the restore, and this is probably the fastest it will go.

Typically during restores, you would generally see sites/accounts come up as they get restored; unfortunately this restore is a different case. I say this because if we would've ran all of the services on the server, the restore would probably take 2 or 3 or possibly even 4 times as long, as the load on the server would just sky rocket and ultimately either slow down the machine to a crawl or ultimately crash it. Due to this, we have, for now, disabled all of the services on the machine until all data is restored.

There is roughly 240GB left to restore. Judging by the current statistics, everything should be done by tomorrow morning between 6-9AM EST

I understand how upsetting this might be, but believe us when I say we really are doing our best here. We've got staff continuously checking on the server 24/7 and most of us are still trying to figure out a way to speed this up (with little success unfortunately).

Paul or someone else from the NOC will update everyone within the next couple hours on the status of the restore.

Again, my deepest and most sincere apologies go out to all of the customers on SH110. This will get finished though, and a brighter day is sure to come

Thanks,

Paul

Hello,

The restore is still on-going. The data is moving at a more steady pace and we will keep everyone updated.

Daniel

Alright guys, we are heading down the last mile of this restore. All system data has been restored, and we have restored all accounts up to the letter "p" (/home data was restored in alphabetical order). This is the hardest and longest part, since there is so much data, and especially a large amount of files as well. It is incredibly difficult at this point to give further ETA's, due to the ridiculous fluctuations in the transfer speed - we're talking at times it will transfer 90MB/s and then slow down to a crawl and do 1.5MB/s and lower.

Instead of giving further "it'll be done soon"'s, I'll give you exact statistics:

There is roughly 520-540GB's of data in /home. We have restored so far ~450GB of it.

There are exactly 1,838,715 files in /home, and we have so far restored 722,695 of them.

What this means is that, a huge chunk of large files were in the beginning of the restore, and the majority of files now should be rather small and transfer much quicker (as there is less than 100GB's left to restore).

We are planning on soon turning up some services to begin serving at least those accounts that are already restored (a through p), but this is dependent on how well the system handles the extra load. As noted in my previous replies - during large file transfers, the system's load averages jump really high, and we wouldn't want to risk slowing the transfer rates further than they are already.

IF we do put the services up, we are definitely going to make a post about it here.

I am currently evaluating how viable the service turn up is, and will let you know the outcome shortly.

Thanks,

Paul

Hello,

we are completing the transfer now. Services will be turned up at 5am at the latest.


Thank you everyone for holding up during this trying time.

Paul

Hello,

Server is now online, Please open a ticket if you have any issues.

Thanks.

Andy

Tuesday night, April 22nd we will be upgrading PHP and MySQL to version 5 on sh110.surpasshosting.com.

There will be some minimum downtime past 12am EST.