VPS12 Downtime and Scheduled /vz Restore

**Daniel** · January 10th, 2008, 4:18 PM

Hello,

Today, earlier in the morning at roughly 2:00AM EST, VPS12 started experiencing heavy filesystem related problems. Upon reboot, it was diagnosed that the /vz filesystem (this fs contains all of your VE pertinent data) required a filesystem check.

Due to the immense of inodes on this filesystem, it is normal for the fsck's to take an extremely long time, so we allowed the server to continue fsck'ing for the next several hours. Unfortunately, after 6 hours of the fsck running, we started noticing that a lot of data was starting to get moved over to /lost+found. If this would continue, chances are, most of the data would end up there at which point it would be incredibly difficult to restore anything at all.

For this reason, to prevent further difficulties, we cancel'd the ongoing fsck and force mounted /vz to get the server and all the VE's booted up, even if temporarily. Meanwhile, we have hooked up an additional drive to this system which we are using to backup all of the /vz data to.

Once the backup of /vz is complete, we will be reformatting the original /vz partition and copying all of the data back.

To make note - this is not a hardware failure, but rather filesystem corruption to some extent. Reformatting the filesystem will fix the issue and it should not reoccur any time soon.

As of right now, the data is copying, but unfortunately we cannot give an ETA for completion for this step, as there are too many factors involved (general server load, ongoing I/O, etc.). During the backup process, the VPS server might feel a little sluggish, but won't affect overall performance by much. You won't experience any downtime either during this time.

We will update you here via this thread once the backup completes. At that point, another announcement will be made as to when the server is going to be going down to have /vz repopulated.

We thank you for your patience and understanding while we are dealing with these issues.

Thanks,

**Daniel** · January 13th, 2008, 4:17 PM

Hello,

A quick update as to what is currently going on with VPS12.

The data copy of /vz completed successfully, and we were planning on restoring /vz the following day. Unfortunately, due to a prolonged rsync, this was not able to take place, thus we are scheduling the downtime for VPS12 to occur tonight at 12:00AM EST. We expect the maintenance window to be no longer than 8 hours.

We thank you for your patience and understanding while we deal with these issues.

Thanks,

January 10th, 2008, 4:18 PM	#1 (permalink)
Daniel Sysadmin Seasoned Poster Joined in Feb 2005 76 posts Gave thanks: 0 Thanked 42 times	VPS12 Downtime and Scheduled /vz Restore Hello, Today, earlier in the morning at roughly 2:00AM EST, VPS12 started experiencing heavy filesystem related problems. Upon reboot, it was diagnosed that the /vz filesystem (this fs contains all of your VE pertinent data) required a filesystem check. Due to the immense of inodes on this filesystem, it is normal for the fsck's to take an extremely long time, so we allowed the server to continue fsck'ing for the next several hours. Unfortunately, after 6 hours of the fsck running, we started noticing that a lot of data was starting to get moved over to /lost+found. If this would continue, chances are, most of the data would end up there at which point it would be incredibly difficult to restore anything at all. For this reason, to prevent further difficulties, we cancel'd the ongoing fsck and force mounted /vz to get the server and all the VE's booted up, even if temporarily. Meanwhile, we have hooked up an additional drive to this system which we are using to backup all of the /vz data to. Once the backup of /vz is complete, we will be reformatting the original /vz partition and copying all of the data back. To make note - this is not a hardware failure, but rather filesystem corruption to some extent. Reformatting the filesystem will fix the issue and it should not reoccur any time soon. As of right now, the data is copying, but unfortunately we cannot give an ETA for completion for this step, as there are too many factors involved (general server load, ongoing I/O, etc.). During the backup process, the VPS server might feel a little sluggish, but won't affect overall performance by much. You won't experience any downtime either during this time. We will update you here via this thread once the backup completes. At that point, another announcement will be made as to when the server is going to be going down to have /vz repopulated. We thank you for your patience and understanding while we are dealing with these issues. Thanks,

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

January 13th, 2008, 4:17 PM	#2 (permalink)
Daniel Sysadmin Seasoned Poster Joined in Feb 2005 76 posts Gave thanks: 0 Thanked 42 times	Hello, A quick update as to what is currently going on with VPS12. The data copy of /vz completed successfully, and we were planning on restoring /vz the following day. Unfortunately, due to a prolonged rsync, this was not able to take place, thus we are scheduling the downtime for VPS12 to occur tonight at 12:00AM EST. We expect the maintenance window to be no longer than 8 hours. We thank you for your patience and understanding while we deal with these issues. Thanks,

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)