Kayla

This thread is for any issues/updates/upgrades occurring with shared server SH111. Please subscribe to receive instant email updates.

register_globals is off
allow_url_fopen is off

Ruby on Rails is installed and the following gems,

rootsh110 [~]# gem list

*** LOCAL GEMS ***

actionmailer (1.3.2)
Service layer for easy email delivery and testing.

actionpack (1.13.2)
Web-flow and rendering framework putting the VC in MVC.

actionwebservice (1.2.2)
Web service support for Action Pack.

activerecord (1.15.2)
Implements the ActiveRecord pattern for ORM.

activesupport (1.4.1)
Support and utility classes used by the Rails framework.

acts_as_taggable (2.0.2)
An acts-as Mixin for easy applying and searching tags/folksnomies on
Active Record objects

archive-tar-minitar (0.5.1)
Provides POSIX tarchive management from Ruby programs.

Bloglines4R (0.1.0)
A library to access the Bloglines API from Ruby

BlueCloth (1.0.0)
BlueCloth is a Ruby implementation of Markdown, a text-to-HTML
conversion tool for web writers. Markdown allows you to write using
an easy-to-read, easy-to-write plain text format, then convert it to
structurally valid XHTML (or HTML).

builder (2.0.0)
Builders for MarkUp.

capistrano (1.4.0)
Capistrano is a framework and utility for executing commands in
parallel on multiple remote machines, via SSH. The primary goal is
to simplify and automate the deployment of web applications.

cmdparse (2.0.2)
Advanced command line parser supporting commands

daemons (1.0.4, 0.4.2)
A toolkit to create and control daemons in different ways

diff-lcs (1.1.2)
Provides a list of changes that represent the difference between two
sequenced collections.

facets (1.8.20, 1.4.5)
If Godzilla were a Programmer's Library

fcgi (0.8.7)
FastCGI library for Ruby.

feedtools (0.2.26)
Parsing, generation, and caching system for xml news feeds.

ferret (0.10.14)
Ruby indexing library.

gd2 (1.1.1)
Ruby interface to gd 2 library.

gen (0.41.0)
A simple code generation system

geoip (0.3.0)
GeoIP looks up a GeoIP database to provide geographical data for an
IP address or Internet hostname. The free version of the GeoIP
database available from www.maxmind.com only contains country
information. This library supports that and the GeoIPCity file (both
revisions). The data is much more reliable than using the country
codes at the end of the hosts' domain names.

gettext (1.9.0)
Ruby-GetText-Package is a libary and tools to localize messages.

glue (0.41.0)
Utility methods and classes for Nitro + Og

gnuplot (2.2)
Utility library to aid in interacting with gnuplot

hoe (1.1.7)
Hoe is a way to write Rakefiles much easier and cleaner.

htmltools (1.10)
This is a Ruby library for building trees representing HTML
structure.

localization_generator (1.0.8)
[Rails] Localization generator.

login_generator (1.2.2)
[Rails] Login generator.

madeleine (0.7.3)
Madeleine is a Ruby implementation of Object Prevalence

mysql (2.7)
MySQL/Ruby provides the same functions for Ruby programs that the
MySQL C API provides for C programs.

needle (1.3.0)
Needle is a Dependency Injection/Inversion of Control container for
Ruby. It supports both type-2 (setter) and type-3 (constructor)
injection. It takes advantage of the dynamic nature of Ruby to
provide a rich and flexible approach to injecting dependencies.

net-sftp (1.1.0)
Net::SFTP is a pure-Ruby implementation of the SFTP client protocol.

net-ssh (1.0.10)
Net::SSH is a pure-Ruby implementation of the SSH2 client protocol.

nitro (0.41.0)
Everything you need to create Web 2.0 applications with Ruby and
Javascript

og (0.41.0)
State of the art object-relational mapping system

PageTemplate (2.2.3)
A simple templating system for Web sites.

ParseTree (1.6.4)
Extract and enumerate ruby parse trees.

payment (1.0.1)
Payment is used to process credit cards and electronic cash through
merchant accounts.

podcast (0.0.4)
Create podcasts from MP3 files

rails (1.2.2)
Web-application framework with template engine, control-flow layer,
and ORM.

rake (0.7.1)
Ruby based make-like utility.

RedCloth (3.0.4)
RedCloth is a module for using Textile and Markdown in Ruby. Textile
and Markdown are text formats. A very simple text format. Another
stab at making readable text that can be converted to HTML.

ruby-breakpoint (0.5.1, 0.5.0)
ruby-breakpoint lets you inspect and modify state at run time.

ruby_odeum (0.2.1)
Ruby/Odeum is a simple full text reverse indexer that lets you index
a set of files and then search through them very quickly.

rubyforge (0.4.0)
A simplistic script which automates a limited set of rubyforge
operations

RubyInline (3.6.2)
Multi-language extension coding within ruby.

rubyzip (0.9.1)
rubyzip is a ruby module for reading and writing zip files

ruwiki (0.9.3)
A simple, extensible, and fast Wiki-clone.

salted_login_generator (1.1.1)
[Rails] Login generator with salted passwords.

shipping (1.5.0)
A general shipping module to find out the shipping prices via UPS or
FedEx.

sources (0.0.1)
This package provides download sources for remote gem installation

syntax (1.0.0)
Syntax is Ruby library for performing simple syntax highlighting.

tagtools (0.0.3)
Folksonomy system for Rails.

tidy (1.1.2)
Ruby interface to HTML Tidy Library Project

uuidtools (1.0.0)
Generation of UUIDs.

webgen (0.4.1)
webgen is a templated based static Web site generator.

wee (0.10.0)
Wee is a framework for building highly dynamic web applications.

Kayla

On our servers we have the following versions of PHP

PHP Version 4.4.2
PHP Version 5.1.6

On 2/14 we are upgrading PHP5 to 5.2 on the following servers:

SH110
SH111
SH112
SH113
SH114

After we see and resolve any common issues or bugs, we will then start upgrading PHP5 on the older servers. Another announcement will follow for that.

Kayla

Server load was nearly reaching 10 - a site had to be suspended on this machine.

It's the blurred out username below:

Kayla

The server has now been running perfectly.

Server Load 0.99 (4 cpus)
Memory Used 19.4 %

Daniel

Hello,

This morning, around 3:00AM EST, SH110 became unresponsive and required a reboot. Upon reboot, the server failed to come up cleanly indicating that some system files were either corrupted or plain missing.

We've faced such scenarios (and seemingly much worse ones) before, so we decided to go ahead and just start piecing things together. Unfortunately there was more missing than what we expected, as at this point right now, the server still refuses to boot properly. We are still going to try to fix the server, but just in case, we have setup a new SH110 server and have begun a restore from backups on it.

We will try our best to fix the old SH110 box, as there is quite a lot of data on this server, thus a restore could take anywhere between 12-32 hours for this server. Clearly, the best scenario would be to just get old SH110 machine up. As a side note - no user data was lost; it seems that only system related files got affected, so do not worry about personal data loss if we get the old machine back up. If the old machine doesn't get fixed - backups are completely up to date, and of course, the old user data is available as well if its needed at any point.

Right now, the restore is about 2-3% completed, so we've got a long way to go. I'll keep this thread updated with the latest news on the status of this machine.

We appreciate your patience and understanding, while we deal with these matters.

Thanks,

Daniel

Hello,

A quick update : we ran into a slight hitch - we are not entirely sure what caused this, but the new server started acting up about an hour ago, which forced us to relocate the drives to yet another different server. The previous server would just crash intermittently, display odd disk/adapter related errors, so we thought the best solution here would be to just entirely switch the machines again.

So far, it has worked out fine. We are restoring data from backups and slowly but surely accounts are getting populated on the server.

Unfortunately, due to the sheer size of the array, the restore as noted before is going to take a long time. At this point we are only 15-20% into the restore. Thankfully, weekend traffic slows down quite a bit, so you shouldn't feel as large of an impact from the outage than say during business hours on weekdays.

Also, we've been continuously working on the old server, but so far we haven't had much luck in getting it running into a decent enough shape.

I'll update again within the next few hours.

Thanks everyone for being understanding of all this

-Dan

david.p

Daniel updated me on the situation regarding SH110. It appears that the backup drive is faulty. Currently we are attempting to salvage off of the old/original machine, as attempts made previously were done with CentOS-5, whereas the current system on there was CentOS-4.5. We are anticipating that this change will allow for a re-install "on top of" which will solve many of the problems regarding this machine. We will update again as progress/information is made/gained.

Daniel

Hello everyone,

Following David's update - a reinstall 'on top of' existing installation won't happen, as there is something else fundamentally wrong with the system. Our current plan of attack is to hook up both, the new and old machines via a local gigabit link and transfering data from old -> new machine, which is essentially going to resemble a network based restore.

I truly apologize for all these problems. No one could've expected this, and I am really happy and thankful that everyone is being as understanding as they are about this. We really are trying our hardest to get your sites up as soon as possible and get everyone back on track.

I'll be able to give an ETA once the machines are hooked up together and the restore has been started; that way i'll have some speed statistics and will be able to give you a better time estimate.

Thanks,

Kayla

We're looking at really slow transfer speeds. The transfer is going though, and is closely monitored by all of us, which is most important as it's better that it's moving along slowly than not at all.

We are estimating at least another 12 hours of complete downtime. This is a very rare occurrence, and a very tough restore. Unlike other restores, when sites go live as they get restored, this cannot be the case here. This is due to the server having a really hard time dealing with huge file sizes and still performing all the services of the shared server.