I'm having a similar issue.
The strange thing is a lot of other sites on the server load at lightning speeds.
I think it might be a plugin I have installed because some other wordpress sites still load fast (on the same server like
Lost Whisper vs
FoxDemon although one has more graphics than the other...).
...vB is freaking me out with all the auto formating things...