Anybody felt that? (Jan05~Jan06)

[Kindom934]

Did the server just went down and came back up? At around 8:28PM HST, I couldn't connect to 3GM. At around 8:37PM HST, I was able to come back on here.

And I was doing a long reply..... :banghead


EDIT: It felt like the (again) server skiped for a minute or two. I am also feeling the server is lagging a bit as in when I sometimes hit 'reply', it takes about 1 minute then it goes. Same with moving to different parts of the forum. Been doing this today.

EDIT2: Is it me or is the server off a few minutes? Maybe it's just on my side...

http://i13.photobucket.com/albums/a255/Kingdom934/3dgameman/Time.jpg

EDIT3: A thread was marked 'new' and I didn't see that thread being mark as 'new' until just now. The 3 threads that were mark as new post were posted right around 7:00PM ~ 7:45PM. They somehow was re-mark as having a new post but they didn't.

[z33k3r]

Yes, the server is somehow 17min off... weird... ug... just another reason for me to hurry up getting us on the new server (which was turned on yesterday btw, just not with 3gm on it yet). I still got to get everything installed and tested before we flip the switch :)

[Kindom934]

Yes, the server is somehow 17min off... weird... ug... just another reason for me to hurry up getting us on the new server (which was turned on yesterday btw, just not with 3gm on it yet). I still got to get everything installed and tested before we flip the switch :)

I just had two messages of 'the post is a duplicate of another post that has been posted within 5 minutes'.

Darn lag. Not sure if it's on my side or the server....

[z33k3r]

Prety sure the clock got delayed on the server...

[robodude666]

Can you guys email midphase and ask them to resync the time? The time difference is rather annoying. Its like 18 minutes off now. Was only 16 before.

[JesterX]

Yup, I can confirm the downtime.

Error was detected at 2007-01-05 23:28:26
Operation restored at 2007-01-05 23:41:16.

Downtime total 12 min(s) 50 sec(s).

It was probably more than 13 minutes.

[Kindom934]

Yup, I can confirm the downtime.

Error was detected at 2007-01-05 23:28:26
Operation restored at 2007-01-05 23:41:16.

Downtime total 12 min(s) 50 sec(s).

It was probably more than 13 minutes.

I'm guessing you had a monitoring program to do that. If you actually kept track of 3GM without something, then man...you are more of a 3GM addict then me, lol.

[robodude666]

Yup, he has a website track 3GM and he posts the info from there. It is like 30minutes +/- correct I believe. Great service for free. Of course pay services give better quality like check once every minute.

[Kindom934]

Yup, he has a website track 3GM and he posts the info from there. It is like 30minutes +/- correct I believe. Great service for free. Of course pay services give better quality like check once every minute.

Dang, I want something like that!!

I gotta be connected with my beloved 3GM forums!! :heartbeat

[JesterX]

Its not my web site, its a free service offered by host-tracker. Heres the URL to keep track of the stats:
http://host-tracker.com/site-monitoring-statistics/225858/

The site has gotten consistently worse, like the downtime amount that I get in the email when it goes down and comes back up is not reflected on that page. I don't know how that page gets its numbers, but at least you know for sure if it was down.

[JesterX]

Can you guys email midphase and ask them to resync the time? The time difference is rather annoying. Its like 18 minutes off now. Was only 16 before.

I thought this was a VPS, if so, Rodney should have SSH access and he can change that time from there.

[efernandez_98]

I thought this was a VPS, if so, Rodney should have SSH access and he can change that time from there.
I think this is a shared server, so he might not have root access.

[JesterX]

I think this is a shared server, so he might not have root access.

If you lookup the hostname for the IP of 3dgameman.com, its vps.3dgameman.com

[efernandez_98]

So, its a virtual private server. He still might not have root access.

[JesterX]

So, its a virtual private server. He still might not have root access.

I agree thats a possibility, but if you look on the midPhase site, it states that all VPS's have "Full ROOT Access"

http://midphase.com/html_files/hosting_plan_vps.php

[efernandez_98]

I agree thats a possibility, but if you look on the midPhase site, it states that all VPS's have "Full ROOT Access"

http://midphase.com/html_files/hosting_plan_vps.php


Yeah, but it also says, "24/7 VPS Monitoring" and " 99.9% Server Uptime Guarantee".

[robodude666]

Yeah, but it also says, "24/7 VPS Monitoring" and " 99.9% Server Uptime Guarantee".

And its also free month if you don't have 99.9% uptime... Not free 1/2 month.

[Kindom934]

And its also free month if you don't have 99.9% uptime... Not free 1/2 month.

So basically, all what the 'things' it suppose to promise is basically...err...not it.

[JesterX]

I'm sure they have "monitoring" for the VPS, since the Virtuzzo VPS software automatically does that, however, monitoring won't work when the VPS Host server is down.

[Kindom934]

Seems like the forums/main site went down again for about 10 minutes (12:14pm~12:23pm HST).

[JesterX]

Again, I can confirm that, hopefully wont have to in the near future...


Error was detected at 2007-01-09 15:11:22
Operation restored at 2007-01-09 15:21:12.

Downtime total 9 min(s) 50 sec(s).

[efernandez_98]

Wow, jester's tracking site site even has a cute little uptime graphic for 3dgameman.com

<a href='http://host-tracker.com/uptime-statistics/225858/'><img alt='web site monitoring' width=88 height=31 border=0 src="http://ext.host-tracker.com/uptime-img/?s=31&amp;t=225858&amp;m=00.09&amp;p=Total"></a><noscript><a href='http://host-tracker.com/'>monitoring web site downtime</a></noscript>

Like they say in Japan, "Kawaii!!!"

[JesterX]

efer, its NOT my tracking site!
Its a free service.

I've got one for my server as well:
<a href='http://host-tracker.com/web-site-uptime-monitor/106555/'><img alt='website server monitoring' width=88 height=31 border=0 src="http://ext.host-tracker.com/uptime-img/?s=31&amp;t=106555&amp;m=00.09&amp;p=Total" ></a>

Its not cute, but its clear.

[efernandez_98]

efer, its NOT my tracking site!
Its a free service.

I've got one for my server as well:
<a href='http://host-tracker.com/web-site-uptime-monitor/106555/'><img alt='website server monitoring' width=88 height=31 border=0 src="http://ext.host-tracker.com/uptime-img/?s=31&amp;t=106555&amp;m=00.09&amp;p=Total" ></a>

Its not cute, but its clear.
Oh yeah, I know. I guess that I misrepresnted that. Sorry, my bad. I only touted it as "your's" as its the site you always quote when confirming these 3gm outages.

[robodude666]

Know whats funny? 2007 started about 9 days ago... And uptime is already under 99.9% :lmao

[Kindom934]

Know whats funny? 2007 started about 9 days ago... And uptime is already under 99.9% :lmao

We'll be lucky if we can pass this month with 80% uptime considering the amount of downtime we've been having so far.

[JesterX]

We'll be lucky if we can pass this month with 80% uptime considering the amount of downtime we've been having so far.

Well, 80% uptime is a lot of downtime. Thats 148.8 hours of downtime per month. The fact is, it is very easy to get 98% uptime or 99%, since thats 14.88 hours and 7.44 hours per month, respectively. midPhase is supposedly a large hosting company and they should have adequate resources to restore a server within 3 hours.

[JesterX]

Apparently there was more downtime today

Error was detected at 2007-01-10 06:36:41
Operation restored at 2007-01-10 06:49:09.

Downtime total 12 min(s) 28 sec(s).

[efernandez_98]

So, I gotta wonder how much of these outages are due to the server being down, and how many are due to router issues.

[Kindom934]

More down time...hmm...

I wonder if there's a pattern to this.















EDIT

The forums got VERY SLOW around 10:13PM-10:18PM HST. It took me about 4~5 minutes to get from the General Hardware to this thread and post. I hit other sites and they were normal speed.

[efernandez_98]

Maybe the server is having a smoke break. :smokin

[z33k3r]

I was thinking the samething about the router...but Midphase is actually just a colo in a large data center and I doubt that the datacenter is having the problem (they host big companies like Cingular, Sprint, Level3 etc...) I am thinking that the server or maybe, possibly, their DNS servers... but what ever, we are actively tring to get parts for our dedicated server, so a bit more time and we will be up and ready to go without MediocrePhase...

[efernandez_98]

Well, I was thinking more along the lines of any of the routers between the user and server. Not limiting their datacenter. Sometimes, I see new posts that occured a "server outage" and sometimes I don't see the outages at all.

I'd recommend writing a script that runs out of cron to check traceroutes to different parts of the world. If it isn't an actual server shutdown that's causing the outages then maybe you can figure out where along the path you're having an issue.

[JesterX]

I was thinking the samething about the router...but Midphase is actually just a colo in a large data center and I doubt that the datacenter is having the problem (they host big companies like Cingular, Sprint, Level3 etc...) I am thinking that the server or maybe, possibly, their DNS servers... but what ever, we are actively tring to get parts for our dedicated server, so a bit more time and we will be up and ready to go without MediocrePhase...

I highly doubt they host Cingular. MidPhase is colo'ed at scnet.net (Server Central).

traceroute to 3dgameman.com (216.246.58.251), 30 hops max, 38 byte packets
1 d1.46.5d45.static.theplanet.com (69.93.70.209) 0.694 ms 0.596 ms 0.634 ms
2 gi3-9.asr01.dllstx2.theplanet.com (69.41.250.81) 0.504 ms 0.487 ms 0.506 ms
3 vl2.dsr02.dllstx2.theplanet.com (12.96.160.42) 0.634 ms 0.480 ms 0.510 ms
4 vl22.dsr02.dllstx3.theplanet.com (70.85.127.74) 0.912 ms 0.882 ms 0.898 ms
5 et3-2.ibr03.dllstx3.theplanet.com (70.87.253.17) 0.901 ms 1.000 ms 0.766 ms
6 dcr2-ge-4-0-0.dallas.savvis.net (208.172.139.129) 1.163 ms 1.141 ms 1.156 ms
7 dcr2-so-7-0-0.Chicago.savvis.net (204.70.192.97) 26.575 ms 27.211 ms 26.631 ms
8 dcr1-so-5-0-0.chicago.savvis.net (204.70.192.45) 26.551 ms bpr1-so-0-0-0.ChicagoEquinix.savvis.net (208.175.10.238) 26.671 ms dcr1-so-5-0-0.chicago.savvis.net (204.70.192.45) 34.604 ms
9 ber2-ge-8-7.chicagoequinix.savvis.net (208.174.224.33) 27.076 ms ber2-ge-4-1.chicagoequinix.savvis.net (208.175.9.133) 26.801 ms bpr2-so-0-0-0.ChicagoEquinix.savvis.net (208.175.10.98) 26.683 ms
10 208.173.176.214 (208.173.176.214) 27.446 ms ber2-ge-4-2.chicagoequinix.savvis.net (204.70.194.253) 26.707 ms 26.791 ms
11 po1-4G.ar1.ord1.us.scnet.net (64.202.110.62) 27.228 ms 208.173.176.214 (208.173.176.214) 27.449 ms po1-4G.ar1.ord1.us.scnet.net (64.202.110.62) 29.030 ms
12 unknown.ord.scnet.net (216.246.92.94) 27.322 ms po1-4G.ar1.ord1.us.scnet.net (64.202.110.62) 27.598 ms unknown.ord.scnet.net (216.246.92.94) 27.357 ms
13 unknown.ord.scnet.net (216.246.92.94) 27.375 ms 27.342 ms *
14 * unknown.midphase.com (216.246.22.244) 30.732 ms 29.858 ms
15 unknown.midphase.com (216.246.22.244) 28.415 ms vps.3dgameman.com (216.246.58.251) 29.108 ms unknown.midphase.com (216.246.22.244) 27.582 ms


They have a list of their large customers on their site:
http://www.scnet.net/customers

I got like 10 emails notifying me of downtime of 3GM today. It was very spotty, some 2 minute downtimes, a 27 minute one, etc.

[Kindom934]

I got like 10 emails notifying me of downtime of 3GM today. It was very spotty, some 2 minute downtimes, a 27 minute one, etc.

Geez!!

10 already?!




No wonder I somehow manage to get a double post when I hit reply ONE TIME. Also, the server suppose to not allow a double post (i.e if text is the same in two back to back post) for a few minutes as it will give you a message about it.



EDIT : As I was posting this, Server went down...


There seems to have been a problem with the database.
Please try again by clicking the Refresh button in your web browser.

An E-Mail has been dispatched to our Technical Staff, whom you can also contact if the problem persists.

We apologise for any inconvenience.

[JesterX]

I believe thats the vBulletin can't connect to database error.
If its sending out emails, Rodney probably has quite a few ;)

[z33k3r]

Sorry, I meant Verizon... you are just looking at the wrong list...


http://www.equinix.com/customers/list/


Featured Customer
This is a partial list of IBX customers that have selected Equinix for their core Internet infrastructure requirements. Through the company's IBX centers, customers can directly interconnect with the providers that serve over 90% of the world's Internet networks and users for their critical peering, transit and traffic exchange requirements. This list will be updated periodically. Click on highlighted customers listed below to view their testimonials.
Networks:

AAPT
AboveNet
Adelphia
Ameritech
Asia Global Crossing
AT&T
Big Pipe (Shaw Fiber)
BT
China Telecom
Chunghwa Telecom (TWGate)
Cogent Communications
Comcast
Cox Communications
DACOM America
Deutsche Telekom
Earthlink


France Telecom
Genuity
Global Crossing
IIJ America
Internap
Hurricane Electric
Japan Telecom
KDDI America
Level 3
Looking Glass Networks
MCI
MSN
Pacific Bell
OnFiber
Optus
Qwest


Reach Services
SAVVIS
SBC
SingTel
Sprint
ST Teleport
TeliaSonera
Telstra
Telus
Tiscali
UUNET
Verio
Verizon
Wiltel
XO Communications
Yipes
Global Enterprises, Service Providers, and Content Companies:
About.com
Adobe
Accenture
Akamai
Aleron
America Online
Carpathia Hosting
DoubleClick
Electronic Arts
General Electric



Google
Hines
Hotwire
IBM
Kontiki
Kyocera
Loral Cyberstar
Micron Technology
Nortel / Kidz Online
PayPal


Radio Free Virgin
Royal & Sun Alliance
Server Central
Sony Online Entertainment
Washingtonpost Newsweek Interactive
WeatherBug®
WV Fiber, LLC
Yahoo!

[JesterX]

Thats referring to Equinix in general, which operates many datacenters throughout the country. Its like Savvis and Level3, they all own many DCs throughout the US and internationally.


Our datacenter facilities are provided by Equinix (www.equinix.com) and SARA Computing (www.sara.nl), and represent the finest space available in each market worldwide. These datacenters have been designed and operated to satisfy even the most discriminating customers.


If you do a traceroute, you don't see Yousendit or Feedburner on scnet. It appears that FeedBurner is on Internap Colo.

Also, that customers list might just represent those who use Equinix's facilities, possibly for peering or what not.


Equinix, Inc. (Nasdaq: EQIX), the leading provider of network-neutral data centers and Internet exchange services, today announced additional details for the company’s Internet Business Exchange™ (IBX®) center expansions in the Chicago, Washington, D.C. and New York markets. The first phase of build outs for these three expansions, previously announced by Equinix, will increase the sellable cabinet capacity by approximately 20%, increasing the company’s global footprint to 37,000 sellable cabinets.

[z33k3r]

True, I was just saying that I was doubting the downtime to routers... It actually is looking like the slowdowns of the site (not necessarily the downtime) is being cause by spiders..! I think we got that part taken care of...:thumb

[JesterX]

True, I was just saying that I was doubting the downtime to routers... It actually is looking like the slowdowns of the site (not necessarily the downtime) is being cause by spiders..! I think we got that part taken care of...:thumb

Interesting, but yeah, they say they have Juniper routers, which are the best for border routers. Did you decide to block google or rate limit them?

[z33k3r]

We just slowed them down...lol.

[JesterX]

lol...nothing to extreme. :)

[Kindom934]

Spiders? You mean Google bots?

[JesterX]

Spiders? You mean Google bots?

Yeah, not these:

http://www.homestead.com/ipmofalaska/files/spidersteatoda.jpg

[z33k3r]

lol, no, it was the Google bot that seems to have been killing us... they had used over a 1.5gigs of bandwidth already this month just in indexing! I wonder if they are doing some kinda beginning of the year index audit...:What the

[Tivon]

6GB's in November... Man, Google is a HOG! I checked the Google logs last night and it said "Last scanned January 10th" And the problem was on the 11th. So it might not have been Google but rather something else pretending to be just to get a scan of everything. There needs to be away to prove Googlebots are what they say they are. If I was running a search engine or ads company I'd pretend to be googlebot because most people want google searches and let them in to scan.

[muteforce]

haha, I thought you might have been talking about the earthquake I felt in the Bay Area this morning...

[Kindom934]

lol, no, it was the Google bot that seems to have been killing us... they had used over a 1.5gigs of bandwidth already this month just in indexing! I wonder if they are doing some kinda beginning of the year index audit...:What the

We need RAID to kill those bots, lol...

[robodude666]

We need RAID to kill those bots, lol...

:lmao Isn't RAID for roachs?

[Kindom934]

:lmao Isn't RAID for roachs?

True but if you get a few cans of it, you could most likey short circuit the bots...thats if you don't die first from the vapors.

[JesterX]

This still works better

http://www.seniorcenterinc.org/programs/Computer%20Hack.gif

[efernandez_98]

Yep, another service outage.


bash-2.03$ traceroute 3dgameman.com
traceroute: Warning: Multiple interfaces found; using hme0
traceroute to 3dgameman.com (216.246.58.251), 30 hops max, 40 byte packets
(1...5) skipped
6 ae-2-52.bbr2.Denver1.Level3.net (4.68.107.33) 3.484 ms 3.168 ms 3.211 ms
7 ae-0-0.bbr2.Chicago1.Level3.net (64.159.1.34) 27.643 ms as-2-0.bbr1.Chicago1.Level3.net (64.159.4.225) 29.829 ms ae-0-0.bbr2.Chicago1.Level3.net (64.159.1.34) 28.478 ms
8 ge-6-0.ipcolo2.Chicago1.Level3.net (4.68.101.9) 28.777 ms ge-11-0.ipcolo2.Chicago1.Level3.net (4.68.101.41) 27.659 ms ge-6-1.ipcolo2.Chicago1.Level3.net (4.68.101.73) 26.328 ms
9 ge3-0-1-300.j2.ord.scnet.net (4.79.66.2) 26.928 ms 26.779 ms 27.492 ms
10 ae1-4G.ar2.ord1.us.scnet.net (64.202.110.66) 27.336 ms 26.811 ms 27.156 ms
11 tge9-2-10G.ar1.ord1.us.scnet.net (216.246.50.161) 26.676 ms 26.688 ms 26.184 ms
12 unknown.ord.scnet.net (216.246.92.94) 26.975 ms 27.161 ms 26.365 ms
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *


and before you question the next hop, lets try the server that's 1 bit away on the last octect. (250 instead of 251)


bash-2.03$ traceroute 216.246.58.250
traceroute: Warning: Multiple interfaces found; using hme0
traceroute to 216.246.58.250 (216.246.58.250), 30 hops max, 40 byte packets
(1...5) skipped
6 ae-2-54.bbr2.Denver1.Level3.net (4.68.107.97) 4.360 ms 3.385 ms 3.301 ms
7 ae-0-0.bbr2.Chicago1.Level3.net (64.159.1.34) 26.831 ms as-2-0.bbr1.Chicago1.Level3.net (64.159.4.225) 26.514 ms ae-0-0.bbr2.Chicago1.Level3.net (64.159.1.34) 27.369 ms
8 ge-6-2.ipcolo2.Chicago1.Level3.net (4.68.101.137) 28.322 ms ge-11-0.ipcolo2.Chicago1.Level3.net (4.68.101.41) 26.774 ms ge-6-2.ipcolo2.Chicago1.Level3.net (4.68.101.137) 26.167 ms
9 ge3-0-1-300.j2.ord.scnet.net (4.79.66.2) 26.325 ms 26.352 ms 26.677 ms
10 ae1-4G.ar2.ord1.us.scnet.net (64.202.110.66) 27.702 ms 25.908 ms 27.046 ms
11 tge9-2-10G.ar1.ord1.us.scnet.net (216.246.50.161) 28.567 ms 28.918 ms 29.661 ms
12 unknown.ord.scnet.net (216.246.92.94) 29.233 ms 28.518 ms 27.640 ms
13 unknown.ord.scnet.net (205.234.228.37) 38.861 ms 38.413 ms *
14 unknown.midphase.com (216.246.58.250) 38.767 ms 38.890 ms 38.622 ms
bash-2.03$


a few hours later...


bash-2.03$ traceroute 3dgameman.com
traceroute: Warning: Multiple interfaces found; using hme0
traceroute to 3dgameman.com (216.246.58.251), 30 hops max, 40 byte packets
(1...5) skipped
6 ae-2-52.bbr2.Denver1.Level3.net (4.68.107.33) 4.258 ms 3.358 ms 4.771 ms
7 as-2-0.bbr1.Chicago1.Level3.net (64.159.4.225) 27.523 ms ae-0-0.bbr2.Chicago1.Level3.net (64.159.1.34) 28.703 ms as-2-0.bbr1.Chicago1.Level3.net (64.159.4.225) 86.451 ms
8 ge-11-2.ipcolo2.Chicago1.Level3.net (4.68.101.169) 26.491 ms ge-6-2.ipcolo2.Chicago1.Level3.net (4.68.101.137) 30.072 ms ge-11-2.ipcolo2.Chicago1.Level3.net (4.68.101.169) 27.319 ms
9 ge3-0-1-300.j2.ord.scnet.net (4.79.66.2) 26.920 ms 26.504 ms 27.357 ms
10 ae1-4G.ar2.ord1.us.scnet.net (64.202.110.66) 58.285 ms 29.478 ms 27.060 ms
11 tge9-2-10G.ar1.ord1.us.scnet.net (216.246.50.161) 29.279 ms 28.159 ms 27.371 ms
12 unknown.ord.scnet.net (216.246.92.94) 27.375 ms 27.126 ms 27.556 ms
13 * unknown.ord.scnet.net (205.234.228.37) 40.756 ms 39.097 ms
14 unknown.midphase.com (216.246.22.244) 39.295 ms 39.510 ms 38.657 ms
15 vps.3dgameman.com (216.246.58.251) 39.021 ms 39.526 ms 41.106 ms
bash-2.03$


and magically the server is up and running. :luxhello

[efernandez_98]

Oh, and if you were wonding about the hop before you... I also pinged that one while your service was unavailable. I grabbed the list off of midphase's noc and pinged everything.


bash-2.03$ for i in `awk '{print $2}' midphase`
> do ping -c3 ${i}
> done
zakserver2.midphase.com is alive
zakserver.midphase.com is alive
win05.midphase.com is alive
win04.midphase.com is alive
win03.midphase.com is alive
win02.midphase.com is alive
win01.midphase.com is alive
vz9.midphase.com is alive
vz8.midphase.com is alive
vz7.midphase.com is alive
vz7.midphase.com is alive
vz6.midphase.com is alive
vz5.midphase.com is alive
vz4.midphase.com is alive
vz3.midphase.com is alive
ping: unknown host vz22
ping: unknown host vz21.midphase.com
ping: unknown host vz20.midphase.com
vz2.midphase.com is alive
vz19.midphase.com is alive
vz18.midphase.com is alive
vz17.midphase.com is alive
vz16.midphase.com is alive
vz15.midphase.com is alive
vz14.midphase.com is alive
vz13.midphase.com is alive
vz12.midphase.com is alive
vz11.midphase.com is alive
vz10.midphase.com is alive
ping: unknown host vs2.driveragent.com
tpm10.midphase.com is alive
server116.midphase.com is alive
server114.midphase.com is alive
server110.midphase.com is alive
server109.midphase.com is alive
server108.midphase.com is alive
lrm102.midphase.com is alive
legend.midphase.com is alive
jsp102.midphase.com is alive
jsp101.midphase.com is alive
esc96.midphase.com is alive
esc95.midphase.com is alive
esc94.midphase.com is alive
esc93.midphase.com is alive
esc92.midphase.com is alive
esc91.midphase.com is alive
esc90.midphase.com is alive
esc89.midphase.com is alive
esc88.midphase.com is alive
esc87.midphase.com is alive
esc86.midphase.com is alive
esc85.midphase.com is alive
esc84.midphase.com is alive
esc83.midphase.com is alive
esc82.midphase.com is alive
esc81.midphase.com is alive
esc80.midphase.com is alive
esc79.midphase.com is alive
esc78.midphase.com is alive
esc77.midphase.com is alive
esc76.midphase.com is alive
esc75.midphase.com is alive
esc74.midphase.com is alive
esc73.midphase.com is alive
esc72.midphase.com is alive
esc71.midphase.com is alive
esc70.midphase.com is alive
esc69.midphase.com is alive
esc68.midphase.com is alive
esc67.midphase.com is alive
esc66.midphase.com is alive
esc65.midphase.com is alive
no answer from esc63.midphase.com
esc62.midphase.com is alive
esc61.midphase.com is alive
esc60.midphase.com is alive
esc59.midphase.com is alive
esc58.midphase.com is alive
esc57.midphase.com is alive
esc56.midphase.com is alive
esc55.midphase.com is alive
esc54.midphase.com is alive
esc52.midphase.com is alive
esc51.midphase.com is alive
esc50.midphase.com is alive
esc49.midphase.com is alive
esc48.midphase.com is alive
esc46.midphase.com is alive
esc45.midphase.com is alive
esc43.midphase.com is alive
esc42.midphase.com is alive
esc41.midphase.com is alive
esc40.midphase.com is alive
esc39.midphase.com is alive
esc38.midphase.com is alive
esc37.midphase.com is alive
esc36.midphase.com is alive
esc35.midphase.com is alive
ping: unknown host esc34.midphase.com
esc33.midphase.com is alive
esc32.midphase.com is alive
esc32.midphase.com is alive
esc31.midphase.com is alive
esc29.midphase.com is alive
esc28.midphase.com is alive
esc27.midphase.com is alive
esc25.midphase.com is alive
esc24.midphase.com is alive
esc23.midphase.com is alive
esc22.midphase.com is alive
esc21.midphase.com is alive
esc20.midphase.com is alive
esc19.midphase.com is alive
esc18.midphase.com is alive
esc17.midphase.com is alive
esc16.midphase.com is alive
esc15.midphase.com is alive
esc14.midphase.com is alive
esc13.midphase.com is alive
esc11.midphase.com is alive
esc04.midphase.com is alive
esc03.midphase.com is alive
esc02.midphase.com is alive
esc01.midphase.com is alive
no answer from backup2.midphase.com
ans07.midphase.com is alive
no answer from ans06.anhosting.com
ans05.anhosting.com is alive
ans04.midphase.com is alive
ans03.midphase.com is alive
ans02.anhosting.com is alive
ans01.anhosting.com is alive


So, in retrospect....


bash-2.03$ traceroute 3dgameman.com
traceroute: Warning: Multiple interfaces found; using hme0
traceroute to 3dgameman.com (216.246.58.251), 30 hops max, 40 byte packets
(1...5) skipped
6 ae-2-52.bbr2.Denver1.Level3.net (4.68.107.33) 4.258 ms 3.358 ms 4.771 ms
7 as-2-0.bbr1.Chicago1.Level3.net (64.159.4.225) 27.523 ms ae-0-0.bbr2.Chicago1.Level3.net (64.159.1.34) 28.703 ms as-2-0.bbr1.Chicago1.Level3.net (64.159.4.225) 86.451 ms
8 ge-11-2.ipcolo2.Chicago1.Level3.net (4.68.101.169) 26.491 ms ge-6-2.ipcolo2.Chicago1.Level3.net (4.68.101.137) 30.072 ms ge-11-2.ipcolo2.Chicago1.Level3.net (4.68.101.169) 27.319 ms
9 ge3-0-1-300.j2.ord.scnet.net (4.79.66.2) 26.920 ms 26.504 ms 27.357 ms
10 ae1-4G.ar2.ord1.us.scnet.net (64.202.110.66) 58.285 ms 29.478 ms 27.060 ms
11 tge9-2-10G.ar1.ord1.us.scnet.net (216.246.50.161) 29.279 ms 28.159 ms 27.371 ms
12 unknown.ord.scnet.net (216.246.92.94) 27.375 ms 27.126 ms 27.556 ms
13 * unknown.ord.scnet.net (205.234.228.37) 40.756 ms 39.097 ms
14 unknown.midphase.com (216.246.22.244) 39.295 ms 39.510 ms 38.657 ms
15 vps.3dgameman.com (216.246.58.251) 39.021 ms 39.526 ms 41.106 ms
bash-2.03$ getent hosts vz17.midphase.com
216.246.58.18 vz17.midphase.com
bash-2.03$ getent hosts vz12.midphase.com
216.246.22.244 vz12.midphase.com
bash-2.03$


For some reason from your forum post, I thought you were behind vz17 but I guess it is vz12.

[JesterX]

Operation restored at 2007-01-18 03:59:34.
Error was detected at 2007-01-17 22:57:32

Downtime total 5 hour(s) 2 min(s) 2 sec(s).

5 hours! midPhase needs to learn how to run servers at this rate.

[Race234]

lol ya but soon hopefully we wont have to worry about this once Rodney gets a dedicated server going.

[Kindom934]

5 hours! midPhase needs to learn how to run servers at this rate.

No wonder I tried. So how much does that bring our uptime now? 90%?

I thought like "Man, out of all the time, why does it happen to go kapoof right when..."

Yea, I fell asleep. I am recovering from a 24-hour flu still. I didn't miss school (as in physically missed a day) so I'm good :).

On the side note, You know how I said how I would continue to go on 3GM? Well the body achs were so severe, I couldn't even type since my fingers were freakin..... :mad

[JesterX]

Well, heres the bad part of Host-Tracker, it takes them forever to update uptime stats if your site goes down.

The percentage will be updated in like 24-48hours, but its probably around 96-98%.
Remember that 90% uptime is a LOT of downtime! Do the math. 98% uptime is super easy to achieve.

[Tivon]

5 hours! midPhase needs to learn how to run servers at this rate.

They sent an e-mail to Rodney about needing to take the site down for an upgrade. They said it would only be 20 minutes. No time to give anyone a notice. I then sent them an e-mail and to get a ticket. They then said 2 hours = 5 hours to midPhase. :negative

[robodude666]

Looks like Rodney might have another 1/2 month credit soon :spin

[JesterX]

It seems like they removed the credit clause from their Terms of Service page. Though its still on the Why Us? > Our Guarantees page
* If midPhase fails to provide 99.9% uptime, you will receive a credit of one month. For full guarantee details please see our Terms of Service agreement.

I think 3GM met the req a few days into January.

[robodude666]

Yup, once you see a company removing guarantees and using their "edit TOS without notification" rights, you know that they know they screwed up ;)

[Kindom934]

Yup, once you see a company removing guarantees and using their "edit TOS without notification" rights, you know that they know they screwed up ;)

But then I bet they will make some other excuse.

[Tivon]

Their satisfaction guaranteed...

[efernandez_98]

If Midphase was doing upgrades for any period of time, then the least they could do was redirect the traffic to a "down for maintenance" splash page. Really, it doesn't take that much effort. Even lowly clerks will put up a "back at x" sign when they leave the shop unattended for breaks.

[JesterX]

So I did some adding, 10h:44m of downtime in 2007
Thats about 97.5% uptime. There is a margin of error, so I place it between 97% and 98%.

[Tivon]

That is just one of many reasons why we are getting our own server on our own Gig backbone. Just going to take time putting the server up and time getting the software installed and time getting the data ported over. Most of this is all lands on Zeek and he has a day job to deal with first.

[Kindom934]

If Midphase was doing upgrades for any period of time, then the least they could do was redirect the traffic to a "down for maintenance" splash page. Really, it doesn't take that much effort. Even lowly clerks will put up a "back at x" sign when they leave the shop unattended for breaks.

Yea, thats what I was wondering. It's very 'unprofessional' to be taking down a site(s) for maintance without putting a 'down for maintenance" thing. Makes like it was unexpected or something.

[efernandez_98]

That is just one of many reasons why we are getting our own server on our own Gig backbone. Just going to take time putting the server up and time getting the software installed and time getting the data ported over. Most of this is all lands on Zeek and he has a day job to deal with first.

If you guys were serious about high availablity, you would consider putting two servers up.

[Tivon]

If you guys were serious about high availablity, you would consider putting two servers up.

C'mon now... Rodney needs to eat. :Wink

[Kindom934]

C'mon now... Rodney needs to eat. :Wink

You might want to add in sleep. It takes a lot of money to maintain two servers...or build them in the first place.

[efernandez_98]

You might want to add in sleep. It takes a lot of money to maintain two servers...or build them in the first place.
Not if you actually know what you're doing.

[JesterX]

If you guys were serious about high availablity, you would consider putting two servers up.

You don't really need 2 servers for high availability. For 3GM, anything higher than 99.5% uptime is just fine IMO. Thats easily achievable with 1 server and good admin'ing.

The only reason its hard to get 100% uptime on one server is your bound to take it offline one time or another for kernel upgrades. Thats been the case for me, I'm trying to go for 365 days up without reboot on the server in my sig, but its already been rebooted twice for kernel upgrades. Apache (web server) and php (scripting engine) upgrades will only cause a 10 second service interruption. On two servers that I manage for 2 clients, they both have more than 200 days uptime, but thats because they don't need the latest anything.

[JesterX]

Got two more monitoring error emails in the morning,

8:17am PST, went down for 5 min(s) 59 sec(s).

9:51am PST, went down for 10 min(s) 41 sec(s).

[robodude666]

If you guys were serious about high availablity, you would consider putting two servers up.

If you have a good reliable datacenter you don't need two servers. Having a hot site or warm site is always good practice, but its not really needed. I ran my server a few servers at a few good datacenters and I never had a problem with downtime (on their part. it was all downtime from updating the domain's DNS).

Where are you guys planning on holding the server? colocating? or "basement datacenter" ?

[efernandez_98]

I guess there's two people in this world that don't believe in hardware failures. Oh well, at least someone still believes in the Easter Bunny and Santa Claus.

[robodude666]

Majority of the time, hardware doesn't just "fail" on its own. If its crappy quality, then yes it will but the servers I were using were good. I never experienced it, so im not bothered by it.

[JesterX]

I guess there's two people in this world that don't believe in hardware failures. Oh well, at least someone still believes in the Easter Bunny and Santa Claus.

Well, its not we don't believe in hardware failures, its that theres costs associated with having a second server.

[JesterX]

Whats up with the site right now, its very slow...

[Tivon]

I've noticed that as well... cleared up after sending Rodney an IM. Might be spiders again...

[Kindom934]

I've noticed that as well... cleared up after sending Rodney an IM. Might be spiders again...


RAID would be nice...or a shoe.

[JesterX]

RAID would be nice...or a shoe.

http://www.hotshotbrand.com/NR/rdonlyres/BC789442-3382-40C5-866A-643E6D9FAF82/0/4490.jpg

[Tivon]

http://www.webmasterworld.com/forum11/

http://www.webmasterworld.com/forum93/page1.htm

[Kindom934]

http://www.hotshotbrand.com/NR/rdonlyres/BC789442-3382-40C5-866A-643E6D9FAF82/0/4490.jpg


You know, where's a bunch of people I know of that could use the super size package of that.


On the side note, Thank you JesterX for posting up the downtime of 3GM :)

[JesterX]

Just to add to the counter:

Operation restored at 2007-02-12 17:12:05.
Url was down as a resultat of:
Http error:Http_client.No_reply
Error was detected at 2007-02-12 16:57:26
Downtime total 14 min(s) 39 sec(s).

[Kindom934]

Ahh...

Thanks for keeping me updated with all the downtimes of 3GM. I really appreciate it.

[Tivon]

I think Rodney said we had 150 people viewing the site today. Perhaps Myspace people are checking in to see what we have? :)

[JesterX]

I think Rodney said we had 150 people viewing the site today. Perhaps Myspace people are checking in to see what we have? :)

Interesting, there was actually a dip in the Alexa traffic stats for today.

http://img407.imageshack.us/img407/8332/graphf555555uwww3dgamemip2.png

http://www.alexa.com/data/details/traffic_details?q=&url=www.3dgameman.com

[Tivon]

Site Stats for 3dgameman.com:

* Traffic Rank for 3dgameman.com: 70,378 (up22,903)

* Speed: Very Slow (85% of sites are faster), Avg Load Time: 5.0 Seconds What's This?

* Other sites that link to this site: 495

* Online Since: 09-Nov-1998

[JesterX]

* Speed: Very Slow (85% of sites are faster), Avg Load Time: 5.0 Seconds What's This?

Unfortunately, thats kind of true. The Joomla CMS implementation isn't exactly fast and tach on all the media on the page and its pretty hefty.

From SiteReportCard.com
Connection Speed Download Time (sec)
14.4K 406.27
28.8K 203.14
56K 104.47
128K ISDN 45.71
1.44MB T1 4.23

HTML: Connect 0.0781 secs / Transfer 2.2968 secs / Total 2.3749 secs / Rate of 361.17Kbps...
It takes 2.3749 seconds just to grab the html from the server, aside from the media on the page.

From Silktide:
Your website responded in 3.50 seconds, and your homepage downloaded in 4.21 seconds. This is quite respectable and suggests your website is running on a fairly powerful web server.

I have a site running XOOPS CMS thats been tweaked out (page gen time is around 0.08secs) and Silktide's sitecore says:
Your website responded in 0.38 seconds, and your homepage downloaded in 0.60 seconds. This is very fast and suggests your website is running on a sufficiently powerful web server.


That 5.0 second figure isn't far off, once you factor in rendering time, TCP connect time for media on the page, etc.

5.0 seconds to load isn't bad, if you have broadband, if your still on dial-up, oh jeez.

[JesterX]

I was just looking through the Watchdog logs on my server and I took a look at the Resource Usage graph

http://img403.imageshack.us/img403/9932/statsbx1.jpg

On well over 4 occasions, I've had DOS attacks on the server. God damn little script kiddies jacking up the requests to Apache and causing it to somewhat overload the server.
Most of the request overloads are unsuccessful (thankfully) due to mod_evasive, which detects DOS attacks on Apache. I've also seen plenty of SSH brute force attempts before I changed the SSH port.
Fortunately, the server has not experienced any real damaging DOS attacks. The internet is a wilderness out there.

[efernandez_98]

God damn little script kiddies jacking up the requests to Apache and causing it to somewhat overload the server.

Sorry, I'll try to be a little less obtrusive next time.

[JesterX]

Operation restored at 2007-02-15 07:43:57.
Url was down as a resultat of:
Http error:Http_client.No_reply
Error was detected at 2007-02-15 07:38:34
Downtime total 5 min(s) 24 sec(s).
Check failures total: 4.

1234567890

[JesterX]

Tacking on to the wall


Operation restored at 2007-02-21 20:23:30.
Url was down as a resultat of:
Http error:Http_client.No_reply
Error was detected at 2007-02-21 18:08:16
Downtime total 2 hour(s) 15 min(s) 14 sec(s).

[Tivon]

Sad is it not?

[JesterX]

Well, hosts are lazy, use easy passwords, don't secure machines, let little kids with hosting accounts run fork bombs
Like: . () { . | . & } ; .

Or just write a shell script to execute itself to take up all the system resources and have it run on startup to screw the machine up.

Its all good, because we all know MidPhase is run by 5 year olds.

:Beer Cheers

[Kindom934]

It was down again? As long as it isn't down when I have time with my beloved 3GM forums...

[JesterX]

hmm..i just got a DB error from the forums

There seems to have been a problem with the database.
Please try again by clicking the Refresh button in your web browser.

An E-Mail has been dispatched to our Technical Staff, whom you can also contact if the problem persists.

We apologise for any inconvenience.

[Tivon]

Taking forever to load.

[Kindom934]

So how much is 3GM uptime after that almost 2 day downtime? :spin

[JesterX]

I reset the stats, since it was down forever.
It was 97% before this, so 96%.

Still at MidPhase eh?

[Tivon]

How much do you want a bet that the DOS attack that made our server slow showed up in some graphs and someone else not related to the problem only noticed the numbers and said we are using too much and the stopped us. So in truth it was their problem that made our side use more than it really did only to elevate us into a higher bracket than we should be. Sounds like a nice CON to me.. It's just an idea though, no real proof this was the problem or if we really became that popular over night. At any case it was a problem with MidPhase, they should have seen this coming a long time ago and did a better forecast or given Rodney a discount on the setup fees.

[JesterX]

You have access logs, you can always check. I have a simple shell command that I used to use.

cat ACCESSFILENAME | awk '{ print $1; }' | sort | uniq -c | sort -rn | less

Puts the IP and the number of requests, sorted highest requests on top. Next best thing is just to install mod_evasive, which definitely works.
http://www.zdziarski.com/projects/mod_evasive/

[JesterX]

Speaking of DOS attacks, my server came under some sort of attack last night. Took out apache for 5-6 minutes (from responding), mail server got killed quite a few times, but luckily watchdog brought them back up.
I've narrowed it down to that its not the web server that was attacked and is likely some sort of exploit on the mail server (probably imap).

[JesterX]

MidPhase Wall of [INSERT WORD HERE]

Operation restored at 2007-02-26 16:11:09.
Url was down as a resultat of:
Http error:Http_client.No_reply
Error was detected at 2007-02-26 15:17:25
Downtime total 53 min(s) 43 sec(s).

[Tivon]

MidPhase Wall of [INSERT WORD HERE]

There might be some downtime tonight.

[JesterX]

There might be some downtime tonight.

Switching to dedicated server? *excitement building*

[3dGameMan]

Site is getting hit hard again now and the VPS can hardly handle it. Site seems kinda slow to me and the VPS stats show it's maxed. So hopefully they can get someone to get their act together and move my site over to a dedicated later tonight. I was told it would be done last night or this morning, but obviously this didn't happen. They told me "something came up".

As Tivon mentioned, there could be downtime later tonight. If there is, please post about it when the site comes back.

[JesterX]

By the way, the downtime may be because the website is taking longer than 40 seconds to respond. I used a web site monitoring application to see what response times the main site was getting and its ranges 4 to 60 seconds, with the majority of the time between like 5-25 secs.

[robodude666]

Yea, it takes me on average 10 seconds to load a page. Sometimes 30seconds or so if it has some pictures. The main site takes a bit longer like 35seconds.. but god, I feel like im on dial up is ISDN :s

[JesterX]

The software actually does an average, didnt see it til now, it says 14.8 seconds for the server to respond. By server, I actually mean main web site index page.

AVG. response(ms): 14873.29
Min response(ms): 4375
Max response(ms): 90000
Success: 77
Failure: 3
Success rate: 96.2500%

Those 3 failures are probably because it took longer than the alloted 90secs (or 90000 ms) to respond.

[Tivon]

Unless they work 24/7 then the switch over is a no go.. Wait, how would we know anyway?

[JesterX]

Use this:
http://www.dnsstuff.com/tools/lookup.ch?name=www.3dgameman.com&type=A

If IP != 216.246.58.251, 3GM site = moved to diff place

[Tivon]

Hmmm... SingleHop SCNET-MIDPHASE-216-246-58-0 ??