Forum Downtime

Alan_Hicks

We'd like to apologize for the forum downtime we've experienced today. This afternoon our circuit went down shutting us off from the Internet. In order to restore functionality, key pieces of network infrastructure had to be restarted. Somehow during the restart our forums systems experienced some corruption and we've been forced to restore from this morning's backup.

jad1097

Circuit went down? No UPS?

At least there are backups now

GA Fin

Did we lose Popeye during the downtime? :crossed

INTREPID377

Back up? What a GREAT idea. Wish you were around 5 years ago. ;-)

Andrewsmith1

GA Fin wrote: »

Did we lose Popeye during the downtime? :crossed

I managed to add him to my ignore list while the forum was actually working. That is better than nothing.

AC Man

Thanks Alan, I appreciate the update, chit happens. Actually, it did not cause significant disruption as far as I'm concerned.

kodiakzach

Suggestion: dual Internet providers, multi-homed running BGP in a Datacenter with dual entry, diverse path = no Internet outage. Another suggestion, ditch the "throw away" bandwidth provider [COGENT] - it's the cheapest and least reliable of business ISP's. Get more reliable providers.

Polar

Thank you Alice

GA Fin

Polar wrote: »

Thank you Alice

:rotflmao

Mackeral Snatcher

Must refrain.

Joey Buttons

With as much activity this forum has seen in the past month, I think 5 people saw a disruption of service.

[email protected] Special

So let me get this right, FS forum is run by the same firm doing the Obamacare web site?

So yesterday didn't exist?

CaptTater

[email protected] Special wrote: »

So let me get this right, FS forum is run by the same firm doing the Obamacare web site?

So yesterday didn't exist?

please reenter your data.

SuperJox

I just registered to the forums I think either monday or tuesday night. I got approved the next day and made 3 posts. Those 3 posts disappeared, my account got terminated I think since I had to reregister with the same username and all. Also it seems like posts in the past day seemed to have been deleted. Is it because of the downtime? I don't think I broke any rulez

SouthFloridaAngler

I did not notice any down time. Thank you.

BnT

Good to see you back and not welcoming people to leave and not let the door hit em on the way out.

johnD

SuperJox wrote: »

I just registered to the forums I think either monday or tuesday night. I got approved the next day and made 3 posts. Those 3 posts disappeared, my account got terminated I think since I had to reregister with the same username and all. Also it seems like posts in the past day seemed to have been deleted. Is it because of the downtime? I don't think I broke any rulez

you should receive your answer in a timely manner...






























just wait for it.

Darcy

:rotflmao

johnD wrote: »

you should receive your answer in a timely manner...






























just wait for it.

Griz

SuperJox wrote: »

I just registered to the forums I think either monday or tuesday night. I got approved the next day and made 3 posts. Those 3 posts disappeared, my account got terminated I think since I had to reregister with the same username and all. Also it seems like posts in the past day seemed to have been deleted. Is it because of the downtime? I don't think I broke any rulez

All activity including sign up approvals was lost yesterday. That is what happened.

CaptTater

You should make this a sticky.

tunaman

Joey Buttons wrote: »

With as much activity this forum has seen in the past month, I think 5 people saw a disruption of service.

Are you sure 5??? Maybe 6!!

Alan_Hicks

kodiakzach wrote: »

Suggestion: dual Internet providers, multi-homed running BGP in a Datacenter with dual entry, diverse path = no Internet outage. Another suggestion, ditch the "throw away" bandwidth provider [COGENT] - it's the cheapest and least reliable of business ISP's. Get more reliable providers.

We have precisely that. Perhaps the biggest problem we're running into right now is a limit in bandwidth. Right now our circuit is limited to 100Mb by our firewall interface. We're working to convert to a 1Gb circuit but it's going to require additional changes and some additional downtime along the way. You may have noticed an outage today around noon. Our colocation provider dropped the ball and failed us over to our back-up circuit before it was ready. We are not running this forum or our sites on a cheap business internet connection from one of our offices. Everything is housed in the second largest data center in the world, but still problems do occasionally occur, especially when you're forced to change things like increasing bandwidth ten fold.

Seriously, looking at our traffic graphs, we are maxing out a 100Mb circuit for at least 16 hours a day.

awat

CaptTater wrote: »

You should make this a sticky.

Tater likes sticky!:kick

gandrfab

Alan_Hicks wrote: »

Seriously, looking at our traffic graphs, we are maxing out a 100Mb circuit for at least 16 hours a day.

Wonder what it was like when the regional sub forums were more active than OT is now?

Popeye

Hi!:wink

Popeye

gandrfab wrote: »

Wonder what it was like when the regional sub forums were more active than OT is now?

It had to be even more cool.

Mister-Jr

Alan_Hicks wrote: »

We have precisely that. Perhaps the biggest problem we're running into right now is a limit in bandwidth. Right now our circuit is limited to 100Mb by our firewall interface. We're working to convert to a 1Gb circuit but it's going to require additional changes and some additional downtime along the way. You may have noticed an outage today around noon. Our colocation provider dropped the ball and failed us over to our back-up circuit before it was ready. We are not running this forum or our sites on a cheap business internet connection from one of our offices. Everything is housed in the second largest data center in the world, but still problems do occasionally occur, especially when you're forced to change things like increasing bandwidth ten fold.

Seriously, looking at our traffic graphs, we are maxing out a 100Mb circuit for at least 16 hours a day.

tater could fix that in 3 minutes.

Popeye

Tates more focused on the other forum.

I miss him.

dstockwell

Alan_Hicks wrote: »

You may have noticed an outage today around noon..

Crap, wheres the back-up, ready the back-up..

GA Fin

Alan_Hicks wrote: »

Seriously, looking at our traffic graphs, we are maxing out a 100Mb circuit for at least 16 hours a day.

If Rich would stop posting GIFs you would only be hitting about 10Mb at peak times.

Please demote him from Regional Admin to Local Admin.

CaptTater

I would suggest you run a layer 7 monitoring tool to find out why you are saturating a 100mbs pipe. If its that **** cabellas full page nonsense behind the screen fix it. I'll bet you a steak dinner it's not user content.
Maybe I'm wrong... But if you aren't (clearly) doing off site replication then it should be mostly user data.

As far as that crashing... How does a saturated pipe create data loss and corruption? A storage snapshots would have the system up and running in minutes. Most of my customers are running with a 15 minutes RPO and a couple of hours RTO for local failures.