FFXIAH Maintenance - 2012-08-20

I provisioned new servers for FFXIAH and made the switch a couple of hours ago. Apparently, something was mis-configured and the server was unable to handle the load from all the visitors. I tried to fix it but was only able to narrow it down. I switched back to the old servers. Sorry for the downtime! AH sales will be delayed.

that was scary over here. I thought the goddamned IT guy blocked the site or something. (he was in here screwing around while I was out on jobsites) I almost cried.

So i take it u are going to use the old servers till u find the problem on the new ones then u are going to try to change over again to the new ones right.

edit

i normally do not like the spam pics but that one is funny.

downforeveryoneorjustme.com

This is what I was wondering.
I'd like to see some cool stuff.

oh snap, that's cool.

tyvm

isup.me (short version of down for everyone)

Correct. I will conduct load testing in the background so I don't interfere with normal site operation.


They have updated hardware and updated OS. The idea is to speed up the site and give me access to better services at no additional cost. The current servers are over 2 years old, and you know how fast computer hardware changes.

funny that is what i was using the time the server was down.So i checked to see that it was not just me.

So with the new servers, will the morning "freeze" still occur?

lol all my computer hardware is probably over 2 years old...

*** please, all my ***is like 5+ years old, I ***you not :|

I can't find it in newegg, but I know I purchased most of my components around uh ... 2007-2008?

Newest thing in my PC right now is my GPU lol, everything is pretty much junk now.

lol the only thing that would be considered new is my laptop, and some other accessories...

all my desktop hardware is at least as old as yours, most of my things are still pata, and agp class :P

/needsnewshit


but it's different for site operation, as loads get heavier etc, It's not cheap.


Scragg, iirc you have a hosting service, no?

any issues with funding for this upgrade?

Well it's different when you are leasing hardware and they are offering the newer stuff at cheaper rates. Besides, I started leasing the servers 2+ yrs ago and I just go the throwback stuff, not the latest and greatest. I digress.


I use Softlayer. It's not costing extra besides the overlap during the transition and the work involved.


That is caused by a DB backup process. The server move is step 1 in solving that. I am likely going to create a db slave to handle backups.

How's your experience with them been? We use them at work and it's been nothing but problems, over and over and over.

It's been very good. I was a customer with theplanet ever since I started FFXIAH around 2005. Softlayer and theplanet merged. I noticed SoftLayer had higher advertised prices but they are willing to work with you and negotiate. I manage the servers myself so I never really contact their support unless something is really wrong. I usually never have to restart my servers... webserver has been up 714 days.

What kind of services do they provide for your work? Dedicated? Do you manage your own servers? Which datacenter? What problems are you having?

Scragg, the new servers you're renting do you know what RAID setup they're running? I find RAID5+ hot spare provides a lot better database performance than RAID6. I'd assume the site is mostly database heavy so that's why I'm asking.

Anyways hope you figure it out and get it up and running good. Love the site so thanks for all your hard work!

It's RAID1. I'm about 90% sure I figured out the issue. It was a misconfiguration with PHP-FPM. I think it will give it a another try tomorrow while junior is at daycare. :) I will look into the RAID5 stuff though, it might be expensive.

We use virtual machine images and a VPN. Just lots of bad luck with both, bad support when things go wrong, and their VPN is awful to get into in the first place. Most machines are in Dallas, not sure about the rest - I'm not the real sysadmin, if I'm in the console things are pretty dire ;)

I'd bug our real sysad to list the problems but even talking about SL makes him agitated, so I avoid it.

While RAID1 is mostly for redundancy it will slightly increase read times (as which ever of the 2 hard drives reaches the data quickest will send it)

RIAD5 greatly improves write speeds by splitting up the data block between multiple drives, often multiplying (minus some overhead) the read AND write speeds. You can do RAID5 with as little as 3 drives, I usually recommend a 4th drive though configured as a hot spare. So it will cost a tad more but I would think would be worth doing. If you do end up going the RAID5 route be sure to configure them in the RAID controller with 64K block size, as this is what mySQL and MS-SQL are usually happiest with.

Edit: Looked up the controller card looks like a fairly decent one, also I notice it says battery which is a big plus! You can make sure Write back is enabled on the controller, this will let it use the 256MB of dram on the controller as cache. This might already be enabled.

If you knew most of that already sorry for ranting! Just trying to be helpful ^^

FFXIAH Donations!

If we can do it for a bike for tigerwoods, we can do it for FFXIAH!

Their VPS prices seem alittle on the high side. As for the VPN server, I never used one that much, and never with softlayer. In general, VPN and "problems" do end up in the same sentence often it seems.

Scragg, I remember looking you up about two years ago when i worked at theplanet sales :) you guys just signed a new server, so i figured i wouldn't bug you until a year when we had new toys. I eventually left, so never got to do that.

As for raid 5, cost of an additional hd + a raid controller (which it looks like you already have). Raid 5 helps with your database reads, but not your writes. Setto, are you thinking of raid 10? 5 still needs to write to the stripe parity and the block, but can read from both. And hotspare is dependent on how mission critical ffxiah is. Chances of losing 2 hdds before sl can replace is meh, and if we talk about 3hdd raid 5 + 1 hotspare, why not just go raid 10? If you're taking care of your backup in house, improved raid or 10k-15k rpm SAS drives will help out. You might already have sas, as 300gb isn't a standard SATA size. If its database sluggishness, hdds are the next upgrade, followed by memory.

I remember an old customer kept seeing his CPU spike on his db backend and thought he needed to keep upgrading the CPU. Ended up with a nehalem six core and a sizable bill before i convinced him that the cpu was spiking because he had old SATA drives that just couldn't keep up.

Kehlor, i left before the merge, but cloud computing was the new buzz as i was on my way out. And i hated every bit of it. I had CLEARLY defined dedicated server customers that wanted to try and fit the intense db on a shared server. Which is a ridiculous nightmare. Especially when you let a company that is fairly hardware-centric (most dedi server providers) manage a now very software intensive process of managing resources and providing small scale it support.

I'm taking a server class this semester, but I've been tinkering with setting up SCCM deployment, just to fiddle, and just in reading I'll require a domain, and a sql server, apparently I have more reading to do, and several hours of frustration, hope I have the time really...

RAID5 boosts write performance over RAID1 too. . This is an example with 4 HDDs but as you can see A1 gets written to drive0, A2 gets writen to drive1, A3 gets written to drive2, and Ap (for parity) gets writen to drive 3. Than on the next round the parity bits get written to a different drive.

While this example is a 4 HDD setup it's the same with a 3 HDD setup (A1,A2,Ap, B1,Bp,B2). Theoretically doubling write performance over a RAID1 (subtracting things like HDD seek time and the time it takes the RAID controller to generate and write the parity bits)

You're right in saying RAID10 would be a good fit as well though, it requires minimum 4 drives and it would give better performance than a 3HDD RAID5 + hot spare as there is less overhead. The main benefit of 3HDD RAID5 + hot spare is any 2 drives can die and you'll still be up, while RAID10 if 2 drives die you could be down.

I understand it may sound a little overkill but I like up time. Also in a RAID5 soon as 1 drive dies performance is degraded, the hot spare allows it to rebuild, get up to full speed while you get that warranty replacement drive in there.

Edit: Also searching up the part number on those HDDs already in there, you'll find they're 15k rpm SAS drives, which is good.

I don't see anything wrong with a RAID 1, especially if you want to be cost effective with 2 drives. The only disconcerting thing is that the two drives are the same model, and possibly the same factory batch. A production-linked failure could occur in both drives simultaneously, or at least there's a higher probability than if the drives were completely different. With different drives, the failure is more likely to occur spaced out, which makes the problem more manageable.

All said, I don't consider ffxiah to be some sort of critical system, so if the cost/time difference were great, there's really no reason to go that route.

They are SAS 15k rpm drives. Maybe you know someone where that would sneak some RAM in the servers. I'm in DAL06 if I remember correctly, they don't seem to provision new servers there, any idea why? How is it working there? You can PM me if you need.


Raid1 has been fine and the DB had a hard drive fail this year and it was promptly replaced. Buying more drives for raid5 / 10 start to get pricey. The drives I have are $100/month *each*. At some point I could just get another server and scale that way.