Battle of the Evil Server Goblin… There can Only be ONE!
I refuse to concede defeat against the evil server goblin that has been stalking me around cyberspace, using my server as a cudgel aimed directly at my un-offending (keep the guffaws to a minimum) skull. That green little bastard is GOING DOWN!
What green server goblin you ask? THIS ONE:
What’s that you say? It’s merely a Photoshopped (Photopainted actually) photo of my daughter’s latest McD’s Happy Meal toy? NONSENSE! What you are looking at is a 100% pure mean green server piss off machine, and it’s out to get me.
All I have to say is thank God for my brother-in-law Elvio (owner of coolcommands.com and probably one of the smartest, no-nonsense grep-heads you could ever meet), and for the quick turn around times from the P2L coding talents of Nick and Jamie.
Here’s the latest scoop on what’s been happening… I left Server Seed awhile back due to another disappearing act and I have been through a rather spectacular list of service providers looking for someone to simply give me what Server Seed had in terms of performance, but with some degree to available support when things go wrong. Well up until recently, it seems like I just can’t get those two together in a single package unless I want to bend over and let Media Temple be my daddy. (MT has great shared hosting prices, but their true dedicated hardware pricing is nothing short of laughable). My latest setup is a box I purchases directly with Softlayer and I bought some management from a third-party provider.
Since then, I am getting great service, but loads on the box are VERY high compared to what I’m used to and performance, in my opinion, sucks. So I’ve been chasing this new management company about it and I’m basically getting nowhere. In a recent request, I asked that SSL be disabled along with imap and some other ports/services I didn’t need and were open for nothing. I won’t get too deep into the response, but I was told that Apache would have to be recompiled in order to shut off SSL. Ummm… no.
So, I asked Elvio, master of all things Unix, to snoop around on my box and check out the settings and see what’s what… I think it was the whole Apache recompile thing to shut off SSL that had him seriously worried about what my server looked like from the inside. Well needless to say, it was a dog’s breakfast, so my tutelage in Unix command prompt began. He basically had to rebuild the config, recompiled Apache and upgraded it to version 2 (yeah, Linux 1.3 on a multicore, multiprocessor box… that wasn’t a great idea) and upgrade PHP, plus clean out a bunch of other crap. That was finally done tonight actually.
Next up was to basically study the server’s performance and resource usage… and to accomplish this, he installed an AWESOME package called Cacti. This is a freeware based network graphing solution with a sweet user interface that even I can figure out in under 5 seconds. Check it out at www.cacti.net
By studying cacti’s reports and staring at the regular top processes and MySQL top processes, we started to identify coding related areas that could be improved to run more efficiently. Once again Elvio made some suggestions based on his analysis on the logs and Nick and Jamie went right to work kicking CPU ass! In fact, load was shaved right down by improving the caching system, combining includes into single files and tonight, Nick re-worked the Boolean search query as per Elvio’s search method and has cut down search times BIG TIME! For those of you with fast connection’s, you may remember the average search time for a tutorial was about 5 seconds or longer you were searching during a peak time. Well I’ve been search this evening with the box on an average load at results were coming back pretty much instantly!
So, things are definitely improving in terms of server performance and we hope people are enjoying the site as we look to speed things up and get the server back to being reliable and stable. That damn green goblin is GOING DOWN!
In summary, here’s what I’ve learned lately:
– Be wary of server management companies… they seem to go to pieces when things go beyond simply restart Apache.
– You don’t have to recompile Apache to disable SSL (Actually I knew this… I just can’t believe that’s what my management company tried to tell me)
– Disabling imap doesn’t affect POP (See previous note about management company)
– You don’t have to run DNS Server services to run your email properly (See previous note again)
– Top and MyTop are your friends
– Nagios is for monitoring, Cacti is for reporting (both rock)
– Multithreading is a feature added to Linux in version 2+
– Natural Boolean results already return results sorted with correct scoring… if you get query results back with 0 scores, your query is screwed. You should not have to re-sort them for correct score values. (Hey, I’m not a programmer, how the hell would I know?!)
– Elvio, Nick and Jamie are EXTREMELY patient… to the point of impossibility even.
That’s it for now guys! 1AM and I’m tired. Battling green server goblins is tough work… gnite!
Dan