Systems Administration

Hardware / Linux / New Toys! / Systems Administration — 6 Comments
27
Jul 11

Home server build, part one – specifications

So between my laptop and herself’s, we have a fair amount of valuable (to us) data – MSc essays and coursework, book manuscripts, half a gigabyte of open source projects, Phd programming work, wedding photos and video, and about 19 gigabytes of other photos and video, eight gigabytes of target shooting documents and images, half a gigabyte of academic papers… well, you get the idea. So when my laptop hard drive started to hiccup and its SMART report started complaining of bad blocks and imminent failure within 24 hours… well, it prompted some concern 🙂 Most of the important data was backed up on my server (which is not just off-site, but out of the country) using rsnapshot, but there’s nothing like an incipient disaster to make you review your disaster recovery protocols 😀

Besides, I had been planning for some time to offload the bulk data storage (video files and so forth) to a NAS, and while buying an off-the-shelf NAS box is certainly an option:

  • building your own can give you more capability for less outlay;
  • building your own allows you more functionality than just NAS storage – in this case, I had a few other tasks in mind for this box;
  • building your own is something every sysadmin should do for CPD if nothing else. 🙂

So, what’s the specification? The list of tasks is fairly straightforward to start with:

  • NAS storage
  • Central print server (and scanner at some point)
  • Backups of both laptops and off-site storage of those backups
  • Central downloading server for bittorrents and so on
  • Media server

None of these need much in the way of CPU oomph, though we would need multiple cores as the storage will be some form of software RAID array (and having multiple cores would give more performance than a faster single-core, at least for a given amount of outlay). Not a huge amount of RAM is needed either. But we do need a number of disk interfaces, gigabyte ethernet (and the central router for the network has since been upgraded from the standard ISP’s Zyxel to a gigabyte ethernet Netgear router), and if this is all on the motherboard, so much the better.

And obviously, outlay’s an issue as well. Buying an off-the-shelf NAS box (a Synology DS411+) and stocking it with disks (4x Hitachi 2Tb Deskstar drives) would cost approximately €960 (priced on scan.co.uk) so the goal was to get in below that threshold.

First off, the CPU. I’m going with the AMD Athlon II X4 645. It’s a quad core processor but quite cheap. It’s a socket AM3 processor, which leads us to the selection of the motherboard, and I’ve chosen a Gigabyte GA-880GA-UD3H, which is the cheapest Gigabyte AM3 board which had six SATA ports and Gigabyte ethernet ports. Add in a fairly cheap 4Gb of RAM (and pause to remember the time back in ’97 just after graduation when we watched with some degree of awe when the TCD sysadmins showed off a whole gigabyte of RAM, which cost about three months disposable income…) and a fairly standard Artic Cooling Freezer 7 for the CPU and that’s the guts of the thing.

Now for storage. The idea here is a degree of future-proofing, and a lot of I-never-want-to-lose-any-data-or-have-much-downtime 😀 So, the OS is going on two 320Gb Seagate Barracuda hard drives in a RAID 1 array. The main data storage will go on four 2TB Hitachi CoolSpin 5K3000 drives, which were chosen after this particularly excellent article from Backblaze about their 135Tb storage pods. And to try to keep stable power lines, an 850W Coolermaster modular PSU. And since this rig will have to last for a while and I hate slicing my hands open on cheap cases, a Lian Li PC-8NB case to mount all of this in, together with an Icy Box IB554SK SATA RAID frame because when the hard drives fail (and all drives will), I don’t want to have to disassemble the entire box to fix the problem (though, yes, if the OS drives I might have to, but you could always add another Icy Box or similar frame – I was watching the outlay myself). And then just to top it off, a DVD RW drive to make installation a bit easier.

For those who’ve been asking on Hacker News, yes, the PSU here is over-specified. That’s a deliberate choice, because I’d rather come nowhere near the limits (or even the 50% mark if possible) of the PSU for two reasons – stability of voltage lines and cooling. And yes, the CPU could be an Atom, but I’ve chosen instead to go with something a bit more conventional and old and unfashionable and debugged because this is an infrastructure box and I’m willing to pay a few euros more upfront to avoid spending a week of my time trying to fix it in a years time, or having to buy more new hardware because it turned out we needed it to do something more than it can do right now and an Atom couldn’t cut it. And yes, 320Gb hard drives. Actually, I changed my mind and went with the 500Gb ones after writing this post; they turned out to be 6 pence cheaper than the 320GB model, which I had thought was the cheapest available hard drive not made by McFlakey Inc. They’re not an example of overspecification, they’re an example of consumer electronics pricing…

So the full list:

Icy Box IB-554SKIcy Box IB-554SK£83.87
Corsair DDR3 XMS Classic4GB (2x2GB) Corsair DDR3 XMS3 Classic, PC3-10666£25.68
Artic Cooling Freezer 7 Pro Arctic Cooling Freezer 7 Pro v2£18.74
850W Coolermaster Silent Pro850W Coolermaster Silent Pro M£109.02
Lian Li PC-8NBLian Li PC-8NB£83.98
Gigabyte GA-880GA-UD3HGigabyte GA-880GA-UD3H£79.98
AMD Athlon II X4 645AMD Athlon II X4 645£77.51
Pioneer DVR-S19LBKPioneer DVR-S19LBK 24x DVD±R£17.92
Seagate Barracuda 320Gb320GB Seagate ST3320413AS Barracuda£57.31
Hitachi Coolspin 5K3000 2Tb2TB Hitachi 0F12117CoolSpin 5K3000£209.95
Total£788.62 (€~€895)

So that’s under the target price, so the order went in on Scan this weekend and is due for delivery in a day or so…

(continued in part two)

General / Linux / Python / Systems Administration — 22 Comments
20
Jul 10

My interviews at Google

Google LogoSo I’ve now completed the interview process twice with Google (once in 2007 and once in 2010), and while I’m not sure advice from someone not hired after two run-throughs is all that useful, I figured that the more information out there for those undergoing pre-Google-Interview stress, the better, so here’s how it went.

In both cases, I was contacted out of the blue by a Google recruiter. The first time I had been considering looking for a new role and pursued it immediately; the second time I hadn’t been and put off the recruitment process for several months, during which the same recruiter contacted me again twice to follow up. If nothing else, that’s a nice ego boost, but a more cynical mind might be considering the shotgun approach to a narrow recruiting filter and commissions 😀

First, a quick data point, I was applying for an SRE(SA) position on both occasions – Site Reliability Engineer (System Administration), because in most of my roles to date, I’ve been doing both sysadmin and development work and I’ve never seemed to drift towards one pigeonhole or another. SRE(SA) seemed optimal – interesting sysadmin work on large-scale systems and quite a bit of tool-writing to boot. This was decided on between myself and the recruiter, based on the self-assessment form you are given to fill out. I would love to know how they get around illusory superiority and the Dunning-Kruger effect with those forms, especially given the wierd bias they’d have in the dataset from having so many of the best in their fields working there.

Both times, the process proceeded in the same way: Continue reading →

Linux / Systems Administration — 5 Comments
18
Mar 10

Nagios notifications via clickatell

Nagios SMS AlertThe most robust solution for sending SMS notifications of server issues detected by Nagios is well-known – plug a GSM modem (or a mobile phone) into your server directly and use that as the delivery mechanism. It’s true out-of-band communications, so if the backhaul fails completely, you still get notified.

However, for someone who has a small server like mine, in a geographically remote location (like Germany, in my case), that may be overkill. Especially as the cost includes both the initial cost of the hardware and the ongoing service charges. And for true out-of-band notification, you need far more admin time than may be appropriate for a small site; you have to have scheduled test of out-of-band notifications lest the hardware fail without detection, and so forth.

In my case, the requirement is far lower-spec; I just want to get a quick ping if the server starts to die so I can log in and try to effect repairs. If it’s an actual hardware failure, or an actual backhaul failure, odds are good that I can’t fix it from my end in a few minutes anyway, and we’re past the “fix it” part of the disaster recovery plan (you do have one, right?), and into the “get another temporary webhost and redirect using DNS” part of the DR plan (you do keep regular backups of your site locally, right?).

So my solution is equally lower-spec. I got an account with clickatell.com (there are several other web SMS gateways, I chose clickatell.com because I’m most familiar with them, not because of any objective evaluation) and enabled the HTTP/S API on the account and then wrote the following additions to the Nagios /etc/nagios3/commands.cfg file (again, this is for Debian, your file location may vary…):

[cc lang=”bash” escaped=”true”]define command{
command_name host_notify_with_sms
command_line wget “http://api.clickatell.com/http/sendmsg?user=INSERT_USERNAME_HERE&password=INSERT_PASSWORD_HERE&api_id=INSERT_API_ID_HERE&to=$CONTACTPAGER$&text=’$NOTIFICATIONTYPE$ Server is $HOSTSTATE$ ($HOSTOUTPUT$) @ $LONGDATETIME$'”
}

define command{
command_name service_notify_with_sms
command_line wget “http://api.clickatell.com/http/sendmsg?user=INSERT_USERNAME_HERE&password=INSERT_PASSWORD_HERE&api_id=INSERT_API_ID_HERE&to=$CONTACTPAGER$&text=’$NOTIFICATIONTYPE$ Server : $SERVICEDESC$ is $SERVICESTATE$ @ $LONGDATETIME$'”
}[/cc]

Then tell Nagios to use these new commands in /etc/nagios3/conf.d/contacts_nagios2.conf :

[cc lang=”ini”]define contact{
…
service_notification_commands   service_notify_with_sms notify-service-by-email
host_notification_commands      host_notify_with_sms notify-host-by-email
…
pager                           INSERT_MOBILE_PHONE_NUMBER_HERE
}
[/cc]

And now Nagios warnings and alerts are dispatched both by email (in a longer format) and by SMS. All very simply done, needing only wget to be installed. It’s easily testable as well – just call the wget line from the command line, like so:

[cc lang=”bash” escaped=”true”]wget “http://api.clickatell.com/http/sendmsg?user=INSERT_USERNAME_HERE&password=INSERT_PASSWORD_HERE&api_id=INSERT_API_ID_HERE&to=INSERT_MOBILE_PHONE_NUMBER_HERE&text=’This is a test message. If this was a real message, you would be panicking right now and desperately looking for a terminal.”[/cc]

Given the pricing structure in clickatell, it’s quite affordable – at the rate of normal usage, I should get a year’s coverage for around €50 or so, with capacity to spare. So it’s easy, cheap and testable. It’s not true out-of-band, but as 80% solutions go, it’s not too shabby.

← Older Entries