Archives

General / Linux / Mobile / New Toys! — 1 Comment
17
Mar 10

Ben Nanonote with WiFi

Ben Nanonote

One of the reasons I love my Nokia e71 so much is that it’s a pretty decent example of convergence. Like the iPhone and others, it rolls so many features into one box that we’ve stopped calling these things mobile phones and started calling them mobile devices, almost without noticing. Heck, the ‘in-crowd’ just talks about ‘mobile’ as though the OED had recategorised that word from adjective to noun. It’s not so much linguistic arrogance as it is necessity – you have to go to science fiction or back to mythology to find examples of the kind of multifunction tool these devices have become and are still becoming.

The iPhone is without a doubt the poster boy for this, as its marketing is, ironically enough, pretty much founded on using it for things other than as an actual phone (and that’s why the iPad, daftly named as it is, will probably be a great success but not as great as its more dimunitive cousin. The whole attraction of the iPhone’s ability to be more than a phone is based on the fact that you are already carrying it around with you). One quick download and your phone becomes a translation device, a 2-D barcode reader, or any one of a few hundred other devices.

My problem is that I don’t really like the iPhone. It’s very slick and very pretty but… no background applications and a hefty price tag and to use it as intended, I pretty much have to have a mac. Sure, you can fake around that need, but it’s a chore. The Nokia e71 is wonderful in hardware (if you overlook the very poor camera which is hard pressed to handle the basic business task of recording the contents of a whiteboard after a brainstorming session — unforgivable given that mid-range phones handled this task better five years ago) but it’s awkward to setup with calendars and contacts and apps, even going through Ovi (which is why I’m still using a paper diary).

Once you decide against the iPhone and Nokia (and Blackberry because support for it in Ireland is again, all tied to one supplier and it’s not the best supported device here even though it’s huge in Asia), you’re pretty much left with the outliers right now, meaning Android. Yes, Android is an outlier. It gets great press without a doubt, but if you’re not a technology or gadget geek, it’s just another phone that’s a bit dingy-looking with its off-white case that doesn’t sit flat in a jacket pocket. Most people don’t know it’s a software platform, not a phone — and most of them wouldn’t understand what you meant if you told them (and amongst the real experts, btw, there are a few who don’t think much of it at all). And if you don’t mean Android, you’re right out there into the fringe at the moment. Which means stuff gets very interesting and individual indeed, which is where things like the 本 (běn) NanoNote come in:

Ben Nanonote from Qi Hardware

The Ben Nanonote looks like it might be a very interesting part of the fringe indeed. It’s small, but has a physical keyboard (humans like haptic interfaces for a good reason) and is completely open (both in hardware and software). Granted, it’s no speed demon – the iPhone ARM chips have a bit more oomph than it does – but even so, it could run a reasonably wide array of applications. It’s a long way from perfect, since it has no camera, no inbuilt wifi or inbuilt 3G or inbuilt WiMAX; but it’s intended as a first model and for a first model it’s got some promise.

Not least of which is that it costs around €70 at the moment. Add in the €60 you have to pay to get a supported microSD wifi card, and you’re still looking at less than a third of the cost of most netbooks over here. It’s a hobbyist platform rather than a serious do-work-on-this box at the moment, but looking at the upcoming Ya and Mu Nanonote platforms and seeing how building in wifi and other hardware is so possible, you have to ask the question of how long it’ll be until a commercial interest starts capitalising on the work Qi’s done here, and creates a larger market than just the hobbyist fringe. There’s a principle in open source software that the fastest way to change how something is done is to do it differently and release the code. Personally, I hope that trend holds true in hardware and we see a new market of palmtops acting as miniature netbooks; I would love to get a platform the size of the Nanonote, just with a few more networking options (as in, all of them – WiFi, WiMAX, 3G, LTE, the works). A true mobile device.

And yes, I still want an N900. If nothing else, it’d make a good stopgap measure 😀 In the meantime… well, €130 isn’t too much to drop to play with a toy like this, right?

General / MySQL / PHP / Systems Administration — 21 Comments
10
Mar 10

Performance tuning a server in less than three minutes while being slashdotted

So you wrote a blog post about something that seemed fairly innocuous, but for whatever reason, it caught the attention of one of the major sites and now your server load is at 110 and climbing, the ssh command line session is taking thirty seconds to respond to anything at all, and given that your post is on the front page of slashdot at primetime, this doesn’t look like it’s a temporary blip. What do you do?

Burning computer

Okay, first things first. You don’t have time to do a proper fine-tuning session. You need a quick & dirty tuneup here. Proper fine tuning you leave till after the traffic spike, and you can then come back at it with a plan and decent tools like siege and so on – but this is the “fix it in three minutes” version. So if you see stuff here that looks crude to you, despair not, everything looks crude when you’re trying to do it in that kind of timeframe.

First, you have to be able to actually use the server command line. The load’s at 110 – that’s not coming down anytime soon and until it does you can’t do the rest of the work you need to since it takes so much time for even a shell to respond. The load is being caused by Apache and MySQL taking on more work than the server can handle, causing the server to swap excessively; and you’ve got to dump that work or shut off the incoming new work to recover. You can try sudo killall -9 apache2 if you can get an open ssh terminal in to do the job (and it’s the first thing you should try), but the odds are that that server has to be reset. Whether that means a phone call to the data centre, or a walk down the hall to the server room, or just clicking a button in a web interface, that’s the first thing to do. Don’t hold off, because unless everyone stops reading your page right now, that load’s not coming down.

Once the box has rebooted (and I mean immediately – sit there watching ping until it comes back), ssh in and shut down apache. MySQL is okay for now, but the work is coming from Apache, so that has to get shut down until everything’s ready to go again. You’re going to lose a few minutes of service, yes, but that’s recoverable from for a blog (and if this is for something more serious than a blog, you’re going to be in trouble for not properly spec’ing the hardware to begin with anyway, so limit the damage now and take your licks later).

At this point — whether you’ve just logged in or whether you managed to run killall successfully —  if it’s a debian server you should run sudo /etc/init.d/apache2 stop (did I mention I love debian for having scripts like that? No man-paging apachectl, just an easy to remember standard interface for every service on the box. Wonderful). It’ll tidy up from killall, or it’ll shut down apache cleanly, depending on how you got here.

I’m going to use my server as the example here, by the way, since it’s what got burned last night and it’s what prompted me to write this refresher — it’s been two or three years since I last had to maintain a server that was near its capacity, the experience was a bit of a flashback 🙂 So, some background on my server – I moved my blog from wordpress.com to here, on a Hetzner server. It’s their entry-level dedicated server offering 2Gb of RAM, a 64-bit Athlon, 160Gb of hard drive in a hardware RAID-1 array and a 1Gbit NIC) — all running Debian Lenny (and no, I’ve no relationship with Hetzner, they were just the cheapest of the places various friends recommended). WordPress is up to date on my server (2.9.2 at the time of writing), as is the Lenny install — if you don’t have the latest security fixes and such in place, or your WordPress is outdated, then that’s probably adding to your problem, but for a quick fix like this, that’s too big a job. Get through the traffic spike and deal with it later.

And yes, that server spec is overkill for my needs really – but I had a bunch of side projects like RangeClerk (don’t bother, not much is up yet) and the blog for Herself Indoors and her book and some other things I wanted to run as well that would be using wierd php and python modules and libraries and the like; and I just hate cpanel and not being able to install anything I wanted. Plus, it was cheap 😀

Right, back to it.

The first thing we need to do is to sort out MySQL’s configuration. Open up the my.cnf file, whereever you’ve put it (it’ll be /etc/mysql/my.cnf for a stock Debian install). We need to tweak just a few settings. First off, key_buffer. This is probably the most critical variable here because by default all the tables will be MyISAM tables (if they’re not, then this isn’t so critical). It’s set to about 16Mb by default; we’re going to turn that up quite a bit. On a dedicated database box, this would be set very high – anything up to 50% of the total available memory. In this case, with a full stack on the one box, we set it a bit lower since Apache’s going to want a lot of RAM too – 256Mb will do for a starting value.

Next we’re going to disable the InnoDB engine completely to cut down on MySQL’s footprint. Again, WordPress by default isn’t using it. Just ensure skip-innodb is either uncommented or inserted into my.cnf.

Lastly, we’re going to enable the query cache. The thing is, MySQL’s query cache is a fairly blunt instrument. If a query is precisely the same the second time it comes in, it’ll hit the cache – but any change at all, no matter how small, and it misses the cache. So it’s not as enormously useful as you’d first imagine. However, it does help, so we’ll increase its size modestly (48Mb of RAM is sufficient here). So our changes to my.cnf look like so:

[cc lang=”ini”]
key_buffer = 256M
query_cache_limit = 16M
query_cache_size = 48M
skip-innodb[/cc]

Once those changes are made, sudo /etc/init.d/mysql restart will get the MySQL server up and running with the new setup.  Once that’s done, let’s look to the next level in the stack – Apache. Under debian the config files are arranged differently than normal; the configuration changes we’ll make will be in /etc/apache2/apache2.conf but in other installations they would be in httpd.conf or elsewhere.

The default Apache install uses the prefork MPM setup – one thread per process. It’s older, slower, less efficient, but less buggy than the worker MPM which isn’t threadsafe. So find the prefork MPM config settings in apache2.conf. They should look like this in a default install:

[cc escaped=”true” lang=”apache”]

StartServers          5
MinSpareServers       5
MaxSpareServers      10
MaxClients          150
MaxRequestsPerChild   0
[/cc]

We’re going to cut down a lot on how much work Apache takes on at once here. Yes, some users will have to wait a few seconds to see your page – but right now, with the load at 110 and climbing, they could wait until their browser timed out and they’d never see anything. So we reduce slightly the number of servers that Apache will farm off to handle requests at any one time from 5 to 4; we’ll increase the number of spare servers it’ll keep around to hand off requests to (we want to reduce the overhead of starting and stopping those processes) from 10 to 12. We’ll set an upper limit on how many we can have though, and we’ll keep it to just under 100. This works on my system, which is an entry-level system; you might get away with more, but for now use these settings and it’ll get you up and running and you can increase a bit and check again as you go (and this guide really isn’t aimed at big sites anyway, just small ones like mine which were caught on the hop). We’re also going to ensure no apache process takes on too much at once by creating a limit of how many requests any process can take on – we’ll keep it low for now (3), but it can be increased later. So our changed config settings now look like this:

[cc escaped=”true” lang=”apache”]
StartServers          4
MinSpareServers       4
MaxSpareServers       12
ServerLimit           96
MaxClients            96
MaxRequestsPerChild   3

[/cc]

Okay. At this stage, you have two options. The first is to start Apache up again and get back to work. Odds are, this will hold up pretty well – but you want to keep a window open with htop running in the background to keep an eye on things (and mainly you’re watching the swap space usage and the load. The former’s critical, the latter indicative that a problem’s arising – if either go sideways, kill apache and edit apache2.conf setting even lower values for ServerLimit, MaxClients and MaxRequestsPerChild before restarting apache). If that’s your preferred option, skip to the end of this post.

However, if you want to take that extra step, we could install memcached quickly here. It’s a very effective load reducer and under debian, it’s far easier than you’d expect:

[cc lang=”bash”]sudo aptitude install build-essential php5-devel php-pear memcached[/cc]

And let that haul in whatever other libraries it needs, then:

[cc lang=”bash”]pecl install memcached[/cc]

And once that’s done, edit the php.ini file (in Debian, that’ll be /etc/php5/apache2/php.ini ) and insert this (anywhere in the file will do, but the extensions section is the tidiest):

[cc lang=”ini”]extension=memcache.so[/cc]

That should be memcached installed and running in a default configuration (we can finetune later). We now need to drop in the backend that WordPress uses to take advantage of memcached. Download object-cache.php and copy it into the wp-content directory of your website and change the permissions and ownership of the file:

[cc lang=”bash”]cd [insert your www/wp-content directory here]
sudo wget http://plugins.trac.wordpress.org/export/215933/memcached/trunk/object-cache.php
sudo chown www-data.www-data object-cache.php
sudo chmod 644 object-cache.php[/cc]

And that’s it done. Quick, dirty, and everything at default, but that’s a three-minute setup for you (well, maybe five if you do the memcached setup as well, and I am assuming you have a fast net connection for the aptitude step, but still).

Now, restart apache and everything should fire up with memcached caching a lot of requests and keeping the server load to a managable level.

[cc lang=”bash”]sudo /etc/init.d/apache2 force-reload
sudo /etc/init.d/apache2 restart[/cc]

And once that traffic spike is past… take the time to tune it properly!

Newer Entries →