...more recent posts
Looks like my fix yesterday did take care of the problem. I'm still keeping my eye on it, but so far so good. One big help in tracking this down was enabling the MySQL slow query log. This lets you set a threshold for number of seconds, and then any query taking longer than that gets logged to a specific file. Watching that file is then a huge help in spotting where potential problems are hiding.
Here's the official MySQL slow query log page, and here's a nice write up on how to activate it. Basically it just amounts to adding these lines to the my.cnf file and restarting MySQL:
long_query_time = 1 log-slow-queries = /var/log/mysql/mysql-slow.logI had to create the /var/log/mysql/ directory (and chown mysql:mysql) in order to get it to work.
I had a bit of a chase this morning tracking down an issue on the new server (new server = ash, old server = tulip, old old server = elm.) CPU usage was being pegged at 99.9% by mysqld. Turns out - I think - to be a problem with some PHP code I had written (who would have guessed!) Anyway, I ended up learning a bunch about MySQL that I didn't know previously, so these things should be easy to track down in the future. For my own reference I will link here to this excellent Jeremy Zawodny post on tuning MySQL.
Keywords: thread_cache : my.cnf
From the Onion: Apple Unveils New Product-Unveiling Product: iLaunch.
Damn, wasted about 7 hours on this one. I so wish I didn't have to run a mail server. Anyway, the issue was that Qmail was not allowing outgoing messages with 15 or more recipients. Up through 14 would be fine, but then at 15 it would reject all messages (including the first 14) with this error: sorry, reached maximum number of recipients for one session (#5.7.1 - chkuser). Error 5571
So obviously Chkuser is the problem. But how to change this? I took a bunch of wrong turns trying to create a /var/qmail/control/maxrcpt file, but it turns out my Qmailtoaster installation is not compiled to understand the maxrcpt control file. So no luck there.
Hours of googling later I finally found the mailing list question that put me on the right track: I needed to edit tcp.smtp (which I was told would be at /etc/tcp.smtp but of course nothing is easy and mine was instead at /etc/tcprules.d/tcp.smtp). So I changed CHKUSER_RCPTLIMIT="15" to CHKUSER_RCPTLIMIT="150" in tcp.smtp. And then ran qmailctl cdb to reload the new rules (I guess this makes the tcp.smtp.cdb file out of the tcp.smtp file that just got edited.)
And now it works. Fun fun fun.
StorageMojo's look at Bianca Schroeder of CMU's Parallel Data Lab paper Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you?
Storage is very conservative, so don't expect overnight change, but these papers will accelerate the consumerization of large-scale storage. High-end drives still have advantages, but those fictive MTBFs aren't one of them anymore.By much different methodology I've come to the same conclusion for a big set of my data: forget RAID and replicate the data.
Further, these results validate the Google File System's central redundancy concept: forget RAID, just replicate the data three times. If I'm an IT architect, the idea that I can spend less money and get higher reliability from simple cluster storage file replication should be very attractive.
Ars Nokia N800 internet tablet review. I'm really intrigued by this thing. Unfortunately I can't actually justify buying it. But if I can ever find a buyer for my uber digital music system idea I think this would be the remote control.
Often hilarious, to me at least, fake Steve Jobs blog: "Dude, I invented the friggin iPod. Have you heard of it?"
POW (Plain Old Webserver) is, amazingly, a FireFox plugin that turns your browser into a web server. It's written entirely in javascript.
Javascript is sure growing up fast.
Sweet looking Samsung Ultra Smart F700 mobile addresses all the complaints against the iPhone. Namely, it has a full qwerty keyboard, a 5 megapixel camera, HSDPA high speed data, SD memory slot, and removable battery. On the downside, this was announced in Korea (where else?) with no specific availability date (so it will hit US shores approximately never) and I assume it will be running Windows Mobile (or if not, there's little chance Samsung will get the software right.) Still, it's nice to dream. I can't wait for the iPhone, but really I can't wait for something like revision 3 of the iPhone.
Steve Jobs comes out against DRM on digital media. I'm only commenting because this was all over the news yesterday. Obviously I'm a big fan of Jobs. But this needs a little clarification. I mean, I believe him, but...
First, keep in mind that Apple is under legal attack in several European countries about the iTunes music store. So, in some ways this is probably a maneuver to shift focus off of the iTMS (and Apple's FairPlay DRM) and onto the record labels. And this seems fair, since it definitely is the record labels who forced Apple to sell music encumbered with DRM.
On the other hand, a little pinch of salt is needed here since Jobs seems to indicate that he would rather sell non-DRM'd music online (just as he points out the labels sell non-DRM'd music on CD,) but the fact is that many bands are interested in selling their music this way (and have asked Apple) yet Apple says it's not possible. Clearly it is technically possible to have both DRM and DRM free tracks for sale at the same time on iTMS.
I'm not really trying to criticize him here, it's a very complex issue, and Apple has to walk a number of very fine lines. I think they have done a very good job given the backwardsness of the record industry. Still, in a perfect world, Apple could be doing more. And these comments of Jobs shouldn't be taken as an indication that they are about to start selling mp3s without digital rights management.