Epic battle with qmail today. I believe I am victorious.
tell more
Pretty boring stuff. Qmail is the mail server I run. It receives mail sent to any address for any domain hosted on my server, and it is what sends outgoing mail to external domains from any of those same accounts on my server. To me it's one of the most complex programs I run. Email transport is very arcane.
But Qmail is great and mostly it just runs. Until something goes wrong which is what happened at the end of last week. It didn't crash, but suddenly everything got real slow, and the server was working much harder than usual. Some people weren't able to send outgoing mail, and some were receiving multiple copies of incoming mail.
So I knew something was wrong. But I had no idea what it was. So in these cases you just try not to panic. Every program like Qmail running on the server has a directory full of logs where it records it's activity. When something goes wrong you go to the logs and start looking through. But there is a ton of information! Qmail is getting hit by incoming spam at a crazy rate, and every transaction hits the logs. And I don't even know what I'm looking for.
But you just keep looking. And when you find something that doesn't seem right you copy and paste that line from the logs into google and see what other people say. This often leads you on a bunch of wild goose chases following the advise of people who had problems but not the same one as yours. Could be DNS is not resolving? Nope. Could be /var/log getting filled up? Nope. Could be some incredibly unlikely kernal bug that only crops up under extreme incoming mail situations? Thought maybe, but nope.
Eventually though, if you don't give up, you find the right message in the logs which leads to the right posting on some mailing list which leads to the actual solution. Oh, that zombie spamd process I saw a few times in top! If I just kill -9 that process everything goes back to normal. Genius.
Still, I don't really know why that happened. I just know I fixed it. Not an ideal situation, but after a day with my head in the Qmail logs that is good enough.
Whenever I'm in that sort of tecnical situation I find that the why and how of the original problem is so stinking interesting.
I think that is precisely the mark of someone who's good at what they do. You just have to want to get to the bottom of it, to further your own knowledge, if you really want to become good. I have it with a lot of things, but mail servers sort of fall below my threshold. I have to prioritize what I can learn and I'm cool with never being a Qmail pro.
|
- jim 3-18-2008 2:33 am
tell more
- Skinny 3-18-2008 2:38 pm
Pretty boring stuff. Qmail is the mail server I run. It receives mail sent to any address for any domain hosted on my server, and it is what sends outgoing mail to external domains from any of those same accounts on my server. To me it's one of the most complex programs I run. Email transport is very arcane.
But Qmail is great and mostly it just runs. Until something goes wrong which is what happened at the end of last week. It didn't crash, but suddenly everything got real slow, and the server was working much harder than usual. Some people weren't able to send outgoing mail, and some were receiving multiple copies of incoming mail.
So I knew something was wrong. But I had no idea what it was. So in these cases you just try not to panic. Every program like Qmail running on the server has a directory full of logs where it records it's activity. When something goes wrong you go to the logs and start looking through. But there is a ton of information! Qmail is getting hit by incoming spam at a crazy rate, and every transaction hits the logs. And I don't even know what I'm looking for.
But you just keep looking. And when you find something that doesn't seem right you copy and paste that line from the logs into google and see what other people say. This often leads you on a bunch of wild goose chases following the advise of people who had problems but not the same one as yours. Could be DNS is not resolving? Nope. Could be /var/log getting filled up? Nope. Could be some incredibly unlikely kernal bug that only crops up under extreme incoming mail situations? Thought maybe, but nope.
Eventually though, if you don't give up, you find the right message in the logs which leads to the right posting on some mailing list which leads to the actual solution. Oh, that zombie spamd process I saw a few times in top! If I just kill -9 that process everything goes back to normal. Genius.
Still, I don't really know why that happened. I just know I fixed it. Not an ideal situation, but after a day with my head in the Qmail logs that is good enough.
- jim 3-20-2008 8:37 pm
Whenever I'm in that sort of tecnical situation I find that the why and how of the original problem is so stinking interesting.
- L.M. 3-20-2008 11:04 pm
I think that is precisely the mark of someone who's good at what they do. You just have to want to get to the bottom of it, to further your own knowledge, if you really want to become good. I have it with a lot of things, but mail servers sort of fall below my threshold. I have to prioritize what I can learn and I'm cool with never being a Qmail pro.
- jim 3-21-2008 1:35 am