Jim, is the Scooter bot back? Something's hitting the "log (but not you)" referer page pretty hard, going through the archive page by page, and I see Scooter listed in useragent. I remember you decided to zap it some point.
You're right that all our defenses are down. We were blocking some at one point, but that file was lost along the way. What do you think? Scooter is the alta-vista bot. I remember we found that it wasn't as well behaved as the google bot (it hit a lot harder.)
But bandwidth isn't really a problem any more. I guess it might be slowing things down, but I sort of doubt it. Or at least not very much.
I'm all for reblocking the turn-it-in and mail syphon bots. I'll try to get a handle back on those.
But should we block alta-vista? I never use it, but there is an argument for letting it index, since we don't want google to be the only one with an index, right? In that same vein I'd think we'd want to keep inktomi as well (yahoo bot.)
I don't really feel too strongly either way. Tell me what you think. Or anyone else?
I think the alta-vista bot should be allowed in.
Scooter (if it is Scooter--it wasn't doing this till yesterday) creates a lot of red numbers (and page length) on the "log (but not you)" page instead of working "behind the scenes" like the others do. I guess I don't really care, especially since I've argued for search engine biodiversity. Getting rid of the turnitinbot (that name!) would be a blessing though.
Oh, I see. Well I could probably rectify the log display problem without banning the robot. I'll think about how to do this. (Now I remember the problem...)
Great, that would be ideal if it's not too much of a pain. The page is taking a long time to load with all the scooting going on.
OK, I have zapped the Scooter bot. He still indexes pages, but I am no longer recording the referering page, so he won't show up in your referer log. Well, he will, but he'll be part of that big first number that is the combinded total of all hits to your page that don't have a referer (which means anyone who just typed in your URL, loaded from a bookmark, the google bot, now the scooter bot, maybe some other web robots too.)
If you change your log view to useragent you can still see how many times the scooter bot hit your page.
Also, I zapped the old scooter bot entries, so this change should be retroactive, and the effects should be immediately apparent.
Please let me know if it seems like it isn't working.
if ($user_agent == 'Scooter/3.3') {
$referer = '';
}
It's working like a charm, thanks!
Jim, this bot just did to my referer log what Scooter was doing before you zapped it: MSNBOT/0.1 (http://search.msn.com/msnbot.htm) -- (986 hits in the referer log)
I gather this is some new breed of MSN bot. The log is taking forever to load; if you could zap it the same way you did the Scooter I'd be much obliged.
|
- tom moody 4-25-2003 6:57 pm
You're right that all our defenses are down. We were blocking some at one point, but that file was lost along the way. What do you think? Scooter is the alta-vista bot. I remember we found that it wasn't as well behaved as the google bot (it hit a lot harder.)
But bandwidth isn't really a problem any more. I guess it might be slowing things down, but I sort of doubt it. Or at least not very much.
I'm all for reblocking the turn-it-in and mail syphon bots. I'll try to get a handle back on those.
But should we block alta-vista? I never use it, but there is an argument for letting it index, since we don't want google to be the only one with an index, right? In that same vein I'd think we'd want to keep inktomi as well (yahoo bot.)
I don't really feel too strongly either way. Tell me what you think. Or anyone else?
- jim 4-25-2003 7:17 pm [add a comment]
I think the alta-vista bot should be allowed in.
- steve 4-25-2003 8:03 pm [add a comment]
Scooter (if it is Scooter--it wasn't doing this till yesterday) creates a lot of red numbers (and page length) on the "log (but not you)" page instead of working "behind the scenes" like the others do. I guess I don't really care, especially since I've argued for search engine biodiversity. Getting rid of the turnitinbot (that name!) would be a blessing though.
- tom moody 4-25-2003 8:14 pm [add a comment]
Oh, I see. Well I could probably rectify the log display problem without banning the robot. I'll think about how to do this. (Now I remember the problem...)
- jim 4-25-2003 8:16 pm [add a comment]
Great, that would be ideal if it's not too much of a pain. The page is taking a long time to load with all the scooting going on.
- tom moody 4-25-2003 11:54 pm [add a comment]
OK, I have zapped the Scooter bot. He still indexes pages, but I am no longer recording the referering page, so he won't show up in your referer log. Well, he will, but he'll be part of that big first number that is the combinded total of all hits to your page that don't have a referer (which means anyone who just typed in your URL, loaded from a bookmark, the google bot, now the scooter bot, maybe some other web robots too.)
If you change your log view to useragent you can still see how many times the scooter bot hit your page.
Also, I zapped the old scooter bot entries, so this change should be retroactive, and the effects should be immediately apparent.
Please let me know if it seems like it isn't working.
if ($user_agent == 'Scooter/3.3') {
$referer = '';
}
- jim 4-26-2003 1:26 am [add a comment]
It's working like a charm, thanks!
- tom moody 4-26-2003 1:39 am [add a comment]
Jim, this bot just did to my referer log what Scooter was doing before you zapped it:
MSNBOT/0.1 (http://search.msn.com/msnbot.htm) -- (986 hits in the referer log)
I gather this is some new breed of MSN bot. The log is taking forever to load; if you could zap it the same way you did the Scooter I'd be much obliged.
- tom moody 6-29-2003 6:39 am [add a comment]