Serv-U filter to block Googlebot?

[Chris Wilson]

Anyone using Serv-U FTP Server software? I want to create a filter to stop Googlebot roaming about. Trouble is they seem to use lots of different IP addresses, but they do use the same anonymous password "[email protected]" Can i filter on a password? I can't seem to find a way. Thanks.

[OhGod]

Don't know anything about that particular software, but you could see if the 'robots.txt' method works for FTP. If you can see the logs (I'm guessing you can) is a request being made for 'robots.txt' in your root folder? If this is the case creating a simple text file with the following contents;

 

User-agent: *

Disallow: /

 

Upload that, as 'robots.txt', to the root of your FTP server. Failing that you'd have to block by password, as you say, or the whole google subnet.

[Chris Wilson]

Thanks for that, sadly it's not working!

 

How else can I block Googlebot from slowing down my connection? I only have a 1 meg phone line connection to my ISP, and when Googlebot starts downloadg files from my FTP server it slows everything right down. I made a robots.txt file and and put it in the root of my ftp directory with the contents

User-agent: *

Disallow: /

 

I have just seen Googlebot is still downloading stuff. Here's a bit of todays log:

 

5] Thu 23Jul09 11:32:21 - (000280) ANONYMOUS logged in, password: [email protected]

[3] Thu 23Jul09 11:32:22 - (000280) Sending file d:\home\our_animals_in_the_snow\dsc02292.jpg

[3] Thu 23Jul09 11:32:42 - (000280) Sent file d:\home\our_animals_in_the_snow\dsc02292.jpg successfully (25.8 kB/sec - 544208 Bytes)

[5] Thu 23Jul09 11:32:43 - (000280) Closing connection for user ANONYMOUS (00:00:23 connected)

[5] Thu 23Jul09 12:08:03 - (000281) Connected to 66.249.67.82 (Local address 192.168.0.2)

[5] Thu 23Jul09 12:08:03 - (000281) ANONYMOUS logged in, password: [email protected]

[3] Thu 23Jul09 12:08:04 - (000281) Sending file d:\home\2jz-gte_oil_squirters\dsc03874_t.jpg

[3] Thu 23Jul09 12:08:04 - (000281) Sent file d:\home\2jz-gte_oil_squirters\dsc03874_t.jpg successfully (32.2 kB/sec - 6520 Bytes)

[5] Thu 23Jul09 12:08:08 - (000281) Closing connection for user ANONYMOUS (00:00:05 connected)

[5] Thu 23Jul09 12:39:12 - (000282) Connected to 66.249.67.82 (Local address 192.168.0.2)

[5] Thu 23Jul09 12:39:13 - (000282) ANONYMOUS logged in, password: [email protected]

[3] Thu 23Jul09 12:39:14 - (000282) Sending file d:\home\our_animals_in_the_snow\dsc02277.jpg

[3] Thu 23Jul09 12:39:37 - (000282) Sent file d:\home\our_animals_in_the_snow\dsc02277.jpg successfully (25.7 kB/sec - 606851 Bytes)

[5] Thu 23Jul09 12:39:38 - (000282) Closing connection for user ANONYMOUS (00:00:26 connected)

[5] Thu 23Jul09 13:14:43 - (000283) Connected to 66.249.67.82 (Local address 192.168.0.2)

[5] Thu 23Jul09 13:14:43 - (000283) ANONYMOUS logged in, password: [email protected]

[3] Thu 23Jul09 13:14:44 - (000283) Sending file d:\home\our_animals_in_the_snow\dsc02280.jpg

[3] Thu 23Jul09 13:15:03 - (000283) Sent file d:\home\our_animals_in_the_snow\dsc02280.jpg successfully (25.3 kB/sec - 495790 Bytes)

[5] Thu 23Jul09 13:15:04 - (000283) Closing connection for user ANONYMOUS (00:00:21 connected)

[5] Thu 23Jul09 13:35:07 - (000284) Connected to 66.249.67.82 (Local address 192.168.0.2)

[5] Thu 23Jul09 13:35:07 - (000284) ANONYMOUS logged in, password: [email protected]

[3] Thu 23Jul09 13:35:08 - (000284) Sending file d:\home\our_animals_in_the_snow\dsc02309.jpg

[3] Thu 23Jul09 13:35:30 - (000284) Sent file d:\home\our_animals_in_the_snow\dsc02309.jpg successfully (26.0 kB/sec - 571608 Bytes)

[5] Thu 23Jul09 13:35:31 - (000284) Closing connection for user ANONYMOUS (00:00:24 connected)

[5] Thu 23Jul09 13:46:13 - (000285) Connected to 66.249.67.82 (Local address 192.168.0.2)

[5] Thu 23Jul09 13:46:13 - (000285) ANONYMOUS logged in, password: [email protected]

[3] Thu 23Jul09 13:46:14 - (000285) Sending file d:\home\our_animals_in_the_snow\dsc02319.jpg

[3] Thu 23Jul09 13:46:36 - (000285) Sent file d:\home\our_animals_in_the_snow\dsc02319.jpg successfully (26.1 kB/sec - 595411 Bytes)

[5] Thu 23Jul09 13:46:37 - (000285) Closing connection for user ANONYMOUS (00:00:24 connected)

[5] Thu 23Jul09 14:34:14 - (000286) Connected to 66.249.67.82 (Local address 192.168.0.2)

[5] Thu 23Jul09 14:34:14 - (000286) ANONYMOUS logged in, password: [email protected]

[3] Thu 23Jul09 14:34:15 - (000286) Sending file d:\home\our_animals_in_the_snow\dsc02287.jpg

[3] Thu 23Jul09 14:34:37 - (000286) Sent file d:\home\our_animals_in_the_snow\dsc02287.jpg successfully (26.1 kB/sec - 579451 Bytes)

[5] Thu 23Jul09 14:34:38 - (000286) Closing connection for user ANONYMOUS (00:00:24 connected)

[5] Thu 23Jul09 14:44:03 - (000287) Connected to 66.249.67.82 (Local address 192.168.0.2)

[5] Thu 23Jul09 14:44:03 - (000287) ANONYMOUS logged in, password: [email protected]

[3] Thu 23Jul09 14:44:04 - (000287) Sending file d:\home\our_animals_in_the_snow\dsc02333.jpg

[3] Thu 23Jul09 14:44:23 - (000287) Sent file d:\home\our_animals_in_the_snow\dsc02333.jpg successfully (29.6 kB/sec - 583851 Bytes)

[5] Thu 23Jul09 14:44:25 - (000287) Closing connection for user ANONYMOUS (00:00:22 connected)

[5] Thu 23Jul09 15:12:58 - (000288) Connected to 66.249.67.82 (Local address 192.168.0.2)

[5] Thu 23Jul09 15:12:58 - (000288) ANONYMOUS logged in, password: [email protected]

[3] Thu 23Jul09 15:12:59 - (000288) Sending file d:\home\our_animals_in_the_snow\dsc02284.jpg

[3] Thu 23Jul09 15:13:21 - (000288) Sent file d:\home\our_animals_in_the_snow\dsc02284.jpg successfully (26.0 kB/sec - 584737 Bytes)

[5] Thu 23Jul09 15:13:22 - (000288) Closing connection for user ANONYMOUS (00:00:24 connected)

 

Can I block on password? Any other suggestions please? Thanks!

[Thorin]

Is turning off anonymous logins not an option?

[Multics]

[cheekymonkey]

When did you create the robots.txt? Perhaps the bot hasn't picked it up yet.

 

Google *should* respect the file:

 

http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40364

[Chris Wilson]

I only put it in this morning. Doesn't it work immediately then?

 

If I turn off anonymous I can't show off my handiwork to you guys, that'd be terrible

 

Seriously though, I do like to have anonymous access as it saves a lot of e-mails asking if Mr X can download something. Cheers for the help.

[cheekymonkey]

Googlebot would query robots.txt first, then decide if it can root through your stuff. If robots.txt didn't exist when the bot started its run then it won't have any effect. Keep an eye on things and I reckon you'll see the bot query robots.txt soon enough and that will be the end of it.

 

It's interesting that you run an FTP server rather than a HTTP server though - FTP is more old-skool these days!

[Thorin]

Chris you could block on IP address presumably? Just deny access to Google's net block 66.249.64.0/19.

 

IP addresses 66.249.64.0 - 66.249.95.255

[Chris Wilson]

Serv-U just sent me this reply, looks like I need to block on IP addresses, thanks everyone

 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Hi Chris,

 

There is no way to block just Google from accessing your anonymous account, and robots.txt applies only to HTTP sites, not FTP sites. The only real way to prevent this is to block access from Google entirely, or contact them and ask them to stop crawling your FTP share.

 

You can block access by Google by adding an IP Access rule blocking access to google.com and another for googlebot.com - this should prevent Google from accessing your server. Alternately you can disable anonymous access.

 

Please let me know if there is anything else I can do for you!

 

Thomas J. Parikka - Technical Support Engineer

http://www.RhinoSoft.com

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~