Unknown robot /bandwith killer

[shakil420]

helow , recently i face a problem. in my hosting somone hit my pages with an unknown robot and eat all of my avilable bandwith here is my statistics:
----------------------------------------------------------------------------
robots Hits Bandwidth Last visit
Unknown robot 1375881 3.91 GB 09 Nov 2010 - 04:22
-----------------------------------------------------------------------------
Now can anyone tell me how does he do that and how can i protect my site from robots?
best regards . thank you

[pmbguy]

The Robots META Tag is meant to provide users who cannot upload or control the /robots.txt file at their websites, with a last chance to keep their content out of search engine indexes and services.

<meta name="robots" content="robots-terms">

Examples of the Robots META Tag

The content="robots-terms" is a comma separated list used in the Robots META Tag that may contain one or more of the following keywords without regard to case: noindex, nofollow, all, index and follow.

noindex
Page may not be indexed by a search service.

<meta name="robots" content="noindex">

nofollow
Robots are not to follow links from this page.

<meta name="robots" content="nofollow">

Admin Note: The robots directives of index, follow or all are not required as it is the default behavior of indexing spiders.

<meta name="robots" content="index, follow">
<meta name="robots" content="all">

index
Robots are welcome to include this page in search services.

<meta name="robots" content="index">

follow
Robots are welcome to follow links from this page to find other pages.

<meta name="robots" content="follow">

If this meta tag is missing, or if there is no content, or the robot terms are not specified, then the robot terms will be assumed to be "index, follow" (e.g. "all"). If the keyword all is found in the robots terms list it overrides all other values. That is, a robots terms that is "nofollow, all, noindex, nofollow", would effectively be "all".

If the robots terms contains contradictory information (e.g. "follow, nofollow, follow") then the robot is free to do whatever it wishes with regard to the behavior being addressed (in this case the follow behavior).

Back to Previous
Common Usage for the Robots META Tag

The Robots META Tag is used for excluding content. We've included three (03) examples below of using the robots meta tag correctly to exclude information from search engine indexes and services.

A robots term of noindex allows the links on that page to be followed, even though the page is not to be indexed.
<meta name="robots" content="noindex">

A robots term of nofollow allows the page to be indexed, but no links from the page are followed.
<meta name="robots" content="nofollow">

A robots terms of noindex, nofollow neither the page or the links on that page will be followed or indexed.
<meta name="robots" content="noindex, nofollow">

source

[shakil420]

your information is nice thanx .but i cant find any solution yet to aviod this robot . thank u

[pmbguy]

lol dude:
<meta name="robots" content="noindex, nofollow">

[morse]

lol dude:
<meta name="robots" content="noindex, nofollow">

That will hardly solve the problem if a robot is "intentionally" used
Find out the ip from which the robot is accessing your site and then block the ip range with htaccess.

[ashuwap]

disallow blank user agent......use .htaccess

[shakil420]

@@ashuwap sound cool . 100% aqgree with u .. thanx bro .. can u tell my what is the .htaccess code of block #blank user agent# ??
hope u reply soon...
===============================
@@ pmbguy , dude i really dont understand ,# <meta name="robots" content="noindex, nofollow"> # where i use the code ??? please reply soon
===============================
@@ morse , really i dont found the IP , may b he use hidden ip(not sure abiut that)
===============================
best regards

[khan]

On Your .htaccess

SetEnvIf User-Agent "^$"  NULL_AGENT
Deny from env=NULL_AGENT

Or

RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector
RewriteRule .* - [F,L]