车东[Blog^2]

IaskSpider的访问来源和频度分布：
grep -i iaskspider *|awk '{print $1}' |awk -F ':' '{print $2}' |sort|uniq -c|sort -rn |head -10
490 219.142.78.87
444 219.142.118.62
429 219.142.78.210
390 219.142.118.63
355 219.142.118.65
352 219.142.78.59
320 219.142.78.85
306 219.142.118.64
304 219.142.78.60
247 219.142.78.179

和国内外其他主流搜索引擎的SPIDER不一样：Sohu和新浪的都不抓取robots.txt文件，这样就有可能存在一些不希望被搜索引擎收录的内容被公开的潜在威胁。

附：AWStats中增加针对Sohu Agent和IaskSpider的统计
Index: robots.pm
===================================================================
RCS file: /cvsroot/awstats/awstats/wwwroot/cgi-bin/lib/robots.pm,v
retrieving revision 1.40
diff -r1.40 robots.pm
214a215
> 'iaskspider',
321a323
> 'sohu',
718a721
> 'iaskspider','Sina Iask',
724a728
> 'sohu','SohuSpider',

国内搜索引擎的定义：
===================================================================
RCS file: /cvsroot/awstats/awstats/wwwroot/cgi-bin/lib/search_engines.pm,v
retrieving revision 1.35
diff -r1.35 search_engines.pm
94c94,96
< 'baidu\.com','search\.sina\.com','search\.sohu\.com',
---
> 'baidu\.com','search\.sohu\.com','iask\.com',
> '\.3721\.com','\.163\.com','search\.tom\.com','search\.21cn\.com','\.qq\.com',
> 'yisou\.com','search\.china\.com','zhongsou\.com', 'search\.online\.', 'sogou\.com',
203c205
< 'search\.sina\.com','sina',
---
> 'iask\.com','iask',
204a207,216
> '\.3721\.com','3721',
> '\.163\.com','netease',
> 'search\.tom\.com','tom',
> 'search\.21cn\.com','21cn',
> '\.qq\.com','qq',
> 'yisou\.com','yisou',
> 'search\.china\.com','china',
> 'zhongsou\.com','zhongsou',
> 'search\.online','shonline',
> 'sogou\.com','sogou',
360c372,374
< 'baidu','word=', 'sina', 'word=', 'sohu','word=',
---
> 'baidu','(bs|wd|word)=', 'iask','(_searchkey|k)', 'sohu','(query|key_word|word)=',
> '3721','(p|name)=', 'netease','q=', 'tom','word=', '21cn','word=', 'qq','(word|w)=',
> 'yisou','p=', 'china','query=', 'zhongsou','word=', 'shonline','word=', 'sogou', 'query=',
473c487,490
< 'baidu','Baidu', 'sina','Sina', 'sohu','Sohu',
---
> 'baidu','Baidu', 'iask','Iask', 'sohu','Sohu',
> '3721','3721', 'tom','Tom', 'netease', 'NetEase', '21cn','21CN',
> 'qq','QQ', 'yisou','YiSou', 'zhongsou','ZhougSou', 'shonline','SHOnline',