前几天我的 bluehost 账号总有 CPU 使用超量的问题，今天甚至因此被暂时停掉。察看了下访问记录，发现有个叫 twiceler 的 robot 很奇怪，每秒种访问同一个页面多次（例子附在帖子最后，懂行的人帮我看看）。猜它就是问题的来源，不过并不确定。似乎有人在做一个叫 Cuill 的新搜索引擎，放出这个叫 Twiceler 的机器人。我现在只好阻止它了。在 robots.txt 里加上
User-agent: twiceler
Disallow: /
略微学习一下这个 Cuill，发现还被人认为是 Google 的有力挑战者呢。他们自己声称 Cuill 检索网页的速度比 Google 快，成本低十倍。
-----------------……
38.117.64.101 - - [05/Oct/2007:21:59:38 -0600] “GET /111/ HTTP/1.0″ 302 0 “http://vonye.com/111″ “Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)”
38.117.64.101 - - [05/Oct/2007:21:59:39 -0600] “GET /111/ HTTP/1.0″ 302 0 “http://vonye.com/111″ “Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)”
38.117.64.101 - - [05/Oct/2007:21:59:39 -0600] “GET /111/ HTTP/1.0″ 302 0 “http://vonye.com/111″ “Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)”
38.117.64.101 - - [05/Oct/2007:21:59:39 -0600] “GET /111/ HTTP/1.0″ 302 0 “http://vonye.com/111″ “Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)”
38.117.64.101 - - [05/Oct/2007:21:59:40 -0600] “GET /111/ HTTP/1.0″ 302 0 “http://vonye.com/111″ “Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)”
38.117.64.101 - - [05/Oct/2007:21:59:40 -0600] “GET /111/ HTTP/1.0″ 302 0 “http://vonye.com/111″ “Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)”
38.117.64.101 - - [05/Oct/2007:21:59:40 -0600] “GET /111/ HTTP/1.0″ 302 0 “http://vonye.com/111″ “Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)”
38.117.64.101 - - [05/Oct/2007:21:59:41 -0600] “GET /111/ HTTP/1.0″ 302 0 “http://vonye.com/111″ “Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)”
38.117.64.101 - - [05/Oct/2007:21:59:41 -0600] “GET /111/ HTTP/1.0″ 302 0 “http://vonye.com/111″ “Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)”
38.117.64.101 - - [05/Oct/2007:21:59:42 -0600] “GET /111/ HTTP/1.0″ 302 0 “http://vonye.com/111″ “Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)”
38.117.64.101 - - [05/Oct/2007:21:59:42 -0600] “GET /111/ HTTP/1.0″ 302 0 “http://vonye.com/111″ “Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)”
38.117.64.101 - - [05/Oct/2007:21:59:42 -0600] “GET /111/ HTTP/1.0″ 302 0 “http://vonye.com/111″ “Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)”
38.117.64.101 - - [05/Oct/2007:21:59:43 -0600] “GET /111/ HTTP/1.0″ 302 0 “http://vonye.com/111″ “Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)”
38.117.64.101 - - [05/Oct/2007:21:59:43 -0600] “GET /111/ HTTP/1.0″ 302 0 “http://vonye.com/111″ “Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)”
38.117.64.101 - - [05/Oct/2007:21:59:43 -0600] “GET /111/ HTTP/1.0″ 302 0 “http://vonye.com/111″ “Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)”
……
(cc) 桑葚，来自桑林志 | 阻止 Twiceler