robots.txt搜索结果
- Focused Crawling and Collection Synthesis
Dec 20, 2002 ... Martin Koster (1994); http://any-server:80/robots.txt; Maintained by the webmaster ; Forbid access to pages, directories; Commonly excluded: ...
metadata-wg.mannlib.cornell.edu - 网页快照 - Diapositiva 1
Solution: Robot Exclusion Protocol; File: /robots.txt. Crawler Architecture. Per- server. queues. Robots. Mercator's scheduler. BACK-END: ensures. politeness ...
medialab.di.unipi.it - 网页快照 - Trespass to Chattels
Apr 23, 2009 ... “Robots.txt” file. Good example of Larry Lessig's concept of “code as law”. Automated software codes shape the legal environment, just as ...
www.law.berkeley.edu - 网页快照 - PKI and storage of PGP and X.509 certificates in LDAP
we should use a similiar approach as in the Web: robots.txt; Crawler policy tells the crawler which behaviour is wanted and which is not; LDAP crawler policy ...
www.openldap.org - 网页快照 - Slide 1
Using the Robot Exclusion Protocol. Web Server Technologies | Part II: Web Applications & Server Maintenance. Place a robots.txt file in the site's document root ...
www.port80software.com - 网页快照 - A Reflection on Spiders, Bots and Aggregators
Mar 13, 2001 ... Bot Friendly Sites. Meta tags for bots to locate; Friendly robots.txt files; Terms of service agreements that allow bot usage. Mata Tags ...
www.jeffheaton.com - 网页快照 - Invasive Browser Sniffing and Countermeasures
Dealing with search engines; Robots.txt “standard” (no problem if cheating). Pollution Policy. Pollute entrance URLs; How to choose pollutants? What about links ...
research.sidstamm.com - 网页快照 - Getting Google to Love Your Website
Misconfigurations (in robots.txt, in the type of redirects used, requiring cookies, etc.) can kill indexation. Utilize XML sitemaps to help indexation & overcome ...
assets.en.oreilly.com - 网页快照
4566文档搜索©2010 www.4566.info