User-agent: * ### Only allow URLs generated with RealURL ### #Disallow: /*?id=* #Disallow: /*&id=* ### L=0 is the default language ### #Disallow: /*?L=0* #Disallow: /*&L=0* Disallow: *.sql Disallow: *.sql.gz #User-agent: Mediapartners-Google ### For Google-AdSense please uncomment the line above and below #Disallow: ### Disable some useless crawler-bots ### User-agent: wget User-agent: sitecheck.internetseer.com User-agent: Zealbot User-agent: MSIECrawler User-agent: SiteSnagger User-agent: WebStripper User-agent: WebCopier User-agent: Teleport User-agent: TeleportPro User-agent: WebZIP User-agent: HTTrack User-agent: k2spider User-agent: WebReaper Disallow: /