URL
GET https://www.redditstatic.com/robots.txt
Response Headers
Thu, 09 Jun 2022 10:52:43 GMT
Response Body
User-Agent: *
Disallow: /
Allow: /desktop2x/img/favicon/*
Allow: /favicon.ico
Allow: /icon.png
Allow: /new-icon.png
Allow: /desktop2x/img/social-sharing/*
Allow: /talk/assets/social-sharing/*
User-agent: Googlebot
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: AdsBot-Google-Mobile-Apps
User-agent: Googlebot-Image
User-agent: Googlebot-Video
User-agent: Applebot
User-agent: ClarityBot
Disallow:
User-agent: bingbot
User-agent: MSNBot-Media
Allow: /