# /images is blocked in general, but social-share images must stay crawlable # or link previews (og:image / twitter:image) break on crawlers that respect # robots.txt (e.g. Twitterbot). User-agent: * Allow: /images/site/ Allow: /images/*-thumbnail Disallow: /images User-agent: archive.org_bot Disallow: / User-agent: ia_archiver Disallow: / Sitemap: https://ryanarnaudin.com/sitemap.xml