docs: add robots.txt for sitemap (#2083)

ghostdevv · delucis · web-flow · commit 7ad711321ccf · 2024-08-06T22:11:46.000+02:00
Co-authored-by: Chris Swithinbank &lt;swithinbank@gmail.com&gt;
diff --git a/docs/src/assets/robots.txt b/docs/src/assets/robots.txt
@@ -0,0 +1,4 @@
+User-agent: *
+Allow: /
+
+Sitemap: https://starlight.astro.build/sitemap-index.xml
diff --git a/docs/src/content/docs/guides/customization.mdx b/docs/src/content/docs/guides/customization.mdx
@@ -107,6 +107,8 @@ export default defineConfig({
 });
 ```
 
+Learn how to [add a sitemap link to `robots.txt`](https://docs.astro.build/en/guides/integrations-guide/sitemap/#sitemap-link-in-robotstxt) in the Astro Docs.
+
 ## Page layout
 
 By default, Starlight pages use a layout with a global navigation sidebar and a table of contents that shows the current page headings.
