preservetube
/
backend
mirror of https://github.com/preservetube/backend.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

updated robots.txt

This commit is contained in:
localhost 2026-01-31 18:19:39 +01:00
parent 478b760af1
commit b83a90674c
1 changed files with 15 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
public/robots.txt
View File

@ -38,4 +38,19 @@ Disallow: /
User-Agent: ImagesiftBot User-Agent: ImagesiftBot
Disallow: / Disallow: /
# ai crawler for alexa... who uses alexa.
User-agent: Amazonbot
Disallow: /
# apple does not know how to behave.
User-agent: Applebot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# the crawling rate is simply insane
User-agent: Bingbot
Crawl-delay: 2
Sitemap: https://api.preservetube.com/sitemap.xml Sitemap: https://api.preservetube.com/sitemap.xml