preservetube
/
backend
mirror of https://github.com/preservetube/backend.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

chill out openai

This commit is contained in:
localhost 2026-02-18 23:27:17 +01:00
parent f4216239f3
commit 9784277c81
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
public/robots.txt
View File

@ -70,6 +70,10 @@ Disallow: /
User-agent: Bingbot User-agent: Bingbot
Crawl-delay: 2 Crawl-delay: 2
User-agent: GPTBot
Crawl-delay: 2
# this is intentionally only accessable for search engine crawlers. # this is intentionally only accessable for search engine crawlers.
# do you want to crawl too? get in touch https://preservetube.com/about # do you want to crawl too? get in touch https://preservetube.com/about
Sitemap: https://api.preservetube.com/sitemap.xml Sitemap: https://api.preservetube.com/sitemap.xml