preservetube
/
backend
mirror of https://github.com/preservetube/backend.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

a few more

This commit is contained in:
localhost 2026-02-01 16:17:37 +01:00
parent 54e7e3daa1
commit 0c9cabaa93
1 changed files with 6 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
public/robots.txt
View File

@ -40,6 +40,12 @@ Disallow: /
User-Agent: ImagesiftBot User-Agent: ImagesiftBot
Disallow: / Disallow: /
User-agent: DataForSeoBot
Disallow: /
User-agent: facebookexternalhit
Disallow: /
# ai crawler for alexa... who uses alexa. # ai crawler for alexa... who uses alexa.
User-agent: Amazonbot User-agent: Amazonbot
Disallow: / Disallow: /