preservetube
/
backend
mirror of https://github.com/preservetube/backend.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

and thse

This commit is contained in:
localhost 2026-02-01 16:19:20 +01:00
parent 0c9cabaa93
commit 4eb81a53cd
1 changed files with 6 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
public/robots.txt
View File

@ -46,6 +46,12 @@ Disallow: /
User-agent: facebookexternalhit User-agent: facebookexternalhit
Disallow: / Disallow: /
User-Agent: MJ12bot
Disallow: /
User-agent: Owler
Disallow: /
# ai crawler for alexa... who uses alexa. # ai crawler for alexa... who uses alexa.
User-agent: Amazonbot User-agent: Amazonbot
Disallow: / Disallow: /