ai.robots.txt
Version without Internet Archive? #74
TheZeldaZone
started this conversation in
Ideas
Replies: 1 comment
-
|
The Internet Archive crawler has user agent ia_archiver (see this question), which is not listed in robots.txt. Please could you say why you believe the Internet Archive is blocked? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
This list, while editable, has the same problem Cloudflare's AI blocker did, it includes Internet Archive for goodness knows why. I'm not sure why they're included at all but a branch without them would be handy at least.
Beta Was this translation helpful? Give feedback.
All reactions