|
2863561c89
|
[chore] Update robots.txt with latest ai bullshit (#4024)
|
2025-04-19 13:34:20 +02:00 |
|
|
fd670c6a27
|
[feature] Use ETag for robots.txt to prevent mishaps (#3829)
* [feature] Use ETag for robots.txt to prevent mishaps
* check incoming if-none-match header
|
2025-02-24 11:17:18 +01:00 |
|
|
baed591a1d
|
[feature] Use X-Robots-Tag headers to instruct scrapers/crawlers (#3737)
* [feature] Use `X-Robots-Tag` headers to instruct scrapers/crawlers
* use switch for RobotsHeaders
|
2025-02-05 12:47:13 +01:00 |
|