diff options
| author | pdp8 <pdp8@pdp8.info> | 2024-03-04 12:51:29 +0100 |
|---|---|---|
| committer | pdp8 <pdp8@pdp8.info> | 2024-03-04 12:51:29 +0100 |
| commit | 0ea7a2404b1d2af11947ca0fbedc614cf586a07e (patch) | |
| tree | d55c4b82c670dce28c7cca28d56beeec707fef0f | |
| parent | 5371b5b4e9950549d74047e608a5fb6d75f80ab7 (diff) | |
disallow AI bots in robots.txt
| -rw-r--r-- | robots.txt | 36 | ||||
| -rw-r--r-- | sitemap.txt | 11 |
2 files changed, 45 insertions, 2 deletions
@@ -1,2 +1,34 @@ -User-agent: * -Disallow: +User-agent: Amazonbot +Disallow: / + +User-agent: CCBot +Disallow: / + +User-agent: ChatGPT-User +Disallow: / + +User-agent: GPTBot +Disallow: / + +User-agent: Google-Extended +Disallow: / + +User-agent: anthropic-ai +Disallow: / + +User-agent: Omgilibot +Disallow: / + +User-agent: Omgili +Disallow: / + +User-agent: FacebookBot +Disallow: / + +User-agent: Bytespider +Disallow: / + +User-agent: ImagesiftBot +Disallow: / + +SITEMAP: https://pdp8.info/sitemap.txt diff --git a/sitemap.txt b/sitemap.txt new file mode 100644 index 0000000..1dce6f3 --- /dev/null +++ b/sitemap.txt @@ -0,0 +1,11 @@ +https://pdp8.info +https://pdp8.info/about.html +https://pdp8.info/climbing.html +https://pdp8.info/code.html +https://pdp8.info/contact.html +https://pdp8.info/index.html +https://pdp8.info/music.html +https://pdp8.info/pictures.html +https://pdp8.info/social/announce.html +https://pdp8.info/social/create.html +https://pdp8.info/videos.html |