Do you hate LLM crawlers? Here's how to block them using traefik

Earlier today I noticed that three days ago the OpenAI GPTBot crawler started spamming my hosted websites with about 120 requests per minute. Firstly, this is an inacceptable waste of my resources, my electricity and cpu capacity. Secondly I hate the LLM industry with a passion and I absolutely do not want companies like OpenAI to train their products on my work and have them profit off of it.

So I decided to block them 🖕

A while ago I did some research into this, but back then the need wasn’t great enough yet, so I didn’t implement anything. That came in handy now, since I knew already that the best way to go about this would probably be to block requests based on the useragent. This of course depends on the requester being honest - which they probably won’t always be. But for now, a quick exploratory analysis of my logs yielded the result that they indeed seem to be setting a real useragent, at least on all of the problematic requests.

An alternative would be to block based on publicly known IP lists. Most of the common crawlers publish such IP lists (here is the offending openai.com/gptbot), but they change regularly and that would require me to track them and update my blocklists somehow. I could automate that, but useragents seem much more stable to me.

I use traefik as a reverse proxy into my hosted services. The more or less maintained (last commit two years ago) plugin “Block User-Agent” does what I want it to do: Provide a middleware that blocks requests with a useragent matching a regex of my choice.
Since depending on unmaintained software isn’t great, I’ve marked the GitHub Repo for watching and I’ll keep tabs on it. Should the need ever arise, I can still fork it and maintain it myself, since it is a rather simple plugin.

Since I am using CLI parameters for the install configuration of my traefik instance, I had to add two lines to the CLI parameters for my traefik instance:

services:
  traefik:
    image: "traefik:v3.7"
    command:
      # ... some other configuration ...
      - "--experimental.plugins.blockuseragent.modulename=github.com/agence-gaya/traefik-plugin-blockuseragent"
      - "--experimental.plugins.blockuseragent.version=v0.1.8"

This activates the experimental auto-install of plugins and installs v0.1.8 of the plugin. Again, this is an experimental feature, so I will keep tabs on traefik and see if configuration around it changes in the future as the feature stabilizes.

Then on to the plugin configuration. Since I am using docker labels for the routing configuration, I’ll format it here accordingly. I added a middleware using the plugin:

services:
  traefik:
    # ... docker compose service configuration for traefik ...
    labels:
      - "traefik.http.middlewares.block-llms.plugin.blockuseragent.regex=openai\\.com"

And then added the label e.g. to my blog’s routing configuration:

services:
  yeldirium:
    # ... docker compose service configuration for my website ...
    labels:
      - "traefik.http.routers.yeldirs-blog.middlewares=block-llms"

And voilà:

You can of course extend the regex to block whatever you like. I expanded it to (openai\\.com|anthropic\\.com) and I am very open to suggestions for more. Those two were the only problematic ones present in my logs.

Have fun and resist.