Dark Visitors

Matthias has written a wonderful post about AI scrapers. Many of us website owners are currently facing the situation of being overrun by bots and crawlers. This goes so far that our sites become unusable because they are either completely overloaded or because the hosting provider has blocked entire countries, as in Matthias's case.

Matthias describes a few ways to counteract this. All of these methods work more or less well and are essentially just treating the symptoms. Of course, it would be better to solve the problem at its root cause. However, this probably won't happen anytime soon, because most of these companies couldn't care less about the consequences of their actions.

I've already written briefly about how I also had to resort to activating geo-blocking. I've also extended the DarkVisitors plugin for Kirby.

Besides the option to create a robots.txt file that asks AI crawlers not to crawl pages, there's now also the option to block crawlers. Most AI crawlers don't pay attention to the robots.txt file, so we need to be a bit stricter. If the corresponding option is enabled, the crawler is rejected directly by Kirby. Matthias describes something similar in his post.

Of course, this can also be done directly in the web server configuration, and that's probably the better place to do it. However, the plugin has the advantage of being able to intervene even where you can't modify the web server configuration, and you also benefit from a constantly updated list of user agents.

Anyone using Kirby who's annoyed by AI crawlers should check out the plugin. Anyone who wants to do even more should read Matthias's article.