Dark Visitors

Dark Visitors is a plugin for Kirby that blocks unwanted AI Crawlers from your website using robots.txt. It uses the Dark Visitors API to identify and block unwanted visitors. It also allows you to add custom rules and your sitemaps to your robots.txt file.

Installation

composer require mauricerenck/darkvisitors

Or download the latest release unzip it, copy it to site/plugins/dark-visitors

Get the access token

You need a Dark Visitors access token to use this plugin.
Go to https://darkvisitors.com/ create an account and your own Project. Open your project and get your access token under settings.

Usage

Edit your config.php and add the following line:

'mauricerenck.dark-visitors.token' => 'YOUR TOKEN'

AI crawlers

Set which types of AI crawlers you want to block:

'mauricerenck.dark-visitors.aiTypes' => ['AI Assistant', 'AI Data Scraper', 'AI Search Crawler'],

Your custom rules

Add your custom rules to the robots.txt file:

'mauricerenck.dark-visitors.agents' => [
    [
        'userAgents' => ['Googlebot', 'Bingbot'],
        'disallow' => ['/admin'],
    ],
    [
        'userAgents' => ['Bingbot'],
        'allow' => ['/microsoft'],
    ],
],

Setting your custom rules will overwrite the default rules, which are:

[
    'userAgents' => ['*'],
    'disallow' => ['/kirby', '/site'],
];

Sitemaps

Add your sitemaps to the robots.txt file:

'mauricerenck.dark-visitors.sitemaps' => [
    'Sitemap: https://your-site.tld/sitemap.xml',
    'Sitemap: https://your-site.tld/sitemap2.xml',
],

Tracking/Analytics

Darkvisitors offers a tracking feature. If you want to use it, you can enable it in the config:

'mauricerenck.dark-visitors.analytics' => true,

Learn more about robots.txt and AI crawlers

You can comment on this post from your blog.

This post reacts to webmentions. You can link it from your website and send a webmention. Your contribution will then appear in the comments here. Does your site not support webmentions? Enter the link to your post here:

Write a comment
By submitting your data, you agree that all entered data may be saved and displayed as a comment.
Like, Share, Reply

I'd love to hear from you! Did you enjoy this post? Leave a comment, link your blog post, or react on Mastodon and Bluesky.