v1.2.1
386
This plugin uses the Known Agent API (formerly known as the DarkVisitors API) to retrieve up-to-date lists of AI user agents and to create a robots.txt file. You will need a free API token for this.
Installation
composer require mauricerenck/darkvisitors
Download the latest version here, un-zip it and copy the folder to site/plugins/dark-visitors.
Setup
You need a Dark Visitors Access Token to use this plugin.
Go to https://knownagents.com/ and create an account and a project. Open the project and get your token under "Settings".
Edit your config.php file and add this line:
'mauricerenck.dark-visitors.token' => 'YOUR TOKEN'
Block ai crawler
Define which types of AI crawlers you want to block:
'mauricerenck.dark-visitors.aiTypes' => ['AI Assistant', 'AI Data Scraper', 'AI Search Crawler'],
The following types are available:
- AI Agent
- AI Assistant
- AI Data Scraper
- AI Search Crawler
- Archiver
- Developer Helper
- Fetcher
- Automated Agent
- Intelligence Gatherer
- Scraper
- SEO Crawler
- Search Engine Crawler
- Security Scanner
- Undocumented AI Agent
- Uncategorized
Force blocking
Some crawlers ignore the robots.txt file and still collect data. To prevent this, you can force blocking:
'mauricerenck.dark-visitors.force-block' => true,
This blocks configured agents via Kirby routing and returns a 403.
Although the UserAgent list is well maintained, you should be aware that incorrect matches may prevent visitors from accessing your site.
The Kirby route method works reliably and should function well with a manageable number of requests. For a high number of requests, it is recommended to block agents server-side, for example, via the .htaccess or nginx configuration.
Custom rules
Add your own rules to the robots.txt:
'mauricerenck.dark-visitors.agents' => [
[
'userAgents' => ['Googlebot', 'Bingbot'],
'disallow' => ['/admin'],
],
[
'userAgents' => ['Bingbot'],
'allow' => ['/microsoft'],
],
],
Your own rules override the default rules. The default rules are:
[
'userAgents' => ['*'],
'disallow' => ['/kirby', '/site'],
];
Sitemaps
Add your sitemaps to the robots.txt:
'mauricerenck.dark-visitors.sitemaps' => [
'Sitemap: https://your-site.tld/sitemap.xml',
'Sitemap: https://your-site.tld/sitemap2.xml',
],
Tracking/Analytics
KnownAgents offers a tracking feature. If you want to use it, you can activate it in the configuration:
'mauricerenck.dark-visitors.analytics' => true,