Yes, it does have a cost. It has a far smaller cost than serving the real thing. It also allows me to firewall them off and stop serving them, even if they come at me with real browsers. That’s a very definitive win: I saved CPU time, I saved RAM, I saved network bandwidth, and I stopped them from accessing my stuff. How is that not a win?
The Daily Mail (vomit) alone publishes 1,500 articles a day. How many do you plan on publishing?
I have an automatically generated infinite maze. It produces roughly a million unique pages each day. It used to produce ~60 million pages / day, but a few months ago I decided to firewall some of the crawlers off instead of serving them garbage.
And I run niche sites. A site with more lucrative traffic than mine (eg, Codeberg, who uses the same software I do) likely generates a lot more garbage.
There was also a paper, commissioned by Anthropic, I believe, that concluded that only 250 malicious pages they fail to remove from the training set is enough to poison even the largest model. Now, I do not trust anything Anthropic says. But even if we’d need a billion pages to poison a model… I alone served that much in the past year.
Unless a significant portion of the internet does this, and we’re talking hundreds of millions of pages, the only cost here is to you.
Fun twist: no! There’s a very neat trick you can do when you serve the crawlers poison: you can hide an identifier in the URLs you serve them, and you can then identify that id when they come back riding on the back of remote controlled chromes. By serving them garbage, you can overload their queue with poisoned ones, which helps you block crawlers that you wouldn’t otherwise be able to block.
Generating and serving garbage is incredibly cheap (cheaper than serving a file from a filesystem on SSD, in most cases), and once you have requests landing on poisoned URLs, you can firewall them off for a day or so, and reduce your costs even more.
We may not be able to poison the models, but we can poison their crawling queues. I have a year’s worth of data to support that. They still haven’t caught on.
Scriptability has been a thing since 2.2.0, released on 2025-06-16, but the built-in script appeared in 3.0 (2025-11-14).
Around here. In the default configuration, it is using the built-in handler. The script can be replaced with something like Nam-Shub of Enki (used by pretty much everything I host, and by Codeberg too, for example).
I’m very late to the party, but: no, iocaine does not expect you to detect the bots. It used to, but it does its own detection for quite a while now (you can replace the detection mechanism, though).
Hungarian: “A nap végén kérlek pakoljátok egymásra a székeket.” (Rough translation back to English: “At the end of the day, I ask you (informal) to stack the chairs on top of each other”).
Same thing, but more formal: “A nap végén kérjük pakolják egymásra a székeket.” (~English: “At the end of the day, we ask you (formatl) to stack the chairs on top of each other”).
In an even more formal / authoritative way: “A nap végén kéretik a székeket egymásra pakolni.” (back to English: “At the end of the day, chairs shall be stacked on top of each other”)
I self-host my email using postfix, dovecot, rspamd and others. The only tradeoff I had to make here is that some of the entities I have to communicate with via email use an allow-list, so some of my outgoing mail is sent through a relay (SMTP2Go).
- Cloud storage / file sync
I self-host a minio for cloud storage. I don’t need file sync, so nothing there. If I would, I would likely use syncthing.
- Maps & navigation
OpenStreetMaps & CoMaps. Works much better than Google Maps did.
- Search engine
Currently a self-hosted YaCy. I have my own index. Not entirely happy with this setup, will switch to something else (still self-hosted, I have no need for a general purpose search engine that indexes the entire internet of slop).
- Web browser
LibreWolf
- Calendar
I’m using Emacs & Org for most calendaring. Wife’s using GNOME Calendar & a Calendar app I found for her on f-droid (unsure which one).
- Contacts management
Nothing on desktop, some random contacts app from f-droid on the phone. I do use EteSync to keep a backup, and potentially sync later. (EteSync syncs her calendar too)
- Notes / to-do lists
Emacs & Org.
- Office suite (docs, spreadsheets, etc.)
Most of my “office” needs are covered by a combination of Emacs, Typst and Zola one way or another. For the rare case where I need Office compatiblity: LibreOffice.
- Messaging / chat
XMPP. Dino on Linux, Conversations on Android. I use Matrix too, from time to time (Element), and have Signal too. Not a big fan of the latter two, because it isn’t practical to self-host those.
- Video calling
XMPP. Dino & Conversations. If I need to video call with someone else, I’ll use whatever they use, usually.
- Social media / microblogging RSS reader / news
For social media, the Fediverse is my only social media. I’m using Tuba on desktop, Tusky on the phone for it. For RSS, self-hosted Miniflux. For Lemmy, the web ui on desktop, Voyager on phone.
- Music streaming / podcast app
Lollypop & Shortwave.
- Video streaming / YouTube alternative
FreeTube or yt-dlp if I need to watch youtube, PeerTube otherwise.
- Password manager
Bitwarden (via a self-hosted Vaultwarden on the server side).
- VPN / DNS / Firewall
The only VPN I use is WireGuard between my systems, but I don’t tunnel everything through it. For DNS, I’m using unbound on my VPS, which in turn dispatches to Quad9. Firewall? nftables.
- Launcher / Android OS (if you use custom ROMs)
I haven’t de-googled my phone, because my bank app refuses to work on rooted phones, and I unfortunately need that for the bank’s 2FA. No, I am not changing banks. I do use a custom launcher (Nova), though.
- App store / APKs
F-droid.
- Photo backup / gallery
I manually copy photos from the phone to my PC, and it gets backed up with the rest of the stuff. I do my backups with restic, and save a copy on my own server, and another at BorgBase. I’ll have a third copy at a third place later.
- Weather
wttr.in, mostly.
- Smart assistant (if any)
My wife. <3
- Anything else you’ve replaced?
Not strictly de-googling, but I’m using Codeberg & my own self-hosted Forgejo instead of GitHub. I replaced LibreWolf’s bookmark manager with Readeck. For push notifications on Android, I’m using a self-hosted nfty.sh.
Would love to hear about your setup — both what works well and any trade-offs you’ve had to make. Always looking for better FOSS or privacy-friendly alternatives
Oh dear. Strap in, for you’re in for a Journey! The entire configuration of both my desktop and the rest of my fleet (my VPS, my homelab server, and my Mom’s miniPC at the moment) are all free software. Based on NixOS, declarative configuration written in a literate programming manner using Org mode. There is a lot of documentation.
I wonder too, why they didn’t, because they’re happily crawling domains that never had anything but junk on them. To me, that suggests they have no idea they’re trapped. Not at crawling time at least.