About TroveBot

Trove is a federated search engine for the small web, seeded by fediverse attention and ranked by domain coherence rather than commercial authority. TroveBot is its crawler — it discovers and indexes URLs shared by people on the fediverse, then follows the citations they make to find more of the small web.

Identity

TroveBot identifies itself with the following User-Agent string:

TroveBot/0.1 (+https://trove.lvl0.xyz/bot)

Crawling behavior

• Respects robots.txt rules under User-agent: TroveBot (and the wildcard User-agent: * as a fallback).
• Polite per-domain rate limit — at most a few requests per minute per host.
• Follows up to 5 redirects per URL.
• Fetches HTML only. PDFs, images, and other binary content are recorded as discovered but never re-fetched.
• Does not execute JavaScript, does not crawl behind authentication, does not crawl URLs containing user credentials.

Opt out

Block TroveBot entirely by adding the following to your site's robots.txt:

User-agent: TroveBot
Disallow: /

Or block specific paths:

User-agent: TroveBot
Disallow: /private/
Disallow: /admin/

Contact & source

• Issues, questions, abuse reports: forge.lvl0.xyz/lvl0/trove/issues
• Source code: forge.lvl0.xyz/lvl0/trove