Hemerolab

HemerolabBot

HemerolabBot is the web crawler of Hemerolab, a small research project. It keeps a private, versioned archive of public web pages and public feeds (news, sports results, economic data) and records when each version was first seen, for research and forecasting. Archived copies are not republished, sold or shared, and they are not used to train AI models.

How to recognise it

Every request carries this User-Agent (the version number may change):

HemerolabBot/0.1 (+https://hemerolab.com/bot; contact address)

Every request is also signed with Web Bot Auth (HTTP Message Signatures, RFC 9421, Ed25519). The Signature-Agent header points to https://hemerolab.com, and the public keys are published at /.well-known/http-message-signatures-directory. Our requests come from changing addresses, so please verify the signature rather than the IP address.

How it behaves

• It reads and obeys robots.txt, including Crawl-delay, and the noarchive directive.
• It waits about two seconds between requests to the same site (never less than one), with a daily limit per site.
• It prefers official feeds and APIs over crawling pages, and keeps its total volume low.
• It slows down on 429 and 503 responses, and stops when a site keeps failing.
• It never logs in, never gets around paywalls, and never solves or bypasses CAPTCHAs or other bot challenges. If a site rejects automated access, it stops visiting that site.
• It does not change its name or address to get around a block.

How to block or limit it

Add this to your robots.txt to block it completely:

User-agent: HemerolabBot
Disallow: /

Or use Disallow rules for specific paths, or Crawl-delay to slow it down. Changes are picked up within an hour. You can also write to us and we will stop visiting your site.

Questions, problems or removal requests: contact [at] hemerolab [dot] com