Crawler policy

NetaRecordBot

This project fetches public records from government portals. If you operate one of them and our crawler is causing you trouble, here is everything you need to identify and stop it.

Our user agent is exactly:

NetaRecordBot/1.0 (+https://netarecord.org/about/bot; data@netarecord.org)

• One request per second per host at most, with random jitter between 250 and 750 milliseconds.
• Where robots.txt sets a Crawl-delay, that figure wins. PRS India asks for ten seconds and gets ten seconds.
• robots.txt is re-read at the start of every run and never cached between runs, so a change takes effect immediately.
• Conditional requests with If-Modified-Since and ETag wherever the host supports them.
• A 429 or 503 causes exponential backoff. A 403 stops the job and is not retried.
• Hosts that disallow crawling are on a hard block list in the fetching library itself, so a job cannot reach them by accident.

Add this to your robots.txt and it will stop within one run:

User-agent: NetaRecordBot
Disallow: /

Or email data@netarecord.org and we will add your host to the block list directly, which takes effect on the next deploy rather than the next run.

It does not attempt to log in, does not submit forms other than the public filter forms a portal exposes for browsing, does not follow links behind an authentication wall, and does not collect personal data beyond what appears in the public record it is reading. It does not fetch phone numbers or home addresses, and it stores constituency-level location at most.