# The crawler > What it reads, and how to keep it out. > Source: https://forgeby.com/crawler.html > Every figure on this site is counted nightly and printed with the date it was counted. A number > here was true when it was counted, not when you asked. # The Forgeby crawler If you found something beginning forgeby- in your access log, this page tells you exactly what it is, what it took, and how to stop it. If you would rather just block us, skip to how to stop us . One line stops all of it. ## Who is running it Two companies appear if you look this up, so here is the honest split rather than a single name that will not match what you find. We build a business directory for technology consultancies. Write to privacy@forgeby.com about anything on this page and a human will answer. If you have already raised it with Amazon, it reaches us either way. The address you saw · Novalo Technologies AB holds the AWS account the crawler runs in, so a WHOIS or abuse lookup on 51.21.44.111 leads to Amazon and then to Novalo. Novalo is an AWS consulting partner and provides the infrastructure. · The crawler itself · Zmart Com West AB (org.nr 559019-9161), trading as Forgeby , Sweden, operates it and decides what it does. For anything about the data it holds, Forgeby is the controller under the GDPR. ## How to recognise it Four of our agents read websites, and every agent we run is named forgeby- something. That is deliberate, so that a single robots.txt line covers all of them and you never have to keep a list up to date. Anything not on this list is not us. Seven more exist, for completeness, though none of them browses a website the way the four above do. forgeby-verify makes a single request to a page we already hold a link to, purely to check that the link still resolves. forgeby-msdocs , forgeby-brreg , forgeby-stats , forgeby-registry-load , forgeby-estate and forgeby-ct only ever call published APIs and open datasets: Microsoft's documentation, the Norwegian company register, the Nordic statistics offices, the Estonian, French and UK company registers, a public address-to-network mapping table and a public certificate-transparency log. They will not appear in your logs unless you are one of those. All of them answer to the same User-agent: forgeby line. forgeby-ownsite · Reads the pages where a company describes itself and its people: an about page, a team page, a contact page. · forgeby-techdetect · Reads your homepage once to record which web technologies it runs on. · forgeby-domainfind · Checks whether a website belongs to a company we already hold a record for. One request, and it stops as soon as the answer is clear. · forgeby-domainfill · The same check, for companies outside the Nordics. ## What it reads, and why Full user agent · Mozilla/5.0 (compatible; forgeby- name /1.0; +https://forgeby.com/crawler.html) · Source address · 51.21.44.111 , and only that address. Anything else claiming to be us is not us. · Requests · One at a time per site, at least 1.5 seconds apart, from every agent. Never concurrent against the same host. · Depth · forgeby-ownsite reads your homepage plus at most four pages that your homepage links to . The other three read one page. None of them ever visits a URL you have not linked. · If we miss twice · Two failed requests in a row and the agent leaves your site alone for the rest of that run, whatever the reason for the failure. Public pages where a company describes itself and its people: an about page, a team page, a contact page. We record the business contact details a company has chosen to publish about its own staff, so that a consultancy looking for the right person to speak to can find one. Everything we hold, and the legal basis for it, is set out in the privacy notice , including how any individual can object. The notice is published in the languages the people in it read, because a notice somebody cannot read is not a notice: English · Svenska · Norsk · Dansk · Suomi · Français ## What they do not do - They do not request URLs you have not linked. They do not guess paths. - They do not touch anything behind a login, and they never attempt a CAPTCHA. - They do not ignore robots.txt , and they obey Crawl-delay where you set one. - They do not fetch anything at all from a domain on our exclusion list, robots.txt or no robots.txt. - They do not collect consumer or private-individual data. Business roles only. - They do not resell your page content. We keep contact details, not your copy. ## How to stop us Any one of these works, and none of them needs to involve us. One line in robots.txt stops every agent we run, now and in future: ``` User-agent: forgeby Disallow: / ``` Because every agent is named forgeby- something, that prefix matches all of them, including any we add later. If you would rather stop just one, name it in full: ``` User-agent: forgeby-ownsite Disallow: / ``` We re-read robots.txt on every visit, so a change takes effect immediately, and we treat a disallow as final rather than as something to work around. We obey Crawl-delay where you set one, and it can only slow us down, never speed us up. You can also block 51.21.44.111 at your edge, or mail privacy@forgeby.com with your domain and we will add it to a permanent exclusion list that every agent checks before it requests anything. It covers your subdomains too. No reason required, and we do not ask for one. ## Changes we made on 4 and 5 August 2026 On 4 August we were reported for requesting a short list of guessed paths on sites that did not have them, which produced a burst of 404s and looked like scanning. That was our error, not a grey area. We removed path guessing entirely: the crawler now requests only URLs a site has itself linked. It also stops visiting a host after two consecutive failed requests, and the minimum gap between requests went from one second to 1.5. On 5 August we found that the opt-out rule this page told you to write did not work. The rule was correct; our software was not reading it correctly, and only a blanket User-agent: * was actually stopping us. If you added a forgeby-ownsite rule before 5 August and wondered why we kept arriving, that is why. It was our defect and we are sorry. It is fixed, we have verified it against a real robots.txt , and the single User-agent: forgeby line above now genuinely stops all four agents. Two other things changed the same day. Our other agents previously identified themselves with names that did not match this page and did not read robots.txt at all; all four now do both. And the permanent exclusion list described above is now a real list that every agent checks before every request, rather than a promise we would have had to keep by hand. We are sorry for the noise in your logs.