The Forgeby crawler
If you found something beginning forgeby- in your access log, this page
tells you exactly what it is, what it took, and how to stop it. If you would rather just block us,
skip to how to stop us. One line stops all of it.
Who is running it
Two companies appear if you look this up, so here is the honest split rather than a single name that will not match what you find.
| The address you saw | Novalo Technologies AB holds the AWS account the
crawler runs in, so a WHOIS or abuse lookup on 51.21.44.111 leads to Amazon and then to
Novalo. Novalo is an AWS consulting partner and provides the infrastructure. |
|---|---|
| The crawler itself | Zmart Com West AB (org.nr 559019-9161), trading as Forgeby, Sweden, operates it and decides what it does. For anything about the data it holds, Forgeby is the controller under the GDPR. |
We build a business directory for technology consultancies. Write to privacy@forgeby.com about anything on this page and a human will answer. If you have already raised it with Amazon, it reaches us either way.
How to recognise it
Seven of our agents read websites, and every agent we run is named forgeby- something.
That is deliberate, so that a single robots.txt line covers all of them and you never have to keep a
list up to date. Anything not on this list is not us.
forgeby-ownsite | Reads the pages where a company describes itself and its people: an about page, a team page, a contact page. |
|---|---|
forgeby-techdetect | Reads your homepage once to record which web technologies it runs on. |
forgeby-domainfind | Checks whether a website belongs to a company we already hold a record for. One request, and it stops as soon as the answer is clear. |
forgeby-domainfill | The same check, for companies outside the Nordics. |
forgeby-chatdetect | Reads your homepage to record which chat or assistant widget it runs, and at most two further pages that your homepage links to. |
forgeby-cloudsig | Reads your homepage once to record which cloud your public assets are served from. |
forgeby-origin | The same check, looking at where your site is hosted. |
Nine more exist, for completeness, though none of them browses a website the way the seven above do.
forgeby-verify makes a single request to a page we already hold a link to, purely to
check that the link still resolves. forgeby-msdocs, forgeby-brreg,
forgeby-stats, forgeby-registry-load, forgeby-estate and
forgeby-ct, forgeby-awsdocs and forgeby-webfetch only ever
call published APIs and open datasets: Microsoft's documentation,
the Norwegian company register, the Nordic statistics offices, the Estonian, French and UK company
registers, a public address-to-network mapping table and a public certificate-transparency log.
They will not appear in your logs unless you are one of those. All of them answer to the same
User-agent: forgeby line.
| Full user agent | Mozilla/5.0 (compatible; forgeby-name/1.0;
+https://forgeby.com/crawler.html) |
|---|---|
| Source address | 51.21.44.111, and only that address. Anything else
claiming to be us is not us. |
| Requests | One at a time per site, at least 1.5 seconds apart. That 1.5 seconds is a
floor: if your robots.txt sets a longer Crawl-delay, we use yours. |
| Depth | forgeby-ownsite reads your homepage plus at most four pages
that your homepage links to; forgeby-chatdetect your homepage plus at most two.
The rest read a single page. None of them constructs a URL: we follow only links you have published.
This was not true of forgeby-chatdetect before 26 August 2026 — see
“Changes we made on 26 August 2026” below. |
| If we miss twice | Two failed requests in a row and the agent leaves your site alone for the rest of that run, whatever the reason for the failure. |
What it reads, and why
Public pages where a company describes itself and its people: an about page, a team page, a contact page. We record the business contact details a company has chosen to publish about its own staff, so that a consultancy looking for the right person to speak to can find one. Everything we hold, and the legal basis for it, is set out in the privacy notice, including how any individual can object.
The notice is published in the languages the people in it read, because a notice somebody cannot read is not a notice: English · Svenska · Norsk · Dansk · Suomi · Français · Nederlands
What they do not do
- They do not request URLs you have not linked. They do not guess paths.
- They do not touch anything behind a login, and they never attempt a CAPTCHA.
- They do not ignore
robots.txt, and they obeyCrawl-delaywhere you set one. - They do not fetch anything at all from a domain on our exclusion list, robots.txt or no robots.txt.
- They do not collect consumer or private-individual data. Business roles only.
- They do not resell your page content. We keep contact details, not your copy.
How to stop us
Any one of these works, and none of them needs to involve us.
One line in robots.txt stops every agent we run, now and in future:
User-agent: forgeby
Disallow: /Because every agent is named forgeby- something, that prefix matches all of them,
including any we add later. If you would rather stop just one, name it in full:
User-agent: forgeby-ownsite
Disallow: /We re-read robots.txt on every visit, so a change takes effect immediately, and we
treat a disallow as final rather than as something to work around. We obey Crawl-delay
where you set one, and it can only slow us down, never speed us up.
You can also block 51.21.44.111 at your edge, or mail
privacy@forgeby.com with your
domain and we will add it to a permanent exclusion list that every agent checks before it requests
anything. It covers your subdomains too. No reason required, and we do not ask for one.
Changes we made on 26 August 2026
On 26 August we were reported a second time. The cause was an agent this page never listed.
One of our own crawlers was not named forgeby- anything, so the opt-out rule
above could not stop it. It identified itself as AlloyChatDetect/1.0, carried no
link back to this page, did not request robots.txt at all, and asked for four guessed
paths — /kontakt, /kundservice, /support and
/contact — on sites that had none of them. If you blocked
User-agent: forgeby and it still arrived, that is why. It was our defect and we are
sorry.
Everything this page said was true of the crawlers it listed and untrue of that one. It ran on a
separate part of our system that the changes of 4 and 5 August never reached, and we had not
enumerated that part before describing our crawling as a whole. That is the real fault, and it is
ours. We have stopped it; added the reporting sites to a permanent exclusion list; renamed every
agent we run to forgeby- something, so the single rule above now genuinely covers all of
them; and removed the guessed paths, so it follows only links your homepage publishes, at most two of
them. The list above is no longer a description of some of what we run. It is all of it.
Changes we made on 4 and 5 August 2026
On 4 August we were reported for requesting a short list of guessed paths on sites that did not have them, which produced a burst of 404s and looked like scanning. That was our error, not a grey area. We removed path guessing entirely: the crawler now requests only URLs a site has itself linked. It also stops visiting a host after two consecutive failed requests, and the minimum gap between requests went from one second to 1.5.
On 5 August we found that the opt-out rule this page told you to write did not work.
The rule was correct; our software was not reading it correctly, and only a blanket
User-agent: * was actually stopping us. If you added a forgeby-ownsite rule
before 5 August and wondered why we kept arriving, that is why. It was our defect and we are sorry.
It is fixed, we have verified it against a real robots.txt, and the single
User-agent: forgeby line above now genuinely stops all four agents.
Two other things changed the same day. Our other agents previously identified themselves with
names that did not match this page and did not read robots.txt at all; all four now do
both. And the permanent exclusion list described above is now a real list that every agent checks
before every request, rather than a promise we would have had to keep by hand.
We are sorry for the noise in your logs.