Skip to content
Octolyze
Start an Audit
Our Crawler

OctolyzeBot

Octolyze crawls a site because somebody asked it to — an owner auditing a property they control. This page tells you how to recognise our crawler, how to check it is really us, and exactly what to put in robots.txt to allow it, slow it down, or refuse it.

Identity

Our Two Crawlers.

They do different jobs, so they answer to different names. You can permit one and refuse the other.

Every request we make comes from a single address — 77.42.19.28 — and carries one of the user-agent strings below. Anything else claiming to be us is not us.

OctolyzeAudit — an audit somebody requested

This is the crawler that runs when a customer audits their own site. A project cannot be created in Octolyze without proving control of the domain first, so this crawler only visits sites whose owner asked for the report.

  • User-agent — Mozilla/5.0 (compatible; OctolyzeAudit/1.0; +https://octolyze.com/bot)
  • robots.txt token — OctolyzeAudit

OctolyzeBot — everything else we fetch

When a report checks whether the links on a page still work, we request those external URLs to read their status code. That means we sometimes touch a site nobody asked us to audit — a page that another site links to. Those requests use this name, and it is the one to refuse if you would rather we did not.

  • User-agent — Mozilla/5.0 (compatible; OctolyzeBot/1.0; +https://octolyze.com/bot)
  • robots.txt token — OctolyzeBot

The older name

Before the split, both used the single token Octolyze. If your robots.txt names it, we still obey that rule — a rename is not a way back into a site that already refused us. You do not need to change anything.

Verification

Checking It Is Really Us.

Anyone can put a name in a user-agent string. Ours is checkable.

Every Octolyze request comes from one address:

77.42.19.28

The same address is published as a machine-readable list at octolyze.com/bot-ips.txt, so a firewall or a bot filter can read it directly instead of a person copying it across. That file is also what Cloudflare fetches to verify us.

If a request claims to be OctolyzeBot and did not come from that address, it is not us. Please tell us — we want to know as much as you do.

Or check it without trusting this page

The address carries our own name in reverse DNS, and the name points back at the address. That round trip is the check search engines publish for their own crawlers, and you can run it in two commands without taking our word for anything:

$ dig +short -x 77.42.19.28
crawl01.octolyze.com.

$ dig +short crawl01.octolyze.com
77.42.19.28

If the first answer is not a name ending in octolyze.com, or the second answer is not the address you were visited from, the request was not ours.

It is called crawl01 because that is what it is: the first machine. If we ever run a second, it gets its own name, its own record and its own line in the list above — not a shared alias that hides which one visited you.

We do not rotate addresses and we do not crawl through proxy pools. That is deliberate: it makes us easy to allow, easy to rate-limit and easy to block. A crawler that is hard to identify is asking you to trust it without giving you any way to check.

Control

Telling Us What to Do.

Put any of these in the robots.txt at the root of your site. We read it fresh at the start of every crawl, never from a cache, so a change applies to the next one.

Allow us

This is the default. If your robots.txt says nothing about us, we follow whatever your User-agent: * rules permit.

Allow your own audit, refuse the rest

User-agent: OctolyzeAudit
Disallow:

User-agent: OctolyzeBot
Disallow: /

Slow us down

Seconds between requests, honoured for the whole crawl.

User-agent: OctolyzeAudit
Crawl-delay: 5

We cap it at 10 seconds. A larger number is read as 10 — below that rate a crawl would not finish, and an audit that never finishes helps nobody. If you need us slower than that, block us instead and we will stay away entirely.

Keep us out of one area

User-agent: OctolyzeAudit
Disallow: /admin/
Disallow: /checkout/

Refuse us completely

User-agent: OctolyzeAudit
Disallow: /

User-agent: OctolyzeBot
Disallow: /
When It Is Not robots.txt

If Your Site Answers 403.

A refused request and a disallowed path are two different things, fixed in two different places. Editing robots.txt does nothing about a 403.

robots.txt is a request we obey. A 403 or 401 is your edge refusing us before a page is served — at that point what your robots.txt says never gets read, because the request never reaches your site.

How to tell which one it is

If your Octolyze report says pages were refused, it is this. If it says paths were blocked, that is robots.txt and the section above is where to change it. Your own server logs settle it too: a robots.txt rule leaves no request at all, while a firewall block usually leaves an entry in your CDN or security plugin, not in the site's access log.

What to allow

Either the name or the address — one is enough, both is fine.

  • User-agent contains OctolyzeAudit (or OctolyzeBot)
  • IP address 77.42.19.28 — the only one we crawl from

Cloudflare

A WAF custom rule, action Skip — or add the address under Security → Tools → IP Access Rules with Allow.

(http.user_agent contains "OctolyzeAudit") or (ip.src eq 77.42.19.28)

If your Cloudflare plan has Bot Fight Mode on, it challenges unverified bots by default; the rule above runs before it.

WordPress and shared hosting

Wordfence (Blocking → Allowlisted IPs), Sucuri, All In One WP Security, and the "bad bot" blocklists some themes ship with all keep their own lists — they do not read robots.txt. On cPanel or Plesk the same block often comes from ModSecurity, and your host can allow the address in one line.

Akamai, DataDome and similar

These are bot-management products with their own directories. We have applied to be listed in them, which is the fix that does not need you to do anything — until it lands, an allowlist entry on your side is the way through.

Why we do not work around it

We could rotate addresses, or send a browser's user-agent, and most of these blocks would stop happening. We do not. A crawler that hides to get in is a crawler you cannot rate-limit, cannot audit and cannot refuse — and the whole point of this page is that you can do all three. One address, one name, published.

Commitments

What We Do, and What We Will Not Do.

  • We read robots.txt before anything else, on every crawl, and obey the group written for our name. If none names us, we follow User-agent: *.
  • We back off when told to. A 429 or 503 slows us down for that host, and we honour Retry-After when you send it.
  • We stay inside the site being audited. External links are requested for their status code only — one request, and we never crawl onward from them.
  • We do not try to get around a block. No proxy rotation, no changing address, no pretending to be a browser. If you refuse us, the crawl reports that it was refused, and the person who ran it sees exactly that.
  • We do not log in. We never submit forms, never use credentials, and never touch anything behind a sign-in.
For Site Owners

Why You Might Allow It.

If you own the site, this crawler is how you get the report — broken links, redirect chains, missing titles, slow pages, pages a search engine cannot index. Blocking it does not protect the site from anything; it only means the person who asked for the audit gets an empty one.

If you do not own the site and we arrived through somebody else's link check, refusing OctolyzeBot costs you nothing at all.

Traffic you did not expect, a rule that is not working, or a request to be excluded permanently — write to [email protected] or use the contact form. A person reads it.