About the PageCheck checker
PageCheck checks websites for speed, search visibility, security and errors. When someone asks for a check of an address, our checker visits that page once, the way a visitor with a browser would, and writes a report for the person who asked. If you found us in your server logs, this page explains what we requested and why.
It is not a crawler
The checker only visits the address a person entered. It does not follow links across your site, build an index, or come back on its own. Each visit was asked for by someone, usually the site's owner or the agency that builds it.
What it requests
- The page itself, first as the server sends it, then opened in a browser with its images, styles and scripts.
- In that browser the checker scrolls the page and moves the mouse, as a visitor does. At the very end it accepts the cookie banner ("Accept all"), to see which statistics and advertising tools start only after consent. That choice stays in the checker's browser, which is discarded after the check; your statistics and your cookie banner's consent log may show the visit.
- robots.txt, the sitemap it names, the site's icons, and one address that does not exist, to see how the site answers a missing page.
- The site's four root addresses (with and without www, over http:// and https://), following each one's redirects to the page it ends on.
- A few standard files of the platform the site runs on, to tell whether its software is up to date (for WordPress, for example, core style and script files).
- For online shops selling in Latvia, whether the shop is listed on the price comparison sites Salidzini.lv (one question about the shop, through a real browser) and Kurpirkt.lv (its public list of shops, read once a day). These requests go to those sites, not to yours.
- A handful of addresses that should not be public, such as log files or admin tools, to warn the owner if they are.
- The domain's public DNS records, for the e-mail and domain checks.
A check takes a minute or two and makes a few dozen requests, most of them the page's own files. Every check is passive: the checker never logs in, never submits forms, never tries passwords and never measures how much load a site can take.
How to recognise it
The checker presents itself as an up-to-date Chrome and adds its own name to the end of the user agent:
Mozilla/5.0 (…) Chrome/… Safari/537.36 PageCheck/1.0 (+https://pagecheck.dev/checker/)
Two exceptions: the speed measurement runs Google's open-source Lighthouse, which uses its own standard identity so that results can be compared with Google's; and when a site sits behind a bot check that the checker has passed, it keeps the identity of the browser that passed it.
Where it comes from
All requests come from these IP addresses (IPv4 only):
65.21.12.114164.68.121.244
When the first address cannot reach a page (no connection, or a bot check it cannot pass), the checker asks for that page once more from the second one, and the report says so.
When we add capacity, the new address is listed here before it sends a single request.
robots.txt
Because every check is asked for by a person and covers one address, the checker reads robots.txt to check it, not as an instruction to stay away, in the same way as Google's PageSpeed Insights. If you want your site never to be checked, use the opt-out below.
One feature reads more than one address: "Find pages on a site" in the bulk check, which looks for pages to check in the sitemap and by following links (at most 100 pages). It follows robots.txt and leaves out the addresses it disallows for PageCheck (or for all crawlers), unless the user confirms that the site is their own or their client's.
Our research crawler
We also run a research crawler that measures how reliable our checks are. About every two weeks it opens the home page of publicly listed websites, sometimes also /favicon.ico, from research-1.pagecheck.dev (157.173.117.113), with the user agent "PageCheck-Research/1.0".
Unlike a check, nobody asked for these visits, so the crawler reads robots.txt first and follows it: the rules for PageCheck-Research, else those for PageCheck, else those for all crawlers. It reads nothing else, and it never visits a site that has opted out (below).
Letting it in
If a firewall or bot protection stops the checker, the report says which checks could not run; it never counts them as passed. To let it in, allow the IP addresses above, or requests whose user agent contains "PageCheck".
Keeping your site out
Write to hello@pagecheck.dev from an address at your domain (or tell us how we can confirm that you manage the site) and name the domain. We add it to our exclusion list, usually within one working day; from then on nobody can check it with PageCheck, its subdomains included.
Questions
If the checker caused you any trouble, or you have a question about a request in your logs, write to hello@pagecheck.dev with the time and address of the request.