Website crawlability depends on whether crawlers can reach a site, interpret its HTML, and understand which URL should represent each page. The latest HTTP Archive Web Almanac SEO measurements show that 84.9% of desktop and mobile sites returned HTTP 200 for robots.txt in 2025, while canonical tags appeared on 68% of desktop pages and 67% of mobile pages. These are technical observations, not a direct census of Google indexing, but they show where crawl and indexability signals are commonly present or absent.
Contents
- Robots.txt availability and file size
- Crawler targeting and emerging bot controls
- Robots directives and indexability controls
- HTML parsing and crawlable metadata
- Canonicalization and duplicate URL consolidation
- Delivery, mobile, and performance conditions
Robots.txt availability and file size
Robots.txt is a small but important entry point for automated crawlers. In the 2025 Web Almanac SEO data, 84.9% of desktop sites and 84.9% of mobile sites returned HTTP 200 for robots.txt. HTTP 404 responses were recorded for 13.3% of desktop sites and 13.2% of mobile sites. Timeouts affected 1.0% of desktop sites and 1.1% of mobile sites, while HTTP 403 appeared for 0.5% of both groups. HTTP 500 was recorded for 0.1% of desktop and mobile requests. Source: SEO | 2025 | The Web Almanac by HTTP Archive
The response distribution suggests that availability is usually straightforward, but a meaningful minority of sites expose a missing, blocked, slow, or server-error response. The measurements describe the robots.txt request itself; they do not establish whether a particular directive was correct or whether a page was ultimately indexed.
File size was generally limited. Robots.txt files of 0–100 bytes covered 97.5% of desktop sites and 97.6% of mobile sites in 2025. Files larger than 500 bytes occurred on 0.1% of both desktop and mobile sites. Source: SEO | 2025 | The Web Almanac by HTTP Archive
For context, the 2024 Web Almanac reported HTTP 200 responses from 83.5% of desktop robots.txt files and 83.9% of mobile robots.txt files. The 2024 and 2025 observations come from separate editions and should be read as period-specific measurements rather than a guaranteed trend for every site. Source: SEO | 2024 | The Web Almanac by HTTP Archive
Crawler targeting and emerging bot controls
Robots.txt files increasingly name crawlers beyond traditional search bots. In the 2025 dataset, Amazonbot was targeted by 3.3% of desktop sites and 3.0% of mobile sites. Google-Extended was targeted by 3.4% of desktop sites and 3.0% of mobile sites. PerplexityBot was targeted by 2.8% of desktop sites and 2.7% of mobile sites.
Other named agents appeared at similar levels. ChatGPT-User was targeted by 2.8% of desktop sites and 2.5% of mobile sites. Facebookbot was targeted by 2.9% of desktop sites and 2.5% of mobile sites, while Meta-ExternalAgent was targeted by 2.8% of desktop sites and 2.5% of mobile sites. These figures identify the presence of targeting rules in robots.txt; they do not measure how often a bot actually crawled a site or whether a rule was obeyed. Source: SEO | 2025 | The Web Almanac by HTTP Archive
Older observations preserve the changing vocabulary of crawler controls. GPTBot appeared in 2.7% of mobile robots.txt crawls in 2024. In the 2022 Web Almanac data, Googlebot was named in 3.3% of desktop robots.txt files and Bingbot in 2.5%. Because these figures come from different editions and agents, they are historical snapshots rather than directly comparable market shares. Source: SEO | 2024 | The Web Almanac by HTTP Archive; Source: SEO | 2022 | The Web Almanac by HTTP Archive
Robots directives and indexability controls
Meta robots directives appeared on 47.0% of desktop pages and 47.9% of mobile pages in 2025. X-Robots-Tag directives were much less common, appearing on 0.6% of desktop pages and 0.7% of mobile pages. Both meta robots and X-Robots-Tag appeared together on 0.4% of desktop and mobile pages. Source: SEO | 2025 | The Web Almanac by HTTP Archive
The Web Almanac also found a difference between page types: inner pages were 50% likely to have a robots directive, compared with 46% for home pages. This matters when auditing a site because a clean homepage does not describe every template, product page, archive, or article URL.
Among named directives, the follow directive appeared on 64.0% of desktop pages and 60.5% of mobile pages. The index directive appeared on 69.0% of desktop pages and 59.3% of mobile pages. The nofollow directive appeared on 2.4% of desktop pages and 2.8% of mobile pages, while noindex appeared on 3.5% of desktop pages and 2.4% of mobile pages.
Less common controls included max-image-preview on 5.0% of desktop pages and 2.8% of mobile pages, and max-snippet on 2.2% of desktop pages and 1.1% of mobile pages. Indexifembedded appeared within iframe content on 88.9% of desktop sites and 87.7% of mobile sites. The measurements describe observed directive usage, not the success of a directive in producing or preventing an index result. Source: SEO | 2025 | The Web Almanac by HTTP Archive
HTML parsing and crawlable metadata
HTML structure can affect how reliably crawlers parse head metadata and page content. Invalid HTML elements in the head occurred on 10.1% of desktop pages and 10.3% of mobile pages in 2025. An invalid img element appeared in the head on 27.2% of desktop pages and 23.0% of mobile pages. Invalid div elements appeared in the head on 11.5% of desktop pages and 10.6% of mobile pages, while invalid a elements appeared on 5.2% of desktop pages and 4.9% of mobile pages. Source: SEO | 2025 | The Web Almanac by HTTP Archive
Basic metadata was more prevalent than complete heading coverage. Title tags were present on 98.6% of desktop pages and 98.5% of mobile pages. Meta descriptions were present on 67.7% of desktop pages and 67.2% of mobile pages. H1 elements were present on 71% of desktop pages and 70% of mobile pages, while H2 elements were present on 72% of desktop pages and 71% of mobile pages.
Presence does not always mean useful content. Non-empty H1 elements appeared on 66% of both desktop and mobile pages. Non-empty H2 elements appeared on 71% of desktop pages and 70% of mobile pages. The gap between an element existing and containing text is therefore visible in the same measurement period. Source: SEO | 2025 | The Web Almanac by HTTP Archive
| Crawlable page signal | Desktop | Mobile |
|---|---|---|
| Title tag present | 98.6% | 98.5% |
| Meta description present | 67.7% | 67.2% |
| H1 present | 71% | 70% |
| Non-empty H1 | 66% | 66% |
| H2 present | 72% | 71% |
Canonicalization and duplicate URL consolidation
Canonical tags appeared on 68% of desktop pages and 67% of mobile pages in 2025. Pages canonicalized to a different URL accounted for 7% of desktop pages and 9% of mobile pages. In the 2024 edition, canonical tags appeared on 65% of desktop and mobile pages, while canonicalized pages represented 6% of desktop pages and 8% of mobile pages. These are edition-specific observations, so they should not be treated as a definitive year-over-year change for all websites. Source: SEO | 2025 | The Web Almanac by HTTP Archive; Source: SEO | 2024 | The Web Almanac by HTTP Archive
The 2025 results also distinguish how a canonical is observed. Raw HTML canonical tags appeared on 64.40% of desktop pages and 64.27% of mobile pages. Rendered canonical tags appeared on 66.01% of desktop pages and 66.11% of mobile pages. Canonical tags can therefore be detected at slightly different rates depending on whether the inspection uses the raw response or the rendered page.
Some sites showed conflicts or changes between representations. Canonical mismatches were detected on 0.72% of desktop sites and 0.71% of mobile sites. Rendered-change canonical issues affected 2.71% of desktop sites and 3.02% of mobile sites, while HTTP-header canonical changes affected 0.25% of desktop sites and 0.21% of mobile sites. Source: SEO | 2025 | The Web Almanac by HTTP Archive
Delivery, mobile, and performance conditions
Secure delivery was common: HTTPS was used by 91.7% of desktop pages and 91.5% of mobile pages in 2025. The meta viewport tag appeared on 93.1% of desktop pages and 95.2% of mobile pages. These figures describe page-level implementation signals, not whether every resource or redirect in a crawl path was secure or mobile-friendly. Source: SEO | 2025 | The Web Almanac by HTTP Archive
The Vary header was absent on 99.3% of client-rendered pages and 99.8% of server-rendered pages. It was present on 0.2% of client-rendered pages and 0.7% of server-rendered pages. The two measurements do not sum to 100% because the source reports separate observed categories and rounding may apply. Source: SEO | 2025 | The Web Almanac by HTTP Archive
Mobile presentation audits were positive for legible font size on 92.07% of mobile home pages and 92.93% of mobile inner pages in 2025. Performance results were more mixed in June 2025:
| June 2025 Core Web Vitals result | Desktop | Mobile |
|---|---|---|
| Good LCP | 74% | 62% |
| Good CLS | 72% | 81% |
| Good INP | 97% | 77% |
| Good overall Core Web Vitals | 56% | 48% |
The June 2025 figures show why crawlability and indexing audits should include delivery conditions without treating performance as an indexing guarantee. A page can expose robots directives, metadata, and canonicals while still producing a different experience across desktop and mobile measurements. Source: SEO | 2025 | The Web Almanac by HTTP Archive