Google Search Console tells you how Googlebot behaves in aggregate. Your server's access log tells you exactly what it did: every URL it requested, when, what your server answered, and how long that took. Compare the two and you find problems no dashboard shows, such as crawl time spent on duplicate URLs, pages Google never requests, and a stream of fake Googlebots probing your site.
This article builds a small, dependency-free Python audit for nginx logs. You will configure nginx to write a dedicated crawler log, verify that a visitor claiming to be Googlebot really is one, and produce a report on status codes, wasted requests and sitemap coverage. Everything here was run against sample and real nginx 1.24 logs.
This applies when you control the web server and can read its logs. Hosted platforms that do not expose access logs, such as Blogger, cannot use it. There, the Crawl Stats report in Search Console is your closest equivalent.
1. What logs can and cannot tell you
Logs can show:
- Which URLs Googlebot requests, how often, and in which sections of your site.
- The status code you returned for each request: successes, redirects, missing pages, server errors.
- How long your server took to respond to crawler requests.
- Which sitemap URLs were never requested in the period you looked at.
- Spoofed crawlers that use Googlebot's name from addresses that do not belong to Google.
Logs cannot show:
- Whether a crawled page was indexed or how it ranks. A request is not an index entry. Use Search Console's page indexing report for that.
- Requests that never reached your server. If a CDN answers from its cache, your origin log has no line for it. For full coverage, use the CDN's logs or run the audit on edge logs.
- Anything about sites where you have no log access.
A note on crawl budget. Google's guidance says active crawl budget management matters mainly for very large sites or sites whose content changes very fast. For most sites the practical value of logs is different: finding errors, wasted URLs and orphaned pages. Do not read every wasted request as a budget crisis.
2. Configure nginx
Two changes: add the response time to the log format, and write crawler requests to their own file. A separate file keeps the audit small and lets you keep ordinary visitor logs for a shorter time, which matters because IP addresses are personal data in many jurisdictions.
Put this inside the http block:
log_format crawl '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time';
map $http_user_agent $is_crawler {
default 0;
~*(googlebot|google-inspectiontool|adsbot-google|googleother|storebot-google) 1;
}
access_log /var/log/nginx/access.log crawl;
access_log /var/log/nginx/crawlers.log crawl if=$is_crawler;The log format keeps nginx's standard combined layout and appends the request time in seconds. The second access_log line writes only requests whose user agent matches, and the if condition is what limits it. Check and reload:
sudo nginx -t && sudo nginx -s reloadIf a proxy or tunnel sits in front of nginx, the address nginx logs is the proxy's, not the visitor's, and every Googlebot check would fail. Tell nginx which sources to trust and which header carries the real address. For a tunnel or proxy on the same host sending the Cloudflare header:
set_real_ip_from 127.0.0.1;
real_ip_header CF-Connecting-IP;Only list addresses you actually control in set_real_ip_from. If you trust a header from anyone, any visitor can write whatever address they like into your logs, which defeats the whole audit.
3. Verify who is really Googlebot
The user agent string is only text, and anyone can send it. Google documents two ways to verify a crawler:
- Reverse and forward DNS. Look up the hostname for the IP address. It must end in googlebot.com, google.com or googleusercontent.com. Then look up that hostname and confirm it resolves back to the same IP.
- Published IP ranges. Match the address against the JSON files Google publishes, which list networks in CIDR form.
The second method is faster for logs, because you check each distinct address once with no network calls. The file for the common crawlers, which include Googlebot, is now called common-crawlers.json. Many older tutorials still refer to googlebot.json. A second file, special-crawlers.json, covers crawlers such as AdsBot. The format is a list of prefixes, each with an ipv4Prefix or ipv6Prefix:
{
"creationTime": "2026-10-19T14:46:25.000000",
"prefixes": [
{ "ipv6Prefix": "2001:4860:4801:10::/64" },
{ "ipv4Prefix": "66.249.64.0/27" }
]
}The ranges change over time, so refresh them regularly. This script downloads both files safely: the f flag makes curl fail on an HTTP error, and the file is replaced only after a complete download.
#!/bin/sh
set -eu
base=https://developers.google.com/static/crawling/ipranges
for name in common-crawlers special-crawlers; do
curl -fsS -o "/opt/audit/$name.json.tmp" "$base/$name.json"
mv "/opt/audit/$name.json.tmp" "/opt/audit/$name.json"
doneThe audit script below uses the ranges by default. Add the rdns flag and any address missing from the files is also checked with Google's DNS method, which is slower but helps when your downloaded ranges are stale.
4. The audit script
Save this as crawl_audit.py. It needs only Python 3.8 or newer and reads plain or gzip-compressed logs:
import argparse
import gzip
import ipaddress
import json
import re
import socket
import xml.etree.ElementTree as ET
from collections import Counter, defaultdict
from datetime import datetime
from urllib.parse import urlsplit
LINE = re.compile(
r'(?P<ip>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] '
r'"(?P<method>[A-Z]+) (?P<path>\S+) (?P<proto>[^"]*)" '
r'(?P<status>\d{3}) (?P<bytes>\d+|-) '
r'"(?P<referer>[^"]*)" "(?P<ua>[^"]*)"'
r'(?: (?P<rt>\d+\.\d+))?')
GOOGLE_SUFFIXES = ('.googlebot.com', '.google.com', '.googleusercontent.com')
DEFAULT_WASTE = [r'\?', r'^/feeds/', r'^/search', r'^/wp-json/', r'^/cart', r'^/cdn-cgi/']
def open_log(path):
return gzip.open(path, 'rt', encoding='utf-8', errors='replace') if path.endswith('.gz') \
else open(path, encoding='utf-8', errors='replace')
def parse_lines(paths):
bad = 0
for path in paths:
with open_log(path) as f:
for raw in f:
m = LINE.match(raw)
if not m:
bad += 1
continue
d = m.groupdict()
d['day'] = datetime.strptime(d['time'], '%d/%b/%Y:%H:%M:%S %z').strftime('%Y-%m-%d')
d['status'] = int(d['status'])
d['rt'] = float(d['rt']) if d['rt'] else None
yield d
parse_lines.bad = bad
def google_family(ua):
u = ua.lower()
for token, name in (('googlebot-image', 'Googlebot-Image'), ('googlebot-video', 'Googlebot-Video'),
('googlebot-news', 'Googlebot-News'), ('google-inspectiontool', 'Google-InspectionTool'),
('adsbot-google', 'AdsBot-Google'), ('storebot-google', 'Storebot-Google'),
('googleother', 'GoogleOther')):
if token in u:
return name
if 'googlebot' in u:
return 'Googlebot smartphone' if 'mobile' in u else 'Googlebot desktop'
return None
def load_ranges(*paths):
nets = []
for p in paths:
with open(p, encoding='utf-8') as f:
for item in json.load(f)['prefixes']:
cidr = item.get('ipv4Prefix') or item.get('ipv6Prefix')
if cidr:
nets.append(ipaddress.ip_network(cidr))
return nets
def in_ranges(ip, nets):
try:
addr = ipaddress.ip_address(ip)
except ValueError:
return False
return any(addr in n for n in nets)
def verify_rdns(ip, reverse=None, forward=None):
# Google's documented method: reverse lookup, check the domain, then forward lookup must return the same IP.
reverse = reverse or (lambda a: socket.gethostbyaddr(a)[0])
forward = forward or (lambda h: {i[4][0] for i in socket.getaddrinfo(h, None)})
try:
host = reverse(ip).rstrip('.').lower()
if not host.endswith(GOOGLE_SUFFIXES):
return False
want = ipaddress.ip_address(ip)
return any(ipaddress.ip_address(a) == want for a in forward(host))
except (OSError, ValueError):
return False
def pct(values, p):
s = sorted(values)
return s[min(len(s) - 1, int(len(s) * p))] if s else 0.0
def sitemap_paths(paths):
out = set()
for p in paths:
for loc in ET.parse(p).getroot().iter('{http://www.sitemaps.org/schemas/sitemap/0.9}loc'):
u = urlsplit(loc.text.strip())
out.add(u.path + ('?' + u.query if u.query else ''))
return out
def audit(log_paths, ranges, use_rdns=False, waste_patterns=None, sitemaps=(), top=10):
waste_re = re.compile('|'.join(waste_patterns or DEFAULT_WASTE))
claimed, verdict = [], {}
total = 0
for rec in parse_lines(log_paths):
total += 1
fam = google_family(rec['ua'])
if fam:
rec['family'] = fam
claimed.append(rec)
for ip in {r['ip'] for r in claimed}: # verify each distinct IP once
ok = in_ranges(ip, ranges)
if not ok and use_rdns:
ok = verify_rdns(ip)
verdict[ip] = ok
real = [r for r in claimed if verdict[r['ip']]]
fake = [r for r in claimed if not verdict[r['ip']]]
print(f'lines parsed: {total} (unparsed: {parse_lines.bad})')
print(f'requests claiming a Google user agent: {len(claimed)}')
print(f' verified: {len(real)} unverified (spoofed or unlisted): {len(fake)}')
if claimed and not real:
print(' WARNING: nothing verified. Check that the log shows real client IPs (proxy or CDN in front?)')
if fake:
by_ip = Counter((r['ip'], r['family']) for r in fake).most_common(top)
print(' top unverified sources:', ', '.join(f'{ip} ({fam}) x{n}' for (ip, fam), n in by_ip))
if not real:
return
print('\nverified crawlers:')
for fam, n in Counter(r['family'] for r in real).most_common():
print(f' {fam:<24}{n:>8}')
print('\nrequests per day (verified):')
for day, n in sorted(Counter(r['day'] for r in real).items()):
print(f' {day} {n:>7}')
print('\nstatus classes (verified):')
classes = Counter(f"{r['status'] // 100}xx" for r in real)
for c, n in sorted(classes.items()):
print(f' {c} {n:>7} {100 * n / len(real):5.1f}%')
for label, test in (('404 URLs', lambda s: s == 404), ('5xx URLs', lambda s: s >= 500),
('redirected URLs (301/302/307/308)', lambda s: s in (301, 302, 307, 308))):
hits = Counter(r['path'] for r in real if test(r['status'])).most_common(top)
if hits:
print(f'\ntop {label}:')
for path, n in hits:
print(f' {n:>6} {path}')
wasted = [r for r in real if waste_re.search(r['path'])]
print(f'\ncrawl waste: {len(wasted)} of {len(real)} verified requests ({100 * len(wasted) / len(real):.1f}%) '
f'match the waste patterns')
for path, n in Counter(r['path'] for r in wasted).most_common(top):
print(f' {n:>6} {path}')
print('\nmost crawled URLs:')
for path, n in Counter(r['path'] for r in real).most_common(top):
print(f' {n:>6} {path}')
sections = Counter('/' + r['path'].lstrip('/').split('/')[0].split('?')[0] for r in real)
print('\nrequests per top-level section:')
for sec, n in sections.most_common(top):
print(f' {n:>6} {sec}')
times = [r['rt'] for r in real if r['rt'] is not None]
if times:
print(f'\nresponse time for verified crawls: p50 {pct(times, 0.5) * 1000:.0f} ms, '
f'p95 {pct(times, 0.95) * 1000:.0f} ms, max {max(times) * 1000:.0f} ms')
if sitemaps:
wanted = sitemap_paths(sitemaps)
seen = {r['path'] for r in real}
missing = sorted(wanted - seen)
print(f'\nsitemap URLs: {len(wanted)}; not requested by a verified crawler in this window: {len(missing)}')
for path in missing[:top]:
print(f' {path}')
if __name__ == '__main__':
ap = argparse.ArgumentParser(description='Audit Googlebot crawling from nginx access logs')
ap.add_argument('logs', nargs='+', help='access log files (.gz supported)')
ap.add_argument('--ranges', nargs='+', required=True, help='Google IP range JSON files')
ap.add_argument('--rdns', action='store_true', help='also verify unlisted IPs by reverse+forward DNS')
ap.add_argument('--waste', action='append', help='extra regex for wasteful URLs (repeatable)')
ap.add_argument('--sitemap', nargs='*', default=[], help='local sitemap XML files')
args = ap.parse_args()
audit(args.logs, load_ranges(*args.ranges), args.rdns,
DEFAULT_WASTE + (args.waste or []), args.sitemap)Run it over at least 30 days of logs, because crawling is bursty and a single week can mislead:
python3 crawl_audit.py /var/log/nginx/crawlers.log /var/log/nginx/crawlers.log.1 \
--ranges /opt/audit/common-crawlers.json /opt/audit/special-crawlers.json \
--sitemap /var/www/sitemap.xmlBefore you trust a report, run a smoke test. Request your own site while claiming to be Googlebot:
curl -s -o /dev/null -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' https://example.com/That request should appear in the crawler log, and the audit should list your address under unverified sources, not among the verified crawlers. If it shows up as verified, something is wrong with your ranges or your real-address setup.
The script has a few design points worth knowing:
- Each distinct address is verified once. A busy log has few distinct crawler addresses, so verification is cheap.
- Unparsed lines are counted, not hidden. Scanner junk and broken requests will not match the format, and the report tells you how many there were.
- The sitemap check compares exact paths and queries. A sitemap URL counts as crawled only if the log has a request for that exact path.
- A verified crawler means a Google address, not necessarily Search. The address belongs to Google's published ranges, and the user agent tells you which crawler it claims to be.
5. Reading the report
The report has these sections, and each one points to a specific action:
- Claimed versus verified. Unverified requests are fake crawlers. They are not a crawl problem, but they often probe for admin pages and vulnerable files, so look at what they request. Consider rate limiting unverified Googlebot user agents, but watch the log first to make sure your ranges are current. If nothing verifies at all, the likely cause is wrong addresses in your log, not a site with no Googlebot.
- Crawler families and requests per day. The split between smartphone and desktop shows how Google sees your site. A sudden drop in daily requests is worth investigating: check robots.txt, server errors and DNS around that date.
- Status classes. A high share of server errors is the first thing to fix. Sustained 5xx and 429 responses make Google slow its crawling, and Google documents this.
- 404 URLs. If your own pages link to them, fix the links or redirect them to the right page. If they are old URLs that nothing links to, a 404 is fine. Do not redirect everything to the home page, since Google may treat that as a soft 404.
- Redirected URLs. Update internal links to point at the final address, and remove redirect chains so a request needs only one hop.
- Crawl waste. The default patterns flag URLs with query strings, feed paths, internal search, JSON API paths, carts and CDN paths. Add your own patterns with the waste flag. To reduce waste, make internal links consistent, add canonical tags, and block endless spaces such as internal search with robots.txt. Remember that robots.txt stops crawling, not indexing, and that a page blocked by robots.txt cannot show a noindex tag. Google's guide to faceted navigation covers the options.
- Most crawled URLs and sections. Compare them with what you consider important. If Googlebot spends most requests on the least valuable section, look at your internal links and your sitemap.
- Response time. The 95th percentile matters more than the median. Slow responses limit how fast Google crawls, so find which kinds of page are slow.
- Sitemap URLs not requested. First check that the pages are linked from other pages, because orphan pages are crawled rarely. Then check that the period is long enough and that the pages are not new. For anything still missing, use the URL Inspection tool in Search Console.
6. Make it a routine
A one-off audit finds problems. A weekly audit finds changes. This cron setup refreshes Google's ranges and writes a dated report every Monday. The percent sign is escaped, because cron treats an unescaped one as a newline:
0 5 * * 1 /opt/audit/update_ranges.sh
0 6 * * 1 /usr/bin/python3 /opt/audit/crawl_audit.py /var/log/nginx/crawlers.log /var/log/nginx/crawlers.log.1 --ranges /opt/audit/common-crawlers.json /opt/audit/special-crawlers.json --sitemap /var/www/sitemap.xml > /var/log/crawl-report-$(date +\%F).txtCheck your log rotation setup. Depending on whether you compress the newest rotated file, it will be named crawlers.log.1 or crawlers.log.1.gz, so pass the names that actually exist.
Track a handful of numbers from week to week and look at the trend, not any single value: verified requests per day, the share of 5xx responses, the share of wasted requests, the 95th percentile response time, and the number of sitemap URLs not crawled. A change in any of them after a deploy tells you what that deploy did to crawling.
On retention, keep the crawler log as long as your analysis needs it, and keep the full visitor log for as short a time as your policy allows. Both contain IP addresses.
7. Limits
- Logs show requests, not indexing. Pair the audit with Search Console's page indexing report.
- CDN caching hides requests. If a cache answers Googlebot, your origin never sees it. Use edge logs for the full picture.
- Log formats vary. The parser expects nginx's combined format with the response time appended. If you use JSON logs or another layout, adapt the regular expression.
- Other Google fetchers are not covered. The ranges used here cover common and special crawlers. User-triggered fetchers have their own files.
- Ranges go stale. Refresh them weekly, and use the rdns flag if you suspect gaps.
- The report is a snapshot of one period. Compare periods before concluding anything.
8. Checklist
- Add the response time to the nginx log format and write crawlers to their own file.
- If a proxy or tunnel is in front, configure the real address and trust only your own proxies.
- Download Google's IP ranges and refresh them weekly.
- Run the smoke test to confirm a fake Googlebot is not verified.
- Audit at least 30 days of logs.
- Fix server errors first, then redirects and broken internal links, then waste.
- Check which sitemap URLs were never requested.
- Run the audit weekly and watch the trend.
Summary
Your access log is the only complete record of what Googlebot actually did on your site. Configure it to capture the right fields, verify crawlers properly instead of trusting the user agent, and read the result as a list of concrete fixes: errors first, then redirects, then waste, then pages Google is ignoring. Repeat it weekly, and the log becomes an early warning system rather than a forensic tool.