By FELFAL Bouabid · Updated October 9, 2026
This article deals with both. You will get an incremental build that touches only pages whose content really changed, using git tree objects as a Merkle tree of your site, working Python you can run, and the quality gates to put in place before you scale from 100 pages to 10,000. A word of warning first: Google's spam policy on scaled content abuse targets pages produced in bulk mainly to rank, whatever tool produced them. Build speed is the easy half of pSEO. Pages worth indexing are the hard half.
1. Why 10k-page builds are slow and noisy
A naive static build deletes the output folder and regenerates everything. Three costs follow:
- Deploy cost. Tools that compare timestamps or sizes (rsync, many S3 sync scripts) see 10,000 'new' files and upload them all, even if 9,900 are byte-identical to the last build.
- Sitemap noise. If lastmod is set to the build time, all 10,000 URLs claim to have changed on every build. Google says it uses lastmod only when it proves consistently accurate, so noisy values teach it to ignore the field.
- No changelog. After a build you cannot answer the simple question: which pages actually changed, and why?
The fix is to make 'changed' mean 'the output bytes are different', and to compute that cheaply.
2. Git trees are a Merkle tree of your site
Git stores a directory as a tree object: a list of names, modes and the hashes of the children. A file is a blob object identified by the hash of its content. Two consequences matter for a static site:
- If any file changes, its blob hash changes, which changes the hash of its parent tree, and so on up to the root tree. If nothing changes, the root tree hash is identical. One comparison tells you whether the whole site is unchanged.
- When you compare two trees, git diff-tree can skip every subtree whose hash is identical. It lists only the paths that differ, with a status: A added, M modified, D deleted.
The whole mechanism is four commands, run inside the output directory:
git add -A
NEW=$(git write-tree)
OLD=$(git rev-parse 'HEAD^{tree}')
git diff-tree -r --name-status --no-renames "$OLD" "$NEW"git add -A still has to look at every file, but it uses the index's stat cache, so repeat runs are cheap. The expensive part of the old workflow, working out what changed, becomes a hash comparison.
3. Prerequisite: deterministic output
Hashing only helps if the same input always produces the same bytes. Treat these as rules:
- No build timestamps anywhere in the HTML. One 'Generated at ...' line in the footer makes every page different on every build and defeats the whole scheme.
- No random ids, and no iteration over unordered collections. Sort dictionary keys and lists.
- Format numbers and dates the same way every time.
- Use one encoding (UTF-8) and one newline style.
- Version your template. When the template changes, bump a version string so every fingerprint changes on purpose.
4. Reference implementation
The build has three layers of skipping. The first is a fingerprint of each data row plus the template version: if it matches the previous build, the page is not even rendered. The second compares the new HTML with the file on disk: if the bytes are the same, nothing is written and lastmod stays put. The third is the git tree diff, which turns the final state of the output directory into an exact upload and delete list.
import hashlib
import html
import json
import shutil
import subprocess
import time
from pathlib import Path
TEMPLATE_VERSION = 'v1' # bump when the template changes: every page rebuilds
EMPTY_TREE = '4b825dc642cb6eb9a060e54bf8d69288fbee4904'
SITE = Path('site') # output directory, also a private git repo
STATE = Path('.build/manifest.json')
BASE_URL = 'https://example.com'
MIN_FACTS = 4 # quality-gate placeholders: tune on your own pages
MIN_SUMMARY_CHARS = 200
def sh(*args):
r = subprocess.run(args, cwd=SITE, check=True, capture_output=True, text=True)
return r.stdout.strip()
def fingerprint(row):
blob = json.dumps(row, sort_keys=True, separators=(',', ':'), ensure_ascii=False)
return hashlib.sha256((TEMPLATE_VERSION + blob).encode('utf-8')).hexdigest()
def indexable(row):
facts = [v for v in row['facts'].values() if str(v).strip()]
return len(facts) >= MIN_FACTS and len(row.get('summary', '')) >= MIN_SUMMARY_CHARS
def render(row):
# Deterministic on purpose: no timestamps, no random ids, sorted keys.
robots = '' if indexable(row) else "<meta name='robots' content='noindex,follow'>"
facts = ''.join(
f'<li>{html.escape(k)}: {html.escape(str(v))}</li>'
for k, v in sorted(row['facts'].items()))
title = html.escape(f"{row['name']}: {row['topic']}")
url = f"{BASE_URL}/p/{row['slug']}/"
return f'''<!doctype html>
<html lang='en'>
<head>
<meta charset='utf-8'>
<title>{title}</title>
<link rel='canonical' href='{url}'>
{robots}
</head>
<body>
<h1>{title}</h1>
<p>{html.escape(row.get('summary', ''))}</p>
<ul>{facts}</ul>
</body>
</html>
'''
def write_sitemaps(entries):
# entries: [(slug, lastmod)] for indexable pages only; 45k per shard (limit is 50k)
size = 45_000
shards = [entries[i:i + size] for i in range(0, len(entries), size)]
ns = "xmlns='http://www.sitemaps.org/schemas/sitemap/0.9'"
head = "<?xml version='1.0' encoding='UTF-8'?>\n"
index = []
for n, shard in enumerate(shards, 1):
urls = ''.join(
f'<url><loc>{BASE_URL}/p/{s}/</loc><lastmod>{m}</lastmod></url>\n'
for s, m in shard)
(SITE / f'sitemap-{n}.xml').write_text(
f'{head}<urlset {ns}>\n{urls}</urlset>\n', encoding='utf-8')
index.append(f'<sitemap><loc>{BASE_URL}/sitemap-{n}.xml</loc></sitemap>\n')
(SITE / 'sitemap.xml').write_text(
f"{head}<sitemapindex {ns}>\n{''.join(index)}</sitemapindex>\n", encoding='utf-8')
def build(rows):
SITE.mkdir(exist_ok=True)
STATE.parent.mkdir(exist_ok=True)
state = json.loads(STATE.read_text()) if STATE.exists() else {}
today = time.strftime('%Y-%m-%d')
new_state, entries, written = {}, [], 0
for row in rows:
slug = row['slug']
fp = fingerprint(row)
out = SITE / 'p' / slug / 'index.html'
prev = state.get(slug)
if prev and prev['fp'] == fp and out.exists():
lastmod = prev['lastmod'] # inputs unchanged: skip the render
else:
text = render(row)
old = out.read_text(encoding='utf-8') if out.exists() else None
if text != old: # lastmod follows the OUTPUT bytes
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(text, encoding='utf-8')
lastmod = today
written += 1
else:
lastmod = prev['lastmod'] if prev else today
new_state[slug] = {'fp': fp, 'lastmod': lastmod}
if indexable(row):
entries.append((slug, lastmod))
for slug in set(state) - set(new_state): # rows that disappeared
shutil.rmtree(SITE / 'p' / slug, ignore_errors=True)
write_sitemaps(sorted(entries))
STATE.write_text(json.dumps(new_state, sort_keys=True), encoding='utf-8')
return written
def snapshot():
"""Commit the output dir; return (tree_hash, [(status, path), ...])."""
if not (SITE / '.git').exists():
sh('git', 'init', '-q')
sh('git', 'config', 'user.email', 'build@localhost')
sh('git', 'config', 'user.name', 'build')
sh('git', 'add', '-A')
new = sh('git', 'write-tree')
try:
old = sh('git', 'rev-parse', 'HEAD^{tree}')
except subprocess.CalledProcessError: # first build: no commits yet
old = EMPTY_TREE
if new == old:
return new, []
diff = sh('git', 'diff-tree', '-r', '--name-status', '--no-renames', old, new)
changes = [line.split('\t', 1) for line in diff.splitlines()]
sh('git', 'commit', '-q', '-m', f'build {new[:12]}')
return new, changes
def deploy_plan(changes):
upload = [p for status, p in changes if status in ('A', 'M')]
delete = [p for status, p in changes if status == 'D']
return upload, deleteSome details worth knowing:
- lastmod is stored per page and only moves when the written bytes change. A data edit that does not alter the HTML does not touch the sitemap.
- Pages that fail the quality gate get noindex,follow and are left out of the sitemap, but they are still built so internal links keep working.
- EMPTY_TREE is the SHA-1 empty tree. If you create the repo with --object-format=sha256, get the right constant with git hash-object -t tree /dev/null.
- Everything inside site/ is part of the diff, including the sitemaps, so they are deployed only when they really change.
5. Quality gates before you scale to 10,000
An efficient pipeline that publishes thin pages just produces unindexed pages faster. Put these controls in place first:
- Generate a page only if the data gives it something specific. The code above requires a minimum number of non-empty facts and a minimum summary length. Both numbers are placeholders, not Google rules. Tune them by reading pages yourself.
- Swapping the city name is not enough. A visitor should learn something on page 4,000 that they cannot learn from page 3,999. If the only difference between two pages is a proper noun, merge them or drop one.
- Roll out in stages: 100 pages, then 1,000, then 10,000. After each stage wait for Search Console to report indexing status. If a large share of the first batch lands in 'Crawled, currently not indexed', fix the pages before adding more. Adding volume to a quality problem makes it worse.
- Link from indexable hubs. Every page should have at least one inbound link from an indexable page, for example a category or index page. Orphan pages are rarely indexed.
- Keep URLs stable and canonical. Each page points to itself, with no query parameters.
- Review a random sample of 20 pages before every stage, read as a visitor and not as a developer.
6. Sitemaps, lastmod and pinging
A sitemap file may hold at most 50,000 URLs or 50 MB uncompressed. The code shards at 45,000 and writes an index file, so the same code keeps working past 10k. Only indexable pages are listed, and lastmod reflects real changes only.
The upload list from deploy_plan maps directly to the page URLs to submit to IndexNow (drop the index.html suffix). Bing, Yandex and other participating engines support it. Google does not use IndexNow, so for Google the sitemap remains the signal.
7. Measure it on your own data
Save this as bench.py next to gitbuild.py and run python bench.py. It needs Python 3.8 or newer and git. The dataset is synthetic, so replace demo_rows with your real data.
import random
import shutil
import time
import gitbuild as gb
def demo_rows(n=10_000, bump=()):
# Synthetic stand-in. Replace with your real dataset.
for i in range(n):
r = random.Random(i)
pop = r.randint(10_000, 5_000_000) + (7 if i in bump else 0)
yield {
'slug': f'city-{i:05d}', 'name': f'City {i}', 'topic': 'Cost of living',
'summary': (f'City {i} has about {pop} residents. ' * 6)[:260],
'facts': {'population': pop, 'rent_1br': r.randint(300, 3000),
'transit_score': r.randint(1, 100), 'climate': r.choice('ABCD')},
}
def run(label, rows):
t0 = time.perf_counter()
written = gb.build(rows)
t1 = time.perf_counter()
tree, changes = gb.snapshot()
t2 = time.perf_counter()
up, de = gb.deploy_plan(changes)
print(f'{label:<22} render={t1 - t0:6.2f}s git={t2 - t1:5.2f}s '
f'written={written:>6} upload={len(up):>6} delete={len(de):>3} tree={tree[:10]}')
if __name__ == '__main__':
for p in ('site', '.build'):
shutil.rmtree(p, ignore_errors=True)
run('cold build', demo_rows())
run('no-op rebuild', demo_rows())
BUMP = set(range(0, 10_000, 100))
run('1% of rows changed', demo_rows(bump=BUMP))
run('50 rows removed', demo_rows(9_950, bump=BUMP))
gb.TEMPLATE_VERSION = 'v2' # same HTML bytes, new fingerprint
run('template bump (no-op)', demo_rows(9_950, bump=BUMP))What to look for in the output:
- No-op rebuild: written, upload and delete must all be 0, and the tree hash must equal the previous one. If not, something non-deterministic is leaking into your HTML.
- 1% of rows changed: upload should be about the number of pages you edited, plus the sitemap if a lastmod date moved.
- Template bump with identical output: every page is fingerprinted again, but nothing is uploaded.
7. When this helps and when it does not
- Check your host first. Several hosting tools, for example the Netlify and Cloudflare Pages CLIs, already upload only files whose hash is new. If you deploy that way, the deploy half of this design adds little. Check your host's documentation before building it.
- Where it still pays off: deploys to S3 with a sync script, rsync to your own server, CDN purge lists, IndexNow pings, accurate lastmod, and an audit trail. Each build is a commit, so git diff between two builds is a readable changelog, and a rollback is a checkout.
- Rendering time is a separate problem. The fingerprint layer is what saves render time. The git layer saves deploy time. Neither makes a slow template fast.
- Scale limits. Git handles tens of thousands of files comfortably. At hundreds of thousands, split the output into one repo per section. A single directory with 10,000 entries produces a large tree object that is rewritten whenever any child changes. That is fine at this size, but it is the first thing to shard.
- It cannot fix thin pages. Faster deploys do nothing for indexing. Section 5 is the part that decides whether Google keeps your pages.
8. Checklist
- Remove every timestamp and unordered iteration from the template.
- Fingerprint each row together with a template version.
- Write a file only when its bytes change, and track lastmod from that.
- Commit the output folder and deploy from git diff-tree.
- Shard sitemaps and list only indexable pages.
- Gate thin pages with noindex or do not generate them.
- Publish in stages and watch indexing before the next stage.
- Run the no-op rebuild test in CI: it must produce zero changes.
Summary
Treat the site as a content-addressed tree: fingerprint inputs, write only changed bytes, and let git tell you exactly what to deploy and what to ping. That gives you faster deploys, honest sitemaps and a changelog for free. It does not give you pages Google wants to index. For that, start small, make every page say something specific, and scale only after the first batch has been indexed.