Before You Delete 244 Pages

Deleting pages sounds like the safest operation in web operations. You are removing something, not changing anything, and if it was duplicate content you are improving the site.

We retired 244 pages across four sites in one afternoon. Three checks ran before anything was deleted, and each one caught something that would otherwise have turned a cleanup into a set of broken links.

Here they are, in the order they have to run.

1. What links to these pages?

Not “what links to your site”. What links, from inside your own site, to the specific pages you are about to remove.

We found 41 internal links across 33 surviving posts pointing at pages on the retirement list.

The naive handling is to let the redirect deal with it: delete the page, add a 301, and the link now resolves via a redirect. That works, and it means every one of those 41 links now costs a redirect hop on every visit and every crawl, permanently, for no reason.

The correct handling is to repoint the links at the destination before deleting anything. Same final page, no hop. It is a find-and-replace across your content, and it has to happen first — once the file is gone you have lost the mapping that tells you what to replace it with.

2. What already redirects to these pages?

This is the one almost nobody checks, and on our estate it was the most dangerous.

Sites accumulate redirects. Old URLs from a previous structure, campaign links, things moved years ago. Those rules point at pages that exist today.

158 existing redirect rules pointed at pages on our retirement list.

Delete those pages without touching the rules and every one of those 158 becomes a 301 into a 404. A visitor follows an old link, gets redirected, and lands on an error — which is a worse experience than the old URL simply 404ing, because it takes longer and looks deliberate.

The fix is mechanical once you know to look: any existing rule whose target is a page you are retiring must be repointed at the same destination the retired page is going to.

A warning about how we found this, because the method matters. The first version of our check asked whether each retiring path appeared anywhere in the redirects file. It reported 158 of 162 paths as “already handled” when in fact none of them were sources — the paths were matching on the right-hand side of existing rules, as targets. A substring search cannot tell a source from a destination. Parse the file into pairs and check the side you mean.

★ Insight ───────────────────────────────────── That bug has a recognisable family. The same shape once ate the /blog/ out of /images/blog/hero.jpg during a link rewrite, because the pattern matched anywhere rather than at a boundary. Any time you search for a path inside a file of paths, ask what else that string appears inside. The answer is usually “several things that mean something different”. ─────────────────────────────────────────────────

3. Is the URL what you think it is?

The check we did not know we needed.

On one of our sites, a post’s URL can be set in its frontmatter, overriding the filename. Twenty-one of the files we were retiring did that — and in two groups, every copy declared the same URL.

Which means those pages were never individually reachable. Fourteen files all claiming one address; one of them wins and thirteen have no URL at all. There is nothing to redirect from, because there was never anything there.

Worse, the page we were keeping also declared an overriding URL. Our redirect had been built from filenames, so it pointed the keeper’s live URL at a URL that does not exist — taking a working page and redirecting it into nothing.

We caught it because the verification compared the redirect map against the built output. The rules were regenerated from the URLs each page actually serves at, 21 never-served paths were dropped, and the final state verified: 614 rules, none targeting a 404, none shadowing a live page.

The checklist

Before deleting any set of pages:

List what links to them internally, and repoint those links at the destination first.
List what already redirects to them, and repoint those rules at the same destination.
Confirm the URL each page actually serves at, from the built site — not from the filename, the title or the CMS entry.
Write redirects for both URL forms, with and without a trailing slash. Once the page is gone, nothing remains to issue the platform’s automatic slash-redirect, so the bare form 404s unless it has its own rule.
Decide which page survives on evidence, not on date. We kept whichever copy held the most inbound links; 214 links would otherwise have been thrown away.
Rebuild and verify that no rule targets a page that no longer builds, and no rule shadows a page that does.
Keep the deletion in one commit, so it can be reverted as a unit.

Why not just leave them?

A fair question, and for a small number of duplicates the answer is genuinely to leave them alone.

Three things change at scale.

The link equity is split. Where several near-identical pages exist, whatever internal links they have are divided between them. Consolidating adds those together on one page. Ours had 214 inbound links spread across pages that were about to be retired — which is the strongest argument for consolidating rather than deleting, and for keeping whichever copy holds the most.

Crawl budget is finite. A search engine allocates a certain amount of attention to your site. Spending it on 244 pages that say the same thing is attention not spent on the pages that sell something.

It is a quality signal. A site that is substantially duplicate is describing itself, accurately, as a site where content is generated rather than written.

Against that, the cost of the operation is real: it is a day of careful work, and done carelessly it breaks links that were previously fine. Below perhaps twenty duplicates, the honest answer is that it is not worth the risk. Above a hundred, it is.

The part that is not technical

Two judgement calls, and both matter more than the mechanics.

Not everything that looks duplicate is duplicate. We compared the body text of every page against the one we planned to keep, and held back any group where a page scored below 60% similarity. Twenty-five groups were held. Those were the same title over genuinely different content — a retitle or a rewrite, not a redirect. Redirecting them would have destroyed content while the report claimed a tidy-up.

A cleanup should be reversible until it is verified. Everything was committed and pushed beforehand, so the whole operation was one revert away from undone for as long as it took to check it.

Neither of those is clever. Both are the difference between a cleanup and an incident.

Cloud Geeks handles website migrations, redirects and infrastructure for Australian small businesses. Web builds come from Cosmos Web Tech, apps from Awesome Apps, and the group is Ganda Tech Services.