Decide what the audit is for
Most technical audits fail before anyone opens a crawler, because nobody agreed what question was being answered. “Audit the site” is not a question. “Why did organic traffic to the category pages fall in March” is. So is “what is stopping new products from being indexed within a week” and “what must be true before this replatform goes live.”
The distinction matters because it determines what you collect and how deep you go. A diagnostic audit chases one symptom to its cause and stops. A baseline audit maps the whole surface and produces a backlog. A pre-migration audit exists to build an inventory. They share techniques and they produce very different documents.
I spoke on this at SearchFest in Portland in 2015 and the framing I used then still holds: an audit is not an inspection, it is a diagnosis, and a diagnosis that does not change what anyone does next was a waste of everyone's time. The measure of a good audit is not how many issues it found. It is how many things got fixed.
Gather access before you gather data
The single biggest determinant of audit quality is what you were given, and access requests take longer to fulfill than the analysis takes to run. Ask for all of it on day one:
- Search Console for every property variant — the HTTP and HTTPS versions, www and non-www, every subdomain, and the domain property if it exists. Sites routinely have coverage problems visible only in a property nobody looks at.
- Raw server access logs, not a summarized report, covering at minimum a few weeks and ideally a period spanning the change being investigated. These are usually held by the hosting provider or behind a content delivery network, and somebody has to be asked for them by name.
- Analytics with permission to see unsampled data and the configuration, including filters, which are a common source of numbers nobody can reproduce.
- The XML sitemaps and whatever generates them.
- A staging environment if one exists, plus a way to identify releases and their dates.
- The content management system and tag manager, at least in read-only form, so you can see revision history and container versions.
- Whitelisting for your crawler, because the firewall will otherwise rate-limit you and you will spend a day debugging your own tooling.
Then establish the baseline before you touch anything: current organic impressions and clicks by page group, current index coverage counts, current crawl volume from the logs. Without a baseline you cannot demonstrate that anything you recommended worked.
Configure the crawl deliberately
A default crawl on a large site produces a large amount of noise. A few configuration decisions do most of the work.
Run at least two crawls. One as a search engine crawler with rendering enabled, following the site as it is actually delivered. One with rendering disabled, so you can see the raw HTML the server returns before JavaScript executes. The difference between those two crawls is where a surprising number of indexation problems hide.
Crawl mobile-first, because that is how the site is primarily evaluated, and confirm the mobile and desktop deliveries contain the same content, links and directives. Respect robots.txt on the first pass, so you see what a crawler sees, then run again ignoring it to find out what is behind the wall — blocked sections often turn out to contain pages that are linked, valuable and quietly invisible.
Feed the crawler additional URL lists rather than relying on discovery alone: the sitemap URLs, the URLs found in logs, the URLs with impressions in Search Console, and any historical URL list from a previous site version. Discovery-only crawling finds what the site links to, which by definition excludes the orphan pages you are looking for.
Reconcile the four lists
This is the core technique, and it is the thing most audits skip. You have four independently produced lists of URLs, and the interesting findings live in the gaps between them.
- Crawled but not in the sitemap. Either the sitemap generation is incomplete, or the site is linking to URLs it does not consider canonical.
- In the sitemap but not crawled. Orphan pages. They exist, the sitemap declares them, and nothing on the site links to them — which means they receive no internal signal and are frequently the weakest performing pages on the site.
- In the logs but not in the crawl. URLs a search engine is spending requests on that the site no longer links to. Old parameters, retired sections, and the long tail of a previous site structure.
- Getting impressions but not indexed as expected, or the reverse: indexed, crawled regularly, and generating nothing at all.
- Receiving organic traffic but absent from the crawl. Always worth chasing. It usually means a section is served from a system nobody mentioned.
Do this as a set comparison in a spreadsheet or a few lines of script, normalizing for protocol, host, trailing slashes and case first. It takes an hour and it consistently produces the findings that a tool's issue list does not.
Work out why pages are excluded
Search Console's index coverage reporting names its exclusion reasons, and each one points at a different fix. Discovered – currently not indexed usually means the site is generating more URLs than the engine considers worth fetching, which is a crawl budget and internal linking problem rather than a content problem. Crawled – currently not indexed is a quality or duplication judgment: the page was fetched and declined. Duplicate, Google chose different canonical means your canonical tag was overruled, which tells you the rest of your signals disagree with it. Alternate page with proper canonical tag is usually fine and usually the largest bucket, and treating it as a problem wastes weeks.
Then go looking for the contradictions directly. A canonical pointing at a URL that redirects. A canonical injected by JavaScript that contradicts the one in the raw HTML. Paginated series where every page canonicalizes to page one, removing the products only listed on later pages. A noindex tag on a template that was applied to more pages than intended. HTTP and HTTPS, www and non-www, trailing slash and not, all answering 200. Hreflang annotations that do not return, which invalidates the cluster. And robots.txt being used to try to solve an indexation problem, which does not work: a blocked URL with inbound links can still appear in results with no description at all.
Check what the crawler actually receives
Rendering is deferred and it is not guaranteed. The question is always the same: what does the page contain when the crawler is finished with it, as opposed to when the server first responds?
Compare the raw HTML against the rendered document for four things specifically — the main content, the internal links, the canonical tag and the title. If any of those exist only after client-side execution, indexation is riding on a queue you do not control. The usual offenders are single-page applications without server-side rendering, infinite-scroll listings with no crawlable pagination behind them, content hidden behind interaction rather than merely visually collapsed, and navigation built on click handlers instead of anchor elements with href attributes. A crawler does not click. If there is no href, there is no link.
While you are in there, check status codes honestly. Soft 404s — pages that say “not found” while returning a 200 — keep dead URLs eligible for indexing indefinitely. Redirect chains waste crawl and dilute signals; every hop should be a single hop. And a site that returns a 200 for literally any URL you invent has an infinite crawl space, which is worth finding before the crawler does.
Then the layers everyone starts with
Site architecture and internal linking come next, and they matter more than their reputation suggests. Look at click depth from the homepage for the pages that make money, at whether category and filter pages are linked in ways that produce a bounded URL space, at orphaned sections, and at whether internal anchor text describes the destination or says “click here” four thousand times.
Structured data should describe what is genuinely on the page, stay synchronized when the page changes, and never mark up content a visitor cannot see. Invented or stale markup is worse than none, because it invites a manual action and teaches the search engine that your data is unreliable.
For performance, use field data from real users rather than the lab score a testing tool produces on demand; a site can pass in the lab and fail in the field because real visitors are on worse connections and older devices. Most performance problems were introduced for marketing reasons — tag managers loading a dozen third-party scripts, chat widgets, consent banners that shift layout after paint. Getting out of the failing range is worth doing. Chasing a perfect score is not.
If the site is international, verify that hreflang annotations are reciprocal, that the language and region codes are valid, and that the alternate URLs are the canonical ones rather than redirecting.
Prioritize, then hand over something usable
Any crawler will produce hundreds of findings, and very few of them are worth an engineer's afternoon. The prioritization is the part of the audit that earns its keep.
I sort by estimated traffic or revenue at stake first, then by implementation cost, then by confidence that the fix will do what I think it will. That ordering pushes template-level problems affecting thousands of URLs above individual page issues, and it pushes anything blocking indexation of commercially important pages to the top regardless of how small the change is.
The deliverable should be a ranked list a development team can put into sprint order, with each item stating the problem, the evidence, the specific change, how to verify it after release, and what happens if it is not done. Not a ninety-page document nobody opens twice. Ten items that will move revenue beat three hundred that fill a spreadsheet, and the extra two hundred and ninety are how audits get shelved.
Then re-measure. Set a date, re-run the crawl and the coverage report, and check the baseline you captured at the start. An audit without a follow-up is a hypothesis. Consulting engagements are handled through Hartzer Consulting.