Technical SEO Foundations

Before content quality, before links, before anything with a dashboard, a search engine has to do four mechanical things with a website: discover its URLs, fetch them, render them, and decide which version of each page is the real one. Technical SEO is the discipline of making those four steps boring. When they go wrong, nothing downstream can compensate — and when they go wrong on a small business site, they usually go wrong silently.

Crawlability: discovery and fetch

• robots.txt controls fetching, not indexing. A page blocked by robots.txt can still appear in results (URL-only, no snippet) if something links to it. Blocking is for crawl management; de-indexing requires noindex — and a page must be crawlable for the noindex to be seen. Confusing these two is the most common serious mistake we encounter in audits.
• Internal links are the discovery mechanism. Every page that matters should be reachable within three clicks of the home page through plain <a href> links. Pages reachable only through JavaScript event handlers, search boxes, or map embeds are effectively invisible to crawlers.
• XML sitemaps declare, they don't command. A sitemap is a hint list of canonical URLs with honest lastmod dates. It does not force indexing, and it should never contain redirects, 404s, or non-canonical variants — a dirty sitemap teaches Google to distrust it.

Response codes: say what you mean

Two failure patterns deserve their names: redirect chains (A→B→C→D wastes crawl budget and dilutes signals; every hop is a place to break) and soft 404s — "not found" pages that return 200, which teach Google that your site serves thin duplicate junk at infinite URLs. Both are detectable mechanically and both are pure engineering hygiene.

Canonicalization: one page, one URL

Every page on the modern web is accessible at several URLs — http/https, www/apex, trailing slash or not, tracking parameters appended. Search engines must collapse these to one, and they'd rather you told them:

• Pick one canonical form (we use https://, apex or www consistently, trailing-slash directories) and 301 every variant to it at the edge.
• Every page carries a self-referencing <link rel="canonical"> — absolute URL, matching the sitemap exactly.
• The canonical tag is a strong hint, not a directive; Google ignores it when signals disagree (internal links pointing at a different variant is the usual cause). Consistency across links, sitemap, and canonicals is what makes it stick.

Rendering: content in the HTML

Googlebot renders JavaScript, but on a deferred second pass — and most other crawlers (Bing's is inconsistent; AI crawlers largely don't execute JS at all) never get that far. The engineering answer is simple and old-fashioned: meaningful content arrives in the initial HTML response. Static generation or server-side rendering for anything that needs to be found; client-side rendering only for what happens after a human is already on the page. This one decision quietly solves indexing speed, AI visibility, and most Core Web Vitals problems at once.

The audit loop

None of this is set-and-forget. Sites rot: a CMS update adds parameters, a redesign orphans pages, a plugin writes a second canonical. Our standing audit — run against every BYHS production site — checks titles and descriptions for duplicates and gaps, heading hierarchy, broken internal links, redirect chains, canonical consistency, sitemap-to-crawl parity, response codes, and Search Console's Page Indexing report, which is the ground truth for how Google actually classified every URL it found. Anything that moves gets investigated before it gets "fixed" — per the lab's methodology, diagnosis precedes treatment.

← All research