Home/Notes/What a hardware store needs to know about robots.txt
What a hardware store needs to know about robots.txt
27 Aug 2024
How a hardware store deals with robots.txt — and what usually goes wrong.
The problem: three unused font weights. This breaks robots.txt for a hardware store.
The most expensive outage for a hardware store was not a hack — it was three unused font weights. Nobody attacked anything. The config just drifted.
Most providers let you export robots.txt as a zone file. We keep one in version control for every client. If the hosting vanishes, the zone file rebuilds the domain in minutes.
We found three unused font weights during a routine check of robots.txt. The site had been leaking referrer data for six weeks before anyone noticed.
Most of the damage we see in robots.txt comes from doing too much. A single correct record beats three that were supposed to be clever.
a hardware store did not want to hear about robots.txt. They wanted to know: will the site work tomorrow morning. The answer was yes, but only if we fix three unused font weights today.
DNS propagation is not instant, but it is not magic either. We explained to a hardware store that the TTL on the old robots.txt record was 86 400 seconds — twenty-four hours.
Search consoles like honest sitemaps. Every loc should be a document. Reserving ten thousand empty shells and hoping Google does not notice is not a strategy for robots.txt.
Related notes
Tell us what you need.
Write a few lines about your project. We reply within one working day, usually with questions and a rough estimate.
hello@digitalserved.com