Why robots.txt matters for a roofing crew

Crawlers and people want the same thing from robots.txt: one URL, a title that matches the heading, and a next link that goes somewhere real.

Test robots.txt from outside your network before you call it done. Your office Wi-Fi might cache the old answer for another six hours.

We checked the WHOIS for a roofing crew and found the domain was registered under a former employee. Transferring it took more time than fixing robots.txt.

Early on we tried to automate robots.txt for a roofing crew with a cron job. It ran as root, silently overwrote a manual change, and took down their mail. Lesson learned.

We moved robots.txt for a roofing crew to a provider that supports DNSSEC. It added five minutes to the migration and closed a class of attacks they did not even know about.

The most expensive outage for a roofing crew was not a hack — it was three unused font weights. Nobody attacked anything. The config just drifted.

More notes

Related notes

• Setting up robots.txt for a plumber
• How an accountant should handle robots.txt
• What a hardware store needs to know about robots.txt
• Setting up robots.txt for a veterinary clinic