Robots.txt generator for blocking AI crawlers

Similar products

by description similarity

1. A bot-free alternative to Otter.ai

   bluedothq.com

   HNHacker News5 pts

2. CockroachCrawler–1 package for crawling,browsers,PDF,extraction,and MCP

   github.com

   Give AI agents bounded eyes on the public web: crawl, search, and normalize evidence without an unrestricted browser.

   HNHacker News5 pts

3. A Node.js CLI tool to generate ai.txt

   npmjs.com

   llms.txt, robots.txt, humans.txt

   HNHacker News7 pts

4. GuardRail

   github.com

   shell guards that stop Claude Code before it pushes to main

   HNHacker News5 pts

5. I built an SDK that scrambles HTML so scrapers get garbage

   obscrd.dev

   Built this because I got tired of AI crawlers reading my HTML in plain text while robots.txt did nothing.

   HNHacker News16 pts