# Bots welcome. Machine access for AI assistants: https://withby.ai/.well-known/ai-catalog.json (MCP server card); our apps are in the ChatGPT and Claude directories. # Terms: https://withby.ai/terms . Bulk copying is not allowed. Our bots and policy: https://yojowa.com/bots.html # SEO/crawler policy. The rules below are the whole file as served. # # MEASURED 2026-09-17 (issue #4763): the public https://withby.ai/robots.txt # carries exactly ONE agent group -- this one -- and is byte-identical to # loopback (1014 bytes). Checked across four UAs, because the Cloudflare # feature described below is UA-TARGETED, so a single fetch cannot tell # "switched off" from "not served to me": # # Googlebot/2.1 | GPTBot/1.0 | Mozilla/5.0 (X11) | curl/8.5.0 # -> 1 agent group, 1014 bytes, all four # # SUPERSEDED (struck, not deleted -- house style, so a reader who inherited # the old model meets the contradiction). Issue-2890 recorded that # Cloudflare's managed AI-crawler / Content-Signal feature PREPENDED its own # allow-everything agent group at the edge, so the served file carried TWO # groups with this file only the second; and that a strict first-match-only # parser would take Cloudflare's group and never reach the disallows below, # leaving /api/, /auth/, /ws and /mocks/ effectively permitted for that class # of crawler -- accepted deliberately at the time. # # BOTH HALVES ARE NOW FALSE: there is no second group, and the exposure it # described does not exist. Do NOT reason about /api/ or /auth/ exposure # from the superseded paragraph above. # # This does NOT assert the feature was disabled -- only that it is not # composing this response for any of those four UAs. That setting lives in # Cloudflare's dashboard, outside this repo. Re-derive before quoting: # curl -s -A "GPTBot/1.0" https://withby.ai/robots.txt | grep -c '^User' # User-agent: * # Listed first: a first-match reader (Python's robotparser, older bots) must meet it before `Allow: /`. Disallow: /export-all Allow: / Disallow: /api/ Disallow: /auth/ Disallow: /ws Disallow: /mocks/ # SEO data resellers: no. User-agent: DataForSeoBot User-agent: MJ12bot User-agent: DotBot Disallow: / # AI training: the named LLM services are welcome. Common Crawl publishes a free copy anyone can reuse, and ByteDance's crawler can't be verified. User-agent: CCBot User-agent: Bytespider Disallow: / Sitemap: https://withby.ai/sitemap.xml # YCT how-it-works pages: how WithBy works, step by step, with real screenshots (an image sitemap). Sitemap: https://withby.ai/how-it-works/sitemap.xml