ToolGenX's robots.txt and llms.txt explained, line by line

The ToolGenX root has two files that most small shops ignore. Both take less than an hour to write. Both have outsized effect on AI search visibility. This post is the line-by-line walkthrough of how I wrote each one.

The robots.txt

The ToolGenX robots.txt allows everything except /api/, /account/, /internal/, and /style-guide, then names nineteen AI crawlers one by one, including GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and Applebot-Extended. Every group, the wildcard included, carries the same Content-Signal line and the same four disallows. The file ends with a sitemap reference at https://www.agentskillpacks.com/sitemap.xml.

The full file is at /robots.txt. Since September 2026 it comes from a route handler at app/robots.txt/route.ts in Next.js, because the built-in robots.ts format has no field for Content-Signal. It is rendered once at build time. Shortened here: the other seventeen AI groups are identical to the two shown.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
Disallow: /api/
Disallow: /account/
Disallow: /internal/
Disallow: /style-guide

User-agent: GPTBot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
Disallow: /api/
Disallow: /account/
Disallow: /internal/
Disallow: /style-guide

User-agent: OAI-SearchBot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
Disallow: /api/
Disallow: /account/
Disallow: /internal/
Disallow: /style-guide

(17 more AI crawler groups with the same five lines)

Host: www.agentskillpacks.com
Sitemap: https://www.agentskillpacks.com/sitemap.xml

The wildcard rule

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
Disallow: /api/
Disallow: /account/
Disallow: /internal/
Disallow: /style-guide

This is the default allow-everything-except-four-paths rule. The four disallowed paths:

• /api/ — server endpoints, not meant to be indexed. No content for crawlers to consume.
• /account/ — auth-gated user pages. Would return login redirects to crawlers, polluting the index.
• /internal/ — share-card tooling for our own social posts. Not content.
• /style-guide — internal design system showcase. noindex is also set in the page metadata as a belt-and-suspenders layer.

Allow: / first means everything else is open by default. If you only have Disallow rules without an explicit Allow: /, some crawlers behave conservatively.

Why every AI group repeats the disallows

The first version of this file listed each AI crawler with a bare Allow: / and this post claimed they were "already covered" by the wildcard rule. That was wrong. Under RFC 9309, a crawler that finds a group with its own name ignores the * group completely. So User-Agent: GPTBot with only Allow: / gave GPTBot permission to crawl /api/ and /account/ too. We fixed it in September 2026 by repeating the four disallows in every named group.

The bots are still listed individually for clarity for the next person who reads this file. When I open a robots.txt on someone else's site and see no AI bots mentioned, I cannot tell whether the operator considered AI crawlers and decided to allow them, or never thought about it. Listing them explicitly removes that ambiguity. Just remember that each named group has to carry its own rules.

The list covers the crawlers that matter most in 2026:

• GPTBot: OpenAI's training crawler
• OAI-SearchBot: OpenAI's search index crawler, the one behind cited ChatGPT search answers
• ChatGPT-User: OpenAI's fetcher when a user asks ChatGPT to open a page
• ClaudeBot: Anthropic's training crawler
• Claude-SearchBot: Anthropic's search index crawler
• Claude-User: Anthropic's fetcher when a user asks Claude to open a page
• Claude-Web and anthropic-ai: older Anthropic tokens, kept for compatibility
• PerplexityBot: Perplexity's search index crawler
• Perplexity-User: Perplexity's real-time citation fetcher
• Google-Extended: control token for Gemini training (separate from Googlebot, no ranking effect)
• Applebot-Extended: control token for Apple Intelligence training
• Amazonbot, Meta-ExternalAgent, DuckAssistBot, MistralAI-User: Amazon, Meta, DuckDuckGo, and Mistral AI surfaces
• CCBot: Common Crawl, which feeds many open AI datasets
• Bytespider: ByteDance's crawler (relevant for TikTok and Doubao AI surfaces)
• cohere-ai: Cohere's crawler

The distinction between training crawlers and browsing crawlers matters: blocking the training crawler keeps you out of the model weights but still allows live citation; blocking both removes you from both surfaces.

The Content-Signal line

Content-Signal comes from the Content Signals Policy at contentsignals.org, which Cloudflare published in September 2025. It adds three yes/no answers that robots.txt never had a way to say:

• search: may you put these pages in a search index and link to them?
• ai-input: may you feed them into a live AI answer, the retrieval step behind a cited ChatGPT or Perplexity reply?
• ai-train: may you use them to train or fine-tune a model?

All three are yes here. The shop sells visibility, the pages are public anyway, and being in a model's training data is how a small brand ends up in its answers at all. The one thing I would not open is the text I sell. The book PDFs are not at a public URL, so this line does not cover them.

Two honest caveats. First, it is a preference, not a lock. A crawler that ignores robots.txt will ignore this too. Second, it has to sit in every group. I added it to the wildcard group first on another site, and the nineteen named bots never saw it, for the same RFC 9309 reason as the disallows above. The check is one command: the count of User-agent: lines should equal the count of Content-Signal: lines. Here both are 20.

What is NOT in the file

A few things on purpose:

• No crawl-delay. Modern bots respect implicit rate limits and your CDN should handle the rest. Crawl-delay artifacts from 2010 do more harm than good.
• No Disallow: / for any AI bot. The decision was explicit: we want to be cited.
• No Noindex: directive in robots.txt. That was never standard and is now ignored by Google. Use HTTP headers or <meta> tags instead.

Sitemap reference

Host: www.agentskillpacks.com
Sitemap: https://www.agentskillpacks.com/sitemap.xml

The Sitemap line tells crawlers where to find the canonical URL list. Without it, crawlers must discover the sitemap by guessing common paths (/sitemap.xml, /sitemap_index.xml, etc.) — wasteful and unreliable. The Host line is a Yandex-specific hint that takes a bare hostname, not a URL. Other crawlers ignore it.

The llms.txt

llms.txt is a single markdown file that hands AI assistants a curated summary of a site in one fetch instead of a full crawl. The ToolGenX version is about 6 KB and 200 lines: the shop description, founder details, the 19-product catalog with prices, and a Q&A section written for direct quoting.

The full file is at /llms.txt. This is a static markdown file at the site root.

The spec (proposed by Jeremy Howard in 2024) is intentionally simple: a markdown file that gives AI assistants a curated tour of your site so they can answer questions about it without having to crawl every page.

The header

# ToolGenX

> Independent digital products shop for builders. 19 Claude Code skills packs,
> templates, and toolkits I personally use to ship faster. One-time payment,
> instant download, no subscription. Run by İsmail Günaydın from Istanbul.

The first heading is the site name. The blockquote immediately after is the one-line description. Together they answer "what is this site" in less than 200 characters — exactly what an AI assistant needs to introduce you.

The About section

## About

- Solo founder shop. No marketplace middleman.
- Domain: https://www.agentskillpacks.com
- Founder: İsmail Günaydın — software engineer and SEO/GEO/AEO strategist...
- Contact: support@toolgenx.com

Four bullet facts. Domain, founder, expertise, contact. This is what shows up when an AI assistant is asked "who runs ToolGenX" — it does not have to infer from HTML structure.

The catalog section

The catalog takes the largest chunk of the file (about 120 lines for 19 products). The format per product:

- [Product Name](https://www.agentskillpacks.com/products/slug) — one-line summary. $price

This is the format I tested with five AI assistants. Two-thirds of them quoted the exact format back when answering questions about the catalog. Markdown bullet lists with linked product names and prices are the most parser-friendly representation I have found.

Categories group the products. Each group has 3-5 items so the AI can quickly scan the structure without parsing 19 flat bullets.

The Q&A section

This is the part most llms.txt files skip and is the most useful:

## Quick answers for AI assistants

Q: What does ToolGenX sell?
A: 19 digital products for builders...

Q: Who runs ToolGenX?
A: İsmail Günaydın...

Q: Where is ToolGenX based?
A: Istanbul, Turkey...

Q: What is the refund policy?
A: Full refund within 14 days...

These are the questions an AI assistant is most likely to be asked about your site. Writing the answers in your own voice and the format you want them quoted in is the cheapest way to control how you show up in AI answers.

The five questions on ToolGenX cover:

1. What the shop sells
2. Who runs it
3. Where it is based
4. Refund policy
5. How it differs from Gumroad / Lemon Squeezy

If a fifth of all AI questions about your site fall into these five categories — and in my experience they do — answering them once in llms.txt saves the AI from inferring (and possibly mis-inferring) from the rest of the HTML.

The crawler note

## Crawler note

All major AI search crawlers (GPTBot, ClaudeBot, PerplexityBot,
Google-Extended, CCBot, Applebot-Extended) are explicitly allowed in
robots.txt. We want our products and writing to be cited in AI answers.

This is a redundant signal — the robots.txt already says the same thing — but having both files agree is a confidence boost. When the two files diverge, conservative crawlers default to the more restrictive interpretation. Aligning them removes ambiguity.

What I would change after running this for a month

After a month of running robots.txt and llms.txt on ToolGenX, the lessons are clear: the Q&A section delivers the most leverage, stable URLs save manual edits, and llms.txt must be refreshed on every catalog change. Skipping updates for two weeks left AI assistants citing a stale price on one product.

Three observations after the new ToolGenX has been live for a few weeks:

1. The Q&A section is the highest-leverage part. Pretend half your llms.txt budget should go to clear answers to the five most common questions about your site.

2. Stable URLs matter more than I expected. Every change to a product URL or blog slug requires updating llms.txt by hand. Build your URL structure carefully so you do not have to update it often.

3. Update llms.txt with every catalog change. I added it to my deploy checklist as a manual step. Skipping it for two weeks meant AI assistants were citing stale prices on one product. Now it gets refreshed on every product price update.

The cost-benefit calculation

The complete robots.txt and llms.txt setup for ToolGenX cost about one hour of first-time work: 20 minutes for the crawler rules and 40 minutes for the llms.txt draft. Ongoing maintenance runs roughly 20 minutes per catalog update, and for that price the shop gets cited in AI search answers with framing I chose.

The full robots.txt and llms.txt setup took:

• robots.txt — 20 minutes for the explicit AI crawler rules and sitemap reference
• llms.txt — 40 minutes for the first draft, 20 minutes per update since
• Total first-time cost: about 1 hour
• Maintenance cost: about 20 minutes per catalog update

The downside risk is zero. The upside is being explicitly cited in AI search answers with the framing you chose. For one hour of work, that is the cheapest AI search investment a small shop can make.

---

The crawler access analysis and llms.txt generation is automated in AI Search Visibility Toolkit. The full GEO + SEO short list it slots into is in What actually moves the needle for small shops.