SEO tool

robots.txt & llms.txt Generator

An llms.txt file generator and robots.txt generator. Set search and AI crawler rules, write an LLM-readable overview, then copy robots.txt, llms.txt, or app/robots.ts. It all stays on your device.

Like the tool? Find a bug?Share with us

Generator

Fill the site, crawler policy, and llms.txt sections, then copy robots.txt, llms.txt, or a Next.js robots.ts. Files stay on this device.

Search crawlers

AI crawlers

llms.txt

Sections

01

02

Live preview

https://html69.com/robots.txt

User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /wp-content/

Sitemap: https://www.html69.com/sitemap.xml

Host: www.html69.com
Supported files: .txt, .md
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /wp-content/

Sitemap: https://www.html69.com/sitemap.xml

Host: www.html69.com
# html69

> Fast, free developer utilities that run in the browser. Convert YAML, CSS, SVG, and Markdown, and generate SEO markup without uploading files.

Every html69 tool stays on-device. There is no account and no API. Start with a tool page, then follow the markdown-friendly links below.

## Tools
- [YAML to JSON Converter](https://www.html69.com/yaml-to-json): Convert YAML maps and lists into JSON in the browser.
- [Schema Markup & JSON-LD Generator](https://www.html69.com/schema-markup-generator): FAQ, Article, and Product JSON-LD.
- [SEO Meta Tag & Open Graph Generator](https://www.html69.com/meta-tag-generator): HTML head tags and a Next.js metadata export.

## Optional
- [Blog](https://www.html69.com/blog): Guides for browser-based developer tools.
import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: "*",
        allow: "/",
        disallow: ["/wp-admin/", "/wp-content/"],
      },
    ],
    sitemap: "https://www.html69.com/sitemap.xml",
    host: "www.html69.com",
  };
}

Guide

How this llms.txt generator works

01

robots.txt for crawlers

Search bots follow Allow and Disallow. Training crawlers like GPTBot and Google-Extended get their own blocks when you choose Block training or Block AI.

02

llms.txt for agents

llms.txt is markdown: an H1, a short blockquote, notes, then H2 file lists. Agents read it on demand. Put the file at /llms.txt or under a docs path.

03

Next.js robots.ts

App Router wants a typed MetadataRoute.Robots export in app/robots.ts. Paste the Next.js output there. Download llms.txt into public/llms.txt.

Use it

What to copy

Use robots.txt at the site root. Use llms.txt at /llms.txt (or /docs/llms.txt). Use Next.js when the app already has a robots.ts file convention. Sample is html69 open to search and AI. Docs is a documentation site. Closed disallows every path and every AI crawler. Input is capped at 50,000 characters.

FAQ

Questions

Does this upload my files?

No. Generation, previews, and copy run in the browser. There is no account and no API call.

robots.txt or llms.txt?

robots.txt tells crawlers what they may fetch. llms.txt tells agents what the site is and which markdown pages to read next. They complement each other; one does not replace the other.

What does Block training do?

It adds Disallow: / for training user-agents such as GPTBot, Google-Extended, CCBot, and ClaudeBot. Assistant fetch bots like ChatGPT-User stay allowed unless you choose Block AI.

Is there a size limit?

Paste is capped at 50,000 characters so a huge dump cannot freeze the tab.