How agents can use TYPENORM

We want agents to be able to read TYPENORM, whether they are crawling it for a model or acting for a person. This page says what is open, what is not, and where to find the same policy in machine-readable form.

What you can read

Every public page is open to every agent. A few paths are not site content and are closed to everyone, people's agents and crawlers alike:

/design-system/
/admin/
/api/
/social/
/f/

These are the same for every user agent in robots.txt.

AI crawlers

AI crawlers are welcome on every public page, and so is training on what they read there.

Each of these has its own group in robots.txt, with the same rules as everyone else:

GPTBot · OpenAI: OpenAI's crawler for training and improving its models.
ClaudeBot · Anthropic: Anthropic's crawler for training and improving its models.
PerplexityBot · Perplexity: Perplexity's crawler for finding and citing pages in its answers.
Google-Extended · Google: Not a separate crawler: a robots control token Google reads to decide whether pages it already crawls may be used for Gemini training and grounding.

Acting agents

Agents acting on a person's behalf, in a browser or through computer use, are welcome on every public read route: articles, the School library and courses, the apps, flows and landings catalogs, and Pulse tools and templates.

Sign-up, sign-in and paid actions are protected by bot checks (BotID). Those checks may refuse an automated client, so an agent should hand those steps back to its person.

There is no agent-specific firewall rule: we do not block these agents, and we do not grant them special access either.

Machine-readable files

The same policy, for machines:

llms.txt: A map of the site for agents, with this policy.
robots.txt: Crawl rules, with a group for each AI crawler above.
sitemap.xml: Every indexable page.