Glossary definition

AI Crawler

/ˌeɪ.aɪ ˈkrɔː.lər/ noun

A bot operated by an AI company that indexes web content for use in training large language models or powering real-time AI search responses. Major AI crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), and Google-Extended (Google). Each operates differently and can be selectively allowed or blocked via robots.txt.

See it in action

Run an AI SEO audit

Check technical SEO, AEO readiness, GEO signals, crawler access, and content gaps in one workflow.

Explore Thoth

Read the full guide

Go deeper on AI Crawler

Read a practical guide with examples, workflows, and implementation advice.

Read the Guide