
Check AI crawler access

Check AI crawler access
Mydentify AI Crawler Access Checker is a free web-based tool that analyzes whether AI crawlers associated with OpenAI and Anthropic can access a specified public web page. The tool evaluates the submitted URL against a set of documented crawler user-agent names, including OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, and ClaudeBot. It reads the site's robots.txt file, inspects page-level directives such as meta tags and X-Robots-Tag headers, and probes the URL to report the HTTP response received for each crawler. The tool treats search discovery, user-requested retrieval, and model training as separate functions, allowing users to see how access rules affect each specific purpose. The workflow involves entering a public page URL, running the access check, and reviewing the results for each crawler individually. This allows users to identify access rules that may be blocking AI tools from retrieving their content. The tool is intended for website owners, developers, and content publishers who want to assess their eligibility for AI discovery. It is important to note that the tool only reports observable access rules and responses. It cannot guarantee that a provider will crawl, index, cite, or rank a page, and the user-agent probe originates from Mydentify infrastructure rather than a verified provider IP, so it cannot certify official crawler behavior. The tool also notes that crawler names and provider policies can change, and users are advised to review primary documentation before altering production access rules.