Right now, someone in your city is asking ChatGPT: "Who's the best divorce lawyer near me?"
If your website is blocking AI crawlers — and statistically, there's about a 1-in-3 chance it is — ChatGPT can't read your site. It can't learn about your firm. And it definitely can't recommend you.
The terrifying part? This happens completely silently. Your website looks fine to you, fine to Google, fine to your clients. But when ChatGPT's crawler (called GPTBot) tries to visit your site, it gets blocked at the door.
Here's how to find out if this is happening to you — in under 60 seconds.
There are two quick ways to check. The first is manual (takes about 60 seconds), and the second is automated (takes about 10 seconds).
Open your browser and type your domain followed by /robots.txt. For example: https://yourfirmname.com/robots.txt
If you see a file, look for lines mentioning these AI crawlers:
If "Disallow: /" appears next to any of those AI crawler names, that platform cannot read your website at all. It's like putting a padlock on your front door and wondering why nobody's visiting.
"Disallow: /" after an AI User-agent = that AI can't see you
"Allow: /" or no mention of the AI crawler = that AI can see you
But here's the catch: robots.txt is only half the story. Even if your robots.txt looks fine, your CDN or firewall might be blocking AI crawlers at the network level — before they ever reach your robots.txt file.
We built a free tool that checks everything at once — not just robots.txt, but whether your CDN is blocking crawlers, whether your page loads fast enough for AI, whether you have schema markup, and 6 more technical factors. Takes about 10 seconds.
Enter your domain and get your AI readiness grade in 10 seconds. No signup required.
There isn't just one AI that needs to read your site — there are at least 8 major AI crawlers actively scanning the web right now, each feeding a different AI platform. If even one is blocked, you're losing potential visibility on that platform.
| Crawler | AI Platform | Why It Matters |
|---|---|---|
| GPTBot | ChatGPT / OpenAI | The #1 AI platform. 200M+ weekly users asking questions about everything — including who to hire as a lawyer. |
| ClaudeBot | Anthropic Claude | The fastest-growing AI assistant, used by professionals and increasingly for research and recommendations. |
| PerplexityBot | Perplexity AI | The "AI search engine" — explicitly designed to find and cite sources. If you're cited here, you get direct referral traffic. |
| Google-Extended | Google Gemini | Powers Gemini and Google AI Overviews (the AI box at the top of Google search results). |
| Bytespider | TikTok / ByteDance AI | TikTok's parent company is building AI products using web data at massive scale. |
| CCBot | Common Crawl | The open dataset that trains many AI models. Blocking this reduces your presence across dozens of AI systems. |
| meta-externalagent | Meta AI | Meta's AI crawler for Facebook, Instagram, and WhatsApp AI features. |
| Applebot-Extended | Apple Intelligence | Apple's AI features on iPhone, iPad, and Mac — reaching over 1 billion devices. |
Cloudflare is the most popular CDN for websites, and law firms love it because it's fast and secure. The problem is that Cloudflare's "Bot Fight Mode" — which is enabled by default on many plans — treats AI crawlers the same as spam bots. It serves them a JavaScript challenge or a 403 Forbidden response.
The fix is simple: in your Cloudflare dashboard, go to Security → Bots → Configure Bot Fight Mode, and add exceptions for AI crawler User-Agents. But most law firms don't even know this setting exists.
Plugins like Wordfence, Sucuri, and iThemes Security are great at blocking hackers. But their aggressive bot filtering often blocks legitimate AI crawlers too. If your security plugin has a "block unknown bots" setting, it's almost certainly blocking ChatGPT.
Check your plugin's bot blocking settings and create allowlist rules for GPTBot, ClaudeBot, and the other AI crawlers listed above.
Some hosting providers (like GoDaddy, WP Engine, and others) have server-level bot protection that you can't control from your WordPress dashboard. If you've checked your robots.txt and your security plugins and everything looks fine, but AI still can't see your site — call your hosting provider and ask specifically about AI crawler access.
The reason this is so dangerous is that you'll never know it's happening unless you check. Your website will load perfectly fine in your browser. Google can still index you. Your clients can still find you. But AI platforms see nothing — and they recommend your competitors instead.
Let's make this concrete. We ran a test on a real law firm in Chicago (with their permission). Their website was blocking GPTBot and ClaudeBot via Cloudflare. When we asked ChatGPT "Who are the best personal injury lawyers in Chicago?", this firm — which has been practicing for 20+ years, has hundreds of 5-star reviews, and ranks on the first page of Google — was not mentioned at all.
After we fixed their bot access (a 15-minute configuration change), ChatGPT started mentioning them within 2 weeks. That's the difference between being invisible and being recommended.
Bot access is the #1 blocker, but it's not the only thing that determines your AI visibility. Luna's 9-point technical audit checks all of the factors that affect whether AI platforms can find, read, understand, and recommend your firm:
| Check | What It Tests |
|---|---|
| AI Bot Access | Can 8 AI crawlers actually reach your site? |
| Page Speed / TTFB | Does your site respond fast enough for AI crawlers? |
| Schema Markup | Do you have JSON-LD that helps AI understand your practice? |
| robots.txt | Are AI crawlers explicitly allowed in your robots.txt? |
| SSL & Security | HTTPS + security headers that AI crawlers expect |
| XML Sitemap | Can AI crawlers discover all your important pages? |
| llms.txt | The new standard file specifically for AI platforms |
| JavaScript Rendering | Can AI read your content or is it locked behind JS? |
| Content Structure | Headers, meta descriptions, and content organization |
Each check is scored individually and combined into an overall grade from A to F. The free audit on our homepage shows you which checks pass and which fail — sign up to get the full breakdown with detailed scores, exactly how we measure each factor, and step-by-step fix instructions. Most law firm websites score between D and F on their first audit. The good news? Every single factor is fixable.
If you've read this far, here's the fastest path to action:
Use Luna's 9-point audit tool at the top of this page. It takes 10 seconds and tells you exactly where you stand.
If your audit shows AI crawlers are being blocked, that's your #1 priority. It's usually a CDN or security plugin setting that takes 15 minutes to fix.
Even if your crawlers aren't blocked, a properly configured robots.txt and an llms.txt file make a measurable difference in how AI platforms index and understand your firm.
JSON-LD schema tells AI exactly what your firm does, where you practice, and what areas you specialize in. It's the difference between AI guessing about your firm and AI knowing about your firm.
Luna's Growth plan ($999/mo) includes a free one-time Generative Engine Optimization (GEO) implementation setup — we deploy the robots.txt, llms.txt, JSON-LD schema, and fix bot access issues for you. Or go Elite ($2,999/mo) and get a dedicated GEO team managing everything ongoing.
See Plans & Pricing →AI is becoming the new front door for legal services. The firms that get this right now will dominate the next decade of client acquisition. The firms that wait will wonder where all their leads went.
Start with the audit. The rest will become obvious.