A crawler blocked in your theme
- The situation
- Your tea pages never show up as a source, even for blends only you sell. Years ago a developer edited the robots.txt template in your theme, and nobody has looked at it since.
- What Rankfender shows you
- The AI-crawler check in the audit lists each crawler on its own line: Google-Extended is allowed, while GPTBot and ClaudeBot are disallowed by a rule in your theme. It sits near the top of the fix list.
- What you do next
- Ask RAISA for robots.txt rules that let AI crawlers in, have them added to the robots.txt template of your theme, then re-run the audit to confirm each crawler is allowed.