Robots.txt & Firecrawl Configuration
Ensure Firecrawl crawler access to allow Wik AI to index your website documentation.
Chatwik uses Firecrawl internally to fetch and index your website content, allowing the Wik AI Assistant to answer customer questions accurately using your site's documentation.
If your website’s robots.txt file contains Disallow rules for crawlers, Firecrawl will skip indexing those pages, resulting in generic or incomplete AI responses.
Configuring Robots.txt for Firecrawl
Check Your Existing robots.txt
Visit https://yourdomain.com/robots.txt in your browser and check for Disallow: rules that might block Web crawlers.
Add Allow Rule for FirecrawlAgent
Add the following lines to your robots.txt file to explicitly allow Firecrawl access:
User-agent: FirecrawlAgent
Allow: /Deploy & Verify Access
Save and publish the updated robots.txt file. Re-visit https://yourdomain.com/robots.txt to confirm that the rule is live. Wik AI will now be able to crawl and index your site's documentation.