Web crawler
Also called Crawler, Spider, Bot
A web crawler is a program that visits web pages automatically, reads their content and follows links or sitemaps to discover more pages. Search engines, AI companies and site tools use crawlers. Well-behaved crawlers identify themselves and respect a site's robots.txt file and noindex rules.
Example
Googlebot reads a site's sitemap.xml, fetches each listed page, and follows links to pages the sitemap missed.
Why it matters for a website chatbot
How a chatbot tool crawls your site decides what it knows. ChatBento finds your pages through your sitemap, including any listed in robots.txt, and your links, then reads each page. You can add pages that aren't linked anywhere one at a time.
See it in practice: Docs and help-center search
Related terms
- Knowledge baseA knowledge base is an organized collection of information, such as help articles, policies, product details and answers to common questions, that people or software can search. In AI chatbots, the knowledge base is the content the chatbot is allowed to answer from, usually built from a company's website, documents and written Q&As.
- llms.txtllms.txt is a proposed web standard for a Markdown file placed at a site's root, at /llms.txt, that gives large language models a concise summary of the site and links to its most useful pages. Jeremy Howard of Answer.AI proposed it in September 2024. Support among AI companies is still uneven.
- Answer engine optimization (AEO)Answer engine optimization (AEO) is the practice of structuring web content so AI answer engines, such as ChatGPT, Perplexity and Google AI Overviews, can find, understand and cite it. It favors direct answers near the top of a page, clear headings, accurate facts, structured data and pages that crawlers can read.