Constile-Crawler
Constile-Crawler is the program Constile uses to read the pages of a publisher's site, so it can match ads to what each page covers. It identifies itself with this user agent:
Mozilla/5.0 (compatible; Constile-Crawler/1.0; +https://constile.app/crawler)Which sites it reads
It reads only sites whose owners installed the gate and confirmed it in their Constile dashboard. It doesn't crawl the open web, and it doesn't follow links off the site.
How it behaves
- It reads your robots.txt first and obeys it.
- It finds pages from your sitemap, or from your homepage's links when there's no sitemap.
- It waits about a second between requests to your site, or the
Crawl-delayin your robots.txt if that's longer.
What it keeps
For each page, it keeps the address, title, meta description, language, and word count. It also keeps a summary, a kind (such as article or product), a category, topics, keywords, and a vector that stands for the page's meaning. It doesn't keep the text.
To work out those details, the network sends each page's text to an AI model provider through OpenRouter. It uses only providers that don't store or train on it, and drops the text afterward. When a page is paywalled, the text goes only to providers that keep no data at all.
Allow it, or keep it out
The crawler follows the rules for User-agent: Constile. If your robots.txt has no group for Constile, it follows the rules for *. The longest matching rule wins, and Allow wins a tie.
To keep it out of part of your site:
User-agent: Constile
Disallow: /members/To keep it out of the whole site:
User-agent: Constile
Disallow: /If a blanket Disallow under * blocks it, allow it by name:
User-agent: Constile
Allow: /To slow it, add a Crawl-delay in seconds to the same group. The crawler honors delays up to 60 seconds and ignores anything longer or invalid:
User-agent: Constile
Crawl-delay: 10You can also exclude any single page on the Pages section of your site's page in the Constile dashboard.
Questions
If the crawler reads a site it shouldn't, or you want a page's record removed, tell us through your dashboard, or see the privacy policy.