Constile
How it worksSetup guidesEarningsAdvertisers
Sign inStart earning

Constile-Crawler

Constile-Crawler is the program Constile uses to read the pages of a publisher's site, so it can match ads to what each page covers. It identifies itself with this user agent:

Mozilla/5.0 (compatible; Constile-Crawler/1.0; +https://constile.app/crawler)

Which sites it reads

It reads only sites whose owners installed the gate and confirmed it in their Constile dashboard. It doesn't crawl the open web, and it doesn't follow links off the site.

How it behaves

  • It reads your robots.txt first and obeys it.
  • It finds pages from your sitemap, or from your homepage's links when there's no sitemap.
  • It waits about a second between requests to your site, or the Crawl-delay in your robots.txt if that's longer.

What it keeps

For each page, it keeps the address, title, meta description, language, and word count. It also keeps a summary, a kind (such as article or product), a category, topics, keywords, and a vector that stands for the page's meaning. It doesn't keep the text.

To work out those details, the network sends each page's text to an AI model provider through OpenRouter. It uses only providers that don't store or train on it, and drops the text afterward. When a page is paywalled, the text goes only to providers that keep no data at all.

Allow it, or keep it out

The crawler follows the rules for User-agent: Constile. If your robots.txt has no group for Constile, it follows the rules for *. The longest matching rule wins, and Allow wins a tie.

To keep it out of part of your site:

User-agent: Constile
Disallow: /members/

To keep it out of the whole site:

User-agent: Constile
Disallow: /

If a blanket Disallow under * blocks it, allow it by name:

User-agent: Constile
Allow: /

To slow it, add a Crawl-delay in seconds to the same group. The crawler honors delays up to 60 seconds and ignores anything longer or invalid:

User-agent: Constile
Crawl-delay: 10

You can also exclude any single page on the Pages section of your site's page in the Constile dashboard.

Questions

If the crawler reads a site it shouldn't, or you want a page's record removed, tell us through your dashboard, or see the privacy policy.

Constile, the Sponsored Access Turnstile, is in early access. Operated by CCC2C Inc.

For publishersSetup guidesFor advertisersTermsPrivacyReport an adAd policySign in
Constile