# Constile gate for Python The Constile gate serves the Constile offer to AI agents that fetch your pages for a person. Everyone else, including people, search engines, and training crawlers, gets your site as usual. This package runs the gate inside your Python app. It works with Django, Flask, FastAPI, Starlette, and any other WSGI or ASGI app. It needs Python 3.9 or later and nothing outside the standard library. ## How the gate works - **People** reach your app untouched. Their responses stream from your app as before. The gate reads only the site config, which it caches. - **Agents** on an eligible page: the gate asks your app for the page in-process, reads its title, description, keywords, and section, and asks the Constile network for an offer. If the network has one, the agent gets the offer. If not, the agent gets the page. - **`/sponsored/*` and `/.well-known/constile`** are answered by the gate. - **When the network is down or slow**, visitors get the page as if the gate weren't there. The gate waits at most 5 seconds by default. After it fails to get the site config, it doesn't ask again for 30 seconds, so an outage doesn't slow your pages down. ## Before you start You need three values from your site's page in the Constile dashboard: the site ID, the site secret, and the network URL. Set them as environment variables where your app runs: ```sh export CONSTILE_SITE_ID="your-site-id" export CONSTILE_SITE_SECRET="your-site-secret" export CONSTILE_NETWORK_URL="https://your-network-url" ``` Keep the site secret out of your repository. Store it the way you store your other secrets, such as in your host's secret settings. If any of the three is missing, the gate passes every request through and logs one warning. ## Install the package To install the package, download it from your dashboard, or install it from the network's URL. Replace `NETWORK_URL` with your network URL: ```sh pip install "NETWORK_URL/install/constile_gate-0.6.0-py3-none-any.whl" ``` Then add the gate to your app with one of the following sections. ## Django 1. In `settings.py`, add the middleware first in `MIDDLEWARE`: ```python MIDDLEWARE = [ "constile_gate.django.ConstileMiddleware", # ...your other middleware ] ``` Putting it first means it runs before page caches and redirects. 1. Set the three values as environment variables, or in `settings.py`: ```python import os CONSTILE_SITE_ID = os.environ["CONSTILE_SITE_ID"] CONSTILE_SITE_SECRET = os.environ["CONSTILE_SITE_SECRET"] CONSTILE_NETWORK_URL = os.environ["CONSTILE_NETWORK_URL"] ``` The gate builds each page's URL with `request.build_absolute_uri()`, so your `ALLOWED_HOSTS` and `SECURE_PROXY_SSL_HEADER` settings apply. ## Flask Wrap your app with the extension: ```python from flask import Flask from constile_gate.flask import Constile app = Flask(__name__) Constile(app) ``` If you use an app factory, create the extension once and call `init_app`: ```python constile = Constile() def create_app(): app = Flask(__name__) constile.init_app(app) return app ``` The extension reads `CONSTILE_SITE_ID`, `CONSTILE_SITE_SECRET`, and `CONSTILE_NETWORK_URL` from `app.config` first, then from the environment. If you use Werkzeug's `ProxyFix`, apply it after the extension, so it runs first and the gate sees the corrected address: ```python Constile(app) app.wsgi_app = ProxyFix(app.wsgi_app, x_for=1, x_proto=1, x_host=1) ``` ## FastAPI Add the middleware after your other middleware, so it's the outermost: ```python from fastapi import FastAPI from constile_gate.fastapi import ConstileMiddleware app = FastAPI() # app.add_middleware(...your other middleware) app.add_middleware(ConstileMiddleware) ``` The gate's network calls run in threads, so they never block the event loop. ## Starlette Put the middleware first in the list, so it's the outermost: ```python from starlette.applications import Starlette from starlette.middleware import Middleware from constile_gate.starlette import ConstileMiddleware app = Starlette(routes=routes, middleware=[Middleware(ConstileMiddleware)]) ``` ## Any other WSGI or ASGI app To gate a WSGI app, such as Bottle or Pyramid, wrap it: ```python from constile_gate.wsgi import ConstileMiddleware application = ConstileMiddleware(application) ``` To gate an ASGI app, such as Quart or Django's ASGI handler, wrap it the same way with `constile_gate.asgi.ConstileMiddleware`. ## Settings Each setting comes from, in order: the middleware's keyword argument, your framework's settings (Django settings or Flask `app.config`), and the environment variable. | Environment variable | Keyword argument | Description | |---|---|---| | `CONSTILE_SITE_ID` | `site_id` | Required. Your site ID. | | `CONSTILE_SITE_SECRET` | `secret` | Required. Your site secret. | | `CONSTILE_NETWORK_URL` | `network_url` | Required. The network URL. | | `CONSTILE_TRUSTED_PROXIES` | `trusted_proxies` | Optional. The addresses or networks of proxies in front of your app, such as `10.0.0.0/8, 192.0.2.1`. | | `CONSTILE_TIMEOUT` | `timeout` | Optional. Seconds to wait for the network before serving the page anyway. The default is 5. | | `CONSTILE_RETRY_AFTER` | `retry_after` | Optional. After the gate fails to get the site config, seconds to pass every request through before it asks again. The default is 30. | | `CONSTILE_DISABLED` | `disabled` | Optional. `1`, `true`, `yes`, or `on` turns the gate off: every request goes to your app unchanged, and nothing goes to the network. Read on every request. | The gate logs to the `constile_gate` logger. ## The visitor's address For a request that claims to come from a registered agent, the gate sends the visitor's address to the network. The network checks it against the agent operator's published addresses, so a request that only claims to be an agent earns no money. The gate reads the address from the connection: `REMOTE_ADDR` for WSGI and Django, and the `client` of the ASGI scope. It reads `X-Forwarded-For` only when you list your proxies in `CONSTILE_TRUSTED_PROXIES`. It then reads the header from the right, skips your proxies, and takes the first address that isn't one of them. A visitor can't forge that address, because your proxies add it. If something in front of the gate already resolves the address, leave `CONSTILE_TRUSTED_PROXIES` unset. For example, uvicorn with `--forwarded-allow-ips` set to your proxies, and Werkzeug's `ProxyFix`, both set the address the gate reads. ## Placement and caches - Install the gate at the root of your host. The gate answers `/sponsored/*` and `/.well-known/constile` there. An app mounted under a sub-path, such as `/blog`, can't answer them. - Keep page caches inside the gate, so an agent never gets a cached page. In Django, list the gate before `UpdateCacheMiddleware`. If a cache sits in front of your app, such as a CDN or nginx's `proxy_cache`, make it skip requests whose `User-Agent` names a registered agent or that carry a `Signature` header. Offers carry `Cache-Control: no-store` and `Vary: User-Agent`, so caches don't store them. - When the gate asks your app for an agent's page, it leaves out the agent's `Accept-Encoding` header, so your app answers uncompressed. If your app compresses anyway, the gate reads gzip and deflate, but not Brotli or Zstandard: an agent then gets the page without an offer. People's requests keep their `Accept-Encoding`, so they still get compressed pages. - The gate reads a page's title and description in the charset its `Content-Type` header names, such as `charset=iso-8859-1`, and as UTF-8 when it names none. A page it passes through keeps its exact bytes. ## Paywalled content Gate 0.6.0 and later can fetch a paywalled article with the release instructions the network sends, so an agent gets the full text and the offer fits it. The network gives the gate a page address on your site's own host and, optionally, headers to send: `Cookie`, `Authorization`, `Constile-Paywall-Key`, or a header starting with `X-`. The gate ignores an address on another host and drops other headers. It strips `Set-Cookie` from what it releases and never logs the headers. To set up a method, open your site's page in the dashboard and choose a content API, a paywall rule, or a Constile login. A paywalled page gets offers only after the method works and the gate is 0.6.0 or later. ## Check the install Open `https://your-site/.well-known/constile`. You see JSON that names your site ID, `"gate_version": "constile-gate 0.6.0"`, and the platform, such as `"django"`. The install check on your dashboard reads the same address. ## Develop the gate The gate's behavior is pinned by the shared conformance cases in `packages/gate/conformance/cases.json`. The runner in `tests/test_conformance.py` replays every case against this port. - To run the tests, run `./test.sh`. To include the framework tests, first install `tests/requirements.txt`. - To run the tests under Python 3.9 and 3.12 in Docker, run `./test.sh docker 3.9 3.12`. - To build the sdist and the wheel, run `./build.sh [outdir]`. The default `outdir` is `apps/network/public/install/` at the repository root.