Several Django sites on one server behind a CDN: the real client IP, hashed static files, the deployment .env

I keep several sites on one server. Each is a Django project with its own Celery and Redis, and all of them sit behind a CDN: Cloudflare or ArvanCloud. This is the layout, and the handful of details that have each broken something at least once.

The layout

Everything lives in one services/ folder under a single docker-compose.yml, on a server prepared by setup-server.sh. Each site's Docker networks and data folder are its own as well.

Why each site keeps its own Redis and Celery

The only Redis knob in the code is the container's name, REDIS_CONTAINER; the database numbers are fixed: the broker on /1, the cache on /2, sessions on /3. Two sites on one Redis would share one queue and one cache. The worker and beat run from the site's own image and .env, so a worker knows only the tasks of its own code, and beat reads its schedule from that site's database through DatabaseScheduler. Trouble stays local too: a site is rebuilt on its own by naming its services, and without the names one failed build keeps them all down.

docker compose up -d --build site_one site_one_celery_worker site_one_celery_beat

nginx in front, the CDN in front of nginx

A request goes visitor → CDN → nginx → gunicorn. nginx reaches gunicorn over the Docker network, so two sites can both listen on 8000. The decision that matters most is that nginx depends on no site: the container's name reaches proxy_pass through a variable, so Docker's DNS (resolver 127.0.0.11) looks it up per request, not when nginx starts. A site that is down is a 502 on its own domain only.

server {
    listen 443 ssl;
    server_name example.com www.example.com;
    ssl_certificate     /etc/nginx/ssl/example.com/origin.pem;
    ssl_certificate_key /etc/nginx/ssl/example.com/origin-key.pem;

    location /static/ { alias /projects/volumes/site_one/static/; }

    location / {
        set $upstream site_one:8000;     # looked up per request
        proxy_pass http://$upstream;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

A host name no file claims, such as the server's bare IP, gets return 444 and never reaches Django. And nginx -t before every reload: on a broken config nginx will not start, and every site goes down.

Cloudflare and ArvanCloud differ

• Cloudflare talks HTTPS to the server, which holds an Origin Certificate issued by Cloudflare (15 years, no renewal), and the SSL mode must be Full (strict). On Flexible it arrives on port 80, nginx sends it back to https, and the site loops.
• ArvanCloud holds and renews the certificate itself and talks plain HTTP to the server. nginx's $scheme is therefore always http; passed on as X-Forwarded-Proto, it makes Django redirect every https request to https again. nginx has to hand on the protocol the visitor came with:

# 00-http.conf
map $http_x_forwarded_proto $visitor_proto {
    https   https;
    default http;
}

# the site's file, in location /
proxy_set_header X-Forwarded-Proto $visitor_proto;

The visitor's real IP: TRUSTED_PROXY_COUNT

Each proxy appends to X-Forwarded-For the address it was reached from. Behind a CDN and nginx, if the visitor also sends a forged header, Django sees:

X-Forwarded-For: 198.51.100.7, 203.0.113.10, 192.0.2.44
# 198.51.100.7   written by the visitor: anything at all
# 203.0.113.10   the visitor, appended by the CDN
# 192.0.2.44     the CDN edge, appended by nginx

The visitor is TRUSTED_PROXY_COUNT entries from the right; anything further left is the visitor's own text. The code used to take the leftmost entry. One forged header was enough to slip past the sign-in slider puzzle's per-address limit, to aim that limit at somebody else's address and lock them out, and to put an invented IP on a sign-in notification. Now get_client_ip() reads:

address = request.META.get("REMOTE_ADDR")
proxies = getattr(settings, "TRUSTED_PROXY_COUNT", 0)
if proxies > 0:
    forwarded = request.META.get("HTTP_X_FORWARDED_FOR", "")
    hops = [hop.strip() for hop in forwarded.split(",") if hop.strip()]
    if hops:
        address = hops[-proxies] if len(hops) >= proxies else hops[0]
return _valid_ip(address) or _valid_ip(request.META.get("REMOTE_ADDR"))

Only the deployment knows the right number, so with DEPLOYED on it has no default. Too low, and every visitor is the proxy, all sharing one rate limit; too high, and the visitor picks the address. A domain whose DNS record skips the CDN (a grey cloud on Cloudflare) is 1.

Hashed static files

A CDN edge caches a static file and keeps serving it; the fix is giving every version of a file a new URL. Deployed, collectstatic writes each file again under a hash of its content (app.css → app.3f2a1b9c0d4e.css), records the pair in staticfiles.json, rewrites the url()s inside CSS, and {% static %} hands out the hashed name. (The setting is STORAGES; since Django 5.1, STATICFILES_STORAGE is ignored without a word.)

Why a forgiving subclass, ForgivingManifestStaticFilesStorage? Django's class raises on a reference to a file that does not exist. collectstatic stops at the first one (the vendor CSS has hundreds: flag-icons' SVGs, KaTeX's fonts, .map files), and because the start script runs under set -e, the container never starts. With DEBUG=False, a {% static %} of a missing file is a 500 for the whole page. The subclass leaves such a reference as written, so it stays a 404 for that one file; manifest_strict = False alone did not get there.

It also hashes with DEBUG on. Under Django's class, a deployment left on DEBUG to chase a problem served an edited stylesheet under the URL the browser and the CDN already had.

collectstatic on every start

The image's entrypoint, _0_deploy/script.sh, does this on every start. The Celery containers pass a command, which the script execs:

if [ "$#" -gt 0 ]; then exec "$@"; fi        # celery worker / beat

while ! nc -z ${DATABASES_HOST:-postgres_service} ${DATABASES_PORT:-5432}; do
    sleep 1
done
python manage.py makemigrations --noinput
python manage.py migrate --noinput
python manage.py collectstatic --noinput
python manage.py translate_sync
exec gunicorn _0_config.wsgi -b 0.0.0.0:"$GUNICORN_BIND_PORT" \
    --workers "${GUNICORN_WORKERS:-3}" --timeout "${GUNICORN_TIMEOUT:-60}"

collectstatic keeps the unhashed file and every earlier hash too, so a page cached with an old URL loads. translate_sync is there because the .mo files are git-ignored.

What a deployment's .env must say

Each site has one .env, handed by compose's env_file to the app, the worker and beat, and kept out of the image. DEPLOYED="True" turns the development defaults off instead of trusting the file to have overridden each one:

• SECRET_KEY, ALLOWED_HOSTS, CSRF_TRUSTED_ORIGINS and TRUSTED_PROXY_COUNT have no default. A missing one stops the start-up naming the key, rather than serving a site on the published "Test" key, open to any host. DEBUG defaults to off.
• Without USE_STATIC_ROOT="True" and STATIC_ROOT, the start-up stops with ImproperlyConfigured; the manifest of hashed names lives there.
• USE_SSL_CONFIG defaults to DEPLOYED: secure cookies, HSTS, the https redirect and SECURE_PROXY_SSL_HEADER. /health/ is exempt from the redirect, because the container's healthcheck asks over plain HTTP and the Celery services wait for it.
• USE_SQLITE_DB="False"; otherwise the app migrates one SQLite file and beat reads another.
• SALT_PASSWORD is set once, before the first account; changing it later locks every account out.

DEBUG="False"
DEPLOYED="True"
SECRET_KEY='-----'
ALLOWED_HOSTS="example.com,www.example.com"
CSRF_TRUSTED_ORIGINS="https://example.com,https://www.example.com"
TRUSTED_PROXY_COUNT="2"
USE_SSL_CONFIG="True"
USE_SQLITE_DB="False"
DATABASES_HOST="postgres_service"
REDIS_CONTAINER="site_one_redis"
USE_STATIC_ROOT="True"
STATIC_ROOT="/projects/volumes/static/"
SERVE_MEDIA_STATIC="False"

An edit reaches the containers with docker compose up -d, not with a restart.

Celery on a small server

By default Celery forks one process per CPU, and each loads all of Django (about 115 MB); on a 22-core laptop, one project's idle worker held 1.7 GB. So there are three limits:

• CELERY_WORKER_CONCURRENCY, default 2, not the core count.
• CELERY_WORKER_MAX_MEMORY_PER_CHILD, default 300000 KB; a process that grows past it is replaced once its task is done.
• mem_limit: 1g on the worker's container in compose.

Bringing a new site up

1. Copy one site's four services in docker-compose.yml under the new name, with its networks and its volume lines under nginx.
2. Copy an nginx file for the new domain; behind Cloudflare, add its Origin Certificate.
3. Clone the project into projects/ and fill in its .env.
4. Create its database by hand: POSTGRES_EXTRA_DATABASES is read only on Postgres's first start.
5. Bring up that site's services, run nginx -t, then recreate nginx.
6. init and createsuperuser: on a server, init creates no admin with the password 123.

docker compose exec postgres_service createdb -U <POSTGRES_USER> site_three_db
docker compose up -d --build site_three site_three_celery_worker site_three_celery_beat
docker compose exec nginx nginx -t
docker compose up -d nginx
docker compose exec site_three python manage.py init
docker compose exec site_three python manage.py createsuperuser