Optional modules in a Django base project: one folder, one .env switch, and a core that never names them

Every project I build starts from the same Django base: a barber-booking platform, a few payment sites, a couple of crypto sites. Over time the base gathered features only some of them needed: a wallet, online payment, payouts, SMS, crypto wallets, a REST API, support tickets. And those features leaked. A project that never sent a text message still had the SMS app in INSTALLED_APPS, a column for the SMS key in its settings table and the package in its image, and every merge from the base brought that code back.

What I ended up with is one rule and a little machinery that keeps it true.

The rule

Every optional feature is an optional module: a folder (occasionally a few) with a module.py, behind one .env switch such as USE_SMS_SERVICE. Off, it does not exist: no app, route, table, column, beat job, block on a page, CSP host or package. And no file of the core names it. The core does not know SMS or a wallet exists; it has places where it asks the modules that are on what they add.

There are twelve today, from wallet and payment to sms, api and compliance. All are off by default except the wallet: projects made before it became a switch had one, and they keep it.

module.py is data, not code

A module describes itself in one dict, MODULE. Here is the SMS module's, shortened:

MODULE = {
    "name": "sms",
    "switch": "USE_SMS_SERVICE",
    "default": False,
    "order": 70,
    "apps": ["apps.sms"],
    "requires": [],
    "paths": ["apps/sms"],
    # its pages: /vN/panel-admin/sms/
    "urls": {
        "panel_admin": {
            "prefix": "sms/",
            "v1": "apps.sms.urls.admin.v1.urls_i18n",
            "v3": "apps.sms.urls.admin.v3.urls_i18n",
        },
    },
    # "Send SMS" and SMS.ir's key, columns of SystemSetting
    "system_setting": "apps.sms.setting_models.SmsSettingMixin",
    "sections": {
        "delivery": ["apps.sms.forms.setting_forms.SmsDeliverySection"],
    },
    # what it draws on the core's pages ({version}: v1 or v3)
    "slots": {
        "admin_rail": "sms/slots/{version}/admin_rail.html",
        "delivery_detail": "sms/slots/{version}/delivery_detail.html",
        # ...
    },
    "words": "apps.sms.locales",
    "wizard": {...},  # setup_project.py's question
}

That file is never imported. MODULES_CONFIG parses it with ast and takes the dict out with ast.literal_eval:

# _0_config/configs/MODULES_CONFIG.py
def read_manifest(path):
    """`MODULE` of a module.py, without running it."""
    tree = ast.parse(Path(path).read_text(encoding="utf-8"))
    for node in tree.body:
        if (
            isinstance(node, ast.Assign)
            and isinstance(node.targets[0], ast.Name)
            and node.targets[0].id == "MODULE"
        ):
            return ast.literal_eval(node.value)


MODULES = tuple({**manifest, "on": _switched_on(manifest)} for manifest in discover())
SWITCHES = {module["switch"]: module["on"] for module in MODULES}
INSTALLED_APPS = [app for module in MODULES if module["on"] for app in module["apps"]]

The reason is who reads it, and when. One reader is settings.py, before Django is set up: importing apps.sms.module would first import the app's package, whose code may expect a ready Django, or a dependency that is not installed while the module is off. The other is setup_project.py, the wizard that creates and updates projects; it runs on the host, outside the containers, and runs no line of the project. And since literal_eval accepts only literals, a module.py cannot quietly grow logic. Each reader has its own parser; a test checks that both see the same modules.

# _0_config/settings.py
MODULES = MODULES_CONFIG.MODULES
globals().update(MODULES_CONFIG.SWITCHES)
# ... the core's settings ...
MODULES_CONFIG.contribute(globals())  # last, so a module sees them whole

At the end of the settings, contribute lets each module that is on add the rest: a package's apps and middleware, CSP hosts (add_csp), or a check that stops start-up; the payment module will not start without its package, or outside Toman and Rial. Dependencies are checked here too: payout on without wallet is an ImproperlyConfigured.

Where the core asks

For each kind of thing a module may add, the core has one place, and there it asks only the modules that are on (module_function.py):

So the root URLconf says nothing about modules beyond urlpatterns += module_urls("root"), and the security settings page has {% module_slot "security_detail" %} where a module's block may go. A slot no module fills draws nothing, which is how the core's templates name no module.

money_refusal shows how far this goes: every flow that moves money asks it first. The compliance module, when on, may answer with a sentence saying why not and where to act; with no module answering, every account moves money, and the payment code never learns such a module exists.

Settings columns that exist only while a module is on

The site's editable settings are columns of SystemSetting, but a project that never turns SMS on should not carry a column for its key. So each module's columns are an abstract mixin in its own folder, and the model is built from the mixins of the modules that are on:

# _1_system_setting/models/models.py
def service_mixins():
    """The mixins of the modules switched on, in their order."""
    return module_setting_mixins()


class SystemSetting(mb.BasicModelHistorical, mx.SitePreferenceMixin, *service_mixins()):
    ...


# apps/sms/setting_models.py
class SmsSettingMixin(models.Model):
    use_sms_service = models.BooleanField(default=False, db_default=False, ...)
    sms_api_key_encrypted = models.TextField(
        blank=True, default="", db_default="", ...
    )

    SENSITIVE_FIELDS = ("sms_api_key_encrypted",)

    class Meta:
        abstract = True

The class is built once per process, from what the deployment switched on; since migrations are generated per project, one made with a switch on carries that module's columns. On is the switch, docker compose up -d, makemigrations and migrate; off is the same, and that migrate drops the columns with their values, leaves the module's own tables unused and deletes its beat jobs. Two rules keep this safe:

• Every column has a db_default beside its default. Between switching off and the next migrate, the model has forgotten the columns but the table has not. A row written in that window, such as the history row every save writes, gets the database's default instead of failing on NOT NULL.
• A column names no code of its module: no callable default, validator or field class. SystemSetting's migrations belong to the core; one that imported the module would stop migrate in a project that later removed it. A list default is functools.partial(list, …) over plain data.

In a project: removing and adding

Off is not enough: a project that does not use a module should not carry its folder either. The wizard deletes the folders of the modules a new project did not switch on, in a commit of their own, and records their switch, .env keys and paths in _0_deploy/removed_modules.json. After that:

python3 src/core/setup_project.py --modules
python3 src/core/setup_project.py --remove-module sms --dry-run
python3 src/core/setup_project.py --remove-module sms
python3 src/core/setup_project.py --add-module sms
python3 src/core/setup_project.py --update

• --update merges the base's latest and deletes again whatever of a removed module comes back, as a modify/delete conflict or a new file, so only real conflicts are left.
• --add-module brings a module back as of the base commit the project last merged, so it matches the core around it.
• --remove-module wants the switch off first, leaves the tables until migrate <app> zero, and refuses a module another one needs: the wallet, while payouts are there.
• A removed module's switch turned on stops start-up with a message giving the way back.

Tests that hold the rule

A rule that lives only in the docs does not last. A module's tests live in its folder and are collected only while it is on: the root conftest.py skips the folders of modules that are off, since most of their tests cannot even be imported then, except test_*_switch_off.py, which hold the off state.

# conftest.py, at the root
OFF_MODULES = [module for module in MODULES if not module["on"]]
OFF_PATHS = [path.rstrip("/") for module in OFF_MODULES for path in module["paths"]]
SWITCH_OFF_TESTS = "test_*_switch_off.py"


def pytest_ignore_collect(collection_path, config):
    ...  # not inside an off module's folder: None
    if Path(collection_path).is_dir():
        return True if not any(Path(collection_path).rglob(SWITCH_OFF_TESTS)) else None
    return None if fnmatch(Path(collection_path).name, SWITCH_OFF_TESTS) else True

The price is that a test cannot turn a module on with override_settings, since SystemSetting is already built. The normal run has every module off but the wallet; for the on state I run the suite with the switches set and --nomigrations, because the local migrations were made with them off. test_optional_modules.py holds the rest:

• the settings and the wizard read every module.py alike;
• a module that is off leaves no app, URL, column or word behind;
• each module's columns serialize without any of its code;
• no core file names a module (its package, folders, switch, templates, static files, URL names), and a module names only the modules it requires;
• a copy of the project without one module's folders, and one without all of them, pass Django's check.

@pytest.mark.parametrize("module", MODULES, ids=IDS)
def test_nothing_of_the_core_names_a_module(module):
    core = [(rel, text) for rel, text in _files() if not _owners(rel)]
    named = _named(module["name"], core)
    assert not named, (
        f"{module['name']} is named outside its folders — removed, it would "
        f"leave these behind: {named}"
    )

It builds its patterns from each module's own module.py and searches the core's code; a module's URL name written into a core template fails it, naming the file and what a project without that module would be left holding.

What it costs, and what it buys

It is not free: two parsers to keep in step, a place to build in the core before a module can use it, and a suite that runs in more than one configuration. In return, a new project has exactly the folders it uses, updates from the base do not bring the rest back, and adding a feature later is one command, a switch and a migration. A feature other projects could use is built in the base as a module first, then merged in.