# Canonical configuration Status: proposed decision for C1. This document defines the target contract; it does not change current behavior. ## Decision summary Use one strict Pydantic schema as the canonical configuration model, but replace the current Pydantic field set with the fields and names actually consumed by `PipelineContext`. Keep named profiles as the user-facing unit. Resolve every run into one validated model, then project that model into the immutable runtime dataclasses used by the pipeline. The current `PstrainConfig` is not that model. It describes a second, substantially inactive configuration system. C2 should provide a migration reader for it and then remove it, rather than preserve two schemas behind a common facade. ## Current state ### The active schema `pstrain build` constructs `PipelineContext` from `project/etc/configs.yaml`. Each top-level key is a named profile. A profile contains `features`, `training`, `split`, and `runner` blocks plus a description. `FeatParams`, `TrainParams`, `TrainingSchedule`, `SplitParams`, and `RunnerParams` in `pstrain/lib/pipeline/context.py` define the accepted fields and defaults. The loader rejects unknown fields and performs several additional value and cross-field checks. This is the active schema because `pstrain/lib/pipeline/tasks.py` reads those objects when it constructs feature, split, initialization, Baum-Welch, decision-tree, packaging, and runner tasks. Effective feature, split, and training values also participate in provenance fingerprints. Built-in profiles live twice: as `DEFAULT_CONFIGS` in the Python module and in the repository's `etc/configs.yaml`. Project setup writes the Python copy into new projects. At load time, project profiles replace built-in profiles by name; the replacement is shallow at the profile boundary, while missing fields inside the selected profile fall back to dataclass defaults. Thus a project `default` profile does not inherit individual values from the built-in `default` mapping. ### The inactive Pydantic schema `PstrainConfig` and its nested models in `pstrain/lib/config/models.py` describe different names and shapes, including `audio.sample_rate`, `features.num_ceps`, `training.n_states`, and `parallel.n_jobs`. `ConfigManager` merges user, project, and experiment YAML into this model. `pstrain setup` writes `etc/config.yaml`, and the `pstrain config show|get|set|list|schema` commands operate on it. The training build does not load that merged object. For example, changing `features.num_ceps` with `pstrain config set` does not change the active `features.ncep`; changing `parallel.n_jobs` does not set the pipeline runner's `runner.jobs`. The config-reference generator imports this inactive Pydantic schema, so `docs/api/config-reference.rst` describes values that the principal training path does not consume. This is the D-bundle deferral that C1 must close. Some validation in the inactive schema now rejects stale feature and training keys and directs users toward profiles. That reduces one failure mode but does not make the schema active. ### Entry surfaces and precedence today There is no single precedence chain today; there are separate consumers: | Surface | Current consumer | Current precedence and effect | | --- | --- | --- | | Dataclass defaults | `PipelineContext` | Lowest active build defaults. | | Built-in named profiles | `load_configs` | Replace dataclass defaults for fields they contain. | | `project/etc/configs.yaml` | `load_configs` | A project profile replaces the same-named built-in profile as a whole; other built-ins remain discoverable only indirectly through an unknown-name error. | | `pstrain build -c/--config NAME` | build CLI | Selects one profile; default is `default`. It does not name a file. | | `pstrain build -j/--jobs N` | pipeline runner | Overrides `runner.jobs` for that invocation. Auto resolution is CPU count minus two. | | `pstrain build --experiment NAME` | paths and provenance | Selects experiment outputs; it does not load experiment configuration. | | `~/.pstrain/config.yaml` | inactive `ConfigManager` and config CLI | Lowest tier in the inactive merge. User defaults are translated into the inactive field names. | | `project/etc/config.yaml` | setup and inactive config CLI | Overrides user defaults in the inactive merge; ignored by `pstrain build`. | | `project/experiments/NAME/config.yaml` | inactive config CLI | Overrides project and user values when an experiment is requested; ignored by `pstrain build`. | | Setup flags | `pstrain setup` | Direct arguments for source files, linking, validation, and overwrite behavior. `--config FILE` is copied and validated as inactive `PstrainConfig`; it does not select a named build profile. | | Individual command and `pstrain step` flags | their command implementations | Direct per-command values. They bypass profile resolution and may have defaults different from the active profile (for example split flags). | | Global `--dry-run` and JSON flags | CLI framework | Invocation behavior and output only; they are not training configuration. | | `PSTRAIN_BIN_DIR`, `PSTRAIN_LIB_PATH`, `PSTRAIN_INCLUDE_DIR`, and platform library-path variables | native path discovery | Process environment overrides installation discovery, not model configuration. | | `PSTRAIN_BW_CHECKPOINTS` | Baum-Welch training | Deprecated enable-only fallback for declared `training.bw_checkpoint_iterations`. | | `PSTRAIN_REQUIRE_CLIB`, `PSTRAIN_TIMINGS_FAULT`, `PSTRAIN_GOLDEN_X86_64_STRICT`, and `PSTRAIN_BENCH_CACHE` | tests, fault injection, or benchmark harness | Operational/test controls outside ordinary project configuration. | The old `pstrain/lib/dictionary.py` module no longer exists. Dictionary code is already a package at `pstrain/lib/dictionary/`; C2 must reconcile stale references and compatibility expectations, not delete a second live module. ### The lying-surface archetype The A4 ledger found that `training.convergence_ratio` could be declared in the active profile yet never reached `run_bw_training`; commit `71da8c6` repaired that parameter plumbing along with other training and tree controls. The later A7b history treated this as a lying configuration surface: accepted syntax and plausible documentation asserted control that the engine did not honor. That failure is more serious than an unknown key because it produces a valid looking, reproducible-looking run with different semantics. ## Truthfulness invariant **A declared configuration value must reach the engine component it governs, or resolution must fail loudly before work starts.** This invariant applies to files, CLI overrides, environment adapters, generated documentation, provenance, and programmatic entry points. A field is not complete merely because it validates. Each canonical field must have: 1. a schema definition and documented default; 2. a traceable resolution source; 3. an explicit runtime consumer or an explicit designation as CLI-only metadata; 4. a test demonstrating the consumer receives a non-default value; and 5. inclusion in the relevant provenance fingerprint. CI should fail if a canonical field has no registered consumer, if a runtime consumer accepts an unregistered configuration value, or if generated reference output differs from the checked-in document. These checks make the A7b lying-config-surface class mechanically difficult to reintroduce. ## Proposed model ### Canonical schema and runtime projection Define a strict Pydantic `Profile` model whose names and nesting initially match the active profile contract: `features`, `training` with per-stage schedules, `split`, `runner`, and metadata. Pydantic is the canonical schema because it supplies validation, descriptions, JSON Schema, migration hooks, and structured introspection. The existing frozen dataclasses remain useful runtime value objects, but become generated projections with no independent defaults or validation policy. Paths and corpus resources belong in a project section of the same canonical document, outside named model profiles. Experiment-specific overrides belong in an explicit experiment overlay. This preserves reusable profiles while bringing project and experiment settings into the same resolver. Remove the present `PstrainConfig` after migration. Do not alias its mismatched fields indefinitely: aliases hide ambiguity about which default and consumer is authoritative. ### Proposed precedence From lowest to highest: 1. canonical schema defaults; 2. installed built-in profile; 3. user defaults in `~/.pstrain/config.yaml`; 4. project configuration and the selected project profile; 5. experiment overlay; 6. supported environment overrides, if any are deliberately registered; 7. explicit CLI overrides. Resolution is a deep, field-level merge after every layer has been migrated to the current schema version. Unknown keys and type errors fail with the layer's file and field path. Environment variables must not acquire generic automatic mapping; each supported variable needs a schema field, parser, documentation, and provenance policy. Test-only fault switches remain outside the canonical user contract and are labeled as such. CLI options that merely select a project, experiment, profile, output format, or dry-run mode are selectors or presentation controls, not extra schema layers. Direct semantic flags such as `--jobs` are recorded as overrides. ### Versioning and migration Every canonical project document carries `config_version`, beginning with version 1. Readers support the current version and a bounded set of older versions. Migration is deterministic, side-effect-free during inspection, and reported as old path, new path, and any changed interpretation. `pstrain config migrate --check` prints the proposed conversion without writing. `pstrain config migrate` writes a canonical file atomically and keeps a timestamped backup. Ambiguous collisions—for example both `features.num_ceps` and active `features.ncep` with different values—stop and request a choice. Removed fields stop with a reason rather than being dropped. Existing directories may contain `etc/configs.yaml`, `etc/config.yaml`, and experiment configs. The migration reader loads all three, reports which were actually effective before migration, converts active profiles first, and only maps inactive values that have an unambiguous canonical consumer. It never claims that a formerly ignored inactive value affected historical builds. ### Explainability and discovery `pstrain config explain [KEY]` resolves exactly as the requested build would. It accepts the same project, experiment, profile, and semantic override selectors as `pstrain build`. For each field it prints: - resolved value and canonical type; - winning source, including file and YAML path or exact CLI flag; - overridden candidates in precedence order; - default and validation constraints; - runtime consumer and relevant provenance scope; and - a short reason, such as “CLI override wins over project profile.” Machine-readable JSON contains stable source-kind and field-path identifiers. An unknown key, invalid layer, or declared field without a runtime consumer is an error. `pstrain config profiles` lists built-in and project profiles, descriptions, origin, schema version, and whether a project profile shadows a built-in. `pstrain config show --resolved` displays the selected effective profile; `--sources` adds source annotations. Unknown-profile errors continue to list available names. ### Generated reference and CI Generate the configuration reference and JSON Schema from the canonical Pydantic model. The generator must import the same model used by the resolver, not a documentation-only facade. Check in the rendered RST for stable docs builds, and add a CI check that regenerates it into a temporary location and fails on a diff. The same job runs schema-consumer coverage tests required by the truthfulness invariant. This completes the config-reference work deferred by the D-bundle. ## Compatibility and C2 landing order C1 is documentation only. C2 should land in this order: 1. Add the canonical versioned models, source-aware resolver, and runtime projection without changing the build CLI's effective defaults. 2. Add contract tests that inject a non-default value for every semantic field and observe it at its registered consumer and in provenance. 3. Add profile discovery, `config explain`, migration check/write commands, and canonical reference generation; wire regeneration and consumer coverage into CI. 4. Make setup and build use the resolver. During one compatibility window, read legacy files with warnings and reject ambiguous combinations. 5. Migrate maintained fixtures and examples, then remove the inactive `PstrainConfig`, `ConfigManager`, and old generator. 6. Remove `pstrain/lib/commands.py` and the legacy `Action` layer after all CLI verbs use direct library calls and the supported dry-run plan abstraction. 7. Reconcile dictionary module/package compatibility. The standalone module is already deleted; remove stale shims, imports, and design notes only after verifying the package API covers supported callers. 8. After the announced compatibility window, remove legacy readers and aliases. Migration warnings must identify the command that performs the conversion and must not be emitted for a freshly generated project. Historical provenance is left untouched; new provenance records the canonical version and sources. ## Open decisions for Kevin 1. **Canonical file layout.** Recommendation: keep `etc/configs.yaml` for named profiles and add project/experiment overlays in versioned canonical files, all parsed by one schema family. A single large file is simpler to locate but creates contention and makes reusable profile sets harder to share. 2. **User-wide semantic defaults.** Recommendation: allow them, below project layers, but require `config explain` and provenance to expose them. Removing them improves portability; retaining them supports consistent local policy across projects. 3. **Built-in shadowing.** Recommendation: deep-merge project profiles over a named built-in only when the project explicitly declares `extends`. Otherwise require a complete profile. Implicit deep merge is convenient but lets installed-version default changes alter an old project silently. 4. **Compatibility duration.** Recommendation: one minor release with legacy reads and loud warnings, followed by removal. A longer window reduces immediate migration cost but prolongs the two-schema truthfulness risk. 5. **Environment overrides.** Recommendation: keep native-library discovery and diagnostic switches outside semantic model configuration, and add no generic `PSTRAIN_*` mapping. Generic mapping is convenient in automation but makes provenance and typo detection substantially weaker. 6. **Consumer registration enforcement.** Recommendation: make missing consumer coverage a required CI failure for every semantic field. The test matrix has maintenance cost, but it directly prevents a repeat of A4/A7b.