pstrain

User Guide

  • Getting started
    • Install the current checkout
    • Train a small model
    • Install a published release instead
    • Further workflows
      • Supplying an existing train/test split
      • Python API
  • Input formats
    • Pronunciation dictionary
    • Transcripts
    • Phoneset
    • Audio
  • Examples
    • Project Setup
    • Project Validation
    • Model Creation
    • Data Structures
    • Log-domain Math
  • Alignment mass and coverage report
    • Where it appears
    • Outcomes
    • What is counted
    • Flags
    • Size
  • Support and dependency policy
    • Platforms
      • Windows Python coverage
    • Dependencies
  • Package replacement safety
    • Concurrent pathname changes
  • Glossary
    • Model Types
      • CI (Context-Independent)
      • CD (Context-Dependent)
      • Tied vs Untied
    • Core Concepts
      • HMM (Hidden Markov Model)
      • GMM (Gaussian Mixture Model)
      • State
      • Senone (Tied State)
    • Features
      • MFCC (Mel-Frequency Cepstral Coefficients)
      • Cepstrum
      • Delta / Delta-Delta
      • CMN (Cepstral Mean Normalization)
      • AGC (Automatic Gain Control)
    • Training Algorithms
      • Baum-Welch
      • Viterbi
      • Forced Alignment
    • Model Parameters
      • Means / Variances
      • Mixture Weights
      • Transition Matrices (tmat)
      • Density
    • Files and Formats
      • mdef (Model Definition)
      • ctl (Control File)
      • fileids
      • transcription
      • feat.params
      • sendump
    • Training Stages
      • Flat Initialization
      • Convergence
      • Iteration
      • Gaussian Splitting
    • Decision Trees
      • Question
      • Quest File
      • Pruning
    • Abbreviations Reference

API Reference

  • API Reference
    • Library API
      • Public API
        • CDModel
        • CIModel
        • CMUDict
        • CMUDictSource
        • Dictionary
        • FeatureConfig
        • FileType
        • Model
        • ModelCompareResult
        • Phoneset
        • Profile
        • PstrainPaths
        • TrainingConfig
        • TutorialExistsError
        • TutorialResult
        • ValidationReport
        • build_lm()
        • build_lm_from_file()
        • compare_auto()
        • compare_features()
        • compare_gaussians()
        • compare_mixw()
        • compare_models()
        • compare_tmat()
        • copy_tutorial()
        • create_model()
        • default_cmudict_cache()
        • describe_file()
        • detect_file_type()
        • extract_features()
        • fetch_cmudict()
        • get_fileids()
        • get_model_class()
        • get_paths()
        • init_flat_model()
        • package_model()
        • parse_transcription_file()
        • print_stats()
        • resolve_config()
        • run_build_lm()
        • run_step_cd_hmm_untied()
        • run_step_ci_hmm()
        • run_step_features()
        • setup_project()
        • step_cd_hmm_untied()
        • step_ci_hmm()
        • step_features()
        • strip_dictionary_stress()
        • strip_stress()
        • validate_file_type()
        • validate_package_destination()
        • validate_project()
        • TUTORIAL_FILENAME
      • Project Setup
      • Project Validation
      • Configuration
      • Data Structures
        • Dictionary
        • Phoneset
        • Transcription
      • Models
      • Native Worker
      • Guarded Logmath Wrapper
    • Alignment API
      • AlignmentCoverage
        • AlignmentCoverage.__init__()
        • AlignmentCoverage.format()
        • AlignmentCoverage.lost_phones
        • AlignmentCoverage.lost_triphones
        • AlignmentCoverage.n_duration_unknown
        • AlignmentCoverage.n_flags
        • AlignmentCoverage.outcomes
        • AlignmentCoverage.phone_carriers
        • AlignmentCoverage.phones
        • AlignmentCoverage.phones_from_alignment
        • AlignmentCoverage.retry_only_phones
        • AlignmentCoverage.retry_only_triphones
        • AlignmentCoverage.run_failure_rate
        • AlignmentCoverage.seconds
        • AlignmentCoverage.speakers
        • AlignmentCoverage.speakers_without_aligned_audio
        • AlignmentCoverage.thin_min_carriers
        • AlignmentCoverage.thin_phones
        • AlignmentCoverage.thin_rate_ratio
        • AlignmentCoverage.thin_tokens
        • AlignmentCoverage.to_dict()
        • AlignmentCoverage.triphones
        • AlignmentCoverage.unexpanded_words
        • AlignmentCoverage.utterances
      • align_corpus()
      • alignment_coverage()
      • collect_phone_report()
      • load_transcripts()
      • save_ctm()
      • save_textgrid()
    • Checkpoint recovery API
      • list_checkpoints()
      • restore_checkpoint()
    • Configuration API
      • generate_markdown_docs()
      • generate_rst_docs()
      • get_schema()
      • get_user_config()
      • list_parameters()
      • list_profiles()
      • migrate_project()
      • resolve_config()
      • CURRENT_CONFIG_VERSION
      • User settings
    • Corpus API
      • split_is_external()
      • train_test_split()
      • validate_external_split()
    • Diagnostics API
      • fp_contract_policy()
      • learning_curves()
      • load_bw_telemetry()
      • native_library_available()
      • resolve_binary()
      • PSTRAIN_BINARIES
    • Dictionary API
      • default_cmudict_cache()
      • fetch_cmudict()
      • CMUDictSource
    • Language model API
      • build_lm()
      • build_lm_from_file()
      • load_transcripts()
    • One-command Training API
      • PromptFormatError
      • identity_difference()
      • input_identity()
      • installed_corpus_identity()
      • validate_inputs()
      • write_training_transcription()
      • write_validation_reports()
      • PROMPT_FORMATS
    • Packaging API
      • create_noisedict()
      • package_model()
      • validate_package_destination()
    • Pipeline API
      • PipelineContext
        • PipelineContext.__init__()
        • PipelineContext.all_fileids()
        • PipelineContext.all_transcription
        • PipelineContext.architecture_dir
        • PipelineContext.audio_dir
        • PipelineContext.audio_fileids()
        • PipelineContext.config_name
        • PipelineContext.description
        • PipelineContext.dist_dir
        • PipelineContext.etc_dir
        • PipelineContext.experiment
        • PipelineContext.experiment_dir
        • PipelineContext.feat
        • PipelineContext.features_dir
        • PipelineContext.filler_dict
        • PipelineContext.filler_dict_path
        • PipelineContext.fingerprint_payload()
        • PipelineContext.from_config()
        • PipelineContext.lm_dir
        • PipelineContext.model_dir()
        • PipelineContext.model_files()
        • PipelineContext.models_dir
        • PipelineContext.project_dir
        • PipelineContext.provenance_document()
        • PipelineContext.provenance_path()
        • PipelineContext.provenance_payload()
        • PipelineContext.read_fileids()
        • PipelineContext.reports_dir
        • PipelineContext.resolved_config
        • PipelineContext.runner
        • PipelineContext.sharding
        • PipelineContext.shared_dir
        • PipelineContext.split
        • PipelineContext.train
        • PipelineContext.trees_dir
        • PipelineContext.validate_training_features()
      • UnknownTargetError
      • build_pipeline()
      • create_pipeline_context()
      • run_pipeline()
      • TARGETS
      • DEFAULT_TARGET
      • DEFAULT_CONFIGS
    • Training Steps API
      • CDHMMUntiedStep
        • CDHMMUntiedStep.add_arguments()
        • CDHMMUntiedStep.default_params
        • CDHMMUntiedStep.description
        • CDHMMUntiedStep.execute()
        • CDHMMUntiedStep.get_inputs()
        • CDHMMUntiedStep.get_outputs()
        • CDHMMUntiedStep.name
        • CDHMMUntiedStep.script
      • CIHMMStep
        • CIHMMStep.add_arguments()
        • CIHMMStep.default_params
        • CIHMMStep.description
        • CIHMMStep.execute()
        • CIHMMStep.get_inputs()
        • CIHMMStep.get_outputs()
        • CIHMMStep.get_params_from_args()
        • CIHMMStep.name
        • CIHMMStep.script
      • FeaturesStep
        • FeaturesStep.add_arguments()
        • FeaturesStep.default_params
        • FeaturesStep.description
        • FeaturesStep.execute()
        • FeaturesStep.get_inputs()
        • FeaturesStep.get_outputs()
        • FeaturesStep.name
        • FeaturesStep.script
      • Step
        • Step.add_arguments()
        • Step.default_params
        • Step.description
        • Step.execute()
        • Step.get_definition()
        • Step.get_inputs()
        • Step.get_outputs()
        • Step.get_params()
        • Step.get_params_from_args()
        • Step.main()
        • Step.name
        • Step.run()
        • Step.script
        • Step.to_dict()
      • StepContext
        • StepContext.__init__()
        • StepContext.blank()
        • StepContext.comment()
        • StepContext.config
        • StepContext.copy()
        • StepContext.dry_run
        • StepContext.etc_dir
        • StepContext.experiment
        • StepContext.experiment_dir
        • StepContext.flat_dir()
        • StepContext.hmm_dir()
        • StepContext.log()
        • StepContext.log_comment()
        • StepContext.mkdir()
        • StepContext.model_dir()
        • StepContext.project_dir
        • StepContext.run_cmd()
        • StepContext.shared_dir
        • StepContext.symlink()
      • StepDefinition
        • StepDefinition.__init__()
        • StepDefinition.description
        • StepDefinition.inputs
        • StepDefinition.name
        • StepDefinition.outputs
        • StepDefinition.params
        • StepDefinition.script
      • run_build_lm()
      • run_step_cd_hmm_untied()
      • run_step_ci_hmm()
      • run_step_features()
      • step_cd_hmm_untied()
      • step_ci_hmm()
      • step_features()
      • features_step
      • ci_hmm_step
      • cd_hmm_untied_step
    • Testing API
      • check_pocketsphinx()
      • create_report()
      • load_transcripts()
      • test_model()
    • Timings API
      • format_summary()
      • load_document()
    • Tutorial API
      • copy_tutorial()
      • TUTORIAL_FILENAME
    • CLI API
      • JSON output
        • Command
        • CommandContext
        • CommandResult
        • ModelCommand
        • ProjectCommand
        • main()
    • Data Utilities
      • get_data_file()
      • get_data_path()
      • get_sample_audio()
      • get_sample_file()
      • get_sample_path()
      • get_sample_transcript()
    • Configuration Reference
      • alignment
      • description
      • features
      • runner
      • sharding
      • split
      • training

Design Notes

  • Design documentation
    • Current architecture
    • Domain references
    • Past decisions
    • Decisions reversed (kept here for context)
  • Parity and declared deviations
    • Declared register
    • Inventory under multiple pronunciations
    • Training front-end boundary
    • Alignment recovery and final silence
    • Checkpoints, sharding, and reduction
    • Dropped base pronunciations in the alignment dictionary
  • Canonical configuration
    • Decision summary
    • Current state
      • The active schema
      • The inactive Pydantic schema
      • Entry surfaces and precedence today
      • The lying-surface archetype
    • Truthfulness invariant
    • Proposed model
      • Canonical schema and runtime projection
      • Proposed precedence
      • Versioning and migration
      • Explainability and discovery
      • Generated reference and CI
    • Compatibility and C2 landing order
    • Open decisions for Kevin
  • One-command training
    • Goal
    • Input contract
    • Pre-normalized prompts are the default
    • Handoff to setup and build
    • Failure UX
    • CLI verb choice
    • Compatibility and rollout
    • Open decisions for Kevin
  • Pipeline runner
    • What it is
    • How it works
      • Dependency resolution
      • Staleness
      • Execution
      • Dry-run
    • Why we built our own
    • What we explicitly don’t support
    • Multi-pronunciation training
    • Stage-specific Baum-Welch control
    • Adding a new pipeline node
    • Testing
    • Experimental split variance regularization
    • Explicit checkpoint recovery
  • Training Pipeline Plan
    • Implementation Status Grid
      • Legend
    • Key Insight: CFFI Bindings Are Complete
    • What’s Missing: Step Orchestration
      • Example: CI Training Step
    • Architecture
    • What’s Complete ✅
      • CFFI Bindings (All Done)
      • Orchestration Layer
      • Working Steps
    • What Needs Implementation ⏳
      • Complete Training Pipeline (execution order)
      • Step Functions to Implement (total order)
      • Step 0: Train/Test Split — ✅ DONE
      • Step 1: BW Training (steps/train.py) — FIRST PRIORITY
      • Step 2: Gaussian Split (steps/split.py)
      • Step 3: CD Untied (steps/cd_untied.py)
      • Step 4: Build Trees (steps/trees.py)
      • Step 5: Tie States (steps/tiestate.py)
    • Implementation Order
    • File Locations
    • Success Criteria
  • Multi-pronunciation training
    • Status
    • What it does in one paragraph
    • Why the prior trainer was wrong
    • How it works
      • The key reuse: forward-backward is already a DAG engine
      • Inspiration from Kaldi, without OpenFST
    • As-built layout
      • New C modules
      • Changes to existing modules
      • Untouched
    • Pipeline graph shape
    • Operations
      • Default behavior
      • Opting out (legacy / SphinxTrain parity)
      • CI fallback survival prior
      • Mixing models across runs
    • What we don’t do
    • M4b SLT growth measurement
    • Empirical signal
    • Commit history
    • Relationship to the upstream graph builders
    • Future work
      • Phase 6: data-estimated pronunciation probabilities
  • Baum-Welch normalization policy
  • Baum-Welch sharding contract
    • Generated gate scope
    • Hand-written limitations and interpretation
    • Cross-count characterization
    • Multipron activation and worker lifetime
  • Failed-alignment policy
    • Deliberate vendored divergence
  • Optional final transcript silence
    • Deliberate vendored divergence
    • Graph ownership
    • What mandatory final silence can measure
    • Scope
  • CLI-to-library boundary
    • What these checks are, and are not
    • Runtime observation
      • Establishing the three roles
      • Command-line origins
      • Route state across a dispatch
      • Transparent infrastructure
      • Consequences for the public API
      • Reporting
    • Static addition
    • Combined coverage and limits
  • Native boundary: what is contained, and what is not
    • Phase contract: contained-all-operations
    • Complete-model value validation
    • Exception hierarchy
    • Lifecycle
    • No silent fallback
  • Numerical-correctness harness
    • Purpose
    • Golden trajectory
    • Floating-point contraction checks
    • Five choke points
    • Acceptance debts
    • Per-pass utterance exclusions
    • Reproducibility
  • Decision-tree and tied-state comparison
  • Per-step profiling
  • Source-determinable re-roll dispositions
  • Project Setup Design
    • Overview
    • Project Structure
      • Key Design Principles
    • Setup Requirements
      • Required Inputs
      • Optional Inputs
      • Outputs
    • Setup Process
      • Step 1: Create Directory Structure
      • Step 2: Create/Validate Configuration File
      • Step 3: Prepare Dictionary and Phoneset
      • Step 4: Prepare Transcription File
      • Step 5: Prepare Audio Files
      • Step 6: Validate Setup
    • CLI Command
    • Implementation for pstrain
      • Phase 0: Project Setup (NEW - First Priority)
        • TODO 0.1: Setup CLI Command
        • TODO 0.2: Setup Function
        • TODO 0.3: Configuration Management
        • TODO 0.4: Dictionary and Phoneset Handling
        • TODO 0.5: Transcription File Handling
        • TODO 0.6: Feature Set Identification
        • TODO 0.7: Project Validation
    • Configuration File Format
    • Validation
    • File Format Details
      • Transcription File Format (etc/all.transcription)
      • Dictionary Format (shared/dictionary.dict)
      • Phoneset Format (shared/phoneset.txt)
      • Audio File Naming
    • Integration with Workflows
    • Next Steps After Setup
    • Design Notes
      • Feature Sharing Design
    • Testing with CMU Arctic
  • CI Model Building Requirements
    • Overview
    • Prerequisites (Dependencies)
      • 1. Flat Models
      • 2. Features
      • 3. Dictionary
      • 4. Training Transcriptions
    • CI Training Process
      • Step 1: Flat Initialization
      • Step 2: Baum-Welch Training
      • Step 3: Normalization (Optional)
      • Step 4: Gaussian Splitting (Optional)
    • CI Training Parameters
      • Required Parameters
      • Optional Parameters
    • Workflow Dependencies
    • Implementation for pstrain
      • Phase 1: Prerequisites
      • Phase 2: CI Training
      • Phase 3: Advanced Features
    • Key Programs Needed
    • Output Products
    • Implementation Notes
    • Next Steps
  • Testing with CMU Arctic Corpus
    • Overview
    • CMU Arctic Structure
      • Available Speakers
      • Packed Archive Format
    • Process for Testing
      • Step 1: Download CMU Arctic Data
      • Step 2: Extract Required Files
      • Step 3: Download CMUDict
      • Step 4: Convert CMU Arctic Format
      • Step 5: Set Up Project
    • Implementation Notes
      • For Development/Testing Only
      • Conversion Process (Local Script)
      • Integration with Project Setup
    • Testing Workflow
    • Required Files Summary
    • Notes
  • Terminology
    • Transcription vs Alignment
    • Usage

Benchmarks and Evidence

  • Arctic benchmark pin
    • Measurement identity
    • Pin conditions
      • Configuration provenance by result cell
    • Provenance correction
    • Comparability
    • Baseline
      • Gap composition
    • Forward gate
    • Decode-path transport
    • Training skips and decode coverage
    • Condition contract maintenance
    • Replicability
  • Arctic oracle provenance and reconstruction
    • Keeping the live comparison resource-matched
      • Why the sidecar names a different commit than the one carrying it
    • Procedure for an independently derived stock arm
  • Historical, incomplete evidence: M4b SLT runs
    • Skip counts
    • Reproduction commands
    • Growth

Contributing

  • Development
    • Canonical verification command
    • Individual build and test commands
    • Changing the native interface
    • Building documentation
    • Code Quality
  • Detached runs
pstrain
  • Overview: module code

All modules for which code is available

  • pstrain.api
    • pstrain.api.checkpoints
    • pstrain.api.config
    • pstrain.api.diagnostics
    • pstrain.api.pipeline
    • pstrain.api.steps
    • pstrain.api.testing
    • pstrain.api.tutorial
  • pstrain.cli.base
  • pstrain.cli.cli
  • pstrain.data
  • pstrain.lib._cffi.core
  • pstrain.lib._cffi.io
  • pstrain.lib._cffi.logmath
  • pstrain.lib.alignment.batch
  • pstrain.lib.alignment.coverage
  • pstrain.lib.alignment.export
  • pstrain.lib.commands
  • pstrain.lib.compare
  • pstrain.lib.config.models
  • pstrain.lib.config.resolver
  • pstrain.lib.config.schema
  • pstrain.lib.config.user
  • pstrain.lib.corpus.split
  • pstrain.lib.dictionary.cmudict
  • pstrain.lib.dictionary.cmudict_source
  • pstrain.lib.dictionary.dictionary
  • pstrain.lib.dictionary.filler
  • pstrain.lib.features
  • pstrain.lib.filetypes
  • pstrain.lib.flat
  • pstrain.lib.lm
  • pstrain.lib.model
  • pstrain.lib.native_worker
  • pstrain.lib.one_command
  • pstrain.lib.paths
  • pstrain.lib.phoneset
  • pstrain.lib.pipeline.context
  • pstrain.lib.pipeline.runner
  • pstrain.lib.pipeline.timings
  • pstrain.lib.runtime
  • pstrain.lib.setup
  • pstrain.lib.steps.base
  • pstrain.lib.steps.cd_hmm_untied
  • pstrain.lib.steps.ci_hmm
  • pstrain.lib.steps.features
  • pstrain.lib.steps.package
  • pstrain.lib.telemetry
  • pstrain.lib.transcription
  • pstrain.lib.validate

© Copyright 2026, Kevin Lenzo.

Built with Sphinx using a theme provided by Read the Docs.