Project Setup Design

This document outlines how to set up a project structure in pstrain.

Overview

Project setup creates the initial directory structure and prepares the project for training. This is the first step before any training can begin.

Project Structure

Design for sharing resources across experiments:

project/
├── etc/
│   └── config.yaml          # Project configuration
├── audio/                    # Audio files (WAV format) - shared across experiments
├── shared/                   # Shared resources across ALL experiments
│   ├── dictionary.dict      # Main pronunciation dictionary
│   ├── filler.dict          # Filler word dictionary
│   └── features/            # Extracted features (shared across experiments)
│       └── {feature_set_id}/  # Feature set identified by audio+feature config
│           ├── {fileid1}.mfc # Feature files (one per audio file)
│           ├── {fileid2}.mfc
│           ├── ...
│           ├── feat.params   # Feature parameters
│           ├── global_mean    # Global statistics
│           └── global_variance
└── experiments/             # Training experiments (isolated)
    └── {experiment_name}/   # Individual experiment
        ├── models/          # Trained models (experiment-specific)
        │   ├── ci/          # Context-Independent models
        │   │   └── {config}/  # Model configuration (e.g., "baseline", "1g", "lda")
        │   │       └── model/
        │   │           ├── flat/  # Flat initialization models
        │   │           └── hmm/   # Trained CI model files
        │   └── cd/          # Context-Dependent models (future)
        │       └── {config}/
        │           └── model/
        │               └── hmm/   # Trained CD model files
        ├── logs/            # Training logs
        └── work/            # Working directory for training

Key Design Principles

1. Shared Resources at Project Level

   • audio/ - All experiments use the same audio files

   • shared/dictionary.dict - All experiments use the same dictionary

   • shared/features/{feature_set_id}/ - Features shared by experiments with same config

2. Feature Set Identification

   • Feature directory name (feature_set_id) is deterministic based on:

     • Audio configuration (sample rate, format)

     • Feature extraction configuration (num_ceps, filters, etc.)

   • Experiments with identical audio+feature config share the same feature directory

   • Different feature configs get different directories (e.g., mfcc_13, mfcc_39, lda_29)

3. Experiment Isolation

   • Each experiment has its own models/, logs/, and work/ directories

   • Experiments can reference shared features without copying them

   • Multiple experiments can run in parallel using the same features

4. Space Efficiency

   • Features are computed once per unique (audio, feature) configuration

   • No duplication of features across experiments

   • Experiments only store their unique outputs (models, logs)

Setup Requirements

Required Inputs

1. Configuration file (etc/config.yaml)

   • Project metadata (name, description)

   • Audio settings (sample rate, format)

   • Feature extraction settings

   • Training parameters

   • Dictionary and phoneset paths

2. Fileid list with transcripts (etc/all.transcription or similar)

   • Format: One line per utterance: <fileid> <word1> <word2> ...

   • Example: arctic_a0001 hello world

   • Fileid is the base name (without extension) of the audio file

   • Used to map audio files to their transcriptions

3. Audio files (in audio/ directory)

   • Named by fileid: {fileid}.wav (e.g., arctic_a0001.wav)

   • Must match fileids in transcription file

   • Format: WAV (16kHz recommended for speech)

   • One audio file per fileid

4. Pronunciation dictionary (shared/dictionary.dict)

   • Format: One entry per line: <word> <phone1> <phone2> ...

   • Example: hello HH AH L OW

   • Must contain all words used in transcripts

   • UTF-8 encoding

5. Phoneset (shared/phoneset.txt or extracted from dictionary)

   • List of all valid phones used in dictionary

   • Format: One phone per line (comments with #)

   • Can be extracted from dictionary automatically

   • Must include special phones (e.g., SIL for silence)

Optional Inputs

• Filler dictionary (shared/filler.dict) - Filler words (SIL, noise, etc.)

• Test split - Separate test transcription file (or split from all.transcription)

Outputs

• Project structure - All directories created

• Configuration file - etc/config.yaml (created or validated)

• Dictionary files - shared/dictionary.dict, shared/filler.dict

• Phoneset file - shared/phoneset.txt (extracted from dictionary)

• Transcription file - etc/all.transcription (validated)

• Directory structure - audio/, shared/, experiments/

Setup Process

Step 1: Create Directory Structure

• Create etc/ directory

• Create audio/ directory

• Create shared/ directory

• Create experiments/ directory

• Create shared/features/ directory

Step 2: Create/Validate Configuration File

• Create etc/config.yaml with:

  • Project metadata (name, description)

  • Audio settings (sample rate, format)

  • Feature extraction settings

  • Training parameters

  • Dictionary and phoneset paths

• Or validate existing config file

Step 3: Prepare Dictionary and Phoneset

• Copy dictionary file to shared/dictionary.dict

• Validate dictionary format (UTF-8, one word per line)

• Extract phoneset from dictionary to shared/phoneset.txt

• Create filler dictionary to shared/filler.dict (if not provided)

• Validate all words in transcripts are in dictionary

Step 4: Prepare Transcription File

• Create or validate etc/all.transcription file

• Format: <fileid> <word1> <word2> ... (one line per utterance)

• Validate format (UTF-8, proper structure)

• Extract list of fileids from transcription file

Step 5: Prepare Audio Files

• Copy or link audio files to audio/ directory

• Name files by fileid: {fileid}.wav (must match transcription fileids)

• Validate audio files:

  • All fileids in transcription have corresponding audio files

  • Audio files are readable WAV format

  • Audio format matches config (sample rate, etc.)

Step 6: Validate Setup

• Check all required directories exist

• Validate dictionary format and phoneset

• Validate transcription file format

• Check all fileids have corresponding audio files

• Verify all words in transcripts are in dictionary

• Verify configuration is valid

CLI Command

pstrain setup [project_dir] [options]

Project Directory:

• If project_dir is provided: Initialize project in that directory (create if needed)

• If project_dir is omitted: Initialize project in current directory (.)

Options:

• --config <path> - Path to config file (or create default)

• --transcription <path> - Path to transcription file (fileid + transcripts)

• --audio <path> - Path to audio files directory (files named by fileid)

• --dictionary <path> - Path to pronunciation dictionary file

• --phoneset <path> - Path to phoneset file (or extract from dictionary)

• --filler-dict <path> - Path to filler dictionary (optional)

• --force - Force recreation if project exists

Examples:

Initialize in current directory:

cd my_project
pstrain setup \
  --transcription /path/to/all.transcription \
  --audio /path/to/audio \
  --dictionary /path/to/dictionary.dict

Initialize in specified directory:

pstrain setup my_project \
  --transcription /path/to/all.transcription \
  --audio /path/to/audio \
  --dictionary /path/to/dictionary.dict

Initialize with all options:

pstrain setup my_project \
  --config /path/to/config.yaml \
  --transcription /path/to/all.transcription \
  --audio /path/to/audio \
  --dictionary /path/to/dictionary.dict \
  --phoneset /path/to/phoneset.txt

Transcription file format (etc/all.transcription):

arctic_a0001 hello world
arctic_a0002 this is a test
arctic_a0003 good morning

Audio file naming:

• Fileid from transcription: arctic_a0001

• Audio file: audio/arctic_a0001.wav

• Must match exactly (case-sensitive)

Implementation for pstrain

Phase 0: Project Setup (NEW - First Priority)

TODO 0.1: Setup CLI Command

• [ ] Create pstrain/cli/setup.py module

• [ ] Add pstrain setup command to main CLI

• [ ] Parse command-line arguments:

  • [ ] project_dir (optional positional) - If provided, use that directory; if omitted, use current directory

  • [ ] --config, --transcription, --audio, --dictionary, --phoneset, --filler-dict

  • [ ] --force flag

• [ ] Resolve project directory:

  • [ ] If project_dir provided: create if needed, use absolute path

  • [ ] If omitted: use Path.cwd() (current directory)

• [ ] Validate inputs

• [ ] Call setup function

TODO 0.2: Setup Function

• [ ] Create pstrain/lib/setup.py module

• [ ] Implement setup_project(project_dir: Path, ...) function

• [ ] Handle project directory:

  • [ ] Accept project_dir as Path (absolute or relative)

  • [ ] Create directory if it doesn’t exist (when project_dir is provided)

  • [ ] If directory exists and not empty, check --force flag

• [ ] Create directory structure

• [ ] Create default config file

• [ ] Copy/prepare dictionary

• [ ] Copy/link audio files

• [ ] Prepare transcripts

• [ ] Validate setup

• [ ] Return setup status and paths

TODO 0.3: Configuration Management

• [ ] Create pstrain/lib/config.py module

• [ ] Implement config loading/saving

• [ ] Create default config template

• [ ] Validate config schema

• [ ] Support config overrides

TODO 0.4: Dictionary and Phoneset Handling

• [ ] Create pstrain/lib/dictionary.py module

• [ ] Implement dictionary loading/validation

• [ ] Support filler dictionary

• [ ] Validate dictionary format (UTF-8, proper structure)

• [ ] Create pstrain/lib/phoneset.py module

• [ ] Implement phoneset loading/validation

• [ ] Extract phoneset from dictionary

• [ ] Validate dictionary phones against phoneset

• [ ] Support phoneset file format (one phone per line, # comments)

TODO 0.5: Transcription File Handling

• [ ] Create pstrain/lib/transcription.py module

• [ ] Implement transcription file parsing

• [ ] Format: <fileid> <word1> <word2> ... (one line per utterance)

• [ ] Extract fileid list from transcription

• [ ] Validate transcription format (UTF-8, proper structure)

• [ ] Validate all words in transcripts are in dictionary

TODO 0.6: Feature Set Identification

• [ ] Create pstrain/lib/features.py module

• [ ] Implement get_feature_set_id(audio_config, feature_config) function

• [ ] Generate deterministic identifier from audio + feature parameters

• [ ] Use hash or descriptive name (e.g., mfcc_13, lda_29)

• [ ] Ensure same config always produces same feature_set_id

• [ ] Document feature directory naming convention

TODO 0.7: Project Validation

• [ ] Create pstrain/lib/validate.py module

• [ ] Implement project structure validation

• [ ] Check required files exist:

  • [ ] Configuration file

  • [ ] Transcription file

  • [ ] Dictionary file

  • [ ] Phoneset file (or extract from dictionary)

• [ ] Validate file formats

• [ ] Validate fileid matching:

  • [ ] All fileids in transcription have corresponding audio files

  • [ ] All audio files have corresponding transcriptions

• [ ] Validate dictionary coverage:

  • [ ] All words in transcripts are in dictionary

  • [ ] All phones in dictionary are in phoneset

• [ ] Check audio files are readable WAV format

• [ ] Validate feature directory structure (if features exist)

Deliverables:

• pstrain/cli/setup.py - Setup CLI command

• pstrain/lib/setup.py - Setup implementation

• pstrain/lib/config.py - Configuration management

• pstrain/lib/dictionary.py - Dictionary handling

• pstrain/lib/phoneset.py - Phoneset handling

• pstrain/lib/transcription.py - Transcription file handling

• pstrain/lib/features.py - Feature set identification

• pstrain/lib/validate.py - Project validation

• tests/test_setup.py - Unit tests

• Documentation

Configuration File Format

# etc/config.yaml
project:
  name: "my_project"
  description: "Acoustic model training project"
  version: "1.0.0"

audio:
  sample_rate: 16000
  format: "wav"
  directory: "audio"

features:
  num_ceps: 13
  num_filters: 25
  lower_freq: 133.33334
  upper_freq: 6855.4976
  preemphasis: 0.97
  transform: "dct"
  lifter: 22
  agc: "max"
  cmn: "batch"
  varnorm: false
  feature_type: "1s_c_d_dd"

dictionary:
  main_dict: "shared/dictionary.dict"
  filler_dict: "shared/filler.dict"
  phoneset: "shared/phoneset.txt"  # Can be extracted from dictionary

corpus:
  transcription_file: "etc/all.transcription"  # Fileid + transcripts
  audio_dir: "audio"  # Audio files named by fileid: {fileid}.wav

features:
  # Feature directory is automatically determined by audio + feature config
  # Path: shared/features/{feature_set_id}/
  # feature_set_id is hash/identifier based on audio and feature parameters
  # Multiple experiments with same config share the same feature directory

training:
  ci:
    n_iterations: 10
    convergence_threshold: 0.001
    min_iterations: 3
    abeam: 1e-100
    bbeam: 1e-100
    varfloor: 0.0001
    mixw_floor: 0.00001
    topn: 1

experiments:
  directory: "experiments"
  default_name: "baseline"

Validation

After setup, validate:

• [ ] All required directories exist

• [ ] Configuration file is valid YAML

• [ ] Transcription file exists and is valid format

• [ ] Dictionary file exists and is readable

• [ ] Phoneset file exists (or can be extracted from dictionary)

• [ ] All fileids in transcription have corresponding audio files

• [ ] All audio files are readable WAV format

• [ ] All words in transcripts are in dictionary

• [ ] All phones in dictionary are in phoneset

• [ ] Project structure matches expected layout

File Format Details

Transcription File Format (etc/all.transcription)

Format 1: Simple (fileid + words)

<fileid1> <word1> <word2> <word3>
<fileid2> <word1> <word2>
<fileid3> <word1> <word2> <word3> <word4>

Format 2: Sphinx format (with sentence markers)

<s> <word1> <word2> <word3> </s> (<fileid1>)
<s> <word1> <word2> </s> (<fileid2>)

• One line per utterance

• First token is fileid (Format 1) or fileid in parentheses at end (Format 2)

• Remaining tokens are words

• UTF-8 encoding

• Words must match dictionary entries

• Fileid is base name without extension (e.g., arctic_a0001 → audio file arctic_a0001.wav)

Dictionary Format (shared/dictionary.dict)

<word1> <phone1> <phone2> <phone3>
<word2> <phone1> <phone2>

• One entry per line

• First token is word

• Remaining tokens are phones

• UTF-8 encoding

• Case-sensitive

Phoneset Format (shared/phoneset.txt)

# Phoneset for acoustic model
AA
AE
AH
SIL

• One phone per line

• Comments start with #

• UTF-8 encoding

• Case-sensitive

• Can be extracted from dictionary automatically

Audio File Naming

• Fileid from transcription: arctic_a0001

• Audio file: audio/arctic_a0001.wav

• Must match exactly (case-sensitive, no extension in fileid)

Integration with Workflows

Once project is set up:

• Workflows can reference project structure

• Config file provides paths to resources

• Shared resources (dictionary, features) are accessible

• Experiments can be created in experiments/ directory

Next Steps After Setup

1. Feature extraction - Extract features from audio

2. Data splitting - Split into train/test sets

3. Flat initialization - Initialize flat models

4. CI training - Train CI models

Design Notes

1. Shared resources - Dictionary, audio, and features are shared across experiments

2. Feature set identification - Features are stored in shared/features/{feature_set_id}/ where feature_set_id is deterministic based on audio+feature config

3. Space efficiency - Features computed once per unique config, shared by all experiments using that config

4. Experiment isolation - Each experiment has its own models and logs

5. Config-driven - All paths come from config file

6. Validation - Setup validates project structure before proceeding

7. Idempotent - Setup can be run multiple times safely (skips if exists)

Feature Sharing Design

Problem: Multiple experiments often use the same audio and feature extraction settings, but features are expensive to compute.

Solution: Features are stored in shared/features/{feature_set_id}/ where feature_set_id is a deterministic identifier based on:

• Audio configuration (sample rate, format)

• Feature extraction configuration (num_ceps, filters, transforms, etc.)

Benefits:

• Features computed once per unique (audio, feature) configuration

• All experiments with matching config automatically share features

• No manual coordination needed - feature directory is determined by config

• Experiments can run in parallel using the same features

• Space efficient - no duplication of large feature files

Example:

• Experiment baseline uses mfcc_13 features → shared/features/mfcc_13/

• Experiment lda_v1 uses lda_29 features → shared/features/lda_29/

• Experiment baseline_v2 also uses mfcc_13 → reuses shared/features/mfcc_13/ (no recomputation)

Testing with CMU Arctic

For testing purposes, CMU Arctic corpus can be used:

• Source: http://festvox.org/cmu_arctic/cmu_arctic/packed/

• Format: Packed archives (tar.bz2) with audio and transcripts

• Process: Download, extract, convert format, set up project

• Note: See docs/design/testing-with-cmu-arctic.md for details (local testing only, do not commit data/scripts)