Corpus API

Public API for corpus utilities.

pstrain.api.corpus.split_is_external(output_dir)[source]

Return whether a complete split is user-owned rather than generated.

Generated files remain generated only while all four content hashes match the sidecar written by train_test_split(). Editing any file transfers authority to the persistent files on the next pipeline construction.

Parameters:

output_dir (Path)

Return type:

bool

pstrain.api.corpus.train_test_split(transcription_file, output_dir, *, train_ratio=None, test_count=None, seed=42)[source]

Split a Sphinx-format transcription into train and test partitions.

Exactly one of train_ratio or test_count may be set; if both are None, the default is DEFAULT_TRAIN_RATIO (95% train).

Parameters:

  • transcription_file (Path) – Path to the input transcription (e.g. etc/all.transcription).

  • output_dir (Path) – Directory to write the four output files into; created if it doesn’t exist.

  • train_ratio (float | None) – Fraction of utterances to put in the training set (e.g. 0.95). Mutually exclusive with test_count.

  • test_count (int | None) – Exact number of utterances to put in the test set. Mutually exclusive with train_ratio.

  • seed (int) – Random seed for the shuffle (default 42).

Returns:

A SplitResult with the four written paths and the train/test utterance counts.

Raises:

  • FileNotFoundError – If transcription_file doesn’t exist.

  • ValueError – If both train_ratio and test_count are set, or if the transcription is empty.

Return type:

SplitResult

pstrain.api.corpus.validate_external_split(transcription_file, output_dir, audio_dir)[source]

Validate and preserve an externally supplied Sphinx train/test split.

Parameters:

  • transcription_file (Path)

  • output_dir (Path)

  • audio_dir (Path)

Return type:

SplitResult