Language model API

Public API for language-model construction.

Decoding needs a language model, so building one is part of the supported surface rather than an internal detail. Callers should use these functions instead of reaching into pstrain.lib.lm.

pstrain.api.lm.build_lm(transcripts, output_path, max_order=3, smoothing='auto')[source]

Build an ARPA language model from transcripts using arpabo.

Parameters:

  • transcripts (list[str] | dict[str, str]) – List of transcript strings or dict mapping utt_id to transcript

  • output_path (Path) – Path to write the ARPA LM file

  • max_order (int) – N-gram order (default 3 for trigrams)

  • smoothing (str) – Smoothing method - “auto” (default), “good_turing”, “kneser_ney”

Returns:

Path to the created LM file

Return type:

Path

Note

“auto” mode uses optimized Katz backoff, which works well for typical speech corpus sizes. For very small corpora, “good_turing” may be better.

pstrain.api.lm.build_lm_from_file(transcript_file, output_path, max_order=3, smoothing='auto')[source]

Build an ARPA LM from a supported transcription file.

Parameters:

  • transcript_file (Path) – Path to a simple or Sphinx-format transcription file

  • output_path (Path) – Path to write ARPA LM

  • max_order (int) – N-gram order (default 3)

  • smoothing (str) – Smoothing method (default “auto”)

Returns:

Path to created LM file

Return type:

Path

pstrain.api.lm.load_transcripts(transcript_file)[source]

Load transcripts from a supported transcription file.

Supports utterance_id words and <s> words </s> (utterance_id).

Parameters:

transcript_file (Path) – Path to transcription file

Returns:

Dict mapping utterance_id to transcript text

Raises:

ValueError – If a nonempty line does not match a supported format

Return type:

dict[str, str]