Freya-TR-Eval

Kuşun kafesini en son ne zaman temizlediğini hatırlıyor musun?

Sentence 0235 of 495

Freya-TR-Eval leaderboardTurkish text-to-speech systems, ranked by how often a speech recognizer mishears them.

495
everyday Turkish sentences, 3 to 13 words, no brands or domain terms
8 kHz
every system band-matched before scoring, wideband or telephony
Whisper-large-v3
one recognizer, one Turkish normalizer, one corpus-level WER

Word error rate

Lower is better. Click a row for the system, the source of its number, and notes.

# System Params WER
Download results (JSON)

One pipeline for every system

Nothing is tuned per system. Every number on the board comes from these five steps. Rows tagged PatientDesk average three takes per sentence (seed 100000·take + sentence index); rows tagged Paper are one take.

  1. SynthesizeAll 495 sentences at the system's native rate. Cloners get one fixed, shared reference clip.
  2. Band-matchDownsample to 8 kHz, then back to 16 kHz, so no system wins on bandwidth alone.
  3. TranscribeWhisper-large-v3 through faster-whisper, Turkish, beam size 5.
  4. NormalizeTurkish-aware casing for I and İ, digits spelled out, punctuation removed on both sides.
  5. ScoreCorpus-level WER with jiwer across all 495 sentences, per take; the board shows the mean over takes.

Noise floor

Whiskers are 95% sentence-bootstrap intervals, computed here from the published transcripts. The top seven systems, all between 1.0% and 1.5%, overlap; read their order as a near tie.

Reference line

Real human recordings (FLEURS-tr) score 9.7% through the identical pipeline, on a different corpus, so treat it as a scale reference rather than a ceiling.

What WER rewards

Regular, phonemizer-driven pronunciation helps a recognizer, so small VITS models score low WER. Compare within a weight class.

Submit a system

Wrap your model in an adapter with one method, synthesize(text), returning a float32 waveform and its sample rate. Register it in eval/benchmark.py, run it on a GPU, then open a discussion on this Space with the results JSON and a checkpoint link.

pip install faster-whisper jiwer librosa datasets
python eval/benchmark.py --system <adapter> \
    --out results/<model>.json

Cite

Freya-TR-Eval is introduced in FreyaTTS: A Compact Tokenizer-Free Flow-Matching Transformer for Turkish-First Speech Synthesis.

@misc{pamuk2026freyattscompacttokenizerfreeflowmatching,
  title={FreyaTTS: A Compact Tokenizer-Free Flow-Matching
         Transformer for Turkish-First Speech Synthesis},
  author={Ahmet Erdem Pamuk and Ömer Yentür and Ahmet Tunga Bayrak
          and Yavuz Alp Sencer Öztürk and Mustafa Yavuz},
  year={2026},
  eprint={2607.09530},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2607.09530},
}