🎙️ FF1 — Phone Dysfluency Leaderboard

Vocab-aware operation-level evaluation for phone recognizers. 5 models × 4 datasets × 3 target vocab sizes = 60 result cells. Headline metric = max strict F1 across 3 vocabs per (model, dataset).

Max Strict F1 — best-of-3-vocabs per dataset

Best vocab shown in parens. Sorted by mean across datasets.


1	wav2vec2-xlsr-53 espeak	mrrubino/wav2vec2-large-xlsr-53-l2-arctic-phoneme	0.358 (v156)	0.479 (v156)	0.280 (v156)	0.596 (v156)	0.669


1	HuPER S1 v1	huper29/huper_recognizer	0.753 (v39)	0.636 (v39)	0.557 (v39)	0.730 (v39)	0.669
2	mrrubino L2A	mrrubino/wav2vec2-large-xlsr-53-l2-arctic-phoneme	0.353 (v39)	0.479 (v156)	0.359 (v39)	0.596 (v156)	0.447
3	vitouphy TIMIT	vitouphy/wav2vec2-xls-r-300m-timit-phoneme	0.622 (v39)	0.201 (v156)	0.449 (v39)	0.415 (v156)	0.422
4	PhoneticXeus	changelinglab/PhoneticXeus	0.358 (v156)	0.370 (v156)	0.280 (v156)	0.637 (v156)	0.411
5	wav2vec2-xlsr-53 espeak	facebook/wav2vec2-xlsr-53-espeak-cv-ft	0.513 (v156)	0.261 (v156)	0.406 (v313)	0.454 (v156)	0.409

What is strict F1? FF1 aligns the canonical phone sequence Phm against both the human annotation Phn0 and the model prediction Phn1, extracts the two sets of non-keep events (sub/del/ins), and computes precision/recall on those events. A G2P-cheater (Phn1 = Phm) gets recall = 0 by construction — this is the framework's anti-shortcut property.

Built on the FF1 framework. Source code & full reproduction stack: https://github.com//ff1