Fix multiclass macro F-score (mean of per-class F, not F of the means)#448
Open
gaoflow wants to merge 1 commit into
Open
Fix multiclass macro F-score (mean of per-class F, not F of the means)#448gaoflow wants to merge 1 commit into
gaoflow wants to merge 1 commit into
Conversation
f_score/f1_score computed the F-beta of the macro-averaged precision and recall for multilabel matrices, but macro F is the mean of the per-class F-beta scores; the two differ whenever classes have unequal precision or recall. Average the per-class one-vs-all F-beta instead (mirroring precision/recall), counting an all-wrong class (0/0) as 0. Binary matrices are unchanged.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #448 +/- ##
==========================================
+ Coverage 77.53% 77.92% +0.39%
==========================================
Files 106 104 -2
Lines 7585 7534 -51
==========================================
- Hits 5881 5871 -10
+ Misses 1704 1663 -41 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Multiclass
f_score/f1_scorereturned the F-beta of the macro-averaged precision and recall, but the macro F-score is the mean of the per-class F-beta scores. Those agree only when every class shares the same precision and recall, so multiclassf1_scorewas wrong for most inputs.Repro (
y_true=[0,0,1,1,2,2],y_pred=[0,1,1,1,2,2]):f1_score()returned 0.860215; the macro F1 is 0.822222 (per-class F1 = [2/3, 4/5, 1], matching sklearnf1_score(average="macro")). Macro precision and recall were already correct, which isolates the error to the F combination.The fix mirrors the existing
precision/recallmacro averaging: average the per-class one-vs-all F-beta. A class the model never gets right has precision = recall = 0 (a 0/0 F); it now counts as 0 (matching sklearn'szero_division=0) instead of turning the whole average into NaN. Binary matrices are unchanged.Two regression tests added; full
cargo test --workspace,cargo fmtandcargo clippypass.One caveat: for beta != 1 the aggregation is now correct, but linfa's per-class precision and recall are oriented opposite to sklearn's, so
f_score(beta)lines up with sklearn'sfbeta_score(beta=1/beta). That orientation is pre-existing (it affects the binary path too) and left out of scope here; f1 is symmetric so it is unaffected.