[WIP] Add new scorer: MaNoScorer #289
Conversation
|
At the time of writing, I haven't changed the skada/deep/tests/test_deep_scorer.py yet and there are already failing test when it comes to torch and to the deep methods. It seems that the error is I will investigate that while also adding the MaNoScorer test for the deep methods. |
|
Yes, deep learning tests are failling because skorch v1.0.0 is not compatible with sklearn 1.6.0. They fixed skorch and we are waiting for a new release (see skorch-dev/skorch#1085). In the meantine, you can test your PR with skorch 1.0.0 and sklearn 1.5.2. |
|
Thanks @antoinecollas, yes I am using skorch 1.0.0 and sklearn 1.5.2 in local, I will push the test_deep_scorers once it works locally for me |
@antoinecollas It seems that the skorch issue has been resolved :) |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #289 +/- ##
==========================================
- Coverage 97.07% 96.92% -0.15%
==========================================
Files 63 63
Lines 6559 6671 +112
==========================================
+ Hits 6367 6466 +99
- Misses 192 205 +13 |
| # Test softmax normalization | ||
| scorer = MaNoScorer(threshold=-1) | ||
| score_mean = scorer._score(estimator, X, y, sample_domain=sample_domain) | ||
| assert isinstance(score_mean, float), "score_mean is not a float" |
There was a problem hiding this comment.
You don't check if it is the right normalization which is considered no?
There was a problem hiding this comment.
Understood, will do
skada/deep/tests/test_deep_scorer.py
Outdated
| MaNoScorer(), | ||
| ], | ||
| ) | ||
| def test_dev_exception_layer_name(scorer, da_dataset): |
There was a problem hiding this comment.
To what? it was the name already given
There was a problem hiding this comment.
understood, will do
Implementation of MaNo as an unsupervised realistic scorer.
This should answer #288.