Drop of accuracy after add new labels

[LaporteThomas]

Hello,
After testing the addition of a new label, the results on previous data have deteriorated significantly. The model used is the ‘answerdotai/ModernBERT-base’ model.
Here is an example of the results obtained.

Is there something I'm doing wrong ? I only use the function add_examples()

[codelion]

What was the config used for the adaptive-classifier? How many examples are in the dataset? Can you check the note in the README - https://github.com/codelion/adaptive-classifier?tab=readme-ov-file#order-dependency-in-online-learning and see if that helps.

[LaporteThomas]

Currently it was a test, so I haven"t apply a config. The first dataset is contains 41 classes and 1140 in total, then I add a new label with 35 examples. I've test with the prototype-only configuration and I still see the drop

[codelion]

41 is actually quite a lot of classes, what is the performance on the dataset when you fine tune the model directly without adaptive classifier? If you can share the dataset I can review it. It will likely require a bit of experimenting to figure out. Most examples for BERT style encoder models work well up to 20-30 classes.

[LaporteThomas]

Unfortunately, I cannot disclose the data I am working on as it is confidential.

[codelion]

I am trying to do a fix in the PR #54 you can test again once it is merged and released