Dear Pecos-Team,
I am working as data scientiest for the german national library. We have testet pecos/xtransformer for our own multi label classification tasks and it promises a significant improvement over the existing state-of-the art.
Our production environemnt is based around a toolkit called annif, which will soon integrate pecos/xtransformer thanks to the work of other collaborators.
We have observed that there have been developments to refit the BERT-Generation of Encoder models, such as ModernBERT or EuroBERT (see below). These promise compatibility with previous BERT-Applications, while improving speed and context length.
Currently, the allowed encoder-classes for xtransformer are hard-coded in the network.py file.
What would be the necessary steps to make XTransformer work for this new model generation, too?
Can you give us directions, so that we can propose a pull request to accomplish this?
Best,
Maximilian
References
Dear Pecos-Team,
I am working as data scientiest for the german national library. We have testet pecos/xtransformer for our own multi label classification tasks and it promises a significant improvement over the existing state-of-the art.
Our production environemnt is based around a toolkit called annif, which will soon integrate pecos/xtransformer thanks to the work of other collaborators.
We have observed that there have been developments to refit the BERT-Generation of Encoder models, such as ModernBERT or EuroBERT (see below). These promise compatibility with previous BERT-Applications, while improving speed and context length.
Currently, the allowed encoder-classes for xtransformer are hard-coded in the network.py file.
What would be the necessary steps to make XTransformer work for this new model generation, too?
Can you give us directions, so that we can propose a pull request to accomplish this?
Best,
Maximilian
References