Fine-tune crosslingual model for language detection

[artitw]

Two approaches to try:

1. Use crosslingual embeddings as input to MLP or tree-based model in transfer learning fashion
2. Fine-tune crosslingual translator with softmax output

[Mofetoluwa]

I'd get started on this

[artitw]

Awesome, I've assigned you to this project. Let's keep track of progress here.

[Mofetoluwa]

Alright, sure :)

[Mofetoluwa]

Hi Artit @artitw

Please I need your help, I’m facing some roadblocks.

I decided to start with the second approach you suggested, which is Fine-tuning the cross-lingual translator with a softmax output. My thought process for this:

1. Get a language detection dataset. I decided to work with wili_2018 (https://arxiv.org/pdf/1801.07779v1.pdf, https://zenodo.org/record/841984#.Yd1XWRPMK3I). What do you think about it, and do you have any other datasets in mind?
2. Looking at the Fitter class in the code, I'm not exactly sure how I can fine-tune the translator to train on the identification dataset. Since it requires both the source and target languages, I was thinking of writing another module to do the fine-tuning but was not sure if that was necessary? Do you have any ideas on how I can apply the softmax output to the already existing translator?

Also, does this sound like the right track for approach number 2?

[artitw]

Very much appreciate the updates on this. The dataset you cite looks appropriate; I suggest filtering for the languages which the pretrained model supports for tokenization.

Your ideas on the second approach seem fine so far. Yes, you are correct that you would have to write another module to finetune with a softmax output. I expect this second approach to be more challenging for this reason. If it helps, consider taking the first approach to get things working and then come back to the second approach to get better performance.

[Mofetoluwa]

Alright then, I'll get started with approach 1

[Mofetoluwa]

Hi Art @artitw

Here is the link to the notebook: https://colab.research.google.com/drive/1VxRRURRAaXBZFsYsXC5hSTkSc-4TGdOj?usp=sharing

Based on the last discussion:

• I decided to work with a batch size of 5, but the session took too long and would stop while going through the dataset.
• So I decided to divide the dataset, create embeddings for each division and save it to a file.
• I'm still doing that, as there is still some crashing. But there are currently embeddings for 2200 samples which was used for training.
• The models trained are very simple since the size of the data used is small.

I would add more embeddings and retrain the models to see how performance improves.

What do you think about it? Thank you :)

[artitw]

Hi @Mofetoluwa

Thanks for all the work and the summary. It looks like the MLP model is best performing. Would you be able to add it to the repo? It would be great if language classification is available for use while we continue improving it with finetuning and other methods.

[Mofetoluwa]

Hi @artitw

Alright then :)

So just to clarify, how would we want to use the model on the repo? Asides from pushing the saved model, are we also creating a module for it to be called e.g identification.py, like the other functionalities?

[artitw]

It would be awesome to create an Identifier class so that we can do sonething like t2t.Handler(["test text"]).identify(). Could we give that a try?

[Mofetoluwa]

Alright, I'll add the model to the repo first. In which of the folders should I put it?

[artitw]

Can we store the model in somewhere like Google Drive and only download it when the Identifier is used? This approach would follow the existing convention to keep the core library lightweight.

[Mofetoluwa]

Alright then :)

[Mofetoluwa]

Hi @artitw

My sincere apologies the updates are just coming in, I wanted to have done some work on the Identifier class before sending the updates.

• A pull request has been made for the code, so you can have a look at it and let me know your thoughts.
• More embeddings were created (~ 6900) and used to train the MLP model, so there has been some improvement as seen in the notebook: https://colab.research.google.com/drive/1Cq1lnDJMI2-ZZxm1VmCWzvLL78E2UmVy?usp=sharing
• One limitation this current model has, is that it doesn't correctly predict the language for some short text sequences (less than 10 tokens). But it correctly identifies the language for longer sequences. Hopefully, this can be resolved using the second approach, and I'll also appreciate your thoughts on this problem too.

Thank you :)