Converting Multilingual Document Representations to English

[owwdesilva]

Hi @MaartenGr , Hi All,

I need to obtain topic representations for a set of multilingual documents. I have already utilized multilingual embeddings for this purpose. However, I have received representations in different languages due to these multilingual documents. Is there any approach that can convert all the final representations to the English language? I have selected the following representation method:
generator = pipeline('text2text-generation', model='google/flan-t5-base') representation_model = TextGeneration(generator)

[MaartenGr]

I would advise using a model that support multi-lingual documents a bit better. I'm not sure but I remember Flan-T5 not to be trained on multi-lingual data. Instead, if you use an LLM like Llama 3, you can ask it in the prompt to generate only English texts.

[owwdesilva]

Thank you for the clarifications. Much appreciated.