Does BERTopic have support for the Sinhala language? (Sri Lankan local anguage)

[Iflal]

Hi,
BERTopic has support for the Sinhala language (Which is use in Sri Lanka)? I noticed that some languages are supported, but I couldn’t find explicit information about Sinhala Language. If it is not supported, could you suggest alternatives or workarounds (like using language-independent embeddings or a custom pipeline)?

[Iflal]

I can see that Sinhala language is included in the list of languages in your source code. But on the SBERT.com website, they say they support 50+ languages but "si" is not included.

[MaartenGr]

I'm not sure which commit you checked out but I would advise focusing on the main branch instead: https://github.com/MaartenGr/BERTopic/blob/master/bertopic/backend/_utils.py

With respect to your question, you will need two things for a given language:

• Embedding model that supports your language (embedding_model)
• Tokenizer that can tokenizer your language (CountVectorizer(tokenizer=my_tokenizer))

For the embedding model, I'm not sure which models might work well with Sinhala, so that might be a bit of research you will have to do. As an alternative, you can also use TF-IDF, which is (somewhat) language-independent.

For the tokenizer, it depends on whether Sinhala can be separated by whitespaces. If that separation is the same as with most western languages, then there is no need to look at the tokenizer and you only need an appropriate embedding model (or use TF-IDF which also works with something like SVD/PCA).

[Iflal]

ooh, alright my bad, I checked and get that source code from the previous discussion. Anyway, thank you so much for your time and detailed response .