BERTopic - Dealing with complaints

[PDuarteLopes]

Hey all,
I am working with a Topic Modelling project which has the goal of categorizing user complaints. Tough BERTopic is a really good model I am struggling due to the nature of having a lot of noise even with some pre-process cleaning (removal internal communication, acronyms and user data anonymization). Is there anything I can do to improve results, make my model deal better with noisy text and make my topics be more specific and concise? I am struggling when creating a lot of meaningless topics and missing the key aspects which make a topic a topic

[MaartenGr]

Thanks for reaching out! Could you give a couple of examples perhaps as to the behavior of BERTopic that you want improved? Without output (or training code) it's very difficult to say whether something is due to the code, algorithm, or perhaps even the data.

[MaartenGr]

Since the label is embedded, I assume that a precise topic label should yield better outcomes than a more general one. For example, should I use a label in a specific format like "sms"_resent_confirmation" or a more general title like "SMS Confirmation"?

It generally depends on the underlying embedding model but I would advise using the latter as that often generates more accurate representations.

[PDuarteLopes]

Thanks for your time. I just have one more question, and I think it might be related to an old bug. When I use Zero-Shot Topic Modeling, I get good clustering, but with seemingly mismatched representations.

I’m sorry that I'm working with a Portuguese dataset, but I think it’s easy to get my point across. As you can see in the following example, the topic names I give to my Zero-Shot model generate good clusters, but with mismatched cluster assignments. The keywords for each zero-shot topic appear in the representations, but not in the 'correct'/supposed ones. I might have cluster based on the topics given «, but not in a correct order. Do you know why this is happening?

Best regards,

[MaartenGr]

@PDuarteLopes This might have something to do with your BERTopic version as I believe this was an issue before. Could you try it with the latest version (0.16.4)?

[PDuarteLopes]

I upgraded my version and made some more tests, and unfortunately, the same "error" keeps happening. Even with different Zero-Shot names, the model cannot make cohesive associations between the Zero-Shot Topics and the respective Representations.

Going deeper, I started analyzing the cosine similarity between the embedding of each Zero-Shot Topic and my documents. I found out that even the documents with cosine similarity above 0.9 are being assigned to new topics during the training of my model. Why does this happen? Is it possible that the model initially attributes the document to my Zero-Shot topic and then finds a new topic with higher cosine similarity and makes the change? How can I avoid it?

I really see the potential of using Zero-Shot topics, but I am having a really hard time taking the full potential out of it. I am making a list of potential topics which are easily detected in my documents, but the model insists on creating new topics and giving non-logical representations to the given ones.

[MaartenGr]

Hmmm, that's strange! I can imagine this must be frustrating...

I don't wont to make you do too much work but do you by change have a reproducible example for me? Maybe with some toy data? If I can reproduce what is happening, then I can start diving into this problem...

Also, if that's not possible. Could you share your code? That might help me understand what is happening here.