Evaluating merge_model approach: incremental vs batch processing

[MRaza98]

For my use case, I am modelling parliamentary debate data.

I modelled initially on a batch of 2020 debates and then made incremental models from 2011 until 2019, merging them year by year.

In parallel, I created a batch model trained on the entire dataset from 2010 to 2019.

I did this to showcase the computational efficiency of the incremental approach. However, now I want to evaluate how well the incremental model captured information for the years.

How would you evaluate the incremental model on its own and would you do anything to compare it to the batch model?

[MaartenGr]

Generally, I would advise creating metrics that work for your particular use case. The difficulty with topic modeling is that it is extremely context-dependent and I'm not sure whether general metrics would typically work.

[MRaza98]

@MaartenGr thank you for your response!

If I may ask, would you say that a batch model could be considered as a baseline model to evaluate the incremental model against?

For example, would it make sense from a topic modelling expert's perspective to have a metric like: % of Top 20 topics covered by the merged/incremental model compared to the batch model i.e. whether the incremental model associated documents in a similar manner to the batch model?

[MaartenGr]

It indeed could be but just note that it would miss certain aspects, like the quality of the resulting topics with respect to both the topic assignment as well as the topic representation. Note that just because one model has more topics covered doesn't mean it is necessarily better. I can easily create thousands of "topics" with 100k documents but also just dozens. Neither is necessarily "better" than another.

This all depends on what you deem important for a given use case.

[MRaza98]

@MaartenGr also, would you be able to guide me on which approach I should be using between the two:

merged_model = model_2018.merge_models([model_2019])
merged_model = BERTopic.merge_models([model_2018, model_2019])

In the first one, the model_2019 is completely overwriting the model_2018 (min. similarity is set to default).

Is this expected behaviour?

[MaartenGr]

You should use BERTopic.merge_models for the expected functionality and not model_2018.merge_models