Topics inference from multiple images

[owwdesilva]

Hi @MaartenGr, Hi All, In the multimodal topic analysis "Images Only" option, if we train a model, is there a way to use the find_topics method for topics inference from multiple images? I am asking because the find_topics method can only process a single image, not multiple images. We have used the transform method on individual images to filter the highest topic weights, but the issue is that it gives incorrect results due to a few unrelated images from the lot. Are there any alternative approaches?

[MaartenGr]

Hi All, In the multimodal topic analysis "Images Only" option, if we train a model, is there a way to use the find_topics method for topics inference from multiple images?

find_topics is not meant for actual assignment but is merely a tool to quickly find topics that you might be interested in.

I am asking because the find_topics method can only process a single image, not multiple images. We have used the transform method on individual images to filter the highest topic weights, but the issue is that it gives incorrect results due to a few unrelated images from the lot. Are there any alternative approaches?

What do you mean by "it gives incorrect result due to a few unrelated images from the lot"? The main method for inference is .transform which comes in two flavors, either assignment through HDBSCAN or assignment through the embeddings themselves. To perform the latter, you will have to save the model using safetensors and then load it in order to get similar functionality as find_topics.

[owwdesilva]

Hi Maarten, Thank you for the reply!

Hi All, In the multimodal topic analysis "Images Only" option, if we train a model, is there a way to use the find_topics method for topics inference from multiple images?

find_topics is not meant for actual assignment but is merely a tool to quickly find topics that you might be interested in.

Thank you for clarifying.

I am asking because the find_topics method can only process a single image, not multiple images. We have used the transform method on individual images to filter the highest topic weights, but the issue is that it gives incorrect results due to a few unrelated images from the lot. Are there any alternative approaches?

What do you mean by "it gives incorrect result due to a few unrelated images from the lot"? The main method for inference is .transform which comes in two flavors, either assignment through HDBSCAN or assignment through the embeddings themselves. To perform the latter, you will have to save the model using safetensors and then load it in order to get similar functionality as find_topics.

Sorry for the confusion. Let me explain our use case in more detail.
Our use case differs a bit from the usual BERTopic applications. We aim to feed a batch of images to a model to identify the topic distribution within those images.
Since we can't provide all images to the multimodal BERTopic (image-only) model at once, we process individual images and assign topics to each image. We then aggregate these assigned topics to understand the dominant themes and the overall context of the images. This aggregation step is performed outside of BERTopic. The issue arises when there are noisy images in the inference sample (e.g., random, unrelated images). For instance, random unrelated images can negatively impact the final outcome.

Steps followed:

1. Train a multimodal BERTopic (image-only) using a large set of images to achieve topic coverage. Save the model.
2. Load the model and use the .transform method on each image in the given batch.
3. As a separate step, aggregate the topics to understand the dominant themes.

As I understand it, BERTopic's scope ends once a topic is assigned to individual images. I think , We need an appropriate approach to identify dominant topics from these individual topics.

Alternatively, is there another approach that we can manage within the BERTopic scope to handle this?

[MaartenGr]

The issue arises when there are noisy images in the inference sample (e.g., random, unrelated images). For instance, random unrelated images can negatively impact the final outcome.

Couldn't you reduce the impact of these images by using the underlying probabilities and only select those that exceed a certain threshold?

Train a multimodal BERTopic (image-only) using a large set of images to achieve topic coverage. Save the model.

Note that the method of saving is very important here as pickle create a different method for extracting topics using .transform compared to saving the model with safetensors.

As I understand it, BERTopic's scope ends once a topic is assigned to individual images. I think , We need an appropriate approach to identify dominant topics from these individual topics.

Alternatively, is there another approach that we can manage within the BERTopic scope to handle this?

If you want the topic distribution, I would advise setting calculate_probabilities=True and saving the model with safetensors. That way, you get the same type of distributions as you would get with find_topics and would also give you the possibility to work with topics-images distributions. Thresholding would then allow you to select images that you find relevant.

[owwdesilva]

Hi Maarten, Thank you so much for the quick reply and the advice.

We used safetensors sterilizer when saving, not pickle, and we did not use calculate_probabilities=True while training the model. We will further explore these combinations and get back to you. Thanks again!