Unable to get topics for image only model

[owwdesilva]

Hi @MaartenGr ,

I have trained a image only model by referring this scetion of BERTopic documentation. I was able to successfully train & save the model and see the topic representation as well.

Then I have tried to load my saved model and find topics for another set of images using .transform() method. But unfortunately when I try to get topics I got an error.

This is how I tried to get the topics
image_topic = image_topic_model.transform(documents=None, images=images)

This is the error I'm getting.

ValueError: text input must be of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).

I have tried the same with .find_topics() method also with a single image, but i got similar issue on that method also.

Any advice on this ?

[MaartenGr]

Have you tried to pass the embedding model when loading in the topic model? For instance, BERTopic.load(embedding_model=MY_EMBEDDING_MODEL). I believe it is using the default model instead which cannot handle images.

[owwdesilva]

Hi @MaartenGr.

Thank you for your reply.

Yes I have passed my embedding model into the load function.

With the given clue, i have tested further with different configurations on model saving and loading still I'm getting the same error for all the tested combinations. However when I use the model without saving and loading, it works perfectly with .transform().

This is how I have saved my model.

topic_model.save(model_dir_local_path, serialization="safetensors", save_embedding_model=image_embedding_model)

This is how i have loaded my model.

image_topic_model = BERTopic.load(image_model_dir_local_path, embedding_model=image_embedding_model)

I have tried below combinations as well

1. Saving the model without embedding model and loading with embedding model.
2. Saving the model with embedding model and loading with embedding model.
3. Saving the model with pytorch serialization (with and without embedding model)
4. Saving the full model -> This got crashed , not sure its a performance issue on my notebook instance.

[MaartenGr]

Hmmm, that's unexpected. I would assume that loading the model like that would suffice.

Could you also try installing BERTopic from its main branch? I believe there was a fix related to loading the embedding model. Also, could you share the full error log?

[owwdesilva]

HI @MaartenGr ,

I have tried with master branch code too. but still I'm getting the same issue.

Here is the full error.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[41], line 1
----> 1 image_topic = mymodel.transform(images=test_images, documents=None)

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/bertopic/_bertopic.py:506, in BERTopic.transform(self, documents, embeddings, images)
    503     documents = [documents]
    505 if embeddings is None:
--> 506     embeddings = self._extract_embeddings(documents,
    507                                           images=images,
    508                                           method="document",
    509                                           verbose=self.verbose)
    511 # Check if an embedding model was found
    512 if embeddings is None:

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/bertopic/_bertopic.py:3410, in BERTopic._extract_embeddings(self, documents, images, method, verbose)
   3408     embeddings = self.embedding_model.embed_words(words=documents, verbose=verbose)
   3409 elif method == "document":
-> 3410     embeddings = self.embedding_model.embed_documents(documents, verbose=verbose)
   3411 elif documents[0] is None and images is None:
   3412     raise ValueError("Make sure to use an embedding model that can either embed documents"
   3413                      "or images depending on which you want to embed.")

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/bertopic/backend/_base.py:69, in BaseEmbedder.embed_documents(self, document, verbose)
     55 def embed_documents(self,
     56                     document: List[str],
     57                     verbose: bool = False) -> np.ndarray:
     58     """ Embed a list of n words into an n-dimensional
     59     matrix of embeddings
     60 
   (...)
     67         that each have an embeddings size of `m`
     68     """
---> 69     return self.embed(document, verbose)

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/bertopic/backend/_sentencetransformers.py:65, in SentenceTransformerBackend.embed(self, documents, verbose)
     51 def embed(self,
     52           documents: List[str],
     53           verbose: bool = False) -> np.ndarray:
     54     """ Embed a list of n documents/words into an n-dimensional
     55     matrix of embeddings
     56 
   (...)
     63         that each have an embeddings size of `m`
     64     """
---> 65     embeddings = self.embedding_model.encode(documents, show_progress_bar=verbose)
     66     return embeddings

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py:352, in SentenceTransformer.encode(self, sentences, prompt_name, prompt, batch_size, show_progress_bar, output_value, precision, convert_to_numpy, convert_to_tensor, device, normalize_embeddings)
    350 for start_index in trange(0, len(sentences), batch_size, desc="Batches", disable=not show_progress_bar):
    351     sentences_batch = sentences_sorted[start_index : start_index + batch_size]
--> 352     features = self.tokenize(sentences_batch)
    353     features = batch_to_device(features, device)
    354     features.update(extra_features)

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py:569, in SentenceTransformer.tokenize(self, texts)
    565 def tokenize(self, texts: Union[List[str], List[Dict], List[Tuple[str, str]]]):
    566     """
    567     Tokenizes the texts
    568     """
--> 569     return self._first_module().tokenize(texts)

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/sentence_transformers/models/CLIPModel.py:70, in CLIPModel.tokenize(self, texts)
     67 if len(images) == 0:
     68     images = None
---> 70 inputs = self.processor(text=texts_values, images=images, return_tensors="pt", padding=True)
     71 inputs["image_text_info"] = image_text_info
     72 return inputs

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/transformers/models/clip/processing_clip.py:101, in CLIPProcessor.__call__(self, text, images, return_tensors, **kwargs)
     98     raise ValueError("You have to specify either text or images. Both cannot be none.")
    100 if text is not None:
--> 101     encoding = self.tokenizer(text, return_tensors=return_tensors, **kwargs)
    103 if images is not None:
    104     image_features = self.image_processor(images, return_tensors=return_tensors, **kwargs)

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2872, in PreTrainedTokenizerBase.__call__(self, text, text_pair, text_target, text_pair_target, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   2870     if not self._in_target_context_manager:
   2871         self._switch_to_input_mode()
-> 2872     encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
   2873 if text_target is not None:
   2874     self._switch_to_target_mode()

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2930, in PreTrainedTokenizerBase._call_one(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   2927         return False
   2929 if not _is_valid_text_input(text):
-> 2930     raise ValueError(
   2931         "text input must be of type `str` (single example), `List[str]` (batch or single pretokenized example) "
   2932         "or `List[List[str]]` (batch of pretokenized examples)."
   2933     )
   2935 if text_pair is not None and not _is_valid_text_input(text_pair):
   2936     raise ValueError(
   2937         "text input must be of type `str` (single example), `List[str]` (batch or single pretokenized example) "
   2938         "or `List[List[str]]` (batch of pretokenized examples)."
   2939     )

ValueError: text input must be of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).

Please find below additional info,
This is how i load my saved model and I'm using clip-ViT-B-32 as my embedding_model.

# Train data
images = list(glob.glob(frames_local_folder_path+'/*.png'))

# Training my image model
embedding_model = 'clip-ViT-B-32'
MultiModalBackend = MultiModalBackend(embedding_model, batch_size=32)
umap_model = UMAP(n_neighbors=15, n_components=5, min_dist=0.0, metric='cosine', random_state=42)
ctfidf_model = ClassTfidfTransformer(reduce_frequent_words=True)
representation_model = {
    "Visual_Aspect": VisualRepresentation(image_to_text_model="nlpconnect/vit-gpt2-image-captioning")
}
hdbscan_model = HDBSCAN(min_cluster_size=5, metric='euclidean', prediction_data=True)
vectorizer = CountVectorizer(stop_words="english", ngram_range=(1, 2))

image_embeddings = MultiModalBackend.embed_images(images, verbose=False)

topic_model = BERTopic(
        embedding_model=MultiModalBackend, 
        representation_model=representation_model,
        umap_model=umap_model,
        hdbscan_model=hdbscan_model,
        vectorizer_model=vectorizer,
        ctfidf_model=ctfidf_model,
        verbose=True,
        top_n_words=5
)

topics, probs = topic_model.fit_transform(documents=None, images=images, embeddings=image_embeddings)

# Saving my model
# Note: here i have tried other combinations also.
topic_model.save(model_dir_local_path, serialization="pytorch")

# Loading my saved model
mymodel = BERTopic.load(model_dir_local_path, embedding_model=embedding_model )

# Trying to get image topics
image_topic = mymodel.transform(documents=None, images=test_images)

If you can suggest any workaround to this blocker I'm facing, That would be really helpful. As I'm using a huge data set as my training image data, at the moment the save is really required for me to reuse my trained model.

[MaartenGr]

I am not entirely sure but there might be something going on with this line:

MultiModalBackend = MultiModalBackend(embedding_model, batch_size=32)

You are overwriting a backend with a variable which might be an issue later on. Instead, do something like this:

embedding_model  = MultiModalBackend('clip-ViT-B-32', batch_size=32)

and pass that to embedding_model.

It seems you are using a different backend between saving and loading the model.

[owwdesilva]

Hi @MaartenGr

Thank you for pointing this out. This really helps me to resolve the issue. Earlier i was just passing 'clip-ViT-B-32' text as the embedding model according to the Docs. But with this approach (by passing MultiModalBackend('clip-ViT-B-32', batch_size=32)) as embedding model, the issue got resolved.

Thanks again for the support.