Whay "Transform" function is not deterministic ?

[StephChev]

Hello,
we are using the bertopic model to cluster documents. Then we are storing the model to pickle. This works well.
But as we are receiving new documents we want to know to each cluster it is related... but when we tried that we were very surprised to see that running the TRANSFORM function multiple time without modify anything is not always giving the same result.
You can see below a code that run 10 time the same transform for a list of documents and in each column run 1..10 you have the cluster ID and the related proba (ie "CLUSTERID" with "PROBA").
Is there an explanation about why relaunching multiple time the same transform, the result can be very different (not same cluster and even not the same proba...). Is Bertoptic TRANSFORM fucntion really not deterministic ?

Thanks a lot for your help
BR
Stephane

[MaartenGr]

Thank you for reaching out! Did you check the FAQ? It will give you some information about reproducibility.

[StephChev]

Thanks a lot for your answer @MaartenGr !
I set this up forcing the random seeds of Umap and HDBScan too

• umap_model.random_state=42
• hdbscan_model.random_state=42
• and set HDBSCAN prediction_data=True

This solved the random behavior of multiples run, but....

However this seems not to solve all "random" behaviors: I restarted the process from the begining:

• create the model with random state=42
• save the model to a pickle
• later... I reload the pickle model and to test that results are the same I give the same documents to the model as the ones used to generate clusters, then one ticket that is totally different

here is the result (2nd colum is the model cluster/proba, and other ones are different TRANSFORM call)

• all the TRANFORM runs give the same cluster/proba : good !
• in orange : for 3 tickets used in model they are classified in the same cluster and almost the same proba : not so bad
• in red 2 tickets are put in "other" cluster.. but they were initially in defined clusters (even with very high proba) : very bad
• in green : a totally different document that has been set to "other" : perfect !

Do you have any clue about what could happen ?

Thnaks a lot for your great help and patience !
Stephane

[MaartenGr]

Without seeing your full code, I believe this is actually expected behavior. Assuming you are using the original and UMAP and HDBSCAN packages, you can indeed set the random state to fix the training process, this will in turn also fix the predictions that it will create through transform. However, the way transform behaves in HDBSCAN is different from its training procedure. HDBSCAN, as a clustering algorithm, assumes that a given datapoint is clustered based on all the data available. This means that during training, it uses the full data to decide how a given point should be clustered. In contrast, during prediction (or transform), it uses a function called approximate_predict. As its name implies, it is merely an approximation of prediction rather than the exact same procedure during training.

As a result, HDBSCAN might create different results during training compared to prediction/transform. In other words, HDBSCAN does something different for fit compared to transform. You can find more information about that here.

[StephChev]

Thanks a lor @MaartenGr for your detailed explanation and links. This is clear now.
In fact we are managing thousands of tickets for an IT departement of a big company, and clustering helps to have a higher vision about global "subjects" identified in tickets (last 3 rolling months).
Next step is to try to match new tickets with existing tickets : so we are using transform to get the most relevant related cluster and get the 3 most representatives tickets of this cluster to give our support an idea about how similar tiockets has been solved.

A last question... when tranform is giving a cluster with a proba, what is really the meaning of this proba ? what we can deducde from this value ? if it is "low" does this mean the cluster is not "sure" ? if it is high, we can be sure that the cluster is the right one ?

best regards
Stephane

[MaartenGr]

A last question... when tranform is giving a cluster with a proba, what is really the meaning of this proba ? what we can deducde from this value ? if it is "low" does this mean the cluster is not "sure" ? if it is high, we can be sure that the cluster is the right one ?

Under the hood, and assuming you are using pickle to load and save the model (not safetensors or pytorch), it is using the approximate_predict function of HDBSCAN to perform the predictions. I would advise reading through the related link to understand what exactly is happening and what the meaning of the output is. Moreover, you can find more information about soft clustering here. Generally, we would indeed assume that a higher value of the probabilities indicates the cluster model is more confident a given point belongs to a given cluster.

[StephChev]

That's all very clear. I will dig a little more into documentation links you provided.
Thanks a lot for all your detailed explanations
BR
Stephane