How to calculate Shap value after training model?

[TengGao]

Hi:

I had used Vaex to train a lightgbm model that had 2bililion rows, and I can get the feature importance just as in the demo. However, if I want to calculate the shap values, I got stuck. Had anyone tried to calculate Shap?

best
Mike

[JovanVeljanoski]

Hi,

First of all, good job on training a 2 billion rows model!

About your problem: are you referring to the large volume of data, or are you asking how to access the model to plug into shap?

If it is about the large volume of data, then I don't know if we can help much. Shap is usually computationally expensive even for small-ish amount of data. Maybe you can try sub-sampling if it feasible for your case. In any case, I think this is unrelated to Vaex. For more ideas maybe better to ask on the Shap side?

If you are asking how to access the underlying model so you can use with shap, this is easily done, assuming you can something like:

model_wrapper = IncrementalPredictor(...)  # Or any other vaex-ml model
model_wrapper.fit(df_train)

# Now you can access the underlying model via
model = model_wrapper.model

After that you can use model as you would normally use without vaex.

[TengGao]

Thank you very very much! My question is indeed both, but thank you for your suggestion I will try subsampling, especially now I know how to access the underlying model.