Calling learn.get_preds() consecutively with the same data gives different results

[kiview]

When calling learn.get_preds(dl=test_dl, with_decoded=True) for new data, as shown in the tutorial notebooks, I get the correct predictions and probabilities when calling it initially. However, calling it again with the same dl leads to completely wrong decoded predictions, and the probabilities tensors consist only of 0 and 1 values.

Obviously, there is something stateful happening in the learn object which I don't understand. I also had the feeling this is related to the reorder parameter, however, I could not really reproduce this.

[oguiza]

Hi @kiview,
Which tutorial nb are you referring to?
I'd like to update it. I recommend using this:

probas, _, preds = learn.get_X_preds(X) # with_decoded is set to Tru by default. The valid tfms and batch tfms will be applied.

[kiview]

Thanks, I was referring to https://github.com/timeseriesAI/tsai/blob/main/tutorial_nbs/01_Intro_to_Time_Series_Classification.ipynb. Could you still explain, why I observed this strange non-idempotent behavior?

[meanpenguin]

It sounds something like the issue I reported here: #135

Try loading the latest from !pip install git+https://github.com/timeseriesAI/tsai.git -U

[kiview]

Thanks a lot @meanpenguin, this sounds indeed like the issue you have linked!

[oguiza]

Hi @kiview,
Should I understand that the issue's disappeared when you have upgraded tsai to the latest version?

[kiview]

I haven't yet updated, but will come back once I verified this.

[bob-mcrae]

It looks like there may have been a bug at some point, but it also looks like it was addressed. However, I am seeing something very similar, or perhaps the same. Here is an example of feeding the same, exact data into learner.get_X_preds, but getting different results each time.

Positive Case (bug?):

active_models = load_active_models()                   # currently, just a single learner
filename      = 'DA 02.25.21.2.csv'                    # just laoding data from a file on disk
df            = load_file(DATA_DIR + filename)
df            = add_channels(df)                       # pre-processing
x_data        = prep_4_model(df, active_model)         # conforming the shape for the learner
for _ in range(3):
  print(_, active_models[0].get_X_preds(x_data))

0 (tensor([[0.0534, 0.9466]]), None, '[1]')
1 (tensor([[0., 1.]]), None, '[1]')
2 (tensor([[0., 1.]]), None, '[1]')

Negative Case -- re-loading the learner each time:

for _ in range(3):
  active_models = load_active_models()
  print(_, active_models[0].get_X_preds(x_data))

0 (tensor([[0.0534, 0.9466]]), None, '[1]')
1 (tensor([[0.0534, 0.9466]]), None, '[1]')
2 (tensor([[0.0534, 0.9466]]), None, '[1]')

[oguiza]

Hi @bob-mcrae,
Thanks for raising this. I've verified it and it was indeed an issue. I've logged it as #282.
I've also fixed it. Here's the code I've used to verify it works as expected:

X, y, splits = get_UCR_data('LSST', split_data=False)
tfms = [None, TSClassification()]
batch_tfms = TSStandardize(by_sample=True)
dls = get_ts_dls(X, y, splits=splits, tfms=tfms, batch_tfms=batch_tfms)
learn = ts_learner(dls, InceptionTimePlus, metrics=accuracy, cbs=[ShowGraph()])
learn.fit_one_cycle(10, 1e-2)
output = learn.get_X_preds(X[splits[1]], with_decoded=True)[0]
output2 = learn.get_X_preds(X[splits[1]], with_decoded=True)[0]
test_eq(output, output2)

I've closed the issue, but please let me know if you find any other issues.

[bob-mcrae]

Thanks again Ignacio. I confirmed 0.2.24 does indeed address the issue.