Keras siamese model history is empty #122182
-
BodyI am making a siamese neural network with triplet loss using keras, and have encountered an odd problem. I tried saving my history twice: once in a callback (saved as a dictionary), and once after training with model.fit(), using pickle (saved as a history object). In both instances, the code runs smoothly, but afterwards the saved files are only several hundred bytes and when I plot them with matplotlib, the plot is empty. Here's the callback I passed to model.fit():
def on_epoch_end(self, epoch, logs=None):
metrics_callback = MetricsCallback(train_set, val_set,simese_model_1,folder_path, file_name) Guidelines
|
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 5 replies
-
Hi, have you tried running simply history = model.fit() Also check the recorded metrics logs in |
Beta Was this translation helpful? Give feedback.
This comment was marked as spam.
This comment was marked as spam.
-
Hey, I was eventually able to solve the issue. The history was empty because i had only trained on one epoch, not because there was an issue saving. The reason I never got past the first epoch is because of another issue I had to solve. as for the miscalculations, it turns out that this was a problem with my data generator, that made image triplets. It has to be recreated every epoch, or it doesn't generate anything and the metrics are zero. Here's the updated code. In it I save each metric as a key in logs, and have a history list (self.logs_history) to which I add this epoch's history to the list. This is because logs resets every epoch. then i save the history (a list of directories) with pickle. Also, I added the functions I used to make my data generators and call them ever epoch to make a new generator in on_epoch_end. `
metrics_callback = MetricsCallback(train_x,train_y, val_x,val_y,siamese_model_1,alpha,folder_path, file_name="siamese_train_hist") Alternatively, instead of using logs it is possible to create a list for every metric in the init function like this: And append them in on_epoch_end(instead of inserting them to logs): And then add them all to a history dictionary in a separate callback:
save_hist_callback=SaveHistoryCallback(folder_path, file_name_savehist, metrics_callback) I hope this helps anyone who stumbles upon this. Huge thanks to @markolalovic for the helpful and quick response :) |
Beta Was this translation helpful? Give feedback.
Hey, I was eventually able to solve the issue. The history was empty because i had only trained on one epoch, not because there was an issue saving. The reason I never got past the first epoch is because of another issue I had to solve. as for the miscalculations, it turns out that this was a problem with my data generator, that made image triplets. It has to be recreated every epoch, or it doesn't generate anything and the metrics are zero.
Here's the updated code. In it I save each metric as a key in logs, and have a history list (self.logs_history) to which I add this epoch's history to the list. This is because logs resets every epoch. then i save the history (a list of directories) w…