curiosity

[zhchaoo]

experiment curiosity, use icm to add intrinsic rewards.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros and Trevor Darrell. Curiosity-driven Exploration by Self-supervised Prediction. In ICML 2017.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[MrSyee]

Hi, @zhchaoo
It is great job! I think it'll be better if you make a few change.

1. The graphs of losses for ICM in result part is needed for showing difference between naive DQN and curiosity.
2. In render part, the result gif video is needed, not an error message.

@Curt-Park What do you think? Curiosity is not rainbow series, so it is different from the concept of this tutorial. But it is also one of algorithms using DQN.

[Curt-Park]

@MrSyee I think this can be added as an extra topic.
@zhchaoo Thanks for your contribution. Please check @MrSyee 's comment.

[Curt-Park]

Colab URL:
https://colab.research.google.com/github/zhchaoo/rainbow-is-all-you-need/blob/feature/curiosity/11.curiosity.ipynb

[Curt-Park]

This seems like the most important part in this tutorial.

Currently, it doesn't look having self-contained information so that people understand it enough by this.

How about adding more information like mathematical formulation? or the difference from the naive DQN?

Reply via ReviewNB

[Curt-Park]

Line #38.            # Forward prediction: predict next state feature, given current state feature and action (one-hot)

Suggestion (Google Docstyle):

"""Forward prediction.
predict next state feature, given current state feature and action (one-hot).

"""

Reply via ReviewNB

[Curt-Park]

Line #39.            pred_s_next = F.relu(self.pred_module1( torch.cat([feature_x, a_vec.float()], dim = -1).detach()))

suggestion (Black Style):

pred_s_next = F.relu(
self.pred_module1(torch.cat([feature_x, a_vec.float()], dim =-1).detach())

)

Reply via ReviewNB

[Curt-Park]

Line #59.            self.use_extrinsic = use_extrinsic

Need to add description in the docstring

Reply via ReviewNB

[Curt-Park]

Line #60.            self.intrinsic_scale = intrinsic_scale

Need to add description in the docstring

Reply via ReviewNB

[Curt-Park]

Line #73.            self.icm = ICM(obs_dim, action_dim).to(self.device)

Need to add description in the docstring

Reply via ReviewNB

[Curt-Park]

Line #201.            a_vec = F.one_hot(action, num_classes = self.env.action_space.n).reshape(-1,self.env.action_space.n) # convert action from int to one-hot format

Line is too long. Suggestion (Black style):

# convert action from int to one-hot format
a_vec = F.one_hot(

action, num_classes=self.env.action_space.n

).reshape(-1, self.env.action_space.n)

Reply via ReviewNB

[Curt-Park]

need to show the result

Reply via ReviewNB

[Curt-Park]

Line #37.        def pred(self, feature_x, a_vec):

Need to add type annotation:

feature_x: torch.Tensor, a_vec: torch.Tensor

Reply via ReviewNB