index.xml

<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Markus Peschl</title>
    <link>https://mlpeschl.com/</link>
      <atom:link href="https://mlpeschl.com/index.xml" rel="self" type="application/rss+xml" />
    <description>Markus Peschl</description>
    <generator>Wowchemy (https://wowchemy.com)</generator><language>en-us</language><lastBuildDate>Thu, 02 Feb 2023 00:16:45 +0100</lastBuildDate>
    <image>
      <url>https://mlpeschl.com/images/icon_hu6a98d456223992a97ada131f146ee6ab_65108_512x512_fill_lanczos_center_3.png</url>
      <title>Markus Peschl</title>
      <link>https://mlpeschl.com/</link>
    </image>
    
    <item>
      <title>Robust Scheduling with GFlowNets</title>
      <link>https://mlpeschl.com/publication/gflownet_scheduling/</link>
      <pubDate>Thu, 02 Feb 2023 00:16:45 +0100</pubDate>
      <guid>https://mlpeschl.com/publication/gflownet_scheduling/</guid>
      <description></description>
    </item>
    
    <item>
      <title>Learning Perturbations for Soft-Output Linear MIMO Demappers</title>
      <link>https://mlpeschl.com/publication/mimo-perturb/</link>
      <pubDate>Tue, 27 Dec 2022 22:49:21 -0800</pubDate>
      <guid>https://mlpeschl.com/publication/mimo-perturb/</guid>
      <description></description>
    </item>
    
    <item>
      <title>MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning</title>
      <link>https://mlpeschl.com/publication/moral-aamas/</link>
      <pubDate>Fri, 27 May 2022 22:34:50 -0800</pubDate>
      <guid>https://mlpeschl.com/publication/moral-aamas/</guid>
      <description></description>
    </item>
    
    <item>
      <title>Constrained versus Multi-Objective Reinforcement Learning. A Fundamental Difference?</title>
      <link>https://mlpeschl.com/post/multi_objective_rl/</link>
      <pubDate>Sat, 17 Apr 2021 21:26:22 +0200</pubDate>
      <guid>https://mlpeschl.com/post/multi_objective_rl/</guid>
      <description>&lt;p&gt;&lt;img src=&#34;blog_safety.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;
&lt;h2 id=&#34;introduction&#34;&gt;Introduction&lt;/h2&gt;
&lt;p&gt;If you have ever taken a class in mathematical optimization, you are probably
familiar with the notion of a &lt;em&gt;constrained optimization problem&lt;/em&gt;. If not, then
don&amp;rsquo;t worry, because the idea is really simple. I give you a function $f(x)$
which maps a quantity $x$ to a real number, and I want to find an $x$, such that
$f(x)$ is maximal. For example, let&amp;rsquo;s say that $f(x)$ is a magic function
that miraculously measures my knowledge about reinforcement learning research
in terms of how many arXiv papers I read per day. Assuming that reading more
papers does actually increase my knowledge (which, given the amount of
papers on deep RL being published at the moment, I am pretty doubtful of) I
would have to spend my whole day reading papers on arXiv to maximize $f$. Of
course, this is not how things go&amp;hellip; I completely forgot about all the constraints!
I cannot read arXiv papers all day, because I actually need to make a living,
write a thesis and &lt;a href=&#34;https://twitter.com/gautamcgoel/status/1377353733873434624&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;thank reviewer #2 for their outstandingly insightful criticism
on my latest submission&lt;/a&gt;. Formally speaking, this could translate into
another (possibly vector-valued) function $g(x)$ that has to satisfy $g(x) \leq c$
for some manually defined $c$.&lt;/p&gt;
&lt;p&gt;Okay great, let&amp;rsquo;s just work with constraints then to avoid certain solutions in
our original optimization problem. But wait, why do we not encode all the other
&lt;em&gt;things&lt;/em&gt; we want to do into $f$ straight away? In fact, if you reread the
first paragraph and roughly replaced &lt;em&gt;constraints&lt;/em&gt; with &lt;em&gt;multiple objectives&lt;/em&gt; then
the story would still make perfect sense. The reason for this is that
at the end of the day, the differences between &lt;em&gt;constrained optimization&lt;/em&gt; and
&lt;em&gt;multi-objective&lt;/em&gt; optimization are rather subtle.&lt;/p&gt;
&lt;p&gt;As it turns out, multi-objective and constrained optimization have potentially
very important applications to reinforcement learning (RL) as well and this should
come to no surprise. It is a well known fact that the correct specification of
a reward function is one of the biggest challenges when designing RL systems
for real-world applications, which has raised many questions in the AI safety
literature [1]. From this point of view, constraining the agent to do certain
things might seem like the most straight forward extension to the classical RL
optimization goal which has led to constrained benchmark tasks such as the
OpenAI Safety Gym [2]. On the other hand, multi-objective decision making
has been a very active branch of RL related research. For a survey, see [3].&lt;/p&gt;
&lt;p&gt;In this post, we will explore the fundamental differences of the two approaches
in the RL sphere from a mathematical perspective. However, note that most
of the details will just as well apply for general optimization problems, which
are not necessarily related to RL. To do so, we will&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Formally define the respective optimization problems.&lt;/li&gt;
&lt;li&gt;Prove a basic equivalence result in the case of linear objective preferences.&lt;/li&gt;
&lt;li&gt;Discuss practical implications for RL safety.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&#34;1-definitions&#34;&gt;1. Definitions&lt;/h2&gt;
&lt;h3 id=&#34;markov-decision-processes&#34;&gt;Markov Decision Processes&lt;/h3&gt;
&lt;p&gt;Let&amp;rsquo;s start with a common framework before we explore each of the approaches. In general, the underlying environments that we consider for reinforcement learning problems are &lt;em&gt;Markov decision processes&lt;/em&gt; (MDPs). Formally, an MDP consists of a tuple $\langle \mathcal{S}, \mathcal{A}, \mathcal{T}, r \rangle$, where $\mathcal{S}$ is the set of possible states, $\mathcal{A}$ is the set of actions, $\mathcal{T}(s&amp;rsquo; | s, a)$ is the transition probability of landing in state $s&amp;rsquo;$ when executing action $a$ in state $s$ and $r(s,a)$ is the associated reward. The finite-horizon reinforcement learning problem then consists of finding a policy $\pi : \mathcal{S} \rightarrow \Delta_\mathcal{A}$, with $\Delta_\mathcal{A}$ denoting the class of probability distributions over $\mathcal{A}$, which for a discount factor $0  &amp;lt; \gamma \leq 1$ maximizes the expected return \begin{equation}
\max_{\pi} J(\pi) = \mathbb{E}_\pi\left[ \sum_{t=0}^T \gamma^t r(s_t,a_t) \right].
\end{equation}&lt;/p&gt;
&lt;h3 id=&#34;constrained-markov-decision-processes&#34;&gt;Constrained Markov Decision Processes&lt;/h3&gt;
&lt;p&gt;Constrained Markov decision processes (CMDP) on the other hand consist of a tuple $\langle \mathcal{S}, \mathcal{A}, \mathcal{T}, r, \mathbf{c}, \mathbf{d}\rangle$, where we extend the traditional MDP with a (multivariate) constraint function $\mathbf{c} = (c_1,\dots, c_k) : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}^k$ as well as a constraint hyperparameter $\mathbf{d} \in \mathbb{R}^k$. The CMDP goal is to solve
\begin{align}
\max_{\pi} \ &amp;amp; \ J(\pi) \
\ \ \ \ \text{s.t.} \ &amp;amp;\ J_\mathbf{c}(\pi)\leq \mathbf{d} \label{cmdp},
\end{align}
where $J_\mathbf{c}(\pi) = \mathbb{E}_\pi\left[ \sum_{t=0}^T \gamma^t \mathbf{c}(s_t,a_t) \right]$ denotes the vector of cumulative expected constraint costs. From this definition it is clear that we divide the space of policies $\Pi$ into a set of feasible and infeasible policies.&lt;/p&gt;
&lt;h3 id=&#34;multi-objective-markov-decision-processes&#34;&gt;Multi-Objective Markov Decision Processes&lt;/h3&gt;
&lt;p&gt;Finally, a multi-objective Markov decision process (MOMDP) is represented by a tuple $\langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathbf{r}, \Omega, f_\Omega\rangle$, where $\mathbf{r}(s,a) \in \mathbb{R}^m$ is now a vector-valued reward function, $\Omega$ is a set of preferences and $f_{\boldsymbol{\omega}}(\mathbf{r})$ are preference functions, which scalarize the multi-objective reward function according to a preference $\boldsymbol{\omega} \in \Omega$. In the following, we will consider only linear preference functions $f_{\boldsymbol{\omega}}(\mathbf{r(s,a)}) = \boldsymbol{\omega}^T \mathbf{r}(s,a)$. Given a fixed preference $\boldsymbol{\omega}$, we see that we can maximize $f_{\boldsymbol{\omega}}(\mathbf{r})$ as usual and the problem reduces to a standard MDP task. When the preferences are a-priori unknown, however, we have to consider a Pareto boundary
$$\mathcal{F} := \begin{Bmatrix}J_\mathbf{r}(\pi) | \pi \in \Pi \wedge \nexists \pi&amp;rsquo; \neq \pi  : J_\mathbf{r}(\pi&amp;rsquo;) \geq J_\mathbf{r}(\pi) \end{Bmatrix},$$ where $J_\mathbf{r}(\pi) =  \mathbb{E}_\pi\left[ \sum_{t=0}^T \gamma^t \mathbf{r}(s_t,a_t) \right]$. In general, we can assume very little about the shape of $\mathcal{F}$, in fact it might not necessarily be convex [4]. For that reason, we define the convex coverage set (CCS) as&lt;/p&gt;
&lt;p&gt;\begin{equation}
\mathcal{F}^* := \begin{Bmatrix}J_\mathbf{r}(\pi) \in \mathcal{F} \ \vert\ \exists\boldsymbol{\omega}\in \Omega : \boldsymbol{\omega}^T J_\mathbf{r}(\pi) \geq   \boldsymbol{\omega}^T J_\mathbf{r}(\pi&amp;rsquo;), \ \forall J_\mathbf{r}(\pi&amp;rsquo;)\in \mathcal{F}  \end{Bmatrix}.
\end{equation}&lt;/p&gt;
&lt;p&gt;To clarify this definition intuitively, consider figure 1. The CCS is a subset of of the Pareto boundary, which consists of its convex parts. More formally, the CCS is the set of optimal solutions that we are optimizing for in the MOMDP task with linear preferences. Thus, our MOMDP goal can be interpreted as calculating (or in practice approximating) the complete CCS.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Figure 1. Taken from Yang et al. [4]. (a) The Pareto frontier might not be convex. In this example, the CCS consists of points A-H but does not include K, whereas A-K form the Pareto boundary. (b) given a preference vector $\boldsymbol{\omega}$ the optimal solution will be a point on the CCS that maximizes the projection onto $\boldsymbol{\omega}$.&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;paretoccs.png&#34; alt=&#34;space-1.jpg&#34;&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id=&#34;2-whats-the-difference&#34;&gt;2. What&amp;rsquo;s the Difference?&lt;/h2&gt;
&lt;p&gt;When comparing the multi-objective and the constrained case, it immediately becomes apparent that there is a close connection between the two approaches. To formalize this connection, we will now consider a fixed MDP $\mathcal{M} = \langle \mathcal{S}, \mathcal{A}, \mathcal{T}, r \rangle$ that we extend by a single constraint function $c\geq 0$ and a hyperparameter $d\in\mathbb{R}$. We will call this CMDP $\mathcal{M_C}$. At the same time, consider the MOMDP $\mathcal{M}_{MO}$ obtained by extending $\mathcal{M}$ by a second reward function $r_2 = -c$. We would now like to investigate, when an optimal solution $\pi^* $ in $\mathcal{M_C}$ can also be obtained in $\mathcal{M}_{MO}$ when given a specific preference. We from now on denote $\Pi^*_C(d)$ the set of optimal policies for $\mathcal{M_C}$ (depending on the hyperparameter $d$). Formally, we would like to find out the following:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; For which values of $d$ does $\Pi^* _C(d) $  have a non empty intersection with policies on the convex coverage set $\mathcal{F^*}$ of $\mathcal{M}_{MO}$?&lt;/p&gt;
&lt;p&gt;(Note that we are abusing notation and will be writing $\pi \in \mathcal{F}$ iff $J_\mathbf{r}(\pi) \in \mathcal{F}$). The reason we are interested in this question is because assuming global optimization, by our definition a MOMDP algorithm will only optimize for policies which result in returns lying on the CCS. This means that if for given $d$ the optimal CMDP solution does not lie on the CCS then we will not be able to recover it when approximating the CCS for the MOMDP problem. We will start by proving a simple Lemma.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Lemma 1:&lt;/strong&gt; For each constraint parameter $d$, if there exists an optimal policy then there exists an optimal policy that lies on the Pareto boundary of the corresponding MOMDP : $\Pi^*_C(d) \bigcap \mathcal{F} \neq \emptyset$.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Proof:&lt;/em&gt; Let $d\in \mathbb{R}$ and $\pi^* \in \Pi^*_C(d)$ be arbitrary. If $\pi^*$ lies on the Pareto boundary, we are done. Otherwise, assume $\pi^*$ does not lie in $\mathcal{F}$. By definition, there must exist a policy $\pi&amp;rsquo;$, such that $J_{r_1}(\pi&amp;rsquo;) \geq J_{r_1}(\pi^*)$ and $J_{r_2}(\pi&amp;rsquo;) \geq J_{r_2}(\pi^*)$ with strict inequality holding for at least one of the two inequalities. In case the first inequality strictly improves, it must be that $J_{r_2}(\pi&amp;rsquo;) \leq -d$, otherwise $\pi&amp;rsquo;$ would be another feasible policy that improves upon $\pi^*$ in the CMDP. However, this would contradict $J_{r_2}(\pi&amp;rsquo;) \geq J_{r_2}(\pi^*) &amp;gt; -d$. Therefore, we can assume that it must be the second inequality that strictly improves, i.e. $J_{r_2}(\pi&amp;rsquo;) &amp;gt; J_{r_2}(\pi^*) &amp;gt; -d$. This implies that $\pi&amp;rsquo;$ is also feasible in the CMDP and must yield the same return, namely $J_{r_1}(\pi&amp;rsquo;) = J_{r_1}(\pi^*)$ which shows that $\pi&amp;rsquo; \in \Pi^*_C(d)$. Since $\pi&amp;rsquo;$ was an arbitrary policy that strictly improves upon a policy in $\Pi^*_C(d)$, we can thus repeat the same argument for $\pi&amp;rsquo;$ until we obtain a policy $\pi_\mathcal{F} \in \Pi^*_C(d) \cap \mathcal{F}$ that can no longer be further improved upon (assuming bounded rewards). $\blacksquare$&lt;/p&gt;
&lt;p&gt;Lemma 1 guarantees that we can always find an optimal solution of the CMDP that also lies on the Pareto boundary of the corresponding MOMDP. Intuitively, this makes sense: From all the feasible optimal solutions of the CMDP we take the optimal solution with the least constraint violations. While this lemma shows us that optimal CMDP solutions are somewhat efficient with respect to the MOMDP, it does not state anything about the CCS. However, what immediately follows from the lemma is that the optimal CMDP solution can be recovered in the MOMDP setting if the Pareto boundary is equal to the CCS.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Theorem 1:&lt;/strong&gt; If $\mathcal{F} = \mathcal{F^* }$ then for each constraint parameter that allows feasible policies, there exists a preference that recovers the optimal CMDP solution, that is: $\forall d \in \mathbb{R}$ s.t. $\Pi^*_C(d) \neq \emptyset \ \exists \boldsymbol{\omega} \in \Omega, \pi^* \in \Pi^*_C(d): \pi^* = \text{arg} \max_{\pi} \boldsymbol{\omega}^T J_{\boldsymbol{r}}(\pi)$.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Proof:&lt;/em&gt; The proof follows from the fact that $\mathcal{F} = \mathcal{F}^* $ combined with lemma $1$ implies that there exists $\pi^* \in \mathcal{F}^* \cap \Pi^*_C(d)$. By definition of $\mathcal{F^*}$ the existence of $\boldsymbol{\omega} \in \Omega$ with  $\pi^* = \text{arg} \max_{\pi} \boldsymbol{\omega}^T J_{\boldsymbol{r}}(\pi)$ follows immediately.  $\blacksquare$&lt;/p&gt;
&lt;p&gt;This theorem shows us that in some cases, the CMDP solution will automatically be optimized for in CCS approximation algorithms with linear preferences. In this sense, we could consider the CMDP problem to be a subset of the MOMDP problem, where hyperparameter tuning of $d$ is equivalent to adapting to a different preference vector $\boldsymbol{\omega}$. Unfortunately, however, the converse of the theorem does not hold, which means that in the non-convex case the CMDP solution will not lie on the CCS and thus cannot be recovered by optimizing for a certain linear preference between constraint and rewards. To illustrate this, consider figure $2$.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Figure 2: When the CCS is not equal to the Pareto boundary, the CMDP solution will generally not be recoverable through linear preferences. In this example $\pi^*$ is an optimal CMDP solution which does not lie on the Pareto boundary $\mathcal{F}$. However, as lemma $1$ suggests, we can find an optimal $\tilde{\pi}^ * $ that does lie in $\mathcal{F}$, which yields higher $r_2$ (lower constraint cost) but the same original reward $r_1$. Since in this case the CCS consists of A-E but does not include $\tilde{\pi}^ * $, the scalarized reward $\boldsymbol{\omega}^T\mathbf{r}$ does not get maximized by $\tilde{\pi}^ *$, but by a point on the CCS (in this example that is C).&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;pareto_ccs_sketch.png&#34; alt=&#34;space-1.jpg&#34;&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id=&#34;discussion&#34;&gt;Discussion&lt;/h2&gt;
&lt;p&gt;The illustrations above are mostly motivated from a practical point of view, namely: What kind of formulation is suitable in which kinds of settings? In general, one could argue that
multi-objective RL is slightly more general than constrained RL, albeit both have their respective advantages and disadvantages. Lemma $1$ tells us that if we optimize for a complete
Pareto boundary, then this will automatically include solutions for constrained problems. When you think about it, this is a rather trivial statement. Pareto optimal solutions
cannot be improved upon any further, so if we have the complete set, then of course this is also where our constrained solutions will lie, since these are always optimal with
respect to the primary objective while lying above a certain threshold for the secondary objectives. One difference, however, is that constrained solutions actually do not
care about the magnitude of the constraints, as long as they meet the threshold. This is also reflected in the Lagrangian of the constrained optimization problem&lt;/p&gt;
&lt;p&gt;$$\min_{\pi}\max_{\lambda \geq 0}\mathcal{L}(\pi, \lambda) = -J(\pi) - \lambda (J_{\mathbf{c}}(\pi) - \mathbf{d}).$$&lt;/p&gt;
&lt;p&gt;If the constraints are not satisfied, then $\lambda (J_{\mathbf{c}}(\pi) - \mathbf{d}) &amp;gt; 0$ which, when maximizing over $\lambda$, will yield an infinite value. However, once the constraints are satisfied
$\lambda (J_{\mathbf{c}}(\pi) - \mathbf{d}) \leq 0$ for each $\lambda$. Maximizing this term for fixed $\pi$ over $\lambda$ would essentially set $\lambda$ to $0$, thus ignoring the magnitude $|J_{\mathbf{c}}(\pi) - \mathbf{d}|$
on the overall objective.&lt;/p&gt;
&lt;p&gt;In conclusion, we have to be somewhat careful when choosing constrained RL over multi-objective RL, especially when it is not entirely clear what the reward and constraint functions actually encode. If we
can clearly measure some kind of constraint that we would like to put into our system which should not at all be violated, then constrained RL might be the right framework. However, we
can expect that optimizing for constrained solutions might lead our agent to take shortcuts in order to just barely fulfill the constraints and this could pose a big problem in real-world
applications where constraints are not so easy to write down. On the other hand, multi-objective RL offers a way to maximally care about all the objectives at hand, and trade them off
by a user specified preference. This generally comes with a larger computational cost and might or might not be feasible depending on the problem. Furthermore, multi-objective RL algorithms
have to make some assumptions about which solutions should be computed at all. Optimizing for linear preferences only results in approximating the CCS, but as depicted in figure $2$ the CCS might
not include our desired solution in very nonconvex cases. Whether or not this difference poses a greater problem for more complex applications is rather unclear at this point. In my opinion, the more complex a problem gets and the more fine grained actions our agents are allowed to take, the CCS might actually be a really good approximation of the Pareto boundary.&lt;/p&gt;
&lt;h2 id=&#34;references&#34;&gt;References&lt;/h2&gt;
&lt;p&gt;[1] Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., &amp;amp; Mané, D. (2016). Concrete Problems in AI Safety. &lt;a href=&#34;http://arxiv.org/abs/1606.06565&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;http://arxiv.org/abs/1606.06565&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;[2] Ray, A., Achiam, J., &amp;amp; Amodei, D. (2019). Benchmarking Safe Exploration in Deep Reinforcement Learning. &lt;a href=&#34;https://www.semanticscholar.org/paper/Benchmarking-Safe-Exploration-in-Deep-Reinforcement-Achiam-Amodei/4d0f6a6ffcd6ab04732ff76420fd9f8a7bb649c3&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;https://www.semanticscholar.org/paper/Benchmarking-Safe-Exploration-in-Deep-Reinforcement-Achiam-Amodei/4d0f6a6ffcd6ab04732ff76420fd9f8a7bb649c3&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;[3] Roijers, D. M., Vamplew, P., Whiteson, S., Nl, A. W., &amp;amp; Dazeley, R. (2013). A Survey of Multi-Objective Sequential Decision-Making. In Journal of Artificial Intelligence Research (Vol. 48).&lt;/p&gt;
&lt;p&gt;[4] Yang, R., Sun, X., &amp;amp; Narasimhan, K. (2019). A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation. &lt;a href=&#34;https://arxiv.org/abs/1908.08342&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;https://arxiv.org/abs/1908.08342&lt;/a&gt;&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>ADRQN-PyTorch: A Torch implementation of the action-specific deep recurrent Q network.</title>
      <link>https://mlpeschl.com/post/tiny_adrqn/</link>
      <pubDate>Fri, 06 Nov 2020 15:23:13 +0100</pubDate>
      <guid>https://mlpeschl.com/post/tiny_adrqn/</guid>
      <description>&lt;h2 id=&#34;before-you-read-source-code&#34;&gt;Before you read (source code)&lt;/h2&gt;
&lt;p&gt;An open source Jupyter notebook including source code for this blog post is available. Click the link below to open an interactive version of the source code in Google Colab.&lt;/p&gt;
&lt;p&gt;Note: Enable GPU usage in the runtime settings for training with cuda.
&lt;a href=&#34;https://colab.research.google.com/github/mlpeschl/stat-notebooks/blob/master/Model_Free_RL/ADRQN.ipynb&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;&lt;img src=&#34;https://colab.research.google.com/assets/colab-badge.svg&#34; alt=&#34;Open In Colab&#34;&gt;&lt;/a&gt;&lt;/p&gt;
&lt;h1 id=&#34;introduction&#34;&gt;Introduction&lt;/h1&gt;
&lt;p&gt;Deep Q networks have proven to be an easy to implement method for solving control problems in both continuous or large discrete state spaces. For instance, a highly cited paper by a group of Google Deepmind researchers [1] proposes the use of a deep (convolutional) neural network for approximating the optimal Q function in Atari games. Since then, numerous improvements to the deep Q network (DQN) algorithm have emerged, one notable example being the Rainbow agent [2], which combines fruitful approaches from different subfields of reinforcement learning including distributional RL, multi-step targets and dueling networks.&lt;/p&gt;
&lt;p&gt;While the above efforts to improve upon the performance of DQNs have proven to be effective, most of the research focuses on fully observable environments, or at least introduces workarounds to overcome partial observability by combining multiple consecutive frames into a single observation [1]. If such design choices are not possible, one needs a different mechanism for keeping track of the history of observations when approximating the Q function with a neural network. A rather straight forward way to achieve this, is by including recurrence in the Q network and this is exactly what the action-specific deep recurrent Q network (ADRQN) does [3].&lt;/p&gt;
&lt;p&gt;In this post, we will go through a simple PyTorch implementation of the ADRQN including a small scale experiment on classical control tasks of the OpenAI gym.&lt;/p&gt;
&lt;h1 id=&#34;model-architecture&#34;&gt;Model Architecture&lt;/h1&gt;
&lt;p&gt;The ADRQN [3] is a modification of the DQN algorithm, which introduces an intermediate LSTM layer for remembering action-observation pairs when dealing with partial observability. An outline of the originally proposed architecture can be seen below in figure 1.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align:center&#34;&gt;&lt;em&gt;Figure 1 (taken from the original paper [3]): ADRQN architecture&lt;/em&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;adrqn_architecture.png&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;At each time step $t$ the model takes an observation $o_{t}$ and the action that led to that observation $a_{t-1}$, as well as the hidden state $h_{t-1}$ from the last forward pass of the LSTM layer. The observation $o_t$ gets fed through a series of convolutional downsampling layers, whereas the action $a_{t-1}$ gets embedded in a higher dimensional space through a fully connected linear layer (named IP in the figure). Then, the downsampled observation and embedded action get concatenated and fed into a LSTM, which updates its hidden state $h_t$ and outputs a sequence that gets fed into another linearly connected layer. The output size of the final linear layer matches the number of actions to approximate Q values for each possible action.&lt;/p&gt;
&lt;p&gt;Since we will be testing this architecture on low-dimensional physics based state vectors of the classical control tasks in the OpenAI gym, we replace the convolutional layers with linear layers and ReLU activations. Furthermore, we implement an act function into the model class for executing epsilon greedy actions given an action-observation pair.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;class ADRQN(nn.Module):
    def __init__(self, n_actions, state_size, embedding_size):
        super(ADRQN, self).__init__()
        self.n_actions = n_actions
        self.embedding_size = embedding_size
        self.embedder = nn.Linear(n_actions, embedding_size)
        self.obs_layer = nn.Linear(state_size, 16)
        self.obs_layer2 = nn.Linear(16, 32)
        self.lstm = nn.LSTM(input_size = 32 + embedding_size, hidden_size = 128, batch_first = True)
        self.out_layer = nn.Linear(128, n_actions)

    def forward(self, observation, action, hidden = None):
        #Takes observations with shape (batch_size, seq_len, state_size)
        #Takes one_hot actions with shape (batch_size, seq_len, n_actions)
        action_embedded = self.embedder(action)
        observation = F.relu(self.obs_layer(observation))
        observation = F.relu(self.obs_layer2(observation))
        lstm_input = torch.cat([observation, action_embedded], dim = -1)
        if hidden is not None:
            lstm_out, hidden_out = self.lstm(lstm_input, hidden)
        else:
            lstm_out, hidden_out = self.lstm(lstm_input)

        q_values = self.out_layer(lstm_out)
        return q_values, hidden_out

    def act(self, observation, last_action, epsilon, hidden = None):
        q_values, hidden_out = self.forward(observation, last_action, hidden)
        if np.random.uniform() &amp;gt; epsilon:
            action = torch.argmax(q_values).item()
        else:
            action = np.random.randint(self.n_actions)
        return action, hidden_out
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;If no hidden state is provided to the forward pass, then we omit specifying the hidden argument of the LSTM layer, which equates to setting the hidden state to a zero tensor.&lt;/p&gt;
&lt;h2 id=&#34;sequential-experience-buffer&#34;&gt;Sequential Experience Buffer&lt;/h2&gt;
&lt;p&gt;Besides the model, it is common for DQN type algorithms to use an experience buffer which stores past transitions and provides batches of random transitions for learning the network parameters. Since we are dealing with a recurrent network, we would like to sample batches of sequences at random, on which we then unroll the LSTM to provide a batch of Q-value predictions. While there are various ways to implement such a buffer, we use a simple list that (in order) stores all visited transition tuples $(a_{t-1}, o_t, r_t, a_t, o_{t+1}, d_{t+1})$, where $r_t$ is the reward gained after executing action $a_t$ in observation $o_t$ and $d_{t+1}$ is true if $o_{t+1}$ is a terminal state or otherwise false.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;class ExpBuffer():
    def __init__(self, max_storage, sample_length):
        self.max_storage = max_storage
        self.sample_length = sample_length
        self.counter = -1
        self.filled = -1
        self.storage = [0 for i in range(max_storage)]

    def write_tuple(self, aoarod):
      if self.counter &amp;lt; self.max_storage-1:
          self.counter +=1
          if self.filled &amp;lt; self.max_storage-1:
            self.filled += 1
      else:
          self.counter = 0
      self.storage[self.counter] = aoarod

    def sample(self, batch_size):
        #Returns tensors of sizes (batch_size, seq_len, *) where * depends on action/observation/return/done
        seq_len = self.sample_length
        last_actions = []
        last_observations = []
        actions = []
        rewards = []
        observations = []
        dones = []

        for i in range(batch_size):
            if self.filled - seq_len &amp;lt; 0 :
                raise Exception(&amp;quot;Reduce seq_len or increase exploration at start.&amp;quot;)
            start_idx = np.random.randint(self.filled-seq_len)
            last_act, last_obs, act, rew, obs, done = zip(*self.storage[start_idx:start_idx+seq_len])
            last_actions.append(list(last_act))
            last_observations.append(last_obs)
            actions.append(list(act))
            rewards.append(list(rew))
            observations.append(list(obs))
            dones.append(list(done))

        return torch.tensor(last_actions).cuda(), torch.tensor(last_observations, dtype = torch.float32).cuda(), torch.tensor(actions).cuda(), torch.tensor(rewards).float().cuda() , torch.tensor(observations, dtype = torch.float32).cuda(), torch.tensor(dones).cuda()
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Note that with this implementation, we allow the buffer to return sequences that stretch over multiple episodes. If you want to try a buffer that restricts to sampling sequences from a single episode, check out the full source code which includes such an alternative buffer.&lt;/p&gt;
&lt;h2 id=&#34;training-the-network&#34;&gt;Training the Network&lt;/h2&gt;
&lt;p&gt;In the main loop, we sample batches of sequences (with a fixed length) from the experience buffer after each time an action gets executed. To stabilize training, we employ a standard trick of splitting the task of predicting and evaluating Q values into two separate networks, a target network with parameters $\theta^-$ and the main network with parameters $\theta$. For each batch of transition sequences $(a_{t-1}, o_t, r_t, a_t, o_{t+1}, d_{t+1})$ we then compute the $Q$ value with respect to the target network&lt;/p&gt;
&lt;p&gt;$$y_j = r_j + \gamma \cdot (1-\text{is_done}(o_{t+1}))\cdot
\max_{a}Q(h_j, a_j, o_{j+1}, a; \theta^-),$$&lt;/p&gt;
&lt;p&gt;where $\text{is_done}(o_{t+1}) = 1$ if $o_{t+1}$ is a terminal state and $0$ otherwise. Subsequently, we update the parameters of our main $Q$ network via the gradient of the quadratic loss&lt;/p&gt;
&lt;p&gt;$$ \nabla_\theta (y_j - Q(h_{j-1}, a_{j-1}, o_j, a_j; \theta))^2.
$$&lt;/p&gt;
&lt;p&gt;In the implementation, we made use of the Adam optimizer for taking these gradient updates. At the end of each episode we update the target network parameters $\theta^- := \theta$. Furthermore, we give the agent some time to explore and fill the experience buffer before updating the networks.&lt;/p&gt;
&lt;p&gt;Besides that, we employ an $\epsilon$-greedy policy with respect to the $Q$ network and an adaptive exploration schedule for decreasing $\epsilon$ over time. To induce partial observability, we also introduce a blinding probability parameter which sets incoming observations to a zero tensor with a specified probability.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;env = gym.make(&#39;MountainCar-v0&#39;)
state_size = env.observation_space.shape[0]
n_actions = env.action_space.n
embedding_size = 8
M_episodes = 2500
replay_buffer_size = 100000
sample_length = 20
replay_buffer = ExpBuffer(replay_buffer_size, sample_length)
batch_size = 64
eps_start = 0.9
eps = eps_start
eps_end = 0.05
eps_decay = 200
gamma = 0.999
learning_rate = 0.005
blind_prob = 0
EXPLORE = 300

adrqn = ADRQN(n_actions, state_size, embedding_size).cuda()
adrqn_target = ADRQN(n_actions, state_size, embedding_size).cuda()
adrqn_target.load_state_dict(adrqn.state_dict())

optimizer = torch.optim.Adam(adrqn.parameters(), lr = learning_rate)

for i_episode in range(M_episodes):
    done = False
    hidden = None
    last_action = 0
    last_observation = env.reset()
    for t in count():
        action, hidden = adrqn.act(torch.tensor(last_observation).float().view(1,1,-1).cuda(), F.one_hot(torch.tensor(last_action), n_actions).view(1,1,-1).float().cuda(), hidden = hidden, epsilon = eps)
        observation, reward, done, info = env.step(action)

        if np.random.rand() &amp;lt; blind_prob:
            #Induce partial observability
            observation = np.zeros_like(observation)

        reward = np.sign(reward)
        replay_buffer.write_tuple((last_action, last_observation, action, reward, observation, done))

        last_action = action
        last_observation = observation

        #Updating Networks
        if i_episode &amp;gt; EXPLORE:
                #Update exploration parameter
                eps = eps_end + (eps_start - eps_end) * math.exp((-1*(i_episode-EXPLORE))/eps_decay)
                #Sample a batch of action/observation/reward sequences
                last_actions, last_observations, actions, rewards, observations, dones = replay_buffer.sample(batch_size)

                #Pass the sequence of last observations and actions through the network
                q_values, _ = adrqn.forward(last_observations, F.one_hot(last_actions, n_actions).float())
                #Get the q_values for the executed actions in the respective observations
                q_values = torch.gather(q_values, -1, actions.unsqueeze(-1)).squeeze(-1)
                #Query the target network for Q value predictions
                predicted_q_values, _ = adrqn_target.forward(observations, F.one_hot(actions, n_actions).float())
                #Compute Q update target
                target_values = rewards + (gamma * (1 - dones.float()) * torch.max(predicted_q_values, dim = -1)[0])
                #Updating network parameters
                optimizer.zero_grad()
                loss = torch.nn.MSELoss()(q_values , target_values.detach())
                loss.backward()
                optimizer.step()

        if done:
            break

    #Update the target network
    adrqn_target.load_state_dict(adrqn.state_dict())
&lt;/code&gt;&lt;/pre&gt;
&lt;h1 id=&#34;results&#34;&gt;Results&lt;/h1&gt;
&lt;p&gt;This section shows some learning results on the MountainCar-v0 and CartPole-v1 tasks of the OpenAI gym. The main point here is to show that the network works and not to discuss computational/sample efficiency. However, the authors of the ADRQN paper [3] state that the ADRQN outperforms other state of the art DQN variants in partially observable environments.&lt;/p&gt;
&lt;h2 id=&#34;cartpole-v1&#34;&gt;CartPole-v1&lt;/h2&gt;
&lt;p&gt;In the cartpole environment, the goal of the agent is learning a policy that manages to balance a pole which is attached to a cart moving along a frictionless track. The observation consists of a $4$-dimensional array, which includes the position and velocity of the cart and the pole respectively.
The rewards directly correspond to the time that the pole can be balanced and provide $+1$ for each time step. OpenAI considers this environment solved when the agent receives an average return of $195$ or higher over $100$ consecutive trials.&lt;/p&gt;
&lt;p&gt;&lt;img src=&#34;cartpole1.gif&#34; alt=&#34;&#34;&gt;&lt;/p&gt;
&lt;p&gt;We test two different scenarios in this environment, one where the agent has full access to the observations and one where the observations drop out with a probability of $\frac{1}{2}$. Although training of our shallow ADRQN version is quite fast, we give the agent $300$ time steps to explore before updating the parameters. Figure 2 and 3 show the undiscounted reward and exploration scheme of the agent in the fully observable and partially observable cases respectively. While in the fully observable case, the agent learns a near optimal policy (reaching the maximum of $500$ time steps) after around 100 episodes of learning (with 300 episodes of stored transitions), learning an optimal policy in the partial observable case takes around twice as many learning iterations. Although the learning process is significantly more noisy in the partially observable case, the agent still manages to learn a policy that somewhat consistently achieves the maximum return.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align:center&#34;&gt;&lt;em&gt;Figure 2: ADRQN learning in Cartpole-v1 (fully observable)&lt;/em&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;Cartpole_ADRQN_MDP.png&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align:center&#34;&gt;&lt;em&gt;Figure 3: ADRQN learning in Cartpole-v1 with 0.5 observation censoring probability&lt;/em&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;Cartpole_ADRQN_halfblind.png&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id=&#34;mountaincar-v0&#34;&gt;MountainCar-v0&lt;/h2&gt;
&lt;p&gt;The mountaincar environment is slightly more challenging due to highly sparse rewards. The agent receives a reward of $-1$ at each timestep and is given $200$ time steps to reach the flag at the top of the mountain. Without any modifications to the algorithm, the $\epsilon$-greedy exploration scheme takes quite a long time to discover where to go with the car. The observations consist of a $2$-dimensional array including position and velocity, whereas the action space is discrete and consists of pushing the car to the left, right or not at all. OpenAI considers this environment solved when the agent gets a reward of at least -110.0 over 100 consecutive trials.&lt;/p&gt;
&lt;p&gt;&lt;img src=&#34;mountaincar.gif&#34; alt=&#34;&#34;&gt;&lt;/p&gt;
&lt;p&gt;Figures 4 and 5 show the learning progress in the fully observable and partially observable cases respectively. We did not train the agent until the environment was solved. However, we can see that (after an initial exploration phase of 300 episodes) the agent learns to consistently reach a reward of around $-110$ and $-125$ in the fully observable and partially observable cases respectively.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align:center&#34;&gt;&lt;em&gt;Figure 4: ADRQN learning in MountainCar-v0 (fully observable)&lt;/em&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;mountaincar00.PNG&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align:center&#34;&gt;&lt;em&gt;Figure 5: ADRQN learning in MountainCar with 0.5 observation censoring probability&lt;/em&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;mountaincar05_1.PNG&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id=&#34;references&#34;&gt;References&lt;/h2&gt;
&lt;p&gt;[1] Volodymyr Mnih, Kavukcuoglu Koray, Silver David, Rusu Andrei A, Veness Joel, Belle-mare Marc G, Graves Alex, Riedmiller Martin, Fidje-land Andreas K, Ostrovski Georg, et al. &lt;em&gt;Human-level control through deep reinforcement learning&lt;/em&gt;. In: Nature, 518 (7540): 529–533, 2015.&lt;/p&gt;
&lt;p&gt;[2] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot,M. Azar, and D. Silver. &lt;em&gt;Rainbow: Combining improvements in deep reinforcement learning.&lt;/em&gt; In: Thirty-Second AAAI Conference on Artificial Intelligence, 2018&lt;/p&gt;
&lt;p&gt;[3] Zhu, P., Li, X., Poupart, P., and Miao, G. &lt;em&gt;On improving deep reinforcementlearning for pomdps.&lt;/em&gt; arXiv preprint arXiv:1804.06309 (2018).&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Making Sense of Particle Filtering for POMDPs</title>
      <link>https://mlpeschl.com/post/particle_filtering_1/</link>
      <pubDate>Tue, 20 Oct 2020 20:02:56 +0200</pubDate>
      <guid>https://mlpeschl.com/post/particle_filtering_1/</guid>
      <description>&lt;h2 id=&#34;before-you-read-source-code&#34;&gt;Before you read (source code)&lt;/h2&gt;
&lt;p&gt;An open source Jupyter notebook including source code for this blog post is available. Click the link below to open an interactive version of the source code in your browser.
&lt;a href=&#34;https://mybinder.org/v2/gh/mlpeschl/stat-notebooks/master?filepath=POMDP_Particle_Filters%2Fparticle%20filtering.ipynb&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;&lt;img src=&#34;https://mybinder.org/badge_logo.svg&#34; alt=&#34;Binder&#34;&gt;&lt;/a&gt;&lt;/p&gt;
&lt;h2 id=&#34;introduction&#34;&gt;Introduction&lt;/h2&gt;
&lt;p&gt;In the field of reinforcement learning, partially observable Markov decision processes (POMDPs) form a mathematical model for the underlying decision process of an agent that is unable to know the exact state it is currently in. While incorporating uncertainty about the state of the environment complicates matters mathematically, it allows for modeling more general problems that arise in various real world problems. Take, for example, a robot which has to navigate through its environment by using sensor data. Such data is inevitably noisy, which gives rise to an inherent uncertainty about the exact state that the robot is currently in.
Even in discrete environments, like Atari games (which have become a standard task in deep reinforcement learning research), a single frame sometimes fails to convey the exact state of the environment, since variables depending on the rate of change (e.g. velocity of an enemy) can only be inferred through multiple consecutive observations.&lt;/p&gt;
&lt;p&gt;In this blog post we will explore a popular method for tracking the unobservable state in a POMDP, called particle filtering, with a simple example of recovering noisy observations in a discrete time Markov process.&lt;/p&gt;
&lt;h2 id=&#34;pomdps-and-beliefs&#34;&gt;POMDPs and Beliefs&lt;/h2&gt;
&lt;p&gt;Formally, a POMDP consists of a tuple $\langle S, A, \Omega, T, O, R \rangle$, where $S$ is the state space, $A$ is the set of actions and $\Omega$ denotes the set of possible observations. A transition function $T(s&amp;rsquo;|s,a)$ gives the probability of transitioning to state $s&amp;rsquo;$ when in state $s \in S$ and executing action $a \in A$, whereas an observation function $O(o | s&amp;rsquo;,a)$ gives the probability of observing observation $o \in \Omega$ after action $a$ has taken the agent to state $s&amp;rsquo;$. Finally, $R(s,a)$ gives the reward when taking action $a$ in state $s$. The goal (as in the fully observable Markov decision process), is usually to maximize an expected long-term reward of the form
$$\mathbb{E}\left[\sum_{t=0}^h \gamma^t R_t\right],$$
where $0 \leq \gamma \leq 1$ is a discount factor, $h$ is the planning horizon and $R_t$ is the reward obtained at timestep $t$.&lt;/p&gt;
&lt;p&gt;To tackle the problem of partial observability, one needs to incorporate some kind of memory into the agents decision process. While tracking a history of past events is one possibility, another approach is to summarize this into a sufficient statistic, called the belief state $b(s)$. The belief state is a probability distribution over all possible states and models the agents believes about the current true state of the environment. Although the exact computation of $b$ with respect to an initial prior belief $b_0$ when observing $o$ after action $a$ is possible through Bayesian updating, this requires knowledge of the POMDP dynamics:&lt;/p&gt;
&lt;p&gt;\begin{equation}b_{a,o}(s&amp;rsquo;) = \frac{O(o|s&amp;rsquo;,a)\sum_{s\in S} T(s&amp;rsquo;|s,a)b(s)}{\sum_{s&amp;rsquo; \in S}O(o|s&amp;rsquo;,a)\sum_{s\in S}T(s&amp;rsquo;|s,a)b(s)}.\end{equation}&lt;/p&gt;
&lt;p&gt;Even if the model is known, updating can turn out to be cumbersome due to computational complexity that quadratically grows with the size of the state space $|S|$. While there are various ways to work around this issue, the theory of particle filtering turns out to be  effective and compatible with the POMDP framework for tracking belief states in an approximate manner.&lt;/p&gt;
&lt;h2 id=&#34;particle-filtering-with-sequential-importance-sampling&#34;&gt;Particle Filtering with Sequential Importance Sampling&lt;/h2&gt;
&lt;p&gt;Particle filtering, also known as Sequential Monte Carlo, is a widely used technique for estimating a noisy and/or unobserved signal in a dynamical system. To be precise, we assume that we have a discrete time Markov process ${X_t}$ ($t \in \mathbb{N}$) governed by dynamics $p(x_t | x_{t-1})$  with initial state distribution $p(x_0)$ (for simplicity, we for now drop the possibility of taking actions in the following example). Furthermore, we assume a model for the observations $Y_t$, which is given by $p(y_t | x_t)$, where the values of $Y_t$ are conditionally independent given the values of $X_t$. The general goal of particle filtering is to then approximate the posterior distribution $p(x_t | y_0, y_1, \dots, y_t)$, i.e. to obtain a probability distribution for the true signal at time $t$, given all the information (observations) up to time $t$.&lt;/p&gt;
&lt;p&gt;While this problem has been studied in various fields for a long time and the mathematical theory behind its possible solutions are rich, we will focus on a simple, but effective Monte Carlo algorithm based on importance sampling. Let&amp;rsquo;s assume we are given a noisy observation $(y_0, y_1, \dots, y_{T-1})$ of the signal $(x_0, x_1, \dots, x_{T-1})$ with length $T$. To recover the signal, consider the following procedure (assuming knowledge of $p(x_t | x_{t-1})$ and $p(y_t | x_t)$ ):&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;Initialize $N$ sequences $x_i^n$ $(i = 0, \dots , T-1)$ with initial values $x_0^n$ drawn according to $p(x_0)$.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Initialize an array of weights $\mathbf{w_0} \in \mathbb{R}^N$ with ones.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;For $t = {1,\dots, T-1}$:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Predict the next value for each sequence by sampling from the model:
$$x_t^i \sim p(x| x_{t-1}^i)$$&lt;/li&gt;
&lt;li&gt;For each sequence $x_i^n$ ($i \leq t$), evaluate how likely the observation $y_t$ would have been, assuming that $x_i^n$ was the true signal:
$$w_t^n := w_{t-1}^n \cdot p(y_t | x_{t-1}^n)$$&lt;/li&gt;
&lt;li&gt;Normalize the weights:
$$w_t^n = \frac{w_t^n}{\sum_{i=1}^n w_i^n}$$&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;By following this simple algorithm, we obtain $N$ sequences (also called &lt;em&gt;particles&lt;/em&gt;) $x_{0:T-1}^n \in \mathbb{R}^{T}$ with corresponding likelihoods $w_{T-1}^n$ that indicate, how likely it was for sequence $x_{0:T-1}^n$ to be the true sequence that led to the observation vector $(y_0, y_1, \dots, y_{T-1})$. Intuitively, if $N$ is chosen large enough, we would expect to find a sequence $x_{0:T-1}^n$ with a high enough weight $w^n$, such that it is a good candidate solution $x_{0:T-1}^n \approx (x_1, x_2, \dots, x_{T-1})$. However, there is a small technical issue with the algorithm above, called &lt;em&gt;particle degeneracy&lt;/em&gt;. Due to randomness, most of the particles $x_{0:t}^n$ will receive a very low weight $w_t^n$ for growing $t$, since propagation of deviations in the generation of $x_t^n$ from the true signal can grow arbitrarily large. Therefore, it can happen that almost all weights will be close to $0$ after the normalization step, while one weight of a particle that most closely resembles the true signal will get all the credit. To avoid this, we can completely resample $N$ particles from the sequences $x_{0:t}^n$ from time to time according to their weights $w_t^n$ and reset the weights $w_t^n = \frac{1}{N}$. Over time, this will discard highly unlikely sequences and only keep promising ones.&lt;/p&gt;
&lt;p&gt;To see this algorithm in action, consider the following example: Let $X_t | X_{t-1} = x_{t-1} \sim \mathcal{N}(x_{t-1} +1, 16)$ be a discrete time random walk with constant upwards drift. Furthermore, let $Y_t | X_t = x_t \sim \Gamma(8,\sqrt{|x_t|})$ follow a Gamma distribution with constant shape parameter.&lt;/p&gt;
&lt;p&gt;We start off our simulation experiment by generating a sequence $(x_0, \dots, x_{49})$ of length $T = 50$ and sample an observation signal $(y_0, \dots, y_{49})$ accordingly. Then, we run the algorithm above with particle resampling using a total of $N = 2000$ particles. Figure 1 shows the resulting posterior mean as well as a level 0.99 confidence band obtained from the empirical particle distribution.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align:center&#34;&gt;&lt;em&gt;Figure 1: Simulation Study&lt;/em&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;SIS_example.png&#34; alt=&#34;space-1.jpg&#34;&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Despite the high noise in the observation, our simple algorithm manages to recover the true signal reasonably well in the mean. Additionally, since we approximate a complete posterior distribution we can make use of the uncertainty quantification that comes with it. This can be particularly useful in reinforcement learning, since the study of decision making inherently involves quantifying ones certainty in particular events.&lt;/p&gt;
&lt;h2 id=&#34;applications-to-pomdps&#34;&gt;Applications to POMDPs&lt;/h2&gt;
&lt;p&gt;While in the example above we excluded the possibility of taking actions in the Markov process $X_t$, particle filtering can be easily extended to the POMDP setting. In fact, sequential importance sampling is a useful tool for approximating the belief state $b(s)$, which in itself can be interpreted as the posterior distribution of the true state given past observations (including actions). For instance, if we had the POMDP dynamics available, we could apply the above algorithm in a slightly modified way:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;Instead of sampling $x_t^i \sim p(x| x_{t-1}^i)$ we predict a next candidate state given action $a$ and current candidate state $s$:
$$s&amp;rsquo; \sim T(\cdot | s,a)$$&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;After taking action $a$ and observing $o$, we then update our weights $w$ according to the observation likelihood given $a$ and $s&amp;rsquo;$:
$$w&amp;rsquo; = O(o| s&amp;rsquo;, a)w$$&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This way, we could effectively keep track of likely state sequences $(s_0, s_1, \dots,)$ without ever needing to explicitly observe them.&lt;/p&gt;
&lt;p&gt;It is important to keep in mind that the sequential importance sampling algorithm in the form it is presented above requires explicit knowledge of the state transition and observation model. In case these are available, particle filtering is a useful tool for approximating the exact posterior update $b_{a,o}(s&amp;rsquo;)$. For example, in Partially Observable Monte-Carlo Planning, an extension of Monte-Carlo Tree Search for POMDPs [1], particle filtering is used to track the belief state while unrolling episodes through a generative model of the POMDP dynamics. When a model is not available, recent deep learning based approaches have been proposed, like deep variational reniforcement learning for POMDPs [2], which learns a deep generative model of the POMDP transition dynamics by sampling likely candidate state sequences through particle filtering. Similarly, particle filters have been used for keeping belief states in Bayesian reinforcement learning for large POMDPs which represent a distribution over possible POMDP models [3].&lt;/p&gt;
&lt;p&gt;Alright, that&amp;rsquo;s it for now. In possible sequels to this post, we will dive into more detail of applying particle filtering to POMDP control problems by using methods like the Bayes-Adaptive and deep variational frameworks.&lt;/p&gt;
&lt;h2 id=&#34;references&#34;&gt;References&lt;/h2&gt;
&lt;p&gt;[1] Silver,D. and Veness, J. Monte-Carlo planning in large POMDPs. In &lt;em&gt;Advances in Neural Information Processing Systems&lt;/em&gt;, pages 2164-2172, 2010.&lt;/p&gt;
&lt;p&gt;[2] Igl, M., Zintgraf, L., Le, T. A., Wood, F., and Whiteson, S. &lt;em&gt;Deep variational reinforcement learning for pomdps&lt;/em&gt;. arXiv:1806.02426, 2018.&lt;/p&gt;
&lt;p&gt;[3] Katt, S., Oliehoek, F. A., and Amato, C. Learning in pomdps with monte carlo tree search.  In &lt;em&gt;International Conference on Machine Learning&lt;/em&gt;, pp. 1819–1827, 2017.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Deep Image Prior: Independently Reproduced in a few lines of PyTorch.</title>
      <link>https://mlpeschl.com/post/deepimageprior/</link>
      <pubDate>Sat, 17 Oct 2020 18:36:42 +0200</pubDate>
      <guid>https://mlpeschl.com/post/deepimageprior/</guid>
      <description>&lt;h2 id=&#34;authors&#34;&gt;Authors&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;Markus Peschl&lt;/li&gt;
&lt;li&gt;Cecilia Casolo&lt;/li&gt;
&lt;li&gt;Mats Van Tongeren&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&#34;remark&#34;&gt;Remark&lt;/h2&gt;
&lt;p&gt;This blog has been submitted to &lt;a href=&#34;https://reproducedpapers.org&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt; https://reproducedpapers.org&lt;/a&gt;, which features a collection of reproducibility attempts of papers in the field of Deep Learning by various people. If you are interested, feel free to check it out!&lt;/p&gt;
&lt;h2 id=&#34;introduction&#34;&gt;Introduction&lt;/h2&gt;
&lt;p&gt;&lt;img src=&#34;imagepriortitle.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;
&lt;p&gt;The &lt;a href=&#34;https://en.wikipedia.org/wiki/Deep_Image_Prior&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Deep Image Prior&lt;/a&gt; is a convolutional neural network (CNN), designed to solve
various inverse problems in computer vision, such as denoising, inpainting and super-resolution. Unlike other CNNs designed for these kinds of tasks, the Deep Image Prior does not need any training data, besides the corrupted input image itself. Generally speaking, the network is trained to reconstruct the corrupted image from noise. However, since the architecture of the Deep Image Prior fits structured (natural) data a lot faster than random noise, one can observe that in many applications recovering the noiseless image can be done by stopping the training process after a predefined number of iterations. The authors of the paper (&lt;a href=&#34;https://arxiv.org/abs/1711.10925&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Ulyanov et al.&lt;/a&gt;) explain this as follows:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;[&amp;hellip;] although in the limit the parametrization can fit un-
structured noise, it does so very reluctantly. In other words,
the parametrization offers high impedance to noise and low
impedance to signal.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;This page features an independent reproduction of some of the results published in the original paper, without making use of the already available open-source code. We will describe the design steps that were necessary to get the architecture running and we will explain which ambiguities had to be resolved when interpreting the text material provided by the authors.&lt;/p&gt;
&lt;h2 id=&#34;the-network-architecture&#34;&gt;The Network Architecture&lt;/h2&gt;
&lt;p&gt;The network architecture consists of several convolutional downsampling blocks followed by convolutional upsampling blocks. Furthermore, after each downsampling block a skip connection is added, which links to a corresponding upsampling layer. For a small visualization, see the figure below (taken from the authors &lt;a href=&#34;https://box.skoltech.ru/index.php/s/ib52BOoV58ztuPM#pdfviewer&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Supplementary Materials&lt;/a&gt;).
&lt;img src=&#34;Data/Visualization/architecture.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;
&lt;p&gt;We reimplemented this in Python 3, making use of the PyTorch framework. To do so, we separately defined one Module Class for each of these blocks. For the full source-code, you can download and experiment with the Jupyter Notebooks attached in the Notebooks directory of this Git repository.&lt;/p&gt;
&lt;p&gt;First of all, the downsampling blocks work as follows:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;class Model_Down(nn.Module):
    &amp;quot;&amp;quot;&amp;quot;
    Convolutional (Downsampling) Blocks.

    nd = Number of Filters
    kd = Kernel size

    &amp;quot;&amp;quot;&amp;quot;
    def __init__(self,in_channels, nd = 128, kd = 3, padding = 1, stride = 2):
        super(Model_Down,self).__init__()
        self.padder = nn.ReflectionPad2d(padding)
        self.conv1 = nn.Conv2d(in_channels = in_channels, out_channels = nd, kernel_size = kd, stride = stride)
        self.bn1 = nn.BatchNorm2d(nd)

        self.conv2 = nn.Conv2d(in_channels = nd, out_channels = nd, kernel_size = kd, stride = 1)
        self.bn2 = nn.BatchNorm2d(nd)

        self.relu = nn.LeakyReLU()

    def forward(self, x):
        x = self.padder(x)
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.padder(x)
        x = self.conv2(x)
        x = self.bn2(x)
        x = self.relu(x)
        return x

&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;As we can see, based on the papers default architecture, we default set the number of filters to 128 with a kernel size of 3. We incorporated the downsampling in the convolutional layer with by using the implemented strides of 2. This is also what the authors describe in the supplementary materials.&lt;/p&gt;
&lt;p&gt;Furthermore, we defined the upsampling blocks work as follows:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;class Model_Up(nn.Module):
    &amp;quot;&amp;quot;&amp;quot;
    Convolutional (Upsampling) Blocks.

    nu = Number of Filters
    ku = Kernel size

    &amp;quot;&amp;quot;&amp;quot;
    def __init__(self, in_channels = 132, nu = 128, ku = 3):
        super(Model_Up, self).__init__()
        self.bn1 = nn.BatchNorm2d(in_channels)
        self.padder = nn.ReflectionPad2d(1)
        self.conv1 = nn.Conv2d(in_channels = in_channels, out_channels = nu, kernel_size = ku, stride = 1, padding = 0)
        self.bn2 = nn.BatchNorm2d(nu)

        self.conv2 =  nn.Conv2d(in_channels = nu, out_channels = nu, kernel_size = 1, stride = 1, padding = 0)
        self.bn3 = nn.BatchNorm2d(nu)

        self.relu = nn.LeakyReLU()

    def forward(self,x):
        x = self.bn1(x)
        x = self.padder(x)
        x = self.conv1(x)
        x = self.bn2(x)
        x = self.relu(x)
        x = self.conv2(x)
        x = self.bn3(x)
        x = self.relu(x)
        x = F.interpolate(x, scale_factor = 2, mode = &#39;bicubic&#39;)
        return x

&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The input channels have to match the number of output channels from the last upsampling block plus the number of output channels of the skip connections (since we concatenate the skip connections with the upsampling blocks, for an explanation see the ambiguity section). In the default architecture, this will always be 132, but can of course be manually defined differently, when needed. Initially, we implemented the upsampling blocks using the bilinear upsampling. Nevertheless, we realized that for a high number of iterations (around 8000) the network sturcture performs better with bicubic upsampling.
The upscaling factor of 2 is chosen to match the stride 2 downsampling. This way, we obtain an output image of the same width and height as the input noise.&lt;/p&gt;
&lt;p&gt;Meanwhile, we defined the skip blocks in the following way:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;class Model_Skip(nn.Module):
    &amp;quot;&amp;quot;&amp;quot;
    Skip Connections
    ns = Number of filters
    ks = Kernel size
    &amp;quot;&amp;quot;&amp;quot;
    def __init__(self,in_channels = 128 ,stride = 1 , ns = 4, ks = 1, padding = 0):
        super(Model_Skip, self).__init__()
        self.conv = nn.Conv2d(in_channels = in_channels, out_channels = ns, kernel_size = ks, stride = stride, padding = padding)
        self.bn = nn.BatchNorm2d(ns)
        self.relu = nn.LeakyReLU()

    def forward(self,x):
        x = self.conv(x)
        x = self.bn(x)
        x = self.relu(x)
        return x
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Then, the final network can be easily defined, by concatenating the modular building blocks together in the same way as described in the figure above.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;class Model(nn.Module):
    def __init__(self, length = 5, in_channels = 32, nu = [128,128,128,128,128] , nd =
                    [128,128,128,128,128], ns = [4,4,4,4,4], ku = [3,3,3,3,3], kd = [3,3,3,3,3], ks = [1,1,1,1,1]):
        super(Model,self).__init__()
        assert length == len(nu), &#39;Hyperparameters do not match network depth.&#39;

        self.length = length

        self.downs = nn.ModuleList([Model_Down(in_channels = nd[i-1], nd = nd[i], kd = kd[i]) if i != 0 else
                                        Model_Down(in_channels = in_channels, nd = nd[i], kd = kd[i]) for i in range(self.length)])

        self.skips = nn.ModuleList([Model_Skip(in_channels = nd[i], ns = ns[i], ks = ks[i]) for i in range(self.length)])

        self.ups = nn.ModuleList([Model_Up(in_channels = ns[i]+nu[i+1], nu = nu[i], ku = ku[i]) if i != self.length-1 else
                                        Model_Up(in_channels = ns[i], nu = nu[i], ku = ku[i]) for i in range(self.length-1,-1,-1)]) #Elements ordered backwards

        self.conv_out = nn.Conv2d(nu[0],3,1,padding = 0)
        self.sigm = nn.Sigmoid()

    def forward(self,x):
        s = [] #Skip Activations

        #Downpass
        for i in range(self.length):
            x = self.downs[i].forward(x)
            s.append(self.skips[i].forward(x))

        #Uppass
        for i in range(self.length):
            if (i == 0):
                x = self.ups[i].forward(s[-1])
            else:
                x = self.ups[i].forward(torch.cat([x,s[self.length-1-i]],axis = 1))

        x = self.sigm(self.conv_out(x)) #Squash to RGB ([0,1]) format
        return x

&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id=&#34;preprocessing-inpainting&#34;&gt;Preprocessing (Inpainting)&lt;/h2&gt;
&lt;p&gt;In order to analyze the image correctly and succeed in the inpainting task, we needed to preprocess the original image and the mask. Indeed, our code takes as inputs the original image and the mask that needs to be placed on it. First of all, the images have been converted to numerical arrays and their sizes have been adjusted in such a way that they were compatible. Furthermore, the two numpy arrays have been converted to PyTorch tensors. The masked image will then be given by the normalized multiplication of the torch tensors corresponding to the masked image and the original one.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;im = Image.open(&#39;kate.png&#39;)
maskim = Image.open(&#39;kate_mask.png&#39;)
maskim = maskim.convert(&#39;1&#39;)

im_np = np.array(im)
mask_np = np.array(maskim,dtype = float)
mask_np = (np.repeat(mask_np[:,:,np.newaxis], 3, axis = 2)/255)

fig, ax = plt.subplots(figsize=(10,10))
plt.imshow(im_np*mask_np)

mask_tensor = torch.from_numpy(mask_np).permute(2,0,1)
im_tensor = torch.from_numpy(im_np).permute(2,0,1)

im_masked_tensor = ((mask_tensor*im_tensor).unsqueeze(0)/255).cuda()
mask_tensor = mask_tensor.unsqueeze(0).cuda()
&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id=&#34;preprocessing-restoration&#34;&gt;Preprocessing (Restoration)&lt;/h2&gt;
&lt;p&gt;The preprocessing steps used in the restoration task are very similar to the ones used in the inpainting task. Since the restoration task is to recover an image, where a certain percentage (e.g. 50%) of the original image pixels have been dropped, we simply initialized a sparse array with half of the entries being equal to 0 and used that as our mask in the same way as in the inpainting task.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;from scipy.sparse import random
mask_np = random(512,512,0.5,dtype = bool).A.astype(float)
&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id=&#34;training-the-model&#34;&gt;Training the Model&lt;/h2&gt;
&lt;p&gt;The main loop follows the standard Pytorch conventions. The only difference here is that the input to the loss function must be carefully chosen to prevent cheating. We are only allowed to compute the loss on the masked pixels, which ensures that we do not need the original image to begin with. To ensure this, we just multiply the mask with the original image before we pass it to the loss function. The input is, as suggested by the authors, a uniformly generated tensor with mean 0.05.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;#Initialize model params
z = (0.1) * torch.rand((1,32,512,512), device = &amp;quot;cuda&amp;quot;)


#Initialize the Model
net = Model() #Using standard architecture, no hyperparams specified
optimizer = torch.optim.Adam(net.parameters(),lr = 0.01)

use_gpu = torch.cuda.is_available()

if use_gpu:
    net = net.cuda()


#Main Training Loop
for epoch in range(2000):
    optimizer.zero_grad()
    output = net.forward(z)
    loss = F.mse_loss(output*mask_tensor, im_masked_tensor)

    if (epoch % 250 == 0):
      print(&#39;EPOCH: &#39; + str(epoch))
      print(&#39;LOSS: &#39; + str(loss.item()), end =&#39;\n\n&#39;)
      plt.imshow(output.cpu().view(3,512,512).permute(1,2,0).detach().numpy())
      plt.show()

    loss.backward()
    optimizer.step()

    z = z + (1/(30))*torch.randn_like(z) #Regularization

plt.imshow(output.cpu().view(3,512,512).permute(1,2,0).detach().numpy())

&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id=&#34;ambiguities&#34;&gt;Ambiguities&lt;/h2&gt;
&lt;p&gt;Here we will list some of the things that were not completely specified in the supplementary materials. For these implementation details, we either tested out multiple implementations and then took the best performing one or we simply used the implementation that seemed most reasonable to us. Some of these ambiguities are the following:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;For the skip-layers connections, we decided to use  concatenative connections. Initially, we considered using the additive skip connections. However, this would not be compatible with their specified hyperparameters, since the number of output channels on the skip connections would not match the number of input channels in the upsampling modules. We nevertheless tried out both versions and came to the conclusion that the flexibility of freely choosing the amount of channels on the skip connections helps with preventing overfitting to the noise. Since this can only be achieved with concatenative skip connections, we chose to stick to them in the end.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;The input and its perturbations: The authors mention that they perturb the input at each iteration by small noise. However, this can be achieved in two different ways:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Initializing the input tensor before starting training and then in every iteration taking the initial tensor plus additive noise.&lt;/li&gt;
&lt;li&gt;Initializing the input tensor before starting training and then in every iteration taking the previous input tensor plus additive noise.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;We tried both versions, and both of them do work in their own way. However, despite our expectations being the other way round, the first version seems to be regularizing too much, whereas the second version does achieve better looking results, while still preventing overfitting to noise sufficiently.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Upsampling factor and Downsampling strides: It was not completely clear which strides were used in the downsampling process. Obviously, any stride is possible, as long as the dimensions of the image are large enough. However, this does change the performance of the network. Since for their standard architecture with a kernel size of 3 in the downsampling convolutional blocks and a stride of 2 in the first convolutional layer corresponds to exactly halving the image size for a 512x512 input, we went with that. This also makes the implementation a little bit less tedious, since one does not have to explicitly calculate the upscaling factor anymore.&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h1 id=&#34;inpainting-results&#34;&gt;Inpainting Results&lt;/h1&gt;
&lt;p&gt;We reconstructed the original inpainting task on an image of Kate (at least that is what the authors called the image file so we&amp;rsquo;ll call her Kate as well). We seem to be getting comparable results, although our architecture seems to benefit from a few more iterations. Furthermore, we also bombarded Kate with large holes to see how much the Deep Image Prior can reconstruct. Obviously this is an almost impossible task, but the network still seems to be able to recover (quite remarkably) some traits of the original image.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Corrupted&lt;/th&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Deep Image Prior&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;Data/Results/kate_masked.png&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;Data/Results/kateout.png&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;Data/Results/kategrade1.jpeg&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;Data/Results/kategrade1_result.jpeg&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;Data/Results/kategrade2.jpeg&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;Data/Results/kategrade2_result.jpeg&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;Data/Results/kategrade3.jpeg&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;Data/Results/kategrade3_result.jpeg&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h1 id=&#34;restoration-results&#34;&gt;Restoration Results&lt;/h1&gt;
&lt;p&gt;The authors of the paper test the restoration task on a variety of images, which can be found in the interactive display of the &lt;a href=&#34;https://dmitryulyanov.github.io/deep_image_prior&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Deep Image Prior Page&lt;/a&gt;. A selection of them can be seen here:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Barbara&lt;/th&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Man&lt;/th&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Hill&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;https://dmitryulyanov.github.io/assets/deep-image-prior/Reconstruction/Gray/50p/barbara_GT.png&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;https://dmitryulyanov.github.io/assets/deep-image-prior/Reconstruction/Gray/50p/man_GT.png&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;https://dmitryulyanov.github.io/assets/deep-image-prior/Reconstruction/Gray/50p/hill_GT.png&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Boat&lt;/th&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Couple&lt;/th&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Lena&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;https://dmitryulyanov.github.io/assets/deep-image-prior/Reconstruction/Gray/50p/boat_GT.png&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;https://dmitryulyanov.github.io/assets/deep-image-prior/Reconstruction/Gray/50p/couple_GT.png&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;img src=&#34;https://dmitryulyanov.github.io/assets/deep-image-prior/Reconstruction/Gray/50p/Lena512_GT.png&#34; alt=&#34;&#34;&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Regarding the PSNR scores of the respective images, we seem to be achieving comparable, and sometimes even higher scores than those mentioned in the paper. However, especially on the Barbara picture, we cannot seem to recover all the details just as well. This might be due to the architecture overfitting to the noise. Stopping the training process earlier than suggested in the paper does help with the picture quality and does not change the PSNR significantly though.&lt;/p&gt;
&lt;p&gt;The following table compares our PSNR scores with those reported in Table 1 of the original paper:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Architecture&lt;/th&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Barbara&lt;/th&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Man&lt;/th&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Lena&lt;/th&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Boat&lt;/th&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Hill&lt;/th&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Couple&lt;/th&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Montage&lt;/th&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Cameraman&lt;/th&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Peppers&lt;/th&gt;
&lt;th style=&#34;text-align:center&#34;&gt;House&lt;/th&gt;
&lt;th style=&#34;text-align:center&#34;&gt;Fingerprint&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align:center&#34;&gt;Ours&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;strong&gt;32.51&lt;/strong&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;strong&gt;32.23&lt;/strong&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;strong&gt;35.07&lt;/strong&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;32.93&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;strong&gt;33.03&lt;/strong&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;32.390&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;strong&gt;35.49&lt;/strong&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;strong&gt;33.31&lt;/strong&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;strong&gt;33.71&lt;/strong&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;34.73&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;31.93&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&#34;text-align:center&#34;&gt;Theirs&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;32.22&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;32.20&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;36.16&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;strong&gt;33.06&lt;/strong&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;32.77&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;strong&gt;32.52&lt;/strong&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;34.54&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;29.80&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;33.05&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;strong&gt;39.16&lt;/strong&gt;&lt;/td&gt;
&lt;td style=&#34;text-align:center&#34;&gt;&lt;strong&gt;32.84&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h1 id=&#34;alternative-images&#34;&gt;Alternative images&lt;/h1&gt;
&lt;p&gt;For these images, we did not do any hyperparameter tuning. Instead we used the standard architecture to see what we get. Surprisingly, inpainting seems to work quite well on a variety of images, although the output is always a little bit blurrier than the original image. To alleviate this issue, more training iterations help. We did not use more than 6000 iterations for any of these images.&lt;/p&gt;
&lt;p&gt;&lt;img src=&#34;Data/Results/Altim1.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;
&lt;p&gt;Unfortunately, the inpainting result on the building did not really blow us away. The fine detail of the windows get blurred out significantly, especially in the areas behind the text. Other high resolution images with fine details suffer from the same effect, which could be considered one of the main weak points of the Deep Image Prior.&lt;/p&gt;
&lt;h1 id=&#34;conclusion&#34;&gt;Conclusion&lt;/h1&gt;
&lt;p&gt;The Deep Image Prior paper seemed to be self-contained enough to make it easy to reproduce from scratch. The supplementary materials provided by the authors list almost all needed hyperparameters, which proved to work for not only their selection of pictures, but also on a wider variety of tasks (at the cost of some performance). We did, however, encounter some ambiguities, which made the reproduction more cumbersome. Furthermore, on some restoration tasks we did sometimes not manage to achieve the same visual quality of images as presented in the paper.&lt;/p&gt;
&lt;p&gt;Overall, reproducibility wise, we would give the paper an &lt;strong&gt;8.5/10&lt;/strong&gt; score.&lt;/p&gt;
&lt;p&gt;(Please note, that this score is merely subjective and could vary heavily depending on someone&amp;rsquo;s background with the subject matter.)&lt;/p&gt;
</description>
    </item>
    
  </channel>
</rss>