Skip to content

Commit

Permalink
Update vision_encoder_decoder_blog.md
Browse files Browse the repository at this point in the history
  • Loading branch information
ydshieh authored Feb 6, 2022
1 parent fbfb225 commit 489b325
Showing 1 changed file with 0 additions and 2 deletions.
2 changes: 0 additions & 2 deletions vision_encoder_decoder_blog.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,9 @@ The above models and their variations focus on pretraining either the encoder or

<a id='figure-3'></a>

<center>
| <img src="https://raw.githubusercontent.com/ydshieh/notebooks/master/images/bert-gpt-bart.JPG" alt="drawing" width="400"/> |
|:--:|
| Figure 3: The 3 pretraining paradigms for Transformer models [<sup>[4]</sup>](https://arxiv.org/abs/1810.04805) [<sup>[5]</sup>](https://openai.com/blog/language-unsupervised/) [<sup>[6]</sup>](https://arxiv.org/abs/1910.13461)|
</center>

In 2020, the paper [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://arxiv.org/abs/1907.12461) studied the effectiveness of initializing sequence-to-sequence models with pretrained encoder/decoder checkpoints for sequence generation tasks. It obtained new state-of-the-art results on machine translation, text summarization, etc.

Expand Down

0 comments on commit 489b325

Please sign in to comment.