PyTorch.md

PyTorch

Installation

pip3 install torch torchvision

Important Feature

Broadcast Semantics: the 1D operation can also apply on higher dimension (In short, if a PyTorch operation supports broadcast, then its Tensor arguments can be automatically expanded to be of equal sizes (without making copies of the data).)

• can't broadcast:
  • torch.dot (unlike numpy.dot)
    • issue #138 torch dot function consistent with numpy

Compare with TensorFlow

• PyTorch use dynamic computational graph while TensorFlow use static one.

Dataset and DataLoader

• Kaggle PyTorch Dataset and DataLoader
• PyTorch Data Loading and Processing Tutorial
• How to get mini-batches in pytorch in a clean and efficient way?
• A detailed example of how to generate your data in parallel with PyTorch
• 莫煩Python - Batch Training

Customized

• torch.utils.data
• from torch.utils.data import DataLoader, Dataset

Ready-made

• from torchtext.datasets import text_classification

Embedding

• torch.nn.Embedding Python Example

Tips

Debug

• Use ipdb. (import ipdb; ipdb.set_trace())
• Usually print tensor shape can find the problem easier

Debug in the nn.Sequential

Use a PrintLayer

class PrintLayer(nn.Module):
    def __init__(self):
        super(PrintLayer, self).__init__()

    def forward(self, x):
        # Do your print / debug stuff here
        print(x)
        return x

model = nn.Sequential(
        nn.Linear(1, 5),
        PrintLayer(), # Add Print layer for debug
        nn.ReLU(),
        nn.Linear(5,1),
        nn.LogSigmoid(),
        )

x = Variable(torch.randn(10, 1))
output = model(x)

Resources

• Get Started
• PyTorch Tutorial
  • github
  • Learning PyTorch with Exmaples
    • Tensors
    • Autograd
    • nn Module
  • Deep Learning for NLP with Pytorch
    • Word Embeddings: Encoding Lexical Semantics
• PyTorch Examples
  • Image classification (MNIST) using Convnets
  • Word level Language Modeling using LSTM RNNs
  • Natural Language Inference (SNLI) with GloVe vectors, LSTMs, and torchtext
• Learning PyTorch with Examples

Tutorial

• yunjey/pytorch-tutorial - PyTorch Tutorial for Deep Learning Researchers
• INTERMT/Awesome-PyTorch-Chinese
  • PyTorch Chinese Tutorial
• PyTorchZeroToAll - Quick 3~4 day lecture materials for HKUST students
  • Youtube videos
  • slides
• L1aoXingyu/pytorch-beginner - pytorch tutorial for beginners
• xiaobaoonline/pytorch-in-action - Source code of book PyTorch機器學習從入門到實戰
• pytorch handbook (Chinese)
• sgrvinod/a-PyTorch-Tutorial-to-Sequence-Labeling - a PyTorch Tutorial to Sequence Labeling
• jcjohnson/pytorch-examples: Simple examples to introduce PyTorch
• chenyuntc/pytorch-book: PyTorch tutorials and fun projects including neural talk, neural style, poem writing, anime generation

Article

• Understanding PyTorch with an example: a step-by-step tutorial
• 深度學習新手村：PyTorch入門
  • Basic MNIST Example
  • github

Example/Approach

BiRNN

• Kaggle baseline: pytorch BiLSTM
• BiRNN

Embedding

• PyTorch / Gensim - How to load pre-trained word embeddings

Attention

• Attention - Pytorch and Keras

Related Project

• williamFalcon/pytorch-lightning - The lightweight PyTorch wrapper for ML researchers. Scale your models. Write less boilerplate
  • Pytorch Lightning code guideline for conferences
• Swall0w/torchstat: Model analyzer in PyTorch
• harvardnlp/pytorch-struct: Fast, general, and tested differentiable structured prediction in PyTorch

NLP

• PyTorch-NLP - Supporting Rapid Prototyping with a Toolkit (including Datasets and Neural Network Layers)
• AllenNLP - An open-source NLP research library, built on PyTorch
• torchtext
  • github
• huggingface/pytorch-transformers
  • documentation
• threelittlemonkeys/lstm-crf-pytorch: LSTM-CRF in PyTorch

CV

• pytorch/vision: Datasets, Transforms and Models specific to Computer Vision
• facebookresearch/pytorch3d: PyTorch3D is FAIR's library of reusable components for deep learning with 3D data

Others

• pytorch/vision: Datasets, Transforms and Models specific to Computer Vision

Others

• VSCode PyTorch: error message “torch has no […] member”

Appendix

• torch.bmm vs. torch.mm vs. torch.matmul
• torch.permute vs. torch.transpose vs. torch.view

Why optimizer.zero_grad()?

nn.Module in list can't be auto assign .to(device), put it in nn.ModuleList and use it as normal list will solve the problem