| layout |
|---|
default |
A new version of this course is being offered in Fall 2019
- When: Mondays and Wednesdays from 9:30 to 11:00
- Where: Soda 405
- Instructors: Ion Stoica and Joseph E. Gonzalez
- Announcements: Piazza
- Sign-up to Present: Google Spreadsheet
- Project Ideas: Google Spreadsheet
- If you have reading suggestions please send a pull request to this course website on Github by modifying the index.md file.
The recent success of AI has been in large part due in part to advances in hardware and software systems. These systems have enabled training increasingly complex models on ever larger datasets. In the process, these systems have also simplified model development, enabling the rapid growth in the machine learning community. These new hardware and software systems include a new generation of GPUs and hardware accelerators (e.g., TPU and Nervana), open source frameworks such as Theano, TensorFlow, PyTorch, MXNet, Apache Spark, Clipper, Horovod, and Ray, and a myriad of systems deployed internally at companies just to name a few. At the same time, we are witnessing a flurry of ML/RL applications to improve hardware and system designs, job scheduling, program synthesis, and circuit layouts.
In this course, we will describe the latest trends in systems designs to better support the next generation of AI applications, and applications of AI to optimize the architecture and the performance of systems. The format of this course will be a mix of lectures, seminar-style discussions, and student presentations. Students will be responsible for paper readings, and completing a hands-on project. Readings will be selected from recent conference proceedings and journals. For projects, we will strongly encourage teams that contains both AI and systems students.
{% capture dates %} 1/23/19 1/28/19 1/30/19 2/4/19 2/6/19 2/11/19 2/13/19 2/18/19 2/20/19 2/25/19 2/27/19 3/4/19 3/6/19 3/11/19 3/13/19 3/18/19 3/20/19 3/25/19 3/27/19 4/1/19 4/3/19 4/8/19 4/10/19 4/15/19 4/17/19 4/22/19 4/24/19 4/29/19 5/1/19 5/6/19 5/8/19 5/13/19 {% endcapture %} {% assign dates = dates | split: " " %}
This is a tentative schedule. Specific readings are subject to change as new material is published.
{% include syllabus_entry %}
This lecture will be an overview of the class, requirements, and an introduction to what makes great AI-Systems research.
{% include syllabus_entry %}
Minor Update: We have moved the reading on auto-encoders to Wednesday.
Reading notes for the two required readings below must be submitted using this google form by Monday the 28th at 9:30AM. We have asked that for each reading you answer the following questions:
- What is the problem that is being solved?
- What are the metrics of success?
- What are the key innovations over prior work?
- What are the key results?
- What are some of the limitations and how might this work be improved?
- How might this work have long term impact?
If you find some of the reading confusing and want a more gentle introduction, the optional reading contains some useful explanatory blog posts that may help.
- Reading Quiz due before class.
- Intro Lecture + AlexNet [pdf, pptx]
- Classic Neural Architectures and Inception-v4 [pdf, pptx]
- The AlexNet paper that both help launch deep learning and also advocate for systems and ML. Take a look at how system constraints affected the model.
- Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. In retrospect, the paper Rethinking the Inception Architecture for Computer Vision provides a better overview of the ideas and motivations behind the latest inception models.
- For a quick introduction to convolutional networks take a look at CS231 Intro to Convolutional Networks and Chris Olah's illustrated posts.
- Much of contemporary computer vision can be traced back to the original LeNet paper and it's corresponding 90's era website.
- There is a line of work that builds on residual networks starting with Highway Networks, then Densely Connected Convolutional Networks, and then more recently Deep Layer Aggregation. This blog post provides a nice overview.
{% include syllabus_entry %}
- Reading Quiz due before class.
- Intro [pdf, pptx]
- Autoencoders [pdf, pptx]
- Graph Neural Networks [pdf, pptx]
- We had originally assigned, Autoencoders, Unsupervised Learning, and Deep Architectures. However this paper is a bit theoretical for the goals of this class. Instead, you may alternatively read this overview paper and use it when filling in the reading form.
- Graph Neural Networks: A Review of Methods and Applications
- An excellent Survey on Autoencoders
- A tutorial on variational auto-encoders (and another tutorial)
- Original work on auto-encoders Learning Internal Representations by Error Propagation by Rumelhart and McClelland.
- The paper "Relational inductive biases, deep learning, and graph networks" provides some background and motivations behind deep learning on relational objects and introduces a general Graph Network framework.
- The paper "Semi-Supervised Classification with Graph Convolutional Networks" introduces graph convolutional networks.
{% include syllabus_entry %}
- Reading Quiz due before class.
- Intro Lecture [pdf, pptx]
- TensorFlow Presentation [pdf, pptx]
{% include syllabus_entry %}
- Reading Quiz due before class.
- RLlib [pdf]
- A3C [pdf]
{% include syllabus_entry %}
- Reading Quiz due before class.
- Learned Indexes [pdf, pptx]
- Learning to Optimize Join Queries [pdf]
{% include syllabus_entry %}
- Reading Quiz due before class.
- Learned Cardinalities [pdf]
{% include syllabus_entry %}
{% include syllabus_entry %}
- Reading Quiz due before class. There was a mix-up in updating the reading and the wrong paper was swapped. You may either read the Hyperband paper (preferred) or the Vizer paper (see optional reading) for the second reading.
- A Generalized Framework for Population Based Training [pdf]
{% include syllabus_entry %}
- Reading Quiz due before class.
- AutoML Overview [pdf, pptx]
- Designing Neural Networks with RL [pdf, pptx]
{% include syllabus_entry %}
- Reading Quiz due before class.
- Semantic Segmentation AutoML slides [pdf]
{% include syllabus_entry %}
- Reading Quiz due before class.
- Autonomous Vehicles Overview [pdf, pptx]
- Presentation: The Architectural Implications of Autonomous Driving[pdf]
{% include syllabus_entry %}
- Reading Quiz due before class.
- DL Compiler Overview [pdf, pptx]
- Presentation PDF
- Learning to Optimize Tensor Programs: The TVM story is two fold. There's a System for ML story (above paper) and this paper is their the ML for System story.
{% include syllabus_entry %}
{% include syllabus_entry %}
- Reading Quiz due before class.
{% include syllabus_entry %}
- Reading Quiz due before class.
- Overview [pdf, pptx]
{% include syllabus_entry %}
- Reading Quiz due before class.
{% include syllabus_entry %}
{% include syllabus_entry %}
{% include syllabus_entry %}
- Reading Quiz due before class.
- Introduction [pdf, pptx]
- AI Applications in Network Congestion Control [pdf, pptx]
{% include syllabus_entry %}
- Reading Quiz due before class.
- Introduction [pdf, pptx]
{% include syllabus_entry %}
- Reading Quiz due before class.
- Introduction [pdf, pptx]
- MobileNetV2: Inverted Residuals and Linear Bottlenecks
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
- Blog Post Comparing MobileNet and ShuffleNet
- SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and less than 0.5MB model size
- EffNet: An Efficient Structure for Convolutional Neural Networks
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Ternary Weight Networks
{% include syllabus_entry %}
- Reading Quiz due before class.
- Helen [pdf, pptx]
{% include syllabus_entry %}
- Reading Quiz due before class.
{% include syllabus_entry %}
- Reading Quiz due before class.
- Introduction [pdf, pptx]
{% include syllabus_entry %}
- Reading Quiz due before class.
- Introduction [pdf, pptx]
{% include syllabus_entry %}
- Reading Quiz due before class.
- DL Scheduling slides [pdf]
- Dominant Resource Fairness (DRF) slides [pdf]
{% include syllabus_entry %}
- Reading Quiz due before class.
{% include syllabus_entry %}
- Reading Quiz due before class.
- Neural Modular Networks Slides [pdf, pptx]
- Gonzalez Course Summary (Reflections on the Field of AI-Systems) [pdf, pptx]
{% include syllabus_entry %}
{% include syllabus_entry %}
{% include syllabus_entry %}
- Due at 11:59 PM
- Format: 8 pages (Google Doc)
- Email link to jegonzal@berkeley.edu and istoica@berkeley.edu
| Week | Date (Lec.) | Topic |
|---|
Detailed candidate project descriptions will be posted shortly. However, students are encourage to find projects that relate to their ongoing research.
Grades will be largely based on class participation and projects. In addition, we will require weekly paper summaries submitted before class.
- Projects: 60%
- Weekly Summaries: 20%
- Class Participation: 20%