Prepared for Golan Levin's Interactivity and Computation at CMU by Joey Lee
Forked from The intro to ML with ml5.js originally prepared for ITP Camp by Yining Shi & Joey Leekeywords: machine learning, ml5.js, p5.js, friendly, introduction, workshop
Support generously provided by: The Frank-Ratchye STUDIO for Creative Inquiry at Carnegie Mellon University and NYU's Interactive Telecommunications Program.
- This is an introductory workshop to machine learning with ml5.js
- Context/Audience: prepared for Golan Levin's Interactivity and Computation at CMU. Students in this course are are creative tech folks or creative tech adjacent and are considered "intromediates" -- "students develop the skills and confidence to produce interactive artworks, discuss their work in relation to current and historic praxes of digital art, and engage new technologies critically." -- but we hope this workshop can be used with audiences of all shapes and types.
- General overview of machine learning concepts and terms with emphasis on learning through making.
- We use ml5.js, a javascript library to do machine learning for the web.
Welcome to this friendly introduction to machine learning with ml5.js. We're happy you're here.
ml5.js is about machine learning and people. A major part of the ml5.js project is about creating broader engagement with machine learning and its effects on you, me, and the people and environment. As a user of ml5.js you are agreeing to abide by the ml5.js code of conduct. It's easy and fun to be nice and respectful to those around you. Let's have a wonderful workshop together. If you're looking for more details on what constitutes being "not nice" please see Mozilla's Community Participation Guidelines which are applicable to the space that ml5.js wants to create.
If you have any questions or need to report any issues please contact: info@ml5js.org
- Agenda
- Key Questions and Takeaways
- Introduction to machine learning
- Introduction to ml5.js
- Coding Session
~3 hours
- Hello! (5 mins)
- Intro to Machine Learning (30 mins)
- Intro to ml5.js (15 mins)
- 10 Min Break
- Coding Session Part 1 (~30 mins)
- Coding Session Part 2 (~45 mins)
- Project Shareouts (20 mins)
- Q&A, Closing notes (10 mins)
This is an introductory workshop to machine learning. We are focused on helping you to not only learn through making but also start to reflect through making. The following questions are here to help focus the learning and reflecting.
- "Machine learning is programming with
__________rather than____________." - Kyle McDonald - Why are data central to machine learning?
- What do we mean by "good" data vs. "bad" data? Why does this matter for machine learning?
- What are the implications of all the advancements and increasing availability in machine learning tools and capabilities?
- What is the role of art and design in machine learning today?
- Link to slides: Slide presentation with introductions, overview of terms, project highlights and more.
AI/ML isn't just about advancements in technology, it is about a broader universe of things that are coming together (or diverging). These include changes (or lackthereof) in governance, cultural and social interactions with, awareness of, and attitudes, and feelings towards these new technologies, the development of new tools and communities around and against AI/ML, expansion of AI/ML education, and use of AI/ML in art and design or other domains outside "traditional AI/ML" spaces (e.g. engineering and computer science).
We increasingly interact with machine learning systems every day.
Our lives -- e.g. how we access content, who gets access to services, and how we get from A to B -- are being shaped by the data we produce and the people using our data.
These increasingly algorithmically and data driven interactions reflect an assemblage of social and technical forces that almost always benefit some and harm others.
It is worth thinking about how the following factors affect or are affected by changes in AI/ML.
- Research
- Policies
- Environment
- Tools
- Communities
- Education
- Art/Design
Collected from Yining's Week 1 of Machine Learning for the Web
- Artificial Intelligence:
- AI is the science of making things smart:
- Human Intelligence Exhibited by Machines
- A broad term for getting computers to perform human tasks
- General AI: fabulous machines that have all our senses (maybe even more), all our reason, and think just like we do.
- Most of AI today is Narrow AI systems that can only do one (or a few) defined things as well or better than humans.
- Machine Learning:
- One of the Approaches to Achieve Artificial Intelligence
- A field of computer science that uses statistical techniques to give computer systems the ability to "learn" with data, without being explicitly programmed.
- LOTS OF DATA
- Types of machine learning learning styles:
- Supervised Learning:
- Training data is labeled
- Unsupervised Learning:
- The machine learn from an unlabelled data set.
- Unsupervised learning can be grouped into clustering and dimensionality reduction.
- Reinforcement Learning:
- The machine learn by trial-and-error through reward or punishment.
- Supervised Learning:
- Types of machine learning outputs:
- Regression: Predict continuous values (price of a stock)
- Classification: Predict a class/label (cat or dog?)
- Clustering: Groupings based on potential relationships in the data - Most similar other examples (Amazon related products recommendation)
- Sequence Prediction: Predict the next value/number/word/sentence in a sequence - e.g. the next chord in a melody
- Types of machine learning use cases:
- Object Recognition
- Speech and Sound Detection
- Natural Language Processing
- Prediction
- Translation
- Deep Learning:
- A technique for implementing Machine Learning
- Deep learning - Uses multi-layered artificial neural networks to deliver state-of-the-art accuracy.
- Can automatically learn representations/features from data such as images, video or text.
The process of machine learning is messy. Like most data related projects, 80% of your time or more will be spent cleaning and preparing data for use. The rest of the time will be the actual "making" of your project. The machine learning process might be broadly (over)simplified into a handful of steps which includes: Data collection, data preparation, Training a machine learning model, and using the ML model in context.
- Data collection
- E.g. an artist writes a script to automatically download all images of cats, dogs, and badgers from the internet.
- Data preparation
- E.g. the artist uses a crowd tasking service to 1. confirm that the image is either a cat, dog or badger, 2. Tag the image with the animal type, 3. Crop the image of the animal to a square aspect ratio. The artist then resizes all the images to 64x64px
- Training a machine learning model
- E.g. the artist: 1. Uses a popular machine learning framework called Tensorflow in python to create a neural network. To create a neural network the artist, writes code that specifies image processing and data crunching methods that aim to pull out unique features for each animal type in the given images. The artist then 2. Iteratively refines and tests the model to achieve the highest accuracy.
- Using the ML model in context...and awesome art projects
- E.g. the artist opensources this new pre-trained dog/cat/badger model. This comes just in time since the ASPCA in NYC has been in great need of a method to differentiate between the animals in their shelters. The artist is hired to create a web application that uses this model to label their shelter animals “cat”, or “dog” or “badger”.
- ml5.js website:
- The main ml5.js website
- ml5 documentation
- the documentation page for ml5
- ml5.js organization:
- The ml5.js github organization. Here you can see all the repositories that make up the ml5.js project.
- ml5.js library:
- The main library for ml5.js. This is where all the ✨happens!
- ml5.js examples:
- The examples that correspond to all of the functions that are available in the ml5.js library. Emphasis is given to p5.js examples.
- Index of all the ml5 examples
- ml5.js examples on p5 web editor
- The examples can be found as well in the p5 web editor. You can run the sketches directly and tweak the examples as you'd like. Note, some of the examples currently do not work in the editor.
- ml5.js Medium blog page
- ml5's Medium blog page.
During this code-a-long session, we will walk through several features of ml5.js. Specifically we will walk through the following features: We will start with a blank ml5 boilerplate on the p5 web editor for these examples.
- Pretrained models:
- Training your own model:
In this short 45 min session, select a pre-trained model (e.g. PoseNet) or build your own custom neural network to make a simple interactive web application you can share with your classmates. Team up in groups of 2 or more.
→ Use this ml5 boilerplate on the p5 web editor to get started. OR feel free use your favorite text editor and development environment.
- ml5 documentation:
- ml5 examples
- ml5 examples in the p5 web editor
- Github Repository with code
- Index of all the ml5 examples Some ideas to spark your creativity:
- ml5 community page
- Unicorn racing game
- Yining Shi's "Machine learning for the web" course at NYU's ITP.
- Machine Learning Workshop by @AndreasRef
- Coding Train Video Series:
- Two Minute Papers
- Udemy - Machine learning with Javascript
- Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems
- 3 Blue 1 Brown - Neural Networks
- Facial Recognition Tech Is Growing Stronger, Thanks to Your Face, New York Times
- Eye Conductor, Andreas Refsgaard
- Training a single AI model can emit as much carbon as five cars in their lifetime, MIT Tech Review
- Deep Learning and Carbon Emissions, Lukas Biewald
- One Month, 500,000 Face Scans: How China Is Using A.I. to Profile a Minority, New York Times
- Red Feed, Blue Feed, Wall Street Journal
- OpenAI's GP-T2 text generation algorithm
- Yining Shi's Machine Learning for the Web course at NYU's ITP
- Courtney Morgan, Processing Fellow
- re:ImageNet Roulette, Gaurdian
- Excavating AI, Crawford & Paglen
- ImageNet Roulette, Crawford & Paglen
- San Francisco bans the use of facial recognition tech, WIRED
- AI Winter
- Book of Shaders - referred to for explaining CPU vs. GPU
- Red Burns, Founder of ITP
- P5.js
- Various curated ML projects
- Dan Shiffman's EyeO Talk 2019
- Experiments with Google
- Food Chain Game by Lillian Ritchie
- Forensic Architecture - triple chaser // Note: great example of training images and taking care of potential background effects.
- ml5.js
- RunwayMl - Machine learning for creators. "Bring the power of artificial intelligence to your creative projects with an intuitive and simple visual interface. Start exploring new ways of creating today."
- Google's teachable machine - do transfer learning in the browser with google's teachable machine.
- Tensorflow for python
- Tensorflowjs - TensorFlow.js is a library for developing and training ML models in JavaScript, and deploying in browser or on Node.js
- Keras - Keras: The Python Deep Learning library
- Google co-lab -
- Spell.run - "The most flexible and powerful end-to-end platform for ML and deep learning engineering."
- Paperspace - "The first cloud built for the future. Powering next-generation applications and cloud ML/AI pipelines."
- Openmoji - "Open-source emojis for designers, developers and everyone else!"