Can we capture the unpredictable evolutionary and emergent properties of nature in software? Can understanding the mathematical principles behind our physical world world help us to create digital worlds? This class focuses on the programming strategies and techniques behind computer simulations of natural systems. We explore topics ranging from basic mathematics and physics concepts to more advanced simulations of complex systems. Subjects covered include forces, trigonometry, fractals, cellular automata, self-organization, and genetic algorithms. No computer programming experience is required, the course will start with the basics of code using HTML, CSS, and JavaScript (with the p5js framework). In addition to learning technical skills and discussing scientific concepts, students will have the opportunity to visit science museums, art galleries, and artist studios in the London area.
- Daniel Shiffman, daniel dot shiffman at nyu dot edu
- Course Number: COREA-AD 17J (Cross-listed with Interactive Media and Technology)
- Meets daily: 10am - 1pm
- Office hours daily: 2pm - 3pm (and by appointment via e-mail)
- Credit Hours 4
- Class google group for discussion / questions / announcements
- Homework wiki
- Field trip schedule see below
- Meets 2-5pm today (as to not conflict with morning orientation).
- What is computational media?
- What is programming?
- How can I apply programming to my field of interest?
- Example projects.
- What is the potential for software within the visual arts?
- As a ____________, why would I want or need to write software?
- Programming language discussion
- history of Processing and p5.js (and processing.js?)
- Basics of web development
- Landscape of HTML, CSS, and JavaScript?
- Running things locally
- What is git and github?
- Drawing with numbers
- Coordinate Systems
- Shapes
- Color
- Code
- Sol Lewitt
- The flow: code blocks, setup, draw, and events
- Variation: mouseX, mouseY
- Variables: Declare, Initialize, Use
- random()
- map()
- FTPing to server
- Basics of blogging
- Introduction to wordpress
- Your blog is at yournetid.imnyuad.com/blog. Follow the install instructions at that link (it should be fairly self explanatory). If you are stuck you can watch this video walkthrough.
- If you want to get a bit more into wordpress, you can start here.
- p5.js
- Tutorial: Getting started with p5.js - Set up your files for editing and viewing.
- Reading / Watching
- Read p. 8-41 of Form + Code
- Supplemental
- This course is now taught with JavaScript and p5.js rather than Processing. The material in the following chapters and videos mirrors the content we do in class, however it is in a different language. I leave it up to you whether you find this helpful to look at or not.
- Chapters 1-2, Learning Processing
- Videos 0.0-2.1
- Homework 1
- Transformations
- Conditionals
- If, else if, else
- Boolean variables
- Relational Operators, Logical Operators
- Buttons, rollovers, switches
- Functions
- Re-usability
- Modularity
- Calling vs. Defining
- Parameter Passing
- Return types
- Recursion
- Event functions
- More on randomness and Perlin Noise
- Intro to Objects
- Reading
- Chapter 8 of Learning Processing or Processing OOP tutorial
- Nature of Code Introduction
- "Hackers and Painters" by Paul Graham
- Loops
- while
- for
- Modulo
- Objects
- Principles and Theory (Encapsulation)
- JS literal object
- Object "constructor" function
- Bletchley Park
- 8:55 am, meet at Guilford House
- Estimated return: 4:30pm
- Review loops
- while
- for
- Arrays
- Resizing an array
- Other array functions
- A String is like an array of characters
- An image is like an array of pixels
- Related Reading
- Chapters 6 and 9 from Learning Processing cover the same concepts in the Java programming language.
- Array tutorial from Mozilla
- Array tutorial from JQuery
- For a JavaScript tutorial about arrays I would suggest Chapter 4 of Eloquent JavaScript though I would warn that it gets a bit more advanced than what we need for p5.js sketches.
- There are video lessons in progress about these topics in Processing. They cover all the same content but the syntax in JavaScript is different. Videos 15.0 - 15.9
- Images
- Load and display images
- Writing pixels to screen
- Reading image pixels, image processing: brightness, threshold, etc
- Chapter 15 examples
- In class exercise: make an interactive image processing filter
- Video:
- Live video (Capture) and movie playback (Movie)
- Drawing shapes on screen colored by pixels
- Chapter 16 examples
- Exercise: Make a mirror that paints your portrait
- Intro to Computer Vision
- Related reading:
- Computer Vision for Artists and Designers by Golan Levin
- Learning Processing, Chapters 15-16
- Studio visit to Random International and the V&A Museum.
- Meet at 10:45 am at Random International (our tour is scheduled for 11, see directions sent via e-mail).
- 12:00 pm - 1:00 pm - trip to V&A. We'll head directly to the V&A after the trip, someone from Random International will meet us there to demo the exhibit.
- 2:30 pm - 3:30 pm - meet at NYU London for one hour class to finish up images and talk about data.
- Guest speaker: Stefanie Posavec
- Guest speaker: Marcin Ignac
- Intro to Data
- review events and callbacks
- Loading local files
- Query strings
- Web services: AJAX, JSON, and APIs
- Homework 6
- Intro to Physics
- Vectors
- Forces
- Physics Libraries
- Additional Reading
- Computational Beauty of Nature: introduction
- Computational Beauty of Nature: Chapter 16
- There are also the Nature of Code videos.
- All examples
- Related and past projects
- Homework 7
- We will meet from 12:30 - 3:30!
- Complex Systems:
- Reading:
- Craig Reynolds Steering Behavior
- Computational Beauty of Nature: Chapter 5-6, 15, 20
- Create a final project proposal
- We will be meeting for a field trip at 10am, check your e-mail for directions
- We are visiting Hirsch and Mann and the Machines Room
- Evolutionary Computing:
- Final Project Proposal discussion
- 10:00 - 12:00 pm: presentations
- 12:30 pm: lunch and exhibition
- Assignments will include readings, programming exercises, and written documentation of your work.
- Each student will be required to post their homework to the class blog (instructions provided in class).
- Over the course of the term, you should have completed ten blog posts (8 exercises, 1 final project proposal, 1 final project summary). In total this should amount to at least 7,000 words (equivalent to 18 double-spaced pages.) Each blog post should contain a written response to material presented in class (specific questions will be provided), a description of your homework assignment, and questions regarding the technical material.
- For your final project proposal and summary, you will be required to write about your project concept and project development process.
- Students will be expected to collaborate, to document their work, to make presentations and to discuss their ideas regularly in class.
- Class time will be divided between technical lectures, independent project development, individual meetings with instructor, and project presentations / critique.
- In addition to regular class meetings, the instructor will organize a series of field trips to visit artist studios, gallery shows, and museum exhibits in the London area.
- Define the fundamentals of computation.
- Build creative software sketches that demonstrate knowledge of these fundamentals.
- Critically examine interactive art in the context of screen, sculpture, and web.
- Apply concepts from physics and mathematics to real-time software animation.
- Develop methodologies for simulating nature using computational algorithms.
- Propose and build a creative project made from computational methods.
- Class Participation (attendance, discussions, project proposal and implementation presentations) : 25%
- Writing Assignments: 25%
- Programming Assignments: 35%
- Final Project: 15%
- One liner – What did you do?
- Content – Why did you do it? Who is the audience? How does it engage with the theoretical and technical concerns we have discussed in this class?
- Demo
- Comments/Critique
- The Nature of Code, Daniel Shiffman
- The Computational Beauty of Nature, Gary Flake
- Form + Code, Casey Reas / Chandler McWilliams
- Learning Processing, Daniel Shiffman
- Sign up for a github account to be able to edit this wiki.
- Sign up for the class google group.
- Read p. 8-41 of Form + Code
- (completed in class) Using p5.js, create your own screen drawing: abstract design, self-portrait, alien, monster, etc. Use only 2D primitive shapes –
arc()
,curve()
,ellipse()
,line()
,point()
,quad()
,rect()
,triangle()
– and basic color functions –background()
,colorMode()
,fill()
,noFill()
,noStroke()
,stroke()
. Here's a simple example. - Create a animated application. Start over from scratch and build something with a very simple design. Focus on the logic of variables and avoid using hard-coded values. Play with mouseX and mouseY. If you are stuck, here are some ideas.
- Experiment with motion using a single simple shape. Can you create a randomly jittering "nervous" square? (Here is a sample). A circle that spirals around the window? How could user interaction affect the shape's motion?
- Use random() to create a painting system. Here are some examples.
- Follow the blog setup instructions at netid.imnyuad.com/blog.
- After reading the beginning of Form+Code and looking at example computational media projects write a blog post about how computation applies to something your interests. This could be your major, a different class you took last term, a personal hobby, or a cause you care about. What projects do you imagine making this term?
- In the same blog post (or a new one), document the process of creating the above two sketches. What pitfalls did you run into? What could you not figure out how to do? Here are some examples.
- Note that the total writing requirement for the term is 7,000 words so you'll want to keep each post around 700 words to keep up.
- Upload your sketch to your server by following these instructions.
- Contribute any questions below.
- Add a link to your sketch running online and your blog post(s) below.
- If object-oriented programming is new to you, read Chapter 8 of Learning Processing or the shorter Processing OOP tutorial. This will help prepare you for tomorrow's discussion on objects in JavaScript.
- We're going to look at Perlin Noise in class soon. For a head start read and to think about objects more read the Nature of Code Introduction.
- For more on fractals and recursion read Chapter 8 from 8.1 - 8.3.
- Create a p5.js sketch that involves interaction with conditionals and/or functions. You can use one of the ideas below or invent your own assignment.
- As an exercise, try making a rollover, button, or slider from scratch. Compare your code to the examples on github. This is something of a silly exercise because as we briefly saw in class we can get all these interface elements for free with HTML. Still this is an opportunity to practice conditional statements and well as invent new GUI elements beyond buttons, sliders, rollovers, etc.
- Take the idea for a design and break it out into its own function. Add parameters to the function so the design can be drawn differently based on the selected parameters. Draw three or more instances of the design on the screen to show the differences possible by changing the parameters. Here is the example demonstrated in class
- Create a design from a recursive function. How could you animate or change this design over time? Can you make it interactive or have it controlled by
noise()
? Here are some samples as shown in class.
- Document the process of creating your homework in a blog post. What were you inspired by? What pitfalls did you run into? What could you not figure out how to do? What are your next steps?
- Contribute any questions below.
- (required) Design a sketch in an object-oriented fashion. Try to eliminate code from setup() and draw()) except for the core requirements (size(), background(), etc.) and calls to objects. For example: Consider building a particle system. A particle system can be used to simulate: rain, snow, fireworks, explosions, smoke, etc. For this first step, you would create a single Particle object or maybe two on the screen using separate variables. Try making a separate file for your object like in this example.
- (optional) A second step for those of you feeling ambitious is to try using an array to create multiple instances of an object. Here is an example you can use as a model. You should probably stop here, but here are some additional (more difficult) possibilities.
- Experiment with the resizable nature of arrays. Can you add objects one at a time? Remove them after a certain amount of time or when they leave the screen? Can objects from on ArrayList trigger the birth of other objects? See this example as demonstrated in class.
- Create an object that stores the history of its own path. This might be a drawing program where you keep a history of mouse locations in an ArrayList or a shape that moves around the screen autonomously and draws its own trail. Can you have an ArrayList of these objects (each of which tracks its own ArrayList?). Example 1 Example 2
- Read Computing Machinery and Intelligence by Alan Turing (1950). Can machines think? Do Alan Turing's questions and arguments hold up today? What has come to fruition and what has not? Write some thoughts about the paper and your impressions from the trip to Bletchley Park. There is no need for a formal essay, just some impressions and ideas.
- Put a link below to your p5.js sketch as well as blog documentation of your process and thoughts from the trip.
- Using arrays, write a program that creates multiple instances of an object (feel free to use an object you developed previously or create something new). Here is an example you can use as a model. You should probably stop here, but here are some additional (more difficult) possibilities.
- Experiment with the resizable nature of arrays. Can you add objects one at a time? Remove them after a certain amount of time or when they leave the screen? Can objects from an array trigger the birth of other objects? See this example as demonstrated in class.
- Create an object that stores the history of its own path. This might be a drawing program where you keep a history of mouse locations in an array or a shape that moves around the screen autonomously and draws its own trail. Can you have an array of these objects (each of which stores its own array?). Example 1 Example 2.
- A String is like an array of characters. Can you create a sketch that plays with text by looping through the "array" inside a String. See: charAt(). We'll get into this in more detail when we look at data, but here is a tutorial you can start with.
- An image contains an array of pixels. Experiment with image processing by looping through all the pixel colors of the canvas or an image. See this example and this one for some ideas. We'll look at images and pixels more closely tomorrow.
- Some reading material / referances:
- Chapters 6 and 9 from Learning Processing cover the same concepts in the Java programming language.
- Array tutorial from Mozilla
- Array tutorial from JQuery
- For a JavaScript tutorial about arrays I would suggest Chapter 4 of Eloquent JavaScript though I would warn that it gets a bit more advanced than what we need for p5.js sketches.
- Put a link below to your sketch as well as documentation of your process.
Note this is due Thursday, Jan 15. If you are stuck here are some ideas:
- Create a software mirror by designing an abstract drawing machine which you color according to pixels from live video. Here are some model examples: Pointillism, Painters, Brightness Mirror, Scribbler.
- Make a slideshow of images. Take a look at this Crossfade Effect.
- Use one of the computer vision examples to augment a video image. Can you put glasses on a face or make Particles come out of a person's mouth, etc.? You can use this example as a model: Simple Face Detection. There are some other new ones too but note they are experimental and not as stable.
- Document your process on the above.
- Write a response to our visit to Random International and the V&A. Some questions to consider (you don't need to answer these, just some ideas to get you started):
- What role does technology play in the fine art world? Is the artwork the process, the result, the code? What do you think about the idea of making "limited editions" of software or physical computing projects?
- What did you like about Swarm, not like?
- Did you visit the Science museum and if so write about any interactive experiences there?
- How can software emulate nature?
- Computer Vision for Artists and Designers by Golan Levin
- Processing images and pixels tutorial -- note this uses Java syntax and also the pixel array has only one spot for each color (not 4!) so it works differently than in JavaScript.
- FaceTracker explained
- You are not required to do any programming for this assignment, but of course you can. Your assignment should include a blog post whether you create a javascript sketch or not. Here are some possibilities (choose one or more).
- Write a response to Stefanie and Marcin's presentation. Some ideas:
- What makes something a "data visualization"?
- What qualities make up good "data visualization"?
- To what extent can data visualization be objective? Should it be objective? What is the role of the artist or designer in telling a story about or with the data?
- Find a data source that interests you. Does it have what you need? What format is it in? Do you know how to access it? What are some ideas you have for visualizing it? What story do you want to tell?
- Try getting JSON into a p5.js sketch. You can use any of the examples as a model. Here is a list of possible data sources.
- Write a response to Stefanie and Marcin's presentation. Some ideas:
- p5.js notes: Loading-external-files:-AJAX,-XML,-JSON
- Robyn's Instagram API How To
- Artistic Data Visualization: Beyond Visual Analytics by Fernanda B. Viégas and Martin Wattenberg
- Flowing Data
Try incorporating the concept of vectors and forces into a sketch. Here are some ideas:
- Convert something you did previously to using vectors.
- Simulate motion affecting only the acceleration. Create a formula for calculating a dynamic acceleration, one that changes over time based on any number of factors. What happens if you make more than one object with an array?
- Using forces, simulate a helium-filled balloon floating upward (and bouncing off the top of a window). Can you add a wind force which changes over time, perhaps according to Perlin noise?
- Create an example where instead of objects bouncing off the edge of the wall, an invisible force pushes back on the objects to keep them in the window. Can you weight the force according to how far the object is from an edge, i.e. the closer it is, the stronger the force?
- Research a force from physics and implement it as a vector.
- Use the concept of forces to visualize some input (could be data, literal example would be get windspeed online and translate to a wind force in Processing, but feel free to think more abstractly)
- Computational Beauty of Nature: introduction
- Computational Beauty of Nature: Chapter 16
- Nature of Code Chapter 1
- Nature of Code Chapter 2
- If you want to use a physics engine like box2d or toxiclibs I would suggest reading Chapter 5 as well.
- You might want to look ahead nature of code chapters 6 - 9 as well. We'll be going over this material on Monday.
- There are also the Nature of Code videos if you prefer this over reading.
Create a blog post documenting your idea for a final project and be prepared to present the idea in class. Your proposal should include the following (though feel free to use this as loose guidelines, you can document your idea however you see fit.)
- Title
- One sentence answering: "What is it?"
- Written description of the idea.
- How did you become interested in this idea?
- What is the audience?
- What questions do you have for the class (conceptual or technical)?
- Visual material (images, videos, hand-drawn sketches etc.) that demonstrate your idea.
- Links to any projects or references that inspired you.
- Links to in progress sketches that demonstrates any part of the idea.
The due date for the project is Thursday, Jan 22nd, however if you would like more time to complete the blog post the absolute deadline is Monday, Jan 26th 8 am GMT. Add a link to your project running online and a blog post documenting your work. It's up to you to figure out how to best document your project, if you are looking for some ideas here are some things to include.
- Title
- Brief written description (describe your process, what inspired you, what you struggled with, etc)
- What are your next steps if you had more time?
- What did you learn from "testing" your project with users (if you write your post after tomorrow).
- Visual Documentation: screenshots, gif animation, video, sketches, etc.
- References: links to related projects, code samples, etc.
- Code citations (what examples and libraries did you use?)
- Link to running project
- 10:00 am -- arrive and last minute questions, etc.
- 10:30 am -- project presentations (~10 minutes each)
- 12:00 pm -- break and course evaluations
- 12:30 pm -- exhibition and lunch