CS 349T / EE 192T: Video and Audio Technology for Live Theater in the Age of COVID, Autumn 2020
Course info
Course basics
| Lectures | Thursday, 4:30 p.m.–5:50 p.m., remote synchronous on Zoom |
| Prerequisites | CS 110 or EE 102A. Recommended: familiarity with Linux, C++, and Git. |
| Optional reading | Steiglitz, "A Digital Signal Processing Primer: With Applications to Digital Audio and Computer Music," Sayood, "Introduction to Data Compression,” Poynton, “Digital Video and HD Algorithms and Interfaces," Montgomery, “A Digital Media Primer for Geeks” & “Digital Show & Tell” |
| Course description | This class is part of a multi-disciplinary collaboration between researchers in the CS, EE, and TAPS departments to design and develop a system to host a live theatrical production that will take place over the Internet in the winter quarter. The performing arts have been greatly affected by a transition to theater over Zoom and its competitors, none of which are great at delivering low-latency audio to actors, or high-quality audio and video to the audience, or feedback from the audience back to actors. These are big technical challenges. During the fall, we’ll build a system that improves on current systems in certain areas: audio quality and latency over spotty Internet connections, video quality and realistic composited scenes with multiple actors, audience feedback, and perhaps digital puppetry. Students will learn to be part of a deadline-driven software development effort working to meet the needs of a theater director and creative specialists—while communicating the effect of resource limits and constraints to a nontechnical audience. This is an experimental hands-on laboratory class, and our direction may shift as the creative needs of the theatrical production evolve. Based on the success of class projects and subsequent needs, some students may be invited to continue in the winter term with a research appointment (for pay or credit) to operate the system you have built and instruct actors and creative professionals how to work with the system through rehearsals and the final performance before spring break. |
Lecturers
Sadjad Fouladi
Email:
Tsachy Weissman
Email:
Keith Winstein
Email:
Michael Rau
Email:
Assignments & Grading
This is a project-based class; students will work in a group on a single project, or series of smaller related projects, over the course of the term. Groups will draft a design document early in the term and will revise it as their approach (and understanding of the problem they have chosen to tackle) necessarily evolves. Students will be evaluated based on their design document, their participation in class, demonstrations of intermediate milestones to the rest of the class, and a final report on the project. Students will not be evaluated based on their success at landing pull requests in the production codebase. There will not be an exam.
Learning Goals
- Develop skill and maturity in software development (with deadlines) to meet the needs of a winter-term theatrical production. Learn fundamentals of robust failure-tolerant modular software design, code review, pull requests, and how to make progress when reading error messages and core dumps.
- Learn the basics of video compression, audio compression and applied signal processing, their practical tradeoffs in a real-time setting, and how to interface with them as a software developer.
- Learn practical protocol development for real-time IP networking under latency constraints.
- Develop the grasshopper on your shoulder to tell you when you should ask for help, and how to elicit it from relevant experts (both technical and creative).
Possible Group Projects
- Measure the impulse response of a Stanford auditorium, then implement a real-time audio filter to make our actors feel like they are in the same space auditorily.
- Extract the chroma-key module from Blender and wrap it in simple C++ code, then demonstrate that it’s possible to composite two separate video feeds (each of one actor filmed in front of a green screen) into a single video that makes it look like the two actors are in the same room and are making believable eye contact with each other.
- Develop a real-time waveform monitor and color-correction controls for live video in the Y’CbCr colorspace, and demonstrate that these are the right controls to produce aesthetically pleasing results.
- Develop software that allows an actor to hold up a color-correction card (with known gray and color swatches) in front of a USB webcam, detects the location of each patch, then adjusts the webcam’s settings to auto-correct the exposure and colors.
- Measure statistics of an Internet connection (patterns of loss, of jitter/queueing under different stimuli), then design a protocol to achieve low-latency data transfer over an uncertain network.
- Take a C++ video compression codebase and optimize it to run reliably at 24 fps (sacrificing quality if necessary).
- Implement software to interface with lights controlled over DMX512-over-RJ45.
- Build a system to monitor equipment in the field (metrics about local computation timing and network communication) and flag anomalies/alerts.
- Create an HTML/JS frontend and C++ backend for the audience to watch a play over WebRTC and approximate the feeling of “being there live,” by giving actors and other audience members real-time feedback (e.g. the sound of audience laughter captured from audience microphones).
- We will happily consider other proposals for group projects.