Processing Piano Tutorial Videos in the Browser

Summary

There are a lot of piano “tutorial” videos out there, but most of them aren’t step-by-step walkthroughs. They mostly look like falling raindrops that light up piano keys. These tutorials are often made with a tool called Synthesia. I’ve struggled with these videos because I can’t keep up with the falling notes or figure out the chords I need to play. It’s up to you to keep track of the light bars. For example, I wanted to learn a Telugu song called Samayama, but it only came in video tutorial format. So I built PianoReader. PianoReader is a web tool that watches piano tutorial videos and spits out the notes and chords. As a bonus, it does it all in the browser without extra server compute. It takes in a tutorial video and outputs piano tablature that looks like this: |Left Hand | Right Hand | | -------- | ---------- | | A Maj | A D | | A Maj | A | | D min | D | How it works I knew this problem was going to involve some level of computer vision. It needed some kind of classifier or bounding box detection to make it work. Alternatively, I considered training a model. But I didn’t want to sit and label hundreds of videos to predict notes. I mean, if I did the work to label just one video, I wouldn’t even have to build this :) HTML Canvas HTML canvas is a cheap and client-first option to do low-cost video processing. No server needed, just browser. Canvas ships with a convenient way to pull in video frames. On every frame of the video, you can hook into it with a callback: const canvas = document.getElementById("myCanvas"); const ctx = canvas.getContext("2d"); const video = document.getElementById("video1"); // Get your video element const drawVideoToCanvas = () => { // do some work here with the frame ctx.getImageData([]); // Re-register the callback to be notified about the next frame video.requestVideoFrameCallback(drawVideoToCanvas); }; // Start the drawing loop when the video starts video.requestVideoFrameCallback(drawVideoToCanvas); Detecting Piano Keys Before I c...