Music For 18 Musicians Gets Generative Music Video Made With Machine Learning

Off topic, but interesting: Damien Henry created this generative music video for the classic Steve Reich composition Music for 18 Musicians.

The graphics are generated completely by an algorithm using machine learning. While the video is generative, low-res and unrealistic, it captures the essence of watching the evolving landscape out the window of a moving vehicle.

Here what Henry has to say about the video:

I used videos recorded from trains windows, with landscapes that moves from right to left and trained a Machine Learning (ML) algorithm with it.

First, it learns how to predict the next frame of the videos, by analyzing examples. Then it produces a frame from a first picture, then another frame from the one just generated, etc. The output becomes the input of the next calculation step. So, excepting the first one that I chose, all the other frames were generated by the algorithm.

The results are low resolution, blurry, and not realistic most of the time. But it resonates with the feeling I have when I travel in a train. It means that the algorithm learned the patterns needed to create this feeling. Unlike classical computer generated content, these patterns are not chosen or written by a software engineer.
In this video, nobody made explicit that the foreground should move faster than the background: thanks to Machine Learning, the algorithm figured that itself. The algorithm can find patterns that a software engineer may haven’t noticed, and is able to reproduce them in a way that would be difficult or impossible to code.

What you see at the beginning is what the algorithm produce after very little learnings. It learns more and more during the video, that’s why there are more and more realistic details. Learnings is updated every 20s.

2 thoughts on “Music For 18 Musicians Gets Generative Music Video Made With Machine Learning

  1. Cool! The next step is to learn the machine to listen and adapt to the flow of the music. I highly recommend to see the video for The Chemical Brothers – Star Guitar.

Leave a Reply