Music To Our Ears: How AI Makes Music

Jun 11 2021

If Frédéric Chopin attained notability by age fifteen, and Wolfgang Amadeus Mozart by 17, how old will AI be when it wins its first Grammy? 

AI already shapes the way that we engage with music in ways we might not even think about. Our Spotify accounts arguably know us better than we know ourselves, as the platform’s deep learning system becomes better at analyzing listening histories and working out what we might want to hear next. Their “collaborative filtering” system profiles your taste and then curates your Discover Weekly playlist for you, effectively “reading” the 50 million plus songs in the Spotify catalog through Natural Language Processing (NLP) technology so it can match you up to the right tracks. Machine-learning tech from a company called Super Hi-Fi works with Sonos Radio to automatically work out how to transition from song to song, adjusting the volume to avoid jumps and switch tracks smoothly. It’s also used on radio stations to prevent gaps between songs, also known as “dead air”.  

AI is not only shaping our music tastes and listening habits, but how we create the music we listen to. Projects with AI-generated compositions are well underway. Lucas Cantor, a Los Angeles-based composer for film and TV, used AI to complete Franz Schubert’s famously unfinished Symphony No. 8 in 2019. Collaborating with Huawei, Cantor fed around 2,000 pieces of piano music from Schubert’s catalog into their AI software to teach the AI to think like Schubert and to finish the song. 

Elsewhere, tech enthusiast and former American Idol contestant Taryn Southern released a pop album to be co-written and co-produced with artificial intelligence, 2018’s I AM AI, and musicians like Arca and Toro Y Moi have collaborated with AI. “I feel like we’re at the end of art, human art,” the musician Grimes said last year on Sean Carroll’s Mindscape podcast. While this statement sparked debate, those working in the field of AI and music claim that one thing is hard to deny: AI can be used to speed up and maximize the musical composition process.

There are other applications of AI in music beyond the entertainment industry. Mercury Orbit Music – selected as one of the 38 qualified teams the $5M IBM Watson AI XPRIZE – uses machine learning as an educational tool, or what they call “edutainment”. Their software is a learning platform for students to test out their musical creativity. “It’s about human-to-AI creative collaboration, that’s our goal, what we do,” says founder and CEO Yang Zhang, who is also a musician with a background in film composition. 

One of Mercury Orbit Music’s AI music generation tools is Chord Wizard, which experiments with chord progression based on the selected music styles, she explains. Another is called Rhygen – a tool that generates and blends rhythm and bassline based on the selected chord progressions based on your choices of music styles. Mercury Orbit Music’s venture has been selected and working intensively on the USC Ed-tech Accelerator Program entitled "USC Ed-Tech Cohort 3" as one of the top 11 ed-tech startups on a global scale. To date, they have signed up 16 schools – located both in California and China – to work with their AI-enabled music applications and programs.

But why not just use traditional instruments, you might ask? “This is a fascinating question,” says Zhang, explaining that their software not only allows people to explore their creativity, but sparks new inspiration, and empowers the next generation to work with AI in music – since this future is clearly coming for the industry. Through their ongoing research project live-coding platform, they expect next-generation music learners to mix the learning experience of music production, live coding, and programming languages by interacting with the AI music environment in one destination.  

“It’s not about replacing human intelligence or creativity,” says Zhang. “It’s a balance – we don’t want people, especially for the next generation of music learners to become lazy and refuse the traditional music learning curve. I think it’s very important to keep stretching your own brain capability and your creativity.” The same goes for human-AI collaboration beyond just music, she points out. There is a difference between the AI-made draft of your music composition and a fully polished, real-time music production. By its very nature, the system forces human input: “You have to learn, make your own creative judgements, think how you can improve it or change it in a different way, make it more polished. You have to combine it with your own work to get a better result,” she says. In this sense, Zhang hits on something many experts seem to agree on when it comes to applications of AI in music: there is room for improvement. 

Even in AI-generated playlist curation, AI struggles to understand music that falls outside of genre. As Sonos Radio general manager Ryan Taylor told Fortune: “The truth is music is entirely subjective. There’s a reason why you listen to Anderson .Paak instead of a song that sounds exactly like Anderson Paak,” referring to a well-known R&B singer. People like a song for a whole bunch of different reasons, he pointed out, ranging from loving the narratives behind their favorite artists’ songs to a cultural connection with them. It’s this type of context that AI might not understand. 

Mick Jagger’s charisma or George Michael coming out as gay in a song, say. “At some point in the future, AI might be able to pick up on that stuff,” Taylor said. “Ultimately neural networks can get there for sure, but they need more input than a catalog of 80 million tracks.” When Bob Dylan wrote ‘Blowin’ In The Wind’ lyrics “How many roads must a man walk down, before you call him a man?” he was singing about civil rights movements. How could a computer ever compute? 

There is also the legal question of what happens as we increasingly move towards a world where AI makes the music we hear on the radio, as an article in The Verge recently pointed out. Who is the author? The human that created the AI? The human who inputs the data? Or the AI itself? And what happens if AI randomly generates two pieces of music that sound too similar? Or if AI recreates existent music too closely? When you buy a song, are you buying the right to feed it into an AI as data? All of these difficult questions will need answering in the near future, as AI-generated music becomes more common. 

So, when it comes to AI compositions taking over human-created compositions: will the two ever compare? Will AI be able to write its own songs, that sweep us away emotionally, that create global dance trends, that become era-defining anthems? At least for the foreseeable, AI is not yet creating music without mimicking an existing data set that originated from human innovation. We’ll need to input tens of thousands of hours’ worth of music to truly get something original, but the possibility is there, says Yang: “I think there are unlimited possibilities long term,” she summarizes. “The more we incorporate AI systems the more they stimulate our own unlimited possibilities when it comes to music creation, too. It’s like a circle. An unlimited creative circle.” 

Read more about the amazing strides we can make when humans work with AI in harmony here