Interview with inventor and programmer Ben Cole, co-founder of ‘Mashup Machine’
Technology is getting smarter, largely due to innovations in systems such as augmented reality, data gathering, and chat bots. A startup company known as MashUp Machine is now creating a platform that enables a creative system to get “smarter” by interacting with users. Namely, MashUp Machine was founded to see if an artificially intelligent computer named “Mashi” can learn how to write an entertaining movie.
The program works like this: a human user interacts with Mashi via a chat-based messaging system. Mashi asks the person about what should happen next in a spoofy screenplay he is writing–titled “Scary Cabin” which is set in the woods with a series of stock characters–and then the person can select an option he offers (similar to a Choose-your-own-adventure styled game) or suggest what should happen based on a series of keywords. As the scenes are developed, an animated storyboard pops up until the segment is complete and the human can play and watch the “mini movie” they have made with Mashi.
Making stories with Mashi is fun and it helps teach the AI system about what stories work. Since users can share their finished movie/segment on social media, the computer can track which versions have the most “likes” and views to determine which sequences of events lead to the greatest success.
“Scary Cabin” is a work in progress and so the completed “movies” are currently only animated vignettes but the interface is constantly being improved and updated. Ultimately, this kind of technology could have a profound impact on how Entertainment is both consumed and created with many opportunities for customization and interactivity.
MashUp Machine was created by a software developer and Academy Award winning creative-technologist named Ben Cole and Christopher Sjoholm, previously a video game creative director at Activision and Electronic Arts and VFX supervisor at Rhythm and Hues, the Academy Award winning visual effects company. Both men are very interested in the gaming industry and its impact on technology, culture, and society.
Co-creator Ben Cole recently granted an exclusive interview where he discussed Mashup Machine, the potential for its brand of technology, and more.
Meagan Meehan (MM) of Entertainment Vine: What initially inspired you to seek out a career as a software developer?
Ben Cole (BC): I’m not sure that I so much sought out that career, as it sought me out. I’ve always had computers around and programming was definitely one of my main hobbies by the time I was six or seven years old. My dad used computers to do economic modelling and several of my other family members have been innovators in the computing space, so I’ve had computers around pretty much my whole life. From very young, I was fascinated with how computers work and in the idea that a computer program might be able to be intelligent – to think for itself and even “feel” or have an “emotional” response. Right from the start, I wanted to create programs that you could have a conversation with and more specifically share a creative experience with. It was pretty much a foregone conclusion that I’d end up working with computers!
MM: How did you find your way into the entertainment industry and what aspects of movie-making most interest you?
BC: During my teenage years, I started playing around with computer graphics, largely because I wanted to make video games. One of my high-school friends and I spent time tinkering around with writing our own 3D engines and things like that.
However, my first jobs had nothing to do with that. I graduated from high school a couple of years early and I started working professionally very young. Before going to university, I did an internship at Meiko Scientific, an early massively-parallel super-computing company. They were looking at scalable database systems and I started working on software to help businesses build workflow solutions. This led to a number of contract software jobs building databases for banks and insurance companies.
While the workflow project was fascinating, insurance wasn’t really my passion. I ended up going to university to study artificial intelligence. While I was there, I had an idea for a social, story-based video game and started looking around for a games company that might want to publish it. What happened instead was that someone asked me move to the South of France to work on an animated TV show. I ended up working on some early facial motion capture software – the same kind of technology that has been used recently to great effect on movies like “Planet of the Apes” and this led to a job in a London post-production studio developing software for motion control rigs. Motion control rigs are giant robots with cameras on them. You can do some really cool things with them, so I started looking at how to help people control them through 3D simulation.
Ultimately, in visual effects, I’ve been drawn mostly to challenges that involve simulation and movement: motion capture, motion-control rigs, crowd-simulation, physics simulation. Part of this for me is a fascination with how fairly simple mathematical systems can scale to make complex and beautiful patterns. Another side of it is figuring out how to tame these complex systems, so you can give artists (and others) the power to control them and make the things that they (or the audiences and movie directors) want to see.
But, what really excites me about movies are stories. Without a story to tell and characters to care about, it doesn’t matter if you have the greatest visuals in the world.
MM: You actually won an academy award for your skills, so what was the experience of attending the ceremony, winning, and holding the golden statue like?
BC: It’s a little-known fact that the Academy of Motion Picture Arts and Sciences actually has two ceremonies each year. One if the televised “Arts” one. The other one happens a week or so earlier, focused on technical and scientific innovation. You usually get to see glimpses of this ceremony during the televised awards.
The sci-tech Oscars, as they are known, work slightly differently to the film awards. Firstly, you get to know ahead of time that you going to be an award recipient, so you get to relax on the night and enjoy the celebration. Another difference is that they only give out statues for the lifetime achievement awards.
My award was what is known as a Technical Achievement Award. I received it, along with two other people, for helping to introduce a particular mathematical technique called finite element methods (FEM) to “set a new bar” for digital on-screen destruction. It was a lot of fun to be there!
Regarding golden statues, I have actually had the chance to hold one, although not at the Oscar ceremony. MPC (the company I worked for before starting MashUp Machine) won one for the VFX work we did on “Life of Pi.” One of the first things that went through my head when I picked it up was, “Wow, this thing is pretty heavy! I hope I don’t drop it.”
MM: How did you come up with the concept for Mashup Machine and why did you decide to put a horror movie at its center?
BC: I’ve always wanted to create something like MashUp Machine. I’ve spent much of my career using technology to augment storytelling and entertainment and I’ve always been fascinated by the profound possibilities that AI and machine learning can open up. MashUp Machine is an opportunity to combine my passions for storytelling and technology and apply them directly to the stories themselves.
Looking at the trends in AI and machine learning, I felt that the timing was right to create a system that made use of computational creativity. We are witnessing something of a renaissance in AI. Recent advances in areas such as machine translation, image recognition, and others, suggested to me that, if applied correctly, similar techniques could be used to help people to make their own animated movies. The current state of the art in machine learning, such as deep-learning, relies on large datasets. When it comes to storytelling, those datasets don’t really exist. There are, however, passionate groups of people out there who want to create them, in the form of fan communities.
I founded MashUp Machine with Christopher Sjöholm, another entertainment industry veteran, who has a long career history as a creative director in the video game industry, working on titles such as “FIFA” and “Fight Night,” as well as a background in visual effects production. Chris and I had noticed the radical shift in the way audiences are engaging with content. They are no longer happy to be passive viewers and are constantly looking for new ways to participate. Fanfiction in particular caught our attention – passionate fans creating a high volume of story content for other fans.
We wondered what would happen if we teamed up a machine learning AI with a community of fans, so that the fans could teach the AI how the fandom works and in return the AI could help the fans with their storytelling. We called our AI “Mashi”.
The largest fandoms in the world revolve around franchises such as “Harry Potter” and “Star Wars” so we are excited to see what fans of that content would create with Mashi. But, because what we are doing is new, we wanted to develop a proof-of-concept first. So, we’ve been building “Scary Cabin.”
“Scary Cabin” is a spoofy, cabin-in-the-woods style horror cartoon, where five teenage kids (a jock, a stoner, a cheerleader, etc.) hang out at a cabin, being chased around by a guy in a mask. It’s not exactly a fandom, but it’s a genre that many people understand. As a genre, horror has many characteristics that make it great for Mashi to learn. There is a small and well-known cast of characters, trapped in a small set of locations, and acting out a set of tropes and rules that most audience members know from watching movies such as “Scream,” “Cabin-in-the-Woods,” “Scary Movie,” “Friday 13th,” etc.
It made sense to us to create animated content, partly because of our backgrounds in film and gaming, but also because we knew this would give us a way to add even more structure to the stories and keep the creation experience as easy and accessible as possible.
MM: How exactly does this technology get “smarter” via the database and how have you seen it improve in the months since the launch?
BC: Mashi gets “smarter” in different ways. Each interaction with a community member and each story created contains a wealth of data to help Mashi learn. This data might be suggestions about what should happen next in a story, which of the characters it should happen to or how they interact with each other. It might also be information about the best camera angles for a shot or the dialogue that a character should say in a given situation.
As we acquire more data we are also able to apply increasingly sophisticated and more data hungry algorithms which allow Mashi to make more nuanced suggestions. So, we are able to use the data we have collected in ever evolving ways. Data-Collection, learning and teaching are not passive processes and there’s a feedback loop between them. We are continuously adding new ways of interacting with Mashi and looking at ways to help Mashi learn and help community members to create the richest stories.
Right now, a lot of our focus is on data collection and assisting community members. Mashi’s storytelling abilities are in their infancy and we are only scratching the surface of the data we’ve been collecting. Mashi would be the first to admit that it’s not the world’s best storyteller… yet.
MM: At present, what are the limitations of the program and how do you think they can be overcome?
BC: “Scary Cabin” is just a proof-of-concept, a constantly evolving work-in-progress. There are many opportunities to expand Mashi’s capabilities and also the storytelling possibilities in the script. Today, most of the lowest-level concepts Mashi knows were taught to it by the MashUp Machine team. Community members have been responsible for teaching it how to use them. We keep adding new concepts but we are looking at ways to make it easy for community members to add concepts of their own, so Mashi can acquire concepts more quickly and we can keep increasing the range of stories that can be told.
Aside from Mashi’s capabilities, we have deliberately started by making the stories visually simple, but everyone in the MashUp Machine team has worked on content with really high production values and we are excited about increasing the overall quality of the animation, visuals and sound.
MM: You started a website where users can interact with the programs and make their own films, or little segments, and then share them. So, how many people have used the interface and how many different paths can the story take?
BC: “Scary Cabin” is still in alpha so we haven’t distributed it all that widely. So far, around 10,000 people have created stories with Mashi. Today, Mashi knows over a hundred concepts about the horror genre and it’s learning more all the time. These concepts can be applied in a vast number of combinations, with different characters and dialogue modifications, so, even with minute long movies, the total number of possible story paths is really large.
MM: Overall, what has the response been like regarding this program?
BC: The response has been really positive. People tell us that they have fun chatting and creating with Mashi. It’s great to see people interacting with Mashi. We’ve tried to make the experience fun and humorous and people definitely appreciate this. It’s exciting to see that people really do enjoy the collaborative experience and find Mashi’s suggestions helpful. People tell us that it feels like a collaboration, and a genuinely new kind of experience.
MM: How do you want Mashup Machine to evolve over the coming decade and is there anything else that you would like to discuss?
BC: AI and machine learning is going to have a profound and transformative impact on many parts of our lives. Entertainment is no exception. As the algorithms get smarter, I believe that the platform we are creating has the potential to radically reshape the way people engage with content. The opportunity exists to offer people truly personal storytelling experiences and to empower people to be more creative and expressive.
I believe that fandoms are going to become increasingly important to the media landscape and that we are going to see more and more non-traditional content in the mainstream and also a more fractured mainstream. This trend is already well underway with YouTube and Snapchat.
Today, even fans of traditional content are crying out for ways to engage with it on their own terms, across a range of different media. For storytelling, traditional fanfiction is just the start. Even within these communities, the barriers to entry for content creation remain too high, while at the same time it can be difficult to control quality. MashUp Machine offers a way for more of the fans to create their own content, while also offering a way to keep the quality high. As the algorithms learn, they can help people to create content at an ever-increasing level.
This is really just touching the surface. Longer term, by ushering in an era of truly adaptive content, AI and machine learning have the potential to unlock a range of new experiences that we’re just starting to imagine. I’m excited by what we’re doing now, but thinking about these new experiences blows my mind!
* * * * *
To learn more about “Scary Cabin” and the Mashup Machine, see here and here. They can also be found on Facebook here. Or follow them on Twitter and Instagram via @scarycabin. To watch a video pertaining to the technology, see here.
