Gamasutra: The Art & Business of Making Gamesspacer
arrowPress Releases
October 31, 2014
PR Newswire
View All
View All     Submit Event

If you enjoy reading this site, you might also want to check out these UBM Tech sites:

The Artistic Computer?
by David Maletz on 10/26/12 04:40:00 pm   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.


Ever since college, I've been fascinated with machine learning, specifically, teaching computers to create art, music, and potentially even entire games. Is it possible for a computer to do what we do - be creative? Could an advanced statistical learning model, with enough data, produce art, music, stories, or even design gameplay and write code? If you're worried about machines becoming independent of humans and taking over the world, I can assure you that we're a long way away from that. However, every now and then I come back to this idea, thinking about how cool it would be if I could create a game that generated itself from an "intelligent" computer program. Once I can teach a computer to generate at least one game-element satisfactorily, I plan to build a full game around that ability.

This blog post is about my most recent attempt to teach computers to be creative. The idea was to create a program that would draw graphics by learning the probability distribution of a color given the pixel above it, to the left of it, and the pixel in the previous mipmap level of the input data (graphics made by humans). Creating a new graphic then involves drawing from the distributions from lowest detail mipmap to the highest, left to right, and top to bottom, ensuring that we always have the inputs for the distributions while generating. I liked the idea, because it mimicked what artists do - first creating the basic shape of an object, and then adding in details. Similarly, the program progresses from low detail to high detail with each mipmap level.

The Data

The first 493 pokemon used as input data for the program.

I wanted to generate something fairly retro, and I needed a lot of consistent sprites to teach the model. So, I chose to use sprites from Pokemon as my data set as they were easy to find and there were a lot of them. The question of whether a program taught using pokemon sprites generates copyrighted material or not is tricky, but this was just for fun/research. Since the model depends on the number of colors in the image (the more colors, the harder it is to learn and the more data it needs), I decided to convert the image to just four grayscale colors. I also downsized the pokemon to 32x32 pixels, all to make the data smaller and easier to learn from.

The Results

Ideally the program would be able to learn how to create infinite reasonable looking original pokemon (imagine catching them all!). Instead, the program generated what I've termed "inkblots," some of which don't look anything at all like pokemon, some of which have interesting pokemon-ish shapes if you look for them, and some of which are actually decent (if you squint a little). The quality of the results is completely subjective, so I'll let you be the judge:

36 randomly generated results. Didn't quite have the coherence and structure I had hoped, but not outright terrible either.

You can also test the program yourself on my blog post here:

Older Experiments

Back in college, I attempted to teach a computer how to compose music by feeding midi data to a Hierarchical Hidden Markov Model. The results for that were not that great, but I still have one midi file generated by it that I thought sounded nice (from a set of anime midi files):, and here's a mp3 version for those who have trouble playing midi files: However, this was one result from many that were less coherent, so take it with a grain of salt.


I still haven't made a program that can teach computers to make games, or even acceptable pixel art yet, but that doesn't mean it won't be possible one day. For now, the results are interesting given that this was just a short project, so I figured I would share. Feel free to share any experiments you have done with machine learning and games. Until my next experiment!

Related Jobs

Activision Publishing
Activision Publishing — Santa Monica, California, United States

Tools Programmer-Central Team
Amazon — Seattle, Washington, United States

Sr. Software Development Engineer - Game Publishing
Intel — Folsom, California, United States

Senior Graphics Software Engineer
Pocket Gems
Pocket Gems — San Francisco, California, United States

Software Engineer - Mobile, Backend & Tools


Dimitri Reinhart
profile image
Well to be honest the music sample is quite good, and your new pokemons are... well interesting XD.

What I'd like to see is generating fully physical walking animations for npcs in 3D games, by using an genetic algorithm. That would be extremely classy to see!

David Maletz
profile image
Keep in mind that the music sample is the only one I kept from the project, there were many others generated! That's like me picking the best pokemon from this project and saying: this is the result!

As for walking animations, check out Nick Harris' link, haha. Although, personally I'd prefer generating a 3D model, and then using motion capture data for the animations.

Harold Myles
profile image
I too liked the music sample.

But music is kinda cheating. As long as you keep a beat and have some sort of repetition your brain will be doing most of the work to make it sound like music.

Same with the 'inkblots.' Kinda like starring at clouds. Depending on your individual imagination it skews how good they truly look.

Not trying to be down on the article, I actually like it a lot. Anything like this is interesting. I've been interested in similar things but more on the lines of creating a narrative.

Like when you play a game and then tell a friend a story about something that happened in the game. Despite modern 'games' being so story driven, they are never as good as the stories you make yourself and tell your friends. We need an algorithm for detecting these stories and re-telling them.

David Maletz
profile image
I didn't cheat with the music anymore than I did with the art - the program had no knowledge of musical theory or beats. All it knew was midi data: note start, note end, note pitch, note volume. Any semblance of beat the music has, it learned from midi songs. If you look at the midi file, it's actually really messy with many mistakes, but it's an "ink blot" just like the art - it hasn't perfectly learned the structure, but it has learned some of the structure of art/music, and we're able to pick up on that if we want to - and I find that pretty awesome.

As for creating a narrative, I've had a few ideas about that :P . I've always thought it would be cool to have an artificial DM controlling the story and adapting it based on what the players want to do. For machine learning, we need two things: A discretization of the data, and sample data to learn from. One possible discretization of a narrative is breaking it down into blocks (where each block is say, a small quest or event), and then have the program learn how to stitch these blocks together into a full story, and then it could change what blocks it selects dynamically based on what the player does and still be moving towards a (perhaps different) ending. Although depending on the size and flexibility of these blocks, it may not be able to create every kind of story, but having a program learn with individual words as blocks would be very difficult to learn from haha.

Nick Harris
profile image
...prepare for your jaw to hit the floor...

'Simulation of Evolution by Natural Selection'

'Polyworld: Using Evolution to Design Artificial Intelligence'

'Daniel Dennett - Is Evolution an Algorithmic Process? Part 4'

Here is Karl Sims' research:

David Maletz
profile image
I actually saw some of these back when I was in college, pretty cool stuff. My favorites are the ones where two of the creatures are battling over the green block! Definitely some cool research - I wonder if he's expanded on the idea since 1994?

Nick Harris
profile image
From what I can make out this is Intelligent Design not Natural Selection, because Karl Sims played the role of God and intervened to select which mutations looked the most promising. When these later evolved to a point where they could compete to grab something he missed a trick to reinstate pure Darwinism and reward successful competitors with continued survival in their ecosystem by making them 'fight for food'.

A system based on this foundation would run without further human intervention and evolve ever more sophisticated artificial lifeforms. There is a striking similarity to Pokemon here.

David Maletz
profile image
To really let evolution play out in a simulation without human intervention, you need a very complex ecosystem and environment simulated. Even the blocks that Karl Sims used were designed, not evolved from single-celled organisms (making it tough to even get them to fight for food at first). The amount of data and computational complexity to do real evolution is pretty high, especially for computers back in 1994. There's some interesting research on simulating bacteria evolution, but no macro-organisms without human intervention that I know of yet.

Stephen Chin
profile image
" Could an advanced statistical learning model, with enough data, produce art, music, stories, or even design gameplay and write code? "

At least for the 'design gameplay' part of the question and given the constraint of designing gameplay for an existing model of gameplay, the answer is unequivocally yes. I say this because that's exactly what Cloudberry Kingdom does.

Now if the question is loosened up to design gameplay for a game not already designed eg innovate a mechanic or create a mechanic without context, that's a slightly different question I think.

Stephen Chin
profile image
Also, I would be interested in seeing what came out of your Pokemon sim given higher resolution images and/or more colors. By reducing the resolution and color, you're not just making it easier but you're losing information altogether.

David Maletz
profile image
The less parameters the program has to learn, the easier it is for it. Learning a few parameters from a rigid model of gameplay is certainly a lot easier than learning how to create entirely new forms of gameplay. Likewise, with more colors and higher resolution, it is harder for it to learn. The results for higher resolution/more color were pretty bad, which is why I didn't show it. But, you asked so here is the program w/colors: , and higher resolution: .

Jake Shapiro
profile image
Have you played/read Ian Bogost's "game poems" collection, A Slow Year? It features a thousand "machined haiku." The whole thing is essentially about what you write about here.