This article was originally posted on Kongregate's Developer Blog.
Early marketing testing allows you to gain insights into how well your game's creative performs during player acquisition. There are two main reasons to run an art test:
It’s important to note that if you aren’t willing to change anything, it’s not worth running a test!
Narrow down your art test to 1 goal. You can run multiple tests if you wish to learn multiple things, but never try to cram a bunch of items into one test or you are likely to invalidate your results. There are several different things you can test; I’ve focused on the main four below.
What theme works best for my art style?
When testing a theme, try to keep everything else in the picture as close as possible. Notice how each image below has 4 characters in a similar grouping. It’s also worth noting that your Click Through Rates (CTR) can be greatly influenced by current pop culture. For example, the release of a new Marvel movie might make people more inclined to click on superhero art. Unfortunately, what’s popular one month might not be so interesting next month.
Which art style works best for my theme?
There are many different styles of art: 3D, pixel, low poly and cartoon, to name a few. The game's art style tells the users a lot about the game. When someone sees Dora the Explorer for the first time, no one needs to tell them it’s a kids' show; they can tell by the art that the show is intended for children. It’s important to think about your audience when considering which art styles to test. Here are a few resources specifically on Art Style if you want to learn more: Creating an Art Style and The Aesthetics of Game Art and Game Design.
Which color palette resonates best with my game?
You can test a color palette, but most of the time it is determined based on the emotions your game is trying to evoke and the intended audience for your game.These guides are pretty interesting if you are trying to figure out your color palette: Color in Video Games: How to Choose a Palette and Picking a Color Palette for Your Game's Artwork.
How marketable is my game art?
This is probably one of the biggest questions we get when it comes to game art. As a general rule, the more casual the game looks the cheaper it is to acquire users for. That said, even within the 3 main categories (casual, mid core & hard core) there is still a huge range! CPI testing is much better than CTR testing to determine marketability.
Is the game built yet? Yes: If there is already a game (even a very basic one) you can get Cost Per Installs (CPI), along with Click Through Rate (CTR) & Conversion Rate (CVR). No: If there is no game you can only get CTR.
CTR Test: CTR lets you gauge general interest in your game.
CPI Test: CPI gives you a deeper read into how marketable a game is. You can gauge general interest (CTR) as well as how likely people are to actually install (CVR) and most importantly, how much it will cost you to acquire users (CPI).
The most important part of the test is the range of confidence for your results! The smaller the difference is between art assets, the more users you will need to resolve the test with confidence.
For example: If you run two pieces of art that have CTRs of 1.82% vs. 2.18%, you need over 96,200 impressions to know for sure which is better. Whereas, if their CTRs are 1.70% vs. 2.30%, you only need about 35,000 impressions. Use this simple Statistical Significance Calculator or something similar to find your range.
One word: targeting! You can target both age and gender to be sure your art is appealing to your target audience. If you run a test without targeting and get great CTRs on some super-kiddy-looking art, but the game is meant to be for males 18-35, then your test results aren’t very helpful. That said, if your game is intended to be for everyone, then you can test it on any network.
How do I know if the results are good or bad?
Unfortunately, there is no “magic” number we can give you as a comp. The market is constantly changing. That’s why we recommend using a baseline. Your baseline is an image you have used for another game that you can test alongside your new art to help provide perspective on your results.
Heather is a Marketing Analyst at Kongregate, where she manages performance marketing and promotions across numerous mobile and web games. She loves playing board games, D&D and MTG with her husband and friends. Her favorite holiday is Halloween.