Gamasutra: The Art & Business of Making Gamesspacer
arrowPress Releases
July 25, 2014
PR Newswire
View All
View All     Submit Event





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
How the IndieCade jurying process works
by Kris Ligman on 10/11/13 08:01:00 pm   Editor Blog   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

Kris Ligman is a news editor for Gamasutra.

With IndieCade's annual festival in Culver City well and truly wrapped up for the year, I thought I would take this opportunity to shed a bit of light on the selection process for the festival. As a two-time juror and one of this year's "super jurors," I've seen a side to the decision-making process many have not.

Whether you are a developer who has not yet applied or a former applicant who has come away confused about how IndieCade selections are chosen, I hope this can provide some insight into why IndieCade selects the games that it does.

Juror criteria

There is not much of a formal process to becoming a juror, though we're always looking for new ones. The best way to get involved is to get in touch with one of the existing jurors or organizers, or come out to one of our special juror sessions held over the summer in Los Angeles.

Jurors are invited based on their knowledge of games. This can mean experience in development, a background in game studies academia, work as a game critic or journalist, and the like. Some applicants might be a bit confused why IndieCade would bring in journalists and the like to participate in jurying, but the idea is to diversify evaluations as much as possible.

In my case, I've dabbled in all of those aforementioned areas: I curate Critical Distance, whose This Week in Video Game Criticism feature appears weekly here on Gamasutra; I studied games throughout college and graduate school; and prior to becoming Gamasutra's newest news editor, I served as a community lead for a kids' free-to-play game. So while I don't have the specialized area of expertise of a triple-A veteran, I'm sufficiently well-rounded that I can talk about games from a number of different angles.

This is useful for IndieCade, since we want to compare applicants to each other as little as possible, while also understanding each applicant in the context of its genre or cultural backdrop.

Also, it's worth mentioning that jurors are unpaid volunteers. Super jurors get a small stipend in exchange for the higher volume of work they perform, which I'll get to below.

Super jurors and first wave evaluations

Super jurors (yes, that is what we're called officially) were first mentioned at this year's Red Carpet Awards, where some of our feedback notes to our 36 finalists were read aloud on stage in the course of the ceremony. In truth, we had been at work all summer alongside regular jurors, as we actually participate as part of the first wave of evaluations all IndieCade submissions undergo.

IndieCade received some 800-odd submissions this year. Provided the files submitted work, those are then assigned to jurors and super jurors. Assuming that I'm anywhere near representative of an average, super jurors review more than twice the number of games assigned to other jurors.

Super jurors are chosen based on the depth and breadth of the feedback we've written to past applicants, on the assumption we'll continue to provide in-depth and potentially constructive responses on every new game we're assigned. In exchange for our efforts, our votes receive slightly extra weight than that of the general juror pool. A super juror's feedback can't determine on its own whether an entry will be accepted as a selection, but it can help the decision process if other jurors are split about a game.

In theory, every submission will be assigned to at least two jurors, although with an almost limitless amount of possible hardware and operating system configurations, it can be difficult to ensure every game receives its due consideration. We have a number of remedies for this, including asking applicants for setup instructions with their submissions and holding the special juror sessions I mentioned before, which has the benefit of herding a lot of jurors into the same room to interact with a game under identical specifications. This is doubly important for multiplayer and "big games" (board games, pervasive games, etc) which are nigh on impossible to set up for solo play.

After playing a game (ideally to completion, or at least long enough to get a good feel for the mechanics), jurors fill out review forms on our online system. I should note, in particular, that even though it's the most visible part of the feedback applicants receive from us, the written response section is just one component of our evaluations. Much of jurying is done through scoring on some undeniably subjective (yet still valuable) criteria, such as world design, innovation and even cultural significance -- things which are difficult to articulate and even harder to rank, and also don't necessarily add up to a "yes" or "no" vote.

From Ienién:te and the Peacemaker's Wampum.

An example: Ienién:te

One of the games I evaluated this year was Ienién:te and the Peacemaker's Wampum, a student game facilitated by Aboriginal Territories in Cyberspace (AbTeC). This is an organization that is all about the sharing of native experiences through new media like video games, something that is sorely underexplored in mainstream media.

Unfortunately, Ienién:te (pronounced yeh-YAWN-day) was not accepted into the festival. This was a shame for me personally, as I saw a ton of potential in the game -- the titular protagonist is sort of an anti-colonial Lara Croft, an archaeologist returning stolen artifacts to her people -- but if you play the game for yourself, you can see why it may not have worked so well in festival exhibition. It is a culturally significant game, for sure, but in a loud public space full of flash and spectacle, it wouldn't garner the attention it deserves. It is also (I'm sorry to say) a bit rough to play, and though polish isn't always a determining factor in IndieCade selections, you can see how in this case, many players would find it a resistant experience.

There are other examples I could name. I evaluated 43 submissions for this year's IndieCade, of which I would say I recommended about a quarter for either further consideration or inclusion at festival. Of those 43, 12 were selected for festival, and five of those 12 were shortlisted for awards. There are a number of games not among that five that I would have thought were just as worthy of Red Carpet Award consideration, but I don't want to turn this into a complaint session. I bring up Ienién:te just to illustrate that a game can be many compelling things -- original, hand-made, funny -- and still not make it past the first wave of evaluations, for a variety of factors.

That is why IndieCade looks for multiple jurors from diverse backgrounds to do these evaluations -- and also why jurors are just the first of multiple rounds of decision-making. A separate group is responsible for finalizing the selections based on juror recommendations; yet another group votes on the Red Carpet Awards. The Closing Awards are determined by community vote.

To be sure, it's not a perfect system. I'm not in a position to propose improvements, really, although I'm sure I speak on behalf of the organizers when I say we're always looking for more jurors. Inasmuch as evaluations of games can never be scientific, the more input a game receives the greater the likelihood that a concensus will emerge. At any rate, it will mean more and hopefully productive feedback for applicants as well.

What not to do

My advice is as subjective as any other super juror's and it is not the final say in any IndieCade decision. Nevertheless, all jurors do have a few guiding principles when we set out to evaluate submissions -- and a lot of it is common sense.

Don't send us your clones. No one likes a clone, even a pretty one. You would think that it wouldn't be quite so dire but I have seen enough physics platformers this year to last me well into my old age. I don't need a single one more. You should absolutely feel encouraged to work within an established genre if that is what you like to do, but please bring something of your own to it. One of this year's selections, Valdis Story: Abyssal City, is a great example of working within the Metroidvania mold while fleshing out an iconic art style and setting.

Seriously, don't send us your clones. I can't believe this needs to be repeated, but it does. And if you absolutely can't resist sending us your cloned or derivative game, for Sid Meier's sake don't mention what it's a derivative of in your application.

Don't send us an instruction manual that's thicker than the rest of your game materials combined. If you can't explain your game simply and elegantly to more-or-less trained professionals, it's not going to be pick-up-and-play material for regular Joe attendees and their youngsters.

It is fine that your game's narrative layer is thin, but don't make it superficial. Few jurors are going to be up for reading oodles of backstory, but your game's narrative layer should at least scale to the game you're trying to create. A dense introduction to explain why your MOBA units have different colors is not necessary and sets up the wrong expectations. Likewise, a lot of exploratory games could benefit from a bit of contextualization.

Don't send us a broken build. Somehow this worked out for Videoball, but in general I wouldn't recommend it.

Videoball, as seen not broken.

Comment is free

There's a lot I haven't addressed here that you should feel free to ask about in the comments. I can't speak for IndieCade outside of my capacity as a super juror, obviously, but inside of that role I am happy to answer questions or concerns about the evaluation process or elaborate on the feedback I provided.

Otherwise, if you're interested in getting involved as a juror yourself, I'd highly recommend it! Feel free to bother me any time on Twitter or contact the IndieCade organizers at volunteer@indiecade.com.


Related Jobs

Galxyz Inc.
Galxyz Inc. — Mountain View, California, United States
[07.24.14]

Narrative Writer for Interactive Media
American Girl
American Girl — Middleton, Wisconsin, United States
[07.24.14]

Game Developer
InnoGames GmbH
InnoGames GmbH — Hamburg, Germany
[07.24.14]

Quest Writer (m/f) for The West
InnoGames GmbH
InnoGames GmbH — Hamburg, Germany
[07.24.14]

Game Designer Tribal Wars (m/f)






Comments


Matt Hackett
profile image
> I don't understand why you have entered something like this into Indicade. [sic]

That's the feedback we got from one of your jurors when we entered. Not exactly welcoming…

Kris Ligman
profile image
I'm very sorry that happened to you. It definitely isn't the sort of remark a super juror would make, but no juror should be making it, frankly. That's not what I signed up to do, and that's not what most jurors sign up for.

If you like I can get in touch with the organizers to find who left that comment and hopefully leave some feedback for THEM about why that isn't appropriate.

Simon Ferrari
profile image
In the interest of greater transparency, the broken build worked for Videoball because I hand-picked it for my tiny, inaugural indie eSports showcase. I'd been following the development of the game for months. Part of my curation process was talking directly to developers about what they were doing to improve their builds/designs in the weeks leading up to the conference.

I don't know if I'll be involved in the eSports showcase next year (I don't know if it will exist, either), but live-streaming the development/gameplay of a multiplayer game on sites like Twitch.tv is a pretty good way to prove that your competitive design is tight/deep/will play well to an audience at a festival.

It should be noted that games selected through special curation processes like the eSports showcase are not eligible for the main competition awards. But they are eligible for audience/critic/developer prizes! One of my selections, Slash Dash, won the audience award this year.

Kris Ligman
profile image
Also in the interest of fairness, even a broken build of Videoball was incredibly fun to play...

David Schoneveld
profile image
We had a remake/update of a 90's game, but didn't get any response. Not to say we didn't make it or any sort of communication. Only that a juror has played our game.

Kris Ligman
profile image
To clarify: you received notice that a juror had been assigned to your game, or had submitted a review of your game?

David Schoneveld
profile image
the only messages I ever received from indiecade was 4
"I have been assigned as a juror for one of your games!" nothing else. Which I should add was frustrating b/c it doesn't say they played it. We really have no idea if it was seen at all.

Kris Ligman
profile image
That's really discouraging! I'm sorry that happened. And you received no sort of contact from the festival about your submission's status?

If you like, I can follow up with the organizers to try to figure out what happened. Send me an email with the name of your game and I'll look into it.

David Schoneveld
profile image
thats kind of you to offer, but at this point what can they say. Our game wasn't what they were looking for thats fine. I'd just hope for future submission candidates they'd do 2 things.

1) confirm its been played, even a auto email like the jury assigned email.
2) tell us some sort of result positive or negative.

I read news articles about who the finalists were and wondered if that meant we might still be in some other way? or are those the only games in the festival? it wasn't clear at all.

Michael Rubin
profile image
Thanks, this is good info, particularly when reviewer feedback is sparse. I did get one brief reviewer comment this year, which basically said, "Very impressed with the look, sound and feel of this game...Very compelling." That was encouraging, but without any other feedback, it's tough to know where it fell short. I'm sure a lot of folks would be interested to know if their game even made the first cut or not.

Rob Steffen
profile image
How many submissions does a typical juror go through? Typical superjuror?

Kris Ligman
profile image
I don't have the stats ready-to-hand for averages among jurors or super jurors, as those numbers can vary wildly. Jurors are asked to specify the top number of games they want to be assigned at any time. I gave myself no maximum, so I suspect I'm on the higher end of the curve in terms of productivity, in both the years I've done it.

Assuming I'm close to average, though, as a regular juror I evaluated 15-20 games, while as a super juror I was assigned 44, of which I filed 43 (and the remaining one I abstained from for professional reasons).


none
 
Comment: