Difficulty in video and computer game development has been on the outs for some time. If a player possibly cannot complete Action Adventure Game X, so the thinking goes, the designer has failed. Most MMORPGs feel like they can be played on autopilot; in fact many can, a fact dear to some players' hearts. When Ninja Gaiden came out and turned out to be a stiff challenge some players were up in arms. That ever-so-quotable Tomonobu Itagaki said that hearing player cries made him want to make the game even harder.
Ahem. Setting aside what we might think of his statement, certainly, difficulty in a video game must be handled carefully. Nothing attracts the ire of those fickle game bloggers quite like them getting their asses handed to them by a game. And it is possible to make a game arbitrarily difficult without too much effort. Increase monster attack strength, decrease player health, remove resources and/or add more foes -- these are just the most obvious ways a game can be made harder, and it doesn't take any great skill to add them.
But this is not to say that games must be easy. The impulse to make video games easier can be traced to a fundamental change in perception over what a game should be. The older school of thought, which dates back and beyond the days of Space Invaders to the era of pinball, is that a game should measure the player's skill. Arcade games, in fact, must make it difficult for a player to last for any great length of time in order to keep money coming into the coin box. The newer concept is that a game should provide an experience to the player. The player is to feel like some character, or like he's participating in a story, or that he's making some difference in a fictional realm.
Tecmo's NES classic Ninja Gaiden
The difference can be seen in Super Mario Bros., but may date to before it. Super Mario Bros. doesn't seem like it's entirely convinced either; many of the later levels are quite hard. The discarding of score, the emphasis on "suspension of disbelief" and storytelling, the trend towards providing a narrative longer than "see those guys; kill them," and the idea that a game can be "finished," these are all substantial indications of the change. A game intended only to measure skill, particularly, shouldn't have a completion state, because it would be like the game has given up.
Because of this, many of the games on this list are older ones. There are some exceptions, and those may be the most instructive examples, games that wish to tell a story but are willing to take the risk that the player won't be able to see it through to its completion. Sometimes the difficulty lies in achieving a "super goal", which is kind of a way to have both things at once, an experience and a testing of skill.
Before I start, let's define that. "Super goal" is a term I use for a special goal in a game for supremely skilled play. Atari Games used them in many of their late 80s/early 90s arcade games for contests. A player could, for example, collect letters spelling TOOBIN' in that game to receive directions to follow for obtaining a free T-Shirt. Super goals are distinguished from ordinary goals in that they are an objective beyond winning the game, and that many players are expected to never achieve them.
Take note: although some of the hardest games ever made are on this list, it is by no means a list of the absolute hardest games. That’s why it includes Mischief Makers before Alien Storm; not because the latter game isn’t harder, but because Mischief Makers’ difficulty is particularly instructive. Of course, both of them pale before the terrible majesty of Sinistar.