There is a passage in the final chapters of Scott McCloud's Reinventing Comics, in which McCloud suggests a future of media in which the ideas embodied in artistic works are no longer tied to a particular medium. His metaphor is one of various kinds of works - art, writing, movies, comics, etc. - escaping from eggshells, and instead living in amorphous combinations.
I have always found this metaphor interesting in application to games, because in the ludic context, the works have already escaped the medium, many, many times. Gaming did not start in the context of video games. We have board games, card games, outdoors games, party games, drinking games, etc.
In fact, even within video games, the format is changing with every game made! The only thing about gaming that has remained consistent is the involvement of players, and this is one of the reasons why video games are considered difficult to make - the format that worked for Space Invaders is not the same one that worked for Pac-Man, and the format that worked for Doom is not the same one that worked for Crysis, either. Every new game brings a new spin on the old problems, and old games that we point to as being "worth playing" tend to be the ones that haven't been superseded by a newer, similar format - which is often a matter of taste, but has quantifiable elements in the development of new mechanics, better control schemes, more refined presentation methods, etc.
I distinguish game format from game engine in that "engine" describes the underlying technology limits, but "format" describes the end-user experience of interface, mechanic conventions, and "standard" recurring elements of the game design - they're templates for the overall flow and feel of a game, but they don't describe the detailed design. id tech, Unreal, and Cryengine are all very different engines, but they all aim towards a similar format with subtle variations in the feel of the camera, collision, graphical effects, UI paradigms, etc. Moreover, formats aren't the same as genres, because even within the first-person viewpoint format, there are examples of different genres - action, role-playing, survival horror, etc.
An example of a format distinction would be the difference between Myst and Doom. The Myst format emphasized detailed environment interaction, while Doom focused on moving, turning, and shooting. You could do a game with Doom's shooting elements in the Myst format, and likewise a Myst-like game in the Doom format, but neither would necessarily play to the other format's strengths. The engine affects what formats are available, of course: a game done with a modern first-person engine could pull off both the Myst and Doom formats, provide higher fidelity for both, and do other formats too - for example, formats where you can drive vehicles with physically accurate motion.We know that when a game has a low budget or technical constraints, then the game tends to work best when focused in a very narrow fashion towards one end. Individuals have written great examples of the shmup, 2D platformer, puzzle, roguelike, and interactive fiction genres, and they have gradually come up with ways big and smal to extend the format of each of these genres - more complex simulations, variations on the interface, unique mechanics, etc.
A game with a high budget and cutting-edge technology, on the other hand, can attack many fronts simultaneously. This could mean a heavily scripted game with elaborate cutscenes and a Hollywood-type production, or it could mean complex simulations with many interlocking parts that capture the essence of a living, breathing world.
One point which I should clarify is that fidelity factors into format limits. Roguelikes and IF can be workable for individuals, despite involving heavyweight simulations or in-depth scripting, because they both sacrifice fidelity by resorting to a simplified world model and more dependence on written language vs. visuals. Giving a Roguelike game more fidelity results in something like Diablo; giving IF more fidelity brings us something more like a point+click adventure. Giving Doom less fidelity gives us DoomRL.
What I suspect is the underlying failing of a lot of games - couched in terms of format terminology - is a failure either to understand how to work within the format, or a failure in successfully developing the format. The latter case is the easier and more obvious one since we have plenty of terms to describe it; not enough time, not enough budget, mistakes in project management, crunch, over scope, etc. With those games, the player gets some sense, conscious or not, that things are unfinished and haven't recieved the attention they deserve. We know this problem well enough already.
The former case, of not understanding the format, is the more troublesome one as it is a bit vague. I have some theories, though. It seems to correlate with oft-heard complaints like "this game is just a clone" or "this game doesn't make sense," complaints that suggest that the game basically feels non-cohesive and doesn't combine its elements well. As well, it borders on a standard industry practice for designers to borrow the design of another game or set of games as a starting point for their game, and that in itself is great illustration of how we fail to understand what we're doing.
At first glance it seems like this borrowing happens all the time in other mediums without trouble - similar plots, characters, settings, etc. The problem is that when designs are copied in video games, it is done with methods roughly akin to copying an entire story word-for-word and substituting in new names for everything. What we want to copy is the format of the successful game, building upon it to make our own, better version, but we actually copy the entire design. We have trouble seeing the forest for the trees, so to speak, so we just take it all in, uncertain of what things made the original work successful, because there are so many concepts involved in a complete deconstruction of everything about any given game's design; many of the decisions made by professional designers come down to "game A tried that and it didn't work, but this other thing in game B seemed to work better, so we'll use that." Seeing anything beyond those simple comparison exercises is much, much harder, and progress is made only gradually, evolving as different genres exploit the various strengths and limitations of a format and the ideas cross-pollinate.
A game exists as an interplay of elements; adding and removing relatively small bits and pieces can change everything about how the game feels. This means different things at different levels of scope.
With small games, this means that the developer has to stay very sharp and consider every element equally; they can't afford to let one part of the game slip just because they like working on another more or lack skills for the others. Small developers that get sidetracked will end up in a sad situation where a considerable effort was made in one or two areas, but the game as a whole suffers from problems like: extremely opaque and difficult to play, unbalanced in all but a few respects, doesn't run on most people's systems, has lots of art assets without the code to make them interesting, etc. Small games have a lot of flexibility of format, but at that scale, it can be all too easy to push past the boundaries of what a single person can hold in their head and create an unplayable end result.
Big games face a different, but also terrifying challenge - a large team(which communicates more slowly) building up a complex, push-the-limits format(which is hard to understand) and also trying to make the best usage possible of that format(which requires time to iterate). There's no shortage of talent and thinking to go around, but it's hard to direct in a precise way.
It is no wonder, then, that high-budget games have a reputation of starting from tried-and-true basics, dumping on a laundry list of features, and applying lots of polish, while small games are known for taking huge gameplay risks with little graphical flair or pizazz.
A parallel thought on this is that the biggest games are the ones most likely to feel bulky and generic because of their multi-directional nature - they're advertised as having everything, so they have everything, and it doesn't all work out, but enough of it succeeds that people remain interested. The smaller games, on the other hand, have no buffer to fall back on if they fail basic playability tests, but when they succeed they can be "razor-sharp" or "white hot" in their concentrated power.
And I don't think any of this is a bad thing; it's just a truth that is easy to forget, since the big games get lots of attention and promotional power, while the smallest games scrabble to get any recognition at all. But neither one is really better - it's more a matter of where you feel comfortable as a developer and how big an audience you want to reach. I think it is always possible to build a powerful design so long as your format is within scope and you understand how it works.