Allow me to present a concept for a fantasy combat-focused MMO as a model to discuss.
Let's take another look at the great grandfather of class-based fantasy games, Dungeons & Dragons. Early D&D had wide selection of classes with different abilities. Different classes had different specialties and different focuses in combat abilities.
Classes that wore heavy armor, such as Fighter classes and Clerics, could survive on the front lines of battle because they could avoid hits easier. Fighter classes could also do tremendous damage with heavy weapons against most opponents.
Rogues preferred more subtlety and stealth, with their abilities focused on sneaking past enemies and striking them while vulnerable. A Rogue class might be able to stand toe-to-toe with an enemy for a while, but that was a risky proposition.
Magic Users and other caster classes were not warriors, but they could bring power attack spells and tremendous utility spells to bear in the game; however, they were often limited in how much they could do in a single encounter.
Even though each class had their specialties, few were exclusive to that class. A Rogue with magical equipment might be as well protected as a Fighter in plate mail. That Fighter may obtain an enchanted weapon and thus be able to do significant damage to match a Rogue class over time.
Party composition was also not as strict. For example, being without a Fighter class in heavy plate armor (a Tank in the trinity design) wasn't always a disadvantage; in fact, the party could use stealth easier without members stomping around in a loud metal suit of armor. Magic items such healing potions, magic wands, spells scrolls, protective items, and so forth could also partially replace a missing role.
D&D had four archetypes -- Fighter, Cleric, Rogue, and Magic User -- but there were many different classes. Fighters, Paladins, and Cavaliers are related but play quite differently. Clerics and Druids cast divine magic, but their differing equipment and ways of using divine power make them distinct. Rogues and Bards had similar skills, but they had different specialties to bring to a group so having one of each rarely felt redundant.
Given the history we find in D&D, we aren't forced to abandon familiar combat systems in order to replace the trinity class design; however, we do need to change what we expect from individual characters. The two main concepts for guiding a different design are to eliminate specialized roles and allow the use of tactical options in combat.
Eliminating specialized roles means that we do away with boxing a class into a single role. Without Tanks, each class would have features that would help them participate in and survive many different encounters like heavy armor, strong avoidance, or some class or magical abilities that allow them to disengage from direct combat.
Without specialized DPS, all classes should be able to do damage in order to defeat enemies. Some classes might be specialize in damage type like area of effect (AoE) damage, others might be able to exploit enemy weaknesses, and some might just be good at swinging a sharpened bit of metal in the right direction at a rapid rate.
This design isn't just about having each class able to fill any trinity role. MMO combat would feel more dynamic in this system. Every player would have to react to combat events and defend against attacks. Some characters might be able to protect others, but it wouldn't always be the heavily armored character trying to draw a majority of the enemy's attention. Healing would be more of an emergency thing done at a cost in combat to help a character that has not been defending well.
Obviously good class design is important in order to provide tradeoffs between the classes. A heavily armored fighter with a big sword might not be able to defend against magic attacks, whereas the magic slinger might fall prey to sneak attacks if not paying attention. Each class would still have strengths, weaknesses, and individual flavor, but they wouldn't fall into the precisely defined roles that the trinity design encourages.
This doesn't necessarily eliminate the trinity of core roles from the game, however. A player could decide to focus on being able to take repeated, punishing hits while protecting other players and therefore fulfill a Tank role. However, eliminating explicit roles means that players are not forced into a specific role through class choices or game design requirements.
Use of tactical options in combat, like in paper D&D games, requires a different paradigm for gameplay than found in current MMOs. Maneuvering and physically blocking movement of enemies becomes much more important than it is in current games if the group needs to protect a wounded member who can't just be healed by the specialized healer standing in the back
Party members might be encouraged to use terrain and location as an important component of strategy. Ranged attacks could become more interesting if the players can make their characters use cover, or if they have to worry about firing projectiles into a melee involving group members. Combat could be a lot more dynamic than standing at optimal range and using special maneuvers.
Unfortunately, the biggest drawback of allowing tactical options in combat is that it will run into technical limitations given internet latency and cheaters. The phrase "using terrain" in MMOs usually refers to the exploit of harming NPC opponents while putting them in a position to be unable to harm the character. As the design is fleshed out, more of these issues will come to light and it may require adjusting some combat mechanics.
Let's review our design goals listed previously and how this new proposal satisfies them.
A diversity of character types. The character types are only limited by class design. As pointed out, the old D&D games had a wide variety of classes with different flavors and each archetype had several subtypes. A game using this proposal may see an increase in character types since they don't have to fit within a small selection of specific roles. If the gameplay doesn't punish some types of combat (e.g., enemies with specific damage immunity), different classes should be equally viable in the game.
Ease of balancing for developers. This is the difficult one. Increasing strategic options in combat will be hard to implement and to balance properly. It would be easy to give a class an overwhelming power, such as restricting enemy movement, which makes the class feel very powerful and therefore "necessary" for group composition. To be fair, this is going to be a problem with most original designs.
Solid direction for player advancement. This is possible with good class design. Players won't be stuck trying to solo with group-focused roles if they can handle most of the elements of combat within their limits. A player should be able to build their character to take more advantage of the class features they truly enjoy using.
Ease of identification of other characters. Once again, a class system gives a good shorthand for identifying other characters. This becomes less of an issue if the classes are balanced well; without specialized roles we won't have to worry about finding someone to fill a specific role to "complete" the party.
Designers often find it easier to simply copy mechanics that seem to work without really analyzing them. The trinity of core classes works well in many games, but the design can be limiting if it is copied without critical analysis. By taking a look at the design principles, we can identify the good parts and refine those into a better system.
Not only does this give a game a different and potentially better gameplay experience, it can also be a unique selling point to allow your game to stand out in a crowd. A top designer must take time to really analyze a game system to understand why it works in another game and how it can improve their own designs.