It seems almost dogmatic; don't limit the players choices, present him with a wealth of options at his disposal to give him a richer experience. The issue with non-linearity is that if a little is good, then more is better... right?
Non-linearity and choices are damaging to a game not because they're inherently bad, but we simply have too much of it. Players are bombarded with games offering them choices. Choices on alignment, fighting style, difficulty, primary weapon, secondary weapon, the type of car, the type of sword, all before level 1.
Designers might present choices early because they think that if the player spends 10 minutes selecting the type of sneakers the player will be wearing before playing, that he's going to be more invested in the game once he starts. There are two sides to this coin however; if the player is invested in the game and he likes what he's playing, it's a slam dunk. If the player grows weary of the game for whatever reason, he may continue to play since he feels so invested, which will ultimately result in much scorn instead of just quietly exiting the game without incident. He may not know exactly why he hates your game, but seducing him with choice is what got them there. Choices can get players into trouble if not handled properly.
Imagine 8 levels where the player can choose the order in which they're played. The designer is pitted with quite a task; make each level accessible to new players (players that just started the game), while still being engaging for advanced players (players that have completed the other 7 levels). A creative designer can give the levels different themes, different gameplay aspects, and different priorities, but he can't do much in regards to skill building or difficulty, he must keep changing things to keep it fresh for both types of players. He's unable to focus on a particular skill set, not in an in-depth way at least.
So the designer makes 8 non-linear levels, waiting for that bottle neck to occur, for that Wily castle, so that he can dig deeper into the gameplay that he wants to explore. Unfortunately the designer has a player that has only scratched the surface on a variety of things, not a player that has specialized a particular skill set. So it's unfair to ask the player to specialize, its contrary to everything else in the game. This is why the 8 Robot Masters are fun, but the Wily Castle will always be more forgettable. The Wily Castle isn't bad, it just isn't as good as the non-linear section of the game because there isn't enough to build on, there's no way there could be.
The reason the classic Megaman formula works is because there's 8 Robot Masters, sure the end is always going to be tricky to design, but the point is that it wouldn't work with 80 Robot Masters. There's a limit, a threshold, for Megaman its 8. For other games there surely must be a limit; one that designers have to be mindful of to avoid putting themselves into a position where they can't build skills within the player. Going into a game with a mindset of "how many options, features, and choices can I stuff into this game?" brings out the worst in non-linear design.
So what's so great about linear games? The linear game offers the designer a far more intimate relationship with the player; he knows what the player knows. He can present ideas in level 1, and can sculpt level 2 to reinforce those ideas and build on them even further. He's doesn't have to consider a spreadsheet of possibilities and outcomes every time he wants to try something new. He is also in a far better position to take risks, experiment, and explore.
You and the player can specialize. If the focus of the game is say, managing an inventory and combining items, you will know (for the most part) what the player has in that inventory. If level 3 is about espionage, you can give him a silencer at the beginning of the level and he'll immediately understand its purpose, combine it with the gun, and advance with success. If you give him nothing, that will make him think about his existing inventory and what he can do with it. As a designer you have a fair idea of what's going through the player's head, and can speak more directly to him without using a large set of rules or conditions as a medium.
Of course linearity has its pitfalls, the main one being the "brick wall". If the player isn't experiencing success, and there's no other choices, he's backed up a against the wall and so is the designer. Frustration sets in, the player curses the game and quits. Every designers nightmare. If the designer can brave that risk however, and understand that there will always be those few players that quit, and of course do what he can to minimize frustration, then the rewards can be great. Nothing feels better for players than failing a number of times, honing their skill, and then emerging victorious. Since they're aware that there aren't any other choices, they can focus on the task that vexes them, not just work around it by traveling to some other section and leaving it for later. When the player always has the prospect of doing something else in the back of his mind, then on some level he's distracted from the task at hand, undermining all that great design work that's been poured into the game.
Linearity is simply more high-stakes; it's risky, it can frustrate the player, things can go wrong, but as we all know big risks is where the big payoffs are. Designing linear games makes you better at... designing linear games, if a designer always fear the brick wall, that designer will never develop any skills to deal linearity in any capacity. The non-linear designer can fall into the trap of just creating a path around a problem instead of addressing the issue, because linear design is not in his repertoire. It's dangerously easy to bury problems with non-linearity, particularly when the player can travel back later with some huge advantage.
I by no means intend to bash non-linear design, but to advance the notion that purely linear games are a viable pursuit in their own right, and that linear design should be utilized within a non-linear space. Also that non-linearity needs to be limited in some way, there needs to be a very deliberate effort to keep it meaningful to leverage it properly.
I'll leave you with a paradox; by giving the player less choices you're actually giving him more choices. When every game has a ton of choices "a game with less choices" is actually a new choice.