When I design a boss, I try to keep all of the above in mind, but that's a whopping huge pile of goals to remember! Because I like to keep things simple for myself, I like to boil all that information down to two points. I try to remember these two points at all times while designing a boss battle:
1. A boss is a test.
2. A boss is a story.
The Mother Brain boss fight from Super Metroid is a great example of the principle "a boss is a story." In addition to hitting all the beats I outline later on, this battle manages (without a single line of dialog) to tell a surprisingly touching story of sacrifice, love, loss and revenge.
As I mentioned above, one of a boss battle's primary duties is to test players on the skills they've learned, and to allow them to demonstrate mastery of those skills. It's the designer's responsibility to administer this test, but figuring out how to do that can be overwhelming at first.
To help with that, there are four "prep-work" tasks I like to perform to start things off:
At a bare minimum, all the basic controls of your game should be on this list -- but often you'll want to test the player on something specific.
In the Legend of Zelda games, the player will often obtain a weapon during the course of a level. During that level, he will be taught how to use the weapon and required to use it again and again. When the level is over, the boss at the end tests him on the use of that weapon.
Example list of skills to test: movement, jumping, melee attacks, blocking, dodging left/right, dodging forward/backwards, ranged attacks, vulnerability points, etc... I would write each of these down so that in the next step I can make sure to come up with an attack to test each one.
Once you know the skills that you want to be on the test, the next step is to brainstorm and make a list of attacks that will test those skills.
It's important to brainstorm these attacks independently from any preconceived notions of what the boss can do, or what he is. Think instead only of what the best attacks are that can test the skills you want to test. By keeping this step separate from the next, you can avoid limiting yourself based on your boss' appearance or theme. Once we know what the attacks are at their basic, lowest level, we can theme them appropriately in the next step.
A diagram of a ground-based ring attack, starring Happy McStickman.
Example: In step 1, I decided that I wanted to test the player's jumping skill. A ground-based ring attack is best countered by jumping over the "edge" of the ring as it comes towards the player. So I can add "Ground-Based Ring Attack" to the list of boss fight attacks.
Once you have a list of boss fight attacks, the next step is to figure out how to theme them so that they're appropriate for the boss character you're using. For example, our "ground-based ring attack" example from the previous section might be the boss stomping his foot and sending out a shockwave.
In the fight with Emperor Tachyon in Ratchet and Clank Future: Tools of Destruction, Tachyon jumps into the air and lands on the ground with a thud. A ring of energy expands out from his position. This is how the fight's designer chose to theme the ground-based ring attack from our previous example.
One common weakness in many boss battle designs is that the players can damage the boss character repeatedly and keep him in a state where he can't effectively use any of his attacks. When the player can do that, the boss isn't able to perform the moves you've designed to test the player's mastery. The battle begins to feel dull and unsatisfying.
To guard against this, you need to design the boss with defensive capabilities so he can withdraw from the fight or otherwise allow himself to attack without being interrupted.
Figuring out how the boss can defend himself is a critical step to building a boss fight that will serve as a good test of the player's abilities.
Example defenses: Does the boss have a force field he can turn on while he does his attacks? Can he fly away out of range and then launch an attack? Does he have a huge attack that knocks players back to a safe distance? There are many effective ways for a boss to give himself space to attack the player.