You've developed an object viewer, a Quake level viewer, a radiosity renderer and a patch tessellator. You know 3D math, OpenGL and the 3DS file format by heart. Now you want to go further -- you want to use this knowledge to develop an entire engine.
Does this seem like a large task for you? Well let me tell you, it is. Anyone who has developed a complete game will agree with me. Many people give up or start over along the way. But there are a number of things you can do that will increase your chances of reaching your goal.
I will not try to tell you how to make a good, playable game. Nor will I tell you how to program any specific parts of an engine. In fact there won't even be a single line of code in this article. Rather, I want to encourage you to think beyond shadow volumes and dotproduct3. Think about what it will all be used for, and work with this goal in mind.
Even though this article talks about the development of a 3D engine, the same techniques should be applicable to, for example, 2D or playing-card game engines. The main goal is to enable you to quickly and (relatively) painlessly develop an engine and a game that uses it.
There are three steps that I undertake before beginning work on an engine:
These three steps might require a few days at the beginning of the project (when you're eager to get busy with your compiler), but I guarantee you that the time will be well spent. It is entirely possible to develop a game without any planning at all, but the further you get in development, the more you will wish you had devoted some time to planning in the beginning.