To broaden the topic somewhat - aren't companies who insist on continuing their long time engine development dragging the quality of their products behind them? Like a in ancient Greece, where prisoners where oftentimes tied by their wrists or ankles to a chariot and rode through rocky passes for kilometers.
The analogy isn't completely absent-minded if you just thought so - looking again at Valve, I absolutely can't jump on the whole L4D hype wagon. I look at my workmates playing it every day during lunch break and it repulses me how crude it is - almost no physics, no character model breakdown on body parts - you just basically mow down a crowd of - well yeah zombies - still people call it arcade, I call it the 90ties and a bespoke excuse.
Don't you get the feeling we're more making game according to a marketing monster's strict guidelines than our vision? Say a Senior Programmer at Valve, fed up one day with struggling with the spaghetti code he must bear with every day goes to his supervisor and says "look, the physics engine needs a re-write". The lead stares at him like its April Fools Day - after all the company policy to build on the existing platform should be burned on everyone's skull. It's not good marketing to forgo years of money and time when the next thing that comes out of the house will sell millions on the 1st day.
Will Valve build something new? Not anytime soon IMO, not when the bulk of their customers are people who can't squeeze in more than 30 - 45 min of quick, story free, dare I say zombie-level complexity game time into their day schedule.

Me and some of my designer pals are making a Payload map. Its great fun but Hammer is not nearly plug and play as much as Unreal. In particular the Visgroups which is how source optimizes the visual space of the map. That said it is in my opinion the best supported set of user tools out there. The tutorials are plentiful and the Valve developer notes superior and the communities at interlopers and TF2maps.net supportive. The Unreal UDN is pretty thin which is why there are three 600+ page books on the engine.

Yeah Valve is going to have to move from Source at some point and I hope to use that set of tools as well. The sad thing is though with an increase in graphical fidelity ie moving to mesh based world structure rather than textured BSP you are going to see less mod and community levels. I'm not the first to say this but the asset creation side of modding is going to/has increase(d) the barrier to entry.

@kevin - i agree, Source is one of the last dinosaurs to the old mod generation. Their tools are still in line with the old unreal, and radiant editors of the late 90's and early 20xx. Though this sounds like a bad thing, it's actually the one thing that keeps the mod community buzzing over it. Development on anything built after 2003 (on fresh tech) has taken a turn against modding. Personally; as much as I'd hate to see it gone, modding as we knew it is dead. It's hard enough to get someone to build a model and drop it into a folder, like in the old days of Quake 3 character creation. Now you have to be a professional who knows how to script animation tables, and define transition graphs, and setup frame events for when to play foot step sounds, and define animation channels for body parts, and define IK locators for weapon placement, and... well you get the picture...

The developer in me will be a little sad to see the last dinosaur pass on, but the gamer in me will be happy to play through a session without constant hardware lock ups and hard reboots either. There is never a more nerve wrecking and gratifying experience than starting over.

First off, I've never had any major issue with Source games. Just the regular stuff you get with any pc games.

But, say all the negative things about it, people are still buying "source powered" games by the butt load and there are more mod communities making mods/maps then any another engine. When I go back and play Half Life 2, I'm not noticing that the graphics are dated, I'm to engrossed in the District 17 to realize that.

Forget graphics, Half Life, Left 4 Dead, Portal, and Team Fortress 2, are great designed games and are fun to play (which go farther then graphics in my book).

I agree with Rick

Source is just one of the best engines for the community at this moment, nowhere else are there so much good mods available. From total conversion mods to skins and models, just great.

And the games Valve made are just top notch and very well balanced and produced. I think a lot of developers could learn from Valves designing and balancing process (you can go and hear the very interesting developer commentary for a glimpse of their process).

But I also think it's maybe time for a complete reboot of the Source engine. They have been updating it by every game they made, but a fresh start couldn't hurt I guess. One of the major problems they have is multi-threading, they have started to implement that since Left 4 Dead but on modern pc's multi-threading (and also GPU processing) is pretty important to get a fluent game out of it. But still, all their games run very good, they have made extensive surveys on hardware so they know the best target market for performance.

So yes, Valve is one of the best game developers out there, and graphics are less important than gameplay (and their games have not bad graphics and very good gameplay).

I've never had problems running Source, and I've played almost all of the Source-engine games, but certain UnrealEngine 3 games gave me no end of problems. Bioshock, pinnacle of game narrative that it is, refused to run at all on one computer, then refused to play sounds on another. Both machines were fully capable of playing it, according to the box, but various technical bugs stopped it. America's Army 3, also a UE3 game, had more bugs that I found in two hours of play than I've found in months of playing TF2.

As for the huge game installs, it's partially because the Steam has certain files shared between games. So, Episode 2 may take up 3 gigs, and TF2 may use 5, but both, together, only use 6. Those probably aren't exact numbers, but you get the idea. If you have many Source games installed at once, it's a huge savings.

I will concede that the engine design is outdated. Having to spend hours baking lightmaps is frustrating. An engine heavily optimized for visleafs causes certain problems in highly destructive environments.

However, Valve has proved time and time again that you can make a better game on a technically worse engine. Using an "outdated" engine also let's you run well on "outdated" hardware. All their games run on DX8, and most of their pre-Orange-Box games will run on a measly DX6.

Most people don't care if the engine doesn't keep up with the latest trends. If the game runs, and is fun to play, it's a good game.

Interesting. First off let me say that I spent most of the last 4 years of my life working as a programmer with Source as a 3rd party developer (for Outerlight, on The Ship and an unannounced follow-up). The engine does have its problems, and it certainly has its limitations, but then what engine doesn't.

In any event that doesn't really relate to the topic, which is the more general idea that re-writing something from scratch is the way to fix problems.

"when you continue to build technology on top of existing frameworks, you get long load times, frequent crashes, data bloat, and much more."

I'm sorry Benjamin, but I don't think any of this is correct :$

What you get is experience, an enormous catalogue of fixes for specific issues on specific platforms, and a widespread knowledge of exactly how your technology will perform under a wide set of circumstances. And you get the stability born out of a decade of testing and fixing. Yes, you have to adapt and constantly re-write individual components, but you must be careful not to throw the baby out with the bathwater. Re-writing, from the ground up, a vast piece of technology - especially one with such enormous community adoption - would almost certainly be a pretty big mistake.

The idea that Source games are less stable than others in general is I guess a fairly subjective, setup-dependent assumption. Even if this is the case - and don't get me wrong, it may well be - it's certainly not because it's 'old'. It's not a car.

Check out Joel Spolsky's ancient but still incredibly relevant piece on the subject: http://www.joelonsoftware.com/articles/fog0000000069.html

I never said that Valve made bad or ugly games; on the contrary, they make great games and I wish they came out more frequent. I think Source graphics are where they need to be for the type of games that Valve creates. I don't think that everyone needs to rest on the top tier of graphics, so don't think that this was my driving point.

@Andrew, you make some solid points, but I do still feel that a compound architecture leads to judgment calls based on what is already available and not what CAN be done. For example, Valve's choice in multi-threading was to break up the client and server code into threads; effectively separating the game logic from the image generator. This will work perfectly fine for at least a couple more years, but it's not exactly a solution, more like a band-aid while we wait for 4+ core hardware to be adopted. Many modern engines have taken a more task-based design to prepare themselves for a future with an ocean of processors. I think that this is going to be a major barrier for Source in the coming years.

The use of lightmaps with solid BSPs and visleafs was a staple of the late 90's and is still the primary architecture for Source. This leads to environment files that can grow in excess of 50 to 100MB of compressed geometry alone despite the fact that some of the HL2 environments could have been handled more gracefully using other spatial systems. I can only imagine that this also imposes some interesting limitations in geometry batch sizes due to BSP eccentrics. Some of their choices may even have prohibited them from implementing background streaming techniques to avoid the sluggish loading between areas.

The list goes on but my point is not to bash Source, I'm just saying that it's not perfect. No, Source is not a car but does still suffer from aging as the hardware around it changes. Maybe not for Netscape or Excel, but games are a different kind of beast. A lot has happened in the world of PC's since 1998; I admit, mostly in the graphics department and the recent rise of widely adopted multi-core.

It is entirely possible that through the past 5 or so years and multiple hardware swaps that I am simply the most unlucky guy on earth to get incompatible hardware configurations for Source, yes. But I'm also the guy that can't get the Steam website to stop talking German on his Vista box, so there is no telling what's going on =). I don't deny that a fresh start will ultimately impose it's own share of bugs but sometimes it takes more than incremental changes to make the generational leap. If Valve can continue to keep Source relevant for 10 more years on incremental changes then I say, good for them. I'll probably still buy the game, and play it with two fingers crossed and one on the quick save key.