1. Small development & testing team
While the small size of the development team was advantageous in many situations, it also posed some unique challenges that many games aren't faced with.
There weren't enough available hours between the team members to properly test each new release, so versions were sometimes released with bugs that needed to be patched immediately.
When version 5.84 was released, only one beta game was played and it was played in "deathmatch" mode, (where once your hero dies, it's dead permanently, and you're assigned a new hero to play), to test all the heroes in a single game.
Unfortunately, the tester playing the "Dragon Knight" hero didn't have the opportunity to test one of his abilities, and the map was released with bugged 400 hp/s regeneration ability instead of the intended 4 hp/s regeneration one.
However, the vision was also bigger than the available man hours allowed for. Guinsoo and his team simply didn't have enough time to implement everything they wanted to, and expanding the team through adding more developers would have posed a new set of challenges.
Ultimately, the call was made to prioritize changes by their overall impact to the release, and many great ideas fell to the cutting room floor as a result.
2. Warcraft III Engine & Map Tools
Using the Warcraft III engine imposed certain limitations on the development team that could have been addressed with relative ease in a full product..
No matchmaking was available; brand new players were often forced to play with seasoned veterans, which created an uneven distribution of skill.
This issue was partially addressed through the creation of meta-communities by the players, such as dota-league.com, GG-Client, DotA-Rank, Clan TDA, etc. Unfortunately, these are largely cumbersome services that are hard to use and still contribute to the high barriers to entry of DotA.
In addition to the matchmaking limitation imposed by the tools, lack of stat-tracking meant that some players weren't motivated to keep playing. While each individual session offered players a unique and diverse gameplay experience, players had to start from scratch after the session was over. Not even wins and losses were tracked, which meant you could play over 9,000 games and not see any upgrades in record, status, notoriety, etc.
Blizzard also released regular updates to both Warcraft III and the World Editor tool. Significant code rewrites were often required because of changes Blizzard made. The map file itself could also not be larger than 4 megabytes -- a limitation imposed by Warcraft III. This meant that developing custom content (skins, animations, and models) was usually out of the question.
Due primarily to the fact that the development was small and they didn't have enough bandwidth to implement everything they would like, optimizations that were relatively important (because of the impediments of the Warcraft III engine) were often overlooked.
Certain characters or abilities would lag the game, and it took a long time to load at the beginning of each match. With each new version, the decision had to be made to either optimize the existing map, or add new content.
While optimization was extremely important to Guinsoo, it often lost out to adding more content. Shortly before Guinsoo handed the map to the current developer, he made optimization a high priority and implemented several performance-increasing additions in his final releases.
3. Difficult Distribution & Installation
Existing only as a modification of another game and lacking network support meant that whenever a new version of DotA was released, the entire playerbase had to somehow be notified of the change, and then provided with a copy of the latest release.
New map versions were hosted on a website, but this became increasingly difficult to manage due to bandwidth consumption increasing as the number of active players grew.
Players could also download from other players (joining a game hosted on Battle.net would automatically download the map if the player didn't have it), but game hosts were often reluctant to allow players to download from them.
Many simply assumed these players weren't good at the game because they hadn't already obtained the latest version from the "official" sources.
Once a player actually had the map file, there was no automated installation process. Each player had to save the map file to the correct sub-folder manually. This was often a confusing process for users -- especially for those who were new to the game and hadn't successfully completed the process at least once before.