Every living thing has a set of genes in their cells. These genes express the qualities of that living thing. And every living thing’s set of genes is subtly different. We inherit a lot of common genes, of course. For example, the hox genes controls the body plan of an embryo, giving it a head-tail axis, limbs, and so forth. Hox genes are highly conserved - that is, they’re present in their usual form in almost every multicellular organism. More unusually, a gene that would otherwise have given a primate its powerful jaw structure is defective in humans. It allowed greater encephalization, or brain development, which is obviously crucial for us as a species.
What does this have to do with game design?
You can think of a roleplaying game, or its cousins like card and board games, as organisms, with their own genes. If you draw a family tree for the traditional RPG, you’ll see “genes” like dice rolling, character traits, a game master, and advancement through experience. Some of these genes are highly conserved. Others mutate at a fast rate. For example, dice systems are often different from game to game, but the mere fact of rolling dice is itself conserved. Similarly, the “GM” role disappears from the gene pool for “story games”, or takes on a different quality in games with "player-facing mechanics" like Apocalypse World or the Cypher System.
Like living things, such as domesticated plants and animals, we can breed for desirable traits through artificial selection. In gaming, we do this by taking elements from a game we like, then mutating or cross-pollinating with elements from elsewhere. For example, D&D 5th Edition’s “Advantage” mechanic is a mutation on d20, while the d20 roll itself has been inherited since the 3rd Edition and has appeared in several other games since then. People who liked 3.x went on to conserve many of its desirable traits while weeding out the undesirable ones, and the result was Pathfinder.
Organisms evolve through selection. Those without a survival advantage will fail to reproduce, while those with advantageous traits will continue. The roleplaying community at large applies a strong selection pressure, voting with their wallets, their mindshare, and their own creative efforts. Games with interesting elements will get played, while games that aren’t fun will gather dust on a shelf. This process forces games to mutate.
Remember the broken primate gene? Sometimes a trait that doesn’t work in one game, or that actively hampers a game, will be an advantage in another. D&D 4th Edition is famous for its resemblance to MMORPGs. An example mechanic that both share is the “tanking role”, where one party member’s job is to absorb damage and keep enemies from attacking weaker allies. In a game where NPC target choice is made by computer, this is vital. In a tabletop context, where the enemies can be as smart and cunning as whoever is portraying them, having a tank with a “taunt” move and extra damage-mitigation abilities might not make sense.
Ron Edwards’ article on Fantasy Heartbreakers gives several examples of games with unique mechanics, and laments that some of these innovations weren’t spun out or better developed. Each of these games mutated the original concept of "Dungeons and Dragons" in some unique way, and many of these mutations were novel and enjoyable. But the creators’ original goal of "make D&D again, but better as I define it" got in the way of appreciating the truly worthwhile parts of the game.
What does this mean for you?
When you sit down to design a game, you will be working with these game genes. Things like “character attributes”, dice systems, the concepts of hit points or damage, and so forth.
Once you know that, learn to sift through your design for new genes you’ve made. If you reach a point where you think your current design won’t work, start reviewing the pieces. Something in your design has value. Find it, then extract it. Put it in your genetic library.
If your design mostly works, but isn’t all the way there yet, apply some artificial selection. Make a copy of your document, then mutate one of the genes, or pull an old genes out of your toolkit and see if it plugs in any better. Tweak, and test, and tweak again. If what you’re changing doesn’t seem to help, now you know that you’re changing the wrong thing. Try and find out what really needs to be fixed.
Your games will get better when pressure is applied to them. You can do that before anyone else sees your design. Pick it apart, find the problems, identify the things you don’t like, and then change them. You don’t need to know the final version, either. The process of changing something, by itself, is teaching you what you need to know. The journey creates the destination.
Nothing you do is a waste. Every design you’re not happy with will give you the ingredients for a design you love. Remember to look at everything you do as an opportunity. Breed a good game through breeding bad ones, then so-so ones, and grow your toolkit with useful genes for the future.