When working on any software system there are various challenges. When a system is entirely new, there’s a ton. Here’s a short and generalized description of what’s involved in the process. In the interest of keeping this non-technical, let’s use some analogies of adventure and countryside (that we’re ruining with dams to power our systems, ironically). A system is a specific place that you want to go to, like Grandma’s Secret Hillside Bakery. The path that that leads to it are parts of the program you have to create.

Feed the Troll, Beware the Midget

If your software is a new implementation of a system that already exists you have the advantage to know it’s probably possible to get there. And optionally you can explore the gear used in the other project to get an idea of what you’re getting yourself into. In adventure-speak, this would equal trying to get to Grandpa’s Bakery instead of Grandma’s. Unfortunately, it’s not that easy. There are a lot of things that need to be mapped out and decided on before you begin; what gear you’ll be using, means of travel, food, etc. In this sense, getting there involves a very crappy and spooky looking bridge you’ll have to fix to get over. Occasionally, that bridge has a troll under it — and you forget to bring it raw meat because you were busy mapping out the way to fix the bridge. This is where you get eaten and have to start over.

However, when inventing entirely new systems (as opposed to improving something) — there isn’t necessarily a bridge. Or rather, there might be a bridge. Or maybe there isn’t even a river — just a small hill to walk over. Hell, there might even be no land at all — just an empty void with a small bearded midget screaming “PWNED!”.

The problem is: you have no real idea of which one it is. Halfway through building a bridge, you could realize that just a few clicks up river there’s a really small faucet you can simply turn off … and then walk across. Note that if you encounter the midget anywhere along the way, it is of utmost importance that you ignore him for he is the physical incarnate of your subconscious dissapointment. Just accept you took a wrong turn somewhere and find another way.

Explore the Landscape

When creating a large system — there are many potential solutions and pits of death. (Let’s lay the Grimm analogies to rest for a bit). You begin by identififying which parts of your system someone might have done before. Maybe that program is used in completely different context, but usually there’s always something that can help. Taking an extreme example, you might want a system that can converse with 5 people at the same time — in which case you don’t have to reinvent speech synthesis. The real problem would lie in the intelligent architecture that uses the synthesizers. In short, you want to refrain from re-inventing any wheels. You already have enough of problems.

Background research can be very frustrating, simply because the Earth is a large place. Digging in the depths of the non-semantic web for information related to your project is like following Hans’ and Gretel’s trail of breadcrumbs. You find bits and pieces of information, that reference other pieces, that reference other pieces, that referaaaaaarrgh. This is where it pays off to have superhuman experts advising you where to look and when to stop. PhD’s, preferably. It’s also a good idea to talk to a lot of people about your idea before you start, as otherwise you might find that you’ll end up in a witch’s oven, roasting on slow spin and cursing yourself for thinking you could actually eat that gingerbread without consequences. Unfortunately, when you get a really great idea™ — your first instinct is not to tell anyone about it.

Construct the Bridge

Once you feel fairly confident that no one has implemented your idea before (often they have), and you’ve identified a lot of small parts that you can use as guiding light along your dark way — you can finally start plotting a path from start to finish. Using your identified parts as roadsigns — connect them together to find out what it really is that you need to create. Sometimes you find that you’re lucky; all you have to do to create your awesome system is to connect existing pieces of technology together in a unique way and poof! You have your system. Most of the time, though, you find that no matter how you configure your pieces and parts, there’s always a gigantic gorge you’ll have to get across to create your system. In that case, you can’t simply fix an existing bridge like we mentioned earlier — you have to design it, find the material, and finally build the whole thing. Depending on the size of the gorge, this ultimately decides the fate of your project.

So that’s basically my spontaneous brainstorm of what it’s like to invent new systems, based on my experience with A.I. software. That being said, I hope you don’t get eaten and look forward to seeing your system.

Links and References

• Broken Bridge image credit: George Pazios
• Totally cool Hans & Gretel drawing from Wikipedia
• Intellectual content from my brain