That Will Be Easy! (It's Never Easy)
Today we’re excited to announce the release of our #1 most-requested feature: the ability to upload the Gemfile for your Ruby projects and build a custom search engine over the documentation for your project. We’re calling it Omniref Custom Search, and we hope you’ll like it.
But more on that later. For now, we want to take a moment to reflect on the surprising journey that got us here….
Ready for anything!
When we first started working on Custom Search, we were coming off of a nice coding high: we’d just launched Omniref, having gone from a blank page to indexing (nearly) all of the public Ruby code in a short time. So when people started to ask us for the ability to upload Gemfiles, we said: no problem!
We knew there would be challenges, but we also thought we knew what they were. At that point we’d only really indexed the current version of every Ruby gem, and we knew that sure, there were some hard-to-fix bugs in some of the docs. Those things would be challenging. But, you see, the rest of the feature was so easy…all we needed to do was upload a text file, then parse out a few lines, get package IDs…and then bam! Pow! Custom search engines!
Like all good journeys, then, the early days were smooth sailing and rum and sea shanties: we made steady progress on expanding our search index to include all versions of every Ruby gem, and even polished off the Ruby standard libraries and fixed a bunch of nasty bugs in the process. As far as we could tell, the “hard” parts of the project were under control. Now time for a bit of UX polish, and…
Hey, are those ominous-looking mountains over there?
We were soon pulling at some loose threads. Once we started working on the flow for uploading and processing Gemfiles, we realized that there are a lot of little complexities that we hadn’t considered. Among them:
- How will we manage dependency resolution process?
- How will we keep dependency data synced with the data in our search indices?
- How can we turn gem name/version dependencies into search queries that don’t take forver to run?
- Will our search relevancy algorithms have to change dramatically when we start allowing custom searches?
- Will doing any of that other stuff cause our servers to burst into flames?
But this is just a taste. We answered all of these questions and more, and a pile of questions that spawned off of those questions, as well. Ultimately, our “simple” feature required careful thought about problems that we hadn’t even remotely considered at the outset of the project, ranging from exotic technical decisions regarding multiprocessor concurrency, to the correct locations of buttons in the UI.
Deep underground, all Bundled Up
One of the biggest surprising challenges we addressed was just figuring out what to do with the Gemfiles once we had them. We started simple: we wrote prototype code that used a home-brewed parser to extract basic Gem information, and we used that prototype to feel out how to build the rest of the system. But Gemfiles are surprisingly complicated beasts, and using a home-grown parser and dependency resolver is a serious limitation. Even if we implemented ours to exactly match Bundler, we’d always be playing catch-up.
We needed to use the real deal, but the real deal was surprisingly difficult to use: Bundler isn’t made to be run as a background process, especially not in a project that’s already using Bundler for its own depdendencies. Unless you’re careful, you can mess up your app’s runtime environment in strange and wonderful ways.
The decision to use Bundler guarantees that we’re always doing the right thing for your projects, but it also led to a long journey into the murky depths of that code. We ended up building a system that uses much of the internals of Bundler to resolve dependencies, while also implementing a lot of our own hooks to ensure that we can identify inaccessible gems and gracefully recover from errors in an automated fashion.
At last! The promised land?
What have we learned? That’s a tough question. Everyone knows about the 80/20 rule, and it’s not surprising that software projects are unpredictable and complicated. This is why methodologies like Agile were invented.
What’s interesting here is that something that we thought would be a simple feature with a few, well-understood hard parts turned into a multi-month adventure where the hard parts weren’t the necessarily hardest parts, and a few things that we hadn’t really thought about much at all came out of nowhere to dominate our lives.
The thing that’s really interesting about all of this is that it’s a concrete lesson for why startups are hard — not only can’t you predict your schedule that well, but you can’t predict your schedule in a way that also makes it incredibly difficult to make intelligent risk/reward tradeoffs. If you can’t reliably predict the hard parts of the project before you begin, how can you decide if the project is worth doing?
For now, for us, the answer to that question is in your hands: upload a Gemfile, try out custom search, and let us know what you think. Want us to add something? We’re @omniref on Twitter, or you can email us: firstname.lastname@example.org