Intuition as a service

The problem

Alex Phillips had a very successful marketing consultancy, working with big-name clients. His speciality was finding affiliates and partners overlooked by other tools and techniques.

Clients loved the results, but there was one problem: the process of identifying these partners was extremely laborious.

First, Alex's team manually researched thousands of keywords and competitors. Then they entered the data into an enormous spreadsheet, where dozens of formulae tried valiantly to collate and rank the results. Finally, Alex would use his years of experience to intuit the ideal candidates.

It was like watching an episode of House, minus the lupus.

Alex hired me to turn this duct tape masterpiece into a product. Over several months, I helped Alex take his idea from spreadsheet to angel-funded start-up.

The solution

I broke the problem down into two parts.

Automate the manual processes.
Turn Alex's "intuition" into an algorithm.

Given that Breezy was self-funded at the time, there was also a very important third requirement: do everything as quickly as possible before we reach the end of our runway.

Automate

There are plenty of existing services that perform research for a given set of keywords or URLs. Unfortunately, none of them did what we wanted.

Rather than reinventing the wheel, I elected to combine the results from multiple third-party APIs, each of which supplies some of the required information. This approach worked well and allowed us to progress quickly, but it brought other challenges.

A typical piece of "partner research" requires around 50,000 requests to external APIs. Each request needs to be queued and processed. Errors need to be handled. Requests need to be throttled. Some requests can run concurrently, others depend on data from previous requests. Some APIs respond in milliseconds, others take 30 seconds or more.

And of course, it all needs to scale.

The right tool for the job

Most web apps or APIs take a few milliseconds to process each incoming request. Each request might necessitate a handful of database queries, or kick-off a couple of background processes.

Scaling a typical web application involves figuring out how to handle a large number of small requests. This is exactly what most web-centric languages are good at.

Breezy is not a typical web application.

Even with a sizeable customer base, we might only need to handle a few dozen research requests per hour. But each request triggers a huge amount of work.

The solution to this problem is to shove everything onto a queue. The application then works through the queue, processing each job in turn until the queue is empty.

Problem solved, right? Not quite. Pushing a job onto a queue takes time. Pushing 50,000 jobs onto a queue takes a long time. Too long for a language such as PHP, which is meant to die.

There are fancy API design patterns to address this problem, but none of them quite solve it. Instead, we need the right tool for the job. In this case, that tool is Node.js.

Node works very differently from PHP. Sure, it might slow down when things get busy, but it won't die. Slow is much easier to cure than dead.

Add in Amazon SQS as a cheap, fast queue, and that's the scaling problem solved.

Cheap, fast, and good; pick three

The second automation challenge was monetary. Most APIs charge per request, and 50,000 requests a pop soon adds up. It's no good using third-party services to reduce development costs if they end up costing more than the developer.

Caching to the rescue.

The search engine results for a given term don't change much from day-to-day. The same goes for the number of monthly visitors to a given website or their geographical distribution.

Rather than requesting the same information over and over again, we cache everything for as long as possible. This simple approach reduced API costs by over 80%. It also had the nice side benefit of speeding up results.

Codify

With automation done, I moved onto the tricky part: turning Alex's intuition into an algorithm.

The problem with intuition is that it's very difficult to codify. Rather than spending weeks trying to guess the "right" algorithm, I instead decided to iterate my way to accurate.

I started with a ranking algorithm based on 7 criteria. Each criterion is weighted. Adjusting the weightings affects the overall ranking of the results.

With this system in place, Alex was able to rapidly test different weightings, fine-tuning the automated results until they closely matched his intuition.