Pivot

Late last weekend was a minor pivot. One of the core limbs of the erstwhile crisp app was a content extractor written in Javascript.

A certain something that wasn’t exactly fast to my liking then, and much more so now when I am chasing milliseconds for entire operations.

So in the new iteration, broke it down to its finest bits and started moulding from the ground up. In every sense, this was a rebuild from scratch with zero parallels to the historical one — in philosophy, pace, or being a panacea of sorts.

Which sparked an idea in my mind and thus was born beetle — a clean way to extract structured content from the web. This is very much a POC for a generic use case at the moment. But, the idea having pivoted from crisp’s need for clean content has made itself a consumer — crisp.

In some sense, it is just dogfooding your own product. But in earnest, it puts things out in perspective — separation of work, monetisation of what can be, and more than any, an internal validation towards a larger outer one.

crisp’s beta website now uses beetle’s core under the hood to get clean short summaries for news. The v0 now powers crisp well, but the idea is to move it beyond just that — a generic content extraction framework that can power things from model training to simple readers.

On the engineering part, there has been something too good to be true — the numbers that I see defy historical observations from my shoes.

~2s run times even when exposed as a CGI API for 20 URLS — a stark contrast from the history. And the best part is the CPU even in the 2s that it shoots its head up, is under 40% consumption — I/O bound for the most part. And comparing it to a competing product, I am sure amazed as much as I still wonder how loudly.

Why is this important to both beetle and crisp? The operational cost is next to zero. But, more than just that, this entire exercise has been insanely liberating.