And another fortnight passes, almost. A bunch of firefighting and a lot more that elevate the carpet later, some time to spend on things that keep the brain functional.

Restarted work on separating our current order system into functional, event-driven components. Even as certain teams express discomfort regarding the delays with refunds, the tunnel vision or the lack of any to begin with at the other side seems only to push further towards a scalable solution.

We decided on moving things to Kafka a few months back, but the backlog of years is certainly going to take a fraction at the least thereof. Since Kafka is its own beast and given the inability to do half-solution anything, decided to use Redis as a datastore, event queue in one with separate domain entities using their own instance.

The single-threaded^ nature is in some way a boon; being forced to isolate solves some. Created four different Redis servers two each for two domain entities and two within each for data queueing and event handling. In all, the biggest pain points probably have been along a single line - coupling.

Isolating the concerns to be loosely coupled both between themselves and also at the point of data - the data layer needs to be something that’s replaceable; a la dependency injection.

There’s more to this come to think of it. The wallet system, for an example, even while it exists on MySQL today is something that’d gain a lot from a solution like QLDB. But, wallet entries becoming an entry point before orders and after orders via refunds, make them either tightly coupled, or in the current system’s worldview, via HTTP APIs.

An alternate approach I’m looking at is to provide the scaffold in common but base implementation details depending on the data provider and individual entities providing their own pre/post massaging flows to accommodate.

On those lines, after a old talk on Event sourcing and in particular how append only logs + functional languages are a fantastic way to approach such entities as financial transactions/reports, wrote a starter implementation (Image at top). Given it still is Redis + J(T)S, this is but a clay dough figurine at this point, but the takeaways after the implementation have been excellent.