Migrating business-critical code safely with golden master logs

At Joko, one of my main roles is to take care of the system that is at the heart of our business model and currently generates tens of millions in revenue per year. It’s a system that has been in place since the very beginning of Joko and that has proven itself over time. It’s a system that works.

OK, but if it works, what is it exactly that you do?

That’s a question I get pretty often from family and friends and that I always have a hard time answering properly.

Here’s my best take:

“The code that serves one million users is not the same as the one that serves ten million."

Over the past few years, Joko has grown from a small startup into a company serving millions of active users every month. In this context of rapid growth, both in our historical market and internationally (the size of our user base tripled in the US in one quarter), one of our core responsibilities as an engineering team is to continuously improve our business code.

In this post, I will tell the story of how we migrated one of our most critical pieces of code to make it more robust and maintainable, and how we played it safe using the golden master logs strategy.

The code

Joko is a universal shopping app that helps people save time and money when they shop online. A large part of our business model relies on affiliate commissions. When users shop online through Joko, we receive an affiliate commission from merchants. We share a portion of that commission with users in the form of cash back.

Let’s take an example. Every year, I buy a pair of Stan Smiths in my size. It’s a debatable fashion choice, but it’s a very efficient process. This year, I did so using Juno, our AI shopping assistant, and I was able to get close to €5 cash back. These €5 came from an affiliate commission that Joko shared with me.

Today, we partner with tens of thousands of merchants in multiple countries, including giants like Amazon, Apple, and Walmart. We work with affiliate networks that allow us to reliably track when a purchase was made via Joko.

The code we migrated during this project performs the following actions once every hour. It:

• fetches recent transactions attributed to Joko from the affiliate networks,
• matches them to a Joko user,
• computes the commission for Joko and the cash back for the user,
• checks the approval rules (positive margin, no fraud, …),
• credits the cash back once approved (or informs the user if it was declined).

This is about as business-critical as our code gets: it moves real money to real users, and it tracks the affiliate commissions that make up the majority of our revenue.

The rewrite: untangling logic from I/O

Why rewrite it all? This logic was created right at the beginning of Joko, before the company found its product-market fit, when the company had very few engineers. It was written without taking into account all the problems that come with the scale we can have today.

One of the main issues was that the business logic (user matching, approval rules, fraud checks, etc.) was tangled together with the I/O (input/output: fetching transactions from the affiliate networks, writing to our database, etc.). That made it hard to read, hard to test, and risky to change.

Let’s take the example of the “match user” step above. It reconciles a transaction with its user using the click event logged when the user activates their cash back. Here is what it looked like in the legacy.

async function matchTransactionToUser(transaction) {
  const clickEvent = await db.getClickEvent(transaction.subId); // IO
  if (!clickEvent) {
    transaction.status = "not_matched"; // logic
    await db.save(transaction); // IO
    return;
  }
  const user = await db.getUser(clickEvent.userId); // IO
  transaction.userId = user.id; // logic
  transaction.status = "matched";
  await db.save(transaction); // IO
}

Before: logic and I/O interleaved

In the new code, the business logic is decoupled from the I/O. It lives in pure functions that take data in and return decisions. The I/O moves to the edges of each step. Reads happen upfront, writes at the end.

function decideUserMatch(transaction, clickEvent, user) {
  // pure, trivial to test
  if (!clickEvent || !user) return { ...transaction, status: "not_matched" };
  return { ...transaction, userId: user.id, status: "matched" };
}

async function matchTransactionToUser(transaction, { db, sns }) {
  const clickEvent = await db.getClickEvent(transaction.subId); // 1. IO (read)
  const user = clickEvent && (await db.getUser(clickEvent.userId));

  const matched = decideUserMatch(transaction, clickEvent, user); // 2. logic

  await db.save(matched); // 3. IO (write)
}

After: logic decoupled from the I/O

I won’t go into more detail about the perks of separating I/O from business logic, but here is an excellent video on the topic if you’re interested.

Refactoring the code this way had a huge impact in terms of code readability, code testability, and robustness. However, it does not change the business logic, and it’s absolutely crucial that this refactoring does not change the output of the pipeline at all. The rest of this post is about how we verified just that.

Migrating with the golden master strategy

Golden master logs

The legacy code we are talking about may be flawed, but it recently took us past a stunning 80M€ in cumulative savings for our users. However confident we are that the new architecture will be more robust in the long run, on the question of what the pipeline should do, the old code speaks with authority:

The new code is correct only when it behaves exactly like what runs in production today. That is the “golden master” idea, borrowed from approval testing: record what the running system does and treat that recording as the reference.

Here is what it looks like.

In both implementations, legacy and new, we emit a structured log line immediately before every side effect (every database write, every message sent), describing what the code is about to do:

goldenMasterLog({
  step: "matchUsers",
  effectType: "dynamodb", // DynamoDB is the database service we use
  effectTarget: "affiliateTransactions",
  transactionId: transaction.id,
  payload: transaction,
});
await db.save(transaction);

One golden master log before each side effect

The two log streams must be comparable line by line, so the logger normalizes everything that could differ for irrelevant reasons:

• One line per transaction. The two implementations group batches differently, so batched effects are flattened into one log line per transaction.
• JSON keys are sorted so lines are byte-comparable.
• Volatile timestamps are rounded to the hour. The two runs never execute at exactly the same millisecond. We are only interested in hour-level precision here, but that may change depending on the use case.

In the new code, this is a few lines at the I/O edge of each step. In the legacy, it meant hunting down every write, which was a good reminder of why the old shape had to go.

Enabling AI to investigate and fix discrepancies

With millions of log lines to compare, we wrote an AI skill that ran daily in four stages.

1. Fetch all golden master logs for a given day, legacy and new, using CloudWatch Logs Insights.
2. Compare the volumes per step × effect type × effect target. This instantly exposes whole classes of discrepancies (an event the new code never emits, a step that writes twice as often as its counterpart, etc.)
3. Dig into examples. Join the two streams by transaction ID and diff the payloads of divergent transactions field by field.
4. Open pull requests for fixes. AI has access to both versions of the code, so it can go from a divergent log line to a proposed fix in the same session.

Each diff fell into one of three buckets: a bug in the new code, a bug in the legacy, or an accepted divergence recorded on an explicit allow-list.

What the logs caught

Here are a few examples of the issues we caught:

• A missing cache in the legacy called the affiliate networks API far more often than needed. The double run doubled the call volume and hit rate limits.
• Many consistency issues in the existing data. One example: the legacy could persist a transaction matched to a user but with no user ID. In the new code, that state does not compile thanks to stronger typing.
• Many unintended behavior changes. For example, the legacy computed a potential cash back for every transaction, whatever its status. The new code computed it only after the transaction was matched to a user. The diff caught each difference of this kind and made us choose: reproduce the legacy behavior, or change it on purpose. Without the logs, these would have shipped as silent issues.

We ran the comparison for 10 weeks over more than 10M transactions, and flipped the switch only when the sole remaining diffs were understood and accepted. Until that moment, the legacy stayed the source of truth, so rollback stayed trivial.

Takeaways

Migrating business-critical code is not as scary as it sounds when you take the time to do it properly: isolate the logic from the I/O so every side effect is observable, treat production behavior as the spec, let an agent do the diffing, and keep the old system running until the logs say you can let go.

Golden master logs are now part of our toolbox for any high-stakes migration. If this is the kind of problem you enjoy, we’re hiring! Come build with us 🧑‍🍳