Idempotency, Deduplication and Memoization Are Not the Same Thing

These three words show up in design reviews all the time. They sound close enough that people use them as if they were the same. They all seem to mean "don't do the same thing twice".

They aren't the same. Each one answers a different question.

Idempotency is a property of an operation. Running it once or running it ten times leaves the system in the same state.

Deduplication is something you do. You notice that the same item or request showed up more than once, and you drop the extras.

Memoization is a speed trick. You remember what a function returned for a given input so you don't have to compute it again.

The rest of this post goes through each one with a small example, and then shows where they overlap. The overlap is where most of the confusion comes from.

Idempotency

An operation is idempotent if repeating it has no extra effect after the first time.

Setting a value is idempotent. If you set a user's email to a@b.com five times, the email is a@b.com. Adding to a value is not. If you add 100 to a balance five times, you've added 500.

// Idempotent. Run it as many times as you like.
user.email = 'a@b.com';
 
// Not idempotent. Every run changes the result.
account.balance += 100;

This matters most when networks are involved, because networks fail in annoying ways. A client sends a request. The server handles it. The response gets lost on the way back. From the client's side, it looks like nothing happened, so it retries. If the operation was "charge this card ₹500", the customer just paid twice.

HTTP already thinks about this. GET, PUT and DELETE are meant to be idempotent. POST is not. Deleting order 42 twice still leaves order 42 deleted. The second call might return a 404 instead of a 200, and that's fine. Idempotency is about the state of the system, not about getting the same response every time.

For operations that aren't naturally idempotent, like creating a payment, the common fix is an idempotency key. The client generates a unique ID for the action and sends it with the request. The server stores the key with the result. If the same key comes in again, the server returns the stored result instead of doing the work a second time.

async function createPayment(req) {
  const key = req.headers['idempotency-key'];
 
  const existing = await db.idempotency.findOne({ key });
  if (existing) return existing.response;
 
  const response = await chargeCard(req.body);
  await db.idempotency.insert({ key, response });
  return response;
}

This is a simplified version. A real one has to handle two requests with the same key arriving at the same moment, usually with a unique constraint or a lock. But the idea is the same. Retrying is now safe.

Deduplication

Deduplication is the act of finding repeats and throwing them away.

You see it a lot with message queues. Most queues promise at-least-once delivery, which means a message might reach your consumer more than once. A worker can crash after processing a message but before acknowledging it. The queue doesn't know the work was done, so it hands the message to someone else.

If that message says "send a welcome email", the user gets two emails. So the consumer keeps track of message IDs it has already handled and skips the ones it has seen.

const seen = new Set();
 
function handleMessage(msg) {
  if (seen.has(msg.id)) return;
  seen.add(msg.id);
  sendWelcomeEmail(msg.userId);
}

In production you wouldn't keep this in memory, since it disappears when the process restarts and isn't shared between workers. You'd use Redis with an expiry, or a database table with a unique index on the message ID.

Deduplication also shows up outside of requests. Cleaning a CSV that has the same customer listed three times is deduplication. So is a unique constraint on an email column, or disabling a submit button after the first click.

Notice one thing about the example above. Deduplicating messages is how that consumer becomes idempotent. That's the first overlap. Idempotency is the goal, and deduplication is one way to get there. It isn't the only way. Writing an operation as "set status to paid" instead of "add a payment row" can make it idempotent with no tracking at all.

Memoization

Memoization means caching the return value of a function based on its arguments. Call it again with the same arguments and you get the saved answer instead of a fresh calculation.

function memoize(fn) {
  const cache = new Map();
  return (n) => {
    if (cache.has(n)) return cache.get(n);
    const result = fn(n);
    cache.set(n, result);
    return result;
  };
}
 
const fib = memoize((n) => (n < 2 ? n : fib(n - 1) + fib(n - 2)));
 
fib(50); // fast, because each value is computed once

Without memoization, this Fibonacci function recomputes the same values millions of times. With it, each value is computed once.

If you write React, you've used this with useMemo and React.memo. The idea is the same. Skip the work if the inputs haven't changed.

Memoization only makes sense for pure functions. A pure function always returns the same output for the same input and doesn't change anything outside itself. Memoizing a function that reads the current time, or one that writes to a database, gives you wrong answers or skipped side effects.

The most important thing about memoization is that it's optional. You can clear the cache at any time and your program still works. It just gets slower.

Where they get mixed up

Look back at the idempotency key code. It stores a result against a key and returns the stored result when the key shows up again. That looks a lot like memoization.

The difference is in what happens when the stored data goes away.

If a memoization cache is lost, the function runs again and returns the same answer. Nothing breaks. If an idempotency store is lost, a retried payment goes through again and the customer is charged twice. That's a real bug with real money attached.

So an idempotency store has to be durable. It has to survive restarts and be shared by every server that can receive the request. A memoization cache can live in memory on one machine and be thrown away whenever it's convenient.

Here's a related trap. Say you wrap chargeCard in a memoize helper and call it done. It will seem to work in testing. Then the process restarts, or the retry lands on a different server, and the cache is empty. The function was never idempotent. You only hid the duplicate in one place for a while.

A quick way to tell them apart

Ask what goes wrong if you remove it.

If removing it only makes things slower, it's memoization. If removing it lets repeated input cause repeated effects, you're dealing with deduplication or idempotency. And if the question is "is it safe to retry this", the word you want is idempotency.

They work well together. A payment API might be idempotent because it deduplicates on a key, and the same service might memoize an exchange rate lookup to save time. They're just doing three different jobs.