Comment by luuio
The KV store had etag support for conditional writes. Etags are only useful to make sure the data didn't change underneath between your read and your write.
Storing the checkpoints along with the mutation was for idempotency. If the checkpoint was in the document, that meant the mutation had succeeded and a retry should be no-op
Hm, I misunderstood, I thought the checkpoint system was also concurrency control to make sure nobody else had changed it from underneath you between read and write, since you had to read and write the whole (mega) document even though you only wanted to change a part (sub-document).
Doesn't the KV provide idempotency on it's own -- so long as you're checking that no changes have happened between read and write, why wouldn't doing the same write twice produce an idempotent result? A change happenening between read and write seems the only reason that would be a problem.
But clearly it's complicated and we don't have the whole picture as to business needs. Definitely sounds awful.