Batch up all deletes/updates in an Update API call

[thegedge]

Closes #106

Previously, when an update call was issued to dateilager, we would run a set of queries for every object being updated. This resulted in A LOT of DB chatter. For example, during application creating in Gadget:

You can see 1000s of pgx spans are nested under update-objects and update-packed-objects. This PR fixes that by bulk inserting. Since we have a dynamic number of objects to insert, we can't do a regular INSERT due to potential limitations on the number of parameters. Instead, we use another trick: create a temporary table that is dropped after the transaction completes where we stage all of the updates, and use COPY to efficiently get data into it.

The end result is pretty good!

Ignoring the packed objects update (discussion below), the regular update call goes from 1k+ queries taking 289ms to 6 calls taking 44ms. I'll have to benchmark a bunch, but on average I'm thinking this could be close to an order of magnitude improvement. Hard to talk specifics, but it's also likely going to be a big win for the production DB too, since it has to deal with less chatter.

Topics of conversation

First, the regular update works really well, but I still need to investigate what's making the packed objects version so slow. It's specifically the CopyFrom call, but I would have suspected that to be significantly faster.

Second, there were some comments about doing certain things outside of a transaction to avoid deadlocks. I don't do that any more, but I suspect we'll be okay because we do everything in single statements. The comments mentioned deadlocks were observed in tests, but I didn't observe any issues with a Gadget-side PR using a DL built from this branch for testing.

Finally, we likely now need to load a bunch more stuff into memory at once. Assuming I can make the packed objects query have gains similar to the update, how do we feel about that? If we're feeling a bit uncomfy, I can set it all up to batch CopyFrom calls so that we stay within some amount of memory per call.

[udnay]

Maybe not for this PR but I wonder if we can fold this loop into the one above, we seem to receive all objects and then write them to the DB which made sense so that we didn't block the RPC stream to do DB calls but now that we are splitting and appending to multiple arrays here, maybe we do it as we receive the objects and then do the bulk update/deletes?

[thegedge]

Yep, no problem at all to do that!

[airhorns]

@thegedge did you want to get this out?

[thegedge]

I'm back and forth, but just kept it lying around to focus on some other things for now.

I was curious what you all thought about this approach. Obviously this creates less back and forth DB chatter, but it is more complicated. With the packed object update being much slower, I think it needs a bit more digging before we ship to prod.

If we are happy with the approach in this PR though, I think the plan would be:

1. Do another run with everything on the latest main, gather some traces before and after, and compare to make sure it's overall better.
   • If packed object updates are still slow, dig deeper to learn why and see if it's something we can overcome.
   • If we can't figure out why it's slow, we can always revert that part of this PR, but keep the other update path, which definitely did get faster.
2. Throw it at a DL staging environment first, try to get something a bit more "real world".
3. Ship to prod (it should be no problem rolling back).

Which I would do after getting through some of the more urgent issues I've been debugging.