Buffered Bulk Updates For Efficient Database Operations
Optimizing import logic with buffered bulk updates: collect updates in a buffer, perform upsert_all when reaching batch_size, reducing DB roundtrip overhead and improving performance.
Today I was working on performance optimisations for import logic. One place was operating on an array of arrays of IDs like so:
id_groups = [
[1, 2],
[42, 43],
...
]
id_groups.each do |ids|
ids.each_slice(5000) do |portion|
SomeModel.where(id: portion).update_all(some_field => some_value)
end
end
While doing an .update_all in reasonable batches is a good start, there's a slight inefficiency here in that if the groups are many and small, we'll be making an update query for each group changing only a handful of records.
We can cut down on the number of u...