Skip to content

Chop TestJob output compression loop

Paweł Wieczorek requested to merge pawiecz/lava:wip/pawiecz/chop-jobs-compression-loop into collabora/staging

This patch uses similar approach to the one from "handle_rm" (L294) to divide TestJob output compression task into smaller chunks.

Enumerating the whole query set (used previously) may result in creating a "SELECT" SQL statement that puts high load on the database. Chopping the query set into slices allows TestJob output compression command to proceed even when operating on large amounts of data (in tested case: 4 months of job runs, about 1.5M records).

Approach from "handle_rm" could not be applied 1:1 because TestJob output compression task doesn't alter the TestJob table - it only uses its data for accessing job output. That is why the slice boundries are not fixed and have to be moved.

This type of loop also runs one additional iteration when the query set slice is empty before it finally breaks. This might get fixed in future for both "handle_rm" and "handle_compress".

Task behavior stays the same, i.e. it sleeps every $SIZE items when "slow" is enabled. Default frequency is also not changed (100).

Signed-off-by: Paweł Wieczorek pawiecz@collabora.com

Merge request reports