Skip to content

Use new Lava throttled API

Edmund Smith requested to merge eds/throttling into main

This prevents us from saturating a Lava instance with more requests than it can handle when we have many jobs to monitor. The numbers chosen here could easily be made configurable, but for now, a maximum of one high level request per second, and a maximum of 20 concurrent queries are reasonable defaults.

I've added a commit for debug tracing, because the conditions on correctly using permits are more subtle than it appears at first. I've written up the problem areas in the commit message, but in brief, when the number of permits is small, you could get into a deadlock situation because the permit to loop over jobs() was held while the permit to read logs() was requested; especially when the call to jobs() is slow, you can end up with all the available tokens owned by tasks calling jobs() that are blocked on obtaining a second permit for logs(). The trace code makes debugging this simple, but is annoying enough to write that it feels like it belongs in the codebase.

This needs lava-api!20 (merged) before it can merge.

Edited by Edmund Smith

Merge request reports