Use new Lava throttled API
This prevents us from saturating a Lava instance with more requests than it can handle when we have many jobs to monitor. The numbers chosen here could easily be made configurable, but for now, a maximum of one high level request per second, and a maximum of 20 concurrent queries are reasonable defaults.
I've added a commit for debug tracing, because the conditions on
correctly using permits are more subtle than it appears at first. I've
written up the problem areas in the commit message, but in brief,
when the number of permits is small, you could get into a deadlock
situation because the permit to loop over jobs()
was held while the
permit to read logs()
was requested; especially when the call to jobs()
is slow, you can end up with all the available tokens owned by tasks
calling jobs()
that are blocked on obtaining a second permit for logs()
.
The trace code makes debugging this simple, but is annoying enough to
write that it feels like it belongs in the codebase.
This needs lava-api!20 (merged) before it can merge.