Draft: Track job failures correctly
Up until now, we have used Health::Complete
to indicate job success
in Gitlab pipelines, and Heath::Incomplete
to indicate job failure.
This is not in fact how LAVA uses these health values, because whether
a job completed does not directly indicate whether the tests within it
passed. An incomplete job will always be a failure, because it will
have aborted. But a complete job may contain many (even all) test
cases having failed, even if the job itself did not abort.
The actual logic for whether the job should be reported to Gitlab as
failed is unfortunately slightly more complicated. We need to identify
results
messages within the log, and check whether any of them are
failures. That means we need to make sure we check the logs, before we
report any successful status to Gitlab.
The biggest change is that the monitor
mode now outputs the logs
(for all jobs) directly. There's no particular reason not to do this,
since it has to query the logs anyway in order to accurately report.
success or failure.
This MR is draft pending final testing of the submit
command,
and full testing of the monitor
command.