The compression is not taken into account when the key is "ramdisk". The
logging was wrong by indicating that the file is decompressed (while it's not).

Also improve _url_to_fname by using self.

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

Keys should not include any directory specific characters.

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

The ramdisk should not be uncompressed.

Found by meta-lava and confirmed on staging:
https://staging.validation.linaro.org/scheduler/job/268035



Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

The action is now taking as parameter the dictionary of the element to download.
Instead of guessing which dictionary to use between parameters["images"][key]
and parameters[key], the action is now using the params passed as argument.

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

No need to call .close() explicitly.

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

_url_to_fname_suffix returns a tuple but only the first element is ever used.

Also make decompress_command_map a class variable (will save some memory).

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

"remote" is an alias for these dictionaries.

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

Remove the need to prepend every commands under nice.

Also make lava-run nicer with the system servers like tftpd or nfsd.

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

The later was already a copy.

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

The corresponding namespace data was overriden by another value.

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

When downloading an archive, extract the full archive into the download
directory.

Also remove the (now unused) feature to untar only some files in the archive.

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

http_url_format_string should be a format string where '%s' will be replaced by
the url to the resource to cache.

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

For checking hash of downloaded files, LAVA supports only md5 and
sha256.
This patch adds support for using sha512 hashs.

Only check for the download size when this size is known.

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>
Reviewed-by: Neil Williams <neil.williams@linaro.org>
Tested-by: _Marge_Bot <https://git.lavasoftware.org/lava/lava/merge_requests/315>

Actually LAVA fail to download objects when "Content-Encoding: gzip" is used by the HTTP server.
This patch permits to always get object size, by disabling "Content-Encoding: gzip" for the first HEAD request (or GEt if HEAD was failling).

Black formatting moved some of the comments used to
override bandit warnings.

Signed-off-by: Neil Williams <neil.williams@linaro.org>
Reviewed-by: Rémi Duraffort <remi.duraffort@linaro.org>
Tested-by: _Marge_Bot <https://git.lavasoftware.org/lava/lava/merge_requests/289>

The key is never used and does change the schema that we might want to fully
validate.

Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>
Reviewed-by: Neil Williams <neil.williams@linaro.org>
Tested-by: _Marge_Bot <https://git.lavasoftware.org/lava/lava/merge_requests/262>

Exclude action.py and parser.py which need some
formatting changes.
Change the black task to exclude the remaining files.

Signed-off-by: Neil Williams <neil.williams@linaro.org>

In fact, the validation() function does already check that the resource exists.
So if downloading fails, it's likelly to be an issue with the infrastructure.

Also fix the timeout duration when retrying.
The fix is in fact not perfect because we have no way to get the parents
timeouts that might be smaller.

Closes #131

Reduce the noise in the SAST report further.
Override bandit warnings in lava-dispatcher as bandit is
overly based on a web UI mindset.
Add overrides and fixes for files in ./share/

Signed-off-by: Neil Williams <neil.williams@linaro.org>

It isn't possible to download a directory as part of a
LAVA URL. Fail in validate() instead of waiting for a
later failure.

Signed-off-by: Neil Williams <neil.williams@linaro.org>

If the data stream closes during a download, don't just assume the
download is finised. If we have an expected size for calculating
the progress, use that to check the file has fully downloaded.

Change-Id: I7d6790d867151c991286f46deb0ce320bbe39a37

Since python 3.3, IOError, socket.error has been replaced by OSError.
See https://docs.python.org/3/library/exceptions.html#OSError

Change-Id: I7318cadf729f9072f80b9537a27eec6da057d993

Change-Id: I9d61efc457fb5e49ac1b3ad56acaa1b453b9cc4a

Change-Id: Ic5c1b533c3479b4fdb8cbfe56d051022d1efa779

Change-Id: I4b3d8df3ab04af5bc3f5aa4a672022d03e665275

This argument is not used anymore.

Change-Id: I9cc98ed14003e79110c820149570cc1248549535

Change-Id: I9b367e3399d53ac5667f8ff615384d030769f1c1

Make a clear separation between lava_server and lava_dispatcher
binary packages so that a master does not need all the dependencies
of lava-dispatcher installed (and does not necessarily have a
local worker).
Update documentation on how lava-common can be used.

Change-Id: I040a44c7ce6d853524ccc33e08df593b0d14c54f

- Test decompressed filesize, shasum and md5sum, for bz2, gz and xz
- Test we don't rename files we aren't decompressing (zip)
- Fix zip files being renamed when not decompressed

Change-Id: I66577d4943029c40eefc285d380892c0d844ab00

Change-Id: I7ad41ca0142a2ed0bcb54e310f725362aca22798