GitLab

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

The current stage separation is the minimum needed to make the dependencies
work, splitting things may result in a easier to understand pipeline.

Since we use `needs:` to explicitly track dependencies between jobs,
adding new stages does not impact the ability of the pipeline to keep the
runners busy.

The artifacts build stage and the upload one are currently by far the
largests since they contain a mixed set of different jobs.

This has the drawback that the template files are quite large and that
it's not easy to track progress across jobs of the same kind using the
GitLab pipeline diagram, since a lot of them end up in the same, overly
tall column.

Try to make things slightly more understandable by splitting them in
smaller chunks, trading vertical scrolling with horizontal scrolling
in the pipeline diagram. The latter tradeoff is usually bad, but in this
case it is compensated by having jobs nicely split in boxes by kind.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Always try to initialize the remote repository, ostree is smart enough that
calling `ostree init` on a existing repository won't do anything.

After that, unconditionally use ostree-push to upload the local commit:
* makes the logic simpler
* avoids races where two jobs may end up doing rsync at the same time
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Some jobs currently take more than 1h to complete.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Complete the pipeline definition adding the missing jobs to cover all
the reference images.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

In GitLab CI we need to pass the devrootpack as an artifact before anything
gets uploaded to the publishing server, so let's assume the devrootpack
is a local file.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Avoid re-uploading and storing artifacts from previous stages and try to upload
only what gets generated from each specific job.

This is important to:
* reduce the time needed to upload artifacts back to the controller
* reduce the amount of data in the artifacts storage
* reduce the traffic across the controller and the runners, which is important
  to keep costs under control with cloud runners as the outbound traffic is
  not free
* avoid having to exclude the local ostree repository
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Briefly document the purposes of the sections in the main file.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Restructure the pipeline to make it less noisy. Rather than keeping everything
in a single file, keep only the short definitions of the top-level jobs in the
main file and then `include:` the actual templates, split by stage.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Prepare for instantiating the current steps for all the supported image
variants and boards.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Set $PIPELINE_VERSION to something like 20200211.0, using the CI caching to
communicate the latest build revision across pipeline runs.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

With the goal of moving the whole image-building pipeline to GitLab CI,
start by defining the flow for the minimal amd64 artifacts (apt image,
ostree image, ostree bundles, installer).

After the flow for a single image flavor is fully defined we can factor out the
common bits in templates and use `extends:` to instantiate the actual jobs with
the right variables.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Move from `$image.filelist.gz` to `$image.img.filelist.gz` to better match
the `$image.img.gz` artifact.

For OSTree commits, produce `$image.commit.{pkg,file}list.gz` to avoid
conflicting with the filelist generated for the image.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Error from hawkBit are returned with this JSON:

    {
      "timestamp": "2020-02-10T13:38:27.023+0000",
      "status": 500,
      "error": "Internal Server Error",
      "message": "org.springframework.jdbc.UncategorizedSQLException: null; uncategorized SQLException; SQL state [28000]; error code [0]; HikariPool-1 - Connection is not available, request timed out after 30020ms.; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30020ms.",
      "path": "/rest/v1/softwaremodules"
    }

The `.errorCode` property the current code was looking for does not exist,
causing the script to fail silently.

Try instead to look for the `status` optional property, check it is indeed an
erro and fail only in that case.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Update everything, including the documentation, to point to v2021dev1.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

We no longer have a wiki. Replace the link with one to the project
homepage.
Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

As iptables-persitent package is GPLv3, replace it by a systemd unit in
charge to load iptables rule on boot.
Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Because we weren't using any btrfs specific features and wanted to
switch back to the more mature ext4 file system

And also, because btrfs-tools package contains some GPLv3 code and
thus is unsuitable to be released on our target images
Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Avoid issues due to missing flags in the ancient non-GPLv3 version of `sed`
shipped by default.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

Upstream MR 186 for Debos protect the 'resolv.conf' from
changes caused by systemd-nspawn and debootstrap action.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

systemd-nspawn set own resolv.conf configuration during UEFI bootloader
setup and this configuration became the part of the final image.
Since we are using the Connman for the name management we end with
incorrect default configuration in resolv.conf.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

In light of e21ecdb6 fixing an issue caused by a missing `def` that makes a
variable to be global and thus unwillingly overridden, do a sanity check on the
whole file and add some missing `def`.

Even if their lack currenly seem harmless, it may prevent issues when moving
things around.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

For a long time we had a race in the sysroot index files generation that
sometimes caused one of the links contained there to point to the wrong
architecture, for instance the `armhf` index pointing to `arm64`.

Commit a221b5f4 was an attempt to fix it, but didn't improve the situation.
Hovewer, it gave us some additional output that we could use to reason a bit
more about the issue.

For instance, the v2019 20191122.0 job has this in the logs at
https://jenkins.apertis.org/view/apertis-v2019/job/apertis-v2019/job/images/job/debos-image-build/140/consoleText:

    [Pipeline] echo
    sysroot sysroot/v2019/sysroot-apertis-v2019-armhf
    version=v2019 20191122.0
    url=https://images.apertis.org/daily/v2019/20191122.0/arm64/sysroot/sysroot-apertis-v2019-arm64-20191122.0.tar.gz

The relevant code:
    sysrootname = "sysroot-${osname}-${release}-${architecture}-${env.PIPELINE_VERSION}"
    sysrooturl = "${image_url_prefix}/daily/${release}/${env.PIPELINE_VERSION}/${architecture}/sysroot/${sysrootname}.tar.gz"
    sh(script: """
        cd ${PIPELINE_VERSION}/${architecture}/${type}
        debos ${debosarguments} \
          --show-boot \
          -t architecture:${architecture} \
          -t ospack:ospack_${release}-${architecture}-${type}_${PIPELINE_VERSION} \
          -t sysroot:${sysrootname} \
          ${WORKSPACE}/${osname}-sysroot.yaml; \
        """)

    // Generate sysroot metadata
    def metadata_file = "sysroot/${release}/sysroot-${osname}-${release}-${architecture}"
    def metadata_contents = "version=${release} ${PIPELINE_VERSION}\nurl=${sysrooturl}\n"
    echo "sysroot ${metadata_file}\n${metadata_contents}"
    writeFile file: metadata_file, text: metadata_contents

From the output we can conclude that `metadata_file` is correct, while
`metadata_contents` has the wrong arch, which comes from `sysrooturl`.

Note the lack of `def` before `sysrooturl`. That means that Groovy is creating
a **global** variable. So every time the stage is run, the shared global
variable is overridden: in this case, the stage is run for `armhf` first and
before it reaches the `Generate sysroot metadata` section the `arm64` stage is
run as well, overridding `sysrooturl`.

Adding a `def` there should finally fix this longstanding issue.

See https://phabricator.apertis.org/T6317 for more details.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

As we now have a working graphics stack for armhf boards (specifically the
Sabrelite reference hardware), enable the target image to be built for
this architecture so that we have a stock image which can be used to test
this.
Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

The target image includes most of the components of a GL stack, however
it is lacking the hardware backends to enable this to be used. As debian
(and thus Apertis) includes these hardware backends in a single package
include these in the target image to the GL stack to be used.
Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

NTP and dnsmasq are used for DUT management and setup during the tests.
Services are disabled by default.
(See APERTIS-6509)
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>