Use username and nickname fields to retrieve the username,
since some backends use username (e.g. GitLab), while others use
nickname (OIDC).

See https://phabricator.apertis.org/T9495



Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

nslookup seems to query *all* the search domains and return 1 if *any*
fail, which in this case means that it never actually succeeds. nc
already will fail if DNS doesn't work, so just remove the nslookup loop
and rely solely on nc instead.

https://phabricator.apertis.org/T9457



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Older verisons of Puma had a bug where chunked uploads would result in
an empty body:

https://github.com/puma/puma/issues/1839

This breaks OBS file uploads that rely chunked transfers, which is
generally the standard mechanism used by http clients for uploading
large files. A fix was backported to 4.3.7:

https://github.com/puma/puma/pull/2496

so this MR updates Puma to the latest 4.3.x release to include this
fix.

(The only reason we didn't see this in practice on the Collabora OBS is
that the reverse proxy in front would buffer the request data and send
a single non-chunked one over to OBS.)

https://phabricator.apertis.org/T9457



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

These addresses are often used by Docker and other cloud setups, and
because of their often complex configurations it’s often easier to
permit the whole range, since usually the operating system’s IP filter
will be configured to block improper access anyway.

OBS_TRUST_PRIVATE_NET is set by default by the backend Docker
entrypoint.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Stephan Kulow authored Jun 30, 2021 and Ariel D'Alessandro committed Sep 26, 2022

Relationships can be users or groups and admins expect involved
groups to see the hidden projects

Fixes #11302

[Backport to v2.10.11]
Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Let's remove the `aptly.log` module specific file. Instead log to
stdout, so the output goes to the caller module log. This makes
debugging the trace of events easier.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Ariel D'Alessandro authored Jun 23, 2022 and Sjoerd Simons committed Aug 18, 2022

Currently, aptly module is logging to `/srv/obs/log/aptly.log` without a
timestamping, which makes it difficult to debug.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Andrej Shadura authored Nov 22, 2021 and Sjoerd Simons committed Aug 17, 2022

When logging in with e.g. OpenID Connect/Azure, two email addresses
may be provided, one in "email" field and another in "username" or
"nickname". Since this is exactly the opposite of what the separate
Azure backend does, migration from Azure to OpenID Connect/Azure
needs to try both emails which may be different.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Aptly is currently re-publishing (dropping and publishing again)
repositories when the project configuration is changed. This is
performed to update the components and architectures of the published
distribution based on the updated project configuration.

OBS/aptly contains a hack allowing the usage of slashes `/` in the
publish distribution field, which we're currently using to have the
snapshots published in the same prefix and using a distribution name
with the form: `$release/snapshots/$timestamp`. This makes the snapshots
folder to be inside the same `dists` directory where the repositories
are published.

There's a bug in the above implementation, where:
* OBS project configuration is updated, triggering a re-publishing of
  the distribution.
* Aptly drops the published distribution.
* If aptly doesn't have any other distribution published in the same
  prefix, it will remove the entire `dists` folder, with all the
  snapshots in it.
* Aptly publishes the distribution again.

This bug is making aptly to drop all the snapshots, as it assumes these
can't be stored within the published distribution.

Let's HACK this quickly by disabling the re-publishing temporarily. Note
that OBS will create the aptly repositories and publish them on the
first project configuration, but it won't re-publish it at all on
further config updates.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

The installed aptly version now allows to disable bzip2 compression for
index files. Enable this to speed up aptly publishing, as speed
performance is more important than Package file size in this case.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

The `bs_aptly_snapshot_policy` script run snapshot retention policies.
Each snapshot retention policy is defined by a YAML file, stored by
default at `/srv/obs/aptly/snapshot.policy.d/`.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

We don't use the latest symlink, just the latest.txt timestamp. So drop
it.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

The old reprepro publisher hook script, made the assumption that
-security and -updates suffixes always existed, but this is not the case
with aptly. This commit updates the aptly publisher hook script to check
if the snapshot directory exists for those suffixes and ignores it if it
doesn't. In that case, a warning is gently printed to stderr.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

In order to rsync this properly, the symlink needs to be relative,
otherwise it's not found.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Ariel D'Alessandro authored Jun 13, 2022 and Sjoerd Simons committed Jun 16, 2022

The latest published snapshot required latest/ dir and latest.txt file
to point to it. Generate those.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Ariel D'Alessandro authored Jun 13, 2022 and Sjoerd Simons committed Jun 16, 2022

Add a function to read aptly global configuration options.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Ariel D'Alessandro authored Jun 13, 2022 and Sjoerd Simons committed Jun 16, 2022

Reprepro was setting the snapshot Codename (a.k.a. Suite) to
snapshots/$distro/$timestamp. Let's use the same convention for
compatability.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

The required aptly version includes custom collabora patches. Install
it instead of the upstream version.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

The gpg-key needs to be passed as an argument when a repository is
re-published. This is now missing, so the wrong gpg-key is being used.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

We want to publish sources by default, which needs to be added to the
architectures list on publishing.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Generating Contents indexes takes ages and and require a lot of data.
Basically, aptly scan through the contents of each .deb file.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

This aptly script performs a database cleanup to remove information
about unreferenced packages. Also deletes files in the package pool that
aren't used by packages anymore.

Aptly documentation suggests that it's a good idea to run this command
after massive deletion of mirrors, snapshots or local repos.

This commit adds the script to be called in a weekly basis by cron.
Note that with this configuration it will be called each Sunday at
00:00 hours.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Cron will be used to run periodic jobs on the backend. Let's add this to
the processed launched and monitored by supervisor.

Such cron functionality is required by aptly, which has been now
integrated to OBS. A few related scripts need to be run on the backend
to perform maintenance tasks.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Package binaries are appended with the release suffix, so the target
filenames need to be updated. For some strange reason, this wasn't
working before, as the obs-build submodule wasn't properly initialized.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

This commit adds helper functions to the aptly module in order to remove
and drop published snapshots.

The script `src/backend/bs_aptly_drop_snapshot` is added as the default
reference logic to drop snapshots that were created using
`src/backend/bs_published_hook_aptly_snapshot`.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

This commit adds test scripts for reprepro to aptly repository
migration.

While here, rename aptly test.sh script as aptly.sh to be more specific
about its functionality.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Split common functions from aptly test that could be used by further
test scripts.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

The script `./src/backend/bs_aptly_migration` can be used to migrate
reprepro repositories to aptly.

Note that reprepro repositories and snapshots cleanup is not implemented
by this script.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Extend the testing configuration to define repositories for both
reprepro and aptly.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

This is the default reprepro snaphost hook script. Let's add it until
OBS reprepro support is finally dropped.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

This function cleans up all contents from a repository.

In order to re-create snapshots from reprepro, we may wanna reset a
repository, then add all packages from an old reprepro snapshot and
create the aptly snapshot from there.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Update the prefix used in the aptly test, to match the one used by
reprepro repositories in general.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

These functions will be used from other perl scripts as well. Let's make
it available from the aptly module.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Andrej Shadura authored May 02, 2022 and Sjoerd Simons committed May 12, 2022

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Ariel D'Alessandro authored Mar 29, 2022 and Sjoerd Simons committed May 12, 2022

This commit adds tests for the OBS aptly integration.

See tests/aptly/README.md for instructions on how to run the test suite.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Ariel D'Alessandro authored Feb 03, 2022 and Sjoerd Simons committed May 12, 2022

This commit integratates aptly backend. The following features are
supported:

* OBS repositories gets automatically created/removed on aptly.
* Each arch with the publish flag enabled in a repository, get published
  on aptly.
* Package binaries get added to the aptly repository right after they
  get built. Old versions are automatically removed.
* Every time a package is published, a snapshot is taken and published.

Related documentation is added to `README.aptly.md`:
* aptly configuration files.
* snapshot and db-cleanup scripts.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Andrej Shadura authored May 12, 2022 and Sjoerd Simons committed May 12, 2022

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>