Build Docker images

Was: Create a Docker image with a Debian package built from the current Git source. This eliminates an extra round trip with a manual upload to OBS and the package getting published and fetched from apt repos. Unfortunately, doing this in a way compatible with what was previously done requires some non-trivial hacks. Since we want fairly recent OmniAuth gems, we install them from external sources directly into the resulting Docker image. ruby-faraday is used by the OAuth2 auth backend, but new versions require newer Ruby than what stretch has, so we preinstall it from packages to avoid pinning it. Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Gbp-Pq: Topic collabora/sso Gbp-Pq: Name Build-a-Docker-image-with-the-Web-UI-only.patch Build Docker images in GitLab CI Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Gbp-Pq: Topic collabora/sso Gbp-Pq: Name Build-Docker-images-in-GitLab-CI.patch Split docker-entrypoint.sh into three separate files Gbp-Pq: Topic collabora/sso Gbp-Pq: Name Split-docker-entrypoint.sh-into-three-separate-files.patch Move Docker-related files under docker/ Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Gbp-Pq: Topic collabora/sso Gbp-Pq: Name Move-Docker-related-files-under-docker.patch Clean up stale pid files on start Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Gbp-Pq: Topic collabora Gbp-Pq: Name Clean-up-stale-pid-files-on-start.patch Install and configure mstmp Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Gbp-Pq: Topic collabora Gbp-Pq: Name Install-and-configure-mstmp.patch Allow changing the session lifetime Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Gbp-Pq: Topic collabora Gbp-Pq: Name Allow-changing-the-session-lifetime.patch Preinstall apt-transport-https ca-certificates build-essential Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Simplify the db config generation Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Remove MariaDB/MySQL from the API container There’s no need to keep the database inside when it can be a separate container. Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Switch to an external memcached Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Drop Apache and TLS termination, use Puma instead of Passenger When the container is deployed, it will have a real HTTP server in front of it, so there is no need to have Apache and TLS inside. Since Passenger (at least of the version in Debian stretch) cannot be easily used without Apache, use the standard solution for such cases which is Puma, and expose OBS_FRONTEND_WORKERS (default: 4) to allow scaling it. Drop no longer necessary supervisord. Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Rename the frontend image to a more sensible name A front-end is what it really is, and it’s also the name the upstream uses for their container. Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Generalise the build process to enable building multiple images Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Teach deb-arch-to-obs-arch to work with an arbitrary argument Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Build the backend Docker image This Docker image installs obs-build and obs-productconverter from packages and everything else directly from the sources. Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> WIP: update Docker stuff Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Pass TAG argument Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Don't purge or install unnecessary packages Don't update packages Install puma Wip wup Support buster tags more deps unify log names with upstream add cloud uploader ensure correct permissions for log/tmp/sphinx init app before db so that permissions are all correct db setup needs command line mysql client init or migrate database remove more gem cruft pre-create run dir with correct perms provide our custom docker-compose move services confs into a subdir because the worker also needs some fix frontend fix scripts build worker image add worker again Build Docker images Was: Create a Docker image with a Debian package built from the current Git source. This eliminates an extra round trip with a manual upload to OBS and the package getting published and fetched from apt repos. Unfortunately, doing this in a way compatible with what was previously done requires some non-trivial hacks. Since we want fairly recent OmniAuth gems, we install them from external sources directly into the resulting Docker image. ruby-faraday is used by the OAuth2 auth backend, but new versions require newer Ruby than what stretch has, so we preinstall it from packages to avoid pinning it. Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Gbp-Pq: Topic collabora/sso Gbp-Pq: Name Build-a-Docker-image-with-the-Web-UI-only.patch Build Docker images in GitLab CI Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Gbp-Pq: Topic collabora/sso Gbp-Pq: Name Build-Docker-images-in-GitLab-CI.patch Split docker-entrypoint.sh into three separate files Gbp-Pq: Topic collabora/sso Gbp-Pq: Name Split-docker-entrypoint.sh-into-three-separate-files.patch Move Docker-related files under docker/ Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Gbp-Pq: Topic collabora/sso Gbp-Pq: Name Move-Docker-related-files-under-docker.patch Clean up stale pid files on start Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Gbp-Pq: Topic collabora Gbp-Pq: Name Clean-up-stale-pid-files-on-start.patch Install and configure mstmp Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Gbp-Pq: Topic collabora Gbp-Pq: Name Install-and-configure-mstmp.patch Allow changing the session lifetime Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Gbp-Pq: Topic collabora Gbp-Pq: Name Allow-changing-the-session-lifetime.patch Preinstall apt-transport-https ca-certificates build-essential Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Simplify the db config generation Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Remove MariaDB/MySQL from the API container There’s no need to keep the database inside when it can be a separate container. Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Switch to an external memcached Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Drop Apache and TLS termination, use Puma instead of Passenger When the container is deployed, it will have a real HTTP server in front of it, so there is no need to have Apache and TLS inside. Since Passenger (at least of the version in Debian stretch) cannot be easily used without Apache, use the standard solution for such cases which is Puma, and expose OBS_FRONTEND_WORKERS (default: 4) to allow scaling it. Drop no longer necessary supervisord. Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Rename the frontend image to a more sensible name A front-end is what it really is, and it’s also the name the upstream uses for their container. Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Generalise the build process to enable building multiple images Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Teach deb-arch-to-obs-arch to work with an arbitrary argument Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Build the backend Docker image This Docker image installs obs-build and obs-productconverter from packages and everything else directly from the sources. Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> WIP: update Docker stuff Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Pass TAG argument Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk> Don't purge or install unnecessary packages Don't update packages Install puma Wip wup Support buster tags more deps unify log names with upstream add cloud uploader ensure correct permissions for log/tmp/sphinx init app before db so that permissions are all correct db setup needs command line mysql client init or migrate database remove more gem cruft pre-create run dir with correct perms provide our custom docker-compose move services confs into a subdir because the worker also needs some fix frontend fix scripts build worker image add worker again depend on xml-structured sphinxsearch is not in bullseye adjust path

Build Docker images
ce11d76a · Andrej Shadura · 7bd1bdf3 · ce11d76a · ce11d76a · ce11d76a
Unverified Commit ce11d76a authored Jun 2, 2021 by Andrej Shadura
--- a/docker/services/backend/obsrepserver.conf
+++ b/docker/services/backend/obsrepserver.conf
+[program:obsrepserver]
+command=/usr/lib/obs/server/bs_repserver
+directory=/usr/lib/obs/server/
+stdout_logfile=/srv/obs/log/rep_server.log
+redirect_stderr=true
+autostart=True
+priority=1
+stopsignal=KILL
+killasgroup=true
+stopasgroup=true
+
--- a/docker/services/backend/obsscheduler@.conf.in
+++ b/docker/services/backend/obsscheduler@.conf.in
+[program:obsscheduler@@ARCH@]
+command=/usr/lib/obs/server/bs_sched @ARCH@
+directory=/usr/lib/obs/server/
+stdout_logfile=/srv/obs/log/scheduler_@ARCH@.log
+redirect_stderr=true
+autostart=True
+priority=1
+stopsignal=KILL
+killasgroup=true
+stopasgroup=true
+
--- a/docker/services/backend/obsservice.conf
+++ b/docker/services/backend/obsservice.conf
+[program:obsservice]
+command=/usr/lib/obs/server/bs_service
+directory=/usr/lib/obs/server/
+stdout_logfile=/srv/obs/log/src_service.log
+redirect_stderr=true
+autostart=True
+priority=1
+stopsignal=KILL
+killasgroup=true
+stopasgroup=true
+
--- a/docker/services/backend/obsservicedispatch.conf
+++ b/docker/services/backend/obsservicedispatch.conf
+[program:obsservice]
+command=/usr/lib/obs/server/bs_servicedispatch
+directory=/usr/lib/obs/server/
+stdout_logfile=/srv/obs/log/servicedispatch.log
+redirect_stderr=true
+autostart=True
+priority=1
+stopsignal=KILL
+killasgroup=true
+stopasgroup=true
+
--- a/docker/services/backend/obssrcserver.conf
+++ b/docker/services/backend/obssrcserver.conf
+[program:obssrcserver]
+command=/usr/lib/obs/server/bs_srcserver
+directory=/usr/lib/obs/server/
+stdout_logfile=/srv/obs/log/obssrcserver.log
+redirect_stderr=true
+autostart=True
+priority=1
+stopsignal=KILL
+killasgroup=true
+stopasgroup=true
+
--- a/docker/services/backend/obswarden.conf
+++ b/docker/services/backend/obswarden.conf
+[program:obswarden]
+command=/usr/lib/obs/server/bs_warden
+directory=/usr/lib/obs/server/
+stdout_logfile=/srv/obs/log/warden.log
+redirect_stderr=true
+autostart=True
+priority=1
+stopsignal=KILL
+killasgroup=true
+stopasgroup=true
+
--- a/docker/services/worker/obsworker.conf
+++ b/docker/services/worker/obsworker.conf
+[program:obsworker]
+command=%(ENV_OBS_WORKER_PATH)s/bs_worker --hardstatus --root /var/cache/build/root_%(process_num)d --statedir /var/cache/build/state_%(process_num)d --id %(ENV_OBS_WORKER_NAME)s:%(process_num)d %(ENV_OBS_WORKER_OPT)s
+process_name=%(program_name)s_%(process_num)d
+directory=%(ENV_OBS_WORKER_PATH)s
+stdout_logfile=/srv/obs/log/worker_%(process_num)d.log
+redirect_stderr=true
+autostart=True
+priority=1
+stopsignal=KILL
+killasgroup=true
+stopasgroup=true
+numprocs=%(ENV_OBS_WORKER_INSTANCES)s
+numprocs_start=1
--- a/docker/worker-docker-entrypoint.sh
+++ b/docker/worker-docker-entrypoint.sh
+#!/bin/sh -ex
+
+obsrundir=/run/obs
+workerdir=/var/cache/build
+workerbootdir="$workerdir/boot"
+obslogdir=/var/log/obs
+
+: mkdir -p "$obsrundir"
+
+: ${OBS_REPO_SERVERS:=obs-server:5252}
+
+repo_param=
+for i in $OBS_REPO_SERVERS
+do
+    repo_param="$REPO_PARAM --reposerver http://$i"
+    WORKER_CODE="http://$i"
+done
+
+: ${OBS_WORKER_NICE_LEVEL:=18}
+
+OBS_WORKER_OPT="--hardstatus $repo_param ${OBS_WORKER_JOBS:+--jobs $OBS_WORKER_JOBS}\
+ ${OBS_WORKER_CLEANUP_CHROOT:+--cleanup-chroot}\
+ ${OBS_WORKER_WIPE_AFTER_BUILD:+--wipeafterbuild}\
+ ${OBS_SRC_SERVER:+--srcserver $OBS_SRC_SERVER}\
+ ${OBS_ARCH:+--arch $OBS_ARCH} ${OBS_WORKER_OPT}"
+
+export OBS_WORKER_OPT
+
+: ${OBS_WORKER_NAME:=$(hostname)}
+
+export OBS_WORKER_NAME
+
+: ${OBS_WORKER_INSTANCES:=$(nproc)}
+
+export OBS_WORKER_INSTANCES
+
+OBS_WORKER_PATH=/usr/lib/obs/server
+
+update_worker() {
+    echo "Fetching initial worker code from $WORKER_CODE/getworkercode"
+    mkdir -p "$workerbootdir"
+    cd "$workerbootdir"
+    for retry in $(seq 10)
+    do
+        if curl -sS "$WORKER_CODE/getworkercode" | cpio --extract
+        then
+            ln -sfn . XML
+            chmod 755 bs_worker
+            return 0
+        fi
+        # we need to wait for rep server maybe
+        echo "WARNING: Could not reach rep server $WORKER_CODE. Trying again." >&2
+        sleep 10
+    done
+    echo "ERROR: Unable to reach rep server $WORKER_CODE!" >&2
+    return 1
+}
+
+if [ -n "$WORKER_CODE" ]
+then
+    update_worker
+    OBS_WORKER_PATH="$workerbootdir"
+fi
+
+export OBS_WORKER_PATH
+
+for i in $(seq 1 $OBS_WORKER_INSTANCES)
+do
+    mkdir -p $workerdir/root_$i $workerdir/state_$i
+done
+
+nice -n "$OBS_WORKER_NICE_LEVEL" /usr/bin/supervisord -n
--- a/src/api/config/environments/production.rb
+++ b/src/api/config/environments/production.rb
@@ -37,7 +37,7 @@ OBSApi::Application.configure do
  config.action_controller.perform_caching = true

  # Disable Rails's static asset server (Apache or nginx will already do this)
-  config.public_file_server.enabled = false
+  config.public_file_server.enabled = true

  # Compress JavaScripts and CSS
  config.assets.compress = true


--- a/src/api/config/options.yml.example
+++ b/src/api/config/options.yml.example
@@ -199,6 +199,7 @@ default: &default

 production:
  <<: *default
+  memcached_host: cache

 test:
  <<: *default
@@ -209,4 +210,3 @@ development:
  <<: *default
  source_host: backend
  memcached_host: cache
-
--- a/src/api/db/migrate/20191011000000_create_allowbuilddeps.rb
+++ b/src/api/db/migrate/20191011000000_create_allowbuilddeps.rb
-class CreateAllowbuilddeps < ActiveRecord::Migration
+class CreateAllowbuilddeps < ActiveRecord::Migration[4.2]
  def self.up
    create_table :allowbuilddeps do |t|
      t.integer :db_project_id, :null => false