Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

#DD-0015: GitLab CI Runner EC2 Instance — Ansible Onboarding

Related ADR: ADR-0015: GitLab CI Runner EC2 Instance — Ansible Onboarding

Status: Draft

Date: 2026-06-03

Authors: Dimitris Katsimpris

#Overview

This document specifies how to onboard the dedicated GitLab CI runner EC2 instance into the Themelio Ansible workflow after the manual AWS provisioning steps in DD-0014 are complete. It describes the static runner inventory, NVMe-backed runner storage, Docker and gitlab-runner installation through Galaxy roles, registration of the runner for the themelio/themelio GitLab project, Makefile entrypoints, and all required validation steps.

#Goals and Non-Goals

#Goals

• Manage the runner host through Ansible after the EC2 instance has been created manually.
• Add a static Ansible inventory entry for the runner without changing the Terraform environment inventories or the inventory generation script.
• Install and configure Docker and gitlab-runner on Debian 12.
• Mount a configured local block device and use it for Docker and runner working data.
• Register the runner for the themelio/themelio GitLab project using the Docker executor.
• Provide repeatable Makefile targets for connectivity checks and runner provisioning.

#Non-Goals

• Provisioning the EC2 instance, Elastic IP, AWS key pair, security group, or reserved instance purchase. These steps are covered by DD-0014.
• Managing the runner AWS account through Terraform or adding cross-account Terraform state access.
• Changing generate_ansible_inventory.sh or the existing Terraform-derived environment inventory workflow.
• Registering a group-level or instance-level GitLab runner.
• Changing the existing GitLab CI jobs to require runner tags.
• Building Kubernetes executor support or joining the runner host to the Kubernetes cluster.

#Design

#1. Preconditions

This design starts after DD-0014 has been completed. The runner EC2 instance already exists in the runner AWS account, runs Debian 12, has the pharos-admin-key installed for the admin user, has an Elastic IP associated, and allows SSH only from the configured GRNET and NTUA VPN CIDR ranges. The operator must know the Elastic IP before creating the static Ansible inventory entry.

#2. Static Runner Inventory

The runner is added as a hand-written inventory file at devops/ansible/inventory/runner/hosts.yml. This file is not generated from Terraform and is not modified by generate_ansible_inventory.sh.

The inventory defines a dedicated gitlab_ci_runner group with one host, pharos-gitlab-ci-runner-1. Unlike the main environment inventories, this host is reached directly through its Elastic IP and does not use the main account bastion as a ProxyJump.

all:
  vars:
    ansible_user: admin
    ansible_ssh_private_key_file: "{{ lookup('env', 'HOME') }}/.ssh/pharos-admin-key"
    ansible_become: true
    ansible_become_method: sudo
    ansible_become_user: root
    ansible_ssh_common_args: "-o ForwardAgent=yes -o StrictHostKeyChecking=no"

  children:
    gitlab_ci_runner:
      hosts:
        pharos-gitlab-ci-runner-1:
          ansible_host: <RUNNER_EIP>
          ip: <RUNNER_EIP>
          access_ip: <RUNNER_EIP>
          os_type: debian

Runner-specific variables live under devops/ansible/inventory/runner/group_vars/. Non-secret defaults go in all.yml; the GitLab API token used to create or update the runner is stored in encrypted vault.yml.

# inventory/runner/group_vars/all.yml
deployment_environment: "runner"

gitlab_ci_runner_gitlab_url: "https://gitlab.grnet.gr"
gitlab_ci_runner_project: "themelio/themelio"
gitlab_ci_runner_description: "pharos-gitlab-ci-runner-1"
gitlab_ci_runner_executor: "docker"
gitlab_ci_runner_default_image: "alpine:3.21"
gitlab_ci_runner_concurrent: 14
gitlab_ci_runner_limit: 14
gitlab_ci_runner_run_untagged: true
gitlab_ci_runner_access_level: "not_protected"
gitlab_ci_runner_storage_enabled: true
gitlab_ci_runner_storage_device: "/dev/nvme1n1"
gitlab_ci_runner_mount_path: "/mnt/gitlab-runner"

# inventory/runner/group_vars/vault.yml
vault_gitlab_ci_runner_api_token: "<encrypted GitLab PAT>"

The token must have permission to create and update a project runner for themelio/themelio. The DD does not prescribe a personal owner for the token; operational ownership should follow the same access-control process used for other Ansible Vault secrets.

#3. Playbook and Role

A dedicated playbook, devops/ansible/playbooks/gitlab-ci-runner.yml, targets only the gitlab_ci_runner group. It applies a new gitlab_ci_runner prep role and does not reuse cluster playbooks, because the runner is not a Kubernetes node and should not be part of the kube-master, kube-node, or k8s_cluster groups.

---
- name: Configure GitLab CI runner host
  hosts: gitlab_ci_runner
  become: true
  gather_facts: true

  roles:
    - role: gitlab_ci_runner
    - role: geerlingguy.docker
    - role: riemers.gitlab_runner

The role sequence is responsible for runner setup:

1. Validate the host is Debian 12.
2. Prepare and mount the configured runner storage device.
3. Create or update the GitLab-side runner using community.general.gitlab_runner.
4. Configure the Galaxy Docker and runner role variables from inventory and the GitLab API result.
5. Install Docker with geerlingguy.docker.
6. Install and configure the official gitlab-runner package with riemers.gitlab_runner.
7. Add the gitlab-runner user to the docker group.
8. Restart and verify the gitlab-runner service.

#4. Runner Storage

The role prepares a local block device configured by inventory instead of auto-detecting an AWS-specific device path. For the runner EC2 instance, gitlab_ci_runner_storage_device is set to /dev/nvme1n1, and the role mounts it at /mnt/gitlab-runner.

The role must not blindly erase disks. It verifies that the configured path exists, is a block device, is not mounted as /, and is not already mounted somewhere unexpected. It formats the device only when it has no filesystem, then mounts it persistently with ansible.posix.mount.

#5. Docker Configuration

The runner uses the GitLab Runner Docker executor, so Docker must be installed directly on the EC2 host. The playbook uses the Galaxy role geerlingguy.docker to install Docker from the official Docker Debian repository and enable the docker service.

The prep role sets docker_daemon_options for geerlingguy.docker so Docker stores container/image data under the mounted runner disk:

{
  "data-root": "/mnt/gitlab-runner/docker"
}

The gitlab-runner user is added to the docker group so runner jobs can start job containers through the local Docker daemon.

#6. GitLab Runner Installation

The playbook uses the Galaxy role riemers.gitlab_runner to install gitlab-runner from the official GitLab Debian package repository and manage /etc/gitlab-runner/config.toml. This keeps runner installation and updates aligned with normal Debian package management while avoiding custom registration and template tasks in this repository.

#7. Runner Registration

Runner registration has two parts: creating the runner record in GitLab and writing the local gitlab-runner configuration on the EC2 host.

The role uses community.general.gitlab_runner to create or update a project-scoped runner for themelio/themelio in https://gitlab.grnet.gr. The runner description is pharos-gitlab-ci-runner-1, which is used as the stable identifier for idempotency. The runner accepts untagged jobs because the current Themelio CI configuration does not define runner tags.

The GitLab-side runner is configured with:

The 14 value comes from ADR-0014's sizing target: two concurrent pipelines, each with up to seven parallel lint jobs.

On first provisioning, the prep role passes the runner authentication token returned by the GitLab API to riemers.gitlab_runner with gitlab_runner_registration_token_type: authentication-token. The Galaxy role then registers the local runner non-interactively and writes the local runner entry into /etc/gitlab-runner/config.toml.

If the GitLab-side runner already exists but the local host has no matching runner configuration, the role fails with a clear message instead of silently creating a duplicate runner. Recreating a missing local configuration requires an explicit operator action, such as deleting the stale GitLab runner record or setting a dedicated recreate variable.

The generated /etc/gitlab-runner/config.toml uses the Docker executor and NVMe-backed runner paths:

concurrent = 14
check_interval = 0

[[runners]]
  name = "pharos-gitlab-ci-runner-1"
  url = "https://gitlab.grnet.gr"
  executor = "docker"
  limit = 14
  builds_dir = "/mnt/gitlab-runner/builds"
  cache_dir = "/mnt/gitlab-runner/cache"

  [runners.docker]
    image = "alpine:3.21"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]

#8. Makefile Integration

Runner provisioning gets explicit Makefile targets instead of being folded into the normal environment targets. This keeps the runner workflow separate from dev, staging, production, and local, and avoids changing the Terraform-backed inventory generation flow.

The Makefile adds:

Both targets use the same virtualenv, SSH key, vault handling, and deploy wrapper patterns as the existing Ansible targets.

#9. Controller Dependencies

The community.general.gitlab_runner module uses the GitLab API through python-gitlab. Because the role uses the modern runner creation workflow, the Ansible controller environment must include python-gitlab >= 4.0.0.

The Ansible bootstrap dependencies are updated so make bootstrap installs this package before the runner playbook is used. The Galaxy dependency installer also installs standalone roles into devops/ansible/vendor/galaxy:

• geerlingguy.docker
• riemers.gitlab_runner

#Testing Strategy

The implementation is verified in three layers.

First, run static Ansible checks for the new playbook and role:

devops/ansible/scripts/ci/run_tests_local.sh --files playbooks/gitlab-ci-runner.yml,roles/gitlab_ci_runner/tasks/main.yml yamllint
devops/ansible/scripts/ci/run_tests_local.sh --files playbooks/gitlab-ci-runner.yml,roles/gitlab_ci_runner/tasks/main.yml lint

Second, verify connectivity and idempotent provisioning from devops/ansible:

make ping-runner
make gitlab-ci-runner
make gitlab-ci-runner

Third, validate the configured host and GitLab integration:

ssh -i ~/.ssh/pharos-admin-key admin@<RUNNER_EIP>

sudo systemctl status docker
sudo systemctl status gitlab-runner
sudo gitlab-runner verify
df -h /mnt/gitlab-runner

The GitLab project settings for themelio/themelio should show pharos-gitlab-ci-runner-1 as online. A normal branch pipeline should start on the runner without adding tags: to CI jobs, and a manual build pipeline should run kaniko jobs successfully.

#Deployment & Rollout

The rollout sequence is:

1. Complete DD-0014 and verify SSH access to the runner EIP.
2. Create inventory/runner/hosts.yml with the runner EIP.
3. Add non-secret runner variables in inventory/runner/group_vars/all.yml.
4. Add vault_gitlab_ci_runner_api_token to encrypted inventory/runner/group_vars/vault.yml.
5. Run make bootstrap and make galaxy if the Ansible controller environment is not already prepared.
6. Run make ping-runner.
7. Run make gitlab-ci-runner.
8. Confirm the runner is online in GitLab and trigger a CI pipeline.

Rollback is manual. If provisioning fails before runner registration, fix the failed host-side configuration and re-run the playbook. If the runner has been registered but should not receive jobs, pause or delete pharos-gitlab-ci-runner-1 from the themelio/themelio project runner settings in GitLab. If the EC2 instance is replaced, update inventory/runner/hosts.yml with the new Elastic IP before re-running Ansible.

#Open Questions

None. ADR-0015's implementation choices are resolved in this DD as follows:

• Runner scope: project-scoped runner for themelio/themelio.
• Concurrency: concurrent = 14 and limit = 14, matching the ADR-0014 two-pipeline lint workload.
• Registration secret: GitLab API token stored in encrypted Ansible Vault.
• Controller dependencies: make bootstrap installs the required Python dependencies, including python-gitlab >= 4.0.0, before the runner playbook is used.

#References

• ADR-0014: GitLab CI Runner on Dedicated AWS EC2 Instance
• DD-0014: GitLab CI Runner EC2 Instance — Manual Operator Steps
• ADR-0015: GitLab CI Runner EC2 Instance — Ansible Onboarding
• GitLab Runner installation on Debian/Ubuntu
• GitLab Runner Docker executor
• Ansible community.general.gitlab_runner module
• Ansible Galaxy geerlingguy.docker role
• Ansible Galaxy riemers.gitlab_runner role