---
title: "DD-0015: GitLab CI Runner EC2 Instance — Ansible Onboarding"
---

# DD-0015: GitLab CI Runner EC2 Instance — Ansible Onboarding

**Related ADR:** [ADR-0015: GitLab CI Runner EC2 Instance — Ansible Onboarding](/adr/0015-gitlab-ci-runner-ec2-ansible-onboarding)

**Status:** Draft

**Date:** 2026-06-03

**Authors:** Dimitris Katsimpris

## Overview

This document specifies how to onboard the dedicated GitLab CI runner EC2 instance into the Themelio Ansible workflow after the manual AWS provisioning steps in DD-0014 are complete. It describes the static runner inventory, NVMe-backed runner storage, Docker and `gitlab-runner` installation through Galaxy roles, registration of the runner for the `themelio/themelio` GitLab project, Makefile entrypoints, and all required validation steps.

## Goals and Non-Goals

### Goals

- Manage the runner host through Ansible after the EC2 instance has been created manually.
- Add a static Ansible inventory entry for the runner without changing the Terraform environment inventories or the inventory generation script.
- Install and configure Docker and `gitlab-runner` on Debian 12.
- Mount a configured local block device and use it for Docker and runner working data.
- Register the runner for the `themelio/themelio` GitLab project using the Docker executor.
- Provide repeatable Makefile targets for connectivity checks and runner provisioning.

### Non-Goals

- Provisioning the EC2 instance, Elastic IP, AWS key pair, security group, or reserved instance purchase. These steps are covered by DD-0014.
- Managing the runner AWS account through Terraform or adding cross-account Terraform state access.
- Changing `generate_ansible_inventory.sh` or the existing Terraform-derived environment inventory workflow.
- Registering a group-level or instance-level GitLab runner.
- Changing the existing GitLab CI jobs to require runner tags.
- Building Kubernetes executor support or joining the runner host to the Kubernetes cluster.

## Design

### 1. Preconditions

This design starts after DD-0014 has been completed. The runner EC2 instance already exists in the runner AWS account, runs Debian 12, has the `pharos-admin-key` installed for the `admin` user, has an Elastic IP associated, and allows SSH only from the configured GRNET and NTUA VPN CIDR ranges. The operator must know the Elastic IP before creating the static Ansible inventory entry.

### 2. Static Runner Inventory

The runner is added as a hand-written inventory file at `devops/ansible/inventory/runner/hosts.yml`. This file is not generated from Terraform and is not modified by `generate_ansible_inventory.sh`.

The inventory defines a dedicated `gitlab_ci_runner` group with one host, `pharos-gitlab-ci-runner-1`. Unlike the main environment inventories, this host is reached directly through its Elastic IP and does not use the main account bastion as a ProxyJump.

```yaml
all:
  vars:
    ansible_user: admin
    ansible_ssh_private_key_file: "{{ lookup('env', 'HOME') }}/.ssh/pharos-admin-key"
    ansible_become: true
    ansible_become_method: sudo
    ansible_become_user: root
    ansible_ssh_common_args: "-o ForwardAgent=yes -o StrictHostKeyChecking=no"

  children:
    gitlab_ci_runner:
      hosts:
        pharos-gitlab-ci-runner-1:
          ansible_host: <RUNNER_EIP>
          ip: <RUNNER_EIP>
          access_ip: <RUNNER_EIP>
          os_type: debian
```

Runner-specific variables live under `devops/ansible/inventory/runner/group_vars/`. Non-secret defaults go in `all.yml`; the GitLab API token used to create or update the runner is stored in encrypted `vault.yml`.

```yaml
# inventory/runner/group_vars/all.yml
deployment_environment: "runner"

gitlab_ci_runner_gitlab_url: "https://gitlab.grnet.gr"
gitlab_ci_runner_project: "themelio/themelio"
gitlab_ci_runner_description: "pharos-gitlab-ci-runner-1"
gitlab_ci_runner_executor: "docker"
gitlab_ci_runner_default_image: "alpine:3.21"
gitlab_ci_runner_concurrent: 14
gitlab_ci_runner_limit: 14
gitlab_ci_runner_run_untagged: true
gitlab_ci_runner_access_level: "not_protected"
gitlab_ci_runner_storage_enabled: true
gitlab_ci_runner_storage_device: "/dev/nvme1n1"
gitlab_ci_runner_mount_path: "/mnt/gitlab-runner"
```

```yaml
# inventory/runner/group_vars/vault.yml
vault_gitlab_ci_runner_api_token: "<encrypted GitLab PAT>"
```

The token must have permission to create and update a project runner for `themelio/themelio`. The DD does not prescribe a personal owner for the token; operational ownership should follow the same access-control process used for other Ansible Vault secrets.

### 3. Playbook and Role

A dedicated playbook, `devops/ansible/playbooks/gitlab-ci-runner.yml`, targets only the `gitlab_ci_runner` group. It applies a new `gitlab_ci_runner` prep role and does not reuse cluster playbooks, because the runner is not a Kubernetes node and should not be part of the `kube-master`, `kube-node`, or `k8s_cluster` groups.

```yaml
---
- name: Configure GitLab CI runner host
  hosts: gitlab_ci_runner
  become: true
  gather_facts: true

  roles:
    - role: gitlab_ci_runner
    - role: geerlingguy.docker
    - role: riemers.gitlab_runner
```

The role sequence is responsible for runner setup:

1. Validate the host is Debian 12.
2. Prepare and mount the configured runner storage device.
3. Create or update the GitLab-side runner using `community.general.gitlab_runner`.
4. Configure the Galaxy Docker and runner role variables from inventory and the GitLab API result.
5. Install Docker with `geerlingguy.docker`.
6. Install and configure the official `gitlab-runner` package with `riemers.gitlab_runner`.
7. Add the `gitlab-runner` user to the `docker` group.
8. Restart and verify the `gitlab-runner` service.

### 4. Runner Storage

The role prepares a local block device configured by inventory instead of auto-detecting an AWS-specific device path. For the runner EC2 instance, `gitlab_ci_runner_storage_device` is set to `/dev/nvme1n1`, and the role mounts it at `/mnt/gitlab-runner`.

The role must not blindly erase disks. It verifies that the configured path exists, is a block device, is not mounted as `/`, and is not already mounted somewhere unexpected. It formats the device only when it has no filesystem, then mounts it persistently with `ansible.posix.mount`.

### 5. Docker Configuration

The runner uses the GitLab Runner Docker executor, so Docker must be installed directly on the EC2 host. The playbook uses the Galaxy role `geerlingguy.docker` to install Docker from the official Docker Debian repository and enable the `docker` service.

The prep role sets `docker_daemon_options` for `geerlingguy.docker` so Docker stores container/image data under the mounted runner disk:

```json
{
  "data-root": "/mnt/gitlab-runner/docker"
}
```

The `gitlab-runner` user is added to the `docker` group so runner jobs can start job containers through the local Docker daemon.

### 6. GitLab Runner Installation

The playbook uses the Galaxy role `riemers.gitlab_runner` to install `gitlab-runner` from the official GitLab Debian package repository and manage `/etc/gitlab-runner/config.toml`. This keeps runner installation and updates aligned with normal Debian package management while avoiding custom registration and template tasks in this repository.

### 7. Runner Registration

Runner registration has two parts: creating the runner record in GitLab and writing the local `gitlab-runner` configuration on the EC2 host.

The role uses `community.general.gitlab_runner` to create or update a project-scoped runner for `themelio/themelio` in `https://gitlab.grnet.gr`. The runner description is `pharos-gitlab-ci-runner-1`, which is used as the stable identifier for idempotency. The runner accepts untagged jobs because the current Themelio CI configuration does not define runner tags.

The GitLab-side runner is configured with:

| Setting | Value |
|---|---|
| Scope | Project runner for `themelio/themelio` |
| Description | `pharos-gitlab-ci-runner-1` |
| Executor | `docker` |
| Default image | `alpine:3.21` |
| Run untagged jobs | `true` |
| Access level | `not_protected` |
| Global `concurrent` | `14` |
| Per-runner `limit` | `14` |

The `14` value comes from ADR-0014's sizing target: two concurrent pipelines, each with up to seven parallel lint jobs.

On first provisioning, the prep role passes the runner authentication token returned by the GitLab API to `riemers.gitlab_runner` with `gitlab_runner_registration_token_type: authentication-token`. The Galaxy role then registers the local runner non-interactively and writes the local runner entry into `/etc/gitlab-runner/config.toml`.

If the GitLab-side runner already exists but the local host has no matching runner configuration, the role fails with a clear message instead of silently creating a duplicate runner. Recreating a missing local configuration requires an explicit operator action, such as deleting the stale GitLab runner record or setting a dedicated recreate variable.

The generated `/etc/gitlab-runner/config.toml` uses the Docker executor and NVMe-backed runner paths:

```toml
concurrent = 14
check_interval = 0

[[runners]]
  name = "pharos-gitlab-ci-runner-1"
  url = "https://gitlab.grnet.gr"
  executor = "docker"
  limit = 14
  builds_dir = "/mnt/gitlab-runner/builds"
  cache_dir = "/mnt/gitlab-runner/cache"

  [runners.docker]
    image = "alpine:3.21"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
```

### 8. Makefile Integration

Runner provisioning gets explicit Makefile targets instead of being folded into the normal environment targets. This keeps the runner workflow separate from `dev`, `staging`, `production`, and `local`, and avoids changing the Terraform-backed inventory generation flow.

The Makefile adds:

| Target | Purpose |
|---|---|
| `ping-runner` | Run Ansible ping against `inventory/runner/hosts.yml` |
| `gitlab-ci-runner` | Run `playbooks/gitlab-ci-runner.yml` against the runner inventory |

Both targets use the same virtualenv, SSH key, vault handling, and deploy wrapper patterns as the existing Ansible targets.

### 9. Controller Dependencies

The `community.general.gitlab_runner` module uses the GitLab API through `python-gitlab`. Because the role uses the modern runner creation workflow, the Ansible controller environment must include `python-gitlab >= 4.0.0`.

The Ansible bootstrap dependencies are updated so `make bootstrap` installs this package before the runner playbook is used. The Galaxy dependency installer also installs standalone roles into `devops/ansible/vendor/galaxy`:

- `geerlingguy.docker`
- `riemers.gitlab_runner`

## Testing Strategy

The implementation is verified in three layers.

First, run static Ansible checks for the new playbook and role:

```bash
devops/ansible/scripts/ci/run_tests_local.sh --files playbooks/gitlab-ci-runner.yml,roles/gitlab_ci_runner/tasks/main.yml yamllint
devops/ansible/scripts/ci/run_tests_local.sh --files playbooks/gitlab-ci-runner.yml,roles/gitlab_ci_runner/tasks/main.yml lint
```

Second, verify connectivity and idempotent provisioning from `devops/ansible`:

```bash
make ping-runner
make gitlab-ci-runner
make gitlab-ci-runner
```

Third, validate the configured host and GitLab integration:

```bash
ssh -i ~/.ssh/pharos-admin-key admin@<RUNNER_EIP>

sudo systemctl status docker
sudo systemctl status gitlab-runner
sudo gitlab-runner verify
df -h /mnt/gitlab-runner
```

The GitLab project settings for `themelio/themelio` should show `pharos-gitlab-ci-runner-1` as online. A normal branch pipeline should start on the runner without adding `tags:` to CI jobs, and a manual build pipeline should run kaniko jobs successfully.

## Deployment & Rollout

The rollout sequence is:

1. Complete DD-0014 and verify SSH access to the runner EIP.
2. Create `inventory/runner/hosts.yml` with the runner EIP.
3. Add non-secret runner variables in `inventory/runner/group_vars/all.yml`.
4. Add `vault_gitlab_ci_runner_api_token` to encrypted `inventory/runner/group_vars/vault.yml`.
5. Run `make bootstrap` and `make galaxy` if the Ansible controller environment is not already prepared.
6. Run `make ping-runner`.
7. Run `make gitlab-ci-runner`.
8. Confirm the runner is online in GitLab and trigger a CI pipeline.

Rollback is manual. If provisioning fails before runner registration, fix the failed host-side configuration and re-run the playbook. If the runner has been registered but should not receive jobs, pause or delete `pharos-gitlab-ci-runner-1` from the `themelio/themelio` project runner settings in GitLab. If the EC2 instance is replaced, update `inventory/runner/hosts.yml` with the new Elastic IP before re-running Ansible.

## Open Questions

None. ADR-0015's implementation choices are resolved in this DD as follows:

- Runner scope: project-scoped runner for `themelio/themelio`.
- Concurrency: `concurrent = 14` and `limit = 14`, matching the ADR-0014 two-pipeline lint workload.
- Registration secret: GitLab API token stored in encrypted Ansible Vault.
- Controller dependencies: `make bootstrap` installs the required Python dependencies, including `python-gitlab >= 4.0.0`, before the runner playbook is used.

## References

- [ADR-0014: GitLab CI Runner on Dedicated AWS EC2 Instance](/adr/0014-gitlab-ci-runner-ec2-instance)
- [DD-0014: GitLab CI Runner EC2 Instance — Manual Operator Steps](/dd/0014-gitlab-ci-runner-ec2-instance)
- [ADR-0015: GitLab CI Runner EC2 Instance — Ansible Onboarding](/adr/0015-gitlab-ci-runner-ec2-ansible-onboarding)
- [GitLab Runner installation on Debian/Ubuntu](https://docs.gitlab.com/runner/install/linux-repository.html)
- [GitLab Runner Docker executor](https://docs.gitlab.com/runner/executors/docker.html)
- [Ansible `community.general.gitlab_runner` module](https://docs.ansible.com/projects/ansible/latest/collections/community/general/gitlab_runner_module.html)
- [Ansible Galaxy `geerlingguy.docker` role](https://galaxy.ansible.com/ui/standalone/roles/geerlingguy/docker/)
- [Ansible Galaxy `riemers.gitlab_runner` role](https://galaxy.ansible.com/ui/standalone/roles/riemers/gitlab-runner)