> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. # ADR-0015: GitLab CI Runner EC2 Instance — Ansible Onboarding **Status:** Proposed **Date:** 2026-06-03 **Deciders:** Thanasis Katsadas, Dimitris Katsimpris **Relates to:** [ADR-0014 (GitLab CI Runner on Dedicated AWS EC2 Instance)](/reference/docs/decisions/0014-gitlab-ci-runner-ec2-instance.md) — this ADR addresses the operational onboarding of the instance type and purpose decided there. **Implementation:** [DD-0015: GitLab CI Runner EC2 Instance — Ansible Onboarding](/dd/0015-gitlab-ci-runner-ec2-ansible-onboarding.md) ## Context ADR-0014 decided to provision a c6id.4xlarge EC2 instance as a self-hosted GitLab CI runner. That instance lives in a **separate AWS account** from all other Themelio infrastructure (jump host, Kubernetes masters, workers, Ceph, GPU nodes). The existing Ansible automation assumes all managed hosts are provisioned by Terraform in the main account: `generate_ansible_inventory.sh` reads `terraform output -json` to auto-generate `hosts.yml`, and all hosts are reached via ProxyJump through the main bastion. Neither mechanism applies to the runner instance. The runner's only integration with GitLab is polling `https://gitlab.grnet.gr`, which is publicly available. The runner initiates all communication to GitLab outbound over HTTPS; GitLab never makes inbound connections to the runner. No inbound network path from `gitlab.grnet.gr` to the runner is required. ADR-0014 covers the remaining setup steps required before Ansible can take over: OS image (Debian 12), Elastic IP allocation, and security group configuration (SSH inbound restricted to GRNET and NTUA VPN CIDRs only). By the time this ADR's steps are applied, the instance is reachable via its EIP from any VPN-connected operator workstation. Two questions remain: how to install and register the `gitlab-runner` agent, and how to integrate the host into the existing Ansible inventory and Makefile workflow. ### Requirements 1. Ansible must be able to provision and update the runner (OS packages, `gitlab-runner` binary, Docker engine, runner registration). 2. The runner must reach `https://gitlab.grnet.gr` outbound to poll for jobs and push Docker images to the registry. 3. The onboarding approach must be maintainable within the existing Makefile and inventory workflow without requiring every operator to hold cross-account AWS credentials. ## Decision Drivers - **Operational simplicity:** Prefer approaches that reuse existing patterns (VPN access, static inventory, apt packages) over introducing new infrastructure or tooling dependencies. - **Security:** SSH access must be tightly scoped; the instance has a public IP and lives outside the main account's VPC. - **Consistency with the fleet:** The runner should behave like other managed nodes — same OS, same user, same Ansible entry point — to the greatest extent possible given the account boundary. - **Inventory automation:** The current auto-generation pipeline must not break. The runner must integrate without requiring changes to `generate_ansible_inventory.sh` or the Terraform state it reads. ## Considered Options ### Decision 1: Runner Installation Method ADR-0014 decided the runner uses the Docker executor. This decision covers how the `gitlab-runner` agent itself is delivered and managed. #### Option A: Official Debian packages managed by Ansible Galaxy roles The playbook uses `geerlingguy.docker` to install Docker and `riemers.gitlab_runner` to install `gitlab-runner` via the official GitLab Debian package repository. A systemd unit manages the runner process lifecycle, handles restarts on failure, and starts the runner on boot. Updates are applied via normal package maintenance runs. Runner registration is split between GitLab-side and host-side automation. The `community.general.gitlab_runner` module calls the GitLab API to create or update the project runner and produce an authentication token. That token is passed to `riemers.gitlab_runner`, which completes the local host registration and writes `/etc/gitlab-runner/config.toml`. **Pros:** - Standard delivery mechanism with upstream support. Systemd handles crash recovery. Host-side package and runner configuration logic is delegated to maintained Galaxy roles instead of bespoke local tasks. - `community.general.gitlab_runner` handles registration idempotently via the GitLab API, so re-running the playbook does not create duplicate runner entries. **Cons:** - None significant. #### Option B: Onboard as a Kubernetes worker node (Kubernetes executor) Join the runner instance to the existing Kubernetes cluster as a worker node and use the Kubernetes executor so each CI job runs as a pod scheduled by the cluster. **Pros:** - Pod-level isolation per CI job. **Cons:** - Requires re-running kubespray to extend the cluster across account boundaries, which demands a private network path between the runner VPC and the main account VPC (the problem Option B in Decision 2 describes). The cluster CNI must route pod traffic across the account boundary. The operational complexity is substantial relative to a standalone runner host. ### Decision 2: Ansible Inventory Integration #### Option A: Static `inventory/runner/hosts.yml` + Makefile auto-include A hand-written, static inventory file is created at `inventory/runner/hosts.yml`. It is not generated or modified by `generate_ansible_inventory.sh`. Makefile playbook targets are updated to detect whether this file exists and, if so, automatically append `-i inventory/runner/hosts.yml` to the `ansible-playbook` invocation. Operators do not need to pass an extra flag. **Pros:** - Zero changes to `generate_ansible_inventory.sh` or the Terraform state it reads. The runner inventory is a one-time write. The Makefile integration makes the runner host transparent to operators. **Cons:** - If the Elastic IP is explicitly released and a new one allocated (see below), the file must be updated manually. This is only expected to occur if the instance is permanently replaced. #### Option B: Extend `generate_ansible_inventory.sh` for cross-account Terraform output The script is extended to assume a cross-account IAM role and run `terraform output -json` against a second Terraform workspace in the runner account. **Pros:** - Fully automated; no manual file to maintain. **Cons:** - Every operator must have cross-account IAM credentials configured. The generate script grows substantially in complexity. The runner account requires its own Terraform state management. ## Decisions - OS: Debian 12. - Runner executor: Docker (decided in ADR-0014). - Runner agent delivery: official `gitlab-runner` Debian package managed by `riemers.gitlab_runner` (Option A, Decision 1). - Docker delivery: `geerlingguy.docker`. - GitLab-side runner management: `community.general.gitlab_runner`. - Runner scope: project runner for `themelio/themelio`. - Runner concurrency: global `concurrent=14`, per-runner `limit=14`. - Runner storage: mount an inventory-configured local block device at `/mnt/gitlab-runner`; do not auto-detect AWS-specific device IDs. ## Consequences - **Positive:** Docker and GitLab Runner host setup are handled by maintained Galaxy roles, reducing bespoke Ansible tasks in the repository. - **Positive:** The `gitlab-runner` Debian package integrates cleanly with systemd. The `community.general.gitlab_runner` module manages the GitLab-side runner idempotently via the GitLab API, so repeated playbook runs do not create duplicate runner entries in `https://gitlab.grnet.gr`. - **Positive:** The static `inventory/runner/hosts.yml` approach (Decision 2, Option A) leaves `generate_ansible_inventory.sh` and the Terraform state untouched. The Makefile auto-include means operators use the same `make` targets as for any other environment without extra flags. - **Neutral:** The static inventory file must be written once with the EIP allocated in ADR-0014 before the first `ansible-playbook` run. This is a one-time manual step; after that, the file is not expected to change for the duration of the 1-year reserved instance. - **Negative:** If the runner instance is permanently terminated and replaced, the EIP must be reallocated (per ADR-0014) and `inventory/runner/hosts.yml` updated manually before Ansible can reach the new instance. This is a low-frequency event but requires operator awareness. - **Negative:** Any automation that runs Ansible against the runner (e.g., a CI pipeline applying updates) must originate from a host on the GRNET or NTUA VPN, or the security group defined in ADR-0014 must be extended to include that host's IP. ## Open Questions None. DD-0015 resolves the remaining runner scope and concurrency choices. ## References - [ADR-0014: GitLab CI Runner on Dedicated AWS EC2 Instance](/reference/docs/decisions/0014-gitlab-ci-runner-ec2-instance.md) - [GitLab Runner installation on Debian/Ubuntu](https://docs.gitlab.com/runner/install/linux-repository.html) - [GitLab Runner Docker executor](https://docs.gitlab.com/runner/executors/docker.html) - [Ansible `community.general.gitlab_runner` module](https://docs.ansible.com/projects/ansible/latest/collections/community/general/gitlab_runner_module.html) - [Ansible Galaxy `geerlingguy.docker` role](https://galaxy.ansible.com/ui/standalone/roles/geerlingguy/docker/) - [Ansible Galaxy `riemers.gitlab_runner` role](https://galaxy.ansible.com/ui/standalone/roles/riemers/gitlab-runner)
Page source Source snapshot · Revision `f928834c8`.