For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /reference/docs/decisions/0015-gitlab-ci-runner-ec2-ansible-onboarding.md.

ADR-0015: GitLab CI Runner EC2 Instance — Ansible Onboarding

Status: Proposed

Date: 2026-06-03

Deciders: Thanasis Katsadas, Dimitris Katsimpris

Relates to: ADR-0014 (GitLab CI Runner on Dedicated AWS EC2 Instance) — this ADR addresses the operational onboarding of the instance type and purpose decided there.

Implementation: DD-0015: GitLab CI Runner EC2 Instance — Ansible Onboarding

Context

ADR-0014 decided to provision a c6id.4xlarge EC2 instance as a self-hosted GitLab CI runner. That instance lives in a separate AWS account from all other Themelio infrastructure (jump host, Kubernetes masters, workers, Ceph, GPU nodes). The existing Ansible automation assumes all managed hosts are provisioned by Terraform in the main account: generate_ansible_inventory.sh reads terraform output -json to auto-generate hosts.yml, and all hosts are reached via ProxyJump through the main bastion. Neither mechanism applies to the runner instance.

The runner's only integration with GitLab is polling https://gitlab.grnet.gr, which is publicly available. The runner initiates all communication to GitLab outbound over HTTPS; GitLab never makes inbound connections to the runner. No inbound network path from gitlab.grnet.gr to the runner is required.

ADR-0014 covers the remaining setup steps required before Ansible can take over: OS image (Debian 12), Elastic IP allocation, and security group configuration (SSH inbound restricted to GRNET and NTUA VPN CIDRs only). By the time this ADR's steps are applied, the instance is reachable via its EIP from any VPN-connected operator workstation. Two questions remain: how to install and register the gitlab-runner agent, and how to integrate the host into the existing Ansible inventory and Makefile workflow.

Requirements

  1. Ansible must be able to provision and update the runner (OS packages, gitlab-runner binary, Docker engine, runner registration).
  2. The runner must reach https://gitlab.grnet.gr outbound to poll for jobs and push Docker images to the registry.
  3. The onboarding approach must be maintainable within the existing Makefile and inventory workflow without requiring every operator to hold cross-account AWS credentials.

Decision Drivers

  • Operational simplicity: Prefer approaches that reuse existing patterns (VPN access, static inventory, apt packages) over introducing new infrastructure or tooling dependencies.
  • Security: SSH access must be tightly scoped; the instance has a public IP and lives outside the main account's VPC.
  • Consistency with the fleet: The runner should behave like other managed nodes — same OS, same user, same Ansible entry point — to the greatest extent possible given the account boundary.
  • Inventory automation: The current auto-generation pipeline must not break. The runner must integrate without requiring changes to generate_ansible_inventory.sh or the Terraform state it reads.

Considered Options

Decision 1: Runner Installation Method

ADR-0014 decided the runner uses the Docker executor. This decision covers how the gitlab-runner agent itself is delivered and managed.

Option A: Official Debian packages managed by Ansible Galaxy roles

The playbook uses geerlingguy.docker to install Docker and riemers.gitlab_runner to install gitlab-runner via the official GitLab Debian package repository. A systemd unit manages the runner process lifecycle, handles restarts on failure, and starts the runner on boot. Updates are applied via normal package maintenance runs.

Runner registration is split between GitLab-side and host-side automation. The community.general.gitlab_runner module calls the GitLab API to create or update the project runner and produce an authentication token. That token is passed to riemers.gitlab_runner, which completes the local host registration and writes /etc/gitlab-runner/config.toml.

Pros:

  • Standard delivery mechanism with upstream support. Systemd handles crash recovery. Host-side package and runner configuration logic is delegated to maintained Galaxy roles instead of bespoke local tasks.
  • community.general.gitlab_runner handles registration idempotently via the GitLab API, so re-running the playbook does not create duplicate runner entries.

Cons:

  • None significant.

Option B: Onboard as a Kubernetes worker node (Kubernetes executor)

Join the runner instance to the existing Kubernetes cluster as a worker node and use the Kubernetes executor so each CI job runs as a pod scheduled by the cluster.

Pros:

  • Pod-level isolation per CI job.

Cons:

  • Requires re-running kubespray to extend the cluster across account boundaries, which demands a private network path between the runner VPC and the main account VPC (the problem Option B in Decision 2 describes). The cluster CNI must route pod traffic across the account boundary. The operational complexity is substantial relative to a standalone runner host.

Decision 2: Ansible Inventory Integration

Option A: Static inventory/runner/hosts.yml + Makefile auto-include

A hand-written, static inventory file is created at inventory/runner/hosts.yml. It is not generated or modified by generate_ansible_inventory.sh. Makefile playbook targets are updated to detect whether this file exists and, if so, automatically append -i inventory/runner/hosts.yml to the ansible-playbook invocation. Operators do not need to pass an extra flag.

Pros:

  • Zero changes to generate_ansible_inventory.sh or the Terraform state it reads. The runner inventory is a one-time write. The Makefile integration makes the runner host transparent to operators.

Cons:

  • If the Elastic IP is explicitly released and a new one allocated (see below), the file must be updated manually. This is only expected to occur if the instance is permanently replaced.

Option B: Extend generate_ansible_inventory.sh for cross-account Terraform output

The script is extended to assume a cross-account IAM role and run terraform output -json against a second Terraform workspace in the runner account.

Pros:

  • Fully automated; no manual file to maintain.

Cons:

  • Every operator must have cross-account IAM credentials configured. The generate script grows substantially in complexity. The runner account requires its own Terraform state management.

Decisions

  • OS: Debian 12.
  • Runner executor: Docker (decided in ADR-0014).
  • Runner agent delivery: official gitlab-runner Debian package managed by riemers.gitlab_runner (Option A, Decision 1).
  • Docker delivery: geerlingguy.docker.
  • GitLab-side runner management: community.general.gitlab_runner.
  • Runner scope: project runner for themelio/themelio.
  • Runner concurrency: global concurrent=14, per-runner limit=14.
  • Runner storage: mount an inventory-configured local block device at /mnt/gitlab-runner; do not auto-detect AWS-specific device IDs.

Consequences

  • Positive: Docker and GitLab Runner host setup are handled by maintained Galaxy roles, reducing bespoke Ansible tasks in the repository.
  • Positive: The gitlab-runner Debian package integrates cleanly with systemd. The community.general.gitlab_runner module manages the GitLab-side runner idempotently via the GitLab API, so repeated playbook runs do not create duplicate runner entries in https://gitlab.grnet.gr.
  • Positive: The static inventory/runner/hosts.yml approach (Decision 2, Option A) leaves generate_ansible_inventory.sh and the Terraform state untouched. The Makefile auto-include means operators use the same make targets as for any other environment without extra flags.
  • Neutral: The static inventory file must be written once with the EIP allocated in ADR-0014 before the first ansible-playbook run. This is a one-time manual step; after that, the file is not expected to change for the duration of the 1-year reserved instance.
  • Negative: If the runner instance is permanently terminated and replaced, the EIP must be reallocated (per ADR-0014) and inventory/runner/hosts.yml updated manually before Ansible can reach the new instance. This is a low-frequency event but requires operator awareness.
  • Negative: Any automation that runs Ansible against the runner (e.g., a CI pipeline applying updates) must originate from a host on the GRNET or NTUA VPN, or the security group defined in ADR-0014 must be extended to include that host's IP.

Open Questions

None. DD-0015 resolves the remaining runner scope and concurrency choices.

References

Page source

Source snapshot · Revision f928834c8.