> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. > **Themelio compatibility reference.** This page describes the retained Themelio implementation or its historical design. Check the [parallel-operation guide](/compatibility/index.md) before applying it to the current platform. # ADR-0014: GitLab CI Runner on Dedicated AWS EC2 Instance **Status:** Accepted **Date:** 2026-06-02 **Deciders:** Thanasis Katsadas, Panos Louridas **Relates to:** [ADR-0015 (GitLab CI Runner EC2 Instance — Ansible Onboarding)](/adr/0015-gitlab-ci-runner-ec2-ansible-onboarding.md) — covers Ansible provisioning, runner installation, and GitLab registration once the instance decided here is running. ## Context The Themelio monorepo relies on shared GitLab CI runners provided by GRNET. These runners are not assigned to the project and are shared across GRNET-managed workloads, resulting in unpredictable queue times and slower feedback cycles. The Themelio CI pipeline has five stages (`lint → build-docs → preview → build → deploy`) and is split across domain-specific sub-pipeline files. The `lint` stage alone runs seven jobs in parallel on every branch push: Node.js-based documentation and service linting, and four Ansible verification jobs. The `build` stage, triggered manually, runs five parallel kaniko image builds — the heaviest of which is the Next.js 15 frontend compile. The target operational scenario is **two concurrent pipelines**, representing the normal dev + staging cadence. This means up to 14 parallel lint jobs or 10 parallel kaniko builds may be active at the same time, with peak memory around 10–18 GB and peak CPU around 14–26 cores depending on the scenario. The primary storage bottleneck is `pnpm install` inside kaniko builds, where large numbers of small-file writes are sensitive to disk I/O latency. EBS-backed instances impose provisioning constraints and add ongoing storage costs. Local NVMe eliminates both. ### Requirements 1. The runner must support two simultaneous CI pipelines without queue blocking. 2. Peak memory across the worst-case scenario (two concurrent full builds) must not exceed available RAM. 3. Storage must handle parallel kaniko builds without I/O contention becoming a bottleneck. 4. The instance must be located in the EU to comply with data residency expectations. 5. The total annual cost should be justified by the developer time saved on CI wait times. ## Decision Drivers - **Parallel pipeline support:** Whether the instance can sustain two concurrent pipelines without significant throughput degradation. - **Storage I/O performance:** Whether `pnpm install` and kaniko layer writes under load stay fast. EBS throughput caps are a known risk on high-concurrency builds. - **Memory fit:** Whether RAM is sufficient for peak concurrent jobs without triggering OOM or swap. - **Cost efficiency:** Annual total cost relative to the alternative of over-provisioned on-demand capacity or non-dedicated shared runners. - **Operational simplicity:** Number of moving parts — provisioned IOPS, EBS volumes, storage billing surprises. - **DevOps maintenance burden:** Ongoing effort to install, register, update, and monitor the runner. A self-hosted runner introduces an operational surface that the shared GRNET runners currently absorb. ## Considered Options ### Option A: m6a.4xlarge (16 vCPU, 64 GB RAM, EBS-backed) — Rejected General-purpose AMD instance with EBS for storage. No local NVMe. Storage performance depends on provisioned gp3 IOPS and throughput. **Pros:** - 64 GB RAM is well above peak requirements — no memory pressure under any scenario. - Familiar EBS-based storage model. **Cons:** - 64 GB RAM is roughly 3× the actual peak need (\~18 GB in the worst case), paying for unused capacity. - EBS I/O caps require explicit provisioning; under-provisioned volumes bottleneck kaniko builds. - EBS storage adds a recurring monthly cost on top of the instance reservation. - On-demand: \~$604/mo; 1-year reserved: \~$4,798/yr (EU Frankfurt). ### Option B: m6a.2xlarge (8 vCPU, 32 GB RAM, EBS-backed) — Rejected Half the specs of Option A at half the price. Memory is adequate; CPU is the constraint. **Pros:** - Lower cost. - 32 GB RAM covers peak memory for two simultaneous lint pipelines. **Cons:** - 8 vCPUs cannot sustain two concurrent lint stages (14–16 cores peak) without significant queuing. - A single full build pipeline consumes most available CPU, leaving little headroom for a second. - EBS storage limitations apply as in Option A. ### Option C: c6id.4xlarge (16 vCPU, 32 GB RAM, 950 GB local NVMe) — Accepted Compute-optimized instance with integrated NVMe SSD. Same vCPU count as Option A, right-sized RAM, and no EBS dependency. **Pros:** - Local NVMe at \~3.5 GB/s read / \~2.5 GB/s write eliminates storage I/O as a bottleneck under all realistic load scenarios. - 32 GB RAM comfortably covers peak concurrent load (\~10–18 GB), with no recurring EBS storage costs. - 16 vCPUs support two parallel lint pipelines (14–16 cores peak) and handle two simultaneous build pipelines with mild CPU saturation only during the Next.js compile phase — a 2–5 minute window on an infrequent manual trigger. - NVMe is included in the instance price; $4,803 upfront is the **total annual cost** with no recurring charges. - 41% saving versus on-demand rate ($0.9324/hr). **Cons:** - Local NVMe is ephemeral — runner workspace data is lost if the instance is terminated. GitLab CI jobs are stateless by design so this is not a concern in practice. - CPU contention during simultaneous full image builds (two concurrent manual triggers) causes 20–40% longer build times. This scenario is infrequent. ## Decision We will provision a **c6id.4xlarge** EC2 instance in the **EU (Frankfurt)** region as a self-hosted GitLab CI runner for the Themelio project, on a **1-year all-upfront reserved** pricing plan at a total cost of **$4,803**. The runner will be registered with the project's GitLab instance using the `gitlab-runner` agent and configured to use the Docker executor, with the local NVMe as the runner work directory. The instance will run **Debian 12**, consistent with all other non-GPU nodes in the Themelio fleet. This ensures existing Ansible base roles apply without modification and avoids introducing a second OS image to manage. A permanent **Elastic IP** will be allocated in the runner AWS account and associated with the instance before first use. Unlike the default auto-assigned public IP, an EIP persists across stop, start, and reboot cycles and only changes if explicitly released. This provides a stable address for the static Ansible inventory entry that ADR-0015 describes. An AWS **security group** will restrict inbound access as follows: - Port 22 (SSH): GRNET VPN and NTUA VPN CIDR blocks only. - All other inbound: deny. - Outbound: allow all (runner needs to reach `https://gitlab.grnet.gr`, Docker image registries, and OS package mirrors). Once the instance is launched with the EIP associated and the security group applied, it is ready to be onboarded via Ansible as described in ADR-0015. ## Consequences - **Positive:** CI feedback latency will decrease significantly. Two parallel pipelines (dev + staging) can proceed without queue contention. - **Positive:** Storage I/O is no longer a limiting factor for kaniko builds or `pnpm install` workloads. The NVMe SSD provides headroom that no realistic pipeline scenario will exhaust. - **Positive:** The $4,803 all-upfront fee is the complete annual cost. No ongoing EBS storage billing. - **Positive:** Debian 12 ensures the runner OS is consistent with the rest of the non-GPU fleet. No new base image, no new Ansible user, and no divergence from existing provisioning roles. - **Positive:** The Elastic IP provides a stable, permanent address for the lifetime of the reservation. The static Ansible inventory entry (described in ADR-0015) never needs updating under normal operations. - **Neutral:** CPU will be mildly saturated for a few minutes when two full image builds are manually triggered simultaneously. Build times increase by 20–40% during this window. Given that image builds are manual-only and infrequent, this is an accepted trade-off. - **Negative:** If pipeline build frequency increases substantially (e.g., automated image builds on every merge), CPU will become the limiting factor. The upgrade path is a **c6id.8xlarge** (32 vCPUs, 64 GB RAM, 1,900 GB NVMe) within the same instance family. But for now we will limit the concurrent jobs if we reach this saturation limit. - **Negative:** Operating a self-hosted runner introduces ongoing DevOps responsibility that the shared GRNET runners currently handle invisibly. Initial setup involves provisioning the EC2 instance, installing and registering `gitlab-runner`, configuring the Docker executor, pointing the work directory at the NVMe mount, and wiring up IAM and security groups. Ongoing maintenance includes OS and `gitlab-runner` version updates, monitoring disk usage on the NVMe volume, reacting to runner outages, and rotating any credentials the runner holds. This is estimated at a few hours of setup and low but non-zero recurring attention each month. - **Negative:** The instance must be kept running continuously to function as a runner. Shutdown for cost savings would remove CI capacity. An instance stop/start policy is possible but not planned at this time. Also we can probably switch the EC2 instance functionality if we have no need of it as a Gitlab CI Runner. - **Negative:** The EIP allocation, security group creation, and instance launch are one-time manual steps in the runner AWS account. They are not automated by the existing Terraform pipeline. Steps will be documented in DD-0014. ## Open Questions ### Q1: GRNET and NTUA VPN CIDR ranges The exact CIDR blocks for both VPN networks must be collected before the security group can be configured. These are required to restrict SSH inbound to VPN-connected operators only. ### Q2: Runner AWS account ownership, IAM access, and EIP allocation Who owns the runner AWS account and which operators have console/CLI access to it? Launching the instance, creating the security group, and allocating the Elastic IP are one-time manual steps performed together in the runner account. They will be documented in DD-0014 and will not be automated. ### Q3: SSH key pair strategy Which SSH key(s) will be injected into the instance at launch and used by Ansible for provisioning? Options include reusing the existing AWS key pair already stored in Ansible Vault (consistent with the rest of the fleet), generating a new dedicated key pair for this runner only, or using both (an AWS key pair for emergency console access and a separate Ansible key added via cloud-init). The chosen key's public half must be registered as an AWS key pair in the runner account before the instance is launched. Should we also add personal ssh keys of the operators? ## References - [AWS EC2 c6id instance family](https://aws.amazon.com/ec2/instance-types/c6i/) - [GitLab Runner Docker executor configuration](https://docs.gitlab.com/runner/executors/docker.html) - [Kaniko layer caching with a registry](https://github.com/GoogleContainerTools/kaniko#caching)
Page source Source snapshot · Revision `f928834c8`.