Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

#ADR-0007: GPU Slicing and Quota Enforcement

Status: Draft

Date: 2026-05-22

Deciders: TBD

#Context

Themelio users need to run multiple GPU-backed processes in parallel, such as several Jupyter notebooks or experiments under the same project allocation. Requiring a full physical GPU for each process would waste capacity and make interactive workflows unnecessarily expensive.

This creates a need to expose GPU capacity in smaller units. At the same time, Themelio must avoid presenting a fractional GPU quota as reserved GPU memory or protected capacity unless that guarantee can actually be enforced.

Themelio needs an architecture decision for fractional GPU access that balances utilization, user experience, quota enforcement, and operational feasibility in Kubernetes.

#Requirements

1. Themelio must support multiple GPU-backed workloads running in parallel within the same project allocation, including cases where those workloads share one physical GPU.
2. Themelio must clearly distinguish guaranteed GPU allocations from best-effort shared GPU access.
3. Themelio must not advertise fractional GPU quotas as reserved GPU memory unless memory isolation can be enforced.
4. The chosen model must expose GPU units as Kubernetes resources that can be requested by workloads and counted against project quotas.
5. The chosen model must be compatible with the GPU hardware available to Themelio.

#Decision Drivers

• Quota enforceability: Whether the model gives Themelio a reliable way to enforce the GPU capacity promised to a project.
• Memory isolation: Whether one workload can exhaust GPU memory needed by another workload sharing the same physical device.
• Parallel workload support: Whether users can run multiple notebooks, experiments, or jobs at the same time without requiring one full physical GPU per workload.
• Kubernetes fit: Whether the model can be exposed through scheduler-visible resources and integrated with project quota accounting.
• Hardware compatibility: Whether the model works across the GPU models Themelio expects to operate.
• Operational complexity: How much node configuration, resource naming, monitoring, and user-facing explanation the model requires.

#Considered Options

#Option A: Whole GPU Allocation

Themelio exposes only full physical GPUs as allocatable resources. A workload that needs GPU access requests one or more complete GPUs, and Kubernetes schedules it onto a node with available devices. Users can still run multiple processes against the same GPU from within a single scheduled workload, but Themelio does not schedule or quota those processes independently.

Pros:

• Strongest and simplest quota model.
• No ambiguity about whether GPU memory is reserved.
• Works across NVIDIA GPUs supported by the Kubernetes device plugin.
• Lower operational complexity than fractional sharing.

Cons:

• Lower utilization when workloads do not use the full GPU.
• Users may consume their GPU quota quickly just to run several parallel lightweight sessions.
• Does not solve the main fractional access problem by itself.

#Option B: Time-Sliced GPU Allocation

Themelio exposes each physical GPU as multiple scheduler-visible GPU replicas using NVIDIA device plugin time-slicing. Workloads request one replica, allowing several notebooks or jobs to be scheduled onto the same physical GPU at the same time.

This option directly supports parallel lightweight workloads, but it should be treated as best-effort shared access rather than quota-enforced fractional GPU capacity. Time-sliced replicas are shared access tickets, not separate memory or compute slices. Themelio should enable failRequestsGreaterThanOne: true so users cannot request multiple replicas and accidentally assume that 2 replicas means 2x memory or 2x compute.

Time-slicing does not provide GPU memory isolation between replicas, so one workload can consume GPU memory needed by another workload on the same device.

Pros:

• Supports multiple independent workloads sharing one physical GPU.
• Improves GPU utilization for lightweight notebooks and experiments.
• Implementable in Kubernetes through the NVIDIA device plugin.
• Works on more GPU models than MIG because it does not require MIG-capable hardware.
• Operationally simpler than managing MIG profiles.

Cons:

• Does not provide GPU memory isolation.
• A failure or disruptive workload can affect other workloads sharing the same physical GPU.
• Fractional GPU replicas are scheduling units, not guaranteed memory slices.
• Workloads must be restricted to requesting one shared replica at a time to avoid misleading quota semantics.
• Quota enforcement is limited to admission and counting, not actual GPU memory usage.
• User experience can be unpredictable if one workload monopolizes memory or compute.

#Option C: MPS-Backed GPU Allocation

Themelio enables NVIDIA Multi-Process Service (MPS) for shared GPU access. Workloads run as MPS clients against an MPS server on the node, and a workload may request one or more advertised GPU replicas depending on how much shared GPU capacity it needs.

MPS does not create virtual GPUs in the way MIG does. Instead, it applies logical memory and compute limits to MPS clients. In the Kubernetes device plugin MPS model, each configured MPS replica/client gets an equal fraction of the physical GPU's memory allocation limit, based on 1 / replicas.

MPS does not provide hard compute enforcement in the sense of assigning dedicated physical Streaming Multiprocessors (SMs) to a pod or client. Instead, it provides a CUDA-level compute cap through CUDA_MPS_ACTIVE_THREAD_PERCENTAGE, which limits the maximum active thread capacity exposed to each MPS client. This means MPS can prevent one client from consuming the full GPU, but it does not reserve or physically isolate compute resources like MIG. Therefore, MPS should be treated as controlled shared GPU access with compute ceilings, not as hard hardware compute partitioning.

Pros:

• Supports multiple independent workloads sharing one physical GPU.
• Provides logical GPU memory limits and CUDA-level compute ceilings per MPS client.
• Gives stronger resource controls than time-slicing.
• Implementable through the NVIDIA Kubernetes device plugin.
• Can improve utilization for parallel notebook and experiment workloads.

Cons:

• Does not provide hardware-backed GPU partitions like MIG.
• MPS fault containment is limited. A fatal GPU fault is contained to the subset of GPUs shared by clients connected to the faulting GPU, not necessarily to the single client that caused it. For example, with GPUs 0, 1, and 2: client A uses 0, client B uses 0+1, client C uses 1, and client D uses 2. If A faults on GPU 0, then B links GPUs 0 and 1, so A, B, and C are affected and must exit. D keeps running. After the affected clients exit, MPS recreates the contexts for GPUs 0 and 1 and becomes active again.
• Early termination of an MPS client without synchronizing outstanding GPU work can leave the MPS server and other clients in an undefined state.
• MPS is only supported on Linux and QNX operating systems.
• Operationally more complex than time-slicing because it depends on MPS server behavior and configuration.
• Compatibility with Themelio's target GPU models and CUDA workloads must be validated.

#Option D: MIG-Backed GPU Allocation

Themelio exposes NVIDIA Multi-Instance GPU (MIG) instances as schedulable GPU resources. A MIG-capable physical GPU is partitioned into fixed-size GPU instances, and workloads request a specific MIG profile instead of a whole GPU.

Unlike time-slicing or MPS, MIG creates hardware-backed GPU partitions. Each MIG instance has dedicated GPU memory and compute resources, so the advertised fractional GPU resource corresponds to reserved capacity.

Pros:

• Provides hardware-backed GPU memory isolation.
• Provides stronger workload isolation than time-slicing or MPS.
• Fractional GPU resources correspond to real reserved capacity.
• Better fit for quota-enforced GPU slicing.
• Can expose different GPU slice sizes for different workload classes.

Cons:

• Only works on MIG-capable NVIDIA GPUs.
• Uses fixed MIG profile shapes, so arbitrary fractions are not possible.
• More operationally complex than whole GPU allocation or time-slicing.
• MIG alone still maps one scheduled workload to one MIG instance. If a user receives one MIG slice, Themelio has not created multiple independently schedulable workloads inside that slice.
• Supporting multiple independently scheduled workloads within a single MIG slice requires an additional sharing layer, such as time-slicing on top of MIG.

#Option E: MIG-Backed Allocation with Time-Slicing

Themelio exposes NVIDIA Multi-Instance GPU (MIG) instances as the quota-enforced GPU unit, then applies NVIDIA device plugin time-slicing to selected MIG resources. A physical GPU is first partitioned into MIG instances with dedicated memory and compute resources. Each MIG instance is then advertised as multiple scheduler-visible replicas, allowing several workloads to share the same MIG slice.

This combines two roles. MIG provides the reserved GPU memory and compute capacity that Themelio can count against a project quota. Time-slicing provides best-effort concurrency inside that MIG allocation. The time-sliced replicas are shared access tickets to a MIG instance, not separate memory or compute slices. Themelio should enable failRequestsGreaterThanOne: true so a single workload, such as one Jupyter notebook, cannot request multiple time-sliced replicas and accidentally assume that 2 replicas means 2x memory or 2x compute.

Pros:

• Provides hardware-backed isolation at the MIG slice level.
• Supports multiple independently scheduled workloads within a MIG-backed allocation.
• Keeps the quota-enforced unit tied to reserved GPU memory and compute capacity.
• Improves utilization for interactive notebook and experiment workflows.
• Implementable with the NVIDIA Kubernetes device plugin.

Cons:

• Only works on MIG-capable NVIDIA GPUs.
• Uses fixed MIG profile shapes, so arbitrary fractions are not possible.
• Time-sliced replicas inside a MIG slice do not have memory isolation from each other.
• Quota enforcement applies cleanly to the MIG slice, but not to each time-sliced replica inside it.
• A single workload must be restricted to requesting one shared replica at a time to avoid misleading quota semantics.
• It is not clear whether multiple requested time-sliced replicas are guaranteed to come from the same MIG slice. If they do not, a project could use time-sliced replicas from separate MIG slices. Because time-sliced replicas are not independently memory- or compute-bound, the workload could monopolize capacity from a MIG slice that another project also uses, recreating the same memory and compute boundary problems as plain time-slicing.
• More operationally complex than using only MIG or only time-slicing, because Themelio must manage both MIG profiles and replica counts.

#Comparison Matrix

#Decision

TBD — pending resolution of open questions.

#Consequences

TBD — will be filled in once a decision is made.

#Open Questions

#Q1: GPU Hardware Inventory

Which GPU models will Themelio operate in production, and which of them support MIG, MPS, and the NVIDIA device plugin sharing modes required by the considered options?

#Q2: Shared Resource Exposure

Should Themelio expose best-effort shared GPU resources to users at all, or should user-facing GPU quotas only represent resources with enforceable memory isolation?

#Q3: MPS Workload Compatibility

Are Themelio's expected CUDA workloads compatible with MPS behavior, including its fault handling, client lifecycle requirements, and Linux/QNX-only support?

#Q4: Project Affinity for Time-Sliced MIG Resources

Can Themelio guarantee that all workloads using a project's time-sliced MIG allocation are scheduled onto replicas of the same underlying MIG instance?

For example, if a project is allocated MIG slice A and it runs three notebooks, the desired placement is:

• MIG slice A -> notebook pod 1 -> time-sliced replica 1
• MIG slice A -> notebook pod 2 -> time-sliced replica 2
• MIG slice A -> notebook pod 3 -> time-sliced replica 3

The unwanted placement is:

• MIG slice 1 -> notebook pod 1 -> time-sliced replica 1
• MIG slice 2 -> notebook pod 2 -> time-sliced replica 2

The second case would spread one project's shared allocation across multiple MIG instances and could let that project consume memory or compute from multiple MIG slices, while also interfering with other projects sharing those slices.

#Q5: User-Facing Quota Model

What GPU resource names and quota units should users see in Themelio and Waldur so they can distinguish whole GPUs, MIG-backed guaranteed slices, and best-effort shared replicas?

#References

• NVIDIA Kubernetes Device Plugin
• NVIDIA Multi-Process Service: When to Use MPS
• NVIDIA Multi-Instance GPU User Guide
• NVIDIA GPU Operator: GPU Sharing

Page source

Source snapshot · Revision f928834c8.