Getting Started

Thalamus is a vendor-neutral, Kubernetes-native inference service based on llm-d, the Gateway API inference extension, and Cortex.

Inference instances are declared as thalamus.cloud/v1alpha1 Model resources. The Thalamus operator reconciles each Model into the required engine, endpoint picker, routing, and service resources.

Prerequisites

Tools

• kubectl — Kubernetes CLI
• helm — Kubernetes package manager (v3.x)
• A Kubernetes cluster, or minikube / any other local cluster for development

Cluster requirements

Thalamus is vendor-neutral and does not install platform infrastructure. The following are expected to already be present in the cluster:

• GPU workloads: a GPU driver/operator and node feature discovery for your hardware vendor (e.g. the NVIDIA GPU Operator for NVIDIA).
• Observability (optional): a monitoring stack such as kube-prometheus-stack if you want Thalamus metrics scraped.

The Gateway API, Gateway API inference extension, and agentgateway CRDs are bundled with the install (Step 1).

Accounts

• A Hugging Face account with a read token and access to the models you want to serve

Step 1 — Deploy the stack

Thalamus installs as a single Helm chart bundling the Thalamus operator, the agentgateway controller, and all required CRDs (Thalamus, Gateway API, inference extension, agentgateway). Everything is enabled by default:

helm upgrade --install thalamus oci://ghcr.io/cobaltcore-dev/charts/thalamus \
  --namespace thalamus --create-namespace --wait \
  --version v0.2.0

Caution: the CRDs are installed as ordinary release resources, not via Helm's protected crds/ mechanism. helm uninstall thalamus therefore deletes every CRD and cascades into deleting all custom resources they own — including every Model and its inference workloads. Only uninstall it when you intend to tear down Thalamus.

Already have some of these components?

If your cluster already provides some of the bundled CRDs, disable the matching dependencies:

• gateway-api — skip with --set gateway-api.enabled=false (common on managed clusters)
• gateway-api-inference-extension — provides the InferencePool CRD Thalamus needs; disable only if your cluster already provides it

helm upgrade --install thalamus oci://ghcr.io/cobaltcore-dev/charts/thalamus \
  --namespace thalamus --create-namespace --wait \
  --version v0.2.0 \
  --set gateway-api.enabled=false

Likewise, if you run agentgateway as separate releases, disable its bundled CRDs with --set agentgateway-crds.enabled=false, and disable its bundled controller with --set agentgateway.enabled=false.

Step 2 — Create the Hugging Face secret

Model pods pull their weights from Hugging Face at startup. tokenSecret is optional — public models work without it — but providing a token avoids anonymous rate limits. To create a secret named hf-token with key HF_TOKEN in the thalamus namespace:

kubectl create secret generic hf-token \
  --from-literal=HF_TOKEN="$HF_TOKEN" \
  --namespace thalamus

Step 3 — Create the inference gateway

The Gateway is not part of the helm release; you create and own it. The operator attaches all of its routes — per-model traffic and the OpenAI-compatible /v1/models endpoint — to the Gateway you point it at.

A Gateway is a gateway.networking.k8s.io resource using the agentgateway GatewayClass that the stack deploys. The operator expects:

• the Gateway to live in the same namespace as the operator (thalamus),
• an HTTP listener that routes from all namespaces.

The operator finds the Gateway and listener by name via the chart's gateway.name and gateway.listener values (defaults inference-gateway and api); set them if yours differ.

Apply the example gateway:

kubectl apply -f https://raw.githubusercontent.com/cobaltcore-dev/thalamus/v0.2.0/examples/gateway.yaml

Wait for it to become programmed:

kubectl wait gateway/inference-gateway \
  --namespace thalamus \
  --for=condition=Programmed=True \
  --timeout=120s

The example also shows how to configure the data plane through an AgentgatewayParameters resource, e.g. to expose the gateway via a LoadBalancer instead of the default ClusterIP.

Gateway example manifest

# Inference gateway the Thalamus operator attaches its routes to; not managed
# by the helm chart. Point the chart at it via gateway.name and
# gateway.listener, and create it in the operator's namespace.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: inference-gateway
  namespace: thalamus
spec:
  gatewayClassName: agentgateway
  listeners:
    # The listener the operator attaches its routes to (gateway.listener).
    - name: api
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: All
    # Optional listener for additional consumers, e.g. open-webui.
    - name: frontend
      port: 8080
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: All
  # Optional: configure the data plane agentgateway deploys for this Gateway,
  # e.g. expose it via a LoadBalancer instead of the default ClusterIP.
  infrastructure:
    parametersRef:
      group: agentgateway.dev
      kind: AgentgatewayParameters
      name: inference-gateway
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayParameters
metadata:
  name: inference-gateway
  namespace: thalamus
spec:
  service:
    spec:
      type: ClusterIP
      # type: LoadBalancer
      # loadBalancerIP: ""

Step 4 — Deploy a model

Models are declared as thalamus.cloud/v1alpha1 Model resources and applied independently of the helm release. See examples/model-qwen3-6-27b-gpu.yaml for a GPU example and examples/model-smollm2-cpu.yaml for a CPU example.

kubectl apply -f https://raw.githubusercontent.com/cobaltcore-dev/thalamus/v0.2.0/examples/model-qwen3-6-27b-gpu.yaml

Wait for the model to become ready:

kubectl wait model/qwen3-6-27b --namespace thalamus --for=condition=Ready --timeout=600s

GPU example manifest

# Copyright SAP SE
# SPDX-License-Identifier: Apache-2.0

---
apiVersion: thalamus.cloud/v1alpha1
kind: Model
metadata:
  name: qwen3-6-27b
  namespace: thalamus
spec:
  backend: native
  replicas: 1
  serving:
    engine:
      containers:
        - name: engine
          image: docker.io/vllm/vllm-openai:v0.27.0
          args:
            - --disable-uvicorn-access-log
            - --gpu-memory-utilization=0.95
            - --tensor-parallel-size=2
            - --default-chat-template-kwargs={"enable_thinking":true}
            - --reasoning-parser=qwen3
            - --speculative-config={"method":"mtp","num_speculative_tokens":2}
            - --enable-auto-tool-choice
            - --tool-call-parser=qwen3_coder
          env:
            - name: NCCL_IB_DISABLE
              value: "1"
          resources:
            requests:
              nvidia.com/gpu: 2
            limits:
              nvidia.com/gpu: 2
    router:
      containers:
        - name: router
          image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: 200m
              memory: 512Mi
  weights:
    type: hf
    hf:
      repoId: Qwen/Qwen3.6-27B
      tokenSecret:
        name: hf-token
        key: HF_TOKEN

For a CPU-only or local development setup, use the SmolLM2 example:

kubectl apply -f https://raw.githubusercontent.com/cobaltcore-dev/thalamus/v0.2.0/examples/model-smollm2-cpu.yaml

Wait for the model to become ready:

kubectl wait model/smollm2-135m --namespace thalamus --for=condition=Ready --timeout=600s

CPU example manifest

# Copyright SAP SE
# SPDX-License-Identifier: Apache-2.0

# A small CPU-suitable Model for local development and the E2E validation suite.
---
apiVersion: thalamus.cloud/v1alpha1
kind: Model
metadata:
  name: smollm2-135m
  namespace: thalamus
spec:
  backend: native
  replicas: 1
  serving:
    engine:
      containers:
        - name: engine
          image: docker.io/vllm/vllm-openai-cpu:v0.27.0
          args:
            - --max-model-len=512
            - --max-num-seqs=1
            - --enforce-eager
            - --enable-auto-tool-choice
            - --tool-call-parser=apertus
            - --override-generation-config={"temperature":0}
          env:
            - name: VLLM_CPU_KVCACHE_SPACE
              value: "1"
            - name: VLLM_CPU_OMP_THREADS_BIND
              value: nobind
          resources:
            requests:
              cpu: 2000m
              memory: 6Gi
            limits:
              cpu: 4000m
              memory: 6Gi
    router:
      containers:
        - name: router
          image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: 200m
              memory: 512Mi
  weights:
    type: hf
    hf:
      repoId: HuggingFaceTB/SmolLM2-135M-Instruct

Step 5 — Access the stack

Once the pods are running, the inference gateway exposes an OpenAI-compatible API. Point GATEWAY_HOST at the endpoint you want to use, then send requests.

With a LoadBalancer (cloud clusters):

GATEWAY_HOST=$(kubectl get svc inference-gateway -n thalamus -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

If the IP comes back empty, the load balancer is still provisioning. Check progress with kubectl get svc inference-gateway -n thalamus.

With port-forward (local clusters without a LoadBalancer):

kubectl port-forward svc/inference-gateway 8080:80 -n thalamus &
GATEWAY_HOST=localhost:8080

Then test the API:

curl "http://$GATEWAY_HOST/v1/models"

curl "http://$GATEWAY_HOST/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-id>",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Next Steps

• Browse the Model CRD API Reference for all available fields.
• Secure the gateway and track usage with Gateway Integrations.
• Upgrading from a previous release? See v0.2.0 release notes.
• Add a web frontend with Open WebUI.
• Load the Perses dashboards to monitor SLOs, usage, and GPU health.
• Read the Architecture overview to understand how the operator, gateway, and endpoint picker fit together.
• Watch the Demo for a visual walkthrough.