Thalamus

Appearance

Getting Started ​

Thalamus is a vendor-neutral, Kubernetes-native inference service based on llm-d, the Gateway API inference extension, and Cortex.

Inference instances are declared as thalamus.cloud/v1alpha1 Model resources. The Thalamus operator reconciles each Model into the required engine, endpoint picker, routing, and service resources.

Prerequisites ​

Tools ​

  • kubectl — Kubernetes CLI
  • helm — Kubernetes package manager (v3.x)
  • A Kubernetes cluster, or minikube / any other local cluster for development

Cluster requirements ​

Thalamus is vendor-neutral and does not install platform infrastructure. The following are expected to already be present in the cluster:

  • GPU workloads: a GPU driver/operator and node feature discovery for your hardware vendor (e.g. the NVIDIA GPU Operator for NVIDIA).
  • Observability (optional): a monitoring stack such as kube-prometheus-stack if you want Thalamus metrics scraped.

The Gateway API, Gateway API inference extension, and agentgateway CRDs are bundled with the install (Step 1).

Accounts ​

Step 1 — Deploy the stack ​

Thalamus installs as a single Helm chart bundling the Thalamus operator, the agentgateway controller, and all required CRDs (Thalamus, Gateway API, inference extension, agentgateway). Everything is enabled by default:

bash
helm upgrade --install thalamus oci://ghcr.io/cobaltcore-dev/charts/thalamus \
  --namespace thalamus --create-namespace --wait \
  --version 0.0.0-main

Caution: the CRDs are installed as ordinary release resources, not via Helm's protected crds/ mechanism. helm uninstall thalamus therefore deletes every CRD and cascades into deleting all custom resources they own — including every Model and its inference workloads. Only uninstall it when you intend to tear down Thalamus.

Already have some of these components? ​

If your cluster already provides some of the bundled CRDs, disable the matching dependencies:

  • gateway-api — skip with --set gateway-api.enabled=false (common on managed clusters)
  • gateway-api-inference-extension — provides the InferencePool CRD Thalamus needs; disable only if your cluster already provides it
bash
helm upgrade --install thalamus oci://ghcr.io/cobaltcore-dev/charts/thalamus \
  --namespace thalamus --create-namespace --wait \
  --version 0.0.0-main \
  --set gateway-api.enabled=false

Likewise, if you run agentgateway as separate releases, disable its bundled CRDs with --set agentgateway-crds.enabled=false, and disable its bundled controller with --set agentgateway.enabled=false.

Step 2 — Create the Hugging Face secret ​

Model pods pull their weights from Hugging Face at startup. tokenSecret is optional — public models work without it — but providing a token avoids anonymous rate limits. To create a secret named hf-token with key HF_TOKEN in the thalamus namespace:

bash
kubectl create secret generic hf-token \
  --from-literal=HF_TOKEN="$HF_TOKEN" \
  --namespace thalamus

Step 3 — Create the inference gateway ​

The Gateway is not part of the helm release; you create and own it. The operator attaches all of its routes — per-model traffic and the OpenAI-compatible /v1/models endpoint — to the Gateway you point it at.

A Gateway is a gateway.networking.k8s.io resource using the agentgateway GatewayClass that the stack deploys. The operator expects:

  • the Gateway to live in the same namespace as the operator (thalamus),
  • an HTTP listener that routes from all namespaces.

The operator finds the Gateway and listener by name via the chart's gateway.name and gateway.listener values (defaults inference-gateway and api); set them if yours differ.

Apply the example gateway:

bash
kubectl apply -f https://raw.githubusercontent.com/cobaltcore-dev/thalamus/main/examples/gateway.yaml

Wait for it to become programmed:

bash
kubectl wait gateway/inference-gateway \
  --namespace thalamus \
  --for=condition=Programmed=True \
  --timeout=120s

The example also shows how to configure the data plane through an AgentgatewayParameters resource, e.g. to expose the gateway via a LoadBalancer instead of the default ClusterIP.

Gateway example manifest
yml
# Inference gateway the Thalamus operator attaches its routes to; not managed
# by the helm chart. Point the chart at it via gateway.name and
# gateway.listener, and create it in the operator's namespace.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: inference-gateway
  namespace: thalamus
spec:
  gatewayClassName: agentgateway
  listeners:
    # The listener the operator attaches its routes to (gateway.listener).
    - name: api
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: All
    # Optional listener for additional consumers, e.g. open-webui.
    - name: frontend
      port: 8080
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: All
  # Optional: configure the data plane agentgateway deploys for this Gateway,
  # e.g. expose it via a LoadBalancer instead of the default ClusterIP.
  infrastructure:
    parametersRef:
      group: agentgateway.dev
      kind: AgentgatewayParameters
      name: inference-gateway
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayParameters
metadata:
  name: inference-gateway
  namespace: thalamus
spec:
  service:
    spec:
      type: ClusterIP
      # type: LoadBalancer
      # loadBalancerIP: ""

Step 4 — Deploy a model ​

Models are declared as thalamus.cloud/v1alpha1 Model resources and applied independently of the helm release. See examples/model-qwen3-6-27b-gpu.yaml for a GPU example and examples/model-smollm2-cpu.yaml for a CPU example.

bash
kubectl apply -f https://raw.githubusercontent.com/cobaltcore-dev/thalamus/main/examples/model-qwen3-6-27b-gpu.yaml

Wait for the model to become ready:

bash
kubectl wait model/qwen3-6-27b --namespace thalamus --for=condition=Ready --timeout=600s
GPU example manifest
yml
# Copyright SAP SE
# SPDX-License-Identifier: Apache-2.0

---
apiVersion: thalamus.cloud/v1alpha1
kind: Model
metadata:
  name: qwen3-6-27b
  namespace: thalamus
spec:
  backend: native
  replicas: 1
  serving:
    engine:
      image: docker.io/vllm/vllm-openai:v0.27.0
      args:
        - --disable-uvicorn-access-log
        - --gpu-memory-utilization=0.95
        - --tensor-parallel-size=2
        - --default-chat-template-kwargs={"enable_thinking":true}
        - --reasoning-parser=qwen3
        - --speculative-config={"method":"mtp","num_speculative_tokens":2}
        - --enable-auto-tool-choice
        - --tool-call-parser=qwen3_coder
      env:
        - name: NCCL_IB_DISABLE
          value: "1"
      resources:
        requests:
          nvidia.com/gpu: 2
        limits:
          nvidia.com/gpu: 2
    epp:
      image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0
      resources:
        requests:
          cpu: 100m
          memory: 256Mi
        limits:
          cpu: 200m
          memory: 512Mi
  weights:
    type: hf
    hf:
      repoId: Qwen/Qwen3.6-27B
      tokenSecret:
        name: hf-token
        key: HF_TOKEN

For a CPU-only or local development setup, use the SmolLM2 example:

bash
kubectl apply -f https://raw.githubusercontent.com/cobaltcore-dev/thalamus/main/examples/model-smollm2-cpu.yaml

Wait for the model to become ready:

bash
kubectl wait model/smollm2-135m --namespace thalamus --for=condition=Ready --timeout=600s
CPU example manifest
yml
# Copyright SAP SE
# SPDX-License-Identifier: Apache-2.0

# A small CPU-suitable Model for local development and the E2E validation suite.
---
apiVersion: thalamus.cloud/v1alpha1
kind: Model
metadata:
  name: smollm2-135m
  namespace: thalamus
spec:
  backend: native
  replicas: 1
  serving:
    engine:
      image: docker.io/vllm/vllm-openai-cpu:v0.27.0
      args:
        - --max-model-len=512
        - --max-num-seqs=1
        - --enforce-eager
        - --enable-auto-tool-choice
        - --tool-call-parser=apertus
        - --override-generation-config={"temperature":0}
      env:
        - name: VLLM_CPU_KVCACHE_SPACE
          value: "1"
        - name: VLLM_CPU_OMP_THREADS_BIND
          value: nobind
      resources:
        requests:
          cpu: 2000m
          memory: 6Gi
        limits:
          cpu: 4000m
          memory: 6Gi
    epp:
      image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0
      resources:
        requests:
          cpu: 100m
          memory: 256Mi
        limits:
          cpu: 200m
          memory: 512Mi
  weights:
    type: hf
    hf:
      repoId: HuggingFaceTB/SmolLM2-135M-Instruct

Step 5 — Access the stack ​

Once the pods are running, the inference gateway exposes an OpenAI-compatible API. Point GATEWAY_HOST at the endpoint you want to use, then send requests.

With a LoadBalancer (cloud clusters):

bash
GATEWAY_HOST=$(kubectl get svc inference-gateway -n thalamus -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

If the IP comes back empty, the load balancer is still provisioning. Check progress with kubectl get svc inference-gateway -n thalamus.

With port-forward (local clusters without a LoadBalancer):

bash
kubectl port-forward svc/inference-gateway 8080:80 -n thalamus &
GATEWAY_HOST=localhost:8080

Then test the API:

bash
curl "http://$GATEWAY_HOST/v1/models"
bash
curl "http://$GATEWAY_HOST/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-id>",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

API key authentication (optional) ​

By default, the Thalamus API accepts unauthenticated requests. To enable token-based authentication, save an AgentgatewayPolicy as a values file. The example policy below requires every request to include a valid Authorization: Bearer <key> header. API keys are loaded from Kubernetes secrets labeled thalamus-apikey: "true".

yaml
# apikey-auth.yaml
extraDeploy:
  apikey-auth:
    apiVersion: agentgateway.dev/v1alpha1
    kind: AgentgatewayPolicy
    metadata:
      namespace: thalamus
    spec:
      targetRefs:
        - group: gateway.networking.k8s.io
          kind: Gateway
          name: inference-gateway
          sectionName: api
      traffic:
        apiKeyAuthentication:
          mode: Strict
          secretSelector:
            matchLabels:
              thalamus-apikey: "true"

Add -f apikey-auth.yaml to the install command in Step 1.

Create one secret per user or client:

bash
kubectl create secret generic apikey-<name> \
  --namespace thalamus \
  --from-literal=api-key=$(openssl rand -base64 32 | tr '+/' '-_' | tr -d '=')
kubectl label secret apikey-<name> --namespace thalamus thalamus-apikey=true

Next Steps ​

EU and German government funding logos

Funded by the European Union – NextGenerationEU.

The views and opinions expressed are solely those of the author(s) and do not necessarily reflect the views of the European Union or the European Commission. Neither the European Union nor the European Commission can be held responsible for them.

Copyright © Linux Foundation Europe.

Thalamus is a project of the NeoNephos Foundation. For applicable policies including privacy policy, terms of use and trademark usage guidelines, please see https://linuxfoundation.eu. Linux is a registered trademark of Linus Torvalds.

Terms of Use|Privacy Statement|Legal Disclosure