Thalamus

Appearance

Getting Started ​

Thalamus is a vendor-neutral, Kubernetes-native inference service based on llm-d, the Gateway API inference extension, and Cortex.

Prerequisites ​

Tools ​

  • kubectl — Kubernetes CLI
  • helm — Kubernetes package manager (v3.x)
  • helmfile — declarative wrapper around helm (v1.x)
  • A Kubernetes cluster with GPU nodes (NVIDIA), or minikube / any other local cluster for development

Accounts ​

Step 1 — Create the Hugging Face secret ​

Thalamus pulls model weights from Hugging Face at pod startup. Create a secret with your Hugging Face token in the thalamus namespace:

bash
kubectl create namespace thalamus

Then create the secret. The chart expects a secret named hf-token with key HF_TOKEN.

bash
kubectl create secret generic hf-token \
  --from-literal=HF_TOKEN="$HF_TOKEN" \
  --namespace thalamus

Step 2 — Create API key secrets (optional) ​

When the apikey-auth policy is enabled (gateway.policies.apikey-auth.enabled), every request to the targeted routes or listeners must carry a valid Authorization: Bearer <token> header. Tokens are stored as Kubernetes Secrets labelled thalamus-apikey: "true".

Set targetRefs on the policy to scope where auth is enforced. targetRefs default to the Gateway, so an entry only needs the fields it overrides — here, the listener section to protect:

yaml
thalamus:
  gateway:
    policies:
      apikey-auth:
        enabled: true
        targetRefs:
          - sectionName: https-api

Create one secret per user or client:

bash
kubectl create secret generic apikey-<name> \
  --namespace thalamus \
  --from-literal=api-key=$(openssl rand -base64 32 | tr '+/' '-_' | tr -d '=')
kubectl label secret apikey-<name> --namespace thalamus thalamus-apikey=true

Open WebUI connects to the inference API internally and also requires a token. Set the following in your cluster values to point Open WebUI at the secret:

yaml
open-webui:
  openaiApiKeyExistingSecret: apikey-openwebui
  openaiApiKeyExistingSecretKey: api-key

Custom gateway policies ​

apikey-auth and bbr are just entries in gateway.policies. Any key you add renders an AgentgatewayPolicy of the same name, with the entry used as its spec. targetRefs default to the Gateway and are merged per-entry, so you only specify overrides. Attach elsewhere with a full targetRefs entry or by label via targetSelectors:

yaml
thalamus:
  gateway:
    policies:
      rate-limit:
        targetRefs:
          - kind: HTTPRoute
            name: aggregated-models
        traffic:
          localRateLimit:
            - maxTokens: 100
              tokensPerFill: 100
              fillInterval: 60s

Step 3 — Deploy the stack ​

The helm/helmfile.yaml.gotmpl manifest installs the full stack as a set of ordered helmfile releases: the Gateway API and Inference Extension CRDs, the Thalamus CRDs, the GPU operator and node feature discovery, the agentgateway with its CRDs, kube-prometheus-stack for observability, the thalamus chart (operator + workloads: inference gateway, models, routes), and finally open-webui. Helmfile registers the required helm repositories and applies the releases in dependency order.

Thalamus operator — under development

The Thalamus operator will automate model instance management and move model declaration from Helm values to the thalamus.cloud/v1alpha1 Model CRD, enabling fully declarative, per-resource lifecycle control. Until then, models are managed through the models: values key described below.

Deploy with chart defaults:

bash
helmfile --file helm/helmfile.yaml.gotmpl apply

To customize values for your cluster, write a release-keyed values file and pass it via --state-values-file. The top-level keys are helmfile release names (e.g. thalamus, open-webui, gpu-operator, kube-prometheus-stack, agentgateway); everything underneath is forwarded to that release as chart values. See example.values.yaml for a reference shape of the thalamus release values.

yaml
# my-cluster.yaml
thalamus:
  models:
    - slug: qwen3-6-27b
      model: Qwen/Qwen3.6-27B
      accelerator: nvidia
      resources:
        requests: { nvidia.com/gpu: "2" }
        limits:   { nvidia.com/gpu: "2" }
bash
helmfile --file helm/helmfile.yaml.gotmpl apply \
  --state-values-file my-cluster.yaml

To preview changes before applying, use helmfile diff in place of apply.

Caveats for values file:

  • Adjust resources for your selected model. If it fails without a visible error, it might be OOM-killed due to RAM overflowing the specified limit.
  • If your resources are limited, you may try setting up "--max-model-len=8192" under baseArgs and explore other options to optimize the model.
  • Model slugs must be valid DNS-1035 labels: lowercase alphanumeric and hyphens only, starting with a letter. Dots and underscores are not allowed (e.g. use qwen3-0-6b, not qwen3-0.6b).

Step 4 — Access the stack ​

Once the pods are running, the stack is reachable in two ways.

Gateway API (OpenAI-compatible endpoint) ​

The inference gateway exposes an OpenAI-compatible API. Use the LoadBalancer IP or internal service address to send requests:

bash
curl http://<gateway-ip>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.6-27B",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

For local clusters without a LoadBalancer, use port-forward:

bash
kubectl port-forward svc/inference-gateway 8080:80 -n thalamus

Open WebUI ​

thalamus includes Open WebUI, a browser-based chat interface. It is reachable via the hostname configured in your open-webui.route.hostnames value, or via port-forward for local access:

bash
kubectl port-forward svc/open-webui 8080:80 -n open-webui

Then open http://localhost:8080 in your browser.

Local development (CPU-only) ​

For a lightweight local setup without a GPU, use helm/example.values.cpu.yaml as your values file. The modest resource requests make it suitable for local clusters like minikube or kind.

bash
helmfile --file helm/helmfile.yaml.gotmpl apply \
  --state-values-file helm/example.values.cpu.yaml

Note: The CPU image has no Apple Silicon / Metal acceleration. Inference will be significantly slower than on a GPU or native macOS runtimes like Ollama. Note: When using the Docker driver (default on macOS), Docker does not fully virtualize memory — vLLM sees the entire host RAM and will attempt to allocate a large fraction of it, exceeding your container limits and causing an OOM kill. Set --gpu-memory-utilization explicitly to avoid this.

Next Steps ​