ThalamusKubernetes-Native LLM Inference

A vendor-neutral, Kubernetes-native inference service based on llm-d, the Gateway API inference extension, and Cortex.

Demo

See the current Proof of Concept in action: stack deployment on a Gardener cluster, images in Keppel, Elektra navigation, and the OpenAI-compatible API endpoint.

Watch the demo

🧠

Model CRD

Describe any LLM as a Kubernetes resource. One YAML manifest covers engine, weights, GPU allocation, autoscaling, and access policy.

Model CRD reference

🏗️

Architecture

Built on llm-d, the Gateway API inference extension, and Cortex. Learn how the components fit together.

Architecture overview