Staff Software Engineer – Inference API

About the role

Cerebras is building a new generation of disaggregated AI inference systems that combine GPU‑accelerated prefill with ultra‑fast decode on its Wafer‑Scale Engine. We are looking for a Staff Software Engineer to design, implement and evolve the ML API layer that makes this heterogeneous serving system accessible, reliable and easy to use.

Key responsibilities

• Build production‑grade ML inference APIs for chat completions, text generation, streaming, model configuration, tool calling, structured outputs and multimodal inputs.
• Create a unified serving experience with consistent request/response semantics across GPU prefill, Cerebras decode and other back‑ends.
• Integrate emerging foundation models, tokenizers, prompt formats and sampling methods into the serving platform.
• Maintain API compatibility while designing Cerebras‑specific extensions, handling versioning, deprecation and validation.
• Extend custom inference services with vLLM, PyTorch, Hugging Face libraries, AMD ROCm and Cerebras runtime components.
• Support disaggregated inference by coordinating GPU prefill with Cerebras decode, including routing, state transfer and error handling.
• Optimize performance metrics such as time‑to‑first‑token, latency, throughput, batching and scheduling.
• Build validation, observability and reliability tooling: structured logging, tracing, metrics, dashboards and health checks.

Required profile

• 5+ years of software engineering experience with ownership of production or distributed systems.
• Strong programming ability in Python and Go; experience with performance‑sensitive or highly concurrent services in C++ or Rust.
• Proven experience designing stable, versioned APIs (gRPC, REST, streaming) and operating latency‑sensitive services in Linux containers, Kubernetes or similar orchestration.
• Ability to diagnose correctness, reliability and performance issues across a distributed serving stack.
• Bachelor’s degree in Computer Science, Computer/ Electrical Engineering or equivalent practical experience.

Required skills

• Python
• Go
• C++
• Rust
• gRPC
• REST
• Kubernetes
• Linux
• CI/CD
• Containerization