Data Infrastructure Engineer, Pre-training

About the role

Anthropic is seeking a senior Data Infrastructure Engineer to join its Pre‑training team. You will work at the intersection of cutting‑edge AI research and production engineering, building the data pipelines that power the next generation of large language models.

Key responsibilities

• Design and implement high‑performance, reproducible data processing infrastructure for large‑scale model training.
• Develop and maintain core processing primitives such as tokenization, deduplication, and chunking with a focus on scalability.
• Build robust systems for data quality assurance and validation at web‑scale.
• Collaborate with research teams to translate novel data processing architectures into production.
• Operate end‑to‑end pipelines that convert raw web corpora into training‑ready datasets.

Required profile

• 5+ years of professional software engineering experience (outside of internships).
• Strong background in building high‑throughput, fault‑tolerant distributed systems.
• Hands‑on experience with distributed computing frameworks, especially Apache Spark.
• Excellent problem‑solving abilities and attention to detail.
• Advanced degree in Computer Science or a related field.
• Experience with language model training infrastructure and MLOps/ML infrastructure.

Required skills

• Python
• Rust
• Apache Spark
• Distributed computing frameworks
• High‑throughput fault‑tolerant distributed systems
• Data processing primitives (tokenization, deduplication, chunking)
• MLOps / ML infrastructure
• Language model training infrastructure