Research Engineer (Evals)

Responsibilities: Build the evaluation systems that reveal whether AI agents are safe, reliable, and aligned with the policies customers care about.

• Design datasets, benchmarks, harnesses, and scoring methods for agent behavior.
• Evaluate prompts, outputs, tool calls, plans, and outcomes across realistic workflows.
• Build repeatable regression suites for prompt injection, jailbreaks, leakage, misuse, and off-task behavior.
• Compare models and policies, investigate surprising results, and make failure modes legible.
• Integrate evaluations into CI, runtime monitoring, and customer-facing evidence.
• Work with research and product engineers to turn findings into better controls.

Ideal candidate: You are a strong engineer with experience in evaluation, testing, ML, or developer tooling. You can build reliable Python systems, design careful experiments, and reason about noisy model behavior. You are comfortable with LLMs, agents, APIs, tool calling, and data analysis. Experience with eval frameworks, red teaming, safety research, or CI-integrated testing is a plus.