AI Evaluation Engineer (Knowledge & Research) Job
IT Jobs.
Role overview
We are looking for an AI Evaluation Engineer with a strong research background to design and evaluate complex, multi-agent tasks used to benchmark next-generation AI systems.
In this role, you will work at the intersection of research, data structuring, and AI evaluation, building high-quality tasks that require deep document understanding, structured reasoning, and multi-step synthesis. You will create datasets and evaluation frameworks that test whether AI agents can truly read, reason, and extract knowledge from large-scale unstructured data.
This is a high-precision, detail-oriented role requiring strong analytical thinking, structured problem decomposition, and the ability to translate research content into measurable evaluation tasks.
Key Responsibilities
- Build multi-agent benchmark tasks that require reading, analyzing, and synthesizing large document collections
- Curate real-world research corpora — academic papers, case studies, technical reports — and design questions that require comprehensive analysis
- Write structured ground-truth oracles (JSON) with specific, verifiable answers that prove the agent actually read the source material
- Design LLM judge prompts that evaluate agent output field-by-field against the oracle
- Create decomposition guides that split research across multiple parallel sub-agents (one per document, one per domain, then synthesis)
Qualifications & Experience
- 5+ years of experience in research (academic or industry) in a scientific, technical, or analytical domain
- Strong ability to read, analyze, and extract structured information from unstructured documents
- Experience designing or working with structured data formats (JSON, schemas, validation)
- Proficiency in Python scripting (data processing, validation, or evaluation scripts)
- Experience with AI evaluation, coding benchmarks, or structured reasoning tasks (e.g., SWE-bench, Terminal-bench, or similar)
- Experience working with Docker (building images, debugging containers)
- Strong attention to detail, especially when defining exact, verifiable outputs
- Ability to design complex, multi-step problem-solving workflows
How to Apply
🎯 Before You Apply for This Job…
Found a job you really want? Don’t lose the opportunity because of your CV.
You may have the right experience and skills, but if your CV doesn’t clearly match what the employer is looking for, you may never get the interview.
Don’t apply with the same CV you use for every job.
Send us your CV + the job advert, and our recruitment team will customise your CV to position you strongly for that specific role.
Plus, you’ll receive 3 months job search support from our recruiters. You’ll have a dedicated recruiter to support you with your job search and help you find suitable jobs..
Don’t just apply. 👉 Click Here & Get Your CV Customised & Claim Your 3-Month Support Today
Read Job Seeker Testimonials Below: Professionals Who Got Jobs Through Corporate Staffing
Related Jobs To Check Out
Jobs Managed by Our Recruitment Team
The vacancies below are managed directly by the Corporate Staffing recruitment team on behalf of our clients. Review the job advert, then submit your application through our recruitment team for consideration.
Browse Other Jobs Posted Today
Explore the latest job opportunities posted now. Apply early to maximize your chances, as many employers begin reviewing applications as soon as suitable candidates apply.