Titlesby GlyphEvals Engineer job title trend since Apr 2011
Engineering · Emerging · in 0% of HN hiring posts · 14 open roles at 8 companies · median $232k
Measures model and agent quality: builds eval suites, graders and the dashboards teams ship against.
Why it exists
Written from the sources below
Language model output cannot be checked with ordinary unit tests, so teams began building evaluation suites and graders as their own discipline. OpenAI open-sourced its Evals framework in 2023 and practitioners such as Hamel Husain argued every AI product needs evals, which turned the work into a job title.
- Postingopenai/evals: a framework for evaluating LLMs and LLM systemsOpenAI on GitHub, 2023 · backs: an open framework and registry for LLM evals
- EssayYour AI Product Needs EvalsHamel Husain, Mar 2024 · backs: evals as the core of improving AI products
Where it is posted
Sectors
- AI lab5
- Robotics and autonomy2
- Security1
Cities
- San Francisco4
- Palo Alto4
- Washington1
- Austin1
- New York1
- Pittsburgh1
Countries and remote
- United States12
- United Kingdom1
Posted pay
Middle half $212k to $256k, median $232k from 10 postings that state a base range (USD a year).
Pay by level and city on SalariesWho is hiring for it
Open postings on their own job boards, read 5h ago| # | Company | Open roles | |
|---|---|---|---|
| 1 | Anduril IndustriesA company hiring most for Systems Engineer and Test Engineer. | 5 | |
| 2 | 2 | ||
| 3 | 2 | ||
| 4 | 1 | ||
| 5 | AvrideA company hiring most for Machine Learning Engineer and Data Scientist. | 1 | |
| 6 | 1 | ||
| 7 | 1 | ||
| 8 | 1 |
Open roles
8 of 14; company pages list the restSenior Software Engineer, Eval Efficiency & Optimization
Mountain View, CA, USA; San Francisco, CA, USA; New York, NY, USA; Pittsburgh, PA, USA
Principal Test & Evaluation Engineer - Networked Battlefield Systems
Costa Mesa, California, United States
Cyber Evaluations Engineer
Remote-Friendly, United States; San Francisco, CA | Washington, DC
| Role | |||
|---|---|---|---|
| Senior Software Engineer, Eval Efficiency & Optimization | Mountain View, CA, USA; San Francisco, CA, USA; New York, NY, USA; Pittsburgh, PA, USA | $213k to $263k | |
| Software Engineer, Evaluation Infrastructure | Palo Alto, CA | $180k to $440k | |
| Principal Test & Evaluation Engineer - Networked Battlefield SystemsAnduril IndustriesCosta Mesa, California, United States$220k to $292k | Anduril Industries | Costa Mesa, California, United States | $220k to $292k |
| Cyber Evaluations Engineer | Remote-Friendly, United States; San Francisco, CA | Washington, DC | $300k to $405k | |
| Senior Software Engineer, Evaluation Engine | Remote U.S. | $195k to $229k | |
| Product Evals Engineer, ChatGPT Learning | San Francisco | $185k to $490k | |
| Software Engineer, Evaluation Platform / Infra | San Francisco | Not stated | |
| Lead Evaluation Engineer - Scenario Coverage & Datasets (Autonomous Driving)AvrideAustin, Texas | Avride | Austin, Texas | Not stated |
Lineage
evaluation split out of AI engineering · source
Share of Hacker News hiring posts
0 posts since Apr 2011Related titles
About this title
Measures model and agent quality: builds eval suites, graders and the dashboards teams ship against.
Also counted: evaluation engineer, model evaluation
Median posted salary $232k from 10 postings that state a base pay range (the median of their midpoints, USD a year). Not a survey of what people earn.