Senior AI Quality & Evaluation Engineer

Solvd

  • Remote - US
  • Remote
  • Senior level
  • 3+ yrs experience
  • Quality Engineering / AI Evaluation
  • Visa: not specified

The role in 30 seconds

This role owns the independent proof that a retrieval-augmented generation system works and that permissions hold, serving as an acceptance gate so the engineers who built the system are not grading their own work. The engineer builds and runs the golden question suite, negative entitlement tests, freshness/retirement checks, and drift alarms, automating the harness in CI and reporting results to client stewards. It is a 50% allocation role drawn from a Quality Engineering or GRC background, reporting to the POD Lead and working day-to-day with the retrieval engineer at Solvd.

Is this role a fit for you?

The backgrounds this role is most likely to hire from, as RoleStack reads the posting.

Strong match

  • Data platform / ML pipeline
  • B2B infrastructure / API / developer tools
  • Enterprise-scale
  • Growth-stage

Workable stretch

  • B2B enterprise platform / workflow software
  • Internal tooling / back-office systems
  • Regulated environment

Harder transition

  • B2C consumer-facing application
  • Embedded systems / hardware-adjacent
  • Startup / greenfield

Industry: Software, SaaS & AI Platform (AI-native consulting delivery). See your personal match score

What they're really screening for

Must-haves

  • CORE 3+ years in software quality engineering or test automation with Python, including building test harnesses that run in CI on a schedule
  • CORE Hands-on evaluation of a search or RAG system in production — writing ground truth sets, scoring relevance and grounding, and tracking drift over time
  • CORE Experience designing security-focused negative tests — authorization, entitlement, data exposure — and reporting them at the severity a client security team expects
  • CORE Ability to work in AWS — invoking Bedrock and OpenSearch APIs, reading CloudWatch, deploying a scheduled job with infrastructure code
  • PLUS Clear written reporting; the suite's output is read by client stewards, not only engineers
  • PLUS Experience building for enterprise client stewards and platform owners (client-facing delivery context)

Nice-to-haves

  • PLUS AWS Certified AI Practitioner or Machine Learning Engineer Associate
  • PLUS Experience with RAG evaluation libraries and LLM-as-judge scoring with human calibration
  • PLUS Contact center or knowledge management domain exposure
  • PLUS Familiarity with SOC 2 or ISO 27001 evidence practices

Résumé keywords for this role

Terms the posting stresses. Include the ones you can honestly back up.

Tools & technologies

  • Python
  • AWS
  • Bedrock
  • OpenSearch
  • CloudWatch
  • RAG
  • LLM-as-judge
  • CI
  • SOC 2
  • ISO 27001
  • AWS Certified AI Practitioner
  • Machine Learning Engineer Associate
  • RAG evaluation libraries

Concepts & phrases

  • golden question suite
  • negative entitlement test set
  • retrieval works and permissions hold
  • acceptance gates
  • relevance, grounding, and citation accuracy
  • severity-one alarm
  • freshness and retirement tests
  • quality dashboard
  • regression alarms
  • QA gate
  • evaluation runbook
  • LLM-as-judge scoring
  • human calibration
  • test harnesses that run in CI
  • ground truth sets

Words that count as the same thing: "RAG" ≈ Retrieval-Augmented Generation, retrieval-augmented generation systems · "ground truth sets" ≈ golden question suite, evaluation datasets, labeled test sets · "grounding" ≈ faithfulness, answer groundedness, citation accuracy · "entitlement" ≈ authorization, access control, permissions · "LLM-as-judge" ≈ model-based evaluation, automated LLM scoring

Check my résumé against these keywords

Hiring at Solvd

Open senior roles RoleStack tracks right now, posted in the last 30 days.

More open roles at Solvd

Full job description from Solvd

From Solvd's posting. View the original

Solvd Inc. is a rapidly growing AI-native consulting and technology services firm delivering enterprise transformation across cloud, data, software engineering, and artificial intelligence. We work with industry-leading organizations to design, build, and operationalize technology solutions that drive measurable business outcomes.

Following the acquisition of Tooploox, a premier AI and product development company, Solvd now offers true end-to-end delivery—from strategic advisory and solution design to custom AI development and enterprise-scale implementation. Our capability centers combine deep technical expertise, proven delivery methodologies, and sector-specific knowledge to address complex business challenges quickly and effectively.

Show full description Show less

We are looking for a Senior AI Quality & Evaluation Engineer to own the independent proof that retrieval works and that permissions hold. You’ll build and run the golden question suite, negative entitlement tests, and drift alarms that serve as acceptance gates — so the engineers who built the system are not grading their own work.

This is a 50% allocation role across weeks 3–10, drawn from a Quality Engineering or GRC background, reporting to the POD Lead and working day-to-day with the retrieval engineer.

WHAT YOU’LL DO

- Build the golden question suite from top contact drivers and needs workshops — question, expected source, expected passage, and acceptance thresholds for relevance, grounding, and citation accuracy, versioned alongside the content it tests.

- Build the negative entitlement test set — queries issued under unentitled identities that must return no restricted passage, with any failure raised as a severity-one alarm.

- Build freshness and retirement tests — source-modified-to-index-available lag per item against the agreed threshold, and verification that retired items disappear on the next sync.

- Automate the harness to run on schedule and after every content, configuration, or model change — publishing scores to the quality dashboard with trend lines and regression alarms.

- Define the QA gate content analysts apply to remediated content.

- Run the suite before every pilot change in weeks 7–9 and report misses with root cause — content, chunking, ranking, or permissions — to the right owner.

- Write the evaluation runbook and train client stewards and platform owners to add questions, re-run the suite, and interpret regressions.

REQUIRED SKILLS & EXPERIENCE

- 3+ years in software quality engineering or test automation with Python, including building test harnesses that run in CI on a schedule.

- Hands-on evaluation of a search or RAG system in production — writing ground truth sets, scoring relevance and grounding, and tracking drift over time.

- Experience designing security-focused negative tests — authorization, entitlement, data exposure — and reporting them at the severity a client security team expects.

- Ability to work in AWS — invoking Bedrock and OpenSearch APIs, reading CloudWatch, deploying a scheduled job with infrastructure code.

- Clear written reporting; the suite’s output is read by client stewards, not only engineers.

PREFERRED QUALIFICATIONS

- AWS Certified AI Practitioner or Machine Learning Engineer Associate.

- Experience with RAG evaluation libraries and LLM-as-judge scoring with human calibration.

- Contact center or knowledge management domain exposure.

- Familiarity with SOC 2 or ISO 27001 evidence practices.

When you join Solvd, you'll…

- Shape real-world AI-driven projects across key industries, working with clients from startup innovation to enterprise transformation.

- Be part of a global team with equal opportunities for collaboration across continents and cultures.

- Thrive in an inclusive environment that prioritizes continuous learning, innovation, and ethical AI standards.

Ready to make an impact?

If you're excited to build things that matter, champion responsible AI, and grow with some of the industry’s sharpest minds. Apply today and let’s innovate together.

Solvd is an equal opportunity employer.

FAQ

How much experience do you need?

The posting asks for 3+ years in software quality engineering / test automation. It is a senior-level role.

Does Solvd sponsor visas for this role?

The posting does not say whether visa sponsorship is available. Ask the recruiter before you apply.

Is this role remote?

It is a remote role.

What tools and technologies does the posting mention?

Python, AWS, Bedrock, OpenSearch, CloudWatch, RAG, LLM-as-judge, CI, SOC 2, ISO 27001, AWS Certified AI Practitioner, Machine Learning Engineer Associate, RAG evaluation libraries.