Member of Technical Staff, Pretraining Evaluations

Hark

  • San Jose, CA
  • On-site
  • Senior level
  • AI/ML Research
  • Visa: not specified
  • $180k–$450k

The role in 30 seconds

Hark is building advanced, personalized intelligence that is proactive, multimodal, and capable of interacting with the world. The Omni team is developing multimodal foundation models for seamless, real-time multimodal intelligence to power intuitive and immersive user experiences beyond text. This role will build and maintain the systems used to evaluate these models during pretraining and mid-training.

Is this role a fit for you?

The backgrounds this role is most likely to hire from, as RoleStack reads the posting.

Strong match

  • Software, SaaS & AI Platform
  • Developers / technical users
  • Internal users (employees, ops teams)
  • Enterprise-scale (high availability, compliance, large distributed systems)

Workable stretch

  • B2B enterprise platform / workflow software
  • B2B infrastructure / API / developer tools
  • Enterprise users (complex workflows, admin panels, role-based access)
  • Growth-stage (scaling existing systems, reliability concerns)

Harder transition

  • Internal tooling / back-office systems
  • B2C consumer-facing application
  • SMB / small business
  • Consumer / general public
  • Startup / greenfield (build from scratch, speed over polish)

Industry: Software, SaaS & AI Platform. See your personal match score

What they're really screening for

Must-haves

  • CORE Experience building model evaluation systems or experimental tools for reproducible comparisons across models or checkpoints.
  • CORE Knowledge of language model evaluation methods, including log-likelihood and generative scoring, prompt sensitivity, few-shot variance, and formatting effects.
  • CORE Strong statistical judgment, including uncertainty estimation, sample-size planning, and distinguishing statistical significance from practical improvements.
  • CORE Experience identifying benchmark contamination, saturation, or gaps between benchmark scores and the intended capability.
  • CORE Strong Python engineering skills, including building reliable evaluation pipelines that run efficiently across many checkpoints.
  • CORE A record of developing useful benchmarks, evaluation methods, or tools that inform research and engineering decisions.

Nice-to-haves

  • PLUS Experience designing benchmarks with human ratings or model-based graders.
  • PLUS Experience studying capability scaling or predicting larger-model performance from smaller experiments.
  • PLUS Experience with code execution sandboxes, math verifiers, or other programmatic graders.
  • CORE Experience evaluating checkpoints during pretraining or mid-training.
  • CORE Multimodal evaluation experience: audio, speech, vision, or interleaved inputs.
  • PLUS Experience evaluating model quality under constrained inference budgets.

Résumé keywords for this role

Terms the posting stresses. Include the ones you can honestly back up.

Tools & technologies

  • AI
  • Hark
  • Omni
  • Python
  • LLM
  • log-likelihood
  • few-shot prompts
  • checkpoints
  • benchmarks
  • audio
  • speech
  • vision

Concepts & phrases

  • Member of Technical Staff
  • Pretraining Evaluations
  • advanced, personalized intelligence
  • next-generation hardware
  • universal interface
  • agentic systems
  • multimodal models
  • next-generation AI hardware
  • unified interface
  • Omni team
  • multimodal foundation models
  • pretraining and mid-training
  • held-out loss
  • multimodal tasks
  • training runs reproducible

Words that count as the same thing: "agentic systems" ≈ autonomous AI systems, AI agents · "multimodal models" ≈ AI models handling multiple data types, AI for text, audio, vision · "held-out loss" ≈ model performance on unseen data, evaluation loss · "few-shot prompts" ≈ prompting with limited examples, few-shot learning evaluation · "checkpoints" ≈ model snapshots, saved model states

Check my résumé against these keywords

Hiring at Hark

Open senior roles RoleStack tracks right now, posted in the last 30 days.

More open roles at Hark

Full job description from Hark

From Hark's posting. View the original

About Hark

Hark is an artificial intelligence company building advanced, personalized intelligence. One that is proactive, multimodal, and capable of interacting with the world through speech, text, vision, and persistent memory.

We're pairing that intelligence with next-generation hardware to create a universal interface between humans and machines. While today's AI largely operates through chat boxes and decade-old devices, Hark is focused on what comes next: agentic systems that interact naturally with people and the real world.

Show full description Show less

To get there, we're developing multimodal models and next-generation AI hardware together - designed from the ground up as a single, unified interface for a new era of intelligent systems.

About the Role

The Omni team at Hark is building the next generation of AI experiences beyond text, enabling models to understand and generate content across multiple modalities, including text, audio, and beyond. Our goal is to create seamless, real-time multimodal intelligence that powers intuitive and immersive user experiences.

As part of the Omni team, you will build the multimodal foundation models at the core of this effort. You'll build the systems used to evaluate Hark's models during pretraining and mid-training. You'll track held-out loss, measure capabilities across text and multimodal tasks, and make comparisons between training runs reproducible. Your analysis will help researchers choose training recipes, select checkpoints, and identify regressions.

Responsibilities

• Build and maintain an evaluation harness for held-out loss, performance across data domains, and capabilities in text, code, math, and reasoning.

• Design evaluations for early checkpoints and smaller models using log-likelihood scoring, few-shot prompts, and generated completions.

• Assess benchmark coverage and saturation, quantify uncertainty in results, and work with data researchers to detect training-data contamination.

• Version datasets, prompts, graders, and inference settings so results can be reproduced and compared across checkpoints and training runs.

• Analyze experiments and work with researchers to identify useful recipe changes, select checkpoints, and prioritize capability improvements.

• Develop new benchmarks for capabilities that existing evaluations do not adequately measure.

• Extend evaluation tools to image, video, speech, and audio tasks in collaboration with multimodal researchers.

Requirements

• Experience building model evaluation systems or experimental tools for reproducible comparisons across models or checkpoints.

• Knowledge of language model evaluation methods, including log-likelihood and generative scoring, prompt sensitivity, few-shot variance, and formatting effects.

• Strong statistical judgment, including uncertainty estimation, sample-size planning, and distinguishing statistical significance from practical improvements.

• Experience identifying benchmark contamination, saturation, or gaps between benchmark scores and the intended capability.

• Strong Python engineering skills, including building reliable evaluation pipelines that run efficiently across many checkpoints.

• A record of developing useful benchmarks, evaluation methods, or tools that inform research and engineering decisions.

Bonus Qualifications

• Experience designing benchmarks with human ratings or model-based graders.

• Experience studying capability scaling or predicting larger-model performance from smaller experiments.

• Experience with code execution sandboxes, math verifiers, or other programmatic graders.

• Experience evaluating checkpoints during pretraining or mid-training.

• Multimodal evaluation experience: audio, speech, vision, or interleaved inputs.

• Experience evaluating model quality under constrained inference budgets.

Compensation

The US base salary range for this full-time position is between $180,000 - $450,000 annually.

The pay offered for this position may vary based on several individual factors, including job-related knowledge, skills, and experience. The total compensation package may also include additional components/benefits depending on the specific role. This information will be shared if an employment offer is extended.

FAQ

How much does a Member of Technical Staff, Pretraining Evaluations at Hark make?

The posted pay for this role is $180k–$450k a year.

Does Hark sponsor visas for this role?

The posting does not say whether visa sponsorship is available. Ask the recruiter before you apply.

Is this role remote?

It is an on-site role.

What tools and technologies does the posting mention?

AI, Hark, Omni, Python, LLM, log-likelihood, few-shot prompts, checkpoints, benchmarks, audio, speech, vision.