Three years of continuous engagement in AI data annotation, benchmark engineering, model evaluation, and adversarial red-teaming — measuring reasoning, tool use, computer interaction, terminal operations, planning, instruction-following, and real-world task completion for model families at the current frontier.
At a glance
The same annotate–review–QA system that built Terminal Bench from Linux fundamentals through autonomous long-horizon agents also produced the mathematics stress-testing behind Mixtral 8x7B, the adversarial terminal benchmarks that separate unreliable elicitation from absent capability, and PhD-level power-electronics problems engineered against current SOTA models.
This is not a generalist labeling operation applying one shallow process to every task. It is a single delivery pipeline that flexes from foundational benchmark construction, to frontier red-teaming, to specialist SME work — without lowering the quality bar.
Audience 01
Adversarial evaluation, reliability gaps, and training signal that shapes the next release — not commodity labels.
Audience 02
Production-scale SFT, tool-calling, long-horizon workflows, and safety data under NDA-ready delivery.
Audience 03
Benchmark construction, Harbor-validated coding agents, GUI/computer-use suites, and process-level execution review.
Three representative outcomes from the portfolio — model contribution, reliability measurement, and SME depth.
Model contribution
Adversarial mathematics and logic tasks engineered against Mistral’s then-current enterprise model. The resulting data fed directly into the creation of Mixtral 8x7B — evaluation that became training signal for the next model.
Reliability signal
Adversarial Terminal Bench tasks where golden solutions pass every test, while top commercial models — including Claude and Qwen — pass only a fraction across repeated runs (e.g. a subset of 16 attempts). That gap isolates unreliable elicitation from absent capability.
SME depth
Complete research problems in Power Electronics, Embedded Systems, Chip Design, and Power Systems — constructed to break Gemini, Claude, and GPT. Not templated questions: work that requires genuine electrical-engineering depth to author.
Ordered from the most technically demanding work to the most foundational. Filter by tier. Confidential rows are marked — not guessed.
12+
Named frontier organizations
21
Distinct engagements
4
Major benchmark systems
Soros — Electrical Engineering
Gemini, Claude, GPT (SOTA)
PhD-level Power Electronics, Embedded Systems, Chip Design & Power Systems problems
Soros — PCB Design
Emerging PCB-design LLMs
Golden-solution PCB designs with full component specs; near-zero prior training signal
Mistral Math & Logic Stress-Test
Mistral → Mixtral 8x7B
Adversarial math/logic design that fed directly into the resulting model
Gemini Spreadsheet Stress-Test
Gemini 3 Pro
7–8-class spreadsheet taxonomy + adversarial structured-data tasks
Qwen Time-Reasoning Stress-Test
Qwen
Interdependent, multi-timezone scheduling to stress temporal logic
Alibaba/Tencent Adversarial Terminal Bench
Claude, Qwen (commercial)
Golden solutions pass all tests; SOTA models pass only a fraction across 16 runs
Google GDM SWE-Bench
Google DeepMind
Harbor validation, Agentic Vet panel, patch & bug-fix remediation
Google Antigravity
Google Antigravity
Live GitHub PR resolution on 1,000+-star repos via remote SSH
Meta Agentic Code Generation
Meta
GitHub-issue-based golden-response construction
Terminal Bench 1.0 → 3.0
Alibaba, Meta, Google, xAI, Reflection AI, DataHub, TML
Flagship Linux/agentic benchmark: fundamentals → workflows → long-horizon agents
OSWorld (+ GUI scripting)
Multi-lab
PyAutoGUI VM capture across Chrome, LibreOffice, Terminal, VS Code
Video / Execution Review
Multi-lab
Full-process scoring of recorded agent runs — not final outputs alone
Meta RLHF Code Evaluation
Meta
Python/JS/SQL execution validation + scoring for training feedback
Meta OpenClaw
Meta
Swarm-agent VM tasks via OpenClaw/Maton + MCP; 3,000+ samples
Apple Agentic AI
Apple (Siri)
Voice-style SFT prompting written as users actually speak
ServiceNow Agentic AI
ServiceNow
Long-horizon, multi-step, end-to-end workflow design
NVIDIA SFT
NVIDIA
100-API workflow tool-calling with script / LLM-judge verification
Penguin SFT & Tool Calling
ConfidentialConfidential engagement
API-call accuracy; hallucination-free JSON handling
Penguin Safety SFT
ConfidentialConfidential engagement
Jailbreak resistance; sensitive-information leakage prevention
Google DeepMind Personalization
Gemini Pro 3.1 + beta models
Cross-service memory evaluation — Gmail, Calendar, search history
Dove Bookkeeping
ConfidentialConfidential engagement
Finance-domain golden trajectories from real invoices, receipts & transactions
Electrical Engineering & PCB (Soros)
Gemini, Claude, GPT (SOTA)
Covered in Tier 1 — highest technical demand in the portfolio
Video-Generation Frame Labeling
Video-generation models
Frame-to-frame instruction labeling for coherent video synthesis
Soros — Electrical Engineering
Gemini, Claude, GPT (SOTA)
PhD-level Power Electronics, Embedded Systems, Chip Design & Power Systems problems
Soros — PCB Design
Emerging PCB-design LLMs
Golden-solution PCB designs with full component specs; near-zero prior training signal
Mistral Math & Logic Stress-Test
Mistral → Mixtral 8x7B
Adversarial math/logic design that fed directly into the resulting model
Gemini Spreadsheet Stress-Test
Gemini 3 Pro
7–8-class spreadsheet taxonomy + adversarial structured-data tasks
Qwen Time-Reasoning Stress-Test
Qwen
Interdependent, multi-timezone scheduling to stress temporal logic
Alibaba/Tencent Adversarial Terminal Bench
Claude, Qwen (commercial)
Golden solutions pass all tests; SOTA models pass only a fraction across 16 runs
Google GDM SWE-Bench
Google DeepMind
Harbor validation, Agentic Vet panel, patch & bug-fix remediation
Google Antigravity
Google Antigravity
Live GitHub PR resolution on 1,000+-star repos via remote SSH
Meta Agentic Code Generation
Meta
GitHub-issue-based golden-response construction
Terminal Bench 1.0 → 3.0
Alibaba, Meta, Google, xAI, Reflection AI, DataHub, TML
Flagship Linux/agentic benchmark: fundamentals → workflows → long-horizon agents
OSWorld (+ GUI scripting)
Multi-lab
PyAutoGUI VM capture across Chrome, LibreOffice, Terminal, VS Code
Video / Execution Review
Multi-lab
Full-process scoring of recorded agent runs — not final outputs alone
Meta RLHF Code Evaluation
Meta
Python/JS/SQL execution validation + scoring for training feedback
Meta OpenClaw
Meta
Swarm-agent VM tasks via OpenClaw/Maton + MCP; 3,000+ samples
Apple Agentic AI
Apple (Siri)
Voice-style SFT prompting written as users actually speak
ServiceNow Agentic AI
ServiceNow
Long-horizon, multi-step, end-to-end workflow design
NVIDIA SFT
NVIDIA
100-API workflow tool-calling with script / LLM-judge verification
Penguin SFT & Tool Calling
Confidential engagement
API-call accuracy; hallucination-free JSON handling
Penguin Safety SFT
Confidential engagement
Jailbreak resistance; sensitive-information leakage prevention
Google DeepMind Personalization
Gemini Pro 3.1 + beta models
Cross-service memory evaluation — Gmail, Calendar, search history
Dove Bookkeeping
Confidential engagement
Finance-domain golden trajectories from real invoices, receipts & transactions
Electrical Engineering & PCB (Soros)
Gemini, Claude, GPT (SOTA)
Covered in Tier 1 — highest technical demand in the portfolio
Video-Generation Frame Labeling
Video-generation models
Frame-to-frame instruction labeling for coherent video synthesis
Showing 23 of 23 engagements · Ordered most technically elite first within each tier
High-quality benchmark and training tasks: realistic Linux and GUI scenarios, multi-step and long-horizon engineering work. Verify expected outputs. Test reproducibility from clean environments. Design for diversity and realistic interaction — not convenience.
Independent validation: remove ambiguous instructions, hidden hints and shortcuts, information leakage. Ensure fairness, clarity, and edge-case coverage. Nothing ships on the annotator’s judgment alone.
Production layer: annotation verification, policy compliance, consistency detection, reproducibility standards, structured feedback to annotators, and formal sign-off before delivery.
Delivery stack
Enterprise posture
NDA-ready delivery
Engagements under internal project codes when end-client names cannot be disclosed. Confidential work is flagged — never invented.
Independent review layer
Nothing ships on a single annotator’s judgment. Ambiguity, leakage, and hidden shortcuts are removed before QA.
Reproducible environments
Tasks verified from clean environments with deterministic execution — script checks and LLM-judge verification at scale.
Production sign-off
Policy compliance, consistency checks, structured feedback, and formal approval before a dataset is production-ready.
Technical reach
Collectively contributing to LLMs, coding assistants, autonomous software-engineering agents, computer-use agents, and evaluation frameworks themselves.
Named where disclosable · confidential engagements flagged in matrix
Designed for procurement, research leads, and evaluation programs that need a clear path from NDA to signed-off data.
Define task families, success criteria, volume, and confidentiality boundary in a technical scoping call.
Small, reviewable batch through the full annotate–review–QA pipeline — calibrated to your quality bar before scale.
Scaled throughput with reproducible environments, automated checks, and formal sign-off on every release.
Frontier labs, foundation-model teams, enterprise agent platforms, and research/evaluation programs that need training data and benchmarks at production quality — not crowd-sourced labels.
Every engagement runs through the same annotate → review → QA pipeline, with independent review, reproducibility checks, and formal sign-off before a dataset is production-ready.
Yes. Several portfolio rows were delivered under internal project codenames. We mark those as confidential rather than inventing or guessing end-client names.
A progressive Linux/agentic benchmark built in three stages — fundamentals, realistic multi-skill workflows, and autonomous long-horizon agents — used across multiple frontier organizations.
Confidential technical briefing
Foundational rigor, frontier-grade red-teaming, and specialist domain depth — inside one pipeline. Request a confidential technical briefing.
