Preference Model (founded by Anthropic) is a San Francisco-based "super-intelligent data research company" focused on building reinforcement learning (RL) environments for cutting-edge AI labs to train more powerful and better-aligned models. The company's core belief is that model behavior is almost entirely shaped by reward signals during training. Therefore, they focus on building high-quality, reward-hacking-resistant RL environments, hardening the training harness and graders as attack surfaces to prevent models from finding shortcuts or vulnerabilities to achieve high scores instead of truly solving problems. Over the past year, the company has built RL environments for numerous leading labs, particularly focusing on ML research and engineering tasks (such as writing CUDA kernels, data curation, and experimental design), as these are currently the most valued areas for capability enhancement in these labs. Core open-source project: Karotte (previously introduced): - Provides a framework for building RL environments. - Provides secure default settings and primitives to proactively defend against common reward-based hacking techniques (sandbox isolation, unprivileged execution, killing residual processes before scoring, rejecting dangerous files, preventing OOM/fork bombs, etc.). - Has undergone over 1 million internal evaluation runs and adversarial red team tests for production use. Team: Jennifer Zhou: Co-founder and CEO. Previously worked in the early stages of the Anthropic data team, responsible for building pre-trained data infrastructure, tokenizers, and Claude's pre-trained datasets; Funding: Announced a $16 million seed round on October 7, 2026, led by a16z. Angel investors include renowned researchers such as Fei-Fei Li, Ian Goodfellow, and Julian Schrittwieser. Karotte: https://(github.com)/preferencemodel/karotte Website: https://(preferencemodel.com)/