I'm a Member of Technical Staff at Reka AI, where I work on training world models. Most recently I worked on Rho-1, an omni model that understands and generates text, images, and video, reasons over them, and takes actions within a single neural network.
Previously, I was a research scientist at Stability AI working on image and controllable video generation; before that, I spent six years at Amazon working on computer vision for robotics and visual search.
I did my PhD at the Max Planck Institute for Informatics and Saarland University with Bernt Schiele and Matthias Hein.
Stable Diffusion 3.5
Open text-to-image models
Top-k demo
My research focuses on generative models that understand, generate, and reason with multimodal data, including language, images, video, and 3D. I'm especially interested in where 2D and 3D vision meet: novel view synthesis, camera control, and world models that keep a consistent 3D scene. Representative work is highlighted.
Instead of a pipeline of specialist models, one network handles text, images, video, and robot actions as tokens in a single context. It is trained from scratch with next-token prediction and flow matching.
What would a cinematographer ask of a video model? 76 filmmaking controls, 0.25M expert annotations of 20K videos from 10+ models, and a VLM judge that beats zero-shot baselines.
Animate a portrait from any driving video. Masked facial regions carry the motion, and cross-identity training keeps the driver's appearance from leaking through.
Blur, Canny edge, and Depth ControlNets for Stable Diffusion 3.5, enabling high-fidelity upscaling, structural guidance, and depth-conditioned generation.
Stability's flagship open text-to-image models: Large, Large Turbo, and Medium.
On 16 accelerators, SD3 trained 1.5× faster on Gaudi 2 than on H100-80GB. Scaled to 256 accelerators, it processed over 3× more images per second than A100-80GB.
Pre-train video encoders against language describing the actions, so that they learn where in time those actions happen.
One framework for multiclass, top-k, and multilabel losses: which are calibrated for top-k error, and how to optimize all of them with SDCA.
If you are graded on top-5 error, train for it. We introduce the smooth top-k SVM and top-k entropy losses, which improve the top-k accuracy they target across 11 datasets.
A multiclass SVM that only asks for the right answer to be among the top k, trained with a fast projection onto the top-k simplex.
Learn how tasks relate to each other at the same time as the tasks themselves, without giving up efficiency.
Many scene categories, one shared representation, learned jointly with multitask learning that scales to many classes.
Weighted SVM can always replicate an SVM+ solution. SVM+ turns out to be a weighted SVM in disguise, so privileged information is best spent deciding which training examples matter.
Treat large populations as continuous and small ones as discrete and stochastic, which makes transient analysis of Markov population models tractable.
Fast multiclass and multilabel training with stochastic dual coordinate ascent, and the reference implementation of the top-k SVM and top-k entropy losses. A new release, with numerical fixes and performance improvements, is on the way.
Applied science and product work at Amazon.
Catch damaged items before they ship. I led the science roadmap and built one of the deployed models, which treats anomaly detection on RGB-D images as retrieval.
Recognize a product without scanning its barcode. I contributed to the multimodal RGB-D embedding model.
Shop the room: products spotted in lifestyle photos link to matching items on Amazon. I designed and shipped the visual-similarity model.