←

Building Ego-Centric Datasets
for Physical AI

Published: August 15, 2026 General Machines Team

Task demo - first-person perspective

Figure 1: First-person video from a robot manipulation task. Each frame carries full spatial and temporal context about how the agent relates to the objects it interacts with.

Training physical AI systems requires data that no benchmark dataset provides. Ego-centric video - first-person recordings of tasks performed in real environments - is the closest proxy to how a robot or spatial agent experiences the world. We describe our approach to collecting, annotating, and structuring ego-centric and embodied datasets at scale, and the principles that guide what we capture and why.

1. The Data Problem in Physical AI

1.1 Why Physical AI Data Is Hard to Collect

The data problem in physical AI is qualitatively different from the data problems that earlier generations of language and vision models faced. You cannot scrape it from the internet. You cannot generate it synthetically at the fidelity that real deployment requires. Every data point requires a human, a camera, a physical environment, and a task sequence - and every one of those has to be logged, synced, and verified before it is usable.¹1. Synthetic data can close distribution gaps for rare or dangerous events, but for core manipulation and navigation competencies, real-world data remains essential.

This means that the teams building frontier physical AI systems are spending enormous fractions of their engineering capacity on data infrastructure rather than on model development. Data collection pipelines, annotation tooling, quality filtering, and dataset curation are full-time problems for teams that nominally exist to build robots. We exist to take those problems off their plate.

1.2 Why the Ego-Centric Perspective Matters

Most existing robot datasets are captured from third-person cameras - a fixed overhead or external view of a robot completing a task. This is useful for studying behavior from the outside. It is not useful for training the robot’s own perception.²2. Third-person data teaches an observer; first-person data teaches a participant. See the EPIC-Kitchens 100 dataset for a large-scale study of what ego-centric annotation enables.

An ego-centric dataset captures the world from the agent’s perspective: what the robot sees as it reaches for an object, what the hand looks like as it grasps and manipulates, how the scene changes as the agent moves. This is the visual substrate that the model needs to close the perception-to-action loop in real deployment.

Robot hands reaching for a mug in a kitchen, with green skeletal annotation overlay and bounding box labels showing contact probability and task phase — Figure 1: Ego-centric frame from a domestic manipulation task. Skeletal hand keypoints (green) and object bounding box annotations capture the spatial relationship between the agent and target object. Labels indicate contact probability (0.91) and current task phase (reach). This annotation density is what separates training-ready data from raw footage.

2. Our Collection Approach

2.1 Environments

We collect data across three primary environment categories: domestic (kitchen, laundry, general household), light industrial (assembly, packaging, material handling), and unstructured outdoor (navigation, tool use in variable terrain). Each environment category presents a different set of perceptual challenges and requires different annotation schema.