What kind of tasks will humanoid robots be able to perform in the future when they enter homes, factories, supermarkets, and other spaces? Shanghai Artificial Intelligence Laboratory, in collaboration with the National and Local Co-constructed Humanoid Robot Innovation Center, and Shanghai Kupas Technology Co., Ltd., recently launched AgiBot World, an open-source project featuring millions of real-machine datasets based on global real-world scenarios. This dataset, designed for global embodied intelligence developers, includes more than 80 work skills, showcasing the broad application prospects of humanoid robots as they master household tasks and beyond.
Humanoid Robots Master Household Tasks
Why is it necessary to open-source so much data? Peng Zhihui explained that in embodied intelligence, data diversity and authenticity are essential for driving algorithmic innovation. However, the cost and challenges associated with collecting real-world machine data are significantly high. As a “unicorn” company, Zhiyuan leverages its commitment to technological openness and industry responsibility, aiming to enable many scientific research teams to train embodied intelligence algorithms using real data, thereby accelerating technological advancements and product applications.
Training Robots in Data Collection Factories
The AgiBot World data set comes from Zhiyuan’s data collection factory. This factory houses hundreds of humanoid robots and features five distinct application scenarios: home, catering, supermarkets, offices, and industrial settings, encompassing over 3,000 real-world objects.
In the home scenario, the company has replicated the layout of a real house, including living rooms, bedrooms, kitchens, bathrooms, and other functional spaces. For the industrial setting, they have constructed warehouses and production lines equipped with sorting systems, packaging equipment, and conveyor belts, allowing robots to be trained for sorting, packaging, and material handling tasks.
Revolutionising Robotics: Precision Training and Advanced Capabilities
In this highly simulated environment, company employees train humanoid robots daily to perform various tasks. Using two handles, data collectors repeat various actions, such as grabbing items from an assembly line and placing them into designated packaging boxes or scanning QR codes with a scanning device before bagging the product and handing it to a customer. Through remote control, the robots mimic these actions precisely. During this “hands-on” training, all data is uploaded to the cloud, becoming part of the AgiBot World dataset.
Peng Zhihui explained that Zhiyuan equips each robot with 8 surround-layout cameras, enabling real-time, 360-degree environmental perception. The robots feature 6-degree-of-freedom dexterous hands designed for precise and flexible movements. These hands are fitted with six-dimensional force sensors and high-precision tactile sensors, allowing the robots to detect subtle changes in force and apply appropriate pressure, achieving “moderation.” Additionally, with 32 active degrees of freedom throughout their bodies, these robots can easily adapt to and handle various complex tasks.
To ensure the quality of the data set, the candidate data collected by employees will be strictly screened on the end and cloud sides. The data collection system will automatically eliminate the data that does not meet the requirements. Then, the auditor will review it frame by frame to ensure that every action made by the robot meets the task standards. Finally, the data will be subject to secondary verification by the algorithm.
AgiBot World Dataset: Setting New Benchmarks in Robotics Training
The AgiBot World dataset spans over 100 real-world scenarios, with home environments making up 40%, catering and industrial settings 20%, and supermarket and office scenarios 10%. Among the tasks performed for data collection, approximately 80% are long-range tasks lasting 60-150 seconds.
Notably, AgiBot World outperforms Google’s Open X-Embodiment dataset, offering 10 times more long-range data and 100 times broader scene coverage.
As AI language models rely on extensive corpus training, humanoid robots require vast datasets to evolve into proficient workers or service providers. By open-sourcing the AgiBot World dataset, countless research teams can leverage its rich data to train embodied intelligence models, enabling humanoid robots to acquire and perfect various standardized skills.
Mastering Precision: AgiBot World’s Comprehensive Action Dataset
The AgiBot World dataset encompasses many actions, from basic operations like grabbing, placing, pushing, and pulling to more intricate tasks such as stirring, folding, and ironing. These represent the essential “atomic actions” required for daily human activities.
Long-range tasks, composed of multiple “atomic actions,” often demand precision and pose significant challenges during robot training and data collection. For instance, after being trained to use a dishwasher, a robot can meticulously load tableware into the correct slots, even when the sink is cluttered with stacked dishes. Similarly, in installing computer memory sticks, repeated training enables the robot to achieve millimeter-level precision, allowing it to insert memory sticks accurately into the designated slots in a computer host.
This dataset’s depth and diversity pave the way for humanoid robots to master fundamental and advanced skills, bringing them closer to seamless integration into everyday life.
A Landmark Open-Source Project for Embodied Intelligence
“We hope this will be a milestone open-source project, marking the ‘ImageNet moment’ in the field of embodied intelligence,” said Peng Zhihui in an interview.
ImageNet, created by Chinese-American scientist Fei-Fei Li and her team, revolutionized deep learning by providing a large-scale visualization database for visual object AI recognition research. AgiBot World’s million-level real machine dataset—comprising sensor, operational, and environmental interaction data—is poised to have a similarly transformative impact.
This dataset is expected to lower the research barrier in embodied intelligence significantly, driving innovation in humanoid robotics, fostering interdisciplinary collaboration, and accelerating real-world industry applications.
AgiBot World Challenges: Shaping the Future of Humanoid Robotics
This year, the Shanghai-based company is launching a series of AgiBot World Challenges to attract scientific research teams and innovative talent from around the globe. These challenges will create a platform for robots developed using the AgiBot World dataset to compete in real-world scenarios. Participants will have the opportunity to engage in technical exchanges and collaborations, working together to explore and establish technical standards and specifications for this rapidly emerging industry.
Peng Zhihui envisions the future of humanoid robots in the coming years. “In the next 2-3 years, humanoid robots will become integral to the manufacturing industry, particularly in flexible production and assembly line environments, where they will play an irreplaceable role,” he stated. “By the next five years, we expect them to enter the housekeeping sector, where they will perform basic household tasks, provide companionship, and assist with caregiving—becoming essential members of many families.”
Leave a Reply