MIT debuts a method inspired by the big-language model to teach robots new skills

MIT this week presented a new model for training robots. Instead of a standard set of focused data used to teach robots new tasks, the method is broad and mimics the vast troves of data used to train large language models (LLMs).

The researchers note that imitation learning – through which an agent learns by following someone performing a task – can fail if small challenges are introduced. These might be things like lighting, different scenery, or new obstacles. In such scenarios, robots simply wouldn’t have enough data to draw on to adapt.

The team checked out models like GPT-4 for a data-driven brute force approach to solving problems.

“In the linguistic domain, all data is just sentences,” says Lirui Wang, lead writer of the new paper. “In robotics, given all the heterogeneity in the data, if we want to do initial training in a similar way, we need a different architecture.”

The team introduced a new architecture called Heterogeneous Pretrained Transformers (HPT), which collects information from different sensors and different environments. A transformer was then used to mix the data into training models. The larger the transformer, the higher the output power.

Users then enter the robot design, configuration, and task they need to perform.

“Our dream is to have a universal robot brain that you can download and use in your robot without any training,” CMU associate professor David Held said of the research. “While we are still in the early stages, we will continue to push hard and hope that scaling will lead to a breakthrough in robotics policy, just as it has with large language models.”

The research was partially funded by the Toyota Research Institute. Last 12 months at the TechCrunch Disrupt conference, TRI presented a method for training robots at night. It recently formed a groundbreaking partnership that can mix robotic learning research with Boston Dynamics equipment.