# Reward Isn't Free: Supervising Robot Learning with Language and Video from the Web

DevFeed: [Reward Isn't Free: Supervising Robot Learning with Language and Video from the Web](<https://devfeed.tech/articles/reward-isn-t-free-supervising-robot-learning-with-language-and-video-from-the-web-7588.md>)

Original publisher: [Read original article](<https://ai.stanford.edu/blog/reward-isnt-free/>)

Author: A Href; Suraj Nair

Published: 2022-01-21T08:00:00Z

Content type: article

Language: en

Sources: [The Stanford AI Lab Blog](<https://devfeed.tech/sources/the-stanford-ai-lab-blog.md>)

Topics: [Robotics](<https://devfeed.tech/topics/robotics.md>), [Deep learning](<https://devfeed.tech/topics/deep-learning.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [foundation-models](<https://devfeed.tech/topics/foundation-models.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>)

Tags: [datasets](<https://devfeed.tech/tags/datasets.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [foundation-models](<https://devfeed.tech/tags/foundation-models.md>), [nlp](<https://devfeed.tech/tags/nlp.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [robotics](<https://devfeed.tech/tags/robotics.md>), [vision](<https://devfeed.tech/tags/vision.md>)

## AI overview

The article examines the challenge of teaching robots to generalize across new environments, tasks, and objects. It discusses using large, diverse datasets and scalable offline learning, while highlighting the limitations of imitation learning and the difficulty of designing or manually annotating reward functions for offline reinforcement learning.

## Source excerpt

This work was conducted as part of SAIL and CRFM. Deep learning has enabled improvements in the capabilities of robots on a range of problems such as grasping 1 and locomotion 2 in recent years. However, building the quintessential home robot that can perform a range of interactive tasks, from cooking to cleaning, in novel environments has remained elusive. While a number of hardware and software challenges remain, a necessary component is robots that can generalize their prior knowledge to new environments, tasks, and objects in a zero or few shot manner. For example, a home robot tasked with setting the dining table cannot afford lengthy re-training for every new dish, piece of cutlery, or dining room it may need to interact with. A natural way to enable such generalization in our robots is to train them on rich data sources that contain a wide range of different environments, tasks, and objects. Indeed, this recipe of massive, diverse datasets combined with scalable offline learning algorithms (e.g. self-supervised or cheaply supervised learning) has been the backbone of the many recent successes of foundation models 3 in NLP 456789 and vision 101112. Replicating these impressive generalization and adaptation capabilities in robot learning algorithms would certainly be a step toward robots that can be used in unstructured, real world environments. However, directly extending this recipe to robotics is nontrivial, as we neither have sufficiently large and diverse datasets of robot interaction, nor is it obvious what type of supervision can enable us to scalably learn useful skills from these datasets. On one hand, the popular imitation learning approach relies on expert data which can be expensive to obtain at scale. On the other hand, offline reinforcement learning, which can be performed using non-expert and autonomously-collected data, requires us to define a suitable reward function. Hard-coded reward functions are often task-specific and difficult to design,