# Self-supervised learning A machine learning technique that generates implicit labels from unstructured data rather than relying on manually labeled datasets. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## LinkBERT: Improving Language Model Training with Document Link DevFeed: [LinkBERT: Improving Language Model Training with Document Link]() Original publisher: [Read original article]() Author: A Href; Michihiro Yasunaga; Jure Leskovec; Percy Liang Published: 2022-05-31T07:00:00Z Content type: article Language: en Sources: [The Stanford AI Lab Blog]() Topics: [Large Language Model](), [Natural language processing](), [Training AI Models](), [Self-supervised learning](), [Web](), [data](), [Fine-tuning]() Tags: [data](), [google](), [language-models](), [model](), [model-training](), [natural-language-processing](), [self-supervised-learning](), [training](), [web]() ### AI overview The article introduces LinkBERT in the context of language-model pretraining. It explains how BERT and GPT use self-supervised objectives to learn from large text corpora, then describes a limitation of treating documents independently: hyperlinks and citation links connect knowledge across documents and can support multi-hop understanding. The supplied text ends while introducing this challenge. ### Source excerpt Language Model Pretraining Language models (LMs), like BERT 1 and the GPT series 2, achieve remarkable performance on many natural language processing (NLP) tasks. They are now the foundation of today's NLP systems. 3 These models serve important roles in products and tools that we use every day, such as search engines like Google 4 and personal assistants like Alexa 5. These LMs are powerful because they can be pretrained via self-supervised learning on massive amounts of text data on the web without the need for labels, after which the pretrained models can be quickly adapted to a wide range of new tasks without much task-specific finetuning. For instance, BERT is pretrained to predict randomly masked words in original text (masked language modeling), e.g. predicting the masked word "dog" from "My __ is fetching the ball". GPTs are pretrained to predict the next word given a previous sequence of text (causal language modeling), e.g. predicting the next word "ball" from "My dog is fetching the". In either cases, through pretraining, LMs learn to encode various knowledge from a text corpus that helps to perform downstream applications involving language understanding or generation. In particular, LMs can learn world knowledge (associations between concepts like "dog", "fetch", "ball") from training text where the concepts appear together, and help for knowledge-intensive applications like question answering. 6 Challenges. A challenge with most common LM pretraining strategies is that they model a single document at a time. That is, one would split a text corpus into a list of documents and draw training instances for LMs from each document independently. Treating each document independently may pose limitations because documents often have rich dependencies with each other. For instance, text from the web 7 or scientific literature 8 is often used for LM training, but they all have document links, such as hyperlinks and citation links. Document links are crucial bec ## Stanford AI Lab Papers at ICCV 2021 DevFeed: [Stanford AI Lab Papers at ICCV 2021]() Original publisher: [Read original article]() Author: Compiled by Drew A. Hudson Published: 2021-10-08T07:00:00Z Content type: article Language: en Sources: [The Stanford AI Lab Blog]() Topics: [Artificial Intelligence](), [Self-supervised learning](), [multimodal](), [Representation learning](), [object-detection](), [Transformer]() Tags: [3d](), [ai](), [computer-vision](), [diffusion](), [fairness-accountability-transparency](), [learning](), [multimodal](), [object-detection](), [reasoning](), [video](), [vision]() ### AI overview Stanford AI Lab presents its accepted work for ICCV 2021, including research on multimodal medical image recognition, self-supervised learning, diffusion-based shape generation, object pose tracking, human-object relationships in videos, 3D human motion, computational photography, and related vision topics. ### Source excerpt The International Conference on Computer Vision (ICCV 2021) will be hosted virtually next week. We're excited to share all the work from SAIL that will be presented, and you'll find links to papers, videos and blogs below. Feel free to reach out to the contact authors directly to learn more about the work that's happening at Stanford! List of Accepted Papers GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition Authors: Mars Huang Contact: mschuang@stanford.edu Keywords: medical image, self-supervised learning, multimodal fusion 3D Shape Generation and Completion Through Point-Voxel Diffusion Authors: Linqi Zhou, Yilun Du, Jiajun Wu Contact: linqizhou@stanford.edu Links: Paper | Video | Website Keywords: diffusion, shape generation CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds Authors: Yijia Weng*, He Wang*, Qiang Zhou, Yuzhe Qin, Yueqi Duan, Qingnan Fan, Baoquan Chen, Hao Su, Leonidas J. Guibas Contact: yijiaw@stanford.edu Award nominations: Oral Presentation Links: Paper | Video | Website Keywords: category-level object pose tracking, articulated objects Detecting Human-Object Relationships in Videos Authors: Jingwei Ji, Rishi Desai, Juan Carlos Niebles Contact: jingweij@cs.stanford.edu Links: Paper Keywords: human-object relationships, video, detection, transformer, spatio-temporal reasoning Geography-Aware Self-Supervised Learning Authors: Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell, Stefano Ermon Contact: kayush@cs.stanford.edu, chenlin@stanford.edu Links: Paper | Website Keywords: self-supervised learning, contrastive learning, remote sensing, spatio-temporal, classification, object detection, segmentation HuMoR: 3D Human Motion Model for Robust Pose Estimation Authors: Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, Leonidas Guibas Contact: drempe@stanford.edu Award nominations: Oral Presentati