# LinkBERT: Improving Language Model Training with Document Link

DevFeed: [LinkBERT: Improving Language Model Training with Document Link](<https://devfeed.tech/articles/linkbert-improving-language-model-training-with-document-link-7585.md>)

Original publisher: [Read original article](<https://ai.stanford.edu/blog/linkbert/>)

Author: A Href; Michihiro Yasunaga; Jure Leskovec; Percy Liang

Published: 2022-05-31T07:00:00Z

Content type: article

Language: en

Sources: [The Stanford AI Lab Blog](<https://devfeed.tech/sources/the-stanford-ai-lab-blog.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Natural language processing](<https://devfeed.tech/topics/nlp.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [Self-supervised learning](<https://devfeed.tech/topics/self-supervised-learning.md>), [Web](<https://devfeed.tech/topics/web.md>), [data](<https://devfeed.tech/topics/data.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [data](<https://devfeed.tech/tags/data.md>), [google](<https://devfeed.tech/tags/google.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [model](<https://devfeed.tech/tags/model.md>), [model-training](<https://devfeed.tech/tags/model-training.md>), [natural-language-processing](<https://devfeed.tech/tags/natural-language-processing.md>), [self-supervised-learning](<https://devfeed.tech/tags/self-supervised-learning.md>), [training](<https://devfeed.tech/tags/training.md>), [web](<https://devfeed.tech/tags/web.md>)

## AI overview

The article introduces LinkBERT in the context of language-model pretraining. It explains how BERT and GPT use self-supervised objectives to learn from large text corpora, then describes a limitation of treating documents independently: hyperlinks and citation links connect knowledge across documents and can support multi-hop understanding. The supplied text ends while introducing this challenge.

## Source excerpt

Language Model Pretraining Language models (LMs), like BERT 1 and the GPT series 2, achieve remarkable performance on many natural language processing (NLP) tasks. They are now the foundation of today's NLP systems. 3 These models serve important roles in products and tools that we use every day, such as search engines like Google 4 and personal assistants like Alexa 5. These LMs are powerful because they can be pretrained via self-supervised learning on massive amounts of text data on the web without the need for labels, after which the pretrained models can be quickly adapted to a wide range of new tasks without much task-specific finetuning. For instance, BERT is pretrained to predict randomly masked words in original text (masked language modeling), e.g. predicting the masked word "dog" from "My __ is fetching the ball". GPTs are pretrained to predict the next word given a previous sequence of text (causal language modeling), e.g. predicting the next word "ball" from "My dog is fetching the". In either cases, through pretraining, LMs learn to encode various knowledge from a text corpus that helps to perform downstream applications involving language understanding or generation. In particular, LMs can learn world knowledge (associations between concepts like "dog", "fetch", "ball") from training text where the concepts appear together, and help for knowledge-intensive applications like question answering. 6 Challenges. A challenge with most common LM pretraining strategies is that they model a single document at a time. That is, one would split a text corpus into a list of documents and draw training instances for LMs from each document independently. Treating each document independently may pose limitations because documents often have rich dependencies with each other. For instance, text from the web 7 or scientific literature 8 is often used for LM training, but they all have document links, such as hyperlinks and citation links. Document links are crucial bec