# Machie Learning

Published articles for Machie Learning.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Why a Tiny GPT Can Know Two Facts and Still Not Connect Them

DevFeed: [Why a Tiny GPT Can Know Two Facts and Still Not Connect Them](<https://devfeed.tech/articles/why-a-tiny-gpt-can-know-two-facts-and-still-not-connect-them-118838.md>)

Original publisher: [Read original article](<https://towardsdatascience.com/why-a-tiny-gpt-can-know-two-facts-and-still-not-connect-them/>)

Author: Utkarsh Mangal

Published: 2026-11-05T12:00:00Z

Content type: article

Language: en

Sources: [Towards Data Science](<https://devfeed.tech/sources/towards-data-science.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [ai-research](<https://devfeed.tech/tags/ai-research.md>), [bridge](<https://devfeed.tech/tags/bridge.md>), [chain-of-thought](<https://devfeed.tech/tags/chain-of-thought.md>), [composition](<https://devfeed.tech/tags/composition.md>), [concrete](<https://devfeed.tech/tags/concrete.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [features](<https://devfeed.tech/tags/features.md>), [frontier-models](<https://devfeed.tech/tags/frontier-models.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [know](<https://devfeed.tech/tags/know.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [llm](<https://devfeed.tech/tags/llm.md>), [machie-learning](<https://devfeed.tech/tags/machie-learning.md>), [tiny](<https://devfeed.tech/tags/tiny.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

In a toy GPT trained on separate one-hop facts, the model answered 0 of 600 two-hop test questions correctly despite perfect one-hop training accuracy. Adding identity examples for bridge entities enabled some composition only for bridges with examples: early accuracy averaged 25%-38%, then declined to 7%-15% by step 2,500; bridges without examples never composed. The author presents this as evidence that the identity bridge helps in this setup but its benefit erodes with continued training. The result does not contradict Lin et al.'s near-perfect results: their experiments used much larger models, more epochs and a lower learning rate. The author says the cause of the decline is unknown and notes that evaluation began only at step 100.

### Source excerpt

A tiny GPT memorized every fact I gave it, then couldn't answer a single question that needed two of them. I tested the one-line fix from a recent paper, and it only half worked. The post Why a Tiny GPT Can Know Two Facts and Still Not Connect Them appeared first on Towards Data Science.