# A Proofless Introduction to Information Theory

DevFeed: [A Proofless Introduction to Information Theory](<https://devfeed.tech/articles/a-proofless-introduction-to-information-theory-40377.md>)

Original publisher: [Read original article](<https://www.jeremykun.com/2015/02/16/a-proofless-introduction-to-information-theory/>)

Published: 2015-02-16T09:00:00Z

Content type: tutorial

Language: en

Sources: [Jeremy Kun](<https://devfeed.tech/sources/jeremy-kun.md>)

Topics: [Encoding](<https://devfeed.tech/topics/encoding.md>), [digital](<https://devfeed.tech/topics/digital.md>)

Tags: [channel](<https://devfeed.tech/tags/channel.md>), [coding-theory](<https://devfeed.tech/tags/coding-theory.md>), [communication](<https://devfeed.tech/tags/communication.md>), [compression](<https://devfeed.tech/tags/compression.md>), [digital](<https://devfeed.tech/tags/digital.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [error](<https://devfeed.tech/tags/error.md>), [error-correction](<https://devfeed.tech/tags/error-correction.md>), [example](<https://devfeed.tech/tags/example.md>), [explain](<https://devfeed.tech/tags/explain.md>), [hamming](<https://devfeed.tech/tags/hamming.md>), [independent](<https://devfeed.tech/tags/independent.md>), [information](<https://devfeed.tech/tags/information.md>), [information-theory](<https://devfeed.tech/tags/information-theory.md>), [kolmogorov-complexity](<https://devfeed.tech/tags/kolmogorov-complexity.md>), [language](<https://devfeed.tech/tags/language.md>), [markov](<https://devfeed.tech/tags/markov.md>), [mathematics](<https://devfeed.tech/tags/mathematics.md>), [messages](<https://devfeed.tech/tags/messages.md>), [models](<https://devfeed.tech/tags/models.md>), [probabilistic-method](<https://devfeed.tech/tags/probabilistic-method.md>), [problems](<https://devfeed.tech/tags/problems.md>), [shannon](<https://devfeed.tech/tags/shannon.md>), [theory](<https://devfeed.tech/tags/theory.md>)

## AI overview

A proofless introduction to information theory explains noiseless and noisy communication problems, encoding schemes, entropy, and how entropy differs from Kolmogorov complexity.

## Source excerpt

There are two basic problems in information theory that are very easy to explain. Two people, Alice and Bob, want to communicate over a digital channel over some long period of time, and they know the probability that certain messages will be sent ahead of time. For example, English language sentences are more likely than gibberish, and "Hi" is much more likely than "asphyxiation." The problems are: Say communication is very expensive.