# information theory Published articles for information theory. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## The Codes of Solomon, Reed, and Muller DevFeed: [The Codes of Solomon, Reed, and Muller]() Original publisher: [Read original article]() Published: 2015-03-23T09:00:00Z Content type: tutorial Language: en Sources: [Jeremy Kun]() Topics: [Code](), [Decoding]() Tags: [code](), [coding-theory](), [decoding](), [hamming-code](), [information-theory](), [linear-codes](), [projection](), [reed-muller-code](), [reed-solomon-code]() ### AI overview This article introduces Reed-Solomon codes, explains the Singleton bound for codes with parameters (n,k,d)q, proves the bound using a projection map, and relates codes that meet the bound to maximum distance separable codes. It also places Reed-Muller codes in the broader discussion. ### Source excerpt Last time we defined the Hamming code. We also saw that it meets the Hamming bound, which is a measure of how densely a code can be packed inside an ambient space and still maintain a given distance. This time we'll define the Reed-Solomon code which optimizes a different bound called the Singleton bound, and then generalize them to a larger class of codes called Reed-Muller codes. In future posts we'll consider algorithmic issues behind decoding the codes, for now we just care about their existence and optimality properties. ## Hamming's Code DevFeed: [Hamming's Code]() Original publisher: [Read original article]() Published: 2015-03-02T09:00:00Z Content type: tutorial Language: en Sources: [Jeremy Kun]() Topics: [Code](), [Encoding](), [digital]() Tags: [coding-theory](), [compression](), [computing](), [encoding](), [error](), [errors](), [finite-fields](), [hamming](), [hamming-code](), [hypercube](), [information-theory](), [linear-algebra](), [linear-codes](), [np-hard](), [reed-solomon-codes](), [solved](), [transmission]() ### AI overview This tutorial introduces Hamming codes as efficiently computable encoding schemes for detecting and correcting errors caused by noise during digital transmission. It defines a code as a subset of binary strings and relates codewords to an injective encoding function. ### Source excerpt Or how to detect and correct errors Last time we made a quick tour through the main theorems of Claude Shannon, which essentially solved the following two problems about communicating over a digital channel. What is the best encoding for information when you are guaranteed that your communication channel is error free? Are there any encoding schemes that can recover from random noise introduced during transmission? The answers to these questions were purely mathematical theorems, of course. ## A Proofless Introduction to Information Theory DevFeed: [A Proofless Introduction to Information Theory]() Original publisher: [Read original article]() Published: 2015-02-16T09:00:00Z Content type: tutorial Language: en Sources: [Jeremy Kun]() Topics: [Encoding](), [digital]() Tags: [channel](), [coding-theory](), [communication](), [compression](), [digital](), [encoding](), [entropy](), [error](), [error-correction](), [example](), [explain](), [hamming](), [independent](), [information](), [information-theory](), [kolmogorov-complexity](), [language](), [markov](), [mathematics](), [messages](), [models](), [probabilistic-method](), [problems](), [shannon](), [theory]() ### AI overview A proofless introduction to information theory explains noiseless and noisy communication problems, encoding schemes, entropy, and how entropy differs from Kolmogorov complexity. ### Source excerpt There are two basic problems in information theory that are very easy to explain. Two people, Alice and Bob, want to communicate over a digital channel over some long period of time, and they know the probability that certain messages will be sent ahead of time. For example, English language sentences are more likely than gibberish, and "Hi" is much more likely than "asphyxiation." The problems are: Say communication is very expensive. ## The Complexity of Communication DevFeed: [The Complexity of Communication]() Original publisher: [Read original article]() Published: 2014-11-10T09:00:25Z Content type: tutorial Language: en Sources: [Jeremy Kun]() Topics: [Computer science](), [Algorithms, Complexity](), [Streaming](), [circuit]() Tags: [communication](), [communication-complexity](), [computational-complexity](), [fourier-analysis](), [information-theory](), [log-rank-conjecture](), [lower-bounds](), [matrices](), [streaming-algorithms](), [theory]() ### AI overview This tutorial introduces communication complexity: how much information two parties must exchange to jointly compute a function of separate inputs. It presents the basic two-player model and explains the subject's use in proving lower bounds, including applications to circuit design and streaming algorithms. ### Source excerpt satellite One of the most interesting questions posed in the last thirty years of computer science is to ask how much "information" must be communicated between two parties in order for them to jointly compute something. One can imagine these two parties living on distant planets, so that the cost of communicating any amount of information is very expensive, but each person has an integral component of the answer that the other does not. ## Information Distance -- A Primer DevFeed: [Information Distance -- A Primer]() Original publisher: [Read original article]() Published: 2012-12-04T08:30:08Z Content type: tutorial Language: en Sources: [Jeremy Kun]() Topics: [Compression](), [Machine learning]() Tags: [compression](), [computing](), [information-theory](), [kolmogorov-complexity](), [machine-learning](), [metric](), [turing-machines](), [universality]() ### AI overview This primer introduces information distance through Kolmogorov complexity. It explains how the shortest program relating two strings can reflect their shared information and behave similarly to a metric, with applications to compression and machine learning. ### Source excerpt This post assumes familiarity with our primer on Kolmogorov complexity. We recommend the uninformed reader begin there. We will do our best to keep consistent notation across both posts. Kolmogorov Complexity as a Metric Over the past fifty years mathematicians have been piling up more and more theorems about Kolmogorov complexity, and for good reason. One of the main interpretations of the Kolmogorov complexity function $ K$ is that for a given string $ x$, $ K(x)$ is the best theoretical compression of $ x$ under any compression scheme.