# Automatically Detecting Text Encodings in C++

DevFeed: [Automatically Detecting Text Encodings in C++](<https://devfeed.tech/articles/automatically-detecting-text-encodings-in-c-21017.md>)

Original publisher: [Read original article](<https://preshing.com/20200727/automatically-detecting-text-encodings-in-cpp>)

Author: Jeff Preshing

Published: 2020-07-27T20:10:00Z

Content type: article

Language: en

Sources: [Jeff Preshing](<https://devfeed.tech/sources/jeff-preshing.md>)

Topics: [C++](<https://devfeed.tech/topics/c-plus-plus.md>), [ASCII](<https://devfeed.tech/topics/ascii.md>), [cross-platform](<https://devfeed.tech/topics/cross-platform.md>), [Framework](<https://devfeed.tech/topics/framework.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Internet](<https://devfeed.tech/topics/internet.md>), [Windows](<https://devfeed.tech/topics/windows.md>), [Python](<https://devfeed.tech/topics/python.md>)

Tags: [ascii](<https://devfeed.tech/tags/ascii.md>), [c-plus-plus](<https://devfeed.tech/tags/c-plus-plus.md>), [cross-platform](<https://devfeed.tech/tags/cross-platform.md>), [files](<https://devfeed.tech/tags/files.md>), [framework](<https://devfeed.tech/tags/framework.md>), [internet](<https://devfeed.tech/tags/internet.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [platform](<https://devfeed.tech/tags/platform.md>), [python](<https://devfeed.tech/tags/python.md>), [windows](<https://devfeed.tech/tags/windows.md>)

## AI overview

This article explains why text-file encodings are ambiguous and describes how the Plywood C++ Framework detects and normalizes text formats. It covers ASCII, UTF-8, UTF-16, Windows-1252, BOMs, and platform-specific line endings.

## Source excerpt

Consider the lowly text file. This text file can take on a surprising number of different formats. The text could be encoded as ASCII, UTF-8, UTF-16 (little or big-endian), Windows-1252, Shift JIS, or any of dozens of other encodings. The file may or may not begin with a byte order mark (BOM). Lines of text could be terminated with a linefeed character \n (typical on UNIX), a CRLF sequence \r\n (typical on Windows) or, if the file was created on an older system, some other character sequence. Sometimes it's impossible to determine the encoding used by a particular text file. For example, suppose a file contains the following bytes: A2 C2 A2 C2 A2 C2 This could be: a UTF-8 file containing "ccc" a little-endian UTF-16 (or UCS-2) file containing "ꋂꋂꋂ" a big-endian UTF-16 file containing "슢슢슢" a Windows-1252 file containing "ÂcÂcÂc" That's obviously an artificial example, but the point is that text files are inherently ambiguous. This poses a challenge to software that loads text. It's a problem that has been around for a while. Fortunately, the text file landscape has gotten simpler over time, with UTF-8 winning out over other character encodings. More than 95% of the Internet is now delivered using UTF-8. It's impressive how quickly that number has changed; it was less than 10% as recently as 2006. UTF-8 hasn't taken over the world just yet, though. The Windows Registry editor, for example, still saves text files as UTF-16. When writing a text file from Python, the default encoding is platform-dependent; on my Windows PC, it's Windows-1252. In other words, the ambiguity problem still exists today. And even if a text file is encoded in UTF-8, there are still variations in format, since the file may or may not start with a BOM and could use either UNIX-style or Windows-style line endings. How the Plywood C++ Framework Loads Text Plywood is a cross-platform open-source C++ framework I released two months ago. When opening a text file using Plywood, you have a couple of o