# Transcoding UTF-8 to UTF-16 with replacement at gigabytes per second

DevFeed: [Transcoding UTF-8 to UTF-16 with replacement at gigabytes per second](<https://devfeed.tech/articles/transcoding-utf-8-to-utf-16-with-replacement-at-gigabytes-per-second-62841.md>)

Original publisher: [Read original article](<https://lemire.me/blog/2026/09/30/transcoding-utf-8-to-utf-16-with-replacement-at-gigabytes-per-second/>)

Author: Daniel Lemire

Published: 2026-09-30T12:33:27Z

Content type: article

Language: en

Sources: [Daniel Lemire](<https://devfeed.tech/sources/daniel-lemire.md>)

Topics: [Encoding](<https://devfeed.tech/topics/encoding.md>), [OpenCV](<https://devfeed.tech/topics/opencv.md>)

Tags: [c-sharp](<https://devfeed.tech/tags/c-sharp.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [engine](<https://devfeed.tech/tags/engine.md>), [gcc](<https://devfeed.tech/tags/gcc.md>), [icelake](<https://devfeed.tech/tags/icelake.md>), [java](<https://devfeed.tech/tags/java.md>), [javascript](<https://devfeed.tech/tags/javascript.md>), [validation](<https://devfeed.tech/tags/validation.md>), [xeon](<https://devfeed.tech/tags/xeon.md>)

## AI overview

The simdutf library adds a function for transcoding invalid UTF-8 input to UTF-16, replacing ill-formed sequences with U+FFFD. The method first validates the input and records error offsets, then transcodes the valid spans. In a reported test on one Xeon Gold 6548N core, simdutf reached 7.0 GB/s on valid input, compared with 1.7 GB/s for a scalar decoder.

## Source excerpt

Our software represents strings using the UTF-16 or the UTF-8 formats. Most text on the web is UTF-8, but Java, C# or JavaScript represents the strings as UTF-16 to the programmer. Sometimes we need to transcode (convert) strings. Your browser probably uses the simdutf library for validating or transcoding. It is part of the widely ... Continue reading Transcoding UTF-8 to UTF-16 with replacement at gigabytes per second