Abstract
Intel includes on its recent processors a powerful set of instructions
capable of processing 512-bit registers with a single instruction
(AVX-512). Some of these instructions have no equivalent in earlier
instruction sets. We leverage these instructions to efficiently
transcode strings between the most common formats: UTF-8 and UTF-16.
With our novel algorithms, we are often twice as fast as the previous
best solutions. For example, we transcode Chinese text from UTF-8 to
UTF-16 at more than 5 GiB s − 1 using fewer than 2 CPU
instructions per character. To ensure reproducibility, we make our
software freely available as an open source library.