Convert File Encoding

Character Encodings Explained

What is a character encoding?

Computers store text as numbers. A character encoding (or charset) is the rule that says which number represents which character, and how those numbers are written as bytes. The same sequence of bytes can therefore mean completely different things depending on the encoding used to read it: the byte 0xE9 is é in ISO-8859-1, щ in windows-1251 and an invalid sequence in UTF-8. When a file is read with the wrong encoding you get the familiar "mojibake": é instead of é, ’ instead of a curly apostrophe, or rows of question marks and �.

Converting a file means decoding its bytes with the encoding it was written in, then encoding the resulting characters with the encoding you want. Getting the source encoding right is the whole difficulty; the preview above lets you check that accented letters and symbols look right before you convert.

Unicode: UTF-8, UTF-16 and UTF-32

Unicode assigns a number (a code point) to every character of every writing system: more than 150,000 characters today. The three UTF encodings are different ways of writing those code points as bytes:

Legacy code pages

Before Unicode, each language or region had its own 8-bit encoding: 128 ASCII characters plus 128 local characters. ISO-8859-1 (Latin-1) covers Western Europe, ISO-8859-2 Central Europe, ISO-8859-5 and KOI8-R Cyrillic, ISO-8859-7 Greek, and Microsoft shipped its own variants (windows-1252, windows-1251…) with a few extra characters such as the euro sign and curly quotes in the 0x80–0x9F range. East Asian languages need thousands of characters and use multi-byte encodings (Shift_JIS, EUC-JP, GBK, GB18030, Big5, EUC-KR) where a character is 1 or 2 bytes (up to 4 for GB18030). These encodings can only represent their own repertoire: converting a Chinese text to ISO-8859-1 will replace every ideogram by ?. Convert legacy files to UTF-8 whenever you can; convert from UTF-8 to a legacy encoding only when an old system demands it.

The byte order mark (BOM)

The BOM is the Unicode character U+FEFF written at the very start of a file. In UTF-16 and UTF-32 it tells the reader the byte order: FF FE means little-endian, FE FF big-endian. In UTF-8 the BOM (EF BB BF) carries no ordering information and is merely a signature; it is added by Windows Notepad and expected by some Microsoft tools (Excel reads CSV files as UTF-8 only if the BOM is present), but it breaks many Unix tools, shell scripts (#! must be the first bytes), PHP files and JSON parsers. Rule of thumb: add a BOM for UTF-16/32, add it for UTF-8 only when a Windows application needs it, and never for source code or configuration files.

Why is auto-detection only a guess?

Files do not carry a label saying which encoding they use. Detection is therefore heuristic: a BOM is a certain signal; a byte sequence that is valid UTF-8 is almost certainly UTF-8 (random Latin-1 text is very unlikely to form valid multi-byte sequences); files with a NUL byte every other position are UTF-16. But a file containing only bytes above 0x7F cannot be told apart between ISO-8859-1, windows-1252, ISO-8859-2, windows-1251… except by statistics on the language. This tool tries UTF-8 first, then falls back to windows-1252 (or windows-1251 when the high bytes look Cyrillic). When the guess is wrong, the "Detect file encoding" table shows the same bytes decoded with several candidate encodings so you can pick the one that reads correctly.

Common encodings

Privacy: decoding, detection and encoding to Unicode or single-byte charsets happen entirely in your browser. Only when the target is a multi-byte legacy encoding that browsers cannot produce (Shift_JIS, EUC-JP, ISO-2022-JP, GBK, GB18030, Big5, EUC-KR) is the text sent to the small api/convert-encoding.php helper, which converts it in memory with PHP's mbstring/iconv and returns the bytes without storing anything.