uconv(1) - phpMan

UCONV(1) ICU 70.1 Manual UCONV(1)

NAME
uconv - convert data from one encoding to another

SYNOPSIS
uconv [ -h, -?, --help ] [ -V, --version ] [ -s, --silent ] [ -v, --verbose ] [ -l, --list
| -l, --list-code code | --default-code | -L, --list-transliterators ] [ --canon ] [ -x
transliteration ] [ --to-callback callback | -c ] [ --from-callback callback | -i ] [
--callback callback ] [ --fallback | --no-fallback ] [ -b, --block-size size ] [ -f,
--from-code encoding ] [ -t, --to-code encoding ] [ --add-signature ] [ --remove-signature
] [ -o, --output file ] [ file... ]

DESCRIPTION
uconv converts, or transcodes, each given file (or its standard input if no file is speci-
fied) from one encoding to another. The transcoding is done using Unicode as a pivot en-
coding (i.e. the data are first transcoded from their original encoding to Unicode, and
then from Unicode to the destination encoding).

If an encoding is not specified or is -, the default encoding is used. Thus, calling uconv
with no encoding provides an easy way to validate and sanitize data files for further con-
sumption by tools requiring data in the default encoding.

When calling uconv, it is possible to specify callbacks that are used to handle invalid
characters in the input, or characters that cannot be transcoded to the destination encod-
ing. Some encodings, for example, offer a default substitution character that can be used
to represent the occurrence of such characters in the input. Other callbacks offer a use-
ful visual representation of the invalid data.

uconv can also run the specified transliteration on the transcoded data, in which case
transliteration will happen as an intermediate step, after the data have been transcoded
to Unicode. The transliteration can be either a list of semicolon-separated translitera-
tor names, or an arbitrarily complex set of rules in the ICU transliteration rules format.

For transcoding purposes, uconv options are compatible with those of iconv(1), making it
easy to replace it in scripts. It is not necessarily the case, however, that the encoding
names used by uconv and ICU are the same as the ones used by iconv(1). Also, options that
provide informational data, such as the -l, --list one offered by some iconv(1) variants
such as GNU's, produce data in a slightly different and easier to parse format.

OPTIONS
-h, -?, --help
Print help about usage and exit.

-V, --version
Print the version of uconv and exit.

-s, --silent
Suppress messages during execution.

-v, --verbose
Display extra informative messages during execution.

-l, --list
List all the available encodings and exit.

-l, --list-code code
List only the code encoding and exit. If code is not a proper encoding, exit with
an error.

--default-code
List only the name of the default encoding and exit.

-L, --list-transliterators
List all the available transliterators and exit.

--canon
If used with -l, --list or --default-code, the list of encodings is produced in a
format compatible with convrtrs.txt(5). If used with -L, --list-transliterators,
print only one transliterator name per line.

-x transliteration
Run the given transliteration on the transcoded Unicode data, and use the translit-
erated data as input for the transcoding to the destination encoding.

--to-callback callback
Use callback to handle characters that cannot be transcoded to the destination en-
coding. See section CALLBACKS for details on valid callbacks.

-c Omit invalid characters from the output. Same as --to-callback skip.

--from-callback callback
Use callback to handle characters that cannot be transcoded from the original en-
coding. See section CALLBACKS for details on valid callbacks.

-i Ignore invalid sequences in the input. Same as --from-callback skip.

--callback callback
Use callback to handle both characters that cannot be transcoded from the original
encoding and characters that cannot be transcoded to the destination encoding. See
section CALLBACKS for details on valid callbacks.

--fallback
Use the fallback mapping when transcoding from Unicode to the destination encoding.

--no-fallback
Do not use the fallback mapping when transcoding from Unicode to the destination
encoding. This is the default.

-b, --block-size size
Read input in blocks of size bytes at a time. The default block size is 4096.

-f, --from-code encoding
Set the original encoding of the data to encoding.

-t, --to-code encoding
Transcode the data to encoding.

--add-signature
Add a U+FEFF Unicode signature character (BOM) if the output charset supports it
and does not add one anyway.

--remove-signature
Remove a U+FEFF Unicode signature character (BOM).

-o, --output file
Write the transcoded data to file.

CALLBACKS
uconv supports specifying callbacks to handle invalid data. Callbacks can be set for both
directions of transcoding: from the original encoding to Unicode, with the --from-callback
option, and from Unicode to the destination encoding, with the --to-callback option.

The following is a list of valid callback names, along with a description of their behav-
ior. The list of callbacks actually supported by uconv is displayed when it is called with
-h, --help.

substitute Write the encoding's substitute sequence, or the Unicode replacement
character U+FFFD when transcoding to Unicode.

skip Ignore the invalid data.

stop Stop with an error when encountering invalid data. This is the default
callback.

escape Same as escape-icu.

escape-icu Replace the missing characters with a string of the format %Uhhhh for
plane 0 characters, and %Uhhhh%Uhhhh for planes 1 and above characters,
where hhhh is the hexadecimal value of one of the UTF-16 code units rep-
resenting the character. Characters from planes 1 and above are written
as a pair of UTF-16 surrogate code units.

escape-java Replace the missing characters with a string of the format \uhhhh for
plane 0 characters, and \uhhhh\uhhhh for planes 1 and above characters,
where hhhh is the hexadecimal value of one of the UTF-16 code units rep-
resenting the character. Characters from planes 1 and above are written
as a pair of UTF-16 surrogate code units.

escape-c Replace the missing characters with a string of the format \uhhhh for
plane 0 characters, and \Uhhhhhhhh for planes 1 and above characters,
where hhhh and hhhhhhhh are the hexadecimal values of the Unicode code-
point.

escape-xml Same as escape-xml-hex.

escape-xml-hex Replace the missing characters with a string of the format &#xhhhh;,
where hhhh is the hexadecimal value of the Unicode codepoint.

escape-xml-dec Replace the missing characters with a string of the format &#nnnn;, where
nnnn is the decimal value of the Unicode codepoint.

escape-unicode Replace the missing characters with a string of the format {U+hhhh},
where hhhh is the hexadecimal value of the Unicode codepoint. That hexa-
decimal string is of variable length and can use from 4 to 6 digits.
This is the format universally used to denote a Unicode codepoint in the
literature, delimited by curly braces for easy recognition of those sub-
stitutions in the output.

EXAMPLES
Convert data from a given encoding to the platform encoding:

$ uconv -f encoding

Check if a file contains valid data for a given encoding:

$ uconv -f encoding -c file >/dev/null

Convert a UTF-8 file to a given encoding and ensure that the resulting text is good for
any version of HTML:

$ uconv -f utf-8 -t encoding \
--callback escape-xml-dec file

Display the names of the Unicode code points in a UTF-file:

$ uconv -f utf-8 -x any-name file

Print the name of a Unicode code point whose value is known (U+30AB in this example):

$ echo '\u30ab' | uconv -x 'hex-any; any-name'; echo
{KATAKANA LETTER KA}{LINE FEED}
$

(The names are delimited by curly braces. Also, the name of the line terminator is also
displayed.)

Normalize UTF-8 data using Unicode NFKC, remove all control characters, and map Katakana
to Hiragana:

$ uconv -f utf-8 -t utf-8 \
-x '::nfkc; [:Cc:] >; ::katakana-hiragana;'

CAVEATS AND BUGS
uconv does report errors as occurring at the first invalid byte encountered. This may be
confusing to users of GNU iconv(1), which reports errors as occurring at the first byte of
an invalid sequence. For multi-byte character sets or encodings, this means that uconv er-
ror positions may be at a later offset in the input stream than would be the case with GNU
iconv(1).

The reporting of error positions when a transliterator is used may be inaccurate or un-
available, in which case uconv will report the offset in the output stream at which the
error occurred.

AUTHORS
Jonas Utterstroem
Yves Arrouye

VERSION
70.1