@node Encoding conversions @chapter Encoding conversions (@file{iconv.h}) This chapter describes the Newlib iconv library. The iconv functions declarations are in @file{iconv.h}. @menu * Function iconv:: Encoding conversion routines * Introduction to iconv:: Introduction to iconv and encodings * Supported encodings:: The list of currently supported encodings * iconv design decisions:: General iconv library design issues * iconv configuration:: iconv-related configure script options * Encoding names:: How encodings are named. * CCS tables:: CCS tables format and 'mktbl.pl' Perl script * CES converters:: CES converters description * The encodings description file:: The 'encoding.deps' file and 'mkdeps.pl' * How to add new encoding:: The steps to add new encoding support * The locale support interfaces:: Locale-related iconv interfaces * Contact:: The author contact @end menu @page @include iconv/lib/iconv.def @page @node Introduction to iconv @section Introduction to iconv @findex encoding @findex character set @findex charset @findex CES @findex CCS @* The iconv library is intended to convert characters from one encoding to another. It implements iconv(), iconv_open() and iconv_close() calls, which are defined by the Single Unix Specification. @* In addition to these user-level interfaces, the iconv library also has several useful interfaces which are needed to support coding capabilities of the Newlib Locale infrastructure. Since Locale support also needs to convert various character sets to and from the @emph{wide characters set}, the iconv library shares it's capabilities with the Newlib Locale subsystem. Moreover, the iconv library supports several features which are only needed for the Locale infrastructure (for example, the MB_CUR_MAX value). @* The Newlib iconv library was created using concepts from another iconv library implemented by Konstantin Chuguev (ver 2.0). The Newlib iconv library was rewritten from scratch and contains a lot of improvements with respect to the original iconv library. @* Terms like @dfn{encoding} or @dfn{character set} aren't well defined and are often used with various meanings. The following are the definitions of terms which are used in this documentation as well as in the iconv library implementation: @itemize @bullet @item @dfn{encoding} - a machine representation of characters by means of bits; @item @dfn{Character Set} or @dfn{Charset} - just a collection of characters, i.e. the encoding is the machine representation of the character set; @item @dfn{CCS} (@dfn{Coded Character Set}) - a mapping from an character set to a set of integers @dfn{character codes}; @item @dfn{CES} (@dfn{Character Encoding Scheme}) - a mapping from a set of character codes to a sequence of bytes; @end itemize @* Users usually deal with encodings, for example, KOI8-R, Unicode, UTF-8, ASCII, etc. Encodings are formed by the following chain of steps: @enumerate @item User has a set of characters which are specific to his or her language (character set). @item Each character from this set is uniquely numbered, resulting in an CCS. @item Each number from the CCS is converted to a sequence of bits or bytes by means of a CES and form some encoding. Thus, CES may be considered as a function of CCS which produces some encoding. Note, that CES may be applied to more than one CCS. @end enumerate @* Thus, an encoding may be considered as one or more CCS + CES. @* Sometimes, there is no CES and in such cases encoding is equivalent to CCS, e.g. KOI8-R or ASCII. @* An example of a more complicated encoding is UTF-8 which is the UCS (or Unicode) CCS plus the UTF-8 CES. @* The following is a brief list of iconv library features: @itemize @item Generic architecture; @item Locale infrastructure support; @item Automatic generation of the program code which handles CES/CCS/Encoding/Names/Aliases dependencies; @item The ability to choose size- or speed-optimazed configuration; @item The ability to exclude a lot of unneeded code and data from the linking step. @end itemize @page @node Supported encodings @section Supported encodings @findex big5 @findex cp775 @findex cp850 @findex cp852 @findex cp855 @findex cp866 @findex euc_jp @findex euc_kr @findex euc_tw @findex iso_8859_1 @findex iso_8859_10 @findex iso_8859_11 @findex iso_8859_13 @findex iso_8859_14 @findex iso_8859_15 @findex iso_8859_2 @findex iso_8859_3 @findex iso_8859_4 @findex iso_8859_5 @findex iso_8859_6 @findex iso_8859_7 @findex iso_8859_8 @findex iso_8859_9 @findex iso_ir_111 @findex koi8_r @findex koi8_ru @findex koi8_u @findex koi8_uni @findex ucs_2 @findex ucs_2_internal @findex ucs_2be @findex ucs_2le @findex ucs_4 @findex ucs_4_internal @findex ucs_4be @findex ucs_4le @findex us_ascii @findex utf_16 @findex utf_16be @findex utf_16le @findex utf_8 @findex win_1250 @findex win_1251 @findex win_1252 @findex win_1253 @findex win_1254 @findex win_1255 @findex win_1256 @findex win_1257 @findex win_1258 @* The following is the list of currently supported encodings. The first column corresponds to the encoding name, the second column is the list of aliases, the third column is its CES and CCS components names, and the fourth column is a short description. @multitable @columnfractions .20 .26 .24 .30 @item Name @tab Aliases @tab CES/CCS @tab Short description @item @tab @tab @tab @item big5 @tab csbig5, big_five, bigfive, cn_big5, cp950 @tab table_pcs / big5, us_ascii @tab The encoding for the Traditional Chinese. @item cp775 @tab ibm775, cspc775baltic @tab table / cp775 @tab The updated version of CP 437 that supports the balitic languages. @item cp850 @tab ibm850, 850, cspc850multilingual @tab table / cp850 @tab IBM 850 - the updated version of CP 437 where several Latin 1 characters have been added instead of some less-often used characters like the line-drawing and the greek ones. @item cp852 @tab ibm852, 852, cspcp852 @tab @tab IBM 852 - the updated version of CP 437 where several Latin 2 characters have been added instead of some less-often used characters like the line-drawing and the greek ones. @item cp855 @tab ibm855, 855, csibm855 @tab table / cp855 @tab IBM 855 - the updated version of CP 437 that supports Cyrillic. @item cp866 @tab 866, IBM866, CSIBM866 @tab table / cp866 @tab IBM 866 - the updated version of CP 855 which follows more the logical Russian alphabet ordering of the alternative variant that is preferred by many Russian users. @item euc_jp @tab eucjp @tab euc / jis_x0208_1990, jis_x0201_1976, jis_x0212_1990 @tab EUC-JP - The EUC for Japanese. @item euc_kr @tab euckr @tab euc / ksx1001 @tab EUC-KR - The EUC for Korean. @item euc_tw @tab euctw @tab euc / cns11643_plane1, cns11643_plane2, cns11643_plane14 @tab EUC-TW - The EUC for Traditional Chinese. @item iso_8859_1 @tab iso8859_1, iso88591, iso_8859_1:1987, iso_ir_100, latin1, l1, ibm819, cp819, csisolatin1 @tab table / iso_8859_1 @tab ISO 8859-1:1987 - Latin 1, West European. @item iso_8859_10 @tab iso_8859_10:1992, iso_ir_157, iso885910, latin6, l6, csisolatin6, iso8859_10 @tab table / iso_8859_10 @tab ISO 8859-10:1992 - Latin 6, Nordic. @item iso_8859_11 @tab iso8859_11, iso885911 @tab table / iso_8859_11 @tab ISO 8859-11 - Thai. @item iso_8859_13 @tab iso_8859_13:1998, iso8859_13, iso885913 @tab table / iso_8859_13 @tab ISO 8859-13:1998 - Latin 7, Baltic Rim. @item iso_8859_14 @tab iso_8859_14:1998, iso885914, iso8859_14 @tab table / iso_8859_14 @tab ISO 8859-14:1998 - Latin 8, Celtic. @item iso_8859_15 @tab iso885915, iso_8859_15:1998, iso8859_15, @tab table / iso_8859_15 @tab ISO 8859-15:1998 - Latin 9, West Europe, successor of Latin 1. @item iso_8859_2 @tab iso8859_2, iso88592, iso_8859_2:1987, iso_ir_101, latin2, l2, csisolatin2 @tab table / iso_8859_2 @tab ISO 8859-2:1987 - Latin 2, East European. @item iso_8859_3 @tab iso_8859_3:1988, iso_ir_109, iso8859_3, latin3, l3, csisolatin3, iso88593 @tab table / iso_8859_3 @tab ISO 8859-3:1988 - Latin 3, South European. @item iso_8859_4 @tab iso8859_4, iso88594, iso_8859_4:1988, iso_ir_110, latin4, l4, csisolatin4 @tab table / iso_8859_4 @tab ISO 8859-4:1988 - Latin 4, North European. @item iso_8859_5 @tab iso8859_5, iso88595, iso_8859_5:1988, iso_ir_144, cyrillic, csisolatincyrillic @tab table / iso_8859_5 @tab ISO 8859-5:1988 - Cyrillic. @item iso_8859_6 @tab iso_8859_6:1987, iso_ir_127, iso8859_6, ecma_114, asmo_708, arabic, csisolatinarabic, iso88596 @tab table / iso_8859_6 @tab ISO i8859-6:1987 - Arabic. @item iso_8859_7 @tab iso_8859_7:1987, iso_ir_126, iso8859_7, elot_928, ecma_118, greek, greek8, csisolatingreek, iso88597 @tab table / iso_8859_7 @tab ISO 8859-7:1987 - Greek. @item iso_8859_8 @tab iso_8859_8:1988, iso_ir_138, iso8859_8, hebrew, csisolatinhebrew, iso88598 @tab table / iso_8859_8 @tab ISO 8859-8:1988 - Hebrew. @item iso_8859_9 @tab iso_8859_9:1989, iso_ir_148, iso8859_9, latin5, l5, csisolatin5, iso88599 @tab table / iso_8859_9 @tab ISO 8859-9:1989 - Latin 5, Turkish. @item iso_ir_111 @tab ecma_cyrillic, koi8_e, koi8e, csiso111ecmacyrillic @tab table / iso_ir_111 @tab ISO IR 111/ECMA Cyrillic. @item koi8_r @tab cskoi8r, koi8r, koi8 @tab table / koi8_r @tab RFC 1489 Cyrillic. @item koi8_ru @tab koi8ru @tab table / koi8_ru @tab The obsolete Ukrainian. @item koi8_u @tab koi8u @tab table / koi8_u @tab RFC 2319 Ukrainian. @item koi8_uni @tab koi8uni @tab table / koi8_uni @tab KOI8 Unified. @item ucs_2 @tab ucs2, iso_10646_ucs_2, iso10646_ucs_2, iso_10646_ucs2, iso10646_ucs2, iso10646ucs2, csUnicode @tab ucs_2 / (UCS) @tab ISO-10646-UCS-2. Big Endian, NBSP is always interpreted as NBSP (BOM isn't supported). @item ucs_2_internal @tab ucs2_internal, ucs_2internal, ucs2internal @tab ucs_2_internal / (UCS) @tab ISO-10646-UCS-2 in system byte order. NBSP is always interpreted as NBSP (BOM isn't supported). @item ucs_2be @tab ucs2be @tab ucs_2 / (UCS) @tab Big Endian version of ISO-10646-UCS-2 (in fact, equivalent to ucs_2). Big Endian, NBSP is always interpreted as NBSP (BOM isn't supported). @item ucs_2le @tab ucs2le @tab ucs_2 / (UCS) @tab Little Endian version of ISO-10646-UCS-2. Little Endian, NBSP is always interpreted as NBSP (BOM isn't supported). @item ucs_4 @tab ucs4, iso_10646_ucs_4, iso10646_ucs_4, iso_10646_ucs4, iso10646_ucs4, iso10646ucs4 @tab ucs_4 / (UCS) @tab ISO-10646-UCS-4. Big Endian, NBSP is always interpreted as NBSP (BOM isn't supported). @item ucs_4_internal @tab ucs4_internal, ucs_4internal, ucs4internal @tab ucs_4_internal / (UCS) @tab ISO-10646-UCS-4 in system byte order. NBSP is always interpreted as NBSP (BOM isn't supported). @item ucs_4be @tab ucs4be @tab ucs_4 / (UCS) @tab Big Endian version of ISO-10646-UCS-4 (in fact, equivalent to ucs_4). Big Endian, NBSP is always interpreted as NBSP (BOM isn't supported). @item ucs_4le @tab ucs4le @tab ucs_4 / (UCS) @tab Little Endian version of ISO-10646-UCS-4. Little Endian, NBSP is always interpreted as NBSP (BOM isn't supported). @item us_ascii @tab ansi_x3.4_1968, ansi_x3.4_1986, iso_646.irv:1991, ascii, iso646_us, us, ibm367, cp367, csascii @tab us_ascii / (ASCII) @tab 7-bit ASCII. @item utf_16 @tab utf16 @tab utf_16 / (UCS) @tab RFC 2781 UTF-16. The very first NBSP code in stream is interpreted as BOM. @item utf_16be @tab utf16be @tab utf_16 / (UCS) @tab Big Endian version of RFC 2781 UTF-16. NBSP is always interpreted as NBSP (BOM isn't supported). @item utf_16le @tab utf16le @tab utf_16 / (UCS) @tab Little Endian version of RFC 2781 UTF-16. NBSP is always interpreted as NBSP (BOM isn't supported). @item utf_8 @tab utf8 @tab utf_8 / (UCS) @tab RFC 3629 UTF-8. @item win_1250 @tab cp1250 @tab @tab Win-1250 Croatian. @item win_1251 @tab cp1251 @tab table / win_1251 @tab Win-1251 - Cyrillic. @item win_1252 @tab cp1252 @tab table / win_1252 @tab Win-1252 - Latin 1. @item win_1253 @tab cp1253 @tab table / win_1253 @tab Win-1253 - Greek. @item win_1254 @tab cp1254 @tab table / win_1254 @tab Win-1254 - Turkish. @item win_1255 @tab cp1255 @tab table / win_1255 @tab Win-1255 - Hebrew. @item win_1256 @tab cp1256 @tab table / win_1256 @tab Win-1256 - Arabic. @item win_1257 @tab cp1257 @tab table / win_1257 @tab Win-1257 - Baltic. @item win_1258 @tab cp1258 @tab table / win_1258 @tab Win-1258 - Vietnamese7 that supports Cyrillic. @end multitable @page @node iconv design decisions @section iconv design decisions @findex CCS table @findex CES converter @findex Speed-optimized tables @findex Size-optimized tables @* The first iconv library design issue arises when considering the following two design approaches: @enumerate @item Have modules which implement conversion from the encoding A to the encoding B and vice versa i.e., one conversion module relates to any two encodings. @item Have modules which implement conversion from the encoding A to the fixed encoding C and vice versa i.e., one conversion module relates to any one encoding A and one fixed encoding C. In this case, to convert from the encoding A to the encoding B, two modules are needed (in order to convert from A to C and then from C to B). @end enumerate @* It's obvious, that we have tradeoff between commonality/flexibility and efficiency: the first method is more efficient since it converts directly; however, it isn't so flexible since for each encoding pair a distinct module is needed. @* The Newlib iconv model uses the second method and always converts through the 32-bit UCS but its design also allows one to write specialized conversion modules if the conversion speed is critical. @* The second design issue is how to break down (decompose) encodings. The Newlib iconv library uses the fact that any encoding may be considered as one or more CCS plus a CES. It also decomposes its conversion modules on @dfn{CES converter} plus one or more @dfn{CCS tables}. CCS tables map CCS to UCS and vice versa; the CES converters map CCS to the encoding and vice versa. @* As the example, let's consider the conversion from the big5 encoding to the EUC-TW encoding. The big5 encoding may be decomposed to the ASCII and BIG5 CCS-es plus the BIG5 CES. EUC-TW may be decomposed on the CNS11643_PLANE1, CNS11643_PLANE2, and CNS11643_PLANE14 CCS-es plus the EUC CES. @* The euc_jp -> big5 conversion is performed as follows: @enumerate @item The EUC converter performs the EUC-TW encoding to the corresponding CCS-es transformation (CNS11643_PLANE1, CNS11643_PLANE2 and CNS11643_PLANE14 CCS-es); @item The obtained CCS codes are transformed to the UCS codes using the CNS11643_PLANE1, CNS11643_PLANE2 and CNS11643_PLANE14 CCS tables; @item The resulting UCS codes are transformed to the ASCII and BIG5 codes using the corresponding CCS tables; @item The obtained CCS codes are transformed to the big5 encoding using the corresponding CES converter. @end enumerate @* Analogously, the backward conversion is performed as follows: @enumerate @item The BIG5 converter performs the big5 encoding to the corresponding CCS-es transformation (the ASCII and BIG5 CCS-es); @item The obtained CCS codes are transformed to the UCS codes using the ASCII and BIG5 CCS tables; @item The resulting UCS codes are transformed to the ASCII and BIG5 codes using the corresponding CCS tables; @item The obtained CCS codes are transformed to the EUC-TW encoding using the corresponding CES converter. @end enumerate @* Note, the above is just an example and real names (which are implemented in the Newlib iconv) of the CES converters and the CCS tables are slightly different. @* The third design issue also relates to flexibility. Obviously, it isn't desirable to always link all the CES converters and the CCS tables to the library but instead, we want to be able to load the needed converters and tables dynamically on demand. This isn't a problem on "big" machines such as a PC, but it may be very problematical within "small" embedded systems. @* Since the CCS tables are just data, it is possible to load them dynamically from external files. The CES converters, on the other hand are algorithms with some code so a dynamic library loading capability is required. @* Apart from possible restrictions applied by embedded systems (small RAM for example), Newlib itself has no dynamic library support and therefore, all the CES converters which will ever be used must be linked into the library. However, loading of the dynamic CCS tables is possible and is implemented in the Newlib iconv library. It may be enabled via the Newlib configure script options. @* The next design issue is fine-tuning the iconv library configuration. One important ability is for iconv to not link all it's converters and tables (if dynamic loading is not enabled) but instead, enable only those encodings which are specified at configuration time (see the section about the configure script options). @* In addition, the Newlib iconv library configure options distinguish between conversion directions. This means that not only are supported encodings selectable, the conversion direction is as well. For example, if user wants the configuration which allows conversions from UTF-8 to UTF-16 and doesn't plan using the "UTF-16 to UTF-8" conversions, he or she can enable only this conversion direction (i.e., no "UTF-16 -> UTF-8"-related code will be included) thus, saving some memory (note, that such technique allows to exclude one half of a CCS table from linking which may be big enough). @* One more design aspect are the speed- and size- optimized tables. Users can select between them using configure script options. The speed-optimized CCS tables are the same as the size-optimized ones in case of 8-bit CCS (e.g.m KOI8-R), but for 16-bit CCS-es the size-optimized CCS tables may be 1.5 to 2 times less then the speed-optimized ones. On the other hand, conversion with speed tables is several times faster. @* Its worth to stress that the new encoding support can't be dynamically added into an already compiled Newlib library, even if it needs only an additional CCS table and iconv is configured to use the external files with CCS tables (this isn't the fundamental restriction and the possibility to add new Table-based encoding support dynamically, by means of just adding new .cct file, may be easily added). @* Theoretically, the compiled-in CCS tables should be more appropriate for embedded systems than dynamically loaded CCS tables. This is because the compiled-in tables are read-only and can be placed in ROM whereas dynamic loading requires RAM. Moreover, in the current iconv implementation, a distinct copy of the dynamic CCS file is loaded for each opened iconv descriptor even in case of the same encoding. This means, for example, that if two iconv descriptors for "KOI8-R -> UCS-4BE" and "KOI8-R -> UTF-16BE" are opened, two copies of koi8-r .cct file will be loaded (actually, iconv loads only the needed part of these files). On the other hand, in the case of compiled-in CCS tables, there will always be only one copy. @page @node iconv configuration @section iconv configuration @findex iconv configuration @findex --enable-newlib-iconv-encodings @findex --enable-newlib-iconv-from-encodings @findex --enable-newlib-iconv-to-encodings @findex --enable-newlib-iconv-external-ccs @findex NLSPATH @* To enable an encoding, the @emph{--enable-newlib-iconv-encodings} configure script option should be used. This option accepts a comma-separated list of @emph{encodings} that should be enabled. The option enables each encoding in both ("to" and "from") directions. @* The @option{--enable-newlib-iconv-from-encodings} configure script option enables "from" support for each encoding that was passed to it. @* The @option{--enable-newlib-iconv-to-encodings} configure script option enables "to" support for each encoding that was passed to it. @* Example: if user plans only the "KOI8-R -> UTF-8", "UTF-8 -> ISO-8859-5" and "KOI8-R -> UCS-2" conversions, the most optimal way (minimal iconv code and data will be linked) is to configure Newlib with the following options: @* @code{--enable-newlib-iconv-encodings=UTF-8 --enable-newlib-iconv-from-encodings=KOI8-R --enable-newlib-iconv-to-encodings=UCS-2,ISO-8859-5} @* which is the same as @* @code{--enable-newlib-iconv-from-encodings=KOI8-R,UTF-8 --enable-newlib-iconv-to-encodings=UCS-2,ISO-8859-5,UTF-8} @* User may also just use the @* @code{--enable-newlib-iconv-encodings=KOI8-R,ISO-8859-5,UTF-8,UCS-2} @* configure script option, but it isn't so optimal since there will be some unneeded data and code. @* The @option{--enable-newlib-iconv-external-ccs} option enables iconv's capabilities to work with the external CCS files. @* The @option{--enable-target-optspace} Newlib configure script option also affects the iconv library. If this option is present, the library uses the size optimized CCS tables. This means, that only the size-optimized CCS tables will be linked or, if the @option{--enable-newlib-iconv-external-ccs} configure script option was used, the iconv library will load the size-optimized tables. If the @option{--enable-target-optspace}configure script option is disabled, the speed-optimized CCS tables are used. @* Note: .cct files are searched by iconv_open in the $NLSPATH/iconv_data/ directory. Thus, the NLSPATH environment variable should be set. @page @node Encoding names @section Encoding names @findex encoding name @findex encoding alias @findex normalized name @* Each encoding has one @dfn{name} and a number of @dfn{aliases}. When user works with the iconv library (i.e., when the @code{iconv_open} call is used) both name or aliases may be used. The same is when encoding names are used in configure script options. @* Names and aliases may be specified in any case (small or capital letters) and the @kbd{-} symbol is equivalent to the @kbd{_} symbol. @* Internally the Newlib iconv library always converts aliases to names. It also converts names and aliases in the @dfn{normalized} form which means that all capital letters are converted to small letters and the @kbd{-} symbols are converted to @kbd{_} symbols. @page @node CCS tables @section CCS tables @findex Size-optimized CCS table @findex Speed-optimized CCS table @findex mktbl.pl Perl script @findex .cct files @findex The CCT tables source files @findex CCS source files @* The iconv library stores files with CCS tables in the the @emph{ccs/} subdirectory. The CCS tables for any CCS may be kept in two forms - in the binary form (@dfn{.cct files}, see the @emph{ccs/binary/} subdirectory) and in form of compilable .c source files. The .cct files are only used when the @option{--enable-newlib-iconv-external-ccs} configure script option is enabled. The .c files are linked to the Newlib library if the corresponding encoding is enabled. @* As stated earlier, the Newlib iconv library performs all conversions through the 32-bit UCS, but the codes which are used in most CCS-es, fit into the first 16-bit subset of the 32-bit UCS set. Thus, in order to make the CCS tables more compact, the 16-bit UCS-2 is used instead of the 32-bit UCS-4. @* CCS tables may be 8- or 16-bit wide. 8-bit CCS tables map 8-bit CCS to 16-bit UCS-2 and vice versa while 16-bit CCS tables map 16-bit CCS to 16-bit UCS-2 and vice versa. 8-bit tables are small (in size) while 16-bit tables may be big enough. Because of this, 16-bit CCS tables may be either speed- or size-optimized. Size-optimized CCS tables are smaller then speed-optimized ones, but the conversion process is slower if the size-optimized CCS tables are used. 8-bit CCS tables have only size-optimized variant. Each CCS table (both speed- and size-optimized) consists of @dfn{from_ucs} and @dfn{to_ucs} subtables. "from_ucs" subtable maps UCS-2 codes to CCS codes, while "to_ucs" subtable maps CCS codes to UCS-2 codes. @* Almost all 16-bit CCS tables contain less then 0xFFFF codes and a lot of gaps exist. @subsection Speed-optimized tables format @* In case of 8-bit speed-optimized CCS tables the "to_ucs" subtables format is trivial - it is just the array of 256 16-bit UCS codes. Therefore, an UCS-2 code @emph{Y} corresponding to a @emph{X} CCS code is calculates as @emph{Y = to_ucs[X]}. @* Obviously, the simplest way to create the "from_ucs" table or the 16-bit "to_ucs" table is to use the huge 16-bit array like in case of the 8-bit "to_ucs" table. But almost all the 16-bit CCS tables contain less then 0xFFFF code maps and this fact may be exploited to reduce the size of the CCS tables. @* In this chapter the "UCS-2 -> CCS" 8-bit CCS table format is described. The 16-bit "CCS -> UCS-2" CCS table format is the same, except the mapping direction and the CCS bits number. @* In case of the 8-bit speed-optimized table the "from_ucs" subtable corresponds the "from_ucs" array and has the following layout: @* from_ucs array: @* ------------------------------------- @* 0xFF mapping (2 bytes) (only for 8-bit table). @* ------------------------------------- @* Heading block @* ------------------------------------- @* Block 1 @* ------------------------------------- @* Block 2 @* ------------------------------------- @* ... @* ------------------------------------- @* Block N @* ------------------------------------- @* The 0x0000-0xFFFF 16-bit code range is divided to 256 code subranges. Each subrange is represented by an 256-element @dfn{block} (256 1-byte elements or 256 2-byte element in case of 16-bit CCS table) with elements which are equivalent to the CCS codes of this subrange. If the "UCS-2 -> CCS" mapping has big enough gaps, some blocks will be absent and there will be less then 256 blocks. @* Any element number @emph{m} of @dfn{the heading block} (which contains 256 2-byte elements) corresponds to the @emph{m}-th 256-element subrange. If the subrange contains some codes, the value of the @emph{m}-th element of the heading block contains the offset of the corresponding block in the "from_ucs" array. If there is no codes in the subrange, the heading block element contains 0xFFFF. @* If there are some gaps in a block, the corresponding block elements have the 0xFF value. If there is an 0xFF code present in the CCS, it's mapping is defined in the first 2-byte element of the "from_ucs" array. @* Having such a table format, the algorithm of searching the CCS code @emph{X} which corresponds to the UCS-2 code @emph{Y} is as follows. @* @enumerate @item If @emph{Y} is equivalent to the value of the first 2-byte element of the "from_ucs" array, @emph{X} is 0xFF. Else, continue to search. @item Calculate the block number: @emph{BlkN = (Y & 0xFF00) >> 8}. @item If the heading block element with number @emph{BlkN} is 0xFFFF, there is no corresponding CCS code (error, wrong input data). Else, fetch the "flom_ucs" array index of the @emph{BlkN}-th block. @item Calculate the offset of the @emph{X} code in its block: @emph{Xindex = Y & 0xFF} @item If the @emph{Xindex}-th element of the block (which is equivalent to @emph{from_ucs[BlkN+Xindex]}) value is 0xFF, there is no corresponding CCS code (error, wrong input data). Else, @emph{X = from_ucs[BlkN+Xindex]}. @end enumerate @subsection Size-optimized tables format @* As it is stated above, size-optimized tables exist only for 16-bit CCS-es. This is because there is too small difference between the speed-optimized and the size-optimized table sizes in case of 8-bit CCS-es. @* Formats of the "to_ucs" and "from_ucs" subtables are equivalent in case of size-optimized tables. This sections describes the format of the "UCS-2 -> CCS" size-optimized CCS table. The format of "CCS -> UCS-2" table is the same. The idea of the size-optimized tables is to split the UCS-2 codes ("from" codes) on @dfn{ranges} (@dfn{range} is a number of consecutive UCS-2 codes). Then CCS codes ("to" codes) are stored only for the codes from these ranges. Distinct "from" codes, which have no range (@dfn{unranged codes}, are stored together with the corresponding "to" codes. @* The following is the layout of the size-optimized table array: @* size_arr array: @* ------------------------------------- @* Ranges number (2 bytes) @* ------------------------------------- @* Unranged codes number (2 bytes) @* ------------------------------------- @* Unranged codes array index (2 bytes) @* ------------------------------------- @* Ranges indexes (triads) @* ------------------------------------- @* Ranges @* ------------------------------------- @* Unranged codes array @* ------------------------------------- @* The @dfn{Unranged codes array index} @emph{size_arr} section helps to find the offset of the needed range in the @emph{size_arr} and has the following format (triads): @* the first code in range, the last code in range, range offset. @* The array of these triads is sorted by the firs element, therefore it is possible to quickly find the needed range index. @* Each range has the corresponding sub-array containing the "to" codes. These sub-arrays are stored in the place marked as "Ranges" in the layout diagram. @* The "Unranged codes array" contains pairs ("from" code, "to" code") for each unranged code. The array of these pairs is sorted by "from" code values, therefore it is possible to find the needed pair quickly. @* Note, that each range requires 6 bytes to form its index. If, for example, there are two ranges (1 - 5 and 9 - 10), and one unranged code (7), 12 bytes are needed for two range indexes and 4 bytes for the unranged code (total 16). But it is better to join both ranges as 1 - 10 and mark codes 6 and 8 as absent. In this case, only 6 additional bytes for the range index and 4 bytes to mark codes 6 and 8 as absent are needed (total 10 bytes). This optimization is done in the size-optimized tables. Thus, ranges may contain small gaps. The absent codes in ranges are marked as 0xFFFF. @* Note, a pair of "from" codes is stored by means of unranged codes since the number of bytes which are needed to form the range is greater than the number of bytes to store two unranged codes (5 against 4). @* The algorithm of searching of the CCS code @emph{X} which corresponds to the UCS-2 code @emph{Y} (input) in the "UCS-2 -> CCS" size-optimized table is as follows. @* @enumerate @item Try to find the corresponding triad in the "Unranged codes array index". Since we are searching in the sorted array, we can do it quickly (divide by 2, compare, etc). @item If the triad is found, fetch the @emph{X} code from the corresponding range array. If it is 0xFFFF, return an error. @item If there is no corresponding triad, search the @emph{X} code among the sorted unranged codes. Return error, if noting was found. @end enumerate @subsection .cct ant .c CCS Table files @* The .c source files for 8-bit CCS tables have "to_ucs" and "from_ucs" speed-optimized tables. The .c source files for 16-bit CCS tables have "to_ucs_speed", "to_ucs_size", "from_ucs_speed" and "from_ucs_size" tables. @* When .c files are compiled and used, all the 16-bit and 32-bit values have the native endian format (Big Endian for the BE systems and Little Endian for the LE systems) since they are compile for the system before they are used. @* In case of .cct files, which are intended for dynamic CCS tables loading, the CCS tables are stored either in LE or BE format. Since the .cct files are generated by the 'mktbl.pl' Perl script, it is possible to choose the endianess of the tables. It is also possible to store two copies (both LE and BE) of the CCS tables in one .cct file. The default .cct files (which come with the Newlib sources) have both LE and BE CCS tables. The Newlib iconv library automatically chooses the needed CCS tables (with appropriate endianess). @* Note, the .cct files are only used when the @option{--enable-newlib-iconv-external-ccs} is used. @subsection The 'mktbl.pl' Perl script @* The 'mktbl.pl' script is intended to generate .cct and .c CCS table files from the @dfn{CCS source files}. @* The CCS source files are just text files which has one or more colons with CCS <-> UCS-2 codes mapping. To see an example of the CCS table source files see one of them using URL-s which will be given bellow. @* The following table describes where the source files for CCS table files provided by the Newlib distribution are located. @multitable @columnfractions .25 .75 @item Name @tab URL @item @tab @item big5 @tab http://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/OTHER/BIG5.TXT @item cns11643_plane1 cns11643_plane14 cns11643_plane2 @tab http://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/OTHER/CNS11643.TXT @item cp775 cp850 cp852 cp855 cp866 @tab http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/ @item iso_8859_1 iso_8859_2 iso_8859_3 iso_8859_4 iso_8859_5 iso_8859_6 iso_8859_7 iso_8859_8 iso_8859_9 iso_8859_10 iso_8859_11 iso_8859_13 iso_8859_14 iso_8859_15 @tab http://www.unicode.org/Public/MAPPINGS/ISO8859/ @item iso_ir_111 @tab http://crl.nmsu.edu/~mleisher/csets/ISOIR111.TXT @item jis_x0201_1976 jis_x0208_1990 jis_x0212_1990 @tab http://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/JIS/JIS0201.TXT @item koi8_r @tab http://www.unicode.org/Public/MAPPINGS/VENDORS/MISC/KOI8-R.TXT @item koi8_ru @tab http://crl.nmsu.edu/~mleisher/csets/KOI8RU.TXT @item koi8_u @tab http://crl.nmsu.edu/~mleisher/csets/KOI8U.TXT @item koi8_uni @tab http://crl.nmsu.edu/~mleisher/csets/KOI8UNI.TXT @item ksx1001 @tab http://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/KSC/KSX1001.TXT @item win_1250 win_1251 win_1252 win_1253 win_1254 win_1255 win_1256 win_1257 win_1258 @tab http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/ @end multitable The CCS source files aren't distributed with Newlib because of License restrictions in most Unicode.org's files. The following are 'mktbl.pl' options which were used to generate .cct files. Note, to generate CCS tables source files @option{-s} option should be added. @enumerate @item For the iso_8859_10.cct, iso_8859_13.cct, iso_8859_14.cct, iso_8859_15.cct, iso_8859_1.cct, iso_8859_2.cct, iso_8859_3.cct, iso_8859_4.cct, iso_8859_5.cct, iso_8859_6.cct, iso_8859_7.cct, iso_8859_8.cct, iso_8859_9.cct, iso_8859_11.cct, win_1250.cct, win_1252.cct, win_1254.cct win_1256.cct, win_1258.cct, win_1251.cct, win_1253.cct, win_1255.cct, win_1257.cct, koi8_r.cct, koi8_ru.cct, koi8_u.cct, koi8_uni.cct, iso_ir_111.cct, big5.cct, cp775.cct, cp850.cct, cp852.cct, cp855.cct, cp866.cct, cns11643.cct files, only the @option{-i } option were used. @item To generate the jis_x0208_1990.cct file, the @option{-i jis_x0208_1990.txt -x 2 -y 3} options were used. @item To generate the cns11643_plane1.cct file, the @option{-i cns11643.txt -p1 -N cns11643_plane1 -o cns11643_plane1.cct} options were used. @item To generate the cns11643_plane2.cct file, the @option{-i cns11643.txt -p2 -N cns11643_plane2 -o cns11643_plane2.cct} options were used. @item To generate the cns11643_plane14.cct file, the @option{-i cns11643.txt -p0xE -N cns11643_plane14 -o cns11643_plane14.cct} options were used. @end enumerate @* For more info about the 'mktbl.pl' options, see the 'mktbl.pl -h' output. @* It is assumed that CCS codes are 16 or less bits wide. If there are wider CCS codes in the CCS source file, the bits which are higher then 16 defines plane (see the cns11643.txt CCS source file). @* Sometimes, it is impossible to map some CCS codes to the 16-bit UCS if, for example, several different CCS codes are mapped to one UCS-2 code or one CCS code is mapped to the pair of UCS-2 codes. In these cases, such CCS codes (@dfn{lost codes}) aren't just rejected but instead, they are mapped to the default UCS-2 code (which is currently the @kbd{?} character's code). @page @node CES converters @section CES converters @findex PCS @* Similar to the CCS tables, CES converters are also split into "from UCS" and "to UCS" parts. Depending on the iconv library configuration, these parts are enabled or disabled. @* The following it the list of CES converters which are currently present in the Newlib iconv library. @itemize @bullet @item @emph{euc} - supports the @emph{euc_jp}, @emph{euc_kr} and @emph{euc_tw} encodings. The @emph{euc} CES converter uses the @emph{table} and the @emph{us_ascii} CES converters. @item @emph{table} - this CES converter corresponds to "null" and just performs tables-based conversion using 8- and 16-bit CCS tables. This converter is also used by any other CES converter which needs the CCS table-based conversions. The @emph{table} converter is also responsible for .cct files loading. @item @emph{table_pcs} - this is the wrapper over the @emph{table} converter which is intended for 16-bit encodings which also use the @dfn{Portable Character Set} (@dfn{PCS}) which is the same as the @emph{US-ASCII}. This means, that if the first byte the CCS code is in range of [0x00-0x7f], this is the 7-bit PCS code. Else, this is the 16-bit CCS code. Of course, the 16-bit codes must not contain bytes in the range of [0x00-0x7f]. The @emph{big5} encoding uses the @emph{table_pcs} CES converter and the @emph{table_pcs} CES converter depends on the @emph{table} CES converter. @item @emph{ucs_2} - intended for the @emph{ucs_2}, @emph{ucs_2be} and @emph{ucs_2le} encodings support. @item @emph{ucs_4} - intended for the @emph{ucs_4}, @emph{ucs_4be} and @emph{ucs_4le} encodings support. @item @emph{ucs_2_internal} - intended for the @emph{ucs_2_internal} encoding support. @item @emph{ucs_4_internal} - intended for the @emph{ucs_4_internal} encoding support. @item @emph{us_ascii} - intended for the @emph{us_ascii} encoding support. In principle, the most natural way to support the @emph{us_ascii} encoding is to define the @emph{us_ascii} CCS and use the @emph{table} CES converter. But for the optimization purposes, the specialized @emph{us_ascii} CES converter was created. @item @emph{utf_16} - intended for the @emph{utf_16}, @emph{utf_16be} and @emph{utf_16le} encodings support. @item @emph{utf_8} - intended for the @emph{utf_8} encoding support. @end itemize @page @node The encodings description file @section The encodings description file @findex encoding.deps description file @findex mkdeps.pl Perl script @* To simplify the process of adding new encodings support allowing to automatically generate a lot of "glue" files. @* There is the 'encoding.deps' file in the @emph{lib/} subdirectory which is used to describe encoding's properties. The 'mkdeps.pl' Perl script uses 'encoding.deps' to generates the "glue" files. @* The 'encoding.deps' file is composed of sections, each section consists of entries, each entry contains some encoding/CES/CCS description. @* The 'encoding.deps' file's syntax is very simple. Currently only two sections are defined: @emph{ENCODINGS} and @emph{CES_DEPENDENCIES}. @* Each @emph{ENCODINGS} section's entry describes one encoding and contains the following information. @itemize @bullet @item Encoding name (the @emph{ENCODING} field). The name should be unique and only one name is possible. @item The encoding's CES converter name (the @emph{CES} field). Only one CES converter is allowed. @item The whitespace-separated list of CCS table names which are used by the encoding (the @emph{CCS} field). @item The whitespace-separated list of aliases names (the @emph{ENCODING} field). @end itemize @* Note all names in the 'encoding.deps' file have to have the normalized form. @* Each @emph{CES_DEPENDENCIES} section's entry describes dependencies of one CES converted. For example, the @emph{euc} CES converter depends on the @emph{table} and the @emph{us_ascii} CES converter since the @emph{euc} CES converter uses them. This means, that both @emph{table} and @emph{us_ascii} CES converters should be linked if the @emph{euc} CES converter is enabled. @* The @emph{CES_DEPENDENCIES} section defines the following: @itemize @bullet @item the CES converter name for which the dependencies are defined in this entry (the @emph{CES} field); @item the whitespace-separated list of CES converters which are needed for this CES converter (the @emph{USED_CES} field). @end itemize @* The 'mktbl.pl' Perl script automatically solves the following tasks. @itemize @bullet @item User works with the iconv library in terms of encodings and doesn't know anything about CES converters and CCS tables. The script automatically generates code which enables all needed CES converters and CCS tables for all encodings, which were enabled by the user. @item The CES converters may have dependencies and the script automatically generates the code which handles these dependencies. @item The list of encoding's aliases is also automatically generated. @item The script uses a lot of macros in order to enable only the minimum set of code/data which is needed to support the requested encodings in the requested directions. @end itemize @* The 'mktbl.pl' Perl script is intended to interpret the 'encoding.deps' file and generates the following files. @itemize @bullet @item @emph{lib/encnames.h} - this header files contains macro definitions for all encoding names @item @emph{lib/aliasesbi.c} - the array of encoding names and aliases. The array is used to find the name of requested encoding by it's alias. @item @emph{ces/cesbi.c} - this file defines two arrays (@code{_iconv_from_ucs_ces} and @code{_iconv_to_ucs_ces}) which contain description of enabled "to UCS" and "from UCS" CES converters and the names of encodings which are supported by these CES converters. @item @emph{ces/cesbi.h} - this file contains the set of macros which defines the set of CES converters which should be enabled if only the set of enabled encodings is given (through macros defined in the @emph{newlib.h} file). Note, that one CES converter may handle several encodings. @item @emph{ces/cesdeps.h} - the CES converters dependencies are handled in this file. @item @emph{ccs/ccsdeps.h} - the array of linked-in CCS tables is defined here. @item @emph{ccs/ccsnames.h} - this header files contains macro definitions for all CCS names. @item @emph{encoding.aliases} - the list of supported encodings and their aliases which is intended for the Newlib configure scripts in order to handle the iconv-related configure script options. @end itemize @page @node How to add new encoding @section How to add new encoding @* At first, the new encoding should be broken down to CCS and CES. Then, the process of adding new encoding is split to the following activities. @enumerate @item Generate the .cct CCS file and the .c source file for the new encoding's CCS (if it isn't already present). To do this, the CCS source file should be had and the 'mktbl.pl' script should be used. @item Write the corresponding CES converter (if it isn't already present). Use the existing CES converters as an example. @item Add the corresponding entries to the 'encoding.deps' file and regenerate the autogenerated "glue" files using the 'mkdeps.pl' script. @item Don't forget to add entries to the newlib/newlib.hin file. @item Of course, the 'Makefile.am'-s should also be updated (if new files were added) and the 'Makefile.in'-s should be regenerated using the correct version of 'automake'. @item Don't forget to update the documentation (the list of supported encodings and CES converters). @end enumerate In case a new encoding doesn't fit to the CES/CCS decomposition model or it is desired to add the specialized (non UCS-based) conversion support, the Newlib iconv library code should be upgraded. @page @node The locale support interfaces @section The locale support interfaces @* The newlib iconv library also has some interface functions (besides the @code{iconv}, @code{iconv_open} and @code{iconv_close} interfaces) which are intended for the Locale subsystem. All the locale-related code is placed in the @emph{lib/iconvnls.c} file. @* The following is the description of the locale-related interfaces: @itemize @bullet @item @code{_iconv_nls_open} - opens two iconv descriptors for "CCS -> wchar_t" and "wchar_t -> CCS" conversions. The normalized CCS name is passed in the function parameters. The @emph{wchar_t} characters encoding is either ucs_2_internal or ucs_4_internal depending on size of @emph{wchar_t}. @item @code{_iconv_nls_conv} - the function is similar to the @code{iconv} functions, but if there is no character in the output encoding which corresponds to the character in the input encoding, the default conversion isn't performed (the @code{iconv} function sets such output characters to the @kbd{?} symbol and this is the behavior, which is specified in SUSv3). @item @code{_iconv_nls_get_state} - returns the current encoding's shift state (the @code{mbstate_t} object). @item @code{_iconv_nls_set_state} sets the current encoding's shift state (the @code{mbstate_t} object). @item @code{_iconv_nls_is_stateful} - checks whether the encoding is stateful or stateless. @item @code{_iconv_nls_get_mb_cur_max} - returns the maximum length (the maximum bytes number) of the encoding's characters. @end itemize @page @node Contact @section Contact @* The author of the original BSD iconv library (Alexander Chuguev) no longer supports that code. @* Any questions regarding the iconv library may be forwarded to Artem B. Bityuckiy (dedekind@@oktetlabs.ru or dedekind@@mail.ru) as well as to the public Newlib mailing list.