postgresql/doc/src/sgml/charset.sgml

 <chapter id="charset">
  <title>Character Sets</title>

  <abstract>
   <para>
    Describes the available language and character set support in
    <productname>Postgres</productname>.
   </para>
  </abstract>

  <para>
   <productname>Postgres</productname> supports non-ASCII character
   sets with two approaches: 

   <itemizedlist>
    <listitem>
     <para>
      Using locale features in underlying
      system libraries. This allows single-byte character sets to be
      configured with a locale-specific collation order, provided that
      the underlying system supports the required locale. This
      technique supports only one character set per server, and can
      not support multi-byte character sets.
     </para>
    </listitem>

    <listitem>
     <para>
      Using explicit multiple-byte character sets defined in the
      <productname>Postgres</productname> server. These character sets
      are also known to some client libraries. The number of character
      sets is fixed at the time the server is compiled, and internal
      operations such as string comparisons require expansion of each
      character into a 32-bit word.
     </para>
    </listitem>
   </itemizedlist>
  </para>

  <sect1>
   <title>Multi-byte Support</title>

   <note>
    <title>Author</title>

    <para>
     <ulink url="mailto:ishii@postgresql.org">Tatsuo Ishii</ulink>,
     last updated 2000-03-22.
     Check <ulink
      url="http://www.sra.co.jp/people/t-ishii/PostgreSQL/">Tatsuo's
      web site</ulink> for more information.
    </para>
   </note>

   <para>
    Multi-byte (<acronym>MB</acronym>) support is intended to allow
    <productname>Postgres</productname> to handle
    multiple-byte character sets such as EUC (Extended Unix Code), Unicode and
    Mule internal code. With <acronym>MB</acronym> enabled you can use multi-byte
    character sets in regular expressions (regexp), LIKE, and some
    other functions. The default
    encoding system is selected while initializing your
    <productname>Postgres</productname> installation using
    <application>initdb</application>. Note that this can be
    overridden when you create a database using
    <application>createdb</application> or by using the SQL command
    CREATE DATABASE. So you can have multiple databases each with
    a different encoding system.
   </para>

   <para>
    <acronym>MB</acronym> also fixes some problems concerning 8-bit single byte
    character sets including ISO8859. (I would not say all of problems
    have been fixed. I just confirmed that the regression test ran fine
    and a few French characters could be used with the patch. Please let
    me know if you find any problem while using 8-bit characters.)
   </para>

   <sect2>
    <title>Enabling MB</title>

    <para>
     Run configure with a multibyte option:

     <programlisting>
% ./configure --enable-multibyte[=<replaceable>encoding_system</replaceable>]
     </programlisting>

     where <replaceable>encoding_system</replaceable> can be one of the
     values in the following table:

     <table tocentry="1">
      <title><productname>Postgres</productname> Character Set Encodings</title>
      <titleabbrev>Encodings</titleabbrev>
      <tgroup cols="2">
       <thead>
	<row>
	 <entry>Encoding</entry>
	 <entry>Description</entry>
	</row>
       </thead>
       <tbody>
	<row>
	 <entry>SQL_ASCII</entry>
	 <entry>ASCII</entry>
	</row>
	<row>
	 <entry>EUC_JP</entry>
	 <entry>Japanese EUC</entry>
	</row>
	<row>
	 <entry>EUC_CN</entry>
	 <entry>Chinese EUC</entry>
	</row>
	<row>
	 <entry>EUC_KR</entry>
	 <entry>Korean EUC</entry>
	</row>
	<row>
	 <entry>EUC_TW</entry>
	 <entry>Taiwan EUC</entry>
	</row>
	<row>
	 <entry>UNICODE</entry>
	 <entry>Unicode(UTF-8)</entry>
	</row>
	<row>
	 <entry>MULE_INTERNAL</entry>
	 <entry>Mule internal</entry>
	</row>
	<row>
	 <entry>LATIN1</entry>
	 <entry>ISO 8859-1 English and some European languages</entry>
	</row>
	<row>
	 <entry>LATIN2</entry>
	 <entry>ISO 8859-2 English and some European languages</entry>
	</row>
	<row>
	 <entry>LATIN3</entry>
	 <entry>ISO 8859-3 English and some European languages</entry>
	</row>
	<row>
	 <entry>LATIN4</entry>
	 <entry>ISO 8859-4 English and some European languages</entry>
	</row>
	<row>
	 <entry>LATIN5</entry>
	 <entry>ISO 8859-5 English and some European languages</entry>
	</row>
	<row>
	 <entry>KOI8</entry>
	 <entry>KOI8-R</entry>
	</row>
	<row>
	 <entry>WIN</entry>
	 <entry>Windows CP1251</entry>
	</row>
	<row>
	 <entry>ALT</entry>
	 <entry>Windows CP866</entry>
	</row>
       </tbody>
      </tgroup>
     </table>
    </para>

    <para>
     Here is an example of configuring
     <productname>Postgres</productname> to use a Japanese encoding by
     default:

     <programlisting>
% ./configure --enable-multibyte=EUC_JP
     </programlisting>
    </para>

    <para>
     If the encoding system is omitted (./configure --enable-multibyte),
     SQL_ASCII is assumed.
    </para>
   </sect2>

   <sect2>
    <title>Setting the Encoding</title>

    <para>
     <application>initdb</application> defines the default encoding
     for a <productname>Postgres</productname> installation. For example:

     <programlisting>
% initdb -E EUC_JP
     </programlisting>

     sets the default encoding to EUC_JP(Extended Unix Code for Japanese).
     Note that you can use "--encoding" instead of "-E" if you prefer
     to type longer option strings. 
     If no -E or --encoding option is given, the encoding
     specified at the compile time is used.
    </para>

    <para>
     You can create a database with a different encoding:

     <programlisting>
% createdb -E EUC_KR korean
     </programlisting>

     will create a database named "korean" with EUC_KR encoding. The
     another way to accomplish this is to use a SQL command:

     <programlisting>
CREATE DATABASE korean WITH ENCODING = 'EUC_KR';
     </programlisting>

     The encoding for a database is represented as an
     <firstterm>encoding column</firstterm>  in the
     <literal>pg_database</literal> system catalog.
     You can see that by using -l or \l of psql
     command.

     <programlisting>
$ psql -l
            List of databases
   Database    |  Owner  |   Encoding    
---------------+---------+---------------
 euc_cn        | t-ishii | EUC_CN
 euc_jp        | t-ishii | EUC_JP
 euc_kr        | t-ishii | EUC_KR
 euc_tw        | t-ishii | EUC_TW
 mule_internal | t-ishii | MULE_INTERNAL
 regression    | t-ishii | SQL_ASCII
 template1     | t-ishii | EUC_JP
 test          | t-ishii | EUC_JP
 unicode       | t-ishii | UNICODE
(9 rows)
     </programlisting>
    </para>
   </sect2>

   <sect2>
    <title>Automatic encoding translation between backend and
     frontend</title>

    <para>
     <productname>Postgres</productname> supports an automatic
     encoding translation between backend
     and frontend for some encodings.

     <table tocentry="1">
      <title><productname>Postgres</productname> Client/Server Character Set Encodings</title>
      <titleabbrev>Communication Encodings</titleabbrev>
      <tgroup cols="2">
       <thead>
	<row>
	 <entry>Server Encoding</entry>
	 <entry>Available Client Encodings</entry>
	</row>
       </thead>
       <tbody>
	<row>
	 <entry>EUC_JP</entry>
	 <entry>EUC_JP, SJIS</entry>
	</row>
	<row>
	 <entry>EUC_TW</entry>
	 <entry>EUC_TW, BIG5</entry>
	</row>
	<row>  
	 <entry>LATIN2</entry>
	 <entry>LATIN2, WIN1250</entry>
	</row>
	<row>
	 <entry>LATIN5</entry>
	 <entry>LATIN5, WIN, ALT</entry>
	</row>
	<row>
	 <entry>MULE_INTERNAL</entry>
	 <entry>EUC_JP, SJIS, EUC_KR, EUC_CN, 
	  EUC_TW, BIG5, LATIN1 to LATIN5, 
	  WIN, ALT, WIN1250</entry>
	</row>
       </tbody>
      </tgroup>
     </table>
    </para>

    <para>
     To enable the automatic encoding translation, you have to tell
     <productname>Postgres</productname> the encoding you would like
     to use in frontend. There are
     several ways to accomplish this.

     <itemizedlist>
      <listitem>
       <para>
	Using the <command>\encoding</command> command in
	<application>psql</application>.
	<command>\encoding</command> allows you to change frontend
	encoding on the fly. For
	example, to change the encoding to SJIS, type:

	<programlisting>
\encoding SJIS
	</programlisting>
       </para>
      </listitem>

      <listitem>
       <para>
	Using libpq functions.
	<command>\encoding</command> actually calls
	PQsetClientEncoding() for its purpose.

	<programlisting>
int PQsetClientEncoding(PGconn *<replaceable>conn</replaceable>, const char *<replaceable>encoding</replaceable>)
	</programlisting>

	where <replaceable>conn</replaceable> is a connection to the backend,
	and <replaceable>encoding</replaceable> is an encoding you
	want to use. If it successfully sets the encoding, it returns 0,
	otherwise -1. The current encoding for this connection can be shown by
	using:

	<programlisting>
int PQclientEncoding(const PGconn *<replaceable>conn</replaceable>)
	</programlisting>

	Note that it returns the "encoding id," not the encoding symbol string
	such as "EUC_JP." To convert an encoding id to an encoding symbol, you
	can use:

	<programlisting>
char *pg_encoding_to_char(int <replaceable>encoding_id</replaceable>)
	</programlisting>
       </para>
      </listitem>

      <listitem>
       <para>
	Using <envar>PGCLIENTENCODING</envar>.

	If an environment variable <envar>PGCLIENTENCODING</envar> is defined in the
	frontend, an automatic encoding translation is done by the backend.
       </para>
      </listitem>

      <listitem>
       <para>
	Using <command>SET CLIENT_ENCODING TO</command>.

	Setting the frontend side encoding can be done a SQL command:

	<programlisting>
SET CLIENT_ENCODING TO 'encoding';
	</programlisting>

	Also you can use SQL92 syntax "SET NAMES" for this purpose:

	<programlisting>
SET NAMES 'encoding';
	</programlisting>

	To query the current the frontend encoding:

	<programlisting>
SHOW CLIENT_ENCODING;
	</programlisting>

	To return to the default encoding:

	<programlisting>
RESET CLIENT_ENCODING;
	</programlisting>
       </para>
      </listitem>
     </itemizedlist>
    </para>
   </sect2>

   <sect2>
    <title>About Unicode</title>

    <para>
     An automatic encoding translation between Unicode and other
     encodings is not yet supported.
    </para>
   </sect2>

   <sect2>
    <title>What happens if the translation is not possible?</title>

    <para>
     Suppose you choose EUC_JP for the backend, LATIN1 for the frontend,
     then some Japanese characters could not be translated into LATIN1. In
     this case, a letter cannot be represented in the LATIN1 character set,
     would be transformed as:

     <programlisting>
(HEXA DECIMAL)
     </programlisting>
    </para>
   </sect2>

   <sect2>
    <title>References</title>

    <para>
     These are good sources to start learning various kind of encoding
     systems.

     <itemizedlist>
      <listitem>
       <para>
	<ulink url="ftp://ftp.ora.com/pub/examples/nutshell/ujip/doc/cjk.inf">
	 ftp://ftp.ora.com/pub/examples/nutshell/ujip/doc/cjk.inf</ulink>
	Detailed explanations of EUC_JP, EUC_CN, EUC_KR, EUC_TW
	appear in section 3.2.
       </para>
      </listitem>

      <listitem>
       <para>
	Unicode: <ulink url="http://www.unicode.org/">http://www.unicode.org/</ulink>
	The homepage of UNICODE.
       </para>
      </listitem>

      <listitem>
       <para>
	<literal>RFC 2044</literal>
	UTF-8 is defined here.
       </para>
      </listitem>
     </itemizedlist>
    </para>
   </sect2>

   <sect2>
    <title>History</title>

    <para>
     <programlisting>
May 20, 2000
	* SJIS UDC (NEC selection IBM kanji) support contributed
	  by Eiji Tokuya
	* Changes above will appear in 7.0.1

Mar 22, 2000
	* Add new libpq functions PQsetClientEncoding, PQclientEncoding
	* ./configure --with-mb=EUC_JP
	  now deprecated. use 
	  ./configure --enable-multibyte=EUC_JP
	  instead
  	* Add SQL_ASCII regression test case
	* Add SJIS User Defined Character (UDC) support
	* All of above will appear in 7.0

July 11, 1999
	* Add support for WIN1250 (Windows Czech) as a client encoding
	  (contributed by Pavel Behal)
	* fix some compiler warnings (contributed by Tomoaki Nishiyama)

Mar 23, 1999
	* Add support for KOI8(KOI8-R), WIN(CP1251), ALT(CP866)
	  (thanks Oleg Broytmann for testing)
	* Fix problem with MB and locale

Jan 26, 1999
	* Add support for Big5 for fronend encoding
	  (you need to create a database with EUC_TW to use Big5)
	* Add regression test case for EUC_TW
	  (contributed by <ulink url="mailto:jonahkuo@mail.ttn.com.tw">Jonah Kuo</ulink>)

Dec 15, 1998
	* Bugs related to SQL_ASCII support fixed

Nov 5, 1998
	* 6.4 release. In this version, pg_database has "encoding"
	  column that represents the database encoding

Jul 22, 1998
	* determine encoding at initdb/createdb rather than compile time
	* support for PGCLIENTENCODING when issuing COPY command
	* support for SQL92 syntax "SET NAMES"
	* support for LATIN2-5
	* add UNICODE regression test case
	* new test suite for MB
	* clean up source files

Jun 5, 1998
	* add support for the encoding translation between the backend
	  and the frontend
	* new command SET CLIENT_ENCODING etc. added
	* add support for LATIN1 character set
	* enhance 8 bit cleaness

April 21, 1998 some enhancements/fixes
	* character_length(), position(), substring() are now aware of 
	  multi-byte characters
	* add octet_length()
	* add --with-mb option to configure
	* new regression tests for EUC_KR
  	  (contributed by <ulink url="mailto:hong@lunaris.hanmesoft.co.kr">Soonmyung. Hong</ulink>)
	* add some test cases to the EUC_JP regression test
	* fix problem in regress/regress.sh in case of System V
	* fix toupper(), tolower() to handle 8bit chars

Mar 25, 1998 MB PL2 is incorporated into PostgreSQL 6.3.1

Mar 10, 1998 PL2 released
	* add regression test for EUC_JP, EUC_CN and MULE_INTERNAL
	* add an English document (this file)
	* fix problems concerning 8-bit single byte characters

Mar 1, 1998 PL1 released
     </programlisting>
    </para>
   </sect2>

   <sect2>
    <title>WIN1250 on Windows/ODBC</title>

    <para>
<!--
[Here is a good documentation explaining how to use WIN1250 on
Windows/ODBC from Pavel Behal. Please note that Installation step 1)
is not necceary in 6.5.1 - Tatsuo]

Version: 0.91 for PgSQL 6.5
Author: Pavel Behal
Revised by: Tatsuo Ishii
Email: <ulink url="mailto:behal@opf.slu.cz">behal@opf.slu.cz</ulink>
Licence: The Same as PostgreSQL

Sorry for my Eglish and C code, I'm not native :-)

!!!!!!!!!!!!!!!!!!!!!!!!! NO WARRANTY !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
-->

     The WIN1250 character set on Windows client platforms can be used
     with <productname>Postgres</productname> with locale support
     enabled.
    </para>

    <para>
     The following should be kept in mind:

     <itemizedlist>
      <listitem>
       <para>
	Success depends on proper system locales. This has been tested
	with RH6.0 and Slackware 3.6, with cs_CZ.iso8859-2 locale.
       </para>
      </listitem>

      <listitem>
       <para>
	Never try to set the server multibyte database encoding to WIN1250.
	Always use LATIN2 instead since there is not a WIN1250 locale
	in Unix.
       </para>
      </listitem>

      <listitem>
       <para>
	WIN1250 encoding is useable only for M$W ODBC clients. The
	characters are recoded on the fly, to be displayed and stored
	back properly.
       </para>
      </listitem>
     </itemizedlist>
    </para>

    <para>
     When running, it is important to remember the following:

     <itemizedlist>
      <listitem>
       <para>
	This configuration reorders your sort order depending on your
	<envar>LC_<replaceable>x</replaceable></envar> settings. Don't be
	confused with the regression test results since they don't use
	locale.
       </para>
      </listitem>

      <listitem>
       <para>
	A locale such as "<literal>ch</literal>" is correctly sorted
	only if your system
	supports that locale; older systems may not do so but new ones
	(e.g. RH6.0) do.
       </para>
      </listitem>

      <listitem>
       <para>
	You have to insert money as '<literal>162,50</literal>' (note
	comma within the single-quotes).
       </para>
      </listitem>

      <listitem>
       <para>
	At the time of writing (early 1999), this configuration has
	not received extensive testing. Please let us know of any
	changes you had to make!
       </para>
      </listitem>
     </itemizedlist>
    </para>

    <procedure>
     <title>WIN1250 on Windows/ODBC</title>
     <step>
      <para>
       Change the three relevant files in the source directories.
      </para>
     </step>

     <step>
      <para>
       Compile <productname>Postgres</productname> with local enabled
       and the multibyte encoding set to <literal>LATIN2</literal>.
      </para>
     </step>

     <step>
      <para>
       Set up your instalation. Do not forget to create locale
       variables in your profile (environment). For example (this may
       not be correct for <emphasis>your</emphasis> environment):

       <programlisting>
LC_ALL=cs_CZ.ISO8859-2
LC_COLLATE=cs_CZ.ISO8859-2
LC_CTYPE=cs_CZ.ISO8859-2
LC_MONETARY=cs_CZ.ISO8859-2
LC_NUMERIC=cs_CZ.ISO8859-2
LC_TIME=cs_CZ.ISO8859-2
       </programlisting>
      </para>
     </step>

     <step>
      <para>
       You have to start the postmaster with locales set!
      </para>
     </step>

     <step>
      <para>
       Try it with Czech language, and have it sort on a query.
      </para>
     </step>

     <step>
      <para>
       Install ODBC driver for PgSQL on your M$ Windows machine.
      </para>
     </step>

     <step>
      <para>
       Setup properly your data source. Include this line in your ODBC
       configuration dialog in the field <literal>Connect Settings</literal>:

       <programlisting>
SET CLIENT_ENCODING = 'WIN1250';
       </programlisting>
      </para>
     </step>

     <step>
      <para>
       Now try it again, but in Windows with ODBC.
      </para>
     </step>
    </procedure>
   </sect2>
  </sect1>
 </chapter>

<!-- Keep this comment at the end of the file
Local variables:
mode:sgml
sgml-omittag:nil
sgml-shorttag:t
sgml-minimize-attributes:nil
sgml-always-quote-attributes:t
sgml-indent-step:1
sgml-indent-data:t
sgml-parent-document:nil
sgml-default-dtd-file:"./reference.ced"
sgml-exposed-tags:nil
sgml-local-catalogs:("/usr/lib/sgml/catalog")
sgml-local-ecat-files:nil
End:
-->