Localization</>
<abstract>
<para>
Describes the available localization features from the point of
view of the administrator.
</para>
</abstract>
<para>
<productname>Postgres</productname> supports localization with
three approaches:
<itemizedlist>
<listitem>
<para>
Using the locale features of the operating system to provide
locale-specific collation order, number formatting, and other
aspects.
</para>
</listitem>
<listitem>
<para>
Using explicit multiple-byte character sets defined in the
<productname>Postgres</productname> server to support languages
that require more characters than will fit into a single byte,
and to provide character set recoding between client and server.
The number of supported character sets is fixed at the time the
server is compiled, and internal operations such as string
comparisons require expansion of each character into a 32-bit
word.
</para>
</listitem>
<listitem>
<para>
Single byte character recoding provides a more light-weight
solution for users of multiple, yet single-byte character sets.
</para>
</listitem>
</itemizedlist>
</para>
<sect1 id="locale">
<title>Locale SupportLocale support refers to an application respecting
cultural preferences regarding alphabets, sorting, number
formatting, etc. PostgreSQL uses the standard ISO
C and POSIX-like locale facilities provided by the server operating
system. For additional information refer to the documentation of your
system.
Overview</>
<para>
Locale support is not built into <productname>PostgreSQL</> by
default; to enable it, supply the <option>--enable-locale</> option
to the <filename>configure</> script:
<informalexample>
<screen>
<prompt>$ </><userinput>./configure --enable-locale</>
</screen>
</informalexample>
Locale support only affects the server; all clients are compatible
with servers with or without locale support.
</para>
<para>
The information about which particular cultural rules to use is
determined by standard environment variables. If you are getting
localized behavior from other programs you probably have them set
up already. The simplest way to set the localization information
is the <envar>LANG</> variable, for example:
<programlisting>
export LANG=sv_SE
</programlisting>
This sets the locale to Swedish (<literal>sv</>) as spoken in
Sweden (<literal>SE</>). Other possibilities might be
<literal>en_US</> (U.S. English) and <literal>fr_CA</> (Canada,
French). If more than one character set can be useful for a locale
then the specifications look like this:
<literal>cs_CZ.ISO8859-2</>. What locales are available under what
names on your system depends on what was provided by the operating
system vendor and what was installed.
</para>
<para>
Occasionally it is useful to mix rules from several locales, e.g.,
use U.S. collation rules but Spanish messages. To do that a set of
environment variables exist that override the default of
<envar>LANG</> for a particular category:
<informaltable>
<tgroup cols="2">
<tbody>
<row>
<entry><envar>LC_COLLATE</></>
<entry>String sort order</>
</row>
<row>
<entry><envar>LC_CTYPE</></>
<entry>Character classification (What is a letter? The upper-case equivalent?)</>
</row>
<row>
<entry><envar>LC_MESSAGES</></>
<entry>Language of messages</>
</row>
<row>
<entry><envar>LC_MONETARY</></>
<entry>Formatting of currency amounts</>
</row>
<row>
<entry><envar>LC_NUMERIC</></>
<entry>Formatting of numbers</>
</row>
<row>
<entry><envar>LC_TIME</></>
<entry>Formatting of dates and times</>
</row>
</tbody>
</tgroup>
</informaltable>
<envar>LC_MESSAGES</> only affects the messages that come from the
operating system, not <productname>PostgreSQL</>.
</para>
<para>
If you want the system to behave as if it had no locale support,
use the special locale <literal>C</> or <literal>POSIX</>, or
simply unset all locale related variables.
</para>
<para>
Note that the locale behavior is determined by the environment
variables seen by the server, not by the environment of any client.
Therefore, be careful to set these variables before starting the
postmaster.
</para>
<para>
The <envar>LC_COLLATE</> and <envar>LC_CTYPE</> variables affect the
sort order of indexes. Therefore, these values must be kept fixed
for any particular database cluster, or indexes on text columns will
become corrupt. <productname>Postgres</productname> enforces this
by recording the values of <envar>LC_COLLATE</> and <envar>LC_CTYPE</>
that are seen by <command>initdb</>. The server automatically adopts
those two values when it is started; only the other <envar>LC_</>
categories can be set from the environment at server startup.
In short, only one collation order can be used in a database cluster,
and it is chosen at <command>initdb</> time.
</para>
</sect2>
<sect2>
<title>Benefits</>
<para>
Locale support influences in particular the following features:
<itemizedlist>
<listitem>
<para>
Sort order in <command>ORDER BY</> queries.
</para>
</listitem>
<listitem>
<para>
The <function>to_char</> family of functions
</para>
</listitem>
<listitem>
<para>
The <literal>LIKE</> and <literal>~</> operators for pattern
matching
</para>
</listitem>
</itemizedlist>
</para>
<para>
The only severe drawback of using the locale support in
<productname>PostgreSQL</> is its speed. So use locale only if you
actually need it. It should be noted in particular that selecting
a non-C locale disables index optimizations for <literal>LIKE</> and
<literal>~</> operators, which can make a huge difference in the
speed of searches that use those operators.
</para>
</sect2>
<sect2>
<title>Problems</>
<para>
If locale support doesn't work in spite of the explanation above,
check that the locale support in your operating system is correctly configured.
To check whether a given locale is installed and functional you
can use <application>Perl</>, for example. Perl has also support
for locales and if a locale is broken <command>perl -v</> will
complain something like this:
<screen>
<prompt>$</> <userinput>export LC_CTYPE='not_exist'</>
<prompt>$</> <userinput>perl -v</>
<computeroutput>
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LC_ALL = (unset),
LC_CTYPE = "not_exist",
LANG = (unset)
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
</computeroutput>
</screen>
</para>
<para>
Check that your locale files are in the right location. Possible
locations include: <filename>/usr/lib/locale</filename> (<systemitem class="osname">Linux</>,
<systemitem class="osname">Solaris</>), <filename>/usr/share/locale</filename> (<systemitem class="osname">Linux</>),
<filename>/usr/lib/nls/loc</filename> (<systemitem class="osname">DUX 4.0</>). Check the locale
man page of your system if you are not sure.
</para>
<para>
The directory <filename>src/test/locale</> contains a test suite
for <productname>PostgreSQL</>'s locale support.
</para>
</sect2>
</sect1>
<sect1 id="multibyte">
<title>Multibyte SupportAuthor
Tatsuo Ishii (ishii@postgresql.org),
last updated 2000-03-22.
Check Tatsuo's
web site for more information.
Multibyte (MB) support is intended to allow
Postgres to handle
multiple-byte character sets such as EUC (Extended Unix Code), Unicode and
Mule internal code. With MB enabled you can use multibyte
character sets in regular expressions (regexp), LIKE, and some
other functions. The default
encoding system is selected while initializing your
Postgres installation using
initdb. Note that this can be
overridden when you create a database using
createdb or by using the SQL command
CREATE DATABASE. So you can have multiple databases each with
a different encoding system.
MB also fixes some problems concerning 8-bit single byte
character sets including ISO8859. (I would not say all problems
have been fixed. I just confirmed that the regression test ran fine
and a few French characters could be used with the patch. Please let
me know if you find any problem while using 8-bit characters.)
Enabling MB
Run configure with the multibyte option:
% ./configure --enable-multibyte[=encoding_system]
where encoding_system can be one of the
values in the following table:
<productname>Postgres</productname> Character Set EncodingsEncodingsEncodingDescriptionSQL_ASCIIASCIIEUC_JPJapanese EUCEUC_CNChinese EUCEUC_KRKorean EUCEUC_TWTaiwan EUCUNICODEUnicode (UTF-8)MULE_INTERNALMule internalLATIN1ISO 8859-1 English and some European languagesLATIN2ISO 8859-2 English and some European languagesLATIN3ISO 8859-3 English and some European languagesLATIN4ISO 8859-4 English and some European languagesLATIN5ISO 8859-5 English and some European languagesKOI8KOI8-R(U)WINWindows CP1251ALTWindows CP866
Here is an example of configuring
Postgres to use a Japanese encoding by
default:
% ./configure --enable-multibyte=EUC_JP
If the encoding system is omitted (./configure --enable-multibyte),
SQL_ASCII is assumed.
Setting the Encodinginitdb defines the default encoding
for a Postgres installation. For example:
% initdb -E EUC_JP
sets the default encoding to EUC_JP (Extended Unix Code for Japanese).
Note that you can use "--encoding" instead of "-E" if you prefer
to type longer option strings.
If no -E or --encoding option is given, the encoding
specified at configure time is used.
You can create a database with a different encoding:
% createdb -E EUC_KR korean
will create a database named korean with EUC_KR encoding.
Another way to accomplish this is to use a SQL command:
CREATE DATABASE korean WITH ENCODING = 'EUC_KR';
The encoding for a database is represented as an
encoding column in the
pg_database system catalog.
You can see that by using or \l of psql
command.
$ psql -l
List of databases
Database | Owner | Encoding
---------------+---------+---------------
euc_cn | t-ishii | EUC_CN
euc_jp | t-ishii | EUC_JP
euc_kr | t-ishii | EUC_KR
euc_tw | t-ishii | EUC_TW
mule_internal | t-ishii | MULE_INTERNAL
regression | t-ishii | SQL_ASCII
template1 | t-ishii | EUC_JP
test | t-ishii | EUC_JP
unicode | t-ishii | UNICODE
(9 rows)
Automatic encoding translation between backend and
frontendPostgres supports an automatic
encoding translation between backend
and frontend for some encodings.
<productname>Postgres</productname> Client/Server Character Set EncodingsCommunication EncodingsServer EncodingAvailable Client EncodingsEUC_JPEUC_JP, SJISEUC_TWEUC_TW, BIG5LATIN2LATIN2, WIN1250LATIN5LATIN5, WIN, ALTMULE_INTERNALEUC_JP, SJIS, EUC_KR, EUC_CN,
EUC_TW, BIG5, LATIN1 to LATIN5,
WIN, ALT, WIN1250
To enable the automatic encoding translation, you have to tell
Postgres the encoding you would like
to use in frontend. There are
several ways to accomplish this.
Using the \encoding command in
psql.
\encoding allows you to change frontend
encoding on the fly. For
example, to change the encoding to SJIS, type:
\encoding SJIS
Using libpq functions.
\encoding actually calls
PQsetClientEncoding() for its purpose.
int PQsetClientEncoding(PGconn *conn, const char *encoding)
where conn is a connection to the backend,
and encoding is an encoding you
want to use. If it successfully sets the encoding, it returns 0,
otherwise -1. The current encoding for this connection can be shown by
using:
int PQclientEncoding(const PGconn *conn)
Note that it returns the "encoding id," not the encoding symbol string
such as EUC_JP. To convert an encoding id to an encoding symbol, you
can use:
char *pg_encoding_to_char(int encoding_id)
Using SET CLIENT_ENCODING TO.
Setting the frontend side encoding can be done by this SQL command:
SET CLIENT_ENCODING TO 'encoding';
Also you can use SQL92 syntax "SET NAMES" for this purpose:
SET NAMES 'encoding';
To query the current frontend encoding:
SHOW CLIENT_ENCODING;
To return to the default encoding:
RESET CLIENT_ENCODING;
Using PGCLIENTENCODING.
If environment variable PGCLIENTENCODING is defined
in the client's environment, that client encoding is automatically
selected when a backend connection is made. (This can subsequently
be overridden using any of the other methods mentioned above.)
About Unicode
An automatic encoding translation between Unicode and other
encodings has been supported since PostgreSQL 7.1.
Because this requires huge conversion tables, it's not enabled by default.
To enable this feature, run configure with the
option. Note that this requires
the option also.
What happens if the translation is not possible?
Suppose you choose EUC_JP for the backend, LATIN1 for the frontend,
then some Japanese characters could not be translated into LATIN1. In
this case, a letter that cannot be represented in the LATIN1 character set
would be transformed as:
(HEXA DECIMAL)
References
These are good sources to start learning about various kinds of encoding
systems.
ftp://ftp.ora.com/pub/examples/nutshell/ujip/doc/cjk.inf
Detailed explanations of EUC_JP, EUC_CN, EUC_KR, EUC_TW
appear in section 3.2.
Unicode: http://www.unicode.org/
The homepage of Unicode.
RFC 2044UTF-8 is defined here.
History
Dec 7, 2000
* An automatic encoding translation between Unicode and other
encodings are implemented
* Changes above will appear in 7.1
May 20, 2000
* SJIS UDC (NEC selection IBM kanji) support contributed
by Eiji Tokuya
* Changes above will appear in 7.0.1
Mar 22, 2000
* Add new libpq functions PQsetClientEncoding, PQclientEncoding
* ./configure --with-mb=EUC_JP
now deprecated. use
./configure --enable-multibyte=EUC_JP
instead
* Add SQL_ASCII regression test case
* Add SJIS User Defined Character (UDC) support
* All of above will appear in 7.0
July 11, 1999
* Add support for WIN1250 (Windows Czech) as a client encoding
(contributed by Pavel Behal)
* fix some compiler warnings (contributed by Tomoaki Nishiyama)
Mar 23, 1999
* Add support for KOI8(KOI8-R), WIN(CP1251), ALT(CP866)
(thanks Oleg Broytmann for testing)
* Fix problem with MB and locale
Jan 26, 1999
* Add support for Big5 for fronend encoding
(you need to create a database with EUC_TW to use Big5)
* Add regression test case for EUC_TW
(contributed by Jonah Kuo jonahkuo@mail.ttn.com.tw)
Dec 15, 1998
* Bugs related to SQL_ASCII support fixed
Nov 5, 1998
* 6.4 release. In this version, pg_database has "encoding"
column that represents the database encoding
Jul 22, 1998
* determine encoding at initdb/createdb rather than compile time
* support for PGCLIENTENCODING when issuing COPY command
* support for SQL92 syntax "SET NAMES"
* support for LATIN2-5
* add UNICODE regression test case
* new test suite for MB
* clean up source files
Jun 5, 1998
* add support for the encoding translation between the backend
and the frontend
* new command SET CLIENT_ENCODING etc. added
* add support for LATIN1 character set
* enhance 8 bit cleaness
April 21, 1998 some enhancements/fixes
* character_length(), position(), substring() are now aware of
multi-byte characters
* add octet_length()
* add --with-mb option to configure
* new regression tests for EUC_KR
(contributed by Soonmyung Hong hong@lunaris.hanmesoft.co.kr)
* add some test cases to the EUC_JP regression test
* fix problem in regress/regress.sh in case of System V
* fix toupper(), tolower() to handle 8bit chars
Mar 25, 1998 MB PL2 is incorporated into PostgreSQL 6.3.1
Mar 10, 1998 PL2 released
* add regression test for EUC_JP, EUC_CN and MULE_INTERNAL
* add an English document (this file)
* fix problems concerning 8-bit single byte characters
Mar 1, 1998 PL1 released
WIN1250 on Windows/ODBC
The WIN1250 character set on Windows client platforms can be used
with Postgres with locale support
enabled.
The following should be kept in mind:
Success depends on proper system locales. This has been tested
with Red Hat 6.0 and Slackware 3.6, with cs_CZ.iso8859-2 locale.
Never try to set the server multibyte database encoding to WIN1250.
Always use LATIN2 instead since there is not a WIN1250 locale
in Unix.
WIN1250 encoding is usable only for Windows ODBC clients. The
characters are recoded on the fly, to be displayed and stored
back properly.
When running, it is important to remember the following:
This configuration reorders your sort order depending on your
LC_x settings. Don't be
confused with the regression test results since they don't use
locale.
A locale such as "ch" is correctly sorted
only if your system
supports that locale; older systems may not do so but new ones
(e.g. RH6.0) do.
You have to insert money as '162,50' (note
comma within the single-quotes).
At the time of writing (early 1999), this configuration has
not received extensive testing. Please let us know of any
changes you had to make!
WIN1250 on Windows/ODBC
Compile Postgres with locale enabled
and the multibyte encoding set to LATIN2.
Set up your installation. Do not forget to create locale
variables in your profile (environment). For example (this may
not be correct for your environment):
LC_ALL=cs_CZ.ISO8859-2
LC_COLLATE=cs_CZ.ISO8859-2
LC_CTYPE=cs_CZ.ISO8859-2
LC_MONETARY=cs_CZ.ISO8859-2
LC_NUMERIC=cs_CZ.ISO8859-2
LC_TIME=cs_CZ.ISO8859-2
You have to start the postmaster with locales set!
Try it with Czech language, and have it sort on a query.
Install ODBC driver for PostgreSQL on your Windows machine.
Setup properly your data source. Include this line in your ODBC
configuration dialog in the field Connect Settings:
SET CLIENT_ENCODING = 'WIN1250';
Now try it again, but in Windows with ODBC.
Single-byte character set recoding</>
<!-- formerly in README.charsets, by Josef Balatka, <balatka@email.cz> -->
<para>
You can set up this feature with the <option>--enable-recode</> option
to <filename>configure</>. This option was formerly described as
<quote>Cyrillic recode support</> which doesn't express all its
power. It can be used for <emphasis>any</> single-byte character
set recoding.
</para>
<para>
This method uses a file <filename>charset.conf</> file located in
the database directory (<envar>PGDATA</>). It's a typical
configuration text file where spaces and newlines separate items
and records and # specifies comments. Three keywords with the
following syntax are recognized here:
<synopsis>
BaseCharset <replaceable>server_charset</>
RecodeTable <replaceable>from_charset</> <replaceable>to_charset</> <replaceable>file_name</>
HostCharset <replaceable>host_spec</> <replaceable>host_charset</>
</synopsis>
</para>
<para>
<token>BaseCharset</> defines the encoding of the database server.
All character set names are only used for mapping inside of
<filename>charset.conf</> so you can freely use typing-friendly
names.
</para>
<para>
<token>RecodeTable</> records specify translation tables between
server and client. The file name is relative to the
<envar>PGDATA</> directory. The table file format is very
simple. There are no keywords and characters are represented by a
pair of decimal or hexadecimal (0x prefixed) values on single
lines:
<synopsis>
<replaceable>char_value</> <replaceable>translated_char_value</>
</synopsis>
</para>
<para>
<token>HostCharset</> records define the client character set by IP
address. You can use a single IP address, an IP mask range starting
from the given address or an IP interval (e.g., 127.0.0.1,
192.168.1.100/24, 192.168.1.20-192.168.1.40).
</para>
<para>
The <filename>charset.conf</> file is always processed up to the
end, so you can easily specify exceptions from the previous
rules. In the <filename>src/data/</> directory you will find an
example <filename>charset.conf</> and a few recoding tables.
</para>
<para>
As this solution is based on the client's IP address and character
set mapping there are obviously some restrictions as well. You
cannot use different encodings on the same host at the same
time. It is also inconvenient when you boot your client hosts into
multiple operating systems. Nevertheless, when these restrictions are
not limiting and you do not need multibyte characters than it is a
simple and effective solution.
</para>
</sect1>
</chapter>
<!-- Keep this comment at the end of the file
Local variables:
mode:sgml
sgml-omittag:nil
sgml-shorttag:t
sgml-minimize-attributes:nil
sgml-always-quote-attributes:t
sgml-indent-step:1
sgml-indent-data:t
sgml-parent-document:nil
sgml-default-dtd-file:"./reference.ced"
sgml-exposed-tags:nil
sgml-local-catalogs:("/usr/lib/sgml/catalog")
sgml-local-ecat-files:nil
End:
-->