2009-08-18 12:34:39 +02:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
*
|
|
|
|
* unaccent.c
|
|
|
|
* Text search unaccent dictionary
|
|
|
|
*
|
2024-01-04 02:49:05 +01:00
|
|
|
* Copyright (c) 2009-2024, PostgreSQL Global Development Group
|
2009-08-18 12:34:39 +02:00
|
|
|
*
|
|
|
|
* IDENTIFICATION
|
2010-09-20 22:08:53 +02:00
|
|
|
* contrib/unaccent/unaccent.c
|
2009-08-18 12:34:39 +02:00
|
|
|
*
|
|
|
|
*-------------------------------------------------------------------------
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include "postgres.h"
|
|
|
|
|
|
|
|
#include "catalog/namespace.h"
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
#include "catalog/pg_ts_dict.h"
|
2009-08-18 12:34:39 +02:00
|
|
|
#include "commands/defrem.h"
|
2014-07-01 17:22:43 +02:00
|
|
|
#include "lib/stringinfo.h"
|
2009-08-18 12:34:39 +02:00
|
|
|
#include "tsearch/ts_cache.h"
|
|
|
|
#include "tsearch/ts_locale.h"
|
|
|
|
#include "tsearch/ts_public.h"
|
|
|
|
#include "utils/builtins.h"
|
2018-09-06 16:49:45 +02:00
|
|
|
#include "utils/lsyscache.h"
|
2017-01-21 02:29:53 +01:00
|
|
|
#include "utils/regproc.h"
|
2018-09-06 16:49:45 +02:00
|
|
|
#include "utils/syscache.h"
|
2009-08-18 12:34:39 +02:00
|
|
|
|
|
|
|
PG_MODULE_MAGIC;
|
|
|
|
|
|
|
|
/*
|
2014-07-01 03:46:29 +02:00
|
|
|
* An unaccent dictionary uses a trie to find a string to replace. Each node
|
|
|
|
* of the trie is an array of 256 TrieChar structs; the N-th element of the
|
|
|
|
* array corresponds to next byte value N. That element can contain both a
|
|
|
|
* replacement string (to be used if the source string ends with this byte)
|
|
|
|
* and a link to another trie node (to be followed if there are more bytes).
|
|
|
|
*
|
|
|
|
* Note that the trie search logic pays no attention to multibyte character
|
|
|
|
* boundaries. This is OK as long as both the data entered into the trie and
|
|
|
|
* the data we're trying to look up are validly encoded; no partial-character
|
|
|
|
* matches will occur.
|
2009-08-18 12:34:39 +02:00
|
|
|
*/
|
2013-05-08 19:57:42 +02:00
|
|
|
typedef struct TrieChar
|
2009-08-18 12:34:39 +02:00
|
|
|
{
|
2013-05-08 19:57:42 +02:00
|
|
|
struct TrieChar *nextChar;
|
2009-08-18 12:34:39 +02:00
|
|
|
char *replaceTo;
|
|
|
|
int replacelen;
|
2013-05-08 19:57:42 +02:00
|
|
|
} TrieChar;
|
2009-08-18 12:34:39 +02:00
|
|
|
|
|
|
|
/*
|
2013-05-08 19:57:42 +02:00
|
|
|
* placeChar - put str into trie's structure, byte by byte.
|
2014-07-01 03:46:29 +02:00
|
|
|
*
|
|
|
|
* If node is NULL, we need to make a new node, which will be returned;
|
|
|
|
* otherwise the return value is the same as node.
|
2009-08-18 12:34:39 +02:00
|
|
|
*/
|
2013-05-08 19:57:42 +02:00
|
|
|
static TrieChar *
|
2014-07-01 03:46:29 +02:00
|
|
|
placeChar(TrieChar *node, const unsigned char *str, int lenstr,
|
|
|
|
const char *replaceTo, int replacelen)
|
2009-08-18 12:34:39 +02:00
|
|
|
{
|
2013-05-08 19:57:42 +02:00
|
|
|
TrieChar *curnode;
|
2009-08-18 12:34:39 +02:00
|
|
|
|
|
|
|
if (!node)
|
2014-07-01 03:46:29 +02:00
|
|
|
node = (TrieChar *) palloc0(sizeof(TrieChar) * 256);
|
|
|
|
|
|
|
|
Assert(lenstr > 0); /* else str[0] doesn't exist */
|
2009-08-18 12:34:39 +02:00
|
|
|
|
|
|
|
curnode = node + *str;
|
|
|
|
|
2014-07-01 03:46:29 +02:00
|
|
|
if (lenstr <= 1)
|
2009-08-18 12:34:39 +02:00
|
|
|
{
|
|
|
|
if (curnode->replaceTo)
|
2014-07-01 04:03:37 +02:00
|
|
|
ereport(WARNING,
|
|
|
|
(errcode(ERRCODE_CONFIG_FILE_ERROR),
|
|
|
|
errmsg("duplicate source strings, first one will be used")));
|
2009-08-18 12:34:39 +02:00
|
|
|
else
|
|
|
|
{
|
|
|
|
curnode->replacelen = replacelen;
|
2014-07-01 03:46:29 +02:00
|
|
|
curnode->replaceTo = (char *) palloc(replacelen);
|
2009-08-18 12:34:39 +02:00
|
|
|
memcpy(curnode->replaceTo, replaceTo, replacelen);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
2014-07-01 03:46:29 +02:00
|
|
|
curnode->nextChar = placeChar(curnode->nextChar, str + 1, lenstr - 1,
|
|
|
|
replaceTo, replacelen);
|
2009-08-18 12:34:39 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
return node;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2013-05-08 19:57:42 +02:00
|
|
|
* initTrie - create trie from file.
|
|
|
|
*
|
|
|
|
* Function converts UTF8-encoded file into current encoding.
|
2009-08-18 12:34:39 +02:00
|
|
|
*/
|
2013-05-08 19:57:42 +02:00
|
|
|
static TrieChar *
|
2017-10-31 15:34:31 +01:00
|
|
|
initTrie(const char *filename)
|
2009-08-18 12:34:39 +02:00
|
|
|
{
|
2013-05-08 19:57:42 +02:00
|
|
|
TrieChar *volatile rootTrie = NULL;
|
2009-08-18 12:34:39 +02:00
|
|
|
MemoryContext ccxt = CurrentMemoryContext;
|
|
|
|
tsearch_readline_state trst;
|
2009-08-18 17:57:26 +02:00
|
|
|
volatile bool skip;
|
2009-08-18 12:34:39 +02:00
|
|
|
|
|
|
|
filename = get_tsearch_config_filename(filename, "rules");
|
|
|
|
if (!tsearch_readline_begin(&trst, filename))
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_CONFIG_FILE_ERROR),
|
|
|
|
errmsg("could not open unaccent file \"%s\": %m",
|
|
|
|
filename)));
|
|
|
|
|
|
|
|
do
|
|
|
|
{
|
2011-11-07 17:48:53 +01:00
|
|
|
/*
|
|
|
|
* pg_do_encoding_conversion() (called by tsearch_readline()) will
|
|
|
|
* emit exception if it finds untranslatable characters in current
|
|
|
|
* locale. We just skip such lines, continuing with the next.
|
|
|
|
*/
|
2009-08-18 12:34:39 +02:00
|
|
|
skip = true;
|
|
|
|
|
|
|
|
PG_TRY();
|
|
|
|
{
|
2011-11-07 17:48:53 +01:00
|
|
|
char *line;
|
|
|
|
|
2009-08-18 12:34:39 +02:00
|
|
|
while ((line = tsearch_readline(&trst)) != NULL)
|
|
|
|
{
|
2014-07-01 02:51:26 +02:00
|
|
|
/*----------
|
|
|
|
* The format of each line must be "src" or "src trg", where
|
|
|
|
* src and trg are sequences of one or more non-whitespace
|
|
|
|
* characters, separated by whitespace. Whitespace at start
|
|
|
|
* or end of line is ignored. If trg is omitted, an empty
|
unaccent: Add support for quoted translated characters
As reported in bug #18057, the extension unaccent removes in its rule
file whitespace characters that are intentionally specified when
building unaccent.rules from UnicodeData.txt, causing an incorrect
translation for some characters like numeric symbols. This is caused by
the fact that all whitespaces before and after the origin and target
characters are all discarded (this limitation is documented).
This commit makes possible the use of quotes around target characters,
so as whitespaces can be considered part of target characters. Some
target characters use a double quote, these require an extra double
quote.
The documentation is updated to show how to use quoted areas,
generate_unaccent_rules.py is updated to generate unaccent.rules and a
couple of tests are added for numeric symbols. While working on this
patch, I have implemented a fake rule file to test the parsing logic
implemented, which is not included here as it would just consume extra
cycles in the tests, and it requires the manipulation of an installation
tree to be able to work correctly.
As this requires a change of format in unaccent.rules, this cannot be
backpatched, unfortunately. The idea to use double quotes as escaped
characters comes from Tom Lane.
Reported-by: Martin Schlossarek
Author: Michael Paquier
Discussion: https://postgr.es/m/18057-62712cad01bd202c@postgresql.org
2023-09-20 05:29:36 +02:00
|
|
|
* string is used as the replacement. trg can be optionally
|
|
|
|
* quoted, in which case whitespaces are included in it.
|
2014-07-01 02:51:26 +02:00
|
|
|
*
|
|
|
|
* We use a simple state machine, with states
|
|
|
|
* 0 initial (before src)
|
|
|
|
* 1 in src
|
|
|
|
* 2 in whitespace after src
|
unaccent: Add support for quoted translated characters
As reported in bug #18057, the extension unaccent removes in its rule
file whitespace characters that are intentionally specified when
building unaccent.rules from UnicodeData.txt, causing an incorrect
translation for some characters like numeric symbols. This is caused by
the fact that all whitespaces before and after the origin and target
characters are all discarded (this limitation is documented).
This commit makes possible the use of quotes around target characters,
so as whitespaces can be considered part of target characters. Some
target characters use a double quote, these require an extra double
quote.
The documentation is updated to show how to use quoted areas,
generate_unaccent_rules.py is updated to generate unaccent.rules and a
couple of tests are added for numeric symbols. While working on this
patch, I have implemented a fake rule file to test the parsing logic
implemented, which is not included here as it would just consume extra
cycles in the tests, and it requires the manipulation of an installation
tree to be able to work correctly.
As this requires a change of format in unaccent.rules, this cannot be
backpatched, unfortunately. The idea to use double quotes as escaped
characters comes from Tom Lane.
Reported-by: Martin Schlossarek
Author: Michael Paquier
Discussion: https://postgr.es/m/18057-62712cad01bd202c@postgresql.org
2023-09-20 05:29:36 +02:00
|
|
|
* 3 in trg (non-quoted)
|
|
|
|
* 4 in trg (quoted)
|
|
|
|
* 5 in whitespace after trg
|
|
|
|
* -1 syntax error detected (two strings)
|
|
|
|
* -2 syntax error detected (unfinished quoted string)
|
2014-07-01 02:51:26 +02:00
|
|
|
*----------
|
2011-11-07 17:48:53 +01:00
|
|
|
*/
|
|
|
|
int state;
|
|
|
|
char *ptr;
|
|
|
|
char *src = NULL;
|
|
|
|
char *trg = NULL;
|
unaccent: Add support for quoted translated characters
As reported in bug #18057, the extension unaccent removes in its rule
file whitespace characters that are intentionally specified when
building unaccent.rules from UnicodeData.txt, causing an incorrect
translation for some characters like numeric symbols. This is caused by
the fact that all whitespaces before and after the origin and target
characters are all discarded (this limitation is documented).
This commit makes possible the use of quotes around target characters,
so as whitespaces can be considered part of target characters. Some
target characters use a double quote, these require an extra double
quote.
The documentation is updated to show how to use quoted areas,
generate_unaccent_rules.py is updated to generate unaccent.rules and a
couple of tests are added for numeric symbols. While working on this
patch, I have implemented a fake rule file to test the parsing logic
implemented, which is not included here as it would just consume extra
cycles in the tests, and it requires the manipulation of an installation
tree to be able to work correctly.
As this requires a change of format in unaccent.rules, this cannot be
backpatched, unfortunately. The idea to use double quotes as escaped
characters comes from Tom Lane.
Reported-by: Martin Schlossarek
Author: Michael Paquier
Discussion: https://postgr.es/m/18057-62712cad01bd202c@postgresql.org
2023-09-20 05:29:36 +02:00
|
|
|
char *trgstore = NULL;
|
2011-11-07 17:48:53 +01:00
|
|
|
int ptrlen;
|
|
|
|
int srclen = 0;
|
|
|
|
int trglen = 0;
|
unaccent: Add support for quoted translated characters
As reported in bug #18057, the extension unaccent removes in its rule
file whitespace characters that are intentionally specified when
building unaccent.rules from UnicodeData.txt, causing an incorrect
translation for some characters like numeric symbols. This is caused by
the fact that all whitespaces before and after the origin and target
characters are all discarded (this limitation is documented).
This commit makes possible the use of quotes around target characters,
so as whitespaces can be considered part of target characters. Some
target characters use a double quote, these require an extra double
quote.
The documentation is updated to show how to use quoted areas,
generate_unaccent_rules.py is updated to generate unaccent.rules and a
couple of tests are added for numeric symbols. While working on this
patch, I have implemented a fake rule file to test the parsing logic
implemented, which is not included here as it would just consume extra
cycles in the tests, and it requires the manipulation of an installation
tree to be able to work correctly.
As this requires a change of format in unaccent.rules, this cannot be
backpatched, unfortunately. The idea to use double quotes as escaped
characters comes from Tom Lane.
Reported-by: Martin Schlossarek
Author: Michael Paquier
Discussion: https://postgr.es/m/18057-62712cad01bd202c@postgresql.org
2023-09-20 05:29:36 +02:00
|
|
|
int trgstorelen = 0;
|
|
|
|
bool trgquoted = false;
|
2011-11-07 17:48:53 +01:00
|
|
|
|
|
|
|
state = 0;
|
|
|
|
for (ptr = line; *ptr; ptr += ptrlen)
|
|
|
|
{
|
|
|
|
ptrlen = pg_mblen(ptr);
|
|
|
|
/* ignore whitespace, but end src or trg */
|
|
|
|
if (t_isspace(ptr))
|
|
|
|
{
|
|
|
|
if (state == 1)
|
|
|
|
state = 2;
|
|
|
|
else if (state == 3)
|
unaccent: Add support for quoted translated characters
As reported in bug #18057, the extension unaccent removes in its rule
file whitespace characters that are intentionally specified when
building unaccent.rules from UnicodeData.txt, causing an incorrect
translation for some characters like numeric symbols. This is caused by
the fact that all whitespaces before and after the origin and target
characters are all discarded (this limitation is documented).
This commit makes possible the use of quotes around target characters,
so as whitespaces can be considered part of target characters. Some
target characters use a double quote, these require an extra double
quote.
The documentation is updated to show how to use quoted areas,
generate_unaccent_rules.py is updated to generate unaccent.rules and a
couple of tests are added for numeric symbols. While working on this
patch, I have implemented a fake rule file to test the parsing logic
implemented, which is not included here as it would just consume extra
cycles in the tests, and it requires the manipulation of an installation
tree to be able to work correctly.
As this requires a change of format in unaccent.rules, this cannot be
backpatched, unfortunately. The idea to use double quotes as escaped
characters comes from Tom Lane.
Reported-by: Martin Schlossarek
Author: Michael Paquier
Discussion: https://postgr.es/m/18057-62712cad01bd202c@postgresql.org
2023-09-20 05:29:36 +02:00
|
|
|
state = 5;
|
|
|
|
/* whitespaces are OK in quoted area */
|
|
|
|
if (state != 4)
|
|
|
|
continue;
|
2011-11-07 17:48:53 +01:00
|
|
|
}
|
|
|
|
switch (state)
|
|
|
|
{
|
|
|
|
case 0:
|
|
|
|
/* start of src */
|
|
|
|
src = ptr;
|
|
|
|
srclen = ptrlen;
|
|
|
|
state = 1;
|
|
|
|
break;
|
|
|
|
case 1:
|
|
|
|
/* continue src */
|
|
|
|
srclen += ptrlen;
|
|
|
|
break;
|
|
|
|
case 2:
|
|
|
|
/* start of trg */
|
unaccent: Add support for quoted translated characters
As reported in bug #18057, the extension unaccent removes in its rule
file whitespace characters that are intentionally specified when
building unaccent.rules from UnicodeData.txt, causing an incorrect
translation for some characters like numeric symbols. This is caused by
the fact that all whitespaces before and after the origin and target
characters are all discarded (this limitation is documented).
This commit makes possible the use of quotes around target characters,
so as whitespaces can be considered part of target characters. Some
target characters use a double quote, these require an extra double
quote.
The documentation is updated to show how to use quoted areas,
generate_unaccent_rules.py is updated to generate unaccent.rules and a
couple of tests are added for numeric symbols. While working on this
patch, I have implemented a fake rule file to test the parsing logic
implemented, which is not included here as it would just consume extra
cycles in the tests, and it requires the manipulation of an installation
tree to be able to work correctly.
As this requires a change of format in unaccent.rules, this cannot be
backpatched, unfortunately. The idea to use double quotes as escaped
characters comes from Tom Lane.
Reported-by: Martin Schlossarek
Author: Michael Paquier
Discussion: https://postgr.es/m/18057-62712cad01bd202c@postgresql.org
2023-09-20 05:29:36 +02:00
|
|
|
if (*ptr == '"')
|
|
|
|
{
|
|
|
|
trgquoted = true;
|
|
|
|
state = 4;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
state = 3;
|
|
|
|
|
2011-11-07 17:48:53 +01:00
|
|
|
trg = ptr;
|
|
|
|
trglen = ptrlen;
|
|
|
|
break;
|
|
|
|
case 3:
|
unaccent: Add support for quoted translated characters
As reported in bug #18057, the extension unaccent removes in its rule
file whitespace characters that are intentionally specified when
building unaccent.rules from UnicodeData.txt, causing an incorrect
translation for some characters like numeric symbols. This is caused by
the fact that all whitespaces before and after the origin and target
characters are all discarded (this limitation is documented).
This commit makes possible the use of quotes around target characters,
so as whitespaces can be considered part of target characters. Some
target characters use a double quote, these require an extra double
quote.
The documentation is updated to show how to use quoted areas,
generate_unaccent_rules.py is updated to generate unaccent.rules and a
couple of tests are added for numeric symbols. While working on this
patch, I have implemented a fake rule file to test the parsing logic
implemented, which is not included here as it would just consume extra
cycles in the tests, and it requires the manipulation of an installation
tree to be able to work correctly.
As this requires a change of format in unaccent.rules, this cannot be
backpatched, unfortunately. The idea to use double quotes as escaped
characters comes from Tom Lane.
Reported-by: Martin Schlossarek
Author: Michael Paquier
Discussion: https://postgr.es/m/18057-62712cad01bd202c@postgresql.org
2023-09-20 05:29:36 +02:00
|
|
|
/* continue non-quoted trg */
|
|
|
|
trglen += ptrlen;
|
|
|
|
break;
|
|
|
|
case 4:
|
|
|
|
/* continue quoted trg */
|
2011-11-07 17:48:53 +01:00
|
|
|
trglen += ptrlen;
|
unaccent: Add support for quoted translated characters
As reported in bug #18057, the extension unaccent removes in its rule
file whitespace characters that are intentionally specified when
building unaccent.rules from UnicodeData.txt, causing an incorrect
translation for some characters like numeric symbols. This is caused by
the fact that all whitespaces before and after the origin and target
characters are all discarded (this limitation is documented).
This commit makes possible the use of quotes around target characters,
so as whitespaces can be considered part of target characters. Some
target characters use a double quote, these require an extra double
quote.
The documentation is updated to show how to use quoted areas,
generate_unaccent_rules.py is updated to generate unaccent.rules and a
couple of tests are added for numeric symbols. While working on this
patch, I have implemented a fake rule file to test the parsing logic
implemented, which is not included here as it would just consume extra
cycles in the tests, and it requires the manipulation of an installation
tree to be able to work correctly.
As this requires a change of format in unaccent.rules, this cannot be
backpatched, unfortunately. The idea to use double quotes as escaped
characters comes from Tom Lane.
Reported-by: Martin Schlossarek
Author: Michael Paquier
Discussion: https://postgr.es/m/18057-62712cad01bd202c@postgresql.org
2023-09-20 05:29:36 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If this is a quote, consider it as the end of
|
|
|
|
* trg except if the follow-up character is itself
|
|
|
|
* a quote.
|
|
|
|
*/
|
|
|
|
if (*ptr == '"')
|
|
|
|
{
|
|
|
|
if (*(ptr + 1) == '"')
|
|
|
|
{
|
|
|
|
ptr++;
|
|
|
|
trglen += 1;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
state = 5;
|
|
|
|
}
|
2011-11-07 17:48:53 +01:00
|
|
|
break;
|
|
|
|
default:
|
|
|
|
/* bogus line format */
|
|
|
|
state = -1;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2014-07-01 02:51:26 +02:00
|
|
|
if (state == 1 || state == 2)
|
|
|
|
{
|
|
|
|
/* trg was omitted, so use "" */
|
|
|
|
trg = "";
|
|
|
|
trglen = 0;
|
|
|
|
}
|
|
|
|
|
unaccent: Add support for quoted translated characters
As reported in bug #18057, the extension unaccent removes in its rule
file whitespace characters that are intentionally specified when
building unaccent.rules from UnicodeData.txt, causing an incorrect
translation for some characters like numeric symbols. This is caused by
the fact that all whitespaces before and after the origin and target
characters are all discarded (this limitation is documented).
This commit makes possible the use of quotes around target characters,
so as whitespaces can be considered part of target characters. Some
target characters use a double quote, these require an extra double
quote.
The documentation is updated to show how to use quoted areas,
generate_unaccent_rules.py is updated to generate unaccent.rules and a
couple of tests are added for numeric symbols. While working on this
patch, I have implemented a fake rule file to test the parsing logic
implemented, which is not included here as it would just consume extra
cycles in the tests, and it requires the manipulation of an installation
tree to be able to work correctly.
As this requires a change of format in unaccent.rules, this cannot be
backpatched, unfortunately. The idea to use double quotes as escaped
characters comes from Tom Lane.
Reported-by: Martin Schlossarek
Author: Michael Paquier
Discussion: https://postgr.es/m/18057-62712cad01bd202c@postgresql.org
2023-09-20 05:29:36 +02:00
|
|
|
/* If still in a quoted area, fallback to an error */
|
|
|
|
if (state == 4)
|
|
|
|
state = -2;
|
|
|
|
|
|
|
|
/* If trg was quoted, remove its quotes and unescape it */
|
|
|
|
if (trgquoted && state > 0)
|
|
|
|
{
|
|
|
|
/* Ignore first and end quotes */
|
2023-09-25 02:31:48 +02:00
|
|
|
trgstore = (char *) palloc(sizeof(char) * (trglen - 2));
|
unaccent: Add support for quoted translated characters
As reported in bug #18057, the extension unaccent removes in its rule
file whitespace characters that are intentionally specified when
building unaccent.rules from UnicodeData.txt, causing an incorrect
translation for some characters like numeric symbols. This is caused by
the fact that all whitespaces before and after the origin and target
characters are all discarded (this limitation is documented).
This commit makes possible the use of quotes around target characters,
so as whitespaces can be considered part of target characters. Some
target characters use a double quote, these require an extra double
quote.
The documentation is updated to show how to use quoted areas,
generate_unaccent_rules.py is updated to generate unaccent.rules and a
couple of tests are added for numeric symbols. While working on this
patch, I have implemented a fake rule file to test the parsing logic
implemented, which is not included here as it would just consume extra
cycles in the tests, and it requires the manipulation of an installation
tree to be able to work correctly.
As this requires a change of format in unaccent.rules, this cannot be
backpatched, unfortunately. The idea to use double quotes as escaped
characters comes from Tom Lane.
Reported-by: Martin Schlossarek
Author: Michael Paquier
Discussion: https://postgr.es/m/18057-62712cad01bd202c@postgresql.org
2023-09-20 05:29:36 +02:00
|
|
|
trgstorelen = 0;
|
|
|
|
for (int i = 1; i < trglen - 1; i++)
|
|
|
|
{
|
|
|
|
trgstore[trgstorelen] = trg[i];
|
|
|
|
trgstorelen++;
|
|
|
|
/* skip second double quotes */
|
|
|
|
if (trg[i] == '"' && trg[i + 1] == '"')
|
|
|
|
i++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
2023-09-25 02:31:48 +02:00
|
|
|
trgstore = (char *) palloc(sizeof(char) * trglen);
|
unaccent: Add support for quoted translated characters
As reported in bug #18057, the extension unaccent removes in its rule
file whitespace characters that are intentionally specified when
building unaccent.rules from UnicodeData.txt, causing an incorrect
translation for some characters like numeric symbols. This is caused by
the fact that all whitespaces before and after the origin and target
characters are all discarded (this limitation is documented).
This commit makes possible the use of quotes around target characters,
so as whitespaces can be considered part of target characters. Some
target characters use a double quote, these require an extra double
quote.
The documentation is updated to show how to use quoted areas,
generate_unaccent_rules.py is updated to generate unaccent.rules and a
couple of tests are added for numeric symbols. While working on this
patch, I have implemented a fake rule file to test the parsing logic
implemented, which is not included here as it would just consume extra
cycles in the tests, and it requires the manipulation of an installation
tree to be able to work correctly.
As this requires a change of format in unaccent.rules, this cannot be
backpatched, unfortunately. The idea to use double quotes as escaped
characters comes from Tom Lane.
Reported-by: Martin Schlossarek
Author: Michael Paquier
Discussion: https://postgr.es/m/18057-62712cad01bd202c@postgresql.org
2023-09-20 05:29:36 +02:00
|
|
|
trgstorelen = trglen;
|
|
|
|
memcpy(trgstore, trg, trgstorelen);
|
|
|
|
}
|
|
|
|
|
2014-07-01 02:51:26 +02:00
|
|
|
if (state > 0)
|
2013-05-08 19:57:42 +02:00
|
|
|
rootTrie = placeChar(rootTrie,
|
2011-11-07 17:48:53 +01:00
|
|
|
(unsigned char *) src, srclen,
|
unaccent: Add support for quoted translated characters
As reported in bug #18057, the extension unaccent removes in its rule
file whitespace characters that are intentionally specified when
building unaccent.rules from UnicodeData.txt, causing an incorrect
translation for some characters like numeric symbols. This is caused by
the fact that all whitespaces before and after the origin and target
characters are all discarded (this limitation is documented).
This commit makes possible the use of quotes around target characters,
so as whitespaces can be considered part of target characters. Some
target characters use a double quote, these require an extra double
quote.
The documentation is updated to show how to use quoted areas,
generate_unaccent_rules.py is updated to generate unaccent.rules and a
couple of tests are added for numeric symbols. While working on this
patch, I have implemented a fake rule file to test the parsing logic
implemented, which is not included here as it would just consume extra
cycles in the tests, and it requires the manipulation of an installation
tree to be able to work correctly.
As this requires a change of format in unaccent.rules, this cannot be
backpatched, unfortunately. The idea to use double quotes as escaped
characters comes from Tom Lane.
Reported-by: Martin Schlossarek
Author: Michael Paquier
Discussion: https://postgr.es/m/18057-62712cad01bd202c@postgresql.org
2023-09-20 05:29:36 +02:00
|
|
|
trgstore, trgstorelen);
|
|
|
|
else if (state == -1)
|
2014-07-01 04:03:37 +02:00
|
|
|
ereport(WARNING,
|
|
|
|
(errcode(ERRCODE_CONFIG_FILE_ERROR),
|
|
|
|
errmsg("invalid syntax: more than two strings in unaccent rule")));
|
unaccent: Add support for quoted translated characters
As reported in bug #18057, the extension unaccent removes in its rule
file whitespace characters that are intentionally specified when
building unaccent.rules from UnicodeData.txt, causing an incorrect
translation for some characters like numeric symbols. This is caused by
the fact that all whitespaces before and after the origin and target
characters are all discarded (this limitation is documented).
This commit makes possible the use of quotes around target characters,
so as whitespaces can be considered part of target characters. Some
target characters use a double quote, these require an extra double
quote.
The documentation is updated to show how to use quoted areas,
generate_unaccent_rules.py is updated to generate unaccent.rules and a
couple of tests are added for numeric symbols. While working on this
patch, I have implemented a fake rule file to test the parsing logic
implemented, which is not included here as it would just consume extra
cycles in the tests, and it requires the manipulation of an installation
tree to be able to work correctly.
As this requires a change of format in unaccent.rules, this cannot be
backpatched, unfortunately. The idea to use double quotes as escaped
characters comes from Tom Lane.
Reported-by: Martin Schlossarek
Author: Michael Paquier
Discussion: https://postgr.es/m/18057-62712cad01bd202c@postgresql.org
2023-09-20 05:29:36 +02:00
|
|
|
else if (state == -2)
|
|
|
|
ereport(WARNING,
|
|
|
|
(errcode(ERRCODE_CONFIG_FILE_ERROR),
|
|
|
|
errmsg("invalid syntax: unfinished quoted string in unaccent rule")));
|
2009-08-18 12:34:39 +02:00
|
|
|
|
unaccent: Add support for quoted translated characters
As reported in bug #18057, the extension unaccent removes in its rule
file whitespace characters that are intentionally specified when
building unaccent.rules from UnicodeData.txt, causing an incorrect
translation for some characters like numeric symbols. This is caused by
the fact that all whitespaces before and after the origin and target
characters are all discarded (this limitation is documented).
This commit makes possible the use of quotes around target characters,
so as whitespaces can be considered part of target characters. Some
target characters use a double quote, these require an extra double
quote.
The documentation is updated to show how to use quoted areas,
generate_unaccent_rules.py is updated to generate unaccent.rules and a
couple of tests are added for numeric symbols. While working on this
patch, I have implemented a fake rule file to test the parsing logic
implemented, which is not included here as it would just consume extra
cycles in the tests, and it requires the manipulation of an installation
tree to be able to work correctly.
As this requires a change of format in unaccent.rules, this cannot be
backpatched, unfortunately. The idea to use double quotes as escaped
characters comes from Tom Lane.
Reported-by: Martin Schlossarek
Author: Michael Paquier
Discussion: https://postgr.es/m/18057-62712cad01bd202c@postgresql.org
2023-09-20 05:29:36 +02:00
|
|
|
pfree(trgstore);
|
2009-08-18 12:34:39 +02:00
|
|
|
pfree(line);
|
|
|
|
}
|
2011-11-07 17:48:53 +01:00
|
|
|
skip = false;
|
2009-08-18 12:34:39 +02:00
|
|
|
}
|
|
|
|
PG_CATCH();
|
|
|
|
{
|
|
|
|
ErrorData *errdata;
|
|
|
|
MemoryContext ecxt;
|
|
|
|
|
|
|
|
ecxt = MemoryContextSwitchTo(ccxt);
|
|
|
|
errdata = CopyErrorData();
|
|
|
|
if (errdata->sqlerrcode == ERRCODE_UNTRANSLATABLE_CHARACTER)
|
|
|
|
{
|
|
|
|
FlushErrorState();
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
MemoryContextSwitchTo(ecxt);
|
|
|
|
PG_RE_THROW();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
PG_END_TRY();
|
|
|
|
}
|
|
|
|
while (skip);
|
|
|
|
|
|
|
|
tsearch_readline_end(&trst);
|
|
|
|
|
2013-05-08 19:57:42 +02:00
|
|
|
return rootTrie;
|
2009-08-18 12:34:39 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2014-07-01 03:46:29 +02:00
|
|
|
* findReplaceTo - find longest possible match in trie
|
|
|
|
*
|
|
|
|
* On success, returns pointer to ending subnode, plus length of matched
|
|
|
|
* source string in *p_matchlen. On failure, returns NULL.
|
2009-08-18 12:34:39 +02:00
|
|
|
*/
|
2013-05-08 19:57:42 +02:00
|
|
|
static TrieChar *
|
2014-07-01 03:46:29 +02:00
|
|
|
findReplaceTo(TrieChar *node, const unsigned char *src, int srclen,
|
|
|
|
int *p_matchlen)
|
2009-08-18 12:34:39 +02:00
|
|
|
{
|
2014-07-01 03:46:29 +02:00
|
|
|
TrieChar *result = NULL;
|
|
|
|
int matchlen = 0;
|
|
|
|
|
|
|
|
*p_matchlen = 0; /* prevent uninitialized-variable warnings */
|
|
|
|
|
|
|
|
while (node && matchlen < srclen)
|
2009-08-18 12:34:39 +02:00
|
|
|
{
|
2014-07-01 03:46:29 +02:00
|
|
|
node = node + src[matchlen];
|
|
|
|
matchlen++;
|
|
|
|
|
|
|
|
if (node->replaceTo)
|
|
|
|
{
|
|
|
|
result = node;
|
|
|
|
*p_matchlen = matchlen;
|
|
|
|
}
|
2009-08-18 12:34:39 +02:00
|
|
|
|
|
|
|
node = node->nextChar;
|
|
|
|
}
|
|
|
|
|
2014-07-01 03:46:29 +02:00
|
|
|
return result;
|
2009-08-18 12:34:39 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
PG_FUNCTION_INFO_V1(unaccent_init);
|
|
|
|
Datum
|
|
|
|
unaccent_init(PG_FUNCTION_ARGS)
|
|
|
|
{
|
|
|
|
List *dictoptions = (List *) PG_GETARG_POINTER(0);
|
2013-05-08 19:57:42 +02:00
|
|
|
TrieChar *rootTrie = NULL;
|
2009-08-18 12:34:39 +02:00
|
|
|
bool fileloaded = false;
|
|
|
|
ListCell *l;
|
|
|
|
|
|
|
|
foreach(l, dictoptions)
|
|
|
|
{
|
|
|
|
DefElem *defel = (DefElem *) lfirst(l);
|
|
|
|
|
Avoid unnecessary use of pg_strcasecmp for already-downcased identifiers.
We have a lot of code in which option names, which from the user's
viewpoint are logically keywords, are passed through the grammar as plain
identifiers, and then matched to string literals during command execution.
This approach avoids making words into lexer keywords unnecessarily. Some
places matched these strings using plain strcmp, some using pg_strcasecmp.
But the latter should be unnecessary since identifiers would have been
downcased on their way through the parser. Aside from any efficiency
concerns (probably not a big factor), the lack of consistency in this area
creates a hazard of subtle bugs due to different places coming to different
conclusions about whether two option names are the same or different.
Hence, standardize on using strcmp() to match any option names that are
expected to have been fed through the parser.
This does create a user-visible behavioral change, which is that while
formerly all of these would work:
alter table foo set (fillfactor = 50);
alter table foo set (FillFactor = 50);
alter table foo set ("fillfactor" = 50);
alter table foo set ("FillFactor" = 50);
now the last case will fail because that double-quoted identifier is
different from the others. However, none of our documentation says that
you can use a quoted identifier in such contexts at all, and we should
discourage doing so since it would break if we ever decide to parse such
constructs as true lexer keywords rather than poor man's substitutes.
So this shouldn't create a significant compatibility issue for users.
Daniel Gustafsson, reviewed by Michael Paquier, small changes by me
Discussion: https://postgr.es/m/29405B24-564E-476B-98C0-677A29805B84@yesql.se
2018-01-27 00:25:02 +01:00
|
|
|
if (strcmp(defel->defname, "rules") == 0)
|
2009-08-18 12:34:39 +02:00
|
|
|
{
|
|
|
|
if (fileloaded)
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
|
|
|
|
errmsg("multiple Rules parameters")));
|
2013-05-08 19:57:42 +02:00
|
|
|
rootTrie = initTrie(defGetString(defel));
|
2009-08-18 12:34:39 +02:00
|
|
|
fileloaded = true;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
|
|
|
|
errmsg("unrecognized Unaccent parameter: \"%s\"",
|
|
|
|
defel->defname)));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!fileloaded)
|
|
|
|
{
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
|
|
|
|
errmsg("missing Rules parameter")));
|
|
|
|
}
|
|
|
|
|
2013-05-08 19:57:42 +02:00
|
|
|
PG_RETURN_POINTER(rootTrie);
|
2009-08-18 12:34:39 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
PG_FUNCTION_INFO_V1(unaccent_lexize);
|
|
|
|
Datum
|
|
|
|
unaccent_lexize(PG_FUNCTION_ARGS)
|
|
|
|
{
|
2013-05-08 19:57:42 +02:00
|
|
|
TrieChar *rootTrie = (TrieChar *) PG_GETARG_POINTER(0);
|
2009-08-18 12:34:39 +02:00
|
|
|
char *srcchar = (char *) PG_GETARG_POINTER(1);
|
|
|
|
int32 len = PG_GETARG_INT32(2);
|
2014-07-01 17:22:43 +02:00
|
|
|
char *srcstart = srcchar;
|
|
|
|
TSLexeme *res;
|
|
|
|
StringInfoData buf;
|
|
|
|
|
|
|
|
/* we allocate storage for the buffer only if needed */
|
|
|
|
buf.data = NULL;
|
2009-08-18 12:34:39 +02:00
|
|
|
|
2014-07-01 03:46:29 +02:00
|
|
|
while (len > 0)
|
2009-08-18 12:34:39 +02:00
|
|
|
{
|
2014-07-01 03:46:29 +02:00
|
|
|
TrieChar *node;
|
|
|
|
int matchlen;
|
2009-08-18 12:34:39 +02:00
|
|
|
|
2014-07-01 03:46:29 +02:00
|
|
|
node = findReplaceTo(rootTrie, (unsigned char *) srcchar, len,
|
|
|
|
&matchlen);
|
2009-08-18 12:34:39 +02:00
|
|
|
if (node && node->replaceTo)
|
|
|
|
{
|
2014-07-01 17:22:43 +02:00
|
|
|
if (buf.data == NULL)
|
2009-08-18 12:34:39 +02:00
|
|
|
{
|
2014-07-01 17:22:43 +02:00
|
|
|
/* initialize buffer */
|
|
|
|
initStringInfo(&buf);
|
|
|
|
/* insert any data we already skipped over */
|
2009-08-18 12:34:39 +02:00
|
|
|
if (srcchar != srcstart)
|
2014-07-01 17:22:43 +02:00
|
|
|
appendBinaryStringInfo(&buf, srcstart, srcchar - srcstart);
|
2009-08-18 12:34:39 +02:00
|
|
|
}
|
2014-07-01 17:22:43 +02:00
|
|
|
appendBinaryStringInfo(&buf, node->replaceTo, node->replacelen);
|
2009-08-18 12:34:39 +02:00
|
|
|
}
|
2014-07-01 03:46:29 +02:00
|
|
|
else
|
2009-08-18 12:34:39 +02:00
|
|
|
{
|
2014-07-01 03:46:29 +02:00
|
|
|
matchlen = pg_mblen(srcchar);
|
2014-07-01 17:22:43 +02:00
|
|
|
if (buf.data != NULL)
|
|
|
|
appendBinaryStringInfo(&buf, srcchar, matchlen);
|
2009-08-18 12:34:39 +02:00
|
|
|
}
|
|
|
|
|
2014-07-01 03:46:29 +02:00
|
|
|
srcchar += matchlen;
|
|
|
|
len -= matchlen;
|
2009-08-18 12:34:39 +02:00
|
|
|
}
|
|
|
|
|
2014-07-01 17:22:43 +02:00
|
|
|
/* return a result only if we made at least one substitution */
|
|
|
|
if (buf.data != NULL)
|
|
|
|
{
|
|
|
|
res = (TSLexeme *) palloc0(sizeof(TSLexeme) * 2);
|
|
|
|
res->lexeme = buf.data;
|
|
|
|
res->flags = TSL_FILTER;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
res = NULL;
|
2009-08-18 12:34:39 +02:00
|
|
|
|
|
|
|
PG_RETURN_POINTER(res);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Function-like wrapper for dictionary
|
|
|
|
*/
|
|
|
|
PG_FUNCTION_INFO_V1(unaccent_dict);
|
|
|
|
Datum
|
|
|
|
unaccent_dict(PG_FUNCTION_ARGS)
|
|
|
|
{
|
|
|
|
text *str;
|
|
|
|
int strArg;
|
|
|
|
Oid dictOid;
|
|
|
|
TSDictionaryCacheEntry *dict;
|
|
|
|
TSLexeme *res;
|
|
|
|
|
|
|
|
if (PG_NARGS() == 1)
|
|
|
|
{
|
2018-09-06 16:49:45 +02:00
|
|
|
/*
|
|
|
|
* Use the "unaccent" dictionary that is in the same schema that this
|
|
|
|
* function is in.
|
|
|
|
*/
|
|
|
|
Oid procnspid = get_func_namespace(fcinfo->flinfo->fn_oid);
|
|
|
|
const char *dictname = "unaccent";
|
|
|
|
|
Remove WITH OIDS support, change oid catalog column visibility.
Previously tables declared WITH OIDS, including a significant fraction
of the catalog tables, stored the oid column not as a normal column,
but as part of the tuple header.
This special column was not shown by default, which was somewhat odd,
as it's often (consider e.g. pg_class.oid) one of the more important
parts of a row. Neither pg_dump nor COPY included the contents of the
oid column by default.
The fact that the oid column was not an ordinary column necessitated a
significant amount of special case code to support oid columns. That
already was painful for the existing, but upcoming work aiming to make
table storage pluggable, would have required expanding and duplicating
that "specialness" significantly.
WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0).
Remove it.
Removing includes:
- CREATE TABLE and ALTER TABLE syntax for declaring the table to be
WITH OIDS has been removed (WITH (oids[ = true]) will error out)
- pg_dump does not support dumping tables declared WITH OIDS and will
issue a warning when dumping one (and ignore the oid column).
- restoring an pg_dump archive with pg_restore will warn when
restoring a table with oid contents (and ignore the oid column)
- COPY will refuse to load binary dump that includes oids.
- pg_upgrade will error out when encountering tables declared WITH
OIDS, they have to be altered to remove the oid column first.
- Functionality to access the oid of the last inserted row (like
plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed.
The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false)
for CREATE TABLE) is still supported. While that requires a bit of
support code, it seems unnecessary to break applications / dumps that
do not use oids, and are explicit about not using them.
The biggest user of WITH OID columns was postgres' catalog. This
commit changes all 'magic' oid columns to be columns that are normally
declared and stored. To reduce unnecessary query breakage all the
newly added columns are still named 'oid', even if a table's column
naming scheme would indicate 'reloid' or such. This obviously
requires adapting a lot code, mostly replacing oid access via
HeapTupleGetOid() with access to the underlying Form_pg_*->oid column.
The bootstrap process now assigns oids for all oid columns in
genbki.pl that do not have an explicit value (starting at the largest
oid previously used), only oids assigned later by oids will be above
FirstBootstrapObjectId. As the oid column now is a normal column the
special bootstrap syntax for oids has been removed.
Oids are not automatically assigned during insertion anymore, all
backend code explicitly assigns oids with GetNewOidWithIndex(). For
the rare case that insertions into the catalog via SQL are called for
the new pg_nextoid() function can be used (which only works on catalog
tables).
The fact that oid columns on system tables are now normal columns
means that they will be included in the set of columns expanded
by * (i.e. SELECT * FROM pg_class will now include the table's oid,
previously it did not). It'd not technically be hard to hide oid
column by default, but that'd mean confusing behavior would either
have to be carried forward forever, or it'd cause breakage down the
line.
While it's not unlikely that further adjustments are needed, the
scope/invasiveness of the patch makes it worthwhile to get merge this
now. It's painful to maintain externally, too complicated to commit
after the code code freeze, and a dependency of a number of other
patches.
Catversion bump, for obvious reasons.
Author: Andres Freund, with contributions by John Naylor
Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de
2018-11-21 00:36:57 +01:00
|
|
|
dictOid = GetSysCacheOid2(TSDICTNAMENSP, Anum_pg_ts_dict_oid,
|
2018-09-06 16:49:45 +02:00
|
|
|
PointerGetDatum(dictname),
|
|
|
|
ObjectIdGetDatum(procnspid));
|
|
|
|
if (!OidIsValid(dictOid))
|
|
|
|
ereport(ERROR,
|
|
|
|
(errcode(ERRCODE_UNDEFINED_OBJECT),
|
|
|
|
errmsg("text search dictionary \"%s.%s\" does not exist",
|
|
|
|
get_namespace_name(procnspid), dictname)));
|
2009-08-18 12:34:39 +02:00
|
|
|
strArg = 0;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
dictOid = PG_GETARG_OID(0);
|
|
|
|
strArg = 1;
|
|
|
|
}
|
2017-03-13 00:35:34 +01:00
|
|
|
str = PG_GETARG_TEXT_PP(strArg);
|
2009-08-18 12:34:39 +02:00
|
|
|
|
|
|
|
dict = lookup_ts_dictionary_cache(dictOid);
|
|
|
|
|
|
|
|
res = (TSLexeme *) DatumGetPointer(FunctionCall4(&(dict->lexize),
|
|
|
|
PointerGetDatum(dict->dictData),
|
2017-03-13 00:35:34 +01:00
|
|
|
PointerGetDatum(VARDATA_ANY(str)),
|
|
|
|
Int32GetDatum(VARSIZE_ANY_EXHDR(str)),
|
2009-08-18 12:34:39 +02:00
|
|
|
PointerGetDatum(NULL)));
|
|
|
|
|
|
|
|
PG_FREE_IF_COPY(str, strArg);
|
|
|
|
|
|
|
|
if (res == NULL)
|
|
|
|
{
|
|
|
|
PG_RETURN_TEXT_P(PG_GETARG_TEXT_P_COPY(strArg));
|
|
|
|
}
|
|
|
|
else if (res->lexeme == NULL)
|
|
|
|
{
|
|
|
|
pfree(res);
|
|
|
|
PG_RETURN_TEXT_P(PG_GETARG_TEXT_P_COPY(strArg));
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
text *txt = cstring_to_text(res->lexeme);
|
|
|
|
|
|
|
|
pfree(res->lexeme);
|
|
|
|
pfree(res);
|
|
|
|
|
|
|
|
PG_RETURN_TEXT_P(txt);
|
|
|
|
}
|
|
|
|
}
|