Make locale-dependent regex character classes work for large char codes.
Previously, we failed to recognize Unicode characters above U+7FF as being members of locale-dependent character classes such as [[:alpha:]]. (Actually, the same problem occurs for large pg_wchar values in any multibyte encoding, but UTF8 is the only case people have actually complained about.) It's impractical to get Spencer's original code to handle character classes or ranges containing many thousands of characters, because it insists on considering each member character individually at regex compile time, whether or not the character will ever be of interest at run time. To fix, choose a cutoff point MAX_SIMPLE_CHR below which we process characters individually as before, and deal with entire ranges or classes as single entities above that. We can actually make things cheaper than before for chars below the cutoff, because the color map can now be a simple linear array for those chars, rather than the multilevel tree structure Spencer designed. It's more expensive than before for chars above the cutoff, because we must do a binary search in a list of high chars and char ranges used in the regex pattern, plus call iswalpha() and friends for each locale-dependent character class used in the pattern. However, multibyte encodings are normally designed to give smaller codes to popular characters, so that we can expect that the slow path will be taken relatively infrequently. In any case, the speed penalty appears minor except when we have to apply iswalpha() etc. to high character codes at runtime --- and the previous coding gave wrong answers for those cases, so whether it was faster is moot. Tom Lane, reviewed by Heikki Linnakangas Discussion: <15563.1471913698@sss.pgh.pa.us>
This commit is contained in:
Tom Lane
parent
f80049f76a
commit
c54159d44c
|
|
@ -27,13 +27,14 @@ and similarly additional source files rege_*.c that are #include'd in
|
|||
regexec. This was done to avoid exposing internal symbols globally;
|
||||
all functions not meant to be part of the library API are static.
|
||||
|
||||
(Actually the above is a lie in one respect: there is one more global
|
||||
symbol, pg_set_regex_collation in regcomp. It is not meant to be part of
|
||||
the API, but it has to be global because both regcomp and regexec call it.
|
||||
It'd be better to get rid of that, as well as the static variables it
|
||||
sets, in favor of keeping the needed locale state in the regex structs.
|
||||
We have not done this yet for lack of a design for how to add
|
||||
application-specific state to the structs.)
|
||||
(Actually the above is a lie in one respect: there are two more global
|
||||
symbols, pg_set_regex_collation and pg_reg_getcolor in regcomp. These are
|
||||
not meant to be part of the API, but they have to be global because both
|
||||
regcomp and regexec call them. It'd be better to get rid of
|
||||
pg_set_regex_collation, as well as the static variables it sets, in favor of
|
||||
keeping the needed locale state in the regex structs. We have not done this
|
||||
yet for lack of a design for how to add application-specific state to the
|
||||
structs.)
|
||||
|
||||
What's where in src/backend/regex/:
|
||||
|
||||
|
|
@ -274,28 +275,65 @@ colors:
|
|||
an existing color has to be subdivided.
|
||||
|
||||
The last two of these are handled with the "struct colordesc" array and
|
||||
the "colorchain" links in NFA arc structs. The color map proper (that
|
||||
is, the per-character lookup array) is handled as a multi-level tree,
|
||||
with each tree level indexed by one byte of a character's value. The
|
||||
code arranges to not have more than one copy of bottom-level tree pages
|
||||
that are all-the-same-color.
|
||||
the "colorchain" links in NFA arc structs.
|
||||
|
||||
Unfortunately, this design does not seem terribly efficient for common
|
||||
cases such as a tree in which all Unicode letters are colored the same,
|
||||
because there aren't that many places where we get a whole page all the
|
||||
same color, except at the end of the map. (It also strikes me that given
|
||||
PG's current restrictions on the range of Unicode values, we could use a
|
||||
3-level rather than 4-level tree; but there's not provision for that in
|
||||
regguts.h at the moment.)
|
||||
Ideally, we'd do the first two operations using a simple linear array
|
||||
storing the current color assignment for each character code.
|
||||
Unfortunately, that's not terribly workable for large charsets such as
|
||||
Unicode. Our solution is to divide the color map into two parts. A simple
|
||||
linear array is used for character codes up to MAX_SIMPLE_CHR, which can be
|
||||
chosen large enough to include all popular characters (so that the
|
||||
significantly-slower code paths about to be described are seldom invoked).
|
||||
Characters above that need be considered at compile time only if they
|
||||
appear explicitly in the regex pattern. We store each such mentioned
|
||||
character or character range as an entry in the "colormaprange" array in
|
||||
the colormap. (Overlapping ranges are split into unique subranges, so that
|
||||
each range in the finished list needs only a single color that describes
|
||||
all its characters.) When mapping a character above MAX_SIMPLE_CHR to a
|
||||
color at runtime, we search this list of ranges explicitly.
|
||||
|
||||
A bigger problem is that it just doesn't seem very reasonable to have to
|
||||
consider each Unicode letter separately at regex parse time for a regex
|
||||
such as "\w"; more than likely, a huge percentage of those codes will
|
||||
never be seen at runtime. We need to fix things so that locale-based
|
||||
character classes are somehow processed "symbolically" without making a
|
||||
full expansion of their contents at parse time. This would mean that we'd
|
||||
have to be ready to call iswalpha() at runtime, but if that only happens
|
||||
for high-code-value characters, it shouldn't be a big performance hit.
|
||||
That's still not quite enough, though, because of locale-dependent
|
||||
character classes such as [[:alpha:]]. In Unicode locales these classes
|
||||
may have thousands of entries that are above MAX_SIMPLE_CHR, and we
|
||||
certainly don't want to be searching large colormaprange arrays at runtime.
|
||||
Nor do we even want to spend the time to initialize cvec structures that
|
||||
exhaustively describe all of those characters. Our solution is to compute
|
||||
exact per-character colors at regex compile time only up to MAX_SIMPLE_CHR.
|
||||
For characters above that, we apply the <ctype.h> or <wctype.h> lookup
|
||||
functions at runtime for each locale-dependent character class used in the
|
||||
regex pattern, constructing a bitmap that describes which classes the
|
||||
runtime character belongs to. The per-character-range data structure
|
||||
mentioned above actually holds, for each range, a separate color entry
|
||||
for each possible combination of character class properties. That is,
|
||||
the color map for characters above MAX_SIMPLE_CHR is really a 2-D array,
|
||||
whose rows correspond to high characters or character ranges that are
|
||||
explicitly mentioned in the regex pattern, and whose columns correspond
|
||||
to sets of the locale-dependent character classes that are used in the
|
||||
regex.
|
||||
|
||||
As an example, given the pattern '\w\u1234[\U0001D100-\U0001D1FF]'
|
||||
(and supposing that MAX_SIMPLE_CHR is less than 0x1234), we will need
|
||||
a high color map with three rows. One row is for the single character
|
||||
U+1234 (represented as a single-element range), one is for the range
|
||||
U+1D100..U+1D1FF, and the other row represents all remaining high
|
||||
characters. The color map has two columns, one for characters that
|
||||
satisfy iswalnum() and one for those that don't.
|
||||
|
||||
We build this color map in parallel with scanning the regex. Each time
|
||||
we detect a new explicit high character (or range) or a locale-dependent
|
||||
character class, we split existing entry(s) in the high color map so that
|
||||
characters we need to be able to distinguish will have distinct entries
|
||||
that can be given separate colors. Often, though, single entries in the
|
||||
high color map will represent very large sets of characters.
|
||||
|
||||
If there are both explicit high characters/ranges and locale-dependent
|
||||
character classes, we may have entries in the high color map array that
|
||||
have non-WHITE colors but don't actually represent any real characters.
|
||||
(For example, in a row representing a singleton range, only one of the
|
||||
columns could possibly be a live entry; it's the one matching the actual
|
||||
locale properties for that single character.) We don't currently make
|
||||
any effort to reclaim such colors. In principle it could be done, but
|
||||
it's not clear that it's worth the trouble.
|
||||
|
||||
|
||||
Detailed semantics of an NFA
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -34,7 +34,8 @@
|
|||
|
||||
/*
|
||||
* Notes:
|
||||
* Only (selected) functions in _this_ file should treat chr* as non-constant.
|
||||
* Only (selected) functions in _this_ file should treat the chr arrays
|
||||
* of a cvec as non-constant.
|
||||
*/
|
||||
|
||||
/*
|
||||
|
|
@ -67,6 +68,7 @@ clearcvec(struct cvec * cv)
|
|||
assert(cv != NULL);
|
||||
cv->nchrs = 0;
|
||||
cv->nranges = 0;
|
||||
cv->cclasscode = -1;
|
||||
return cv;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -349,6 +349,19 @@ static const struct cname
|
|||
}
|
||||
};
|
||||
|
||||
/*
|
||||
* The following arrays define the valid character class names.
|
||||
*/
|
||||
static const char *const classNames[NUM_CCLASSES + 1] = {
|
||||
"alnum", "alpha", "ascii", "blank", "cntrl", "digit", "graph",
|
||||
"lower", "print", "punct", "space", "upper", "xdigit", NULL
|
||||
};
|
||||
|
||||
enum classes
|
||||
{
|
||||
CC_ALNUM, CC_ALPHA, CC_ASCII, CC_BLANK, CC_CNTRL, CC_DIGIT, CC_GRAPH,
|
||||
CC_LOWER, CC_PRINT, CC_PUNCT, CC_SPACE, CC_UPPER, CC_XDIGIT
|
||||
};
|
||||
|
||||
/*
|
||||
* We do not use the hard-wired Unicode classification tables that Tcl does.
|
||||
|
|
@ -543,21 +556,6 @@ cclass(struct vars * v, /* context */
|
|||
int i,
|
||||
index;
|
||||
|
||||
/*
|
||||
* The following arrays define the valid character class names.
|
||||
*/
|
||||
|
||||
static const char *const classNames[] = {
|
||||
"alnum", "alpha", "ascii", "blank", "cntrl", "digit", "graph",
|
||||
"lower", "print", "punct", "space", "upper", "xdigit", NULL
|
||||
};
|
||||
|
||||
enum classes
|
||||
{
|
||||
CC_ALNUM, CC_ALPHA, CC_ASCII, CC_BLANK, CC_CNTRL, CC_DIGIT, CC_GRAPH,
|
||||
CC_LOWER, CC_PRINT, CC_PUNCT, CC_SPACE, CC_UPPER, CC_XDIGIT
|
||||
};
|
||||
|
||||
/*
|
||||
* Map the name to the corresponding enumerated value.
|
||||
*/
|
||||
|
|
@ -593,18 +591,20 @@ cclass(struct vars * v, /* context */
|
|||
* pg_ctype_get_cache so that we can cache the results. Other classes
|
||||
* have definitions that are hard-wired here, and for those we just
|
||||
* construct a transient cvec on the fly.
|
||||
*
|
||||
* NB: keep this code in sync with cclass_column_index(), below.
|
||||
*/
|
||||
|
||||
switch ((enum classes) index)
|
||||
{
|
||||
case CC_PRINT:
|
||||
cv = pg_ctype_get_cache(pg_wc_isprint);
|
||||
cv = pg_ctype_get_cache(pg_wc_isprint, index);
|
||||
break;
|
||||
case CC_ALNUM:
|
||||
cv = pg_ctype_get_cache(pg_wc_isalnum);
|
||||
cv = pg_ctype_get_cache(pg_wc_isalnum, index);
|
||||
break;
|
||||
case CC_ALPHA:
|
||||
cv = pg_ctype_get_cache(pg_wc_isalpha);
|
||||
cv = pg_ctype_get_cache(pg_wc_isalpha, index);
|
||||
break;
|
||||
case CC_ASCII:
|
||||
/* hard-wired meaning */
|
||||
|
|
@ -625,10 +625,10 @@ cclass(struct vars * v, /* context */
|
|||
addrange(cv, 0x7f, 0x9f);
|
||||
break;
|
||||
case CC_DIGIT:
|
||||
cv = pg_ctype_get_cache(pg_wc_isdigit);
|
||||
cv = pg_ctype_get_cache(pg_wc_isdigit, index);
|
||||
break;
|
||||
case CC_PUNCT:
|
||||
cv = pg_ctype_get_cache(pg_wc_ispunct);
|
||||
cv = pg_ctype_get_cache(pg_wc_ispunct, index);
|
||||
break;
|
||||
case CC_XDIGIT:
|
||||
|
||||
|
|
@ -646,16 +646,16 @@ cclass(struct vars * v, /* context */
|
|||
}
|
||||
break;
|
||||
case CC_SPACE:
|
||||
cv = pg_ctype_get_cache(pg_wc_isspace);
|
||||
cv = pg_ctype_get_cache(pg_wc_isspace, index);
|
||||
break;
|
||||
case CC_LOWER:
|
||||
cv = pg_ctype_get_cache(pg_wc_islower);
|
||||
cv = pg_ctype_get_cache(pg_wc_islower, index);
|
||||
break;
|
||||
case CC_UPPER:
|
||||
cv = pg_ctype_get_cache(pg_wc_isupper);
|
||||
cv = pg_ctype_get_cache(pg_wc_isupper, index);
|
||||
break;
|
||||
case CC_GRAPH:
|
||||
cv = pg_ctype_get_cache(pg_wc_isgraph);
|
||||
cv = pg_ctype_get_cache(pg_wc_isgraph, index);
|
||||
break;
|
||||
}
|
||||
|
||||
|
|
@ -665,6 +665,47 @@ cclass(struct vars * v, /* context */
|
|||
return cv;
|
||||
}
|
||||
|
||||
/*
|
||||
* cclass_column_index - get appropriate high colormap column index for chr
|
||||
*/
|
||||
static int
|
||||
cclass_column_index(struct colormap * cm, chr c)
|
||||
{
|
||||
int colnum = 0;
|
||||
|
||||
/* Shouldn't go through all these pushups for simple chrs */
|
||||
assert(c > MAX_SIMPLE_CHR);
|
||||
|
||||
/*
|
||||
* Note: we should not see requests to consider cclasses that are not
|
||||
* treated as locale-specific by cclass(), above.
|
||||
*/
|
||||
if (cm->classbits[CC_PRINT] && pg_wc_isprint(c))
|
||||
colnum |= cm->classbits[CC_PRINT];
|
||||
if (cm->classbits[CC_ALNUM] && pg_wc_isalnum(c))
|
||||
colnum |= cm->classbits[CC_ALNUM];
|
||||
if (cm->classbits[CC_ALPHA] && pg_wc_isalpha(c))
|
||||
colnum |= cm->classbits[CC_ALPHA];
|
||||
assert(cm->classbits[CC_ASCII] == 0);
|
||||
assert(cm->classbits[CC_BLANK] == 0);
|
||||
assert(cm->classbits[CC_CNTRL] == 0);
|
||||
if (cm->classbits[CC_DIGIT] && pg_wc_isdigit(c))
|
||||
colnum |= cm->classbits[CC_DIGIT];
|
||||
if (cm->classbits[CC_PUNCT] && pg_wc_ispunct(c))
|
||||
colnum |= cm->classbits[CC_PUNCT];
|
||||
assert(cm->classbits[CC_XDIGIT] == 0);
|
||||
if (cm->classbits[CC_SPACE] && pg_wc_isspace(c))
|
||||
colnum |= cm->classbits[CC_SPACE];
|
||||
if (cm->classbits[CC_LOWER] && pg_wc_islower(c))
|
||||
colnum |= cm->classbits[CC_LOWER];
|
||||
if (cm->classbits[CC_UPPER] && pg_wc_isupper(c))
|
||||
colnum |= cm->classbits[CC_UPPER];
|
||||
if (cm->classbits[CC_GRAPH] && pg_wc_isgraph(c))
|
||||
colnum |= cm->classbits[CC_GRAPH];
|
||||
|
||||
return colnum;
|
||||
}
|
||||
|
||||
/*
|
||||
* allcases - supply cvec for all case counterparts of a chr (including itself)
|
||||
*
|
||||
|
|
|
|||
|
|
@ -736,7 +736,7 @@ store_match(pg_ctype_cache *pcc, pg_wchar chr1, int nchrs)
|
|||
* Note that the result must not be freed or modified by caller.
|
||||
*/
|
||||
static struct cvec *
|
||||
pg_ctype_get_cache(pg_wc_probefunc probefunc)
|
||||
pg_ctype_get_cache(pg_wc_probefunc probefunc, int cclasscode)
|
||||
{
|
||||
pg_ctype_cache *pcc;
|
||||
pg_wchar max_chr;
|
||||
|
|
@ -770,31 +770,43 @@ pg_ctype_get_cache(pg_wc_probefunc probefunc)
|
|||
pcc->cv.ranges = (chr *) malloc(pcc->cv.rangespace * sizeof(chr) * 2);
|
||||
if (pcc->cv.chrs == NULL || pcc->cv.ranges == NULL)
|
||||
goto out_of_memory;
|
||||
pcc->cv.cclasscode = cclasscode;
|
||||
|
||||
/*
|
||||
* Decide how many character codes we ought to look through. For C locale
|
||||
* there's no need to go further than 127. Otherwise, if the encoding is
|
||||
* UTF8 go up to 0x7FF, which is a pretty arbitrary cutoff but we cannot
|
||||
* extend it as far as we'd like (say, 0xFFFF, the end of the Basic
|
||||
* Multilingual Plane) without creating significant performance issues due
|
||||
* to too many characters being fed through the colormap code. This will
|
||||
* need redesign to fix reasonably, but at least for the moment we have
|
||||
* all common European languages covered. Otherwise (not C, not UTF8) go
|
||||
* up to 255. These limits are interrelated with restrictions discussed
|
||||
* at the head of this file.
|
||||
* Decide how many character codes we ought to look through. In general
|
||||
* we don't go past MAX_SIMPLE_CHR; chr codes above that are handled at
|
||||
* runtime using the "high colormap" mechanism. However, in C locale
|
||||
* there's no need to go further than 127, and if we only have a 1-byte
|
||||
* <ctype.h> API there's no need to go further than that can handle.
|
||||
*
|
||||
* If it's not MAX_SIMPLE_CHR that's constraining the search, mark the
|
||||
* output cvec as not having any locale-dependent behavior, since there
|
||||
* will be no need to do any run-time locale checks. (The #if's here
|
||||
* would always be true for production values of MAX_SIMPLE_CHR, but it's
|
||||
* useful to allow it to be small for testing purposes.)
|
||||
*/
|
||||
switch (pg_regex_strategy)
|
||||
{
|
||||
case PG_REGEX_LOCALE_C:
|
||||
#if MAX_SIMPLE_CHR >= 127
|
||||
max_chr = (pg_wchar) 127;
|
||||
pcc->cv.cclasscode = -1;
|
||||
#else
|
||||
max_chr = (pg_wchar) MAX_SIMPLE_CHR;
|
||||
#endif
|
||||
break;
|
||||
case PG_REGEX_LOCALE_WIDE:
|
||||
case PG_REGEX_LOCALE_WIDE_L:
|
||||
max_chr = (pg_wchar) 0x7FF;
|
||||
max_chr = (pg_wchar) MAX_SIMPLE_CHR;
|
||||
break;
|
||||
case PG_REGEX_LOCALE_1BYTE:
|
||||
case PG_REGEX_LOCALE_1BYTE_L:
|
||||
#if MAX_SIMPLE_CHR >= UCHAR_MAX
|
||||
max_chr = (pg_wchar) UCHAR_MAX;
|
||||
pcc->cv.cclasscode = -1;
|
||||
#else
|
||||
max_chr = (pg_wchar) MAX_SIMPLE_CHR;
|
||||
#endif
|
||||
break;
|
||||
default:
|
||||
max_chr = 0; /* can't get here, but keep compiler quiet */
|
||||
|
|
|
|||
|
|
@ -55,7 +55,6 @@ static void cbracket(struct vars *, struct state *, struct state *);
|
|||
static void brackpart(struct vars *, struct state *, struct state *);
|
||||
static const chr *scanplain(struct vars *);
|
||||
static void onechr(struct vars *, chr, struct state *, struct state *);
|
||||
static void dovec(struct vars *, struct cvec *, struct state *, struct state *);
|
||||
static void wordchrs(struct vars *);
|
||||
static void processlacon(struct vars *, struct state *, struct state *, int,
|
||||
struct state *, struct state *);
|
||||
|
|
@ -96,16 +95,19 @@ static chr chrnamed(struct vars *, const chr *, const chr *, chr);
|
|||
/* === regc_color.c === */
|
||||
static void initcm(struct vars *, struct colormap *);
|
||||
static void freecm(struct colormap *);
|
||||
static void cmtreefree(struct colormap *, union tree *, int);
|
||||
static color setcolor(struct colormap *, chr, color);
|
||||
static color maxcolor(struct colormap *);
|
||||
static color newcolor(struct colormap *);
|
||||
static void freecolor(struct colormap *, color);
|
||||
static color pseudocolor(struct colormap *);
|
||||
static color subcolor(struct colormap *, chr c);
|
||||
static color subcolor(struct colormap *, chr);
|
||||
static color subcolorhi(struct colormap *, color *);
|
||||
static color newsub(struct colormap *, color);
|
||||
static void subrange(struct vars *, chr, chr, struct state *, struct state *);
|
||||
static void subblock(struct vars *, chr, struct state *, struct state *);
|
||||
static int newhicolorrow(struct colormap *, int);
|
||||
static void newhicolorcols(struct colormap *);
|
||||
static void subcolorcvec(struct vars *, struct cvec *, struct state *, struct state *);
|
||||
static void subcoloronechr(struct vars *, chr, struct state *, struct state *, color *);
|
||||
static void subcoloronerange(struct vars *, chr, chr, struct state *, struct state *, color *);
|
||||
static void subcoloronerow(struct vars *, int, struct state *, struct state *, color *);
|
||||
static void okcolors(struct nfa *, struct colormap *);
|
||||
static void colorchain(struct colormap *, struct arc *);
|
||||
static void uncolorchain(struct colormap *, struct arc *);
|
||||
|
|
@ -114,7 +116,6 @@ static void colorcomplement(struct nfa *, struct colormap *, int, struct state *
|
|||
|
||||
#ifdef REG_DEBUG
|
||||
static void dumpcolors(struct colormap *, FILE *);
|
||||
static void fillcheck(struct colormap *, union tree *, int, FILE *);
|
||||
static void dumpchr(chr, FILE *);
|
||||
#endif
|
||||
/* === regc_nfa.c === */
|
||||
|
|
@ -215,6 +216,7 @@ static struct cvec *range(struct vars *, chr, chr, int);
|
|||
static int before(chr, chr);
|
||||
static struct cvec *eclass(struct vars *, chr, int);
|
||||
static struct cvec *cclass(struct vars *, const chr *, const chr *, int);
|
||||
static int cclass_column_index(struct colormap *, chr);
|
||||
static struct cvec *allcases(struct vars *, chr);
|
||||
static int cmp(const chr *, const chr *, size_t);
|
||||
static int casecmp(const chr *, const chr *, size_t);
|
||||
|
|
@ -1467,7 +1469,7 @@ brackpart(struct vars * v,
|
|||
NOERR();
|
||||
cv = eclass(v, startc, (v->cflags & REG_ICASE));
|
||||
NOERR();
|
||||
dovec(v, cv, lp, rp);
|
||||
subcolorcvec(v, cv, lp, rp);
|
||||
return;
|
||||
break;
|
||||
case CCLASS:
|
||||
|
|
@ -1477,7 +1479,7 @@ brackpart(struct vars * v,
|
|||
NOERR();
|
||||
cv = cclass(v, startp, endp, (v->cflags & REG_ICASE));
|
||||
NOERR();
|
||||
dovec(v, cv, lp, rp);
|
||||
subcolorcvec(v, cv, lp, rp);
|
||||
return;
|
||||
break;
|
||||
default:
|
||||
|
|
@ -1523,7 +1525,7 @@ brackpart(struct vars * v,
|
|||
NOTE(REG_UUNPORT);
|
||||
cv = range(v, startc, endc, (v->cflags & REG_ICASE));
|
||||
NOERR();
|
||||
dovec(v, cv, lp, rp);
|
||||
subcolorcvec(v, cv, lp, rp);
|
||||
}
|
||||
|
||||
/*
|
||||
|
|
@ -1565,46 +1567,14 @@ onechr(struct vars * v,
|
|||
{
|
||||
if (!(v->cflags & REG_ICASE))
|
||||
{
|
||||
newarc(v->nfa, PLAIN, subcolor(v->cm, c), lp, rp);
|
||||
color lastsubcolor = COLORLESS;
|
||||
|
||||
subcoloronechr(v, c, lp, rp, &lastsubcolor);
|
||||
return;
|
||||
}
|
||||
|
||||
/* rats, need general case anyway... */
|
||||
dovec(v, allcases(v, c), lp, rp);
|
||||
}
|
||||
|
||||
/*
|
||||
* dovec - fill in arcs for each element of a cvec
|
||||
*/
|
||||
static void
|
||||
dovec(struct vars * v,
|
||||
struct cvec * cv,
|
||||
struct state * lp,
|
||||
struct state * rp)
|
||||
{
|
||||
chr ch,
|
||||
from,
|
||||
to;
|
||||
const chr *p;
|
||||
int i;
|
||||
|
||||
/* ordinary characters */
|
||||
for (p = cv->chrs, i = cv->nchrs; i > 0; p++, i--)
|
||||
{
|
||||
ch = *p;
|
||||
newarc(v->nfa, PLAIN, subcolor(v->cm, ch), lp, rp);
|
||||
NOERR();
|
||||
}
|
||||
|
||||
/* and the ranges */
|
||||
for (p = cv->ranges, i = cv->nranges; i > 0; p += 2, i--)
|
||||
{
|
||||
from = *p;
|
||||
to = *(p + 1);
|
||||
if (from <= to)
|
||||
subrange(v, from, to, lp, rp);
|
||||
NOERR();
|
||||
}
|
||||
subcolorcvec(v, allcases(v, c), lp, rp);
|
||||
}
|
||||
|
||||
/*
|
||||
|
|
|
|||
|
|
@ -28,10 +28,6 @@
|
|||
|
||||
#include "regex/regexport.h"
|
||||
|
||||
static void scancolormap(struct colormap * cm, int co,
|
||||
union tree * t, int level, chr partial,
|
||||
pg_wchar **chars, int *chars_len);
|
||||
|
||||
|
||||
/*
|
||||
* Get total number of NFA states.
|
||||
|
|
@ -187,10 +183,7 @@ pg_reg_colorisend(const regex_t *regex, int co)
|
|||
*
|
||||
* Note: we return -1 if the color number is invalid, or if it is a special
|
||||
* color (WHITE or a pseudocolor), or if the number of members is uncertain.
|
||||
* The latter case cannot arise right now but is specified to allow for future
|
||||
* improvements (see musings about run-time handling of higher character codes
|
||||
* in regex/README). Callers should not try to extract the members if -1 is
|
||||
* returned.
|
||||
* Callers should not try to extract the members if -1 is returned.
|
||||
*/
|
||||
int
|
||||
pg_reg_getnumcharacters(const regex_t *regex, int co)
|
||||
|
|
@ -205,7 +198,18 @@ pg_reg_getnumcharacters(const regex_t *regex, int co)
|
|||
if (cm->cd[co].flags & PSEUDO) /* also pseudocolors (BOS etc) */
|
||||
return -1;
|
||||
|
||||
return cm->cd[co].nchrs;
|
||||
/*
|
||||
* If the color appears anywhere in the high colormap, treat its number of
|
||||
* members as uncertain. In principle we could determine all the specific
|
||||
* chrs corresponding to each such entry, but it would be expensive
|
||||
* (particularly if character class tests are required) and it doesn't
|
||||
* seem worth it.
|
||||
*/
|
||||
if (cm->cd[co].nuchrs != 0)
|
||||
return -1;
|
||||
|
||||
/* OK, return the known number of member chrs */
|
||||
return cm->cd[co].nschrs;
|
||||
}
|
||||
|
||||
/*
|
||||
|
|
@ -222,6 +226,7 @@ pg_reg_getcharacters(const regex_t *regex, int co,
|
|||
pg_wchar *chars, int chars_len)
|
||||
{
|
||||
struct colormap *cm;
|
||||
chr c;
|
||||
|
||||
assert(regex != NULL && regex->re_magic == REMAGIC);
|
||||
cm = &((struct guts *) regex->re_guts)->cmap;
|
||||
|
|
@ -231,62 +236,17 @@ pg_reg_getcharacters(const regex_t *regex, int co,
|
|||
if (cm->cd[co].flags & PSEUDO)
|
||||
return;
|
||||
|
||||
/* Recursively search the colormap tree */
|
||||
scancolormap(cm, co, cm->tree, 0, 0, &chars, &chars_len);
|
||||
}
|
||||
|
||||
/*
|
||||
* Recursively scan the colormap tree to find chrs belonging to color "co".
|
||||
* See regex/README for info about the tree structure.
|
||||
*
|
||||
* t: tree block to scan
|
||||
* level: level (from 0) of t
|
||||
* partial: partial chr code for chrs within t
|
||||
* chars, chars_len: output area
|
||||
*/
|
||||
static void
|
||||
scancolormap(struct colormap * cm, int co,
|
||||
union tree * t, int level, chr partial,
|
||||
pg_wchar **chars, int *chars_len)
|
||||
{
|
||||
int i;
|
||||
|
||||
if (level < NBYTS - 1)
|
||||
/*
|
||||
* We need only examine the low character map; there should not be any
|
||||
* matching entries in the high map.
|
||||
*/
|
||||
for (c = CHR_MIN; c <= MAX_SIMPLE_CHR; c++)
|
||||
{
|
||||
/* non-leaf node */
|
||||
for (i = 0; i < BYTTAB; i++)
|
||||
if (cm->locolormap[c - CHR_MIN] == co)
|
||||
{
|
||||
/*
|
||||
* We do not support search for chrs of color 0 (WHITE), so
|
||||
* all-white subtrees need not be searched. These can be
|
||||
* recognized because they are represented by the fill blocks in
|
||||
* the colormap struct. This typically allows us to avoid
|
||||
* scanning large regions of higher-numbered chrs.
|
||||
*/
|
||||
if (t->tptr[i] == &cm->tree[level + 1])
|
||||
continue;
|
||||
|
||||
/* Recursively scan next level down */
|
||||
scancolormap(cm, co,
|
||||
t->tptr[i], level + 1,
|
||||
(partial | (chr) i) << BYTBITS,
|
||||
chars, chars_len);
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
/* leaf node */
|
||||
for (i = 0; i < BYTTAB; i++)
|
||||
{
|
||||
if (t->tcolor[i] == co)
|
||||
{
|
||||
if (*chars_len > 0)
|
||||
{
|
||||
**chars = partial | (chr) i;
|
||||
(*chars)++;
|
||||
(*chars_len)--;
|
||||
}
|
||||
}
|
||||
*chars++ = c;
|
||||
if (--chars_len == 0)
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -194,7 +194,10 @@ findprefix(struct cnfa * cnfa,
|
|||
if (thiscolor == COLORLESS)
|
||||
break;
|
||||
/* The color must be a singleton */
|
||||
if (cm->cd[thiscolor].nchrs != 1)
|
||||
if (cm->cd[thiscolor].nschrs != 1)
|
||||
break;
|
||||
/* Must not have any high-color-map entries */
|
||||
if (cm->cd[thiscolor].nuchrs != 0)
|
||||
break;
|
||||
|
||||
/*
|
||||
|
|
|
|||
|
|
@ -77,6 +77,16 @@ typedef unsigned uchr; /* unsigned type that will hold a chr */
|
|||
*/
|
||||
#define CHR_IS_IN_RANGE(c) ((c) <= CHR_MAX)
|
||||
|
||||
/*
|
||||
* MAX_SIMPLE_CHR is the cutoff between "simple" and "complicated" processing
|
||||
* in the color map logic. It should usually be chosen high enough to ensure
|
||||
* that all common characters are <= MAX_SIMPLE_CHR. However, very large
|
||||
* values will be counterproductive since they cause more regex setup time.
|
||||
* Also, small values can be helpful for testing the high-color-map logic
|
||||
* with plain old ASCII input.
|
||||
*/
|
||||
#define MAX_SIMPLE_CHR 0x7FF /* suitable value for Unicode */
|
||||
|
||||
/* functions operating on chr */
|
||||
#define iscalnum(x) pg_wc_isalnum(x)
|
||||
#define iscalpha(x) pg_wc_isalpha(x)
|
||||
|
|
|
|||
|
|
@ -172,6 +172,5 @@ extern int pg_regexec(regex_t *, const pg_wchar *, size_t, size_t, rm_detail_t *
|
|||
extern int pg_regprefix(regex_t *, pg_wchar **, size_t *);
|
||||
extern void pg_regfree(regex_t *);
|
||||
extern size_t pg_regerror(int, const regex_t *, char *, size_t);
|
||||
extern void pg_set_regex_collation(Oid collation);
|
||||
|
||||
#endif /* _REGEX_H_ */
|
||||
|
|
|
|||
|
|
@ -127,23 +127,6 @@
|
|||
#define ISBSET(uv, sn) ((uv)[(sn)/UBITS] & ((unsigned)1 << ((sn)%UBITS)))
|
||||
|
||||
|
||||
|
||||
/*
|
||||
* We dissect a chr into byts for colormap table indexing. Here we define
|
||||
* a byt, which will be the same as a byte on most machines... The exact
|
||||
* size of a byt is not critical, but about 8 bits is good, and extraction
|
||||
* of 8-bit chunks is sometimes especially fast.
|
||||
*/
|
||||
#ifndef BYTBITS
|
||||
#define BYTBITS 8 /* bits in a byt */
|
||||
#endif
|
||||
#define BYTTAB (1<<BYTBITS) /* size of table with one entry per byt value */
|
||||
#define BYTMASK (BYTTAB-1) /* bit mask for byt */
|
||||
#define NBYTS ((CHRBITS+BYTBITS-1)/BYTBITS)
|
||||
/* the definition of GETCOLOR(), below, assumes NBYTS <= 4 */
|
||||
|
||||
|
||||
|
||||
/*
|
||||
* As soon as possible, we map chrs into equivalence classes -- "colors" --
|
||||
* which are of much more manageable number.
|
||||
|
|
@ -153,42 +136,9 @@ typedef short color; /* colors of characters */
|
|||
#define MAX_COLOR 32767 /* max color (must fit in 'color' datatype) */
|
||||
#define COLORLESS (-1) /* impossible color */
|
||||
#define WHITE 0 /* default color, parent of all others */
|
||||
/* Note: various places in the code know that WHITE is zero */
|
||||
|
||||
|
||||
|
||||
/*
|
||||
* A colormap is a tree -- more precisely, a DAG -- indexed at each level
|
||||
* by a byt of the chr, to map the chr to a color efficiently. Because
|
||||
* lower sections of the tree can be shared, it can exploit the usual
|
||||
* sparseness of such a mapping table. The tree is always NBYTS levels
|
||||
* deep (in the past it was shallower during construction but was "filled"
|
||||
* to full depth at the end of that); areas that are unaltered as yet point
|
||||
* to "fill blocks" which are entirely WHITE in color.
|
||||
*
|
||||
* Leaf-level tree blocks are of type "struct colors", while upper-level
|
||||
* blocks are of type "struct ptrs". Pointers into the tree are generally
|
||||
* declared as "union tree *" to be agnostic about what level they point to.
|
||||
*/
|
||||
|
||||
/* the tree itself */
|
||||
struct colors
|
||||
{
|
||||
color ccolor[BYTTAB];
|
||||
};
|
||||
struct ptrs
|
||||
{
|
||||
union tree *pptr[BYTTAB];
|
||||
};
|
||||
union tree
|
||||
{
|
||||
struct colors colors;
|
||||
struct ptrs ptrs;
|
||||
};
|
||||
|
||||
/* use these pseudo-field names when dereferencing a "union tree" pointer */
|
||||
#define tcolor colors.ccolor
|
||||
#define tptr ptrs.pptr
|
||||
|
||||
/*
|
||||
* Per-color data structure for the compile-time color machinery
|
||||
*
|
||||
|
|
@ -203,26 +153,56 @@ union tree
|
|||
*/
|
||||
struct colordesc
|
||||
{
|
||||
uchr nchrs; /* number of chars of this color */
|
||||
int nschrs; /* number of simple chars of this color */
|
||||
int nuchrs; /* number of upper map entries of this color */
|
||||
color sub; /* open subcolor, if any; or free-chain ptr */
|
||||
#define NOSUB COLORLESS /* value of "sub" when no open subcolor */
|
||||
struct arc *arcs; /* chain of all arcs of this color */
|
||||
chr firstchr; /* char first assigned to this color */
|
||||
chr firstchr; /* simple char first assigned to this color */
|
||||
int flags; /* bit values defined next */
|
||||
#define FREECOL 01 /* currently free */
|
||||
#define PSEUDO 02 /* pseudocolor, no real chars */
|
||||
#define UNUSEDCOLOR(cd) ((cd)->flags & FREECOL)
|
||||
union tree *block; /* block of solid color, if any */
|
||||
};
|
||||
|
||||
/*
|
||||
* The color map itself
|
||||
*
|
||||
* Much of the data in the colormap struct is only used at compile time.
|
||||
* However, the bulk of the space usage is in the "tree" structure, so it's
|
||||
* not clear that there's much point in converting the rest to a more compact
|
||||
* form when compilation is finished.
|
||||
* This struct holds both data used only at compile time, and the chr to
|
||||
* color mapping information, used at both compile and run time. The latter
|
||||
* is the bulk of the space, so it's not really worth separating out the
|
||||
* compile-only portion.
|
||||
*
|
||||
* Ideally, the mapping data would just be an array of colors indexed by
|
||||
* chr codes; but for large character sets that's impractical. Fortunately,
|
||||
* common characters have smaller codes, so we can use a simple array for chr
|
||||
* codes up to MAX_SIMPLE_CHR, and do something more complex for codes above
|
||||
* that, without much loss of performance. The "something more complex" is a
|
||||
* 2-D array of color entries, where row indexes correspond to individual chrs
|
||||
* or chr ranges that have been mentioned in the regex (with row zero
|
||||
* representing all other chrs), and column indexes correspond to different
|
||||
* sets of locale-dependent character classes such as "isalpha". The
|
||||
* classbits[k] entry is zero if we do not care about the k'th character class
|
||||
* in this regex, and otherwise it is the bit to be OR'd into the column index
|
||||
* if the character in question is a member of that class. We find the color
|
||||
* of a high-valued chr by identifying which colormaprange it is in to get
|
||||
* the row index (use row zero if it's in none of them), identifying which of
|
||||
* the interesting cclasses it's in to get the column index, and then indexing
|
||||
* into the 2-D hicolormap array.
|
||||
*
|
||||
* The colormapranges are required to be nonempty, nonoverlapping, and to
|
||||
* appear in increasing chr-value order.
|
||||
*/
|
||||
|
||||
#define NUM_CCLASSES 13 /* must match data in regc_locale.c */
|
||||
|
||||
typedef struct colormaprange
|
||||
{
|
||||
chr cmin; /* range represents cmin..cmax inclusive */
|
||||
chr cmax;
|
||||
int rownum; /* row index in hicolormap array (>= 1) */
|
||||
} colormaprange;
|
||||
|
||||
struct colormap
|
||||
{
|
||||
int magic;
|
||||
|
|
@ -233,27 +213,27 @@ struct colormap
|
|||
color free; /* beginning of free chain (if non-0) */
|
||||
struct colordesc *cd; /* pointer to array of colordescs */
|
||||
#define CDEND(cm) (&(cm)->cd[(cm)->max + 1])
|
||||
|
||||
/* mapping data for chrs <= MAX_SIMPLE_CHR: */
|
||||
color *locolormap; /* simple array indexed by chr code */
|
||||
|
||||
/* mapping data for chrs > MAX_SIMPLE_CHR: */
|
||||
int classbits[NUM_CCLASSES]; /* see comment above */
|
||||
int numcmranges; /* number of colormapranges */
|
||||
colormaprange *cmranges; /* ranges of high chrs */
|
||||
color *hicolormap; /* 2-D array of color entries */
|
||||
int maxarrayrows; /* number of array rows allocated */
|
||||
int hiarrayrows; /* number of array rows in use */
|
||||
int hiarraycols; /* number of array columns (2^N) */
|
||||
|
||||
/* If we need up to NINLINECDS, we store them here to save a malloc */
|
||||
#define NINLINECDS ((size_t)10)
|
||||
#define NINLINECDS ((size_t) 10)
|
||||
struct colordesc cdspace[NINLINECDS];
|
||||
union tree tree[NBYTS]; /* tree top, plus lower-level fill blocks */
|
||||
};
|
||||
|
||||
/* optimization magic to do fast chr->color mapping */
|
||||
#define B0(c) ((c) & BYTMASK)
|
||||
#define B1(c) (((c)>>BYTBITS) & BYTMASK)
|
||||
#define B2(c) (((c)>>(2*BYTBITS)) & BYTMASK)
|
||||
#define B3(c) (((c)>>(3*BYTBITS)) & BYTMASK)
|
||||
#if NBYTS == 1
|
||||
#define GETCOLOR(cm, c) ((cm)->tree->tcolor[B0(c)])
|
||||
#endif
|
||||
/* beware, for NBYTS>1, GETCOLOR() is unsafe -- 2nd arg used repeatedly */
|
||||
#if NBYTS == 2
|
||||
#define GETCOLOR(cm, c) ((cm)->tree->tptr[B1(c)]->tcolor[B0(c)])
|
||||
#endif
|
||||
#if NBYTS == 4
|
||||
#define GETCOLOR(cm, c) ((cm)->tree->tptr[B3(c)]->tptr[B2(c)]->tptr[B1(c)]->tcolor[B0(c)])
|
||||
#endif
|
||||
/* fetch color for chr; beware of multiple evaluation of c argument */
|
||||
#define GETCOLOR(cm, c) \
|
||||
((c) <= MAX_SIMPLE_CHR ? (cm)->locolormap[(c) - CHR_MIN] : pg_reg_getcolor(cm, c))
|
||||
|
||||
|
||||
/*
|
||||
|
|
@ -264,6 +244,11 @@ struct colormap
|
|||
* Representation of a set of characters. chrs[] represents individual
|
||||
* code points, ranges[] represents ranges in the form min..max inclusive.
|
||||
*
|
||||
* If the cvec represents a locale-specific character class, eg [[:alpha:]],
|
||||
* then the chrs[] and ranges[] arrays contain only members of that class
|
||||
* up to MAX_SIMPLE_CHR (inclusive). cclasscode is set to regc_locale.c's
|
||||
* code for the class, rather than being -1 as it is in an ordinary cvec.
|
||||
*
|
||||
* Note that in cvecs gotten from newcvec() and intended to be freed by
|
||||
* freecvec(), both arrays of chrs are after the end of the struct, not
|
||||
* separately malloc'd; so chrspace and rangespace are effectively immutable.
|
||||
|
|
@ -276,6 +261,7 @@ struct cvec
|
|||
int nranges; /* number of ranges (chr pairs) */
|
||||
int rangespace; /* number of ranges allocated in ranges[] */
|
||||
chr *ranges; /* pointer to vector of chr pairs */
|
||||
int cclasscode; /* value of "enum classes", or -1 */
|
||||
};
|
||||
|
||||
|
||||
|
|
@ -489,3 +475,8 @@ struct guts
|
|||
int nlacons; /* size of lacons[]; note that only slots
|
||||
* numbered 1 .. nlacons-1 are used */
|
||||
};
|
||||
|
||||
|
||||
/* prototypes for functions that are exported from regcomp.c to regexec.c */
|
||||
extern void pg_set_regex_collation(Oid collation);
|
||||
extern color pg_reg_getcolor(struct colormap * cm, chr c);
|
||||
|
|
|
|||
|
|
@ -0,0 +1,164 @@
|
|||
/*
|
||||
* This test is for Linux/glibc systems and others that implement proper
|
||||
* locale classification of Unicode characters with high code values.
|
||||
* It must be run in a database with UTF8 encoding and a Unicode-aware locale.
|
||||
*/
|
||||
SET client_encoding TO UTF8;
|
||||
--
|
||||
-- Test the "high colormap" logic with single characters and ranges that
|
||||
-- exceed the MAX_SIMPLE_CHR cutoff, here assumed to be less than U+2000.
|
||||
--
|
||||
-- trivial cases:
|
||||
SELECT 'aⓐ' ~ U&'a\24D0' AS t;
|
||||
t
|
||||
---
|
||||
t
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⓐ' ~ U&'a\24D1' AS f;
|
||||
f
|
||||
---
|
||||
f
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⓕ' ~ 'a[ⓐ-ⓩ]' AS t;
|
||||
t
|
||||
---
|
||||
t
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⒻ' ~ 'a[ⓐ-ⓩ]' AS f;
|
||||
f
|
||||
---
|
||||
f
|
||||
(1 row)
|
||||
|
||||
-- cases requiring splitting of ranges:
|
||||
SELECT 'aⓕⓕ' ~ 'aⓕ[ⓐ-ⓩ]' AS t;
|
||||
t
|
||||
---
|
||||
t
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⓕⓐ' ~ 'aⓕ[ⓐ-ⓩ]' AS t;
|
||||
t
|
||||
---
|
||||
t
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⓐⓕ' ~ 'aⓕ[ⓐ-ⓩ]' AS f;
|
||||
f
|
||||
---
|
||||
f
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⓕⓕ' ~ 'a[ⓐ-ⓩ]ⓕ' AS t;
|
||||
t
|
||||
---
|
||||
t
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⓕⓐ' ~ 'a[ⓐ-ⓩ]ⓕ' AS f;
|
||||
f
|
||||
---
|
||||
f
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⓐⓕ' ~ 'a[ⓐ-ⓩ]ⓕ' AS t;
|
||||
t
|
||||
---
|
||||
t
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⒶⓜ' ~ 'a[Ⓐ-ⓜ][ⓜ-ⓩ]' AS t;
|
||||
t
|
||||
---
|
||||
t
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⓜⓜ' ~ 'a[Ⓐ-ⓜ][ⓜ-ⓩ]' AS t;
|
||||
t
|
||||
---
|
||||
t
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⓜⓩ' ~ 'a[Ⓐ-ⓜ][ⓜ-ⓩ]' AS t;
|
||||
t
|
||||
---
|
||||
t
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⓩⓩ' ~ 'a[Ⓐ-ⓜ][ⓜ-ⓩ]' AS f;
|
||||
f
|
||||
---
|
||||
f
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⓜ⓪' ~ 'a[Ⓐ-ⓜ][ⓜ-ⓩ]' AS f;
|
||||
f
|
||||
---
|
||||
f
|
||||
(1 row)
|
||||
|
||||
SELECT 'a0' ~ 'a[a-ⓩ]' AS f;
|
||||
f
|
||||
---
|
||||
f
|
||||
(1 row)
|
||||
|
||||
SELECT 'aq' ~ 'a[a-ⓩ]' AS t;
|
||||
t
|
||||
---
|
||||
t
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⓜ' ~ 'a[a-ⓩ]' AS t;
|
||||
t
|
||||
---
|
||||
t
|
||||
(1 row)
|
||||
|
||||
SELECT 'a⓪' ~ 'a[a-ⓩ]' AS f;
|
||||
f
|
||||
---
|
||||
f
|
||||
(1 row)
|
||||
|
||||
-- Locale-dependent character classes
|
||||
SELECT 'aⒶⓜ⓪' ~ '[[:alpha:]][[:alpha:]][[:alpha:]][[:graph:]]' AS t;
|
||||
t
|
||||
---
|
||||
t
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⒶⓜ⓪' ~ '[[:alpha:]][[:alpha:]][[:alpha:]][[:alpha:]]' AS f;
|
||||
f
|
||||
---
|
||||
f
|
||||
(1 row)
|
||||
|
||||
-- Locale-dependent character classes with high ranges
|
||||
SELECT 'aⒶⓜ⓪' ~ '[a-z][[:alpha:]][ⓐ-ⓩ][[:graph:]]' AS t;
|
||||
t
|
||||
---
|
||||
t
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⓜⒶ⓪' ~ '[a-z][[:alpha:]][ⓐ-ⓩ][[:graph:]]' AS f;
|
||||
f
|
||||
---
|
||||
f
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⓜⒶ⓪' ~ '[a-z][ⓐ-ⓩ][[:alpha:]][[:graph:]]' AS t;
|
||||
t
|
||||
---
|
||||
t
|
||||
(1 row)
|
||||
|
||||
SELECT 'aⒶⓜ⓪' ~ '[a-z][ⓐ-ⓩ][[:alpha:]][[:graph:]]' AS f;
|
||||
f
|
||||
---
|
||||
f
|
||||
(1 row)
|
||||
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
/*
|
||||
* This test is for Linux/glibc systems and others that implement proper
|
||||
* locale classification of Unicode characters with high code values.
|
||||
* It must be run in a database with UTF8 encoding and a Unicode-aware locale.
|
||||
*/
|
||||
|
||||
SET client_encoding TO UTF8;
|
||||
|
||||
--
|
||||
-- Test the "high colormap" logic with single characters and ranges that
|
||||
-- exceed the MAX_SIMPLE_CHR cutoff, here assumed to be less than U+2000.
|
||||
--
|
||||
|
||||
-- trivial cases:
|
||||
SELECT 'aⓐ' ~ U&'a\24D0' AS t;
|
||||
SELECT 'aⓐ' ~ U&'a\24D1' AS f;
|
||||
SELECT 'aⓕ' ~ 'a[ⓐ-ⓩ]' AS t;
|
||||
SELECT 'aⒻ' ~ 'a[ⓐ-ⓩ]' AS f;
|
||||
-- cases requiring splitting of ranges:
|
||||
SELECT 'aⓕⓕ' ~ 'aⓕ[ⓐ-ⓩ]' AS t;
|
||||
SELECT 'aⓕⓐ' ~ 'aⓕ[ⓐ-ⓩ]' AS t;
|
||||
SELECT 'aⓐⓕ' ~ 'aⓕ[ⓐ-ⓩ]' AS f;
|
||||
SELECT 'aⓕⓕ' ~ 'a[ⓐ-ⓩ]ⓕ' AS t;
|
||||
SELECT 'aⓕⓐ' ~ 'a[ⓐ-ⓩ]ⓕ' AS f;
|
||||
SELECT 'aⓐⓕ' ~ 'a[ⓐ-ⓩ]ⓕ' AS t;
|
||||
SELECT 'aⒶⓜ' ~ 'a[Ⓐ-ⓜ][ⓜ-ⓩ]' AS t;
|
||||
SELECT 'aⓜⓜ' ~ 'a[Ⓐ-ⓜ][ⓜ-ⓩ]' AS t;
|
||||
SELECT 'aⓜⓩ' ~ 'a[Ⓐ-ⓜ][ⓜ-ⓩ]' AS t;
|
||||
SELECT 'aⓩⓩ' ~ 'a[Ⓐ-ⓜ][ⓜ-ⓩ]' AS f;
|
||||
SELECT 'aⓜ⓪' ~ 'a[Ⓐ-ⓜ][ⓜ-ⓩ]' AS f;
|
||||
SELECT 'a0' ~ 'a[a-ⓩ]' AS f;
|
||||
SELECT 'aq' ~ 'a[a-ⓩ]' AS t;
|
||||
SELECT 'aⓜ' ~ 'a[a-ⓩ]' AS t;
|
||||
SELECT 'a⓪' ~ 'a[a-ⓩ]' AS f;
|
||||
|
||||
-- Locale-dependent character classes
|
||||
|
||||
SELECT 'aⒶⓜ⓪' ~ '[[:alpha:]][[:alpha:]][[:alpha:]][[:graph:]]' AS t;
|
||||
SELECT 'aⒶⓜ⓪' ~ '[[:alpha:]][[:alpha:]][[:alpha:]][[:alpha:]]' AS f;
|
||||
|
||||
-- Locale-dependent character classes with high ranges
|
||||
|
||||
SELECT 'aⒶⓜ⓪' ~ '[a-z][[:alpha:]][ⓐ-ⓩ][[:graph:]]' AS t;
|
||||
SELECT 'aⓜⒶ⓪' ~ '[a-z][[:alpha:]][ⓐ-ⓩ][[:graph:]]' AS f;
|
||||
SELECT 'aⓜⒶ⓪' ~ '[a-z][ⓐ-ⓩ][[:alpha:]][[:graph:]]' AS t;
|
||||
SELECT 'aⒶⓜ⓪' ~ '[a-z][ⓐ-ⓩ][[:alpha:]][[:graph:]]' AS f;
|
||||
Loading…
Reference in New Issue