Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
%top{
|
2004-02-19 20:40:09 +01:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
*
|
|
|
|
* psqlscan.l
|
2016-03-25 01:28:47 +01:00
|
|
|
* lexical scanner for SQL commands
|
2004-02-19 20:40:09 +01:00
|
|
|
*
|
2016-03-25 01:28:47 +01:00
|
|
|
* This lexer used to be part of psql, and that heritage is reflected in
|
|
|
|
* the file name as well as function and typedef names, though it can now
|
|
|
|
* be used by other frontend programs as well. It's also possible to extend
|
|
|
|
* this lexer with a compatible add-on lexer to handle program-specific
|
|
|
|
* backslash commands.
|
|
|
|
*
|
|
|
|
* This code is mainly concerned with determining where the end of a SQL
|
|
|
|
* statement is: we are looking for semicolons that are not within quotes,
|
|
|
|
* comments, or parentheses. The most reliable way to handle this is to
|
|
|
|
* borrow the backend's flex lexer rules, lock, stock, and barrel. The rules
|
|
|
|
* below are (except for a few) the same as the backend's, but their actions
|
|
|
|
* are just ECHO whereas the backend's actions generally do other things.
|
2004-02-19 20:40:09 +01:00
|
|
|
*
|
2005-05-26 03:24:29 +02:00
|
|
|
* XXX The rules in this file must be kept in sync with the backend lexer!!!
|
|
|
|
*
|
|
|
|
* XXX Avoid creating backtracking cases --- see the backend lexer for info.
|
2004-02-19 20:40:09 +01:00
|
|
|
*
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
* See psqlscan_int.h for additional commentary.
|
2004-02-19 20:40:09 +01:00
|
|
|
*
|
2018-11-13 18:57:52 +01:00
|
|
|
*
|
2023-01-02 21:00:37 +01:00
|
|
|
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
|
2004-02-19 20:40:09 +01:00
|
|
|
* Portions Copyright (c) 1994, Regents of the University of California
|
|
|
|
*
|
|
|
|
* IDENTIFICATION
|
2016-03-25 01:28:47 +01:00
|
|
|
* src/fe_utils/psqlscan.l
|
2004-02-19 20:40:09 +01:00
|
|
|
*
|
|
|
|
*-------------------------------------------------------------------------
|
|
|
|
*/
|
|
|
|
#include "postgres_fe.h"
|
|
|
|
|
2019-05-14 20:19:49 +02:00
|
|
|
#include "common/logging.h"
|
2016-03-25 01:28:47 +01:00
|
|
|
#include "fe_utils/psqlscan.h"
|
2004-02-19 20:40:09 +01:00
|
|
|
|
2016-03-18 20:05:49 +01:00
|
|
|
#include "libpq-fe.h"
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
}
|
2004-02-19 20:40:09 +01:00
|
|
|
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
%{
|
2018-11-13 18:57:52 +01:00
|
|
|
|
|
|
|
/* LCOV_EXCL_START */
|
|
|
|
|
2016-03-25 01:28:47 +01:00
|
|
|
#include "fe_utils/psqlscan_int.h"
|
2004-02-19 20:40:09 +01:00
|
|
|
|
2016-03-21 02:59:03 +01:00
|
|
|
/*
|
|
|
|
* We must have a typedef YYSTYPE for yylex's first argument, but this lexer
|
|
|
|
* doesn't presently make use of that argument, so just declare it as int.
|
|
|
|
*/
|
|
|
|
typedef int YYSTYPE;
|
|
|
|
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
/*
|
|
|
|
* Set the type of yyextra; we use it as a pointer back to the containing
|
|
|
|
* PsqlScanState.
|
|
|
|
*/
|
|
|
|
#define YY_EXTRA_TYPE PsqlScanState
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
|
|
|
|
/* Return values from yylex() */
|
|
|
|
#define LEXRES_EOL 0 /* end of input */
|
|
|
|
#define LEXRES_SEMI 1 /* command-terminating semicolon found */
|
|
|
|
#define LEXRES_BACKSLASH 2 /* backslash command start */
|
|
|
|
|
|
|
|
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
#define ECHO psqlscan_emit(cur_state, yytext, yyleng)
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Work around a bug in flex 2.5.35: it emits a couple of functions that
|
|
|
|
* it forgets to emit declarations for. Since we use -Wmissing-prototypes,
|
|
|
|
* this would cause warnings. Providing our own declarations should be
|
|
|
|
* harmless even when the bug gets fixed.
|
|
|
|
*/
|
|
|
|
extern int psql_yyget_column(yyscan_t yyscanner);
|
|
|
|
extern void psql_yyset_column(int column_no, yyscan_t yyscanner);
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
%}
|
|
|
|
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
%option reentrant
|
2016-03-21 02:59:03 +01:00
|
|
|
%option bison-bridge
|
2004-02-19 20:40:09 +01:00
|
|
|
%option 8bit
|
|
|
|
%option never-interactive
|
2004-02-24 22:45:18 +01:00
|
|
|
%option nodefault
|
2008-05-09 17:36:31 +02:00
|
|
|
%option noinput
|
2004-02-19 20:40:09 +01:00
|
|
|
%option nounput
|
|
|
|
%option noyywrap
|
2011-08-25 19:55:57 +02:00
|
|
|
%option warn
|
2016-03-18 20:05:49 +01:00
|
|
|
%option prefix="psql_yy"
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* All of the following definitions and rules should exactly match
|
|
|
|
* src/backend/parser/scan.l so far as the flex patterns are concerned.
|
|
|
|
* The rule bodies are just ECHO as opposed to what the backend does,
|
|
|
|
* however. (But be sure to duplicate code that affects the lexing process,
|
2016-03-19 19:36:22 +01:00
|
|
|
* such as BEGIN() and yyless().) Also, psqlscan uses a single <<EOF>> rule
|
|
|
|
* whereas scan.l has a separate one for each exclusive state.
|
2004-02-19 20:40:09 +01:00
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
|
|
|
* OK, here is a short description of lex/flex rules behavior.
|
|
|
|
* The longest pattern which matches an input string is always chosen.
|
|
|
|
* For equal-length patterns, the first occurring in the rules list is chosen.
|
|
|
|
* INITIAL is the starting state, to which all non-conditional rules apply.
|
|
|
|
* Exclusive states change parsing rules while the state is active. When in
|
|
|
|
* an exclusive state, only those rules defined for that state apply.
|
|
|
|
*
|
|
|
|
* We use exclusive states for quoted strings, extended comments,
|
|
|
|
* and to eliminate parsing troubles for numeric strings.
|
|
|
|
* Exclusive states:
|
|
|
|
* <xb> bit string literal
|
|
|
|
* <xc> extended C-style comments
|
|
|
|
* <xd> delimited identifiers (double-quoted identifiers)
|
2021-11-24 09:10:32 +01:00
|
|
|
* <xh> hexadecimal byte string
|
2006-03-06 20:49:20 +01:00
|
|
|
* <xq> standard quoted strings
|
Reduce size of backend scanner's tables.
Previously, the core scanner's yy_transition[] array had 37045 elements.
Since that number is larger than INT16_MAX, Flex generated the array to
contain 32-bit integers. By reimplementing some of the bulkier scanner
rules, this patch reduces the array to 20495 elements. The much smaller
total length, combined with the consequent use of 16-bit integers for
the array elements reduces the binary size by over 200kB. This was
accomplished in two ways:
1. Consolidate handling of quote continuations into a new start condition,
rather than duplicating that logic for five different string types.
2. Treat Unicode strings and identifiers followed by a UESCAPE sequence
as three separate tokens, rather than one. The logic to de-escape
Unicode strings is moved to the filter code in parser.c, which already
had the ability to provide special processing for token sequences.
While we could have implemented the conversion in the grammar, that
approach was rejected for performance and maintainability reasons.
Performance in microbenchmarks of raw parsing seems equal or slightly
faster in most cases, and it's reasonable to expect that in real-world
usage (with more competition for the CPU cache) there will be a larger
win. The exception is UESCAPE sequences; lexing those is about 10%
slower, primarily because the scanner now has to be called three times
rather than one. This seems acceptable since that feature is very
rarely used.
The psql and epcg lexers are likewise modified, primarily because we
want to keep them all in sync. Since those lexers don't use the
space-hogging -CF option, the space savings is much less, but it's
still good for perhaps 10kB apiece.
While at it, merge the ecpg lexer's handling of C-style comments used
in SQL and in C. Those have different rules regarding nested comments,
but since we already have the ability to keep track of the previous
start condition, we can use that to handle both cases within a single
start condition. This matches the core scanner more closely.
John Naylor
Discussion: https://postgr.es/m/CACPNZCvaoa3EgVWm5yZhcSTX6RAtaLgniCPcBVOCwm8h3xpWkw@mail.gmail.com
2020-01-13 21:04:31 +01:00
|
|
|
* <xqs> quote stop (detect continued strings)
|
2006-03-06 20:49:20 +01:00
|
|
|
* <xe> extended quoted strings (support backslash escape sequences)
|
2004-02-24 22:45:18 +01:00
|
|
|
* <xdolq> $foo$ quoted strings
|
2008-10-29 09:04:54 +01:00
|
|
|
* <xui> quoted identifier with Unicode escapes
|
2014-02-04 01:47:57 +01:00
|
|
|
* <xus> quoted string with Unicode escapes
|
2010-10-27 04:23:04 +02:00
|
|
|
*
|
|
|
|
* Note: we intentionally don't mimic the backend's <xeu> state; we have
|
|
|
|
* no need to distinguish it from <xe> state, and no good way to get out
|
|
|
|
* of it in error cases. The backend just throws yyerror() in those
|
|
|
|
* cases, but that's not an option here.
|
2004-02-19 20:40:09 +01:00
|
|
|
*/
|
|
|
|
|
|
|
|
%x xb
|
|
|
|
%x xc
|
|
|
|
%x xd
|
|
|
|
%x xh
|
|
|
|
%x xq
|
Reduce size of backend scanner's tables.
Previously, the core scanner's yy_transition[] array had 37045 elements.
Since that number is larger than INT16_MAX, Flex generated the array to
contain 32-bit integers. By reimplementing some of the bulkier scanner
rules, this patch reduces the array to 20495 elements. The much smaller
total length, combined with the consequent use of 16-bit integers for
the array elements reduces the binary size by over 200kB. This was
accomplished in two ways:
1. Consolidate handling of quote continuations into a new start condition,
rather than duplicating that logic for five different string types.
2. Treat Unicode strings and identifiers followed by a UESCAPE sequence
as three separate tokens, rather than one. The logic to de-escape
Unicode strings is moved to the filter code in parser.c, which already
had the ability to provide special processing for token sequences.
While we could have implemented the conversion in the grammar, that
approach was rejected for performance and maintainability reasons.
Performance in microbenchmarks of raw parsing seems equal or slightly
faster in most cases, and it's reasonable to expect that in real-world
usage (with more competition for the CPU cache) there will be a larger
win. The exception is UESCAPE sequences; lexing those is about 10%
slower, primarily because the scanner now has to be called three times
rather than one. This seems acceptable since that feature is very
rarely used.
The psql and epcg lexers are likewise modified, primarily because we
want to keep them all in sync. Since those lexers don't use the
space-hogging -CF option, the space savings is much less, but it's
still good for perhaps 10kB apiece.
While at it, merge the ecpg lexer's handling of C-style comments used
in SQL and in C. Those have different rules regarding nested comments,
but since we already have the ability to keep track of the previous
start condition, we can use that to handle both cases within a single
start condition. This matches the core scanner more closely.
John Naylor
Discussion: https://postgr.es/m/CACPNZCvaoa3EgVWm5yZhcSTX6RAtaLgniCPcBVOCwm8h3xpWkw@mail.gmail.com
2020-01-13 21:04:31 +01:00
|
|
|
%x xqs
|
2018-11-13 18:57:52 +01:00
|
|
|
%x xe
|
2004-02-24 22:45:18 +01:00
|
|
|
%x xdolq
|
2008-10-29 09:04:54 +01:00
|
|
|
%x xui
|
|
|
|
%x xus
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* In order to make the world safe for Windows and Mac clients as well as
|
|
|
|
* Unix ones, we accept either \n or \r as a newline. A DOS-style \r\n
|
|
|
|
* sequence will be seen as two successive newlines, but that doesn't cause
|
|
|
|
* any problems. Comments that start with -- and extend to the next
|
|
|
|
* newline are treated as equivalent to a single whitespace character.
|
|
|
|
*
|
|
|
|
* NOTE a fine point: if there is no newline following --, we will absorb
|
|
|
|
* everything to the end of the input as a comment. This is correct. Older
|
|
|
|
* versions of Postgres failed to recognize -- as a comment if the input
|
|
|
|
* did not end with a newline.
|
|
|
|
*
|
|
|
|
* XXX perhaps \f (formfeed) should be treated as a newline as well?
|
2009-09-27 05:27:24 +02:00
|
|
|
*
|
|
|
|
* XXX if you change the set of whitespace characters, fix scanner_isspace()
|
2018-11-13 18:57:52 +01:00
|
|
|
* to agree.
|
2004-02-19 20:40:09 +01:00
|
|
|
*/
|
|
|
|
|
|
|
|
space [ \t\n\r\f]
|
|
|
|
horiz_space [ \t\f]
|
|
|
|
newline [\n\r]
|
|
|
|
non_newline [^\n\r]
|
|
|
|
|
|
|
|
comment ("--"{non_newline}*)
|
|
|
|
|
|
|
|
whitespace ({space}+|{comment})
|
|
|
|
|
|
|
|
/*
|
|
|
|
* SQL requires at least one newline in the whitespace separating
|
|
|
|
* string literals that are to be concatenated. Silly, but who are we
|
|
|
|
* to argue? Note that {whitespace_with_newline} should not have * after
|
|
|
|
* it, whereas {whitespace} should generally have a * after it...
|
|
|
|
*/
|
|
|
|
|
|
|
|
special_whitespace ({space}+|{comment}{newline})
|
|
|
|
horiz_whitespace ({horiz_space}|{comment})
|
|
|
|
whitespace_with_newline ({horiz_whitespace}*{newline}{special_whitespace}*)
|
|
|
|
|
Reduce size of backend scanner's tables.
Previously, the core scanner's yy_transition[] array had 37045 elements.
Since that number is larger than INT16_MAX, Flex generated the array to
contain 32-bit integers. By reimplementing some of the bulkier scanner
rules, this patch reduces the array to 20495 elements. The much smaller
total length, combined with the consequent use of 16-bit integers for
the array elements reduces the binary size by over 200kB. This was
accomplished in two ways:
1. Consolidate handling of quote continuations into a new start condition,
rather than duplicating that logic for five different string types.
2. Treat Unicode strings and identifiers followed by a UESCAPE sequence
as three separate tokens, rather than one. The logic to de-escape
Unicode strings is moved to the filter code in parser.c, which already
had the ability to provide special processing for token sequences.
While we could have implemented the conversion in the grammar, that
approach was rejected for performance and maintainability reasons.
Performance in microbenchmarks of raw parsing seems equal or slightly
faster in most cases, and it's reasonable to expect that in real-world
usage (with more competition for the CPU cache) there will be a larger
win. The exception is UESCAPE sequences; lexing those is about 10%
slower, primarily because the scanner now has to be called three times
rather than one. This seems acceptable since that feature is very
rarely used.
The psql and epcg lexers are likewise modified, primarily because we
want to keep them all in sync. Since those lexers don't use the
space-hogging -CF option, the space savings is much less, but it's
still good for perhaps 10kB apiece.
While at it, merge the ecpg lexer's handling of C-style comments used
in SQL and in C. Those have different rules regarding nested comments,
but since we already have the ability to keep track of the previous
start condition, we can use that to handle both cases within a single
start condition. This matches the core scanner more closely.
John Naylor
Discussion: https://postgr.es/m/CACPNZCvaoa3EgVWm5yZhcSTX6RAtaLgniCPcBVOCwm8h3xpWkw@mail.gmail.com
2020-01-13 21:04:31 +01:00
|
|
|
quote '
|
|
|
|
/* If we see {quote} then {quotecontinue}, the quoted string continues */
|
|
|
|
quotecontinue {whitespace_with_newline}{quote}
|
|
|
|
|
2005-05-26 03:24:29 +02:00
|
|
|
/*
|
Reduce size of backend scanner's tables.
Previously, the core scanner's yy_transition[] array had 37045 elements.
Since that number is larger than INT16_MAX, Flex generated the array to
contain 32-bit integers. By reimplementing some of the bulkier scanner
rules, this patch reduces the array to 20495 elements. The much smaller
total length, combined with the consequent use of 16-bit integers for
the array elements reduces the binary size by over 200kB. This was
accomplished in two ways:
1. Consolidate handling of quote continuations into a new start condition,
rather than duplicating that logic for five different string types.
2. Treat Unicode strings and identifiers followed by a UESCAPE sequence
as three separate tokens, rather than one. The logic to de-escape
Unicode strings is moved to the filter code in parser.c, which already
had the ability to provide special processing for token sequences.
While we could have implemented the conversion in the grammar, that
approach was rejected for performance and maintainability reasons.
Performance in microbenchmarks of raw parsing seems equal or slightly
faster in most cases, and it's reasonable to expect that in real-world
usage (with more competition for the CPU cache) there will be a larger
win. The exception is UESCAPE sequences; lexing those is about 10%
slower, primarily because the scanner now has to be called three times
rather than one. This seems acceptable since that feature is very
rarely used.
The psql and epcg lexers are likewise modified, primarily because we
want to keep them all in sync. Since those lexers don't use the
space-hogging -CF option, the space savings is much less, but it's
still good for perhaps 10kB apiece.
While at it, merge the ecpg lexer's handling of C-style comments used
in SQL and in C. Those have different rules regarding nested comments,
but since we already have the ability to keep track of the previous
start condition, we can use that to handle both cases within a single
start condition. This matches the core scanner more closely.
John Naylor
Discussion: https://postgr.es/m/CACPNZCvaoa3EgVWm5yZhcSTX6RAtaLgniCPcBVOCwm8h3xpWkw@mail.gmail.com
2020-01-13 21:04:31 +01:00
|
|
|
* {quotecontinuefail} is needed to avoid lexer backup when we fail to match
|
|
|
|
* {quotecontinue}. It might seem that this could just be {whitespace}*,
|
|
|
|
* but if there's a dash after {whitespace_with_newline}, it must be consumed
|
|
|
|
* to see if there's another dash --- which would start a {comment} and thus
|
|
|
|
* allow continuation of the {quotecontinue} token.
|
2005-05-26 03:24:29 +02:00
|
|
|
*/
|
Reduce size of backend scanner's tables.
Previously, the core scanner's yy_transition[] array had 37045 elements.
Since that number is larger than INT16_MAX, Flex generated the array to
contain 32-bit integers. By reimplementing some of the bulkier scanner
rules, this patch reduces the array to 20495 elements. The much smaller
total length, combined with the consequent use of 16-bit integers for
the array elements reduces the binary size by over 200kB. This was
accomplished in two ways:
1. Consolidate handling of quote continuations into a new start condition,
rather than duplicating that logic for five different string types.
2. Treat Unicode strings and identifiers followed by a UESCAPE sequence
as three separate tokens, rather than one. The logic to de-escape
Unicode strings is moved to the filter code in parser.c, which already
had the ability to provide special processing for token sequences.
While we could have implemented the conversion in the grammar, that
approach was rejected for performance and maintainability reasons.
Performance in microbenchmarks of raw parsing seems equal or slightly
faster in most cases, and it's reasonable to expect that in real-world
usage (with more competition for the CPU cache) there will be a larger
win. The exception is UESCAPE sequences; lexing those is about 10%
slower, primarily because the scanner now has to be called three times
rather than one. This seems acceptable since that feature is very
rarely used.
The psql and epcg lexers are likewise modified, primarily because we
want to keep them all in sync. Since those lexers don't use the
space-hogging -CF option, the space savings is much less, but it's
still good for perhaps 10kB apiece.
While at it, merge the ecpg lexer's handling of C-style comments used
in SQL and in C. Those have different rules regarding nested comments,
but since we already have the ability to keep track of the previous
start condition, we can use that to handle both cases within a single
start condition. This matches the core scanner more closely.
John Naylor
Discussion: https://postgr.es/m/CACPNZCvaoa3EgVWm5yZhcSTX6RAtaLgniCPcBVOCwm8h3xpWkw@mail.gmail.com
2020-01-13 21:04:31 +01:00
|
|
|
quotecontinuefail {whitespace}*"-"?
|
2005-05-26 03:24:29 +02:00
|
|
|
|
2004-02-19 20:40:09 +01:00
|
|
|
/* Bit string
|
|
|
|
* It is tempting to scan the string for only those characters
|
|
|
|
* which are allowed. However, this leads to silently swallowed
|
|
|
|
* characters if illegal characters are included in the string.
|
|
|
|
* For example, if xbinside is [01] then B'ABCD' is interpreted
|
|
|
|
* as a zero-length string, and the ABCD' is lost!
|
|
|
|
* Better to pass the string forward and let the input routines
|
|
|
|
* validate the contents.
|
|
|
|
*/
|
|
|
|
xbstart [bB]{quote}
|
|
|
|
xbinside [^']*
|
|
|
|
|
2021-11-24 09:10:32 +01:00
|
|
|
/* Hexadecimal byte string */
|
2004-02-19 20:40:09 +01:00
|
|
|
xhstart [xX]{quote}
|
|
|
|
xhinside [^']*
|
|
|
|
|
2005-06-26 21:16:07 +02:00
|
|
|
/* National character */
|
2004-02-19 20:40:09 +01:00
|
|
|
xnstart [nN]{quote}
|
|
|
|
|
2005-06-26 21:16:07 +02:00
|
|
|
/* Quoted string that allows backslash escapes */
|
|
|
|
xestart [eE]{quote}
|
2006-03-06 20:49:20 +01:00
|
|
|
xeinside [^\\']+
|
|
|
|
xeescape [\\][^0-7]
|
|
|
|
xeoctesc [\\][0-7]{1,3}
|
|
|
|
xehexesc [\\]x[0-9A-Fa-f]{1,2}
|
2009-09-27 05:27:24 +02:00
|
|
|
xeunicode [\\](u[0-9A-Fa-f]{4}|U[0-9A-Fa-f]{8})
|
|
|
|
xeunicodefail [\\](u[0-9A-Fa-f]{0,3}|U[0-9A-Fa-f]{0,7})
|
2005-06-26 21:16:07 +02:00
|
|
|
|
2004-02-19 20:40:09 +01:00
|
|
|
/* Extended quote
|
2005-06-26 21:16:07 +02:00
|
|
|
* xqdouble implements embedded quote, ''''
|
2004-02-19 20:40:09 +01:00
|
|
|
*/
|
|
|
|
xqstart {quote}
|
|
|
|
xqdouble {quote}{quote}
|
2006-03-06 20:49:20 +01:00
|
|
|
xqinside [^']+
|
2004-02-19 20:40:09 +01:00
|
|
|
|
2004-02-24 22:45:18 +01:00
|
|
|
/* $foo$ style quotes ("dollar quoting")
|
|
|
|
* The quoted string starts with $foo$ where "foo" is an optional string
|
2010-11-23 21:27:50 +01:00
|
|
|
* in the form of an identifier, except that it may not contain "$",
|
|
|
|
* and extends to the first occurrence of an identical string.
|
2004-02-24 22:45:18 +01:00
|
|
|
* There is *no* processing of the quoted text.
|
2005-05-26 03:24:29 +02:00
|
|
|
*
|
|
|
|
* {dolqfailed} is an error rule to avoid scanner backup when {dolqdelim}
|
|
|
|
* fails to match its trailing "$".
|
2004-02-24 22:45:18 +01:00
|
|
|
*/
|
|
|
|
dolq_start [A-Za-z\200-\377_]
|
|
|
|
dolq_cont [A-Za-z\200-\377_0-9]
|
|
|
|
dolqdelim \$({dolq_start}{dolq_cont}*)?\$
|
2005-05-26 03:24:29 +02:00
|
|
|
dolqfailed \${dolq_start}{dolq_cont}*
|
2004-02-24 22:45:18 +01:00
|
|
|
dolqinside [^$]+
|
|
|
|
|
2004-02-19 20:40:09 +01:00
|
|
|
/* Double quote
|
|
|
|
* Allows embedded spaces and other special characters into identifiers.
|
|
|
|
*/
|
|
|
|
dquote \"
|
|
|
|
xdstart {dquote}
|
|
|
|
xdstop {dquote}
|
|
|
|
xddouble {dquote}{dquote}
|
|
|
|
xdinside [^"]+
|
|
|
|
|
2008-10-29 09:04:54 +01:00
|
|
|
/* Quoted identifier with Unicode escapes */
|
|
|
|
xuistart [uU]&{dquote}
|
|
|
|
|
|
|
|
/* Quoted string with Unicode escapes */
|
|
|
|
xusstart [uU]&{quote}
|
2013-03-14 19:31:27 +01:00
|
|
|
|
2008-10-29 09:04:54 +01:00
|
|
|
/* error rule to avoid backup */
|
|
|
|
xufailed [uU]&
|
|
|
|
|
|
|
|
|
2004-02-19 20:40:09 +01:00
|
|
|
/* C-style comments
|
|
|
|
*
|
|
|
|
* The "extended comment" syntax closely resembles allowable operator syntax.
|
|
|
|
* The tricky part here is to get lex to recognize a string starting with
|
|
|
|
* slash-star as a comment, when interpreting it as an operator would produce
|
|
|
|
* a longer match --- remember lex will prefer a longer match! Also, if we
|
|
|
|
* have something like plus-slash-star, lex will think this is a 3-character
|
|
|
|
* operator whereas we want to see it as a + operator and a comment start.
|
|
|
|
* The solution is two-fold:
|
|
|
|
* 1. append {op_chars}* to xcstart so that it matches as much text as
|
|
|
|
* {operator} would. Then the tie-breaker (first matching rule of same
|
|
|
|
* length) ensures xcstart wins. We put back the extra stuff with yyless()
|
|
|
|
* in case it contains a star-slash that should terminate the comment.
|
|
|
|
* 2. In the operator rule, check for slash-star within the operator, and
|
|
|
|
* if found throw it back with yyless(). This handles the plus-slash-star
|
|
|
|
* problem.
|
|
|
|
* Dash-dash comments have similar interactions with the operator rule.
|
|
|
|
*/
|
|
|
|
xcstart \/\*{op_chars}*
|
|
|
|
xcstop \*+\/
|
|
|
|
xcinside [^*/]+
|
|
|
|
|
|
|
|
ident_start [A-Za-z\200-\377_]
|
|
|
|
ident_cont [A-Za-z\200-\377_0-9\$]
|
|
|
|
|
|
|
|
identifier {ident_start}{ident_cont}*
|
|
|
|
|
Make operator precedence follow the SQL standard more closely.
While the SQL standard is pretty vague on the overall topic of operator
precedence (because it never presents a unified BNF for all expressions),
it does seem reasonable to conclude from the spec for <boolean value
expression> that OR has the lowest precedence, then AND, then NOT, then IS
tests, then the six standard comparison operators, then everything else
(since any non-boolean operator in a WHERE clause would need to be an
argument of one of these).
We were only sort of on board with that: most notably, while "<" ">" and
"=" had properly low precedence, "<=" ">=" and "<>" were treated as generic
operators and so had significantly higher precedence. And "IS" tests were
even higher precedence than those, which is very clearly wrong per spec.
Another problem was that "foo NOT SOMETHING bar" constructs, such as
"x NOT LIKE y", were treated inconsistently because of a bison
implementation artifact: they had the documented precedence with respect
to operators to their right, but behaved like NOT (i.e., very low priority)
with respect to operators to their left.
Fixing the precedence issues is just a small matter of rearranging the
precedence declarations in gram.y, except for the NOT problem, which
requires adding an additional lookahead case in base_yylex() so that we
can attach a different token precedence to NOT LIKE and allied two-word
operators.
The bulk of this patch is not the bug fix per se, but adding logic to
parse_expr.c to allow giving warnings if an expression has changed meaning
because of these precedence changes. These warnings are off by default
and are enabled by the new GUC operator_precedence_warning. It's believed
that very few applications will be affected by these changes, but it was
agreed that a warning mechanism is essential to help debug any that are.
2015-03-11 18:22:52 +01:00
|
|
|
/* Assorted special-case operators and operator-like tokens */
|
2004-02-19 20:40:09 +01:00
|
|
|
typecast "::"
|
2009-09-27 05:27:24 +02:00
|
|
|
dot_dot \.\.
|
|
|
|
colon_equals ":="
|
Fix lexing of standard multi-character operators in edge cases.
Commits c6b3c939b (which fixed the precedence of >=, <=, <> operators)
and 865f14a2d (which added support for the standard => notation for
named arguments) created a class of lexer tokens which look like
multi-character operators but which have their own token IDs distinct
from Op. However, longest-match rules meant that following any of
these tokens with another operator character, as in (1<>-1), would
cause them to be incorrectly returned as Op.
The error here isn't immediately obvious, because the parser would
usually still find the correct operator via the Op token, but there
were more subtle problems:
1. If immediately followed by a comment or +-, >= <= <> would be given
the old precedence of Op rather than the correct new precedence;
2. If followed by a comment, != would be returned as Op rather than as
NOT_EQUAL, causing it not to be found at all;
3. If followed by a comment or +-, the => token for named arguments
would be lexed as Op, causing the argument to be mis-parsed as a
simple expression, usually causing an error.
Fix by explicitly checking for the operators in the {operator} code
block in addition to all the existing special cases there.
Backpatch to 9.5 where the problem was introduced.
Analysis and patch by me; review by Tom Lane.
Discussion: https://postgr.es/m/87va851ppl.fsf@news-spur.riddles.org.uk
2018-08-23 19:29:18 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* These operator-like tokens (unlike the above ones) also match the {operator}
|
|
|
|
* rule, which means that they might be overridden by a longer match if they
|
|
|
|
* are followed by a comment start or a + or - character. Accordingly, if you
|
|
|
|
* add to this list, you must also add corresponding code to the {operator}
|
|
|
|
* block to return the correct token in such cases. (This is not needed in
|
|
|
|
* psqlscan.l since the token value is ignored there.)
|
|
|
|
*/
|
2015-03-10 16:48:34 +01:00
|
|
|
equals_greater "=>"
|
Make operator precedence follow the SQL standard more closely.
While the SQL standard is pretty vague on the overall topic of operator
precedence (because it never presents a unified BNF for all expressions),
it does seem reasonable to conclude from the spec for <boolean value
expression> that OR has the lowest precedence, then AND, then NOT, then IS
tests, then the six standard comparison operators, then everything else
(since any non-boolean operator in a WHERE clause would need to be an
argument of one of these).
We were only sort of on board with that: most notably, while "<" ">" and
"=" had properly low precedence, "<=" ">=" and "<>" were treated as generic
operators and so had significantly higher precedence. And "IS" tests were
even higher precedence than those, which is very clearly wrong per spec.
Another problem was that "foo NOT SOMETHING bar" constructs, such as
"x NOT LIKE y", were treated inconsistently because of a bison
implementation artifact: they had the documented precedence with respect
to operators to their right, but behaved like NOT (i.e., very low priority)
with respect to operators to their left.
Fixing the precedence issues is just a small matter of rearranging the
precedence declarations in gram.y, except for the NOT problem, which
requires adding an additional lookahead case in base_yylex() so that we
can attach a different token precedence to NOT LIKE and allied two-word
operators.
The bulk of this patch is not the bug fix per se, but adding logic to
parse_expr.c to allow giving warnings if an expression has changed meaning
because of these precedence changes. These warnings are off by default
and are enabled by the new GUC operator_precedence_warning. It's believed
that very few applications will be affected by these changes, but it was
agreed that a warning mechanism is essential to help debug any that are.
2015-03-11 18:22:52 +01:00
|
|
|
less_equals "<="
|
|
|
|
greater_equals ">="
|
|
|
|
less_greater "<>"
|
|
|
|
not_equals "!="
|
2009-09-27 05:27:24 +02:00
|
|
|
|
2004-02-19 20:40:09 +01:00
|
|
|
/*
|
|
|
|
* "self" is the set of chars that should be returned as single-character
|
|
|
|
* tokens. "op_chars" is the set of chars that can make up "Op" tokens,
|
|
|
|
* which can be one or more characters long (but if a single-char token
|
|
|
|
* appears in the "self" set, it is not to be returned as an Op). Note
|
|
|
|
* that the sets overlap, but each has some chars that are not in the other.
|
|
|
|
*
|
|
|
|
* If you change either set, adjust the character lists appearing in the
|
|
|
|
* rule for "operator"!
|
|
|
|
*/
|
|
|
|
self [,()\[\].;\:\+\-\*\/\%\^\<\>\=]
|
|
|
|
op_chars [\~\!\@\#\^\&\|\`\?\+\-\*\/\%\<\>\=]
|
|
|
|
operator {op_chars}+
|
|
|
|
|
2021-11-24 09:10:32 +01:00
|
|
|
/*
|
|
|
|
* Numbers
|
|
|
|
*
|
|
|
|
* Unary minus is not part of a number here. Instead we pass it separately to
|
|
|
|
* the parser, and there it gets coerced via doNegate().
|
2005-05-26 03:24:29 +02:00
|
|
|
*
|
2022-12-14 05:40:38 +01:00
|
|
|
* {numericfail} is used because we would like "1..10" to lex as 1, dot_dot, 10.
|
2009-11-12 01:13:00 +01:00
|
|
|
*
|
2022-02-16 10:32:36 +01:00
|
|
|
* {realfail} is added to prevent the need for scanner
|
2005-05-26 03:24:29 +02:00
|
|
|
* backup when the {real} rule fails to match completely.
|
2004-02-19 20:40:09 +01:00
|
|
|
*/
|
2022-12-14 05:40:38 +01:00
|
|
|
decdigit [0-9]
|
|
|
|
hexdigit [0-9A-Fa-f]
|
|
|
|
octdigit [0-7]
|
|
|
|
bindigit [0-1]
|
|
|
|
|
2023-02-04 10:48:51 +01:00
|
|
|
decinteger {decdigit}(_?{decdigit})*
|
|
|
|
hexinteger 0[xX](_?{hexdigit})+
|
|
|
|
octinteger 0[oO](_?{octdigit})+
|
|
|
|
bininteger 0[bB](_?{bindigit})+
|
2022-12-14 05:40:38 +01:00
|
|
|
|
2023-02-04 10:48:51 +01:00
|
|
|
hexfail 0[xX]_?
|
|
|
|
octfail 0[oO]_?
|
|
|
|
binfail 0[bB]_?
|
2022-12-14 05:40:38 +01:00
|
|
|
|
|
|
|
numeric (({decinteger}\.{decinteger}?)|(\.{decinteger}))
|
|
|
|
numericfail {decdigit}+\.\.
|
|
|
|
|
2023-02-04 10:48:51 +01:00
|
|
|
real ({decinteger}|{numeric})[Ee][-+]?{decinteger}
|
2022-12-14 05:40:38 +01:00
|
|
|
realfail ({decinteger}|{numeric})[Ee][-+]
|
|
|
|
|
|
|
|
decinteger_junk {decinteger}{ident_start}
|
|
|
|
hexinteger_junk {hexinteger}{ident_start}
|
|
|
|
octinteger_junk {octinteger}{ident_start}
|
|
|
|
bininteger_junk {bininteger}{ident_start}
|
|
|
|
numeric_junk {numeric}{ident_start}
|
2022-02-16 10:32:36 +01:00
|
|
|
real_junk {real}{ident_start}
|
2004-02-19 20:40:09 +01:00
|
|
|
|
2022-12-14 05:40:38 +01:00
|
|
|
param \${decinteger}
|
|
|
|
param_junk \${decinteger}{ident_start}
|
2004-02-19 20:40:09 +01:00
|
|
|
|
2011-08-26 16:41:31 +02:00
|
|
|
/* psql-specific: characters allowed in variable names */
|
|
|
|
variable_char [A-Za-z\200-\377_0-9]
|
|
|
|
|
2004-02-19 20:40:09 +01:00
|
|
|
other .
|
|
|
|
|
|
|
|
/*
|
2004-02-24 22:45:18 +01:00
|
|
|
* Dollar quoted strings are totally opaque, and no escaping is done on them.
|
|
|
|
* Other quoted strings must allow some special characters such as single-quote
|
2004-02-19 20:40:09 +01:00
|
|
|
* and newline.
|
|
|
|
* Embedded single-quotes are implemented both in the SQL standard
|
|
|
|
* style of two adjacent single quotes "''" and in the Postgres/Java style
|
|
|
|
* of escaped-quote "\'".
|
|
|
|
* Other embedded escaped characters are matched explicitly and the leading
|
|
|
|
* backslash is dropped from the string.
|
|
|
|
* Note that xcstart must appear before operator, as explained above!
|
|
|
|
* Also whitespace (comment) must appear before operator.
|
|
|
|
*/
|
|
|
|
|
|
|
|
%%
|
|
|
|
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
%{
|
|
|
|
/* Declare some local variables inside yylex(), for convenience */
|
|
|
|
PsqlScanState cur_state = yyextra;
|
|
|
|
PQExpBuffer output_buf = cur_state->output_buf;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Force flex into the state indicated by start_state. This has a
|
2016-03-19 19:36:22 +01:00
|
|
|
* couple of purposes: it lets some of the functions below set a new
|
|
|
|
* starting state without ugly direct access to flex variables, and it
|
|
|
|
* allows us to transition from one flex lexer to another so that we
|
|
|
|
* can lex different parts of the source string using separate lexers.
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
*/
|
|
|
|
BEGIN(cur_state->start_state);
|
|
|
|
%}
|
|
|
|
|
2004-02-19 20:40:09 +01:00
|
|
|
{whitespace} {
|
|
|
|
/*
|
|
|
|
* Note that the whitespace rule includes both true
|
|
|
|
* whitespace and single-line ("--" style) comments.
|
2021-12-01 18:06:31 +01:00
|
|
|
* We suppress whitespace until we have collected some
|
|
|
|
* non-whitespace data. (This interacts with some
|
|
|
|
* decisions in MainLoop(); see there for details.)
|
2004-02-19 20:40:09 +01:00
|
|
|
*/
|
2021-12-01 18:06:31 +01:00
|
|
|
if (output_buf->len > 0)
|
2004-02-19 20:40:09 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
|
|
|
{xcstart} {
|
|
|
|
cur_state->xcdepth = 0;
|
|
|
|
BEGIN(xc);
|
|
|
|
/* Put back any characters past slash-star; see above */
|
|
|
|
yyless(2);
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
2018-11-13 18:57:52 +01:00
|
|
|
<xc>{
|
|
|
|
{xcstart} {
|
2004-02-19 20:40:09 +01:00
|
|
|
cur_state->xcdepth++;
|
|
|
|
/* Put back any characters past slash-star; see above */
|
|
|
|
yyless(2);
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
2018-11-13 18:57:52 +01:00
|
|
|
{xcstop} {
|
2004-02-19 20:40:09 +01:00
|
|
|
if (cur_state->xcdepth <= 0)
|
|
|
|
BEGIN(INITIAL);
|
|
|
|
else
|
|
|
|
cur_state->xcdepth--;
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
2018-11-13 18:57:52 +01:00
|
|
|
{xcinside} {
|
2004-02-19 20:40:09 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
2018-11-13 18:57:52 +01:00
|
|
|
{op_chars} {
|
2004-02-19 20:40:09 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
2018-11-13 18:57:52 +01:00
|
|
|
\*+ {
|
2005-05-26 03:24:29 +02:00
|
|
|
ECHO;
|
|
|
|
}
|
2018-11-13 18:57:52 +01:00
|
|
|
} /* <xc> */
|
2005-05-26 03:24:29 +02:00
|
|
|
|
2004-02-19 20:40:09 +01:00
|
|
|
{xbstart} {
|
|
|
|
BEGIN(xb);
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
<xh>{xhinside} |
|
|
|
|
<xb>{xbinside} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
|
|
|
{xhstart} {
|
|
|
|
/* Hexadecimal bit type.
|
|
|
|
* At some point we should simply pass the string
|
|
|
|
* forward to the parser and label it there.
|
|
|
|
* In the meantime, place a leading "x" on the string
|
|
|
|
* to mark it for the input routine as a hex string.
|
|
|
|
*/
|
|
|
|
BEGIN(xh);
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
|
|
|
{xnstart} {
|
2016-03-19 19:36:22 +01:00
|
|
|
yyless(1); /* eat only 'n' this time */
|
2004-02-19 20:40:09 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
|
|
|
{xqstart} {
|
2016-03-18 20:05:49 +01:00
|
|
|
if (cur_state->std_strings)
|
2006-03-06 20:49:20 +01:00
|
|
|
BEGIN(xq);
|
|
|
|
else
|
|
|
|
BEGIN(xe);
|
2004-02-19 20:40:09 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
2005-06-26 21:16:07 +02:00
|
|
|
{xestart} {
|
2006-03-06 20:49:20 +01:00
|
|
|
BEGIN(xe);
|
2005-06-26 21:16:07 +02:00
|
|
|
ECHO;
|
|
|
|
}
|
2008-10-29 09:04:54 +01:00
|
|
|
{xusstart} {
|
|
|
|
BEGIN(xus);
|
|
|
|
ECHO;
|
|
|
|
}
|
Reduce size of backend scanner's tables.
Previously, the core scanner's yy_transition[] array had 37045 elements.
Since that number is larger than INT16_MAX, Flex generated the array to
contain 32-bit integers. By reimplementing some of the bulkier scanner
rules, this patch reduces the array to 20495 elements. The much smaller
total length, combined with the consequent use of 16-bit integers for
the array elements reduces the binary size by over 200kB. This was
accomplished in two ways:
1. Consolidate handling of quote continuations into a new start condition,
rather than duplicating that logic for five different string types.
2. Treat Unicode strings and identifiers followed by a UESCAPE sequence
as three separate tokens, rather than one. The logic to de-escape
Unicode strings is moved to the filter code in parser.c, which already
had the ability to provide special processing for token sequences.
While we could have implemented the conversion in the grammar, that
approach was rejected for performance and maintainability reasons.
Performance in microbenchmarks of raw parsing seems equal or slightly
faster in most cases, and it's reasonable to expect that in real-world
usage (with more competition for the CPU cache) there will be a larger
win. The exception is UESCAPE sequences; lexing those is about 10%
slower, primarily because the scanner now has to be called three times
rather than one. This seems acceptable since that feature is very
rarely used.
The psql and epcg lexers are likewise modified, primarily because we
want to keep them all in sync. Since those lexers don't use the
space-hogging -CF option, the space savings is much less, but it's
still good for perhaps 10kB apiece.
While at it, merge the ecpg lexer's handling of C-style comments used
in SQL and in C. Those have different rules regarding nested comments,
but since we already have the ability to keep track of the previous
start condition, we can use that to handle both cases within a single
start condition. This matches the core scanner more closely.
John Naylor
Discussion: https://postgr.es/m/CACPNZCvaoa3EgVWm5yZhcSTX6RAtaLgniCPcBVOCwm8h3xpWkw@mail.gmail.com
2020-01-13 21:04:31 +01:00
|
|
|
|
|
|
|
<xb,xh,xq,xe,xus>{quote} {
|
|
|
|
/*
|
|
|
|
* When we are scanning a quoted string and see an end
|
|
|
|
* quote, we must look ahead for a possible continuation.
|
|
|
|
* If we don't see one, we know the end quote was in fact
|
|
|
|
* the end of the string. To reduce the lexer table size,
|
|
|
|
* we use a single "xqs" state to do the lookahead for all
|
|
|
|
* types of strings.
|
|
|
|
*/
|
|
|
|
cur_state->state_before_str_stop = YYSTATE;
|
|
|
|
BEGIN(xqs);
|
2013-03-14 19:31:27 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
Reduce size of backend scanner's tables.
Previously, the core scanner's yy_transition[] array had 37045 elements.
Since that number is larger than INT16_MAX, Flex generated the array to
contain 32-bit integers. By reimplementing some of the bulkier scanner
rules, this patch reduces the array to 20495 elements. The much smaller
total length, combined with the consequent use of 16-bit integers for
the array elements reduces the binary size by over 200kB. This was
accomplished in two ways:
1. Consolidate handling of quote continuations into a new start condition,
rather than duplicating that logic for five different string types.
2. Treat Unicode strings and identifiers followed by a UESCAPE sequence
as three separate tokens, rather than one. The logic to de-escape
Unicode strings is moved to the filter code in parser.c, which already
had the ability to provide special processing for token sequences.
While we could have implemented the conversion in the grammar, that
approach was rejected for performance and maintainability reasons.
Performance in microbenchmarks of raw parsing seems equal or slightly
faster in most cases, and it's reasonable to expect that in real-world
usage (with more competition for the CPU cache) there will be a larger
win. The exception is UESCAPE sequences; lexing those is about 10%
slower, primarily because the scanner now has to be called three times
rather than one. This seems acceptable since that feature is very
rarely used.
The psql and epcg lexers are likewise modified, primarily because we
want to keep them all in sync. Since those lexers don't use the
space-hogging -CF option, the space savings is much less, but it's
still good for perhaps 10kB apiece.
While at it, merge the ecpg lexer's handling of C-style comments used
in SQL and in C. Those have different rules regarding nested comments,
but since we already have the ability to keep track of the previous
start condition, we can use that to handle both cases within a single
start condition. This matches the core scanner more closely.
John Naylor
Discussion: https://postgr.es/m/CACPNZCvaoa3EgVWm5yZhcSTX6RAtaLgniCPcBVOCwm8h3xpWkw@mail.gmail.com
2020-01-13 21:04:31 +01:00
|
|
|
<xqs>{quotecontinue} {
|
|
|
|
/*
|
|
|
|
* Found a quote continuation, so return to the in-quote
|
|
|
|
* state and continue scanning the literal. Nothing is
|
|
|
|
* added to the literal's contents.
|
|
|
|
*/
|
|
|
|
BEGIN(cur_state->state_before_str_stop);
|
2013-03-14 19:31:27 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
Reduce size of backend scanner's tables.
Previously, the core scanner's yy_transition[] array had 37045 elements.
Since that number is larger than INT16_MAX, Flex generated the array to
contain 32-bit integers. By reimplementing some of the bulkier scanner
rules, this patch reduces the array to 20495 elements. The much smaller
total length, combined with the consequent use of 16-bit integers for
the array elements reduces the binary size by over 200kB. This was
accomplished in two ways:
1. Consolidate handling of quote continuations into a new start condition,
rather than duplicating that logic for five different string types.
2. Treat Unicode strings and identifiers followed by a UESCAPE sequence
as three separate tokens, rather than one. The logic to de-escape
Unicode strings is moved to the filter code in parser.c, which already
had the ability to provide special processing for token sequences.
While we could have implemented the conversion in the grammar, that
approach was rejected for performance and maintainability reasons.
Performance in microbenchmarks of raw parsing seems equal or slightly
faster in most cases, and it's reasonable to expect that in real-world
usage (with more competition for the CPU cache) there will be a larger
win. The exception is UESCAPE sequences; lexing those is about 10%
slower, primarily because the scanner now has to be called three times
rather than one. This seems acceptable since that feature is very
rarely used.
The psql and epcg lexers are likewise modified, primarily because we
want to keep them all in sync. Since those lexers don't use the
space-hogging -CF option, the space savings is much less, but it's
still good for perhaps 10kB apiece.
While at it, merge the ecpg lexer's handling of C-style comments used
in SQL and in C. Those have different rules regarding nested comments,
but since we already have the ability to keep track of the previous
start condition, we can use that to handle both cases within a single
start condition. This matches the core scanner more closely.
John Naylor
Discussion: https://postgr.es/m/CACPNZCvaoa3EgVWm5yZhcSTX6RAtaLgniCPcBVOCwm8h3xpWkw@mail.gmail.com
2020-01-13 21:04:31 +01:00
|
|
|
<xqs>{quotecontinuefail} |
|
|
|
|
<xqs>{other} {
|
|
|
|
/*
|
|
|
|
* Failed to see a quote continuation. Throw back
|
|
|
|
* everything after the end quote, and handle the string
|
|
|
|
* according to the state we were in previously.
|
|
|
|
*/
|
2013-03-14 19:31:27 +01:00
|
|
|
yyless(0);
|
2008-10-29 09:04:54 +01:00
|
|
|
BEGIN(INITIAL);
|
Reduce size of backend scanner's tables.
Previously, the core scanner's yy_transition[] array had 37045 elements.
Since that number is larger than INT16_MAX, Flex generated the array to
contain 32-bit integers. By reimplementing some of the bulkier scanner
rules, this patch reduces the array to 20495 elements. The much smaller
total length, combined with the consequent use of 16-bit integers for
the array elements reduces the binary size by over 200kB. This was
accomplished in two ways:
1. Consolidate handling of quote continuations into a new start condition,
rather than duplicating that logic for five different string types.
2. Treat Unicode strings and identifiers followed by a UESCAPE sequence
as three separate tokens, rather than one. The logic to de-escape
Unicode strings is moved to the filter code in parser.c, which already
had the ability to provide special processing for token sequences.
While we could have implemented the conversion in the grammar, that
approach was rejected for performance and maintainability reasons.
Performance in microbenchmarks of raw parsing seems equal or slightly
faster in most cases, and it's reasonable to expect that in real-world
usage (with more competition for the CPU cache) there will be a larger
win. The exception is UESCAPE sequences; lexing those is about 10%
slower, primarily because the scanner now has to be called three times
rather than one. This seems acceptable since that feature is very
rarely used.
The psql and epcg lexers are likewise modified, primarily because we
want to keep them all in sync. Since those lexers don't use the
space-hogging -CF option, the space savings is much less, but it's
still good for perhaps 10kB apiece.
While at it, merge the ecpg lexer's handling of C-style comments used
in SQL and in C. Those have different rules regarding nested comments,
but since we already have the ability to keep track of the previous
start condition, we can use that to handle both cases within a single
start condition. This matches the core scanner more closely.
John Naylor
Discussion: https://postgr.es/m/CACPNZCvaoa3EgVWm5yZhcSTX6RAtaLgniCPcBVOCwm8h3xpWkw@mail.gmail.com
2020-01-13 21:04:31 +01:00
|
|
|
/* There's nothing to echo ... */
|
2008-10-29 09:04:54 +01:00
|
|
|
}
|
Reduce size of backend scanner's tables.
Previously, the core scanner's yy_transition[] array had 37045 elements.
Since that number is larger than INT16_MAX, Flex generated the array to
contain 32-bit integers. By reimplementing some of the bulkier scanner
rules, this patch reduces the array to 20495 elements. The much smaller
total length, combined with the consequent use of 16-bit integers for
the array elements reduces the binary size by over 200kB. This was
accomplished in two ways:
1. Consolidate handling of quote continuations into a new start condition,
rather than duplicating that logic for five different string types.
2. Treat Unicode strings and identifiers followed by a UESCAPE sequence
as three separate tokens, rather than one. The logic to de-escape
Unicode strings is moved to the filter code in parser.c, which already
had the ability to provide special processing for token sequences.
While we could have implemented the conversion in the grammar, that
approach was rejected for performance and maintainability reasons.
Performance in microbenchmarks of raw parsing seems equal or slightly
faster in most cases, and it's reasonable to expect that in real-world
usage (with more competition for the CPU cache) there will be a larger
win. The exception is UESCAPE sequences; lexing those is about 10%
slower, primarily because the scanner now has to be called three times
rather than one. This seems acceptable since that feature is very
rarely used.
The psql and epcg lexers are likewise modified, primarily because we
want to keep them all in sync. Since those lexers don't use the
space-hogging -CF option, the space savings is much less, but it's
still good for perhaps 10kB apiece.
While at it, merge the ecpg lexer's handling of C-style comments used
in SQL and in C. Those have different rules regarding nested comments,
but since we already have the ability to keep track of the previous
start condition, we can use that to handle both cases within a single
start condition. This matches the core scanner more closely.
John Naylor
Discussion: https://postgr.es/m/CACPNZCvaoa3EgVWm5yZhcSTX6RAtaLgniCPcBVOCwm8h3xpWkw@mail.gmail.com
2020-01-13 21:04:31 +01:00
|
|
|
|
2008-10-29 09:04:54 +01:00
|
|
|
<xq,xe,xus>{xqdouble} {
|
2004-02-19 20:40:09 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
2008-10-29 09:04:54 +01:00
|
|
|
<xq,xus>{xqinside} {
|
2004-02-19 20:40:09 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
2006-03-06 20:49:20 +01:00
|
|
|
<xe>{xeinside} {
|
|
|
|
ECHO;
|
|
|
|
}
|
2009-09-27 05:27:24 +02:00
|
|
|
<xe>{xeunicode} {
|
|
|
|
ECHO;
|
|
|
|
}
|
2010-10-27 04:23:04 +02:00
|
|
|
<xe>{xeunicodefail} {
|
2009-09-27 05:27:24 +02:00
|
|
|
ECHO;
|
|
|
|
}
|
2006-03-06 20:49:20 +01:00
|
|
|
<xe>{xeescape} {
|
2004-02-19 20:40:09 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
2006-03-06 20:49:20 +01:00
|
|
|
<xe>{xeoctesc} {
|
2004-02-19 20:40:09 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
2006-03-06 20:49:20 +01:00
|
|
|
<xe>{xehexesc} {
|
2005-06-02 03:23:48 +02:00
|
|
|
ECHO;
|
|
|
|
}
|
2006-03-06 20:49:20 +01:00
|
|
|
<xe>. {
|
2004-02-24 22:45:18 +01:00
|
|
|
/* This is only needed for \ just before EOF */
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
|
|
|
{dolqdelim} {
|
|
|
|
cur_state->dolqstart = pg_strdup(yytext);
|
|
|
|
BEGIN(xdolq);
|
|
|
|
ECHO;
|
|
|
|
}
|
2005-05-26 03:24:29 +02:00
|
|
|
{dolqfailed} {
|
|
|
|
/* throw back all but the initial "$" */
|
|
|
|
yyless(1);
|
|
|
|
ECHO;
|
|
|
|
}
|
2004-02-24 22:45:18 +01:00
|
|
|
<xdolq>{dolqdelim} {
|
|
|
|
if (strcmp(yytext, cur_state->dolqstart) == 0)
|
|
|
|
{
|
|
|
|
free(cur_state->dolqstart);
|
|
|
|
cur_state->dolqstart = NULL;
|
|
|
|
BEGIN(INITIAL);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* When we fail to match $...$ to dolqstart, transfer
|
|
|
|
* the $... part to the output, but put back the final
|
|
|
|
* $ for rescanning. Consider $delim$...$junk$delim$
|
|
|
|
*/
|
2016-03-19 19:36:22 +01:00
|
|
|
yyless(yyleng - 1);
|
2004-02-24 22:45:18 +01:00
|
|
|
}
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
<xdolq>{dolqinside} {
|
|
|
|
ECHO;
|
|
|
|
}
|
2005-05-26 03:24:29 +02:00
|
|
|
<xdolq>{dolqfailed} {
|
|
|
|
ECHO;
|
|
|
|
}
|
2004-02-24 22:45:18 +01:00
|
|
|
<xdolq>. {
|
|
|
|
/* This is only needed for $ inside the quoted text */
|
|
|
|
ECHO;
|
|
|
|
}
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
{xdstart} {
|
|
|
|
BEGIN(xd);
|
|
|
|
ECHO;
|
|
|
|
}
|
2008-10-29 09:04:54 +01:00
|
|
|
{xuistart} {
|
|
|
|
BEGIN(xui);
|
|
|
|
ECHO;
|
|
|
|
}
|
2004-02-19 20:40:09 +01:00
|
|
|
<xd>{xdstop} {
|
|
|
|
BEGIN(INITIAL);
|
|
|
|
ECHO;
|
|
|
|
}
|
Reduce size of backend scanner's tables.
Previously, the core scanner's yy_transition[] array had 37045 elements.
Since that number is larger than INT16_MAX, Flex generated the array to
contain 32-bit integers. By reimplementing some of the bulkier scanner
rules, this patch reduces the array to 20495 elements. The much smaller
total length, combined with the consequent use of 16-bit integers for
the array elements reduces the binary size by over 200kB. This was
accomplished in two ways:
1. Consolidate handling of quote continuations into a new start condition,
rather than duplicating that logic for five different string types.
2. Treat Unicode strings and identifiers followed by a UESCAPE sequence
as three separate tokens, rather than one. The logic to de-escape
Unicode strings is moved to the filter code in parser.c, which already
had the ability to provide special processing for token sequences.
While we could have implemented the conversion in the grammar, that
approach was rejected for performance and maintainability reasons.
Performance in microbenchmarks of raw parsing seems equal or slightly
faster in most cases, and it's reasonable to expect that in real-world
usage (with more competition for the CPU cache) there will be a larger
win. The exception is UESCAPE sequences; lexing those is about 10%
slower, primarily because the scanner now has to be called three times
rather than one. This seems acceptable since that feature is very
rarely used.
The psql and epcg lexers are likewise modified, primarily because we
want to keep them all in sync. Since those lexers don't use the
space-hogging -CF option, the space savings is much less, but it's
still good for perhaps 10kB apiece.
While at it, merge the ecpg lexer's handling of C-style comments used
in SQL and in C. Those have different rules regarding nested comments,
but since we already have the ability to keep track of the previous
start condition, we can use that to handle both cases within a single
start condition. This matches the core scanner more closely.
John Naylor
Discussion: https://postgr.es/m/CACPNZCvaoa3EgVWm5yZhcSTX6RAtaLgniCPcBVOCwm8h3xpWkw@mail.gmail.com
2020-01-13 21:04:31 +01:00
|
|
|
<xui>{dquote} {
|
2008-10-29 09:04:54 +01:00
|
|
|
BEGIN(INITIAL);
|
2004-02-19 20:40:09 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
2008-10-29 09:04:54 +01:00
|
|
|
<xd,xui>{xddouble} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
<xd,xui>{xdinside} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
|
|
|
{xufailed} {
|
|
|
|
/* throw back all but the initial u/U */
|
|
|
|
yyless(1);
|
2004-02-19 20:40:09 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
|
|
|
{typecast} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
2009-09-27 05:27:24 +02:00
|
|
|
{dot_dot} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
|
|
|
{colon_equals} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
2015-03-10 16:48:34 +01:00
|
|
|
{equals_greater} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
Make operator precedence follow the SQL standard more closely.
While the SQL standard is pretty vague on the overall topic of operator
precedence (because it never presents a unified BNF for all expressions),
it does seem reasonable to conclude from the spec for <boolean value
expression> that OR has the lowest precedence, then AND, then NOT, then IS
tests, then the six standard comparison operators, then everything else
(since any non-boolean operator in a WHERE clause would need to be an
argument of one of these).
We were only sort of on board with that: most notably, while "<" ">" and
"=" had properly low precedence, "<=" ">=" and "<>" were treated as generic
operators and so had significantly higher precedence. And "IS" tests were
even higher precedence than those, which is very clearly wrong per spec.
Another problem was that "foo NOT SOMETHING bar" constructs, such as
"x NOT LIKE y", were treated inconsistently because of a bison
implementation artifact: they had the documented precedence with respect
to operators to their right, but behaved like NOT (i.e., very low priority)
with respect to operators to their left.
Fixing the precedence issues is just a small matter of rearranging the
precedence declarations in gram.y, except for the NOT problem, which
requires adding an additional lookahead case in base_yylex() so that we
can attach a different token precedence to NOT LIKE and allied two-word
operators.
The bulk of this patch is not the bug fix per se, but adding logic to
parse_expr.c to allow giving warnings if an expression has changed meaning
because of these precedence changes. These warnings are off by default
and are enabled by the new GUC operator_precedence_warning. It's believed
that very few applications will be affected by these changes, but it was
agreed that a warning mechanism is essential to help debug any that are.
2015-03-11 18:22:52 +01:00
|
|
|
{less_equals} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
|
|
|
{greater_equals} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
|
|
|
{less_greater} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
|
|
|
{not_equals} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
2004-02-19 20:40:09 +01:00
|
|
|
/*
|
|
|
|
* These rules are specific to psql --- they implement parenthesis
|
|
|
|
* counting and detection of command-ending semicolon. These must
|
|
|
|
* appear before the {self} rule so that they take precedence over it.
|
|
|
|
*/
|
|
|
|
|
|
|
|
"(" {
|
|
|
|
cur_state->paren_depth++;
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
|
|
|
")" {
|
|
|
|
if (cur_state->paren_depth > 0)
|
|
|
|
cur_state->paren_depth--;
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
|
|
|
";" {
|
|
|
|
ECHO;
|
2021-04-07 21:30:08 +02:00
|
|
|
if (cur_state->paren_depth == 0 && cur_state->begin_depth == 0)
|
2004-02-19 20:40:09 +01:00
|
|
|
{
|
|
|
|
/* Terminate lexing temporarily */
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
cur_state->start_state = YY_START;
|
2021-04-07 21:30:08 +02:00
|
|
|
cur_state->identifier_count = 0;
|
2004-02-19 20:40:09 +01:00
|
|
|
return LEXRES_SEMI;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* psql-specific rules to handle backslash commands and variable
|
|
|
|
* substitution. We want these before {self}, also.
|
|
|
|
*/
|
|
|
|
|
2019-03-25 16:16:07 +01:00
|
|
|
"\\"[;:] {
|
|
|
|
/* Force a semi-colon or colon into the query buffer */
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
psqlscan_emit(cur_state, yytext + 1, 1);
|
2021-04-07 21:30:08 +02:00
|
|
|
if (yytext[1] == ';')
|
|
|
|
cur_state->identifier_count = 0;
|
2004-02-19 20:40:09 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
"\\" {
|
|
|
|
/* Terminate lexing temporarily */
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
cur_state->start_state = YY_START;
|
2004-02-19 20:40:09 +01:00
|
|
|
return LEXRES_BACKSLASH;
|
|
|
|
}
|
|
|
|
|
2011-08-26 16:41:31 +02:00
|
|
|
:{variable_char}+ {
|
2004-02-19 20:40:09 +01:00
|
|
|
/* Possible psql variable substitution */
|
2016-03-19 19:36:22 +01:00
|
|
|
char *varname;
|
|
|
|
char *value;
|
2004-02-19 20:40:09 +01:00
|
|
|
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
varname = psqlscan_extract_substring(cur_state,
|
|
|
|
yytext + 1,
|
|
|
|
yyleng - 1);
|
2016-03-18 20:05:49 +01:00
|
|
|
if (cur_state->callbacks->get_variable)
|
|
|
|
value = cur_state->callbacks->get_variable(varname,
|
Allow psql variable substitution to occur in backtick command strings.
Previously, text between backquotes in a psql metacommand's arguments
was always passed to the shell literally. That considerably hobbles
the usefulness of the feature for scripting, so we'd foreseen for a long
time that we'd someday want to allow substitution of psql variables into
the shell command. IMO the addition of \if metacommands has brought us to
that point, since \if can greatly benefit from some sort of client-side
expression evaluation capability, and psql itself is not going to grow any
such thing in time for v10. Hence, this patch. It allows :VARIABLE to be
replaced by the exact contents of the named variable, while :'VARIABLE'
is replaced by the variable's contents suitably quoted to become a single
shell-command argument. (The quoting rules for that are different from
those for SQL literals, so this is a bit of an abuse of the :'VARIABLE'
notation, but I doubt anyone will be confused.)
As with other situations in psql, no substitution occurs if the word
following a colon is not a known variable name. That limits the risk of
compatibility problems for existing psql scripts; but the risk isn't zero,
so this needs to be called out in the v10 release notes.
Discussion: https://postgr.es/m/9561.1490895211@sss.pgh.pa.us
2017-04-02 03:44:54 +02:00
|
|
|
PQUOTE_PLAIN,
|
2017-03-13 22:14:46 +01:00
|
|
|
cur_state->cb_passthrough);
|
2016-03-18 20:05:49 +01:00
|
|
|
else
|
|
|
|
value = NULL;
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
if (value)
|
|
|
|
{
|
2010-05-06 00:18:56 +02:00
|
|
|
/* It is a variable, check for recursion */
|
2016-03-25 01:28:47 +01:00
|
|
|
if (psqlscan_var_is_current_source(cur_state, varname))
|
2010-05-06 00:18:56 +02:00
|
|
|
{
|
|
|
|
/* Recursive expansion --- don't go there */
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_warning("skipping recursive expansion of variable \"%s\"",
|
2016-03-18 20:05:49 +01:00
|
|
|
varname);
|
2010-05-06 00:18:56 +02:00
|
|
|
/* Instead copy the string as is */
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* OK, perform substitution */
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
psqlscan_push_new_buffer(cur_state, value, varname);
|
2010-05-06 00:18:56 +02:00
|
|
|
/* yy_scan_string already made buffer active */
|
|
|
|
}
|
2016-03-18 20:05:49 +01:00
|
|
|
free(value);
|
2004-02-19 20:40:09 +01:00
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/*
|
2016-03-19 19:36:22 +01:00
|
|
|
* if the variable doesn't exist we'll copy the string
|
|
|
|
* as is
|
2004-02-19 20:40:09 +01:00
|
|
|
*/
|
|
|
|
ECHO;
|
|
|
|
}
|
2011-08-26 16:41:31 +02:00
|
|
|
|
|
|
|
free(varname);
|
2004-02-19 20:40:09 +01:00
|
|
|
}
|
|
|
|
|
2011-08-26 16:41:31 +02:00
|
|
|
:'{variable_char}+' {
|
Allow psql variable substitution to occur in backtick command strings.
Previously, text between backquotes in a psql metacommand's arguments
was always passed to the shell literally. That considerably hobbles
the usefulness of the feature for scripting, so we'd foreseen for a long
time that we'd someday want to allow substitution of psql variables into
the shell command. IMO the addition of \if metacommands has brought us to
that point, since \if can greatly benefit from some sort of client-side
expression evaluation capability, and psql itself is not going to grow any
such thing in time for v10. Hence, this patch. It allows :VARIABLE to be
replaced by the exact contents of the named variable, while :'VARIABLE'
is replaced by the variable's contents suitably quoted to become a single
shell-command argument. (The quoting rules for that are different from
those for SQL literals, so this is a bit of an abuse of the :'VARIABLE'
notation, but I doubt anyone will be confused.)
As with other situations in psql, no substitution occurs if the word
following a colon is not a known variable name. That limits the risk of
compatibility problems for existing psql scripts; but the risk isn't zero,
so this needs to be called out in the v10 release notes.
Discussion: https://postgr.es/m/9561.1490895211@sss.pgh.pa.us
2017-04-02 03:44:54 +02:00
|
|
|
psqlscan_escape_variable(cur_state, yytext, yyleng,
|
|
|
|
PQUOTE_SQL_LITERAL);
|
2010-01-29 18:44:12 +01:00
|
|
|
}
|
|
|
|
|
2011-08-26 16:41:31 +02:00
|
|
|
:\"{variable_char}+\" {
|
Allow psql variable substitution to occur in backtick command strings.
Previously, text between backquotes in a psql metacommand's arguments
was always passed to the shell literally. That considerably hobbles
the usefulness of the feature for scripting, so we'd foreseen for a long
time that we'd someday want to allow substitution of psql variables into
the shell command. IMO the addition of \if metacommands has brought us to
that point, since \if can greatly benefit from some sort of client-side
expression evaluation capability, and psql itself is not going to grow any
such thing in time for v10. Hence, this patch. It allows :VARIABLE to be
replaced by the exact contents of the named variable, while :'VARIABLE'
is replaced by the variable's contents suitably quoted to become a single
shell-command argument. (The quoting rules for that are different from
those for SQL literals, so this is a bit of an abuse of the :'VARIABLE'
notation, but I doubt anyone will be confused.)
As with other situations in psql, no substitution occurs if the word
following a colon is not a known variable name. That limits the risk of
compatibility problems for existing psql scripts; but the risk isn't zero,
so this needs to be called out in the v10 release notes.
Discussion: https://postgr.es/m/9561.1490895211@sss.pgh.pa.us
2017-04-02 03:44:54 +02:00
|
|
|
psqlscan_escape_variable(cur_state, yytext, yyleng,
|
|
|
|
PQUOTE_SQL_IDENT);
|
2010-01-29 18:44:12 +01:00
|
|
|
}
|
|
|
|
|
2017-09-22 01:02:23 +02:00
|
|
|
:\{\?{variable_char}+\} {
|
|
|
|
psqlscan_test_variable(cur_state, yytext, yyleng);
|
|
|
|
}
|
|
|
|
|
2011-08-25 20:33:08 +02:00
|
|
|
/*
|
|
|
|
* These rules just avoid the need for scanner backup if one of the
|
2017-09-22 01:02:23 +02:00
|
|
|
* three rules above fails to match completely.
|
2011-08-25 20:33:08 +02:00
|
|
|
*/
|
|
|
|
|
2011-08-26 16:41:31 +02:00
|
|
|
:'{variable_char}* {
|
2011-08-25 20:33:08 +02:00
|
|
|
/* Throw back everything but the colon */
|
|
|
|
yyless(1);
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
2011-08-26 16:41:31 +02:00
|
|
|
:\"{variable_char}* {
|
2011-08-25 20:33:08 +02:00
|
|
|
/* Throw back everything but the colon */
|
|
|
|
yyless(1);
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
2017-09-22 01:02:23 +02:00
|
|
|
:\{\?{variable_char}* {
|
|
|
|
/* Throw back everything but the colon */
|
|
|
|
yyless(1);
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
:\{ {
|
|
|
|
/* Throw back everything but the colon */
|
|
|
|
yyless(1);
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
2004-02-19 20:40:09 +01:00
|
|
|
/*
|
|
|
|
* Back to backend-compatible rules.
|
|
|
|
*/
|
|
|
|
|
|
|
|
{self} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
|
|
|
{operator} {
|
|
|
|
/*
|
|
|
|
* Check for embedded slash-star or dash-dash; those
|
|
|
|
* are comment starts, so operator must stop there.
|
|
|
|
* Note that slash-star or dash-dash at the first
|
|
|
|
* character will match a prior rule, not this one.
|
|
|
|
*/
|
2016-03-19 19:36:22 +01:00
|
|
|
int nchars = yyleng;
|
|
|
|
char *slashstar = strstr(yytext, "/*");
|
|
|
|
char *dashdash = strstr(yytext, "--");
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
if (slashstar && dashdash)
|
|
|
|
{
|
|
|
|
/* if both appear, take the first one */
|
|
|
|
if (slashstar > dashdash)
|
|
|
|
slashstar = dashdash;
|
|
|
|
}
|
|
|
|
else if (!slashstar)
|
|
|
|
slashstar = dashdash;
|
|
|
|
if (slashstar)
|
|
|
|
nchars = slashstar - yytext;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* For SQL compatibility, '+' and '-' cannot be the
|
|
|
|
* last char of a multi-char operator unless the operator
|
|
|
|
* contains chars that are not in SQL operators.
|
|
|
|
* The idea is to lex '=-' as two operators, but not
|
|
|
|
* to forbid operator names like '?-' that could not be
|
|
|
|
* sequences of SQL operators.
|
|
|
|
*/
|
2018-08-23 17:35:33 +02:00
|
|
|
if (nchars > 1 &&
|
|
|
|
(yytext[nchars - 1] == '+' ||
|
|
|
|
yytext[nchars - 1] == '-'))
|
2004-02-19 20:40:09 +01:00
|
|
|
{
|
2016-03-19 19:36:22 +01:00
|
|
|
int ic;
|
2004-02-19 20:40:09 +01:00
|
|
|
|
2016-03-19 19:36:22 +01:00
|
|
|
for (ic = nchars - 2; ic >= 0; ic--)
|
2004-02-19 20:40:09 +01:00
|
|
|
{
|
2018-08-23 17:35:33 +02:00
|
|
|
char c = yytext[ic];
|
|
|
|
if (c == '~' || c == '!' || c == '@' ||
|
|
|
|
c == '#' || c == '^' || c == '&' ||
|
|
|
|
c == '|' || c == '`' || c == '?' ||
|
|
|
|
c == '%')
|
2004-02-19 20:40:09 +01:00
|
|
|
break;
|
|
|
|
}
|
2018-08-23 17:35:33 +02:00
|
|
|
if (ic < 0)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* didn't find a qualifying character, so remove
|
|
|
|
* all trailing [+-]
|
|
|
|
*/
|
|
|
|
do {
|
|
|
|
nchars--;
|
|
|
|
} while (nchars > 1 &&
|
|
|
|
(yytext[nchars - 1] == '+' ||
|
|
|
|
yytext[nchars - 1] == '-'));
|
|
|
|
}
|
2004-02-19 20:40:09 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
if (nchars < yyleng)
|
|
|
|
{
|
|
|
|
/* Strip the unwanted chars from the token */
|
|
|
|
yyless(nchars);
|
|
|
|
}
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
|
|
|
{param} {
|
|
|
|
ECHO;
|
|
|
|
}
|
2022-02-16 10:32:36 +01:00
|
|
|
{param_junk} {
|
|
|
|
ECHO;
|
|
|
|
}
|
2004-02-19 20:40:09 +01:00
|
|
|
|
2022-12-14 05:40:38 +01:00
|
|
|
{decinteger} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
{hexinteger} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
{octinteger} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
{bininteger} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
{hexfail} {
|
2004-02-19 20:40:09 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
2022-12-14 05:40:38 +01:00
|
|
|
{octfail} {
|
2004-02-19 20:40:09 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
2022-12-14 05:40:38 +01:00
|
|
|
{binfail} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
{numeric} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
{numericfail} {
|
2009-11-12 01:13:00 +01:00
|
|
|
/* throw back the .., and treat as integer */
|
2016-03-19 19:36:22 +01:00
|
|
|
yyless(yyleng - 2);
|
2009-11-12 01:13:00 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
2004-02-19 20:40:09 +01:00
|
|
|
{real} {
|
|
|
|
ECHO;
|
|
|
|
}
|
2022-02-16 10:32:36 +01:00
|
|
|
{realfail} {
|
2005-05-26 03:24:29 +02:00
|
|
|
ECHO;
|
|
|
|
}
|
2022-12-14 05:40:38 +01:00
|
|
|
{decinteger_junk} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
{hexinteger_junk} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
{octinteger_junk} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
{bininteger_junk} {
|
2022-02-16 10:32:36 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
2022-12-14 05:40:38 +01:00
|
|
|
{numeric_junk} {
|
2022-02-16 10:32:36 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
{real_junk} {
|
2005-05-26 03:24:29 +02:00
|
|
|
ECHO;
|
|
|
|
}
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
|
|
|
|
{identifier} {
|
2021-04-16 11:46:01 +02:00
|
|
|
/*
|
|
|
|
* We need to track if we are inside a BEGIN .. END block
|
|
|
|
* in a function definition, so that semicolons contained
|
|
|
|
* therein don't terminate the whole statement. Short of
|
|
|
|
* writing a full parser here, the following heuristic
|
|
|
|
* should work. First, we track whether the beginning of
|
|
|
|
* the statement matches CREATE [OR REPLACE]
|
|
|
|
* {FUNCTION|PROCEDURE}
|
|
|
|
*/
|
|
|
|
|
|
|
|
if (cur_state->identifier_count == 0)
|
|
|
|
memset(cur_state->identifiers, 0, sizeof(cur_state->identifiers));
|
|
|
|
|
|
|
|
if (pg_strcasecmp(yytext, "create") == 0 ||
|
|
|
|
pg_strcasecmp(yytext, "function") == 0 ||
|
|
|
|
pg_strcasecmp(yytext, "procedure") == 0 ||
|
|
|
|
pg_strcasecmp(yytext, "or") == 0 ||
|
|
|
|
pg_strcasecmp(yytext, "replace") == 0)
|
2021-04-07 21:30:08 +02:00
|
|
|
{
|
2021-04-16 11:46:01 +02:00
|
|
|
if (cur_state->identifier_count < sizeof(cur_state->identifiers))
|
|
|
|
cur_state->identifiers[cur_state->identifier_count] = pg_tolower((unsigned char) yytext[0]);
|
2021-04-07 21:30:08 +02:00
|
|
|
}
|
2021-04-16 11:46:01 +02:00
|
|
|
|
|
|
|
cur_state->identifier_count++;
|
|
|
|
|
|
|
|
if (cur_state->identifiers[0] == 'c' &&
|
|
|
|
(cur_state->identifiers[1] == 'f' || cur_state->identifiers[1] == 'p' ||
|
|
|
|
(cur_state->identifiers[1] == 'o' && cur_state->identifiers[2] == 'r' &&
|
|
|
|
(cur_state->identifiers[3] == 'f' || cur_state->identifiers[3] == 'p'))) &&
|
|
|
|
cur_state->paren_depth == 0)
|
2021-04-07 21:30:08 +02:00
|
|
|
{
|
2021-04-16 11:46:01 +02:00
|
|
|
if (pg_strcasecmp(yytext, "begin") == 0)
|
|
|
|
cur_state->begin_depth++;
|
|
|
|
else if (pg_strcasecmp(yytext, "case") == 0)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* CASE also ends with END. We only need to track
|
|
|
|
* this if we are already inside a BEGIN.
|
|
|
|
*/
|
|
|
|
if (cur_state->begin_depth >= 1)
|
|
|
|
cur_state->begin_depth++;
|
|
|
|
}
|
|
|
|
else if (pg_strcasecmp(yytext, "end") == 0)
|
|
|
|
{
|
|
|
|
if (cur_state->begin_depth > 0)
|
|
|
|
cur_state->begin_depth--;
|
|
|
|
}
|
2021-04-07 21:30:08 +02:00
|
|
|
}
|
2021-04-16 11:46:01 +02:00
|
|
|
|
2004-02-19 20:40:09 +01:00
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
|
|
|
{other} {
|
|
|
|
ECHO;
|
|
|
|
}
|
|
|
|
|
|
|
|
<<EOF>> {
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
if (cur_state->buffer_stack == NULL)
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
{
|
|
|
|
cur_state->start_state = YY_START;
|
2016-03-19 19:36:22 +01:00
|
|
|
return LEXRES_EOL; /* end of input reached */
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
}
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We were expanding a variable, so pop the inclusion
|
|
|
|
* stack and keep lexing
|
|
|
|
*/
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
psqlscan_pop_buffer_stack(cur_state);
|
|
|
|
psqlscan_select_top_buffer(cur_state);
|
2004-02-19 20:40:09 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
%%
|
|
|
|
|
2017-08-11 05:33:47 +02:00
|
|
|
/* LCOV_EXCL_STOP */
|
|
|
|
|
2004-02-19 20:40:09 +01:00
|
|
|
/*
|
|
|
|
* Create a lexer working state struct.
|
2016-03-18 20:05:49 +01:00
|
|
|
*
|
|
|
|
* callbacks is a struct of function pointers that encapsulate some
|
|
|
|
* behavior we need from the surrounding program. This struct must
|
|
|
|
* remain valid for the lifespan of the PsqlScanState.
|
2004-02-19 20:40:09 +01:00
|
|
|
*/
|
|
|
|
PsqlScanState
|
2016-03-18 20:05:49 +01:00
|
|
|
psql_scan_create(const PsqlScanCallbacks *callbacks)
|
2004-02-19 20:40:09 +01:00
|
|
|
{
|
|
|
|
PsqlScanState state;
|
|
|
|
|
2012-10-02 21:35:10 +02:00
|
|
|
state = (PsqlScanStateData *) pg_malloc0(sizeof(PsqlScanStateData));
|
2004-02-19 20:40:09 +01:00
|
|
|
|
2016-03-18 20:05:49 +01:00
|
|
|
state->callbacks = callbacks;
|
|
|
|
|
2016-03-19 06:02:18 +01:00
|
|
|
yylex_init(&state->scanner);
|
|
|
|
|
|
|
|
yyset_extra(state, state->scanner);
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
|
2004-02-19 20:40:09 +01:00
|
|
|
psql_scan_reset(state);
|
|
|
|
|
|
|
|
return state;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Destroy a lexer working state struct, releasing all resources.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
psql_scan_destroy(PsqlScanState state)
|
|
|
|
{
|
|
|
|
psql_scan_finish(state);
|
|
|
|
|
2004-02-24 22:45:18 +01:00
|
|
|
psql_scan_reset(state);
|
|
|
|
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
yylex_destroy(state->scanner);
|
|
|
|
|
2004-02-19 20:40:09 +01:00
|
|
|
free(state);
|
|
|
|
}
|
|
|
|
|
2017-03-13 22:14:46 +01:00
|
|
|
/*
|
|
|
|
* Set the callback passthrough pointer for the lexer.
|
|
|
|
*
|
|
|
|
* This could have been integrated into psql_scan_create, but keeping it
|
|
|
|
* separate allows the application to change the pointer later, which might
|
|
|
|
* be useful.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
psql_scan_set_passthrough(PsqlScanState state, void *passthrough)
|
|
|
|
{
|
|
|
|
state->cb_passthrough = passthrough;
|
|
|
|
}
|
|
|
|
|
2004-02-19 20:40:09 +01:00
|
|
|
/*
|
|
|
|
* Set up to perform lexing of the given input line.
|
|
|
|
*
|
|
|
|
* The text at *line, extending for line_len bytes, will be scanned by
|
|
|
|
* subsequent calls to the psql_scan routines. psql_scan_finish should
|
|
|
|
* be called when scanning is complete. Note that the lexer retains
|
|
|
|
* a pointer to the storage at *line --- this string must not be altered
|
|
|
|
* or freed until after psql_scan_finish is called.
|
2016-03-18 20:05:49 +01:00
|
|
|
*
|
|
|
|
* encoding is the libpq identifier for the character encoding in use,
|
|
|
|
* and std_strings says whether standard_conforming_strings is on.
|
2004-02-19 20:40:09 +01:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
psql_scan_setup(PsqlScanState state,
|
2016-03-18 20:05:49 +01:00
|
|
|
const char *line, int line_len,
|
|
|
|
int encoding, bool std_strings)
|
2004-02-19 20:40:09 +01:00
|
|
|
{
|
|
|
|
/* Mustn't be scanning already */
|
2012-12-15 00:03:07 +01:00
|
|
|
Assert(state->scanbufhandle == NULL);
|
|
|
|
Assert(state->buffer_stack == NULL);
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
/* Do we need to hack the character set encoding? */
|
2016-03-18 20:05:49 +01:00
|
|
|
state->encoding = encoding;
|
|
|
|
state->safe_encoding = pg_valid_server_encoding_id(encoding);
|
|
|
|
|
|
|
|
/* Save standard-strings flag as well */
|
|
|
|
state->std_strings = std_strings;
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
/* Set up flex input buffer with appropriate translation and padding */
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
state->scanbufhandle = psqlscan_prepare_buffer(state, line, line_len,
|
|
|
|
&state->scanbuf);
|
2004-02-19 20:40:09 +01:00
|
|
|
state->scanline = line;
|
|
|
|
|
|
|
|
/* Set lookaside data in case we have to map unsafe encoding */
|
|
|
|
state->curline = state->scanbuf;
|
|
|
|
state->refline = state->scanline;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Do lexical analysis of SQL command text.
|
|
|
|
*
|
|
|
|
* The text previously passed to psql_scan_setup is scanned, and appended
|
|
|
|
* (possibly with transformation) to query_buf.
|
|
|
|
*
|
|
|
|
* The return value indicates the condition that stopped scanning:
|
|
|
|
*
|
|
|
|
* PSCAN_SEMICOLON: found a command-ending semicolon. (The semicolon is
|
|
|
|
* transferred to query_buf.) The command accumulated in query_buf should
|
|
|
|
* be executed, then clear query_buf and call again to scan the remainder
|
|
|
|
* of the line.
|
|
|
|
*
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
* PSCAN_BACKSLASH: found a backslash that starts a special command.
|
2004-02-19 20:40:09 +01:00
|
|
|
* Any previous data on the line has been transferred to query_buf.
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
* The caller will typically next apply a separate flex lexer to scan
|
|
|
|
* the special command.
|
2004-02-19 20:40:09 +01:00
|
|
|
*
|
|
|
|
* PSCAN_INCOMPLETE: the end of the line was reached, but we have an
|
|
|
|
* incomplete SQL command. *prompt is set to the appropriate prompt type.
|
|
|
|
*
|
|
|
|
* PSCAN_EOL: the end of the line was reached, and there is no lexical
|
|
|
|
* reason to consider the command incomplete. The caller may or may not
|
|
|
|
* choose to send it. *prompt is set to the appropriate prompt type if
|
|
|
|
* the caller chooses to collect more input.
|
|
|
|
*
|
|
|
|
* In the PSCAN_INCOMPLETE and PSCAN_EOL cases, psql_scan_finish() should
|
|
|
|
* be called next, then the cycle may be repeated with a fresh input line.
|
|
|
|
*
|
|
|
|
* In all cases, *prompt is set to an appropriate prompt type code for the
|
|
|
|
* next line-input operation.
|
|
|
|
*/
|
|
|
|
PsqlScanResult
|
|
|
|
psql_scan(PsqlScanState state,
|
|
|
|
PQExpBuffer query_buf,
|
|
|
|
promptStatus_t *prompt)
|
|
|
|
{
|
|
|
|
PsqlScanResult result;
|
|
|
|
int lexresult;
|
|
|
|
|
|
|
|
/* Must be scanning already */
|
2012-12-15 00:03:07 +01:00
|
|
|
Assert(state->scanbufhandle != NULL);
|
2004-02-19 20:40:09 +01:00
|
|
|
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
/* Set current output target */
|
|
|
|
state->output_buf = query_buf;
|
2004-02-19 20:40:09 +01:00
|
|
|
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
/* Set input source */
|
2004-02-19 20:40:09 +01:00
|
|
|
if (state->buffer_stack != NULL)
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
yy_switch_to_buffer(state->buffer_stack->buf, state->scanner);
|
2004-02-19 20:40:09 +01:00
|
|
|
else
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
yy_switch_to_buffer(state->scanbufhandle, state->scanner);
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
/* And lex. */
|
2016-03-21 02:59:03 +01:00
|
|
|
lexresult = yylex(NULL, state->scanner);
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Check termination state and return appropriate result info.
|
|
|
|
*/
|
|
|
|
switch (lexresult)
|
|
|
|
{
|
|
|
|
case LEXRES_EOL: /* end of input */
|
|
|
|
switch (state->start_state)
|
|
|
|
{
|
|
|
|
case INITIAL:
|
Reduce size of backend scanner's tables.
Previously, the core scanner's yy_transition[] array had 37045 elements.
Since that number is larger than INT16_MAX, Flex generated the array to
contain 32-bit integers. By reimplementing some of the bulkier scanner
rules, this patch reduces the array to 20495 elements. The much smaller
total length, combined with the consequent use of 16-bit integers for
the array elements reduces the binary size by over 200kB. This was
accomplished in two ways:
1. Consolidate handling of quote continuations into a new start condition,
rather than duplicating that logic for five different string types.
2. Treat Unicode strings and identifiers followed by a UESCAPE sequence
as three separate tokens, rather than one. The logic to de-escape
Unicode strings is moved to the filter code in parser.c, which already
had the ability to provide special processing for token sequences.
While we could have implemented the conversion in the grammar, that
approach was rejected for performance and maintainability reasons.
Performance in microbenchmarks of raw parsing seems equal or slightly
faster in most cases, and it's reasonable to expect that in real-world
usage (with more competition for the CPU cache) there will be a larger
win. The exception is UESCAPE sequences; lexing those is about 10%
slower, primarily because the scanner now has to be called three times
rather than one. This seems acceptable since that feature is very
rarely used.
The psql and epcg lexers are likewise modified, primarily because we
want to keep them all in sync. Since those lexers don't use the
space-hogging -CF option, the space savings is much less, but it's
still good for perhaps 10kB apiece.
While at it, merge the ecpg lexer's handling of C-style comments used
in SQL and in C. Those have different rules regarding nested comments,
but since we already have the ability to keep track of the previous
start condition, we can use that to handle both cases within a single
start condition. This matches the core scanner more closely.
John Naylor
Discussion: https://postgr.es/m/CACPNZCvaoa3EgVWm5yZhcSTX6RAtaLgniCPcBVOCwm8h3xpWkw@mail.gmail.com
2020-01-13 21:04:31 +01:00
|
|
|
case xqs: /* we treat this like INITIAL */
|
2004-02-19 20:40:09 +01:00
|
|
|
if (state->paren_depth > 0)
|
|
|
|
{
|
|
|
|
result = PSCAN_INCOMPLETE;
|
|
|
|
*prompt = PROMPT_PAREN;
|
|
|
|
}
|
2021-04-29 09:04:31 +02:00
|
|
|
else if (state->begin_depth > 0)
|
2021-04-07 21:30:08 +02:00
|
|
|
{
|
|
|
|
result = PSCAN_INCOMPLETE;
|
|
|
|
*prompt = PROMPT_CONTINUE;
|
|
|
|
}
|
2004-02-19 20:40:09 +01:00
|
|
|
else if (query_buf->len > 0)
|
|
|
|
{
|
|
|
|
result = PSCAN_EOL;
|
|
|
|
*prompt = PROMPT_CONTINUE;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* never bother to send an empty buffer */
|
|
|
|
result = PSCAN_INCOMPLETE;
|
|
|
|
*prompt = PROMPT_READY;
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case xb:
|
|
|
|
result = PSCAN_INCOMPLETE;
|
|
|
|
*prompt = PROMPT_SINGLEQUOTE;
|
|
|
|
break;
|
|
|
|
case xc:
|
|
|
|
result = PSCAN_INCOMPLETE;
|
|
|
|
*prompt = PROMPT_COMMENT;
|
|
|
|
break;
|
|
|
|
case xd:
|
|
|
|
result = PSCAN_INCOMPLETE;
|
|
|
|
*prompt = PROMPT_DOUBLEQUOTE;
|
|
|
|
break;
|
|
|
|
case xh:
|
|
|
|
result = PSCAN_INCOMPLETE;
|
|
|
|
*prompt = PROMPT_SINGLEQUOTE;
|
|
|
|
break;
|
2010-10-27 04:23:04 +02:00
|
|
|
case xe:
|
2004-02-19 20:40:09 +01:00
|
|
|
result = PSCAN_INCOMPLETE;
|
|
|
|
*prompt = PROMPT_SINGLEQUOTE;
|
|
|
|
break;
|
2010-10-27 04:23:04 +02:00
|
|
|
case xq:
|
2006-03-06 20:49:20 +01:00
|
|
|
result = PSCAN_INCOMPLETE;
|
|
|
|
*prompt = PROMPT_SINGLEQUOTE;
|
|
|
|
break;
|
2004-02-24 22:45:18 +01:00
|
|
|
case xdolq:
|
|
|
|
result = PSCAN_INCOMPLETE;
|
|
|
|
*prompt = PROMPT_DOLLARQUOTE;
|
|
|
|
break;
|
2010-10-27 04:23:04 +02:00
|
|
|
case xui:
|
|
|
|
result = PSCAN_INCOMPLETE;
|
|
|
|
*prompt = PROMPT_DOUBLEQUOTE;
|
|
|
|
break;
|
|
|
|
case xus:
|
|
|
|
result = PSCAN_INCOMPLETE;
|
|
|
|
*prompt = PROMPT_SINGLEQUOTE;
|
|
|
|
break;
|
2004-02-19 20:40:09 +01:00
|
|
|
default:
|
|
|
|
/* can't get here */
|
|
|
|
fprintf(stderr, "invalid YY_START\n");
|
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case LEXRES_SEMI: /* semicolon */
|
|
|
|
result = PSCAN_SEMICOLON;
|
|
|
|
*prompt = PROMPT_READY;
|
|
|
|
break;
|
|
|
|
case LEXRES_BACKSLASH: /* backslash */
|
|
|
|
result = PSCAN_BACKSLASH;
|
|
|
|
*prompt = PROMPT_READY;
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
/* can't get here */
|
|
|
|
fprintf(stderr, "invalid yylex result\n");
|
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Clean up after scanning a string. This flushes any unread input and
|
|
|
|
* releases resources (but not the PsqlScanState itself). Note however
|
|
|
|
* that this does not reset the lexer scan state; that can be done by
|
|
|
|
* psql_scan_reset(), which is an orthogonal operation.
|
|
|
|
*
|
|
|
|
* It is legal to call this when not scanning anything (makes it easier
|
|
|
|
* to deal with error recovery).
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
psql_scan_finish(PsqlScanState state)
|
|
|
|
{
|
|
|
|
/* Drop any incomplete variable expansions. */
|
|
|
|
while (state->buffer_stack != NULL)
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
psqlscan_pop_buffer_stack(state);
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
/* Done with the outer scan buffer, too */
|
|
|
|
if (state->scanbufhandle)
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
yy_delete_buffer(state->scanbufhandle, state->scanner);
|
2004-02-19 20:40:09 +01:00
|
|
|
state->scanbufhandle = NULL;
|
|
|
|
if (state->scanbuf)
|
|
|
|
free(state->scanbuf);
|
|
|
|
state->scanbuf = NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Reset lexer scanning state to start conditions. This is appropriate
|
|
|
|
* for executing \r psql commands (or any other time that we discard the
|
|
|
|
* prior contents of query_buf). It is not, however, necessary to do this
|
|
|
|
* when we execute and clear the buffer after getting a PSCAN_SEMICOLON or
|
|
|
|
* PSCAN_EOL scan result, because the scan state must be INITIAL when those
|
|
|
|
* conditions are returned.
|
|
|
|
*
|
|
|
|
* Note that this is unrelated to flushing unread input; that task is
|
|
|
|
* done by psql_scan_finish().
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
psql_scan_reset(PsqlScanState state)
|
|
|
|
{
|
|
|
|
state->start_state = INITIAL;
|
|
|
|
state->paren_depth = 0;
|
|
|
|
state->xcdepth = 0; /* not really necessary */
|
2004-02-24 22:45:18 +01:00
|
|
|
if (state->dolqstart)
|
|
|
|
free(state->dolqstart);
|
|
|
|
state->dolqstart = NULL;
|
2021-04-07 21:30:08 +02:00
|
|
|
state->identifier_count = 0;
|
|
|
|
state->begin_depth = 0;
|
2004-02-19 20:40:09 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
* Reselect this lexer (psqlscan.l) after using another one.
|
2004-02-19 20:40:09 +01:00
|
|
|
*
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
* Currently and for foreseeable uses, it's sufficient to reset to INITIAL
|
|
|
|
* state, because we'd never switch to another lexer in a different state.
|
|
|
|
* However, we don't want to reset e.g. paren_depth, so this can't be
|
|
|
|
* the same as psql_scan_reset().
|
2004-02-19 20:40:09 +01:00
|
|
|
*
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
* Note: psql setjmp error recovery just calls psql_scan_reset(), so that
|
|
|
|
* must be a superset of this.
|
2004-02-19 20:40:09 +01:00
|
|
|
*
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
* Note: it seems likely that other lexers could just assign INITIAL for
|
|
|
|
* themselves, since that probably has the value zero in every flex-generated
|
|
|
|
* lexer. But let's not assume that.
|
2004-02-19 20:40:09 +01:00
|
|
|
*/
|
|
|
|
void
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
psql_scan_reselect_sql_lexer(PsqlScanState state)
|
2004-02-19 20:40:09 +01:00
|
|
|
{
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
state->start_state = INITIAL;
|
2004-02-19 20:40:09 +01:00
|
|
|
}
|
|
|
|
|
2011-08-26 19:52:23 +02:00
|
|
|
/*
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
* Return true if lexer is currently in an "inside quotes" state.
|
2011-08-26 19:52:23 +02:00
|
|
|
*
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
* This is pretty grotty but is needed to preserve the old behavior
|
|
|
|
* that mainloop.c drops blank lines not inside quotes without even
|
|
|
|
* echoing them.
|
2011-08-26 19:52:23 +02:00
|
|
|
*/
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
bool
|
|
|
|
psql_scan_in_quote(PsqlScanState state)
|
2011-08-26 19:52:23 +02:00
|
|
|
{
|
Reduce size of backend scanner's tables.
Previously, the core scanner's yy_transition[] array had 37045 elements.
Since that number is larger than INT16_MAX, Flex generated the array to
contain 32-bit integers. By reimplementing some of the bulkier scanner
rules, this patch reduces the array to 20495 elements. The much smaller
total length, combined with the consequent use of 16-bit integers for
the array elements reduces the binary size by over 200kB. This was
accomplished in two ways:
1. Consolidate handling of quote continuations into a new start condition,
rather than duplicating that logic for five different string types.
2. Treat Unicode strings and identifiers followed by a UESCAPE sequence
as three separate tokens, rather than one. The logic to de-escape
Unicode strings is moved to the filter code in parser.c, which already
had the ability to provide special processing for token sequences.
While we could have implemented the conversion in the grammar, that
approach was rejected for performance and maintainability reasons.
Performance in microbenchmarks of raw parsing seems equal or slightly
faster in most cases, and it's reasonable to expect that in real-world
usage (with more competition for the CPU cache) there will be a larger
win. The exception is UESCAPE sequences; lexing those is about 10%
slower, primarily because the scanner now has to be called three times
rather than one. This seems acceptable since that feature is very
rarely used.
The psql and epcg lexers are likewise modified, primarily because we
want to keep them all in sync. Since those lexers don't use the
space-hogging -CF option, the space savings is much less, but it's
still good for perhaps 10kB apiece.
While at it, merge the ecpg lexer's handling of C-style comments used
in SQL and in C. Those have different rules regarding nested comments,
but since we already have the ability to keep track of the previous
start condition, we can use that to handle both cases within a single
start condition. This matches the core scanner more closely.
John Naylor
Discussion: https://postgr.es/m/CACPNZCvaoa3EgVWm5yZhcSTX6RAtaLgniCPcBVOCwm8h3xpWkw@mail.gmail.com
2020-01-13 21:04:31 +01:00
|
|
|
return state->start_state != INITIAL &&
|
|
|
|
state->start_state != xqs;
|
2011-08-26 19:52:23 +02:00
|
|
|
}
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Push the given string onto the stack of stuff to scan.
|
|
|
|
*
|
|
|
|
* NOTE SIDE EFFECT: the new buffer is made the active flex input buffer.
|
|
|
|
*/
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
void
|
|
|
|
psqlscan_push_new_buffer(PsqlScanState state, const char *newstr,
|
|
|
|
const char *varname)
|
2004-02-19 20:40:09 +01:00
|
|
|
{
|
|
|
|
StackElem *stackelem;
|
|
|
|
|
|
|
|
stackelem = (StackElem *) pg_malloc(sizeof(StackElem));
|
2010-05-06 00:18:56 +02:00
|
|
|
|
|
|
|
/*
|
2016-03-19 19:36:22 +01:00
|
|
|
* In current usage, the passed varname points at the current flex input
|
|
|
|
* buffer; we must copy it before calling psqlscan_prepare_buffer()
|
2010-05-06 00:18:56 +02:00
|
|
|
* because that will change the buffer state.
|
|
|
|
*/
|
|
|
|
stackelem->varname = varname ? pg_strdup(varname) : NULL;
|
|
|
|
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
stackelem->buf = psqlscan_prepare_buffer(state, newstr, strlen(newstr),
|
|
|
|
&stackelem->bufstring);
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
state->curline = stackelem->bufstring;
|
|
|
|
if (state->safe_encoding)
|
2004-02-19 20:40:09 +01:00
|
|
|
{
|
|
|
|
stackelem->origstring = NULL;
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
state->refline = stackelem->bufstring;
|
2004-02-19 20:40:09 +01:00
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
stackelem->origstring = pg_strdup(newstr);
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
state->refline = stackelem->origstring;
|
2004-02-19 20:40:09 +01:00
|
|
|
}
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
stackelem->next = state->buffer_stack;
|
|
|
|
state->buffer_stack = stackelem;
|
2004-02-19 20:40:09 +01:00
|
|
|
}
|
|
|
|
|
2010-05-06 00:18:56 +02:00
|
|
|
/*
|
|
|
|
* Pop the topmost buffer stack item (there must be one!)
|
|
|
|
*
|
|
|
|
* NB: after this, the flex input state is unspecified; caller must
|
|
|
|
* switch to an appropriate buffer to continue lexing.
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
* See psqlscan_select_top_buffer().
|
2010-05-06 00:18:56 +02:00
|
|
|
*/
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
void
|
|
|
|
psqlscan_pop_buffer_stack(PsqlScanState state)
|
2010-05-06 00:18:56 +02:00
|
|
|
{
|
|
|
|
StackElem *stackelem = state->buffer_stack;
|
|
|
|
|
|
|
|
state->buffer_stack = stackelem->next;
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
yy_delete_buffer(stackelem->buf, state->scanner);
|
2010-05-06 00:18:56 +02:00
|
|
|
free(stackelem->bufstring);
|
|
|
|
if (stackelem->origstring)
|
|
|
|
free(stackelem->origstring);
|
|
|
|
if (stackelem->varname)
|
|
|
|
free(stackelem->varname);
|
|
|
|
free(stackelem);
|
|
|
|
}
|
|
|
|
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
/*
|
|
|
|
* Select the topmost surviving buffer as the active input.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
psqlscan_select_top_buffer(PsqlScanState state)
|
|
|
|
{
|
|
|
|
StackElem *stackelem = state->buffer_stack;
|
|
|
|
|
|
|
|
if (stackelem != NULL)
|
|
|
|
{
|
|
|
|
yy_switch_to_buffer(stackelem->buf, state->scanner);
|
|
|
|
state->curline = stackelem->bufstring;
|
|
|
|
state->refline = stackelem->origstring ? stackelem->origstring : stackelem->bufstring;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
yy_switch_to_buffer(state->scanbufhandle, state->scanner);
|
|
|
|
state->curline = state->scanbuf;
|
|
|
|
state->refline = state->scanline;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2010-05-06 00:18:56 +02:00
|
|
|
/*
|
|
|
|
* Check if specified variable name is the source for any string
|
|
|
|
* currently being scanned
|
|
|
|
*/
|
2016-03-25 01:28:47 +01:00
|
|
|
bool
|
|
|
|
psqlscan_var_is_current_source(PsqlScanState state, const char *varname)
|
2010-05-06 00:18:56 +02:00
|
|
|
{
|
|
|
|
StackElem *stackelem;
|
|
|
|
|
|
|
|
for (stackelem = state->buffer_stack;
|
|
|
|
stackelem != NULL;
|
|
|
|
stackelem = stackelem->next)
|
|
|
|
{
|
|
|
|
if (stackelem->varname && strcmp(stackelem->varname, varname) == 0)
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2004-02-19 20:40:09 +01:00
|
|
|
/*
|
|
|
|
* Set up a flex input buffer to scan the given data. We always make a
|
|
|
|
* copy of the data. If working in an unsafe encoding, the copy has
|
|
|
|
* multibyte sequences replaced by FFs to avoid fooling the lexer rules.
|
|
|
|
*
|
|
|
|
* NOTE SIDE EFFECT: the new buffer is made the active flex input buffer.
|
|
|
|
*/
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
YY_BUFFER_STATE
|
|
|
|
psqlscan_prepare_buffer(PsqlScanState state, const char *txt, int len,
|
|
|
|
char **txtcopy)
|
2004-02-19 20:40:09 +01:00
|
|
|
{
|
|
|
|
char *newtxt;
|
|
|
|
|
|
|
|
/* Flex wants two \0 characters after the actual data */
|
|
|
|
newtxt = pg_malloc(len + 2);
|
|
|
|
*txtcopy = newtxt;
|
|
|
|
newtxt[len] = newtxt[len + 1] = YY_END_OF_BUFFER_CHAR;
|
|
|
|
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
if (state->safe_encoding)
|
2004-02-19 20:40:09 +01:00
|
|
|
memcpy(newtxt, txt, len);
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Gotta do it the hard way */
|
2016-03-19 19:36:22 +01:00
|
|
|
int i = 0;
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
while (i < len)
|
|
|
|
{
|
2016-03-19 19:36:22 +01:00
|
|
|
int thislen = PQmblen(txt + i, state->encoding);
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
/* first byte should always be okay... */
|
|
|
|
newtxt[i] = txt[i];
|
|
|
|
i++;
|
2012-12-02 13:11:15 +01:00
|
|
|
while (--thislen > 0 && i < len)
|
2004-02-19 20:40:09 +01:00
|
|
|
newtxt[i++] = (char) 0xFF;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
return yy_scan_buffer(newtxt, len + 2, state->scanner);
|
2004-02-19 20:40:09 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
* psqlscan_emit() --- body for ECHO macro
|
2004-02-19 20:40:09 +01:00
|
|
|
*
|
|
|
|
* NB: this must be used for ALL and ONLY the text copied from the flex
|
|
|
|
* input data. If you pass it something that is not part of the yytext
|
|
|
|
* string, you are making a mistake. Internally generated text can be
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
* appended directly to state->output_buf.
|
2004-02-19 20:40:09 +01:00
|
|
|
*/
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
void
|
|
|
|
psqlscan_emit(PsqlScanState state, const char *txt, int len)
|
2004-02-19 20:40:09 +01:00
|
|
|
{
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
PQExpBuffer output_buf = state->output_buf;
|
|
|
|
|
|
|
|
if (state->safe_encoding)
|
2004-02-19 20:40:09 +01:00
|
|
|
appendBinaryPQExpBuffer(output_buf, txt, len);
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Gotta do it the hard way */
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
const char *reference = state->refline;
|
2016-03-19 19:36:22 +01:00
|
|
|
int i;
|
2004-02-19 20:40:09 +01:00
|
|
|
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
reference += (txt - state->curline);
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
for (i = 0; i < len; i++)
|
|
|
|
{
|
2016-03-19 19:36:22 +01:00
|
|
|
char ch = txt[i];
|
2004-02-19 20:40:09 +01:00
|
|
|
|
|
|
|
if (ch == (char) 0xFF)
|
|
|
|
ch = reference[i];
|
|
|
|
appendPQExpBufferChar(output_buf, ch);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
2009-09-27 05:27:24 +02:00
|
|
|
|
2011-08-26 16:41:31 +02:00
|
|
|
/*
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
* psqlscan_extract_substring --- fetch value of (part of) the current token
|
2011-08-26 16:41:31 +02:00
|
|
|
*
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
* This is like psqlscan_emit(), except that the data is returned as a
|
|
|
|
* malloc'd string rather than being pushed directly to state->output_buf.
|
2011-08-26 16:41:31 +02:00
|
|
|
*/
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
char *
|
|
|
|
psqlscan_extract_substring(PsqlScanState state, const char *txt, int len)
|
2011-08-26 16:41:31 +02:00
|
|
|
{
|
|
|
|
char *result = (char *) pg_malloc(len + 1);
|
|
|
|
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
if (state->safe_encoding)
|
2011-08-26 16:41:31 +02:00
|
|
|
memcpy(result, txt, len);
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Gotta do it the hard way */
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
const char *reference = state->refline;
|
2016-03-19 19:36:22 +01:00
|
|
|
int i;
|
2011-08-26 16:41:31 +02:00
|
|
|
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
reference += (txt - state->curline);
|
2011-08-26 16:41:31 +02:00
|
|
|
|
|
|
|
for (i = 0; i < len; i++)
|
|
|
|
{
|
2016-03-19 19:36:22 +01:00
|
|
|
char ch = txt[i];
|
2011-08-26 16:41:31 +02:00
|
|
|
|
|
|
|
if (ch == (char) 0xFF)
|
|
|
|
ch = reference[i];
|
|
|
|
result[i] = ch;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
result[len] = '\0';
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
* psqlscan_escape_variable --- process :'VARIABLE' or :"VARIABLE"
|
2011-08-26 16:41:31 +02:00
|
|
|
*
|
|
|
|
* If the variable name is found, escape its value using the appropriate
|
|
|
|
* quoting method and emit the value to output_buf. (Since the result is
|
2016-03-19 19:36:22 +01:00
|
|
|
* surely quoted, there is never any reason to rescan it.) If we don't
|
2016-03-18 20:05:49 +01:00
|
|
|
* find the variable or escaping fails, emit the token as-is.
|
2011-08-26 16:41:31 +02:00
|
|
|
*/
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
void
|
|
|
|
psqlscan_escape_variable(PsqlScanState state, const char *txt, int len,
|
Allow psql variable substitution to occur in backtick command strings.
Previously, text between backquotes in a psql metacommand's arguments
was always passed to the shell literally. That considerably hobbles
the usefulness of the feature for scripting, so we'd foreseen for a long
time that we'd someday want to allow substitution of psql variables into
the shell command. IMO the addition of \if metacommands has brought us to
that point, since \if can greatly benefit from some sort of client-side
expression evaluation capability, and psql itself is not going to grow any
such thing in time for v10. Hence, this patch. It allows :VARIABLE to be
replaced by the exact contents of the named variable, while :'VARIABLE'
is replaced by the variable's contents suitably quoted to become a single
shell-command argument. (The quoting rules for that are different from
those for SQL literals, so this is a bit of an abuse of the :'VARIABLE'
notation, but I doubt anyone will be confused.)
As with other situations in psql, no substitution occurs if the word
following a colon is not a known variable name. That limits the risk of
compatibility problems for existing psql scripts; but the risk isn't zero,
so this needs to be called out in the v10 release notes.
Discussion: https://postgr.es/m/9561.1490895211@sss.pgh.pa.us
2017-04-02 03:44:54 +02:00
|
|
|
PsqlScanQuoteType quote)
|
2010-01-29 18:44:12 +01:00
|
|
|
{
|
2011-08-26 16:41:31 +02:00
|
|
|
char *varname;
|
2016-03-18 20:05:49 +01:00
|
|
|
char *value;
|
2010-01-29 18:44:12 +01:00
|
|
|
|
|
|
|
/* Variable lookup. */
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
varname = psqlscan_extract_substring(state, txt + 2, len - 3);
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
if (state->callbacks->get_variable)
|
Allow psql variable substitution to occur in backtick command strings.
Previously, text between backquotes in a psql metacommand's arguments
was always passed to the shell literally. That considerably hobbles
the usefulness of the feature for scripting, so we'd foreseen for a long
time that we'd someday want to allow substitution of psql variables into
the shell command. IMO the addition of \if metacommands has brought us to
that point, since \if can greatly benefit from some sort of client-side
expression evaluation capability, and psql itself is not going to grow any
such thing in time for v10. Hence, this patch. It allows :VARIABLE to be
replaced by the exact contents of the named variable, while :'VARIABLE'
is replaced by the variable's contents suitably quoted to become a single
shell-command argument. (The quoting rules for that are different from
those for SQL literals, so this is a bit of an abuse of the :'VARIABLE'
notation, but I doubt anyone will be confused.)
As with other situations in psql, no substitution occurs if the word
following a colon is not a known variable name. That limits the risk of
compatibility problems for existing psql scripts; but the risk isn't zero,
so this needs to be called out in the v10 release notes.
Discussion: https://postgr.es/m/9561.1490895211@sss.pgh.pa.us
2017-04-02 03:44:54 +02:00
|
|
|
value = state->callbacks->get_variable(varname, quote,
|
2017-03-13 22:14:46 +01:00
|
|
|
state->cb_passthrough);
|
2016-03-18 20:05:49 +01:00
|
|
|
else
|
|
|
|
value = NULL;
|
2011-08-26 16:41:31 +02:00
|
|
|
free(varname);
|
2010-01-29 18:44:12 +01:00
|
|
|
|
|
|
|
if (value)
|
|
|
|
{
|
2016-03-18 20:05:49 +01:00
|
|
|
/* Emit the suitably-escaped value */
|
Convert psql's flex lexer to be re-entrant, and make it compile standalone.
Change psqlscan.l to specify '%option reentrant', adjust internal APIs
to match, and get rid of its internal static variables. While this is
good cleanup in an abstract sense, the reason to do it right now is that
it seems the only practical way to support use of separate flex lexers
with common PsqlScanState infrastructure. If we build two non-reentrant
lexers then we are going to have problems with dangling buffer pointers
in whichever lexer isn't active when we transition from one buffer to
another, as well as curious side-effects if we try to share any code
between the files. (Horiguchi-san had a different solution to that in his
pending patch, but I find it ugly and probably broken for corner cases.)
Depending on which version of flex you're using, this may result in getting
a "warning: unused variable 'yyg'" warning from psqlscan, similar to the
one you'd have seen for a long time in backend/parser/scan.l. I put a
local -Wno-error into CFLAGS for the file, for the convenience of those
who compile with -Werror.
Also, stop compiling psqlscan as part of mainloop.c, and make it a
standalone build target instead. This is a lot cleaner than before, though
it doesn't really change much in practice as of this commit. (I'm not sure
whether the MSVC build scripts will need some help with this part, but the
buildfarm will soon tell us.)
2016-03-19 02:21:52 +01:00
|
|
|
appendPQExpBufferStr(state->output_buf, value);
|
2016-03-18 20:05:49 +01:00
|
|
|
free(value);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Emit original token as-is */
|
Split psql's lexer into two separate .l files for SQL and backslash cases.
This gets us to a point where psqlscan.l can be used by other frontend
programs for the same purpose psql uses it for, ie to detect when it's
collected a complete SQL command from input that is divided across
line boundaries. Moreover, other programs can supply their own lexers
for backslash commands of their own choosing. A follow-on patch will
use this in pgbench.
The end result here is roughly the same as in Kyotaro Horiguchi's
0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although
the details of the method for switching between lexers are quite different.
Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE
stack, *and* yyscan_t between different lexers. The only thing we need
to do to switch to a different lexer is to make sure the start_state is
valid for the new lexer. This works because flex doesn't keep any other
persistent state that depends on the specific lexing tables generated for
a particular .l file. (We are assuming that both lexers are built with
the same flex version, or at least versions that are compatible with
respect to the contents of yyscan_t; but that doesn't seem likely to
be a big problem in practice, considering how slowly flex changes.)
Aside from being more efficient than Horiguchi-san's original solution,
this avoids possible corner-case changes in semantics: the original code
was capable of popping the input buffer stack while still staying in
backslash-related parsing states. I'm not sure that that equates to any
useful user-visible behaviors, but I'm not sure it doesn't either, so
I'm loath to assume that we only need to consider the topmost buffer when
parsing a backslash command.
I've attempted to update the MSVC build scripts for the added .l file,
but will rely on the buildfarm to see if I missed anything.
Kyotaro Horiguchi and Tom Lane
2016-03-19 05:24:55 +01:00
|
|
|
psqlscan_emit(state, txt, len);
|
2010-01-29 18:44:12 +01:00
|
|
|
}
|
|
|
|
}
|
2017-09-22 01:02:23 +02:00
|
|
|
|
|
|
|
void
|
|
|
|
psqlscan_test_variable(PsqlScanState state, const char *txt, int len)
|
|
|
|
{
|
|
|
|
char *varname;
|
|
|
|
char *value;
|
|
|
|
|
|
|
|
varname = psqlscan_extract_substring(state, txt + 3, len - 4);
|
|
|
|
if (state->callbacks->get_variable)
|
|
|
|
value = state->callbacks->get_variable(varname, PQUOTE_PLAIN,
|
|
|
|
state->cb_passthrough);
|
|
|
|
else
|
|
|
|
value = NULL;
|
|
|
|
free(varname);
|
|
|
|
|
|
|
|
if (value != NULL)
|
|
|
|
{
|
|
|
|
psqlscan_emit(state, "TRUE", 4);
|
|
|
|
free(value);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
psqlscan_emit(state, "FALSE", 5);
|
|
|
|
}
|
|
|
|
}
|