2011-02-20 06:17:18 +01:00
|
|
|
<!-- doc/src/sgml/fdwhandler.sgml -->
|
|
|
|
|
|
|
|
<chapter id="fdwhandler">
|
|
|
|
<title>Writing A Foreign Data Wrapper</title>
|
|
|
|
|
|
|
|
<indexterm zone="fdwhandler">
|
|
|
|
<primary>foreign data wrapper</primary>
|
|
|
|
<secondary>handler for</secondary>
|
|
|
|
</indexterm>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
All operations on a foreign table are handled through its foreign data
|
2012-04-06 21:02:35 +02:00
|
|
|
wrapper, which consists of a set of functions that the core server
|
|
|
|
calls. The foreign data wrapper is responsible for fetching
|
2011-02-20 06:17:18 +01:00
|
|
|
data from the remote data source and returning it to the
|
2013-03-10 19:14:53 +01:00
|
|
|
<productname>PostgreSQL</productname> executor. If updating foreign
|
|
|
|
tables is to be supported, the wrapper must handle that, too.
|
|
|
|
This chapter outlines how to write a new foreign data wrapper.
|
2011-02-20 06:17:18 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The foreign data wrappers included in the standard distribution are good
|
|
|
|
references when trying to write your own. Look into the
|
2017-10-09 03:44:17 +02:00
|
|
|
<filename>contrib</filename> subdirectory of the source tree.
|
2017-11-23 15:39:47 +01:00
|
|
|
The <xref linkend="sql-createforeigndatawrapper"/> reference page also has
|
2011-02-20 06:17:18 +01:00
|
|
|
some useful details.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<note>
|
|
|
|
<para>
|
|
|
|
The SQL standard specifies an interface for writing foreign data wrappers.
|
|
|
|
However, PostgreSQL does not implement that API, because the effort to
|
|
|
|
accommodate it into PostgreSQL would be large, and the standard API hasn't
|
|
|
|
gained wide adoption anyway.
|
|
|
|
</para>
|
|
|
|
</note>
|
|
|
|
|
2011-07-05 21:54:00 +02:00
|
|
|
<sect1 id="fdw-functions">
|
|
|
|
<title>Foreign Data Wrapper Functions</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The FDW author needs to implement a handler function, and optionally
|
|
|
|
a validator function. Both functions must be written in a compiled
|
|
|
|
language such as C, using the version-1 interface.
|
|
|
|
For details on C language calling conventions and dynamic loading,
|
2017-11-23 15:39:47 +01:00
|
|
|
see <xref linkend="xfunc-c"/>.
|
2011-07-05 21:54:00 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The handler function simply returns a struct of function pointers to
|
2012-04-06 21:02:35 +02:00
|
|
|
callback functions that will be called by the planner, executor, and
|
|
|
|
various maintenance commands.
|
2011-07-05 21:54:00 +02:00
|
|
|
Most of the effort in writing an FDW is in implementing these callback
|
|
|
|
functions.
|
|
|
|
The handler function must be registered with
|
|
|
|
<productname>PostgreSQL</productname> as taking no arguments and
|
|
|
|
returning the special pseudo-type <type>fdw_handler</type>. The
|
|
|
|
callback functions are plain C functions and are not visible or
|
|
|
|
callable at the SQL level. The callback functions are described in
|
2017-11-23 15:39:47 +01:00
|
|
|
<xref linkend="fdw-callbacks"/>.
|
2011-07-05 21:54:00 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The validator function is responsible for validating options given in
|
|
|
|
<command>CREATE</command> and <command>ALTER</command> commands for its
|
|
|
|
foreign data wrapper, as well as foreign servers, user mappings, and
|
|
|
|
foreign tables using the wrapper.
|
|
|
|
The validator function must be registered as taking two arguments, a
|
|
|
|
text array containing the options to be validated, and an OID
|
|
|
|
representing the type of object the options are associated with (in
|
|
|
|
the form of the OID of the system catalog the object would be stored
|
|
|
|
in, either
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>ForeignDataWrapperRelationId</literal>,
|
|
|
|
<literal>ForeignServerRelationId</literal>,
|
|
|
|
<literal>UserMappingRelationId</literal>,
|
|
|
|
or <literal>ForeignTableRelationId</literal>).
|
2011-07-05 21:54:00 +02:00
|
|
|
If no validator function is supplied, options are not checked at object
|
|
|
|
creation time or object alteration time.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
</sect1>
|
|
|
|
|
|
|
|
<sect1 id="fdw-callbacks">
|
2011-02-20 06:17:18 +01:00
|
|
|
<title>Foreign Data Wrapper Callback Routines</title>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
The FDW handler function returns a palloc'd <structname>FdwRoutine</structname>
|
2013-03-10 19:14:53 +01:00
|
|
|
struct containing pointers to the callback functions described below.
|
|
|
|
The scan-related functions are required, the rest are optional.
|
2011-02-20 06:17:18 +01:00
|
|
|
</para>
|
|
|
|
|
2013-03-10 19:14:53 +01:00
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
The <structname>FdwRoutine</structname> struct type is declared in
|
|
|
|
<filename>src/include/foreign/fdwapi.h</filename>, which see for additional
|
2013-03-10 19:14:53 +01:00
|
|
|
details.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<sect2 id="fdw-callbacks-scan">
|
|
|
|
<title>FDW Routines For Scanning Foreign Tables</title>
|
|
|
|
|
2011-02-20 06:17:18 +01:00
|
|
|
<para>
|
|
|
|
<programlisting>
|
2012-03-05 22:15:59 +01:00
|
|
|
void
|
2017-09-06 17:38:28 +02:00
|
|
|
GetForeignRelSize(PlannerInfo *root,
|
|
|
|
RelOptInfo *baserel,
|
|
|
|
Oid foreigntableid);
|
2011-02-20 06:17:18 +01:00
|
|
|
</programlisting>
|
|
|
|
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
Obtain relation size estimates for a foreign table. This is called
|
2013-03-10 19:14:53 +01:00
|
|
|
at the beginning of planning for a query that scans a foreign table.
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>root</literal> is the planner's global information about the query;
|
|
|
|
<literal>baserel</literal> is the planner's information about this table; and
|
|
|
|
<literal>foreigntableid</literal> is the <structname>pg_class</structname> OID of the
|
|
|
|
foreign table. (<literal>foreigntableid</literal> could be obtained from the
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
planner data structures, but it's passed explicitly to save effort.)
|
2012-03-05 22:15:59 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
This function should update <literal>baserel->rows</literal> to be the
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
expected number of rows returned by the table scan, after accounting for
|
|
|
|
the filtering done by the restriction quals. The initial value of
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>baserel->rows</literal> is just a constant default estimate, which
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
should be replaced if at all possible. The function may also choose to
|
2017-10-09 03:44:17 +02:00
|
|
|
update <literal>baserel->width</literal> if it can compute a better estimate
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
of the average result row width.
|
2011-02-20 06:17:18 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-11-23 15:39:47 +01:00
|
|
|
See <xref linkend="fdw-planning"/> for additional information.
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
2017-09-06 17:38:28 +02:00
|
|
|
GetForeignPaths(PlannerInfo *root,
|
|
|
|
RelOptInfo *baserel,
|
|
|
|
Oid foreigntableid);
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
Create possible access paths for a scan on a foreign table.
|
|
|
|
This is called during query planning.
|
2017-10-09 03:44:17 +02:00
|
|
|
The parameters are the same as for <function>GetForeignRelSize</function>,
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
which has already been called.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
This function must generate at least one access path
|
2017-10-09 03:44:17 +02:00
|
|
|
(<structname>ForeignPath</structname> node) for a scan on the foreign table and
|
|
|
|
must call <function>add_path</function> to add each such path to
|
|
|
|
<literal>baserel->pathlist</literal>. It's recommended to use
|
|
|
|
<function>create_foreignscan_path</function> to build the
|
|
|
|
<structname>ForeignPath</structname> nodes. The function can generate multiple
|
|
|
|
access paths, e.g., a path which has valid <literal>pathkeys</literal> to
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
represent a pre-sorted result. Each access path must contain cost
|
|
|
|
estimates, and can contain any FDW-private information that is needed to
|
|
|
|
identify the specific scan method intended.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-11-23 15:39:47 +01:00
|
|
|
See <xref linkend="fdw-planning"/> for additional information.
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
ForeignScan *
|
2017-09-06 17:38:28 +02:00
|
|
|
GetForeignPlan(PlannerInfo *root,
|
|
|
|
RelOptInfo *baserel,
|
|
|
|
Oid foreigntableid,
|
|
|
|
ForeignPath *best_path,
|
|
|
|
List *tlist,
|
|
|
|
List *scan_clauses,
|
|
|
|
Plan *outer_plan);
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
</programlisting>
|
|
|
|
|
2017-10-09 03:44:17 +02:00
|
|
|
Create a <structname>ForeignScan</structname> plan node from the selected foreign
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
access path. This is called at the end of query planning.
|
2017-10-09 03:44:17 +02:00
|
|
|
The parameters are as for <function>GetForeignRelSize</function>, plus
|
|
|
|
the selected <structname>ForeignPath</structname> (previously produced by
|
|
|
|
<function>GetForeignPaths</function>, <function>GetForeignJoinPaths</function>,
|
|
|
|
or <function>GetForeignUpperPaths</function>),
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
the target list to be emitted by the plan node,
|
2016-03-08 20:30:12 +01:00
|
|
|
the restriction clauses to be enforced by the plan node,
|
2017-10-09 03:44:17 +02:00
|
|
|
and the outer subplan of the <structname>ForeignScan</structname>,
|
|
|
|
which is used for rechecks performed by <function>RecheckForeignScan</function>.
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
(If the path is for a join rather than a base
|
2017-10-09 03:44:17 +02:00
|
|
|
relation, <literal>foreigntableid</literal> is <literal>InvalidOid</literal>.)
|
2011-02-20 06:17:18 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
This function must create and return a <structname>ForeignScan</structname> plan
|
|
|
|
node; it's recommended to use <function>make_foreignscan</function> to build the
|
|
|
|
<structname>ForeignScan</structname> node.
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-11-23 15:39:47 +01:00
|
|
|
See <xref linkend="fdw-planning"/> for additional information.
|
2011-02-20 06:17:18 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
2017-09-06 17:38:28 +02:00
|
|
|
BeginForeignScan(ForeignScanState *node,
|
|
|
|
int eflags);
|
2011-02-20 06:17:18 +01:00
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
Begin executing a foreign scan. This is called during executor startup.
|
2011-07-25 15:37:06 +02:00
|
|
|
It should perform any initialization needed before the scan can start,
|
|
|
|
but not start executing the actual scan (that should be done upon the
|
2017-10-09 03:44:17 +02:00
|
|
|
first call to <function>IterateForeignScan</function>).
|
|
|
|
The <structname>ForeignScanState</structname> node has already been created, but
|
|
|
|
its <structfield>fdw_state</structfield> field is still NULL. Information about
|
2011-02-20 06:17:18 +01:00
|
|
|
the table to scan is accessible through the
|
2017-10-09 03:44:17 +02:00
|
|
|
<structname>ForeignScanState</structname> node (in particular, from the underlying
|
|
|
|
<structname>ForeignScan</structname> plan node, which contains any FDW-private
|
|
|
|
information provided by <function>GetForeignPlan</function>).
|
|
|
|
<literal>eflags</literal> contains flag bits describing the executor's
|
2013-03-10 19:14:53 +01:00
|
|
|
operating mode for this plan node.
|
2011-02-20 06:17:18 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
Note that when <literal>(eflags & EXEC_FLAG_EXPLAIN_ONLY)</literal> is
|
2011-02-20 06:17:18 +01:00
|
|
|
true, this function should not perform any externally-visible actions;
|
|
|
|
it should only do the minimum required to make the node state valid
|
2017-10-09 03:44:17 +02:00
|
|
|
for <function>ExplainForeignScan</function> and <function>EndForeignScan</function>.
|
2011-02-20 06:17:18 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
TupleTableSlot *
|
2017-09-06 17:38:28 +02:00
|
|
|
IterateForeignScan(ForeignScanState *node);
|
2011-02-20 06:17:18 +01:00
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
Fetch one row from the foreign source, returning it in a tuple table slot
|
2017-10-09 03:44:17 +02:00
|
|
|
(the node's <structfield>ScanTupleSlot</structfield> should be used for this
|
2011-02-20 06:17:18 +01:00
|
|
|
purpose). Return NULL if no more rows are available. The tuple table
|
|
|
|
slot infrastructure allows either a physical or virtual tuple to be
|
|
|
|
returned; in most cases the latter choice is preferable from a
|
|
|
|
performance standpoint. Note that this is called in a short-lived memory
|
|
|
|
context that will be reset between invocations. Create a memory context
|
2017-10-09 03:44:17 +02:00
|
|
|
in <function>BeginForeignScan</function> if you need longer-lived storage, or use
|
|
|
|
the <structfield>es_query_cxt</structfield> of the node's <structname>EState</structname>.
|
2011-02-20 06:17:18 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
The rows returned must match the <structfield>fdw_scan_tlist</structfield> target
|
2015-09-11 03:22:21 +02:00
|
|
|
list if one was supplied, otherwise they must match the row type of the
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
foreign table being scanned. If you choose to optimize away fetching
|
|
|
|
columns that are not needed, you should insert nulls in those column
|
2017-10-09 03:44:17 +02:00
|
|
|
positions, or else generate a <structfield>fdw_scan_tlist</structfield> list with
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
those columns omitted.
|
2011-02-20 06:17:18 +01:00
|
|
|
</para>
|
|
|
|
|
2011-06-13 14:28:41 +02:00
|
|
|
<para>
|
|
|
|
Note that <productname>PostgreSQL</productname>'s executor doesn't care
|
2015-04-21 23:46:47 +02:00
|
|
|
whether the rows returned violate any constraints that were defined on
|
|
|
|
the foreign table — but the planner does care, and may optimize
|
|
|
|
queries incorrectly if there are rows visible in the foreign table that
|
|
|
|
do not satisfy a declared constraint. If a constraint is violated when
|
|
|
|
the user has declared that the constraint should hold true, it may be
|
|
|
|
appropriate to raise an error (just as you would need to do in the case
|
|
|
|
of a data type mismatch).
|
2011-06-13 14:28:41 +02:00
|
|
|
</para>
|
|
|
|
|
2011-02-20 06:17:18 +01:00
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
2017-09-06 17:38:28 +02:00
|
|
|
ReScanForeignScan(ForeignScanState *node);
|
2011-02-20 06:17:18 +01:00
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
Restart the scan from the beginning. Note that any parameters the
|
|
|
|
scan depends on may have changed value, so the new scan does not
|
|
|
|
necessarily return exactly the same rows.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
2017-09-06 17:38:28 +02:00
|
|
|
EndForeignScan(ForeignScanState *node);
|
2011-02-20 06:17:18 +01:00
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
End the scan and release resources. It is normally not important
|
|
|
|
to release palloc'd memory, but for example open files and connections
|
|
|
|
to remote servers should be cleaned up.
|
|
|
|
</para>
|
|
|
|
|
2013-03-10 19:14:53 +01:00
|
|
|
</sect2>
|
|
|
|
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
<sect2 id="fdw-callbacks-join-scan">
|
|
|
|
<title>FDW Routines For Scanning Foreign Joins</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
If an FDW supports performing foreign joins remotely (rather than
|
|
|
|
by fetching both tables' data and doing the join locally), it should
|
|
|
|
provide this callback function:
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
2017-09-06 17:38:28 +02:00
|
|
|
GetForeignJoinPaths(PlannerInfo *root,
|
|
|
|
RelOptInfo *joinrel,
|
|
|
|
RelOptInfo *outerrel,
|
|
|
|
RelOptInfo *innerrel,
|
|
|
|
JoinType jointype,
|
|
|
|
JoinPathExtraData *extra);
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
</programlisting>
|
|
|
|
Create possible access paths for a join of two (or more) foreign tables
|
|
|
|
that all belong to the same foreign server. This optional
|
|
|
|
function is called during query planning. As
|
2017-10-09 03:44:17 +02:00
|
|
|
with <function>GetForeignPaths</function>, this function should
|
|
|
|
generate <structname>ForeignPath</structname> path(s) for the
|
Split create_foreignscan_path() into three functions.
Up to now postgres_fdw has been using create_foreignscan_path() to
generate not only base-relation paths, but also paths for foreign joins
and foreign upperrels. This is wrong, because create_foreignscan_path()
calls get_baserel_parampathinfo() which will only do the right thing for
baserels. It accidentally fails to fail for unparameterized paths, which
are the only ones postgres_fdw (thought it) was handling, but we really
need different APIs for the baserel and join cases.
In HEAD, the best thing to do seems to be to split up the baserel,
joinrel, and upperrel cases into three functions so that they can
have different APIs. I haven't actually given create_foreign_join_path
a different API in this commit: we should spend a bit of time thinking
about just what we want to do there, since perhaps FDWs would want to
do something different from the build-up-a-join-pairwise approach that
get_joinrel_parampathinfo expects. In the meantime, since postgres_fdw
isn't prepared to generate parameterized joins anyway, just give it a
defense against trying to plan joins with lateral refs.
In addition (and this is what triggered this whole mess) fix bug #15613
from Srinivasan S A, by teaching file_fdw and postgres_fdw that plain
baserel foreign paths still have outer refs if the relation has
lateral_relids. Add some assertions in relnode.c to catch future
occurrences of the same error --- in particular, to catch other FDWs
doing that, but also as backstop against core-code mistakes like the
one fixed by commit bdd9a99aa.
Bug #15613 also needs to be fixed in the back branches, but the
appropriate fix will look quite a bit different there, since we don't
want to assume that existing FDWs get the word right away.
Discussion: https://postgr.es/m/15613-092be1be9576c728@postgresql.org
2019-02-07 18:59:47 +01:00
|
|
|
supplied <literal>joinrel</literal>
|
|
|
|
(use <function>create_foreign_join_path</function> to build them),
|
|
|
|
and call <function>add_path</function> to add these
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
paths to the set of paths considered for the join. But unlike
|
2017-10-09 03:44:17 +02:00
|
|
|
<function>GetForeignPaths</function>, it is not necessary that this function
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
succeed in creating at least one path, since paths involving local
|
|
|
|
joining are always possible.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Note that this function will be invoked repeatedly for the same join
|
|
|
|
relation, with different combinations of inner and outer relations; it is
|
|
|
|
the responsibility of the FDW to minimize duplicated work.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If a <structname>ForeignPath</structname> path is chosen for the join, it will
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
represent the entire join process; paths generated for the component
|
|
|
|
tables and subsidiary joins will not be used. Subsequent processing of
|
|
|
|
the join path proceeds much as it does for a path scanning a single
|
2017-10-09 03:44:17 +02:00
|
|
|
foreign table. One difference is that the <structfield>scanrelid</structfield> of
|
|
|
|
the resulting <structname>ForeignScan</structname> plan node should be set to zero,
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
since there is no single relation that it represents; instead,
|
2017-10-09 03:44:17 +02:00
|
|
|
the <structfield>fs_relids</structfield> field of the <structname>ForeignScan</structname>
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
node represents the set of relations that were joined. (The latter field
|
|
|
|
is set up automatically by the core planner code, and need not be filled
|
|
|
|
by the FDW.) Another difference is that, because the column list for a
|
|
|
|
remote join cannot be found from the system catalogs, the FDW must
|
2017-10-09 03:44:17 +02:00
|
|
|
fill <structfield>fdw_scan_tlist</structfield> with an appropriate list
|
|
|
|
of <structfield>TargetEntry</structfield> nodes, representing the set of columns
|
2015-09-11 03:22:21 +02:00
|
|
|
it will supply at run time in the tuples it returns.
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-11-23 15:39:47 +01:00
|
|
|
See <xref linkend="fdw-planning"/> for additional information.
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
</para>
|
|
|
|
</sect2>
|
|
|
|
|
2016-03-15 01:04:44 +01:00
|
|
|
<sect2 id="fdw-callbacks-upper-planning">
|
|
|
|
<title>FDW Routines For Planning Post-Scan/Join Processing</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
If an FDW supports performing remote post-scan/join processing, such as
|
|
|
|
remote aggregation, it should provide this callback function:
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
2017-09-06 17:38:28 +02:00
|
|
|
GetForeignUpperPaths(PlannerInfo *root,
|
|
|
|
UpperRelationKind stage,
|
|
|
|
RelOptInfo *input_rel,
|
2018-04-02 16:51:50 +02:00
|
|
|
RelOptInfo *output_rel,
|
|
|
|
void *extra);
|
2016-03-15 01:04:44 +01:00
|
|
|
</programlisting>
|
2017-10-09 03:44:17 +02:00
|
|
|
Create possible access paths for <firstterm>upper relation</firstterm> processing,
|
2016-03-15 01:04:44 +01:00
|
|
|
which is the planner's term for all post-scan/join query processing, such
|
|
|
|
as aggregation, window functions, sorting, and table updates. This
|
|
|
|
optional function is called during query planning. Currently, it is
|
|
|
|
called only if all base relation(s) involved in the query belong to the
|
2017-10-09 03:44:17 +02:00
|
|
|
same FDW. This function should generate <structname>ForeignPath</structname>
|
2016-07-01 19:12:34 +02:00
|
|
|
path(s) for any post-scan/join processing that the FDW knows how to
|
Split create_foreignscan_path() into three functions.
Up to now postgres_fdw has been using create_foreignscan_path() to
generate not only base-relation paths, but also paths for foreign joins
and foreign upperrels. This is wrong, because create_foreignscan_path()
calls get_baserel_parampathinfo() which will only do the right thing for
baserels. It accidentally fails to fail for unparameterized paths, which
are the only ones postgres_fdw (thought it) was handling, but we really
need different APIs for the baserel and join cases.
In HEAD, the best thing to do seems to be to split up the baserel,
joinrel, and upperrel cases into three functions so that they can
have different APIs. I haven't actually given create_foreign_join_path
a different API in this commit: we should spend a bit of time thinking
about just what we want to do there, since perhaps FDWs would want to
do something different from the build-up-a-join-pairwise approach that
get_joinrel_parampathinfo expects. In the meantime, since postgres_fdw
isn't prepared to generate parameterized joins anyway, just give it a
defense against trying to plan joins with lateral refs.
In addition (and this is what triggered this whole mess) fix bug #15613
from Srinivasan S A, by teaching file_fdw and postgres_fdw that plain
baserel foreign paths still have outer refs if the relation has
lateral_relids. Add some assertions in relnode.c to catch future
occurrences of the same error --- in particular, to catch other FDWs
doing that, but also as backstop against core-code mistakes like the
one fixed by commit bdd9a99aa.
Bug #15613 also needs to be fixed in the back branches, but the
appropriate fix will look quite a bit different there, since we don't
want to assume that existing FDWs get the word right away.
Discussion: https://postgr.es/m/15613-092be1be9576c728@postgresql.org
2019-02-07 18:59:47 +01:00
|
|
|
perform remotely
|
|
|
|
(use <function>create_foreign_upper_path</function> to build them),
|
|
|
|
and call <function>add_path</function> to add these paths to
|
2017-10-09 03:44:17 +02:00
|
|
|
the indicated upper relation. As with <function>GetForeignJoinPaths</function>,
|
2016-07-01 19:12:34 +02:00
|
|
|
it is not necessary that this function succeed in creating any paths,
|
|
|
|
since paths involving local processing are always possible.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
The <literal>stage</literal> parameter identifies which post-scan/join step is
|
|
|
|
currently being considered. <literal>output_rel</literal> is the upper relation
|
2016-07-01 19:12:34 +02:00
|
|
|
that should receive paths representing computation of this step,
|
2017-10-09 03:44:17 +02:00
|
|
|
and <literal>input_rel</literal> is the relation representing the input to this
|
2018-04-02 16:51:50 +02:00
|
|
|
step. The <literal>extra</literal> parameter provides additional details,
|
2018-04-02 19:56:40 +02:00
|
|
|
currently, it is set only for <literal>UPPERREL_PARTIAL_GROUP_AGG</literal>
|
2018-04-02 16:51:50 +02:00
|
|
|
or <literal>UPPERREL_GROUP_AGG</literal>, in which case it points to a
|
2018-04-02 19:56:40 +02:00
|
|
|
<literal>GroupPathExtraData</literal> structure.
|
2018-04-02 16:51:50 +02:00
|
|
|
(Note that <structname>ForeignPath</structname> paths added
|
2017-10-09 03:44:17 +02:00
|
|
|
to <literal>output_rel</literal> would typically not have any direct dependency
|
|
|
|
on paths of the <literal>input_rel</literal>, since their processing is expected
|
2016-07-01 19:12:34 +02:00
|
|
|
to be done externally. However, examining paths previously generated for
|
|
|
|
the previous processing step can be useful to avoid redundant planning
|
|
|
|
work.)
|
2016-03-15 01:04:44 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-11-23 15:39:47 +01:00
|
|
|
See <xref linkend="fdw-planning"/> for additional information.
|
2016-03-15 01:04:44 +01:00
|
|
|
</para>
|
|
|
|
</sect2>
|
|
|
|
|
2013-03-10 19:14:53 +01:00
|
|
|
<sect2 id="fdw-callbacks-update">
|
|
|
|
<title>FDW Routines For Updating Foreign Tables</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
If an FDW supports writable foreign tables, it should provide
|
|
|
|
some or all of the following callback functions depending on
|
|
|
|
the needs and capabilities of the FDW:
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
2017-09-06 17:38:28 +02:00
|
|
|
AddForeignUpdateTargets(Query *parsetree,
|
|
|
|
RangeTblEntry *target_rte,
|
|
|
|
Relation target_relation);
|
2013-03-10 19:14:53 +01:00
|
|
|
</programlisting>
|
|
|
|
|
2017-10-09 03:44:17 +02:00
|
|
|
<command>UPDATE</command> and <command>DELETE</command> operations are performed
|
2013-03-10 19:14:53 +01:00
|
|
|
against rows previously fetched by the table-scanning functions. The
|
|
|
|
FDW may need extra information, such as a row ID or the values of
|
|
|
|
primary-key columns, to ensure that it can identify the exact row to
|
|
|
|
update or delete. To support that, this function can add extra hidden,
|
2017-10-09 03:44:17 +02:00
|
|
|
or <quote>junk</quote>, target columns to the list of columns that are to be
|
|
|
|
retrieved from the foreign table during an <command>UPDATE</command> or
|
|
|
|
<command>DELETE</command>.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
To do that, add <structname>TargetEntry</structname> items to
|
|
|
|
<literal>parsetree->targetList</literal>, containing expressions for the
|
2013-03-10 19:14:53 +01:00
|
|
|
extra values to be fetched. Each such entry must be marked
|
2017-10-09 03:44:17 +02:00
|
|
|
<structfield>resjunk</structfield> = <literal>true</literal>, and must have a distinct
|
|
|
|
<structfield>resname</structfield> that will identify it at execution time.
|
|
|
|
Avoid using names matching <literal>ctid<replaceable>N</replaceable></literal>,
|
2014-03-23 07:16:34 +01:00
|
|
|
<literal>wholerow</literal>, or
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>wholerow<replaceable>N</replaceable></literal>, as the core system can
|
2013-03-10 19:14:53 +01:00
|
|
|
generate junk columns of these names.
|
Fix creation of resjunk tlist entries for inherited mixed UPDATE/DELETE.
rewriteTargetListUD's processing is dependent on the relkind of the query's
target table. That was fine at the time it was made to act that way, even
for queries on inheritance trees, because all tables in an inheritance tree
would necessarily be plain tables. However, the 9.5 feature addition
allowing some members of an inheritance tree to be foreign tables broke the
assumption that rewriteTargetListUD's output tlist could be applied to all
child tables with nothing more than column-number mapping. This led to
visible failures if foreign child tables had row-level triggers, and would
also break in cases where child tables belonged to FDWs that used methods
other than CTID for row identification.
To fix, delay running rewriteTargetListUD until after the planner has
expanded inheritance, so that it is applied separately to the (already
mapped) tlist for each child table. We can conveniently call it from
preprocess_targetlist. Refactor associated code slightly to avoid the
need to heap_open the target relation multiple times during
preprocess_targetlist. (The APIs remain a bit ugly, particularly around
the point of which steps scribble on parse->targetList and which don't.
But avoiding such scribbling would require a change in FDW callback APIs,
which is more pain than it's worth.)
Also fix ExecModifyTable to ensure that "tupleid" is reset to NULL when
we transition from rows providing a CTID to rows that don't. (That's
really an independent bug, but it manifests in much the same cases.)
Add a regression test checking one manifestation of this problem, which
was that row-level triggers on a foreign child table did not work right.
Back-patch to 9.5 where the problem was introduced.
Etsuro Fujita, reviewed by Ildus Kurbangaliev and Ashutosh Bapat
Discussion: https://postgr.es/m/20170514150525.0346ba72@postgrespro.ru
2017-11-27 23:53:56 +01:00
|
|
|
If the extra expressions are more complex than simple Vars, they
|
|
|
|
must be run through <function>eval_const_expressions</function>
|
|
|
|
before adding them to the targetlist.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
Fix creation of resjunk tlist entries for inherited mixed UPDATE/DELETE.
rewriteTargetListUD's processing is dependent on the relkind of the query's
target table. That was fine at the time it was made to act that way, even
for queries on inheritance trees, because all tables in an inheritance tree
would necessarily be plain tables. However, the 9.5 feature addition
allowing some members of an inheritance tree to be foreign tables broke the
assumption that rewriteTargetListUD's output tlist could be applied to all
child tables with nothing more than column-number mapping. This led to
visible failures if foreign child tables had row-level triggers, and would
also break in cases where child tables belonged to FDWs that used methods
other than CTID for row identification.
To fix, delay running rewriteTargetListUD until after the planner has
expanded inheritance, so that it is applied separately to the (already
mapped) tlist for each child table. We can conveniently call it from
preprocess_targetlist. Refactor associated code slightly to avoid the
need to heap_open the target relation multiple times during
preprocess_targetlist. (The APIs remain a bit ugly, particularly around
the point of which steps scribble on parse->targetList and which don't.
But avoiding such scribbling would require a change in FDW callback APIs,
which is more pain than it's worth.)
Also fix ExecModifyTable to ensure that "tupleid" is reset to NULL when
we transition from rows providing a CTID to rows that don't. (That's
really an independent bug, but it manifests in much the same cases.)
Add a regression test checking one manifestation of this problem, which
was that row-level triggers on a foreign child table did not work right.
Back-patch to 9.5 where the problem was introduced.
Etsuro Fujita, reviewed by Ildus Kurbangaliev and Ashutosh Bapat
Discussion: https://postgr.es/m/20170514150525.0346ba72@postgrespro.ru
2017-11-27 23:53:56 +01:00
|
|
|
Although this function is called during planning, the
|
|
|
|
information provided is a bit different from that available to other
|
2013-03-10 19:14:53 +01:00
|
|
|
planning routines.
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>parsetree</literal> is the parse tree for the <command>UPDATE</command> or
|
|
|
|
<command>DELETE</command> command, while <literal>target_rte</literal> and
|
|
|
|
<literal>target_relation</literal> describe the target foreign table.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If the <function>AddForeignUpdateTargets</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, no extra target expressions are added.
|
|
|
|
(This will make it impossible to implement <command>DELETE</command>
|
|
|
|
operations, though <command>UPDATE</command> may still be feasible if the FDW
|
2013-03-10 19:14:53 +01:00
|
|
|
relies on an unchanging primary key to identify rows.)
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
List *
|
2017-09-06 17:38:28 +02:00
|
|
|
PlanForeignModify(PlannerInfo *root,
|
|
|
|
ModifyTable *plan,
|
|
|
|
Index resultRelation,
|
|
|
|
int subplan_index);
|
2013-03-10 19:14:53 +01:00
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
Perform any additional planning actions needed for an insert, update, or
|
|
|
|
delete on a foreign table. This function generates the FDW-private
|
2017-10-09 03:44:17 +02:00
|
|
|
information that will be attached to the <structname>ModifyTable</structname> plan
|
2013-03-10 19:14:53 +01:00
|
|
|
node that performs the update action. This private information must
|
2017-10-09 03:44:17 +02:00
|
|
|
have the form of a <literal>List</literal>, and will be delivered to
|
|
|
|
<function>BeginForeignModify</function> during the execution stage.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>root</literal> is the planner's global information about the query.
|
|
|
|
<literal>plan</literal> is the <structname>ModifyTable</structname> plan node, which is
|
|
|
|
complete except for the <structfield>fdwPrivLists</structfield> field.
|
|
|
|
<literal>resultRelation</literal> identifies the target foreign table by its
|
|
|
|
range table index. <literal>subplan_index</literal> identifies which target of
|
|
|
|
the <structname>ModifyTable</structname> plan node this is, counting from zero;
|
|
|
|
use this if you want to index into <literal>plan->plans</literal> or other
|
|
|
|
substructure of the <literal>plan</literal> node.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-11-23 15:39:47 +01:00
|
|
|
See <xref linkend="fdw-planning"/> for additional information.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If the <function>PlanForeignModify</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, no additional plan-time actions are taken, and the
|
|
|
|
<literal>fdw_private</literal> list delivered to
|
|
|
|
<function>BeginForeignModify</function> will be NIL.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
2017-09-06 17:38:28 +02:00
|
|
|
BeginForeignModify(ModifyTableState *mtstate,
|
|
|
|
ResultRelInfo *rinfo,
|
|
|
|
List *fdw_private,
|
|
|
|
int subplan_index,
|
|
|
|
int eflags);
|
2013-03-10 19:14:53 +01:00
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
Begin executing a foreign table modification operation. This routine is
|
|
|
|
called during executor startup. It should perform any initialization
|
|
|
|
needed prior to the actual table modifications. Subsequently,
|
2017-10-09 03:44:17 +02:00
|
|
|
<function>ExecForeignInsert</function>, <function>ExecForeignUpdate</function> or
|
|
|
|
<function>ExecForeignDelete</function> will be called for each tuple to be
|
2013-03-10 19:14:53 +01:00
|
|
|
inserted, updated, or deleted.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>mtstate</literal> is the overall state of the
|
|
|
|
<structname>ModifyTable</structname> plan node being executed; global data about
|
2013-03-10 19:14:53 +01:00
|
|
|
the plan and execution state is available via this structure.
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>rinfo</literal> is the <structname>ResultRelInfo</structname> struct describing
|
|
|
|
the target foreign table. (The <structfield>ri_FdwState</structfield> field of
|
|
|
|
<structname>ResultRelInfo</structname> is available for the FDW to store any
|
2013-03-10 19:14:53 +01:00
|
|
|
private state it needs for this operation.)
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>fdw_private</literal> contains the private data generated by
|
|
|
|
<function>PlanForeignModify</function>, if any.
|
|
|
|
<literal>subplan_index</literal> identifies which target of
|
|
|
|
the <structname>ModifyTable</structname> plan node this is.
|
|
|
|
<literal>eflags</literal> contains flag bits describing the executor's
|
2013-03-10 19:14:53 +01:00
|
|
|
operating mode for this plan node.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
Note that when <literal>(eflags & EXEC_FLAG_EXPLAIN_ONLY)</literal> is
|
2013-03-10 19:14:53 +01:00
|
|
|
true, this function should not perform any externally-visible actions;
|
|
|
|
it should only do the minimum required to make the node state valid
|
2017-10-09 03:44:17 +02:00
|
|
|
for <function>ExplainForeignModify</function> and <function>EndForeignModify</function>.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If the <function>BeginForeignModify</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, no action is taken during executor startup.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
TupleTableSlot *
|
2017-09-06 17:38:28 +02:00
|
|
|
ExecForeignInsert(EState *estate,
|
|
|
|
ResultRelInfo *rinfo,
|
|
|
|
TupleTableSlot *slot,
|
|
|
|
TupleTableSlot *planSlot);
|
2013-03-10 19:14:53 +01:00
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
Insert one tuple into the foreign table.
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>estate</literal> is global execution state for the query.
|
|
|
|
<literal>rinfo</literal> is the <structname>ResultRelInfo</structname> struct describing
|
2013-03-10 19:14:53 +01:00
|
|
|
the target foreign table.
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>slot</literal> contains the tuple to be inserted; it will match the
|
2014-07-17 04:20:15 +02:00
|
|
|
row-type definition of the foreign table.
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>planSlot</literal> contains the tuple that was generated by the
|
|
|
|
<structname>ModifyTable</structname> plan node's subplan; it differs from
|
|
|
|
<literal>slot</literal> in possibly containing additional <quote>junk</quote>
|
|
|
|
columns. (The <literal>planSlot</literal> is typically of little interest
|
|
|
|
for <command>INSERT</command> cases, but is provided for completeness.)
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The return value is either a slot containing the data that was actually
|
|
|
|
inserted (this might differ from the data supplied, for example as a
|
|
|
|
result of trigger actions), or NULL if no row was actually inserted
|
|
|
|
(again, typically as a result of triggers). The passed-in
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>slot</literal> can be re-used for this purpose.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
The data in the returned slot is used only if the <command>INSERT</command>
|
Fix WITH CHECK OPTION on views referencing postgres_fdw tables.
If a view references a foreign table, and the foreign table has a
BEFORE INSERT trigger, then it's possible for a tuple inserted or
updated through the view to be changed such that it violates the
view's WITH CHECK OPTION constraint.
Before this commit, postgres_fdw handled this case inconsistently. A
RETURNING clause on the INSERT or UPDATE statement targeting the view
would cause the finally-inserted tuple to be read back, and the WITH
CHECK OPTION violation would throw an error. But without a RETURNING
clause, postgres_fdw would not read the final tuple back, and WITH
CHECK OPTION would not throw an error for the violation (or may throw
an error when there is no real violation). AFTER ROW triggers on the
foreign table had a similar effect as a RETURNING clause on the INSERT
or UPDATE statement.
To fix, this commit retrieves the attributes needed to enforce the
WITH CHECK OPTION constraint along with the attributes needed for the
RETURNING clause (if any) from the remote side. Thus, the WITH CHECK
OPTION constraint is always evaluated against the final tuple after
any triggers on the remote side.
This fix may be considered inconsistent with CHECK constraints
declared on foreign tables, which are not enforced locally at all
(because the constraint is on a remote object). The discussion
concluded that this difference is reasonable, because the WITH CHECK
OPTION is a constraint on the local view (not any remote object);
therefore it only makes sense to enforce its WITH CHECK OPTION
constraint locally.
Author: Etsuro Fujita
Reviewed-by: Arthur Zakirov, Stephen Frost
Discussion: https://www.postgresql.org/message-id/7eb58fab-fd3b-781b-ac33-f7cfec96021f%40lab.ntt.co.jp
2018-07-08 09:14:51 +02:00
|
|
|
statement has a <literal>RETURNING</literal> clause or involves a view
|
|
|
|
<literal>WITH CHECK OPTION</literal>; or if the foreign table has
|
|
|
|
an <literal>AFTER ROW</literal> trigger. Triggers require all columns,
|
|
|
|
but the FDW could choose to optimize away returning some or all columns
|
|
|
|
depending on the contents of the <literal>RETURNING</literal> clause or
|
|
|
|
<literal>WITH CHECK OPTION</literal> constraints. Regardless, some slot
|
|
|
|
must be returned to indicate success, or the query's reported row count
|
|
|
|
will be wrong.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If the <function>ExecForeignInsert</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, attempts to insert into the foreign table will fail
|
2013-03-10 19:14:53 +01:00
|
|
|
with an error message.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
TupleTableSlot *
|
2017-09-06 17:38:28 +02:00
|
|
|
ExecForeignUpdate(EState *estate,
|
|
|
|
ResultRelInfo *rinfo,
|
|
|
|
TupleTableSlot *slot,
|
|
|
|
TupleTableSlot *planSlot);
|
2013-03-10 19:14:53 +01:00
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
Update one tuple in the foreign table.
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>estate</literal> is global execution state for the query.
|
|
|
|
<literal>rinfo</literal> is the <structname>ResultRelInfo</structname> struct describing
|
2013-03-10 19:14:53 +01:00
|
|
|
the target foreign table.
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>slot</literal> contains the new data for the tuple; it will match the
|
2014-07-17 04:20:15 +02:00
|
|
|
row-type definition of the foreign table.
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>planSlot</literal> contains the tuple that was generated by the
|
|
|
|
<structname>ModifyTable</structname> plan node's subplan; it differs from
|
|
|
|
<literal>slot</literal> in possibly containing additional <quote>junk</quote>
|
2013-03-10 19:14:53 +01:00
|
|
|
columns. In particular, any junk columns that were requested by
|
2017-10-09 03:44:17 +02:00
|
|
|
<function>AddForeignUpdateTargets</function> will be available from this slot.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The return value is either a slot containing the row as it was actually
|
|
|
|
updated (this might differ from the data supplied, for example as a
|
|
|
|
result of trigger actions), or NULL if no row was actually updated
|
|
|
|
(again, typically as a result of triggers). The passed-in
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>slot</literal> can be re-used for this purpose.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
The data in the returned slot is used only if the <command>UPDATE</command>
|
Fix WITH CHECK OPTION on views referencing postgres_fdw tables.
If a view references a foreign table, and the foreign table has a
BEFORE INSERT trigger, then it's possible for a tuple inserted or
updated through the view to be changed such that it violates the
view's WITH CHECK OPTION constraint.
Before this commit, postgres_fdw handled this case inconsistently. A
RETURNING clause on the INSERT or UPDATE statement targeting the view
would cause the finally-inserted tuple to be read back, and the WITH
CHECK OPTION violation would throw an error. But without a RETURNING
clause, postgres_fdw would not read the final tuple back, and WITH
CHECK OPTION would not throw an error for the violation (or may throw
an error when there is no real violation). AFTER ROW triggers on the
foreign table had a similar effect as a RETURNING clause on the INSERT
or UPDATE statement.
To fix, this commit retrieves the attributes needed to enforce the
WITH CHECK OPTION constraint along with the attributes needed for the
RETURNING clause (if any) from the remote side. Thus, the WITH CHECK
OPTION constraint is always evaluated against the final tuple after
any triggers on the remote side.
This fix may be considered inconsistent with CHECK constraints
declared on foreign tables, which are not enforced locally at all
(because the constraint is on a remote object). The discussion
concluded that this difference is reasonable, because the WITH CHECK
OPTION is a constraint on the local view (not any remote object);
therefore it only makes sense to enforce its WITH CHECK OPTION
constraint locally.
Author: Etsuro Fujita
Reviewed-by: Arthur Zakirov, Stephen Frost
Discussion: https://www.postgresql.org/message-id/7eb58fab-fd3b-781b-ac33-f7cfec96021f%40lab.ntt.co.jp
2018-07-08 09:14:51 +02:00
|
|
|
statement has a <literal>RETURNING</literal> clause or involves a view
|
|
|
|
<literal>WITH CHECK OPTION</literal>; or if the foreign table has
|
|
|
|
an <literal>AFTER ROW</literal> trigger. Triggers require all columns,
|
|
|
|
but the FDW could choose to optimize away returning some or all columns
|
|
|
|
depending on the contents of the <literal>RETURNING</literal> clause or
|
|
|
|
<literal>WITH CHECK OPTION</literal> constraints. Regardless, some slot
|
|
|
|
must be returned to indicate success, or the query's reported row count
|
|
|
|
will be wrong.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If the <function>ExecForeignUpdate</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, attempts to update the foreign table will fail
|
2013-03-10 19:14:53 +01:00
|
|
|
with an error message.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
TupleTableSlot *
|
2017-09-06 17:38:28 +02:00
|
|
|
ExecForeignDelete(EState *estate,
|
|
|
|
ResultRelInfo *rinfo,
|
|
|
|
TupleTableSlot *slot,
|
|
|
|
TupleTableSlot *planSlot);
|
2013-03-10 19:14:53 +01:00
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
Delete one tuple from the foreign table.
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>estate</literal> is global execution state for the query.
|
|
|
|
<literal>rinfo</literal> is the <structname>ResultRelInfo</structname> struct describing
|
2013-03-10 19:14:53 +01:00
|
|
|
the target foreign table.
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>slot</literal> contains nothing useful upon call, but can be used to
|
2013-03-10 19:14:53 +01:00
|
|
|
hold the returned tuple.
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>planSlot</literal> contains the tuple that was generated by the
|
|
|
|
<structname>ModifyTable</structname> plan node's subplan; in particular, it will
|
2013-03-10 19:14:53 +01:00
|
|
|
carry any junk columns that were requested by
|
2017-10-09 03:44:17 +02:00
|
|
|
<function>AddForeignUpdateTargets</function>. The junk column(s) must be used
|
2013-03-10 19:14:53 +01:00
|
|
|
to identify the tuple to be deleted.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The return value is either a slot containing the row that was deleted,
|
|
|
|
or NULL if no row was deleted (typically as a result of triggers). The
|
2017-10-09 03:44:17 +02:00
|
|
|
passed-in <literal>slot</literal> can be used to hold the tuple to be returned.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
The data in the returned slot is used only if the <command>DELETE</command>
|
|
|
|
query has a <literal>RETURNING</literal> clause or the foreign table has
|
|
|
|
an <literal>AFTER ROW</literal> trigger. Triggers require all columns, but the
|
2014-03-23 07:16:34 +01:00
|
|
|
FDW could choose to optimize away returning some or all columns depending
|
2017-10-09 03:44:17 +02:00
|
|
|
on the contents of the <literal>RETURNING</literal> clause. Regardless, some
|
2014-03-23 07:16:34 +01:00
|
|
|
slot must be returned to indicate success, or the query's reported row
|
|
|
|
count will be wrong.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If the <function>ExecForeignDelete</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, attempts to delete from the foreign table will fail
|
2013-03-10 19:14:53 +01:00
|
|
|
with an error message.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
2017-09-06 17:38:28 +02:00
|
|
|
EndForeignModify(EState *estate,
|
|
|
|
ResultRelInfo *rinfo);
|
2013-03-10 19:14:53 +01:00
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
End the table update and release resources. It is normally not important
|
|
|
|
to release palloc'd memory, but for example open files and connections
|
|
|
|
to remote servers should be cleaned up.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If the <function>EndForeignModify</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, no action is taken during executor shutdown.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
2018-04-07 01:16:11 +02:00
|
|
|
<para>
|
|
|
|
Tuples inserted into a partitioned table by <command>INSERT</command> or
|
|
|
|
<command>COPY FROM</command> are routed to partitions. If an FDW
|
|
|
|
supports routable foreign-table partitions, it should also provide the
|
|
|
|
following callback functions. These functions are also called when
|
|
|
|
<command>COPY FROM</command> is executed on a foreign table.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
|
|
|
BeginForeignInsert(ModifyTableState *mtstate,
|
|
|
|
ResultRelInfo *rinfo);
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
Begin executing an insert operation on a foreign table. This routine is
|
|
|
|
called right before the first tuple is inserted into the foreign table
|
|
|
|
in both cases when it is the partition chosen for tuple routing and the
|
|
|
|
target specified in a <command>COPY FROM</command> command. It should
|
|
|
|
perform any initialization needed prior to the actual insertion.
|
|
|
|
Subsequently, <function>ExecForeignInsert</function> will be called for
|
|
|
|
each tuple to be inserted into the foreign table.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<literal>mtstate</literal> is the overall state of the
|
|
|
|
<structname>ModifyTable</structname> plan node being executed; global data about
|
|
|
|
the plan and execution state is available via this structure.
|
|
|
|
<literal>rinfo</literal> is the <structname>ResultRelInfo</structname> struct describing
|
|
|
|
the target foreign table. (The <structfield>ri_FdwState</structfield> field of
|
|
|
|
<structname>ResultRelInfo</structname> is available for the FDW to store any
|
|
|
|
private state it needs for this operation.)
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
When this is called by a <command>COPY FROM</command> command, the
|
|
|
|
plan-related global data in <literal>mtstate</literal> is not provided
|
|
|
|
and the <literal>planSlot</literal> parameter of
|
|
|
|
<function>ExecForeignInsert</function> subsequently called for each
|
|
|
|
inserted tuple is <literal>NULL</literal>, whether the foreign table is
|
|
|
|
the partition chosen for tuple routing or the target specified in the
|
|
|
|
command.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
If the <function>BeginForeignInsert</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, no action is taken for the initialization.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
|
|
|
EndForeignInsert(EState *estate,
|
|
|
|
ResultRelInfo *rinfo);
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
End the insert operation and release resources. It is normally not important
|
|
|
|
to release palloc'd memory, but for example open files and connections
|
|
|
|
to remote servers should be cleaned up.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
If the <function>EndForeignInsert</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, no action is taken for the termination.
|
|
|
|
</para>
|
|
|
|
|
2013-06-12 23:52:54 +02:00
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
int
|
2017-09-06 17:38:28 +02:00
|
|
|
IsForeignRelUpdatable(Relation rel);
|
2013-06-12 23:52:54 +02:00
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
Report which update operations the specified foreign table supports.
|
2014-07-17 04:20:15 +02:00
|
|
|
The return value should be a bit mask of rule event numbers indicating
|
2013-06-12 23:52:54 +02:00
|
|
|
which operations are supported by the foreign table, using the
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>CmdType</literal> enumeration; that is,
|
|
|
|
<literal>(1 << CMD_UPDATE) = 4</literal> for <command>UPDATE</command>,
|
|
|
|
<literal>(1 << CMD_INSERT) = 8</literal> for <command>INSERT</command>, and
|
|
|
|
<literal>(1 << CMD_DELETE) = 16</literal> for <command>DELETE</command>.
|
2013-06-12 23:52:54 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If the <function>IsForeignRelUpdatable</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, foreign tables are assumed to be insertable, updatable,
|
|
|
|
or deletable if the FDW provides <function>ExecForeignInsert</function>,
|
|
|
|
<function>ExecForeignUpdate</function>, or <function>ExecForeignDelete</function>
|
2013-06-12 23:52:54 +02:00
|
|
|
respectively. This function is only needed if the FDW supports some
|
|
|
|
tables that are updatable and some that are not. (Even then, it's
|
|
|
|
permissible to throw an error in the execution routine instead of
|
|
|
|
checking in this function. However, this function is used to determine
|
2017-10-09 03:44:17 +02:00
|
|
|
updatability for display in the <literal>information_schema</literal> views.)
|
2013-06-12 23:52:54 +02:00
|
|
|
</para>
|
|
|
|
|
2016-03-18 18:48:58 +01:00
|
|
|
<para>
|
|
|
|
Some inserts, updates, and deletes to foreign tables can be optimized
|
|
|
|
by implementing an alternative set of interfaces. The ordinary
|
|
|
|
interfaces for inserts, updates, and deletes fetch rows from the remote
|
|
|
|
server and then modify those rows one at a time. In some cases, this
|
|
|
|
row-by-row approach is necessary, but it can be inefficient. If it is
|
|
|
|
possible for the foreign server to determine which rows should be
|
|
|
|
modified without actually retrieving them, and if there are no local
|
|
|
|
triggers which would affect the operation, then it is possible to
|
|
|
|
arrange things so that the entire operation is performed on the remote
|
|
|
|
server. The interfaces described below make this possible.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
bool
|
2017-09-06 17:38:28 +02:00
|
|
|
PlanDirectModify(PlannerInfo *root,
|
|
|
|
ModifyTable *plan,
|
|
|
|
Index resultRelation,
|
|
|
|
int subplan_index);
|
2016-03-18 18:48:58 +01:00
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
Decide whether it is safe to execute a direct modification
|
2017-10-09 03:44:17 +02:00
|
|
|
on the remote server. If so, return <literal>true</literal> after performing
|
|
|
|
planning actions needed for that. Otherwise, return <literal>false</literal>.
|
2016-03-18 18:48:58 +01:00
|
|
|
This optional function is called during query planning.
|
2017-10-09 03:44:17 +02:00
|
|
|
If this function succeeds, <function>BeginDirectModify</function>,
|
|
|
|
<function>IterateDirectModify</function> and <function>EndDirectModify</function> will
|
2016-03-18 18:48:58 +01:00
|
|
|
be called at the execution stage, instead. Otherwise, the table
|
|
|
|
modification will be executed using the table-updating functions
|
|
|
|
described above.
|
2017-10-09 03:44:17 +02:00
|
|
|
The parameters are the same as for <function>PlanForeignModify</function>.
|
2016-03-18 18:48:58 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
To execute the direct modification on the remote server, this function
|
2017-10-09 03:44:17 +02:00
|
|
|
must rewrite the target subplan with a <structname>ForeignScan</structname> plan
|
2016-03-18 18:48:58 +01:00
|
|
|
node that executes the direct modification on the remote server. The
|
2017-10-09 03:44:17 +02:00
|
|
|
<structfield>operation</structfield> field of the <structname>ForeignScan</structname> must
|
|
|
|
be set to the <literal>CmdType</literal> enumeration appropriately; that is,
|
|
|
|
<literal>CMD_UPDATE</literal> for <command>UPDATE</command>,
|
|
|
|
<literal>CMD_INSERT</literal> for <command>INSERT</command>, and
|
|
|
|
<literal>CMD_DELETE</literal> for <command>DELETE</command>.
|
2016-03-18 18:48:58 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-11-23 15:39:47 +01:00
|
|
|
See <xref linkend="fdw-planning"/> for additional information.
|
2016-03-18 18:48:58 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If the <function>PlanDirectModify</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, no attempts to execute a direct modification on the
|
2016-03-18 18:48:58 +01:00
|
|
|
remote server are taken.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
2017-09-06 17:38:28 +02:00
|
|
|
BeginDirectModify(ForeignScanState *node,
|
|
|
|
int eflags);
|
2016-03-18 18:48:58 +01:00
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
Prepare to execute a direct modification on the remote server.
|
|
|
|
This is called during executor startup. It should perform any
|
|
|
|
initialization needed prior to the direct modification (that should be
|
2017-10-09 03:44:17 +02:00
|
|
|
done upon the first call to <function>IterateDirectModify</function>).
|
|
|
|
The <structname>ForeignScanState</structname> node has already been created, but
|
|
|
|
its <structfield>fdw_state</structfield> field is still NULL. Information about
|
2016-03-18 18:48:58 +01:00
|
|
|
the table to modify is accessible through the
|
2017-10-09 03:44:17 +02:00
|
|
|
<structname>ForeignScanState</structname> node (in particular, from the underlying
|
|
|
|
<structname>ForeignScan</structname> plan node, which contains any FDW-private
|
|
|
|
information provided by <function>PlanDirectModify</function>).
|
|
|
|
<literal>eflags</literal> contains flag bits describing the executor's
|
2016-03-18 18:48:58 +01:00
|
|
|
operating mode for this plan node.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
Note that when <literal>(eflags & EXEC_FLAG_EXPLAIN_ONLY)</literal> is
|
2016-03-18 18:48:58 +01:00
|
|
|
true, this function should not perform any externally-visible actions;
|
|
|
|
it should only do the minimum required to make the node state valid
|
2017-10-09 03:44:17 +02:00
|
|
|
for <function>ExplainDirectModify</function> and <function>EndDirectModify</function>.
|
2016-03-18 18:48:58 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If the <function>BeginDirectModify</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, no attempts to execute a direct modification on the
|
2016-03-18 18:48:58 +01:00
|
|
|
remote server are taken.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
TupleTableSlot *
|
2017-09-06 17:38:28 +02:00
|
|
|
IterateDirectModify(ForeignScanState *node);
|
2016-03-18 18:48:58 +01:00
|
|
|
</programlisting>
|
|
|
|
|
2017-10-09 03:44:17 +02:00
|
|
|
When the <command>INSERT</command>, <command>UPDATE</command> or <command>DELETE</command>
|
|
|
|
query doesn't have a <literal>RETURNING</literal> clause, just return NULL
|
2016-03-18 18:48:58 +01:00
|
|
|
after a direct modification on the remote server.
|
|
|
|
When the query has the clause, fetch one result containing the data
|
2017-10-09 03:44:17 +02:00
|
|
|
needed for the <literal>RETURNING</literal> calculation, returning it in a
|
|
|
|
tuple table slot (the node's <structfield>ScanTupleSlot</structfield> should be
|
2016-03-18 18:48:58 +01:00
|
|
|
used for this purpose). The data that was actually inserted, updated
|
|
|
|
or deleted must be stored in the
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>es_result_relation_info->ri_projectReturning->pi_exprContext->ecxt_scantuple</literal>
|
|
|
|
of the node's <structname>EState</structname>.
|
2016-03-18 18:48:58 +01:00
|
|
|
Return NULL if no more rows are available.
|
|
|
|
Note that this is called in a short-lived memory context that will be
|
|
|
|
reset between invocations. Create a memory context in
|
2017-10-09 03:44:17 +02:00
|
|
|
<function>BeginDirectModify</function> if you need longer-lived storage, or use
|
|
|
|
the <structfield>es_query_cxt</structfield> of the node's <structname>EState</structname>.
|
2016-03-18 18:48:58 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
The rows returned must match the <structfield>fdw_scan_tlist</structfield> target
|
2016-03-18 18:48:58 +01:00
|
|
|
list if one was supplied, otherwise they must match the row type of the
|
|
|
|
foreign table being updated. If you choose to optimize away fetching
|
2017-10-09 03:44:17 +02:00
|
|
|
columns that are not needed for the <literal>RETURNING</literal> calculation,
|
2016-03-18 18:48:58 +01:00
|
|
|
you should insert nulls in those column positions, or else generate a
|
2017-10-09 03:44:17 +02:00
|
|
|
<structfield>fdw_scan_tlist</structfield> list with those columns omitted.
|
2016-03-18 18:48:58 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2016-05-31 19:56:25 +02:00
|
|
|
Whether the query has the clause or not, the query's reported row count
|
2016-03-18 18:48:58 +01:00
|
|
|
must be incremented by the FDW itself. When the query doesn't have the
|
|
|
|
clause, the FDW must also increment the row count for the
|
2017-10-09 03:44:17 +02:00
|
|
|
<structname>ForeignScanState</structname> node in the <command>EXPLAIN ANALYZE</command>
|
2016-03-18 18:48:58 +01:00
|
|
|
case.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If the <function>IterateDirectModify</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, no attempts to execute a direct modification on the
|
2016-03-18 18:48:58 +01:00
|
|
|
remote server are taken.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
2017-09-06 17:38:28 +02:00
|
|
|
EndDirectModify(ForeignScanState *node);
|
2016-03-18 18:48:58 +01:00
|
|
|
</programlisting>
|
|
|
|
|
2016-07-12 19:30:48 +02:00
|
|
|
Clean up following a direct modification on the remote server. It is
|
2016-03-18 18:48:58 +01:00
|
|
|
normally not important to release palloc'd memory, but for example open
|
|
|
|
files and connections to the remote server should be cleaned up.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If the <function>EndDirectModify</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, no attempts to execute a direct modification on the
|
2016-03-18 18:48:58 +01:00
|
|
|
remote server are taken.
|
|
|
|
</para>
|
|
|
|
|
2013-03-10 19:14:53 +01:00
|
|
|
</sect2>
|
|
|
|
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
<sect2 id="fdw-callbacks-row-locking">
|
|
|
|
<title>FDW Routines For Row Locking</title>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If an FDW wishes to support <firstterm>late row locking</firstterm> (as described
|
2017-11-23 15:39:47 +01:00
|
|
|
in <xref linkend="fdw-row-locking"/>), it must provide the following
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
callback functions:
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
RowMarkType
|
2017-09-06 17:38:28 +02:00
|
|
|
GetForeignRowMarkType(RangeTblEntry *rte,
|
|
|
|
LockClauseStrength strength);
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
Report which row-marking option to use for a foreign table.
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>rte</literal> is the <structname>RangeTblEntry</structname> node for the table
|
|
|
|
and <literal>strength</literal> describes the lock strength requested by the
|
|
|
|
relevant <literal>FOR UPDATE/SHARE</literal> clause, if any. The result must be
|
|
|
|
a member of the <literal>RowMarkType</literal> enum type.
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
This function is called during query planning for each foreign table that
|
2017-10-09 03:44:17 +02:00
|
|
|
appears in an <command>UPDATE</command>, <command>DELETE</command>, or <command>SELECT
|
|
|
|
FOR UPDATE/SHARE</command> query and is not the target of <command>UPDATE</command>
|
|
|
|
or <command>DELETE</command>.
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If the <function>GetForeignRowMarkType</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, the <literal>ROW_MARK_COPY</literal> option is always used.
|
|
|
|
(This implies that <function>RefetchForeignRow</function> will never be called,
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
so it need not be provided either.)
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-11-23 15:39:47 +01:00
|
|
|
See <xref linkend="fdw-row-locking"/> for more information.
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
Store tuples for EvalPlanQual in slots, rather than as HeapTuples.
For the upcoming pluggable table access methods it's quite
inconvenient to store tuples as HeapTuples, as that'd require
converting tuples from a their native format into HeapTuples. Instead
use slots to manage epq tuples.
To fit into that scheme, change the foreign data wrapper callback
RefetchForeignRow, to store the tuple in a slot. Insist on using the
caller provided slot, so it conveniently can be stored in the
corresponding EPQ slot. As there is no in core user of
RefetchForeignRow, that change was done blindly, but we plan to test
that soon.
To avoid duplicating that work for row locks, move row locks to just
directly use the EPQ slots - it previously temporarily stored tuples
in LockRowsState.lr_curtuples, but that doesn't seem beneficial, given
we'd possibly end up with a significant number of additional slots.
The behaviour of es_epqTupleSet[rti -1] is now checked by
es_epqTupleSlot[rti -1] != NULL, as that is distinguishable from a
slot containing an empty tuple.
Author: Andres Freund, Haribabu Kommi, Ashutosh Bapat
Discussion: https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
2019-03-01 19:37:57 +01:00
|
|
|
void
|
2017-09-06 17:38:28 +02:00
|
|
|
RefetchForeignRow(EState *estate,
|
|
|
|
ExecRowMark *erm,
|
|
|
|
Datum rowid,
|
Store tuples for EvalPlanQual in slots, rather than as HeapTuples.
For the upcoming pluggable table access methods it's quite
inconvenient to store tuples as HeapTuples, as that'd require
converting tuples from a their native format into HeapTuples. Instead
use slots to manage epq tuples.
To fit into that scheme, change the foreign data wrapper callback
RefetchForeignRow, to store the tuple in a slot. Insist on using the
caller provided slot, so it conveniently can be stored in the
corresponding EPQ slot. As there is no in core user of
RefetchForeignRow, that change was done blindly, but we plan to test
that soon.
To avoid duplicating that work for row locks, move row locks to just
directly use the EPQ slots - it previously temporarily stored tuples
in LockRowsState.lr_curtuples, but that doesn't seem beneficial, given
we'd possibly end up with a significant number of additional slots.
The behaviour of es_epqTupleSet[rti -1] is now checked by
es_epqTupleSlot[rti -1] != NULL, as that is distinguishable from a
slot containing an empty tuple.
Author: Andres Freund, Haribabu Kommi, Ashutosh Bapat
Discussion: https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
2019-03-01 19:37:57 +01:00
|
|
|
TupleTableSlot *slot,
|
2017-09-06 17:38:28 +02:00
|
|
|
bool *updated);
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
</programlisting>
|
|
|
|
|
Store tuples for EvalPlanQual in slots, rather than as HeapTuples.
For the upcoming pluggable table access methods it's quite
inconvenient to store tuples as HeapTuples, as that'd require
converting tuples from a their native format into HeapTuples. Instead
use slots to manage epq tuples.
To fit into that scheme, change the foreign data wrapper callback
RefetchForeignRow, to store the tuple in a slot. Insist on using the
caller provided slot, so it conveniently can be stored in the
corresponding EPQ slot. As there is no in core user of
RefetchForeignRow, that change was done blindly, but we plan to test
that soon.
To avoid duplicating that work for row locks, move row locks to just
directly use the EPQ slots - it previously temporarily stored tuples
in LockRowsState.lr_curtuples, but that doesn't seem beneficial, given
we'd possibly end up with a significant number of additional slots.
The behaviour of es_epqTupleSet[rti -1] is now checked by
es_epqTupleSlot[rti -1] != NULL, as that is distinguishable from a
slot containing an empty tuple.
Author: Andres Freund, Haribabu Kommi, Ashutosh Bapat
Discussion: https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
2019-03-01 19:37:57 +01:00
|
|
|
Re-fetch one tuple slot from the foreign table, after locking it if required.
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>estate</literal> is global execution state for the query.
|
|
|
|
<literal>erm</literal> is the <structname>ExecRowMark</structname> struct describing
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
the target foreign table and the row lock type (if any) to acquire.
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>rowid</literal> identifies the tuple to be fetched.
|
Store tuples for EvalPlanQual in slots, rather than as HeapTuples.
For the upcoming pluggable table access methods it's quite
inconvenient to store tuples as HeapTuples, as that'd require
converting tuples from a their native format into HeapTuples. Instead
use slots to manage epq tuples.
To fit into that scheme, change the foreign data wrapper callback
RefetchForeignRow, to store the tuple in a slot. Insist on using the
caller provided slot, so it conveniently can be stored in the
corresponding EPQ slot. As there is no in core user of
RefetchForeignRow, that change was done blindly, but we plan to test
that soon.
To avoid duplicating that work for row locks, move row locks to just
directly use the EPQ slots - it previously temporarily stored tuples
in LockRowsState.lr_curtuples, but that doesn't seem beneficial, given
we'd possibly end up with a significant number of additional slots.
The behaviour of es_epqTupleSet[rti -1] is now checked by
es_epqTupleSlot[rti -1] != NULL, as that is distinguishable from a
slot containing an empty tuple.
Author: Andres Freund, Haribabu Kommi, Ashutosh Bapat
Discussion: https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
2019-03-01 19:37:57 +01:00
|
|
|
<literal>slot</literal> contains nothing useful upon call, but can be used to
|
|
|
|
hold the returned tuple. <literal>updated</literal> is an output parameter.
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
Store tuples for EvalPlanQual in slots, rather than as HeapTuples.
For the upcoming pluggable table access methods it's quite
inconvenient to store tuples as HeapTuples, as that'd require
converting tuples from a their native format into HeapTuples. Instead
use slots to manage epq tuples.
To fit into that scheme, change the foreign data wrapper callback
RefetchForeignRow, to store the tuple in a slot. Insist on using the
caller provided slot, so it conveniently can be stored in the
corresponding EPQ slot. As there is no in core user of
RefetchForeignRow, that change was done blindly, but we plan to test
that soon.
To avoid duplicating that work for row locks, move row locks to just
directly use the EPQ slots - it previously temporarily stored tuples
in LockRowsState.lr_curtuples, but that doesn't seem beneficial, given
we'd possibly end up with a significant number of additional slots.
The behaviour of es_epqTupleSet[rti -1] is now checked by
es_epqTupleSlot[rti -1] != NULL, as that is distinguishable from a
slot containing an empty tuple.
Author: Andres Freund, Haribabu Kommi, Ashutosh Bapat
Discussion: https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
2019-03-01 19:37:57 +01:00
|
|
|
This function should store the tuple into the provided, or clear it if if
|
|
|
|
the row lock couldn't be obtained. The row lock type to acquire is
|
|
|
|
defined by <literal>erm->markType</literal>, which is the value
|
|
|
|
previously returned by <function>GetForeignRowMarkType</function>.
|
|
|
|
(<literal>ROW_MARK_REFERENCE</literal> means to just re-fetch the tuple
|
|
|
|
without acquiring any lock, and <literal>ROW_MARK_COPY</literal> will
|
|
|
|
never be seen by this routine.)
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
In addition, <literal>*updated</literal> should be set to <literal>true</literal>
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
if what was fetched was an updated version of the tuple rather than
|
|
|
|
the same version previously obtained. (If the FDW cannot be sure about
|
2017-10-09 03:44:17 +02:00
|
|
|
this, always returning <literal>true</literal> is recommended.)
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Note that by default, failure to acquire a row lock should result in
|
Store tuples for EvalPlanQual in slots, rather than as HeapTuples.
For the upcoming pluggable table access methods it's quite
inconvenient to store tuples as HeapTuples, as that'd require
converting tuples from a their native format into HeapTuples. Instead
use slots to manage epq tuples.
To fit into that scheme, change the foreign data wrapper callback
RefetchForeignRow, to store the tuple in a slot. Insist on using the
caller provided slot, so it conveniently can be stored in the
corresponding EPQ slot. As there is no in core user of
RefetchForeignRow, that change was done blindly, but we plan to test
that soon.
To avoid duplicating that work for row locks, move row locks to just
directly use the EPQ slots - it previously temporarily stored tuples
in LockRowsState.lr_curtuples, but that doesn't seem beneficial, given
we'd possibly end up with a significant number of additional slots.
The behaviour of es_epqTupleSet[rti -1] is now checked by
es_epqTupleSlot[rti -1] != NULL, as that is distinguishable from a
slot containing an empty tuple.
Author: Andres Freund, Haribabu Kommi, Ashutosh Bapat
Discussion: https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
2019-03-01 19:37:57 +01:00
|
|
|
raising an error; returning with an empty slot is only appropriate if
|
2017-10-09 03:44:17 +02:00
|
|
|
the <literal>SKIP LOCKED</literal> option is specified
|
|
|
|
by <literal>erm->waitPolicy</literal>.
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
The <literal>rowid</literal> is the <structfield>ctid</structfield> value previously read
|
|
|
|
for the row to be re-fetched. Although the <literal>rowid</literal> value is
|
|
|
|
passed as a <type>Datum</type>, it can currently only be a <type>tid</type>. The
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
function API is chosen in hopes that it may be possible to allow other
|
2015-09-11 03:22:21 +02:00
|
|
|
data types for row IDs in future.
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If the <function>RefetchForeignRow</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, attempts to re-fetch rows will fail
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
with an error message.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-11-23 15:39:47 +01:00
|
|
|
See <xref linkend="fdw-row-locking"/> for more information.
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
</para>
|
|
|
|
|
Allow foreign and custom joins to handle EvalPlanQual rechecks.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 provided basic
infrastructure for allowing a foreign data wrapper or custom scan
provider to replace a join of one or more tables with a scan.
However, this infrastructure failed to take into account the need
for possible EvalPlanQual rechecks, and ExecScanFetch would fail
an assertion (or just overwrite memory) if such a check was attempted
for a plan containing a pushed-down join. To fix, adjust the EPQ
machinery to skip some processing steps when scanrelid == 0, making
those the responsibility of scan's recheck method, which also has
the responsibility in this case of correctly populating the relevant
slot.
To allow foreign scans to gain control in the right place to make
use of this new facility, add a new, optional RecheckForeignScan
method. Also, allow a foreign scan to have a child plan, which can
be used to correctly populate the slot (or perhaps for something
else, but this is the only use currently envisioned).
KaiGai Kohei, reviewed by Robert Haas, Etsuro Fujita, and Kyotaro
Horiguchi.
2015-12-08 18:31:03 +01:00
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
bool
|
2017-09-06 17:38:28 +02:00
|
|
|
RecheckForeignScan(ForeignScanState *node,
|
|
|
|
TupleTableSlot *slot);
|
Allow foreign and custom joins to handle EvalPlanQual rechecks.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 provided basic
infrastructure for allowing a foreign data wrapper or custom scan
provider to replace a join of one or more tables with a scan.
However, this infrastructure failed to take into account the need
for possible EvalPlanQual rechecks, and ExecScanFetch would fail
an assertion (or just overwrite memory) if such a check was attempted
for a plan containing a pushed-down join. To fix, adjust the EPQ
machinery to skip some processing steps when scanrelid == 0, making
those the responsibility of scan's recheck method, which also has
the responsibility in this case of correctly populating the relevant
slot.
To allow foreign scans to gain control in the right place to make
use of this new facility, add a new, optional RecheckForeignScan
method. Also, allow a foreign scan to have a child plan, which can
be used to correctly populate the slot (or perhaps for something
else, but this is the only use currently envisioned).
KaiGai Kohei, reviewed by Robert Haas, Etsuro Fujita, and Kyotaro
Horiguchi.
2015-12-08 18:31:03 +01:00
|
|
|
</programlisting>
|
|
|
|
Recheck that a previously-returned tuple still matches the relevant
|
|
|
|
scan and join qualifiers, and possibly provide a modified version of
|
|
|
|
the tuple. For foreign data wrappers which do not perform join pushdown,
|
2017-10-09 03:44:17 +02:00
|
|
|
it will typically be more convenient to set this to <literal>NULL</literal> and
|
Allow foreign and custom joins to handle EvalPlanQual rechecks.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 provided basic
infrastructure for allowing a foreign data wrapper or custom scan
provider to replace a join of one or more tables with a scan.
However, this infrastructure failed to take into account the need
for possible EvalPlanQual rechecks, and ExecScanFetch would fail
an assertion (or just overwrite memory) if such a check was attempted
for a plan containing a pushed-down join. To fix, adjust the EPQ
machinery to skip some processing steps when scanrelid == 0, making
those the responsibility of scan's recheck method, which also has
the responsibility in this case of correctly populating the relevant
slot.
To allow foreign scans to gain control in the right place to make
use of this new facility, add a new, optional RecheckForeignScan
method. Also, allow a foreign scan to have a child plan, which can
be used to correctly populate the slot (or perhaps for something
else, but this is the only use currently envisioned).
KaiGai Kohei, reviewed by Robert Haas, Etsuro Fujita, and Kyotaro
Horiguchi.
2015-12-08 18:31:03 +01:00
|
|
|
instead set <structfield>fdw_recheck_quals</structfield> appropriately.
|
|
|
|
When outer joins are pushed down, however, it isn't sufficient to
|
|
|
|
reapply the checks relevant to all the base tables to the result tuple,
|
|
|
|
even if all needed attributes are present, because failure to match some
|
|
|
|
qualifier might result in some attributes going to NULL, rather than in
|
2017-10-09 03:44:17 +02:00
|
|
|
no tuple being returned. <literal>RecheckForeignScan</literal> can recheck
|
Allow foreign and custom joins to handle EvalPlanQual rechecks.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 provided basic
infrastructure for allowing a foreign data wrapper or custom scan
provider to replace a join of one or more tables with a scan.
However, this infrastructure failed to take into account the need
for possible EvalPlanQual rechecks, and ExecScanFetch would fail
an assertion (or just overwrite memory) if such a check was attempted
for a plan containing a pushed-down join. To fix, adjust the EPQ
machinery to skip some processing steps when scanrelid == 0, making
those the responsibility of scan's recheck method, which also has
the responsibility in this case of correctly populating the relevant
slot.
To allow foreign scans to gain control in the right place to make
use of this new facility, add a new, optional RecheckForeignScan
method. Also, allow a foreign scan to have a child plan, which can
be used to correctly populate the slot (or perhaps for something
else, but this is the only use currently envisioned).
KaiGai Kohei, reviewed by Robert Haas, Etsuro Fujita, and Kyotaro
Horiguchi.
2015-12-08 18:31:03 +01:00
|
|
|
qualifiers and return true if they are still satisfied and false
|
|
|
|
otherwise, but it can also store a replacement tuple into the supplied
|
|
|
|
slot.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
To implement join pushdown, a foreign data wrapper will typically
|
|
|
|
construct an alternative local join plan which is used only for
|
|
|
|
rechecks; this will become the outer subplan of the
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>ForeignScan</literal>. When a recheck is required, this subplan
|
Allow foreign and custom joins to handle EvalPlanQual rechecks.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 provided basic
infrastructure for allowing a foreign data wrapper or custom scan
provider to replace a join of one or more tables with a scan.
However, this infrastructure failed to take into account the need
for possible EvalPlanQual rechecks, and ExecScanFetch would fail
an assertion (or just overwrite memory) if such a check was attempted
for a plan containing a pushed-down join. To fix, adjust the EPQ
machinery to skip some processing steps when scanrelid == 0, making
those the responsibility of scan's recheck method, which also has
the responsibility in this case of correctly populating the relevant
slot.
To allow foreign scans to gain control in the right place to make
use of this new facility, add a new, optional RecheckForeignScan
method. Also, allow a foreign scan to have a child plan, which can
be used to correctly populate the slot (or perhaps for something
else, but this is the only use currently envisioned).
KaiGai Kohei, reviewed by Robert Haas, Etsuro Fujita, and Kyotaro
Horiguchi.
2015-12-08 18:31:03 +01:00
|
|
|
can be executed and the resulting tuple can be stored in the slot.
|
|
|
|
This plan need not be efficient since no base table will return more
|
2015-12-10 17:13:24 +01:00
|
|
|
than one row; for example, it may implement all joins as nested loops.
|
2017-10-09 03:44:17 +02:00
|
|
|
The function <literal>GetExistingLocalJoinPath</literal> may be used to search
|
2016-03-04 17:41:23 +01:00
|
|
|
existing paths for a suitable local join path, which can be used as the
|
2017-10-09 03:44:17 +02:00
|
|
|
alternative local join plan. <literal>GetExistingLocalJoinPath</literal>
|
2016-03-04 17:41:23 +01:00
|
|
|
searches for an unparameterized path in the path list of the specified
|
|
|
|
join relation. (If it does not find such a path, it returns NULL, in
|
|
|
|
which case a foreign data wrapper may build the local path by itself or
|
|
|
|
may choose not to create access paths for that join.)
|
Allow foreign and custom joins to handle EvalPlanQual rechecks.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 provided basic
infrastructure for allowing a foreign data wrapper or custom scan
provider to replace a join of one or more tables with a scan.
However, this infrastructure failed to take into account the need
for possible EvalPlanQual rechecks, and ExecScanFetch would fail
an assertion (or just overwrite memory) if such a check was attempted
for a plan containing a pushed-down join. To fix, adjust the EPQ
machinery to skip some processing steps when scanrelid == 0, making
those the responsibility of scan's recheck method, which also has
the responsibility in this case of correctly populating the relevant
slot.
To allow foreign scans to gain control in the right place to make
use of this new facility, add a new, optional RecheckForeignScan
method. Also, allow a foreign scan to have a child plan, which can
be used to correctly populate the slot (or perhaps for something
else, but this is the only use currently envisioned).
KaiGai Kohei, reviewed by Robert Haas, Etsuro Fujita, and Kyotaro
Horiguchi.
2015-12-08 18:31:03 +01:00
|
|
|
</para>
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
</sect2>
|
|
|
|
|
2013-03-10 19:14:53 +01:00
|
|
|
<sect2 id="fdw-callbacks-explain">
|
2017-10-09 03:44:17 +02:00
|
|
|
<title>FDW Routines for <command>EXPLAIN</command></title>
|
2013-03-10 19:14:53 +01:00
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
2017-09-06 17:38:28 +02:00
|
|
|
ExplainForeignScan(ForeignScanState *node,
|
|
|
|
ExplainState *es);
|
2013-03-10 19:14:53 +01:00
|
|
|
</programlisting>
|
|
|
|
|
2017-10-09 03:44:17 +02:00
|
|
|
Print additional <command>EXPLAIN</command> output for a foreign table scan.
|
|
|
|
This function can call <function>ExplainPropertyText</function> and
|
|
|
|
related functions to add fields to the <command>EXPLAIN</command> output.
|
|
|
|
The flag fields in <literal>es</literal> can be used to determine what to
|
|
|
|
print, and the state of the <structname>ForeignScanState</structname> node
|
2013-03-10 19:14:53 +01:00
|
|
|
can be inspected to provide run-time statistics in the <command>EXPLAIN
|
2017-10-09 03:44:17 +02:00
|
|
|
ANALYZE</command> case.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If the <function>ExplainForeignScan</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, no additional information is printed during
|
|
|
|
<command>EXPLAIN</command>.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
2017-09-06 17:38:28 +02:00
|
|
|
ExplainForeignModify(ModifyTableState *mtstate,
|
|
|
|
ResultRelInfo *rinfo,
|
|
|
|
List *fdw_private,
|
|
|
|
int subplan_index,
|
|
|
|
struct ExplainState *es);
|
2013-03-10 19:14:53 +01:00
|
|
|
</programlisting>
|
|
|
|
|
2017-10-09 03:44:17 +02:00
|
|
|
Print additional <command>EXPLAIN</command> output for a foreign table update.
|
|
|
|
This function can call <function>ExplainPropertyText</function> and
|
|
|
|
related functions to add fields to the <command>EXPLAIN</command> output.
|
|
|
|
The flag fields in <literal>es</literal> can be used to determine what to
|
|
|
|
print, and the state of the <structname>ModifyTableState</structname> node
|
2013-03-10 19:14:53 +01:00
|
|
|
can be inspected to provide run-time statistics in the <command>EXPLAIN
|
2017-10-09 03:44:17 +02:00
|
|
|
ANALYZE</command> case. The first four arguments are the same as for
|
|
|
|
<function>BeginForeignModify</function>.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If the <function>ExplainForeignModify</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, no additional information is printed during
|
|
|
|
<command>EXPLAIN</command>.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
2016-03-18 18:48:58 +01:00
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
2017-09-06 17:38:28 +02:00
|
|
|
ExplainDirectModify(ForeignScanState *node,
|
|
|
|
ExplainState *es);
|
2016-03-18 18:48:58 +01:00
|
|
|
</programlisting>
|
|
|
|
|
2017-10-09 03:44:17 +02:00
|
|
|
Print additional <command>EXPLAIN</command> output for a direct modification
|
2016-03-18 18:48:58 +01:00
|
|
|
on the remote server.
|
2017-10-09 03:44:17 +02:00
|
|
|
This function can call <function>ExplainPropertyText</function> and
|
|
|
|
related functions to add fields to the <command>EXPLAIN</command> output.
|
|
|
|
The flag fields in <literal>es</literal> can be used to determine what to
|
|
|
|
print, and the state of the <structname>ForeignScanState</structname> node
|
2016-03-18 18:48:58 +01:00
|
|
|
can be inspected to provide run-time statistics in the <command>EXPLAIN
|
2017-10-09 03:44:17 +02:00
|
|
|
ANALYZE</command> case.
|
2016-03-18 18:48:58 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If the <function>ExplainDirectModify</function> pointer is set to
|
|
|
|
<literal>NULL</literal>, no additional information is printed during
|
|
|
|
<command>EXPLAIN</command>.
|
2016-03-18 18:48:58 +01:00
|
|
|
</para>
|
|
|
|
|
2013-03-10 19:14:53 +01:00
|
|
|
</sect2>
|
|
|
|
|
|
|
|
<sect2 id="fdw-callbacks-analyze">
|
2017-10-09 03:44:17 +02:00
|
|
|
<title>FDW Routines for <command>ANALYZE</command></title>
|
2013-03-10 19:14:53 +01:00
|
|
|
|
2012-04-06 21:02:35 +02:00
|
|
|
<para>
|
|
|
|
<programlisting>
|
2012-04-06 22:04:10 +02:00
|
|
|
bool
|
2017-09-06 17:38:28 +02:00
|
|
|
AnalyzeForeignTable(Relation relation,
|
|
|
|
AcquireSampleRowsFunc *func,
|
|
|
|
BlockNumber *totalpages);
|
2012-04-06 21:02:35 +02:00
|
|
|
</programlisting>
|
|
|
|
|
2017-11-23 15:39:47 +01:00
|
|
|
This function is called when <xref linkend="sql-analyze"/> is executed on
|
2012-04-06 22:04:10 +02:00
|
|
|
a foreign table. If the FDW can collect statistics for this
|
2017-10-09 03:44:17 +02:00
|
|
|
foreign table, it should return <literal>true</literal>, and provide a pointer
|
2012-04-06 22:04:10 +02:00
|
|
|
to a function that will collect sample rows from the table in
|
2017-10-09 03:44:17 +02:00
|
|
|
<parameter>func</parameter>, plus the estimated size of the table in pages in
|
|
|
|
<parameter>totalpages</parameter>. Otherwise, return <literal>false</literal>.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2012-04-06 22:04:10 +02:00
|
|
|
If the FDW does not support collecting statistics for any tables, the
|
2017-10-09 03:44:17 +02:00
|
|
|
<function>AnalyzeForeignTable</function> pointer can be set to <literal>NULL</literal>.
|
2012-04-06 21:02:35 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
If provided, the sample collection function must have the signature
|
|
|
|
<programlisting>
|
|
|
|
int
|
2017-09-06 17:38:28 +02:00
|
|
|
AcquireSampleRowsFunc(Relation relation,
|
|
|
|
int elevel,
|
|
|
|
HeapTuple *rows,
|
|
|
|
int targrows,
|
|
|
|
double *totalrows,
|
|
|
|
double *totaldeadrows);
|
2012-04-06 21:02:35 +02:00
|
|
|
</programlisting>
|
|
|
|
|
2017-10-09 03:44:17 +02:00
|
|
|
A random sample of up to <parameter>targrows</parameter> rows should be collected
|
|
|
|
from the table and stored into the caller-provided <parameter>rows</parameter>
|
2012-04-06 21:02:35 +02:00
|
|
|
array. The actual number of rows collected must be returned. In
|
2012-04-06 22:04:10 +02:00
|
|
|
addition, store estimates of the total numbers of live and dead rows in
|
2017-10-09 03:44:17 +02:00
|
|
|
the table into the output parameters <parameter>totalrows</parameter> and
|
|
|
|
<parameter>totaldeadrows</parameter>. (Set <parameter>totaldeadrows</parameter> to zero
|
2012-04-06 22:04:10 +02:00
|
|
|
if the FDW does not have any concept of dead rows.)
|
2012-04-06 21:02:35 +02:00
|
|
|
</para>
|
|
|
|
|
2013-03-10 19:14:53 +01:00
|
|
|
</sect2>
|
2011-02-20 06:17:18 +01:00
|
|
|
|
2014-07-10 21:01:31 +02:00
|
|
|
<sect2 id="fdw-callbacks-import">
|
2017-10-09 03:44:17 +02:00
|
|
|
<title>FDW Routines For <command>IMPORT FOREIGN SCHEMA</command></title>
|
2014-07-10 21:01:31 +02:00
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
List *
|
2017-09-06 17:38:28 +02:00
|
|
|
ImportForeignSchema(ImportForeignSchemaStmt *stmt, Oid serverOid);
|
2014-07-10 21:01:31 +02:00
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
Obtain a list of foreign table creation commands. This function is
|
2017-11-23 15:39:47 +01:00
|
|
|
called when executing <xref linkend="sql-importforeignschema"/>, and is
|
2014-07-10 21:01:31 +02:00
|
|
|
passed the parse tree for that statement, as well as the OID of the
|
|
|
|
foreign server to use. It should return a list of C strings, each of
|
2017-11-23 15:39:47 +01:00
|
|
|
which must contain a <xref linkend="sql-createforeigntable"/> command.
|
2014-07-10 21:01:31 +02:00
|
|
|
These strings will be parsed and executed by the core server.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
Within the <structname>ImportForeignSchemaStmt</structname> struct,
|
|
|
|
<structfield>remote_schema</structfield> is the name of the remote schema from
|
2014-07-10 21:01:31 +02:00
|
|
|
which tables are to be imported.
|
2017-10-09 03:44:17 +02:00
|
|
|
<structfield>list_type</structfield> identifies how to filter table names:
|
|
|
|
<literal>FDW_IMPORT_SCHEMA_ALL</literal> means that all tables in the remote
|
|
|
|
schema should be imported (in this case <structfield>table_list</structfield> is
|
|
|
|
empty), <literal>FDW_IMPORT_SCHEMA_LIMIT_TO</literal> means to include only
|
|
|
|
tables listed in <structfield>table_list</structfield>,
|
|
|
|
and <literal>FDW_IMPORT_SCHEMA_EXCEPT</literal> means to exclude the tables
|
|
|
|
listed in <structfield>table_list</structfield>.
|
|
|
|
<structfield>options</structfield> is a list of options used for the import process.
|
2014-07-10 21:01:31 +02:00
|
|
|
The meanings of the options are up to the FDW.
|
|
|
|
For example, an FDW could use an option to define whether the
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>NOT NULL</literal> attributes of columns should be imported.
|
2014-07-10 21:01:31 +02:00
|
|
|
These options need not have anything to do with those supported by the
|
|
|
|
FDW as database object options.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
The FDW may ignore the <structfield>local_schema</structfield> field of
|
|
|
|
the <structname>ImportForeignSchemaStmt</structname>, because the core server
|
2014-07-10 21:01:31 +02:00
|
|
|
will automatically insert that name into the parsed <command>CREATE
|
2017-10-09 03:44:17 +02:00
|
|
|
FOREIGN TABLE</command> commands.
|
2014-07-10 21:01:31 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The FDW does not have to concern itself with implementing the filtering
|
2017-10-09 03:44:17 +02:00
|
|
|
specified by <structfield>list_type</structfield> and <structfield>table_list</structfield>,
|
2014-07-10 21:01:31 +02:00
|
|
|
either, as the core server will automatically skip any returned commands
|
|
|
|
for tables excluded according to those options. However, it's often
|
|
|
|
useful to avoid the work of creating commands for excluded tables in the
|
2017-10-09 03:44:17 +02:00
|
|
|
first place. The function <function>IsImportableForeignTable()</function> may be
|
2014-07-10 21:01:31 +02:00
|
|
|
useful to test whether a given foreign-table name will pass the filter.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
If the FDW does not support importing table definitions, the
|
2017-10-09 03:44:17 +02:00
|
|
|
<function>ImportForeignSchema</function> pointer can be set to <literal>NULL</literal>.
|
2014-07-10 21:01:31 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
</sect2>
|
|
|
|
|
2016-02-03 18:46:18 +01:00
|
|
|
<sect2 id="fdw-callbacks-parallel">
|
|
|
|
<title>FDW Routines for Parallel Execution</title>
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
A <structname>ForeignScan</structname> node can, optionally, support parallel
|
|
|
|
execution. A parallel <structname>ForeignScan</structname> will be executed
|
Separate reinitialization of shared parallel-scan state from ExecReScan.
Previously, the parallel executor logic did reinitialization of shared
state within the ExecReScan code for parallel-aware scan nodes. This is
problematic, because it means that the ExecReScan call has to occur
synchronously (ie, during the parent Gather node's ReScan call). That is
swimming very much against the tide so far as the ExecReScan machinery is
concerned; the fact that it works at all today depends on a lot of fragile
assumptions, such as that no plan node between Gather and a parallel-aware
scan node is parameterized. Another objection is that because ExecReScan
might be called in workers as well as the leader, hacky extra tests are
needed in some places to prevent unwanted shared-state resets.
Hence, let's separate this code into two functions, a ReInitializeDSM
call and the ReScan call proper. ReInitializeDSM is called only in
the leader and is guaranteed to run before we start new workers.
ReScan is returned to its traditional function of resetting only local
state, which means that ExecReScan's usual habits of delaying or
eliminating child rescan calls are safe again.
As with the preceding commit 7df2c1f8d, it doesn't seem to be necessary
to make these changes in 9.6, which is a good thing because the FDW and
CustomScan APIs are impacted.
Discussion: https://postgr.es/m/CAA4eK1JkByysFJNh9M349u_nNjqETuEnY_y1VUc_kJiU0bxtaQ@mail.gmail.com
2017-08-30 19:18:16 +02:00
|
|
|
in multiple processes and must return each row exactly once across
|
2016-02-03 18:46:18 +01:00
|
|
|
all cooperating processes. To do this, processes can coordinate through
|
Separate reinitialization of shared parallel-scan state from ExecReScan.
Previously, the parallel executor logic did reinitialization of shared
state within the ExecReScan code for parallel-aware scan nodes. This is
problematic, because it means that the ExecReScan call has to occur
synchronously (ie, during the parent Gather node's ReScan call). That is
swimming very much against the tide so far as the ExecReScan machinery is
concerned; the fact that it works at all today depends on a lot of fragile
assumptions, such as that no plan node between Gather and a parallel-aware
scan node is parameterized. Another objection is that because ExecReScan
might be called in workers as well as the leader, hacky extra tests are
needed in some places to prevent unwanted shared-state resets.
Hence, let's separate this code into two functions, a ReInitializeDSM
call and the ReScan call proper. ReInitializeDSM is called only in
the leader and is guaranteed to run before we start new workers.
ReScan is returned to its traditional function of resetting only local
state, which means that ExecReScan's usual habits of delaying or
eliminating child rescan calls are safe again.
As with the preceding commit 7df2c1f8d, it doesn't seem to be necessary
to make these changes in 9.6, which is a good thing because the FDW and
CustomScan APIs are impacted.
Discussion: https://postgr.es/m/CAA4eK1JkByysFJNh9M349u_nNjqETuEnY_y1VUc_kJiU0bxtaQ@mail.gmail.com
2017-08-30 19:18:16 +02:00
|
|
|
fixed-size chunks of dynamic shared memory. This shared memory is not
|
|
|
|
guaranteed to be mapped at the same address in every process, so it
|
|
|
|
must not contain pointers. The following functions are all optional,
|
|
|
|
but most are required if parallel execution is to be supported.
|
2016-02-03 18:46:18 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
2016-07-08 18:40:51 +02:00
|
|
|
bool
|
2016-02-26 11:44:46 +01:00
|
|
|
IsForeignScanParallelSafe(PlannerInfo *root, RelOptInfo *rel,
|
|
|
|
RangeTblEntry *rte);
|
|
|
|
</programlisting>
|
|
|
|
Test whether a scan can be performed within a parallel worker. This
|
|
|
|
function will only be called when the planner believes that a parallel
|
|
|
|
plan might be possible, and should return true if it is safe for that scan
|
|
|
|
to run within a parallel worker. This will generally not be the case if
|
|
|
|
the remote data source has transaction semantics, unless the worker's
|
|
|
|
connection to the data can somehow be made to share the same transaction
|
|
|
|
context as the leader.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
Separate reinitialization of shared parallel-scan state from ExecReScan.
Previously, the parallel executor logic did reinitialization of shared
state within the ExecReScan code for parallel-aware scan nodes. This is
problematic, because it means that the ExecReScan call has to occur
synchronously (ie, during the parent Gather node's ReScan call). That is
swimming very much against the tide so far as the ExecReScan machinery is
concerned; the fact that it works at all today depends on a lot of fragile
assumptions, such as that no plan node between Gather and a parallel-aware
scan node is parameterized. Another objection is that because ExecReScan
might be called in workers as well as the leader, hacky extra tests are
needed in some places to prevent unwanted shared-state resets.
Hence, let's separate this code into two functions, a ReInitializeDSM
call and the ReScan call proper. ReInitializeDSM is called only in
the leader and is guaranteed to run before we start new workers.
ReScan is returned to its traditional function of resetting only local
state, which means that ExecReScan's usual habits of delaying or
eliminating child rescan calls are safe again.
As with the preceding commit 7df2c1f8d, it doesn't seem to be necessary
to make these changes in 9.6, which is a good thing because the FDW and
CustomScan APIs are impacted.
Discussion: https://postgr.es/m/CAA4eK1JkByysFJNh9M349u_nNjqETuEnY_y1VUc_kJiU0bxtaQ@mail.gmail.com
2017-08-30 19:18:16 +02:00
|
|
|
If this function is not defined, it is assumed that the scan must take
|
2016-02-26 11:44:46 +01:00
|
|
|
place within the parallel leader. Note that returning true does not mean
|
|
|
|
that the scan itself can be done in parallel, only that the scan can be
|
|
|
|
performed within a parallel worker. Therefore, it can be useful to define
|
|
|
|
this method even when parallel execution is not supported.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
Size
|
2016-02-03 18:46:18 +01:00
|
|
|
EstimateDSMForeignScan(ForeignScanState *node, ParallelContext *pcxt);
|
|
|
|
</programlisting>
|
|
|
|
Estimate the amount of dynamic shared memory that will be required
|
|
|
|
for parallel operation. This may be higher than the amount that will
|
|
|
|
actually be used, but it must not be lower. The return value is in bytes.
|
Separate reinitialization of shared parallel-scan state from ExecReScan.
Previously, the parallel executor logic did reinitialization of shared
state within the ExecReScan code for parallel-aware scan nodes. This is
problematic, because it means that the ExecReScan call has to occur
synchronously (ie, during the parent Gather node's ReScan call). That is
swimming very much against the tide so far as the ExecReScan machinery is
concerned; the fact that it works at all today depends on a lot of fragile
assumptions, such as that no plan node between Gather and a parallel-aware
scan node is parameterized. Another objection is that because ExecReScan
might be called in workers as well as the leader, hacky extra tests are
needed in some places to prevent unwanted shared-state resets.
Hence, let's separate this code into two functions, a ReInitializeDSM
call and the ReScan call proper. ReInitializeDSM is called only in
the leader and is guaranteed to run before we start new workers.
ReScan is returned to its traditional function of resetting only local
state, which means that ExecReScan's usual habits of delaying or
eliminating child rescan calls are safe again.
As with the preceding commit 7df2c1f8d, it doesn't seem to be necessary
to make these changes in 9.6, which is a good thing because the FDW and
CustomScan APIs are impacted.
Discussion: https://postgr.es/m/CAA4eK1JkByysFJNh9M349u_nNjqETuEnY_y1VUc_kJiU0bxtaQ@mail.gmail.com
2017-08-30 19:18:16 +02:00
|
|
|
This function is optional, and can be omitted if not needed; but if it
|
|
|
|
is omitted, the next three functions must be omitted as well, because
|
|
|
|
no shared memory will be allocated for the FDW's use.
|
2016-02-03 18:46:18 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
|
|
|
InitializeDSMForeignScan(ForeignScanState *node, ParallelContext *pcxt,
|
|
|
|
void *coordinate);
|
|
|
|
</programlisting>
|
|
|
|
Initialize the dynamic shared memory that will be required for parallel
|
2017-10-09 03:44:17 +02:00
|
|
|
operation. <literal>coordinate</literal> points to a shared memory area of
|
|
|
|
size equal to the return value of <function>EstimateDSMForeignScan</function>.
|
Separate reinitialization of shared parallel-scan state from ExecReScan.
Previously, the parallel executor logic did reinitialization of shared
state within the ExecReScan code for parallel-aware scan nodes. This is
problematic, because it means that the ExecReScan call has to occur
synchronously (ie, during the parent Gather node's ReScan call). That is
swimming very much against the tide so far as the ExecReScan machinery is
concerned; the fact that it works at all today depends on a lot of fragile
assumptions, such as that no plan node between Gather and a parallel-aware
scan node is parameterized. Another objection is that because ExecReScan
might be called in workers as well as the leader, hacky extra tests are
needed in some places to prevent unwanted shared-state resets.
Hence, let's separate this code into two functions, a ReInitializeDSM
call and the ReScan call proper. ReInitializeDSM is called only in
the leader and is guaranteed to run before we start new workers.
ReScan is returned to its traditional function of resetting only local
state, which means that ExecReScan's usual habits of delaying or
eliminating child rescan calls are safe again.
As with the preceding commit 7df2c1f8d, it doesn't seem to be necessary
to make these changes in 9.6, which is a good thing because the FDW and
CustomScan APIs are impacted.
Discussion: https://postgr.es/m/CAA4eK1JkByysFJNh9M349u_nNjqETuEnY_y1VUc_kJiU0bxtaQ@mail.gmail.com
2017-08-30 19:18:16 +02:00
|
|
|
This function is optional, and can be omitted if not needed.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
|
|
|
ReInitializeDSMForeignScan(ForeignScanState *node, ParallelContext *pcxt,
|
|
|
|
void *coordinate);
|
|
|
|
</programlisting>
|
|
|
|
Re-initialize the dynamic shared memory required for parallel operation
|
|
|
|
when the foreign-scan plan node is about to be re-scanned.
|
|
|
|
This function is optional, and can be omitted if not needed.
|
|
|
|
Recommended practice is that this function reset only shared state,
|
2017-10-09 03:44:17 +02:00
|
|
|
while the <function>ReScanForeignScan</function> function resets only local
|
Separate reinitialization of shared parallel-scan state from ExecReScan.
Previously, the parallel executor logic did reinitialization of shared
state within the ExecReScan code for parallel-aware scan nodes. This is
problematic, because it means that the ExecReScan call has to occur
synchronously (ie, during the parent Gather node's ReScan call). That is
swimming very much against the tide so far as the ExecReScan machinery is
concerned; the fact that it works at all today depends on a lot of fragile
assumptions, such as that no plan node between Gather and a parallel-aware
scan node is parameterized. Another objection is that because ExecReScan
might be called in workers as well as the leader, hacky extra tests are
needed in some places to prevent unwanted shared-state resets.
Hence, let's separate this code into two functions, a ReInitializeDSM
call and the ReScan call proper. ReInitializeDSM is called only in
the leader and is guaranteed to run before we start new workers.
ReScan is returned to its traditional function of resetting only local
state, which means that ExecReScan's usual habits of delaying or
eliminating child rescan calls are safe again.
As with the preceding commit 7df2c1f8d, it doesn't seem to be necessary
to make these changes in 9.6, which is a good thing because the FDW and
CustomScan APIs are impacted.
Discussion: https://postgr.es/m/CAA4eK1JkByysFJNh9M349u_nNjqETuEnY_y1VUc_kJiU0bxtaQ@mail.gmail.com
2017-08-30 19:18:16 +02:00
|
|
|
state. Currently, this function will be called
|
2017-10-09 03:44:17 +02:00
|
|
|
before <function>ReScanForeignScan</function>, but it's best not to rely on
|
Separate reinitialization of shared parallel-scan state from ExecReScan.
Previously, the parallel executor logic did reinitialization of shared
state within the ExecReScan code for parallel-aware scan nodes. This is
problematic, because it means that the ExecReScan call has to occur
synchronously (ie, during the parent Gather node's ReScan call). That is
swimming very much against the tide so far as the ExecReScan machinery is
concerned; the fact that it works at all today depends on a lot of fragile
assumptions, such as that no plan node between Gather and a parallel-aware
scan node is parameterized. Another objection is that because ExecReScan
might be called in workers as well as the leader, hacky extra tests are
needed in some places to prevent unwanted shared-state resets.
Hence, let's separate this code into two functions, a ReInitializeDSM
call and the ReScan call proper. ReInitializeDSM is called only in
the leader and is guaranteed to run before we start new workers.
ReScan is returned to its traditional function of resetting only local
state, which means that ExecReScan's usual habits of delaying or
eliminating child rescan calls are safe again.
As with the preceding commit 7df2c1f8d, it doesn't seem to be necessary
to make these changes in 9.6, which is a good thing because the FDW and
CustomScan APIs are impacted.
Discussion: https://postgr.es/m/CAA4eK1JkByysFJNh9M349u_nNjqETuEnY_y1VUc_kJiU0bxtaQ@mail.gmail.com
2017-08-30 19:18:16 +02:00
|
|
|
that ordering.
|
2016-02-03 18:46:18 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
|
|
|
InitializeWorkerForeignScan(ForeignScanState *node, shm_toc *toc,
|
|
|
|
void *coordinate);
|
|
|
|
</programlisting>
|
Separate reinitialization of shared parallel-scan state from ExecReScan.
Previously, the parallel executor logic did reinitialization of shared
state within the ExecReScan code for parallel-aware scan nodes. This is
problematic, because it means that the ExecReScan call has to occur
synchronously (ie, during the parent Gather node's ReScan call). That is
swimming very much against the tide so far as the ExecReScan machinery is
concerned; the fact that it works at all today depends on a lot of fragile
assumptions, such as that no plan node between Gather and a parallel-aware
scan node is parameterized. Another objection is that because ExecReScan
might be called in workers as well as the leader, hacky extra tests are
needed in some places to prevent unwanted shared-state resets.
Hence, let's separate this code into two functions, a ReInitializeDSM
call and the ReScan call proper. ReInitializeDSM is called only in
the leader and is guaranteed to run before we start new workers.
ReScan is returned to its traditional function of resetting only local
state, which means that ExecReScan's usual habits of delaying or
eliminating child rescan calls are safe again.
As with the preceding commit 7df2c1f8d, it doesn't seem to be necessary
to make these changes in 9.6, which is a good thing because the FDW and
CustomScan APIs are impacted.
Discussion: https://postgr.es/m/CAA4eK1JkByysFJNh9M349u_nNjqETuEnY_y1VUc_kJiU0bxtaQ@mail.gmail.com
2017-08-30 19:18:16 +02:00
|
|
|
Initialize a parallel worker's local state based on the shared state
|
2017-10-09 03:44:17 +02:00
|
|
|
set up by the leader during <function>InitializeDSMForeignScan</function>.
|
Separate reinitialization of shared parallel-scan state from ExecReScan.
Previously, the parallel executor logic did reinitialization of shared
state within the ExecReScan code for parallel-aware scan nodes. This is
problematic, because it means that the ExecReScan call has to occur
synchronously (ie, during the parent Gather node's ReScan call). That is
swimming very much against the tide so far as the ExecReScan machinery is
concerned; the fact that it works at all today depends on a lot of fragile
assumptions, such as that no plan node between Gather and a parallel-aware
scan node is parameterized. Another objection is that because ExecReScan
might be called in workers as well as the leader, hacky extra tests are
needed in some places to prevent unwanted shared-state resets.
Hence, let's separate this code into two functions, a ReInitializeDSM
call and the ReScan call proper. ReInitializeDSM is called only in
the leader and is guaranteed to run before we start new workers.
ReScan is returned to its traditional function of resetting only local
state, which means that ExecReScan's usual habits of delaying or
eliminating child rescan calls are safe again.
As with the preceding commit 7df2c1f8d, it doesn't seem to be necessary
to make these changes in 9.6, which is a good thing because the FDW and
CustomScan APIs are impacted.
Discussion: https://postgr.es/m/CAA4eK1JkByysFJNh9M349u_nNjqETuEnY_y1VUc_kJiU0bxtaQ@mail.gmail.com
2017-08-30 19:18:16 +02:00
|
|
|
This function is optional, and can be omitted if not needed.
|
2016-02-03 18:46:18 +01:00
|
|
|
</para>
|
2017-02-26 09:06:49 +01:00
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
void
|
|
|
|
ShutdownForeignScan(ForeignScanState *node);
|
|
|
|
</programlisting>
|
|
|
|
Release resources when it is anticipated the node will not be executed
|
|
|
|
to completion. This is not called in all cases; sometimes,
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>EndForeignScan</literal> may be called without this function having
|
2017-02-26 09:06:49 +01:00
|
|
|
been called first. Since the DSM segment used by parallel query is
|
|
|
|
destroyed just after this callback is invoked, foreign data wrappers that
|
|
|
|
wish to take some action before the DSM segment goes away should implement
|
|
|
|
this method.
|
|
|
|
</para>
|
2016-02-03 18:46:18 +01:00
|
|
|
</sect2>
|
|
|
|
|
Basic partition-wise join functionality.
Instead of joining two partitioned tables in their entirety we can, if
it is an equi-join on the partition keys, join the matching partitions
individually. This involves teaching the planner about "other join"
rels, which are related to regular join rels in the same way that
other member rels are related to baserels. This can use significantly
more CPU time and memory than regular join planning, because there may
now be a set of "other" rels not only for every base relation but also
for every join relation. In most practical cases, this probably
shouldn't be a problem, because (1) it's probably unusual to join many
tables each with many partitions using the partition keys for all
joins and (2) if you do that scenario then you probably have a big
enough machine to handle the increased memory cost of planning and (3)
the resulting plan is highly likely to be better, so what you spend in
planning you'll make up on the execution side. All the same, for now,
turn this feature off by default.
Currently, we can only perform joins between two tables whose
partitioning schemes are absolutely identical. It would be nice to
cope with other scenarios, such as extra partitions on one side or the
other with no match on the other side, but that will have to wait for
a future patch.
Ashutosh Bapat, reviewed and tested by Rajkumar Raghuwanshi, Amit
Langote, Rafia Sabih, Thomas Munro, Dilip Kumar, Antonin Houska, Amit
Khandekar, and by me. A few final adjustments by me.
Discussion: http://postgr.es/m/CAFjFpRfQ8GrQvzp3jA2wnLqrHmaXna-urjm_UY9BqXj=EaDTSA@mail.gmail.com
Discussion: http://postgr.es/m/CAFjFpRcitjfrULr5jfuKWRPsGUX0LQ0k8-yG0Qw2+1LBGNpMdw@mail.gmail.com
2017-10-06 17:11:10 +02:00
|
|
|
<sect2 id="fdw-callbacks-reparameterize-paths">
|
|
|
|
<title>FDW Routines For reparameterization of paths</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
List *
|
|
|
|
ReparameterizeForeignPathByChild(PlannerInfo *root, List *fdw_private,
|
|
|
|
RelOptInfo *child_rel);
|
|
|
|
</programlisting>
|
|
|
|
This function is called while converting a path parameterized by the
|
2017-10-09 03:44:17 +02:00
|
|
|
top-most parent of the given child relation <literal>child_rel</literal> to be
|
Basic partition-wise join functionality.
Instead of joining two partitioned tables in their entirety we can, if
it is an equi-join on the partition keys, join the matching partitions
individually. This involves teaching the planner about "other join"
rels, which are related to regular join rels in the same way that
other member rels are related to baserels. This can use significantly
more CPU time and memory than regular join planning, because there may
now be a set of "other" rels not only for every base relation but also
for every join relation. In most practical cases, this probably
shouldn't be a problem, because (1) it's probably unusual to join many
tables each with many partitions using the partition keys for all
joins and (2) if you do that scenario then you probably have a big
enough machine to handle the increased memory cost of planning and (3)
the resulting plan is highly likely to be better, so what you spend in
planning you'll make up on the execution side. All the same, for now,
turn this feature off by default.
Currently, we can only perform joins between two tables whose
partitioning schemes are absolutely identical. It would be nice to
cope with other scenarios, such as extra partitions on one side or the
other with no match on the other side, but that will have to wait for
a future patch.
Ashutosh Bapat, reviewed and tested by Rajkumar Raghuwanshi, Amit
Langote, Rafia Sabih, Thomas Munro, Dilip Kumar, Antonin Houska, Amit
Khandekar, and by me. A few final adjustments by me.
Discussion: http://postgr.es/m/CAFjFpRfQ8GrQvzp3jA2wnLqrHmaXna-urjm_UY9BqXj=EaDTSA@mail.gmail.com
Discussion: http://postgr.es/m/CAFjFpRcitjfrULr5jfuKWRPsGUX0LQ0k8-yG0Qw2+1LBGNpMdw@mail.gmail.com
2017-10-06 17:11:10 +02:00
|
|
|
parameterized by the child relation. The function is used to reparameterize
|
|
|
|
any paths or translate any expression nodes saved in the given
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>fdw_private</literal> member of a <structname>ForeignPath</structname>. The
|
|
|
|
callback may use <literal>reparameterize_path_by_child</literal>,
|
|
|
|
<literal>adjust_appendrel_attrs</literal> or
|
|
|
|
<literal>adjust_appendrel_attrs_multilevel</literal> as required.
|
Basic partition-wise join functionality.
Instead of joining two partitioned tables in their entirety we can, if
it is an equi-join on the partition keys, join the matching partitions
individually. This involves teaching the planner about "other join"
rels, which are related to regular join rels in the same way that
other member rels are related to baserels. This can use significantly
more CPU time and memory than regular join planning, because there may
now be a set of "other" rels not only for every base relation but also
for every join relation. In most practical cases, this probably
shouldn't be a problem, because (1) it's probably unusual to join many
tables each with many partitions using the partition keys for all
joins and (2) if you do that scenario then you probably have a big
enough machine to handle the increased memory cost of planning and (3)
the resulting plan is highly likely to be better, so what you spend in
planning you'll make up on the execution side. All the same, for now,
turn this feature off by default.
Currently, we can only perform joins between two tables whose
partitioning schemes are absolutely identical. It would be nice to
cope with other scenarios, such as extra partitions on one side or the
other with no match on the other side, but that will have to wait for
a future patch.
Ashutosh Bapat, reviewed and tested by Rajkumar Raghuwanshi, Amit
Langote, Rafia Sabih, Thomas Munro, Dilip Kumar, Antonin Houska, Amit
Khandekar, and by me. A few final adjustments by me.
Discussion: http://postgr.es/m/CAFjFpRfQ8GrQvzp3jA2wnLqrHmaXna-urjm_UY9BqXj=EaDTSA@mail.gmail.com
Discussion: http://postgr.es/m/CAFjFpRcitjfrULr5jfuKWRPsGUX0LQ0k8-yG0Qw2+1LBGNpMdw@mail.gmail.com
2017-10-06 17:11:10 +02:00
|
|
|
</para>
|
|
|
|
</sect2>
|
|
|
|
|
2011-02-20 06:17:18 +01:00
|
|
|
</sect1>
|
|
|
|
|
2012-03-08 00:20:58 +01:00
|
|
|
<sect1 id="fdw-helpers">
|
|
|
|
<title>Foreign Data Wrapper Helper Functions</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Several helper functions are exported from the core server so that
|
|
|
|
authors of foreign data wrappers can get easy access to attributes of
|
|
|
|
FDW-related objects, such as FDW options.
|
|
|
|
To use any of these functions, you need to include the header file
|
|
|
|
<filename>foreign/foreign.h</filename> in your source file.
|
|
|
|
That header also defines the struct types that are returned by
|
|
|
|
these functions.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
ForeignDataWrapper *
|
2018-12-14 00:59:35 +01:00
|
|
|
GetForeignDataWrapperExtended(Oid fdwid, bits16 flags);
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
This function returns a <structname>ForeignDataWrapper</structname>
|
|
|
|
object for the foreign-data wrapper with the given OID. A
|
|
|
|
<structname>ForeignDataWrapper</structname> object contains properties
|
|
|
|
of the FDW (see <filename>foreign/foreign.h</filename> for details).
|
|
|
|
<structfield>flags</structfield> is a bitwise-or'd bit mask indicating
|
|
|
|
an extra set of options. It can take the value
|
|
|
|
<literal>FDW_MISSING_OK</literal>, in which case a <literal>NULL</literal>
|
|
|
|
result is returned to the caller instead of an error for an undefined
|
|
|
|
object.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
ForeignDataWrapper *
|
2012-03-08 00:20:58 +01:00
|
|
|
GetForeignDataWrapper(Oid fdwid);
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
This function returns a <structname>ForeignDataWrapper</structname>
|
|
|
|
object for the foreign-data wrapper with the given OID. A
|
|
|
|
<structname>ForeignDataWrapper</structname> object contains properties
|
|
|
|
of the FDW (see <filename>foreign/foreign.h</filename> for details).
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
ForeignServer *
|
2018-12-14 00:59:35 +01:00
|
|
|
GetForeignServerExtended(Oid serverid, bits16 flags);
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
This function returns a <structname>ForeignServer</structname> object
|
|
|
|
for the foreign server with the given OID. A
|
|
|
|
<structname>ForeignServer</structname> object contains properties
|
|
|
|
of the server (see <filename>foreign/foreign.h</filename> for details).
|
|
|
|
<structfield>flags</structfield> is a bitwise-or'd bit mask indicating
|
|
|
|
an extra set of options. It can take the value
|
|
|
|
<literal>FSV_MISSING_OK</literal>, in which case a <literal>NULL</literal>
|
|
|
|
result is returned to the caller instead of an error for an undefined
|
|
|
|
object.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
ForeignServer *
|
2012-03-08 00:20:58 +01:00
|
|
|
GetForeignServer(Oid serverid);
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
This function returns a <structname>ForeignServer</structname> object
|
|
|
|
for the foreign server with the given OID. A
|
|
|
|
<structname>ForeignServer</structname> object contains properties
|
|
|
|
of the server (see <filename>foreign/foreign.h</filename> for details).
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
UserMapping *
|
|
|
|
GetUserMapping(Oid userid, Oid serverid);
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
This function returns a <structname>UserMapping</structname> object for
|
|
|
|
the user mapping of the given role on the given server. (If there is no
|
|
|
|
mapping for the specific user, it will return the mapping for
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>PUBLIC</literal>, or throw error if there is none.) A
|
2012-03-08 00:20:58 +01:00
|
|
|
<structname>UserMapping</structname> object contains properties of the
|
|
|
|
user mapping (see <filename>foreign/foreign.h</filename> for details).
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
ForeignTable *
|
|
|
|
GetForeignTable(Oid relid);
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
This function returns a <structname>ForeignTable</structname> object for
|
|
|
|
the foreign table with the given OID. A
|
|
|
|
<structname>ForeignTable</structname> object contains properties of the
|
|
|
|
foreign table (see <filename>foreign/foreign.h</filename> for details).
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
List *
|
2012-12-22 21:01:29 +01:00
|
|
|
GetForeignColumnOptions(Oid relid, AttrNumber attnum);
|
2012-03-08 00:20:58 +01:00
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
This function returns the per-column FDW options for the column with the
|
|
|
|
given foreign table OID and attribute number, in the form of a list of
|
|
|
|
<structname>DefElem</structname>. NIL is returned if the column has no
|
|
|
|
options.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Some object types have name-based lookup functions in addition to the
|
|
|
|
OID-based ones:
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
ForeignDataWrapper *
|
|
|
|
GetForeignDataWrapperByName(const char *name, bool missing_ok);
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
This function returns a <structname>ForeignDataWrapper</structname>
|
|
|
|
object for the foreign-data wrapper with the given name. If the wrapper
|
|
|
|
is not found, return NULL if missing_ok is true, otherwise raise an
|
|
|
|
error.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
<programlisting>
|
|
|
|
ForeignServer *
|
|
|
|
GetForeignServerByName(const char *name, bool missing_ok);
|
|
|
|
</programlisting>
|
|
|
|
|
|
|
|
This function returns a <structname>ForeignServer</structname> object
|
|
|
|
for the foreign server with the given name. If the server is not found,
|
|
|
|
return NULL if missing_ok is true, otherwise raise an error.
|
|
|
|
</para>
|
|
|
|
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
</sect1>
|
|
|
|
|
|
|
|
<sect1 id="fdw-planning">
|
|
|
|
<title>Foreign Data Wrapper Query Planning</title>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
The FDW callback functions <function>GetForeignRelSize</function>,
|
|
|
|
<function>GetForeignPaths</function>, <function>GetForeignPlan</function>,
|
|
|
|
<function>PlanForeignModify</function>, <function>GetForeignJoinPaths</function>,
|
|
|
|
<function>GetForeignUpperPaths</function>, and <function>PlanDirectModify</function>
|
|
|
|
must fit into the workings of the <productname>PostgreSQL</productname> planner.
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
Here are some notes about what they must do.
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
The information in <literal>root</literal> and <literal>baserel</literal> can be used
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
to reduce the amount of information that has to be fetched from the
|
|
|
|
foreign table (and therefore reduce the cost).
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>baserel->baserestrictinfo</literal> is particularly interesting, as
|
|
|
|
it contains restriction quals (<literal>WHERE</literal> clauses) that should be
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
used to filter the rows to be fetched. (The FDW itself is not required
|
|
|
|
to enforce these quals, as the core executor can check them instead.)
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>baserel->reltarget->exprs</literal> can be used to determine which
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
columns need to be fetched; but note that it only lists columns that
|
2017-10-09 03:44:17 +02:00
|
|
|
have to be emitted by the <structname>ForeignScan</structname> plan node, not
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
columns that are used in qual evaluation but not output by the query.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Various private fields are available for the FDW planning functions to
|
|
|
|
keep information in. Generally, whatever you store in FDW private fields
|
|
|
|
should be palloc'd, so that it will be reclaimed at the end of planning.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>baserel->fdw_private</literal> is a <type>void</type> pointer that is
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
available for FDW planning functions to store information relevant to
|
|
|
|
the particular foreign table. The core planner does not touch it except
|
2017-10-09 03:44:17 +02:00
|
|
|
to initialize it to NULL when the <literal>RelOptInfo</literal> node is created.
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
It is useful for passing information forward from
|
2017-10-09 03:44:17 +02:00
|
|
|
<function>GetForeignRelSize</function> to <function>GetForeignPaths</function> and/or
|
|
|
|
<function>GetForeignPaths</function> to <function>GetForeignPlan</function>, thereby
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
avoiding recalculation.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
<function>GetForeignPaths</function> can identify the meaning of different
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
access paths by storing private information in the
|
2017-10-09 03:44:17 +02:00
|
|
|
<structfield>fdw_private</structfield> field of <structname>ForeignPath</structname> nodes.
|
|
|
|
<structfield>fdw_private</structfield> is declared as a <type>List</type> pointer, but
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
could actually contain anything since the core planner does not touch
|
|
|
|
it. However, best practice is to use a representation that's dumpable
|
2017-10-09 03:44:17 +02:00
|
|
|
by <function>nodeToString</function>, for use with debugging support available
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
in the backend.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
<function>GetForeignPlan</function> can examine the <structfield>fdw_private</structfield>
|
|
|
|
field of the selected <structname>ForeignPath</structname> node, and can generate
|
|
|
|
<structfield>fdw_exprs</structfield> and <structfield>fdw_private</structfield> lists to be
|
|
|
|
placed in the <structname>ForeignScan</structname> plan node, where they will be
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
available at execution time. Both of these lists must be
|
2017-10-09 03:44:17 +02:00
|
|
|
represented in a form that <function>copyObject</function> knows how to copy.
|
|
|
|
The <structfield>fdw_private</structfield> list has no other restrictions and is
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
not interpreted by the core backend in any way. The
|
2017-10-09 03:44:17 +02:00
|
|
|
<structfield>fdw_exprs</structfield> list, if not NIL, is expected to contain
|
2012-06-07 23:06:20 +02:00
|
|
|
expression trees that are intended to be executed at run time. These
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
trees will undergo post-processing by the planner to make them fully
|
|
|
|
executable.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
In <function>GetForeignPlan</function>, generally the passed-in target list can
|
|
|
|
be copied into the plan node as-is. The passed <literal>scan_clauses</literal> list
|
|
|
|
contains the same clauses as <literal>baserel->baserestrictinfo</literal>,
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
but may be re-ordered for better execution efficiency. In simple cases
|
2017-10-09 03:44:17 +02:00
|
|
|
the FDW can just strip <structname>RestrictInfo</structname> nodes from the
|
|
|
|
<literal>scan_clauses</literal> list (using <function>extract_actual_clauses</function>) and put
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
all the clauses into the plan node's qual list, which means that all the
|
2012-06-07 23:06:20 +02:00
|
|
|
clauses will be checked by the executor at run time. More complex FDWs
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
may be able to check some of the clauses internally, in which case those
|
|
|
|
clauses can be removed from the plan node's qual list so that the
|
|
|
|
executor doesn't waste time rechecking them.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
As an example, the FDW might identify some restriction clauses of the
|
2017-10-09 03:44:17 +02:00
|
|
|
form <replaceable>foreign_variable</replaceable> <literal>=</literal>
|
|
|
|
<replaceable>sub_expression</replaceable>, which it determines can be executed on
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
the remote server given the locally-evaluated value of the
|
2017-10-09 03:44:17 +02:00
|
|
|
<replaceable>sub_expression</replaceable>. The actual identification of such a
|
|
|
|
clause should happen during <function>GetForeignPaths</function>, since it would
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
affect the cost estimate for the path. The path's
|
2017-10-09 03:44:17 +02:00
|
|
|
<structfield>fdw_private</structfield> field would probably include a pointer to
|
|
|
|
the identified clause's <structname>RestrictInfo</structname> node. Then
|
|
|
|
<function>GetForeignPlan</function> would remove that clause from <literal>scan_clauses</literal>,
|
|
|
|
but add the <replaceable>sub_expression</replaceable> to <structfield>fdw_exprs</structfield>
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
to ensure that it gets massaged into executable form. It would probably
|
|
|
|
also put control information into the plan node's
|
2017-10-09 03:44:17 +02:00
|
|
|
<structfield>fdw_private</structfield> field to tell the execution functions what
|
2012-06-07 23:06:20 +02:00
|
|
|
to do at run time. The query transmitted to the remote server would
|
2017-10-09 03:44:17 +02:00
|
|
|
involve something like <literal>WHERE <replaceable>foreign_variable</replaceable> =
|
2012-06-07 23:06:20 +02:00
|
|
|
$1</literal>, with the parameter value obtained at run time from
|
2017-10-09 03:44:17 +02:00
|
|
|
evaluation of the <structfield>fdw_exprs</structfield> expression tree.
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
</para>
|
|
|
|
|
2015-10-15 19:00:40 +02:00
|
|
|
<para>
|
|
|
|
Any clauses removed from the plan node's qual list must instead be added
|
2017-10-09 03:44:17 +02:00
|
|
|
to <literal>fdw_recheck_quals</literal> or rechecked by
|
|
|
|
<literal>RecheckForeignScan</literal> in order to ensure correct behavior
|
|
|
|
at the <literal>READ COMMITTED</literal> isolation level. When a concurrent
|
2015-10-15 19:00:40 +02:00
|
|
|
update occurs for some other table involved in the query, the executor
|
|
|
|
may need to verify that all of the original quals are still satisfied for
|
Allow foreign and custom joins to handle EvalPlanQual rechecks.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 provided basic
infrastructure for allowing a foreign data wrapper or custom scan
provider to replace a join of one or more tables with a scan.
However, this infrastructure failed to take into account the need
for possible EvalPlanQual rechecks, and ExecScanFetch would fail
an assertion (or just overwrite memory) if such a check was attempted
for a plan containing a pushed-down join. To fix, adjust the EPQ
machinery to skip some processing steps when scanrelid == 0, making
those the responsibility of scan's recheck method, which also has
the responsibility in this case of correctly populating the relevant
slot.
To allow foreign scans to gain control in the right place to make
use of this new facility, add a new, optional RecheckForeignScan
method. Also, allow a foreign scan to have a child plan, which can
be used to correctly populate the slot (or perhaps for something
else, but this is the only use currently envisioned).
KaiGai Kohei, reviewed by Robert Haas, Etsuro Fujita, and Kyotaro
Horiguchi.
2015-12-08 18:31:03 +01:00
|
|
|
the tuple, possibly against a different set of parameter values. Using
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>fdw_recheck_quals</literal> is typically easier than implementing checks
|
|
|
|
inside <literal>RecheckForeignScan</literal>, but this method will be
|
Allow foreign and custom joins to handle EvalPlanQual rechecks.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 provided basic
infrastructure for allowing a foreign data wrapper or custom scan
provider to replace a join of one or more tables with a scan.
However, this infrastructure failed to take into account the need
for possible EvalPlanQual rechecks, and ExecScanFetch would fail
an assertion (or just overwrite memory) if such a check was attempted
for a plan containing a pushed-down join. To fix, adjust the EPQ
machinery to skip some processing steps when scanrelid == 0, making
those the responsibility of scan's recheck method, which also has
the responsibility in this case of correctly populating the relevant
slot.
To allow foreign scans to gain control in the right place to make
use of this new facility, add a new, optional RecheckForeignScan
method. Also, allow a foreign scan to have a child plan, which can
be used to correctly populate the slot (or perhaps for something
else, but this is the only use currently envisioned).
KaiGai Kohei, reviewed by Robert Haas, Etsuro Fujita, and Kyotaro
Horiguchi.
2015-12-08 18:31:03 +01:00
|
|
|
insufficient when outer joins have been pushed down, since the join tuples
|
|
|
|
in that case might have some fields go to NULL without rejecting the
|
|
|
|
tuple entirely.
|
2015-10-15 19:00:40 +02:00
|
|
|
</para>
|
|
|
|
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
Another <structname>ForeignScan</structname> field that can be filled by FDWs
|
|
|
|
is <structfield>fdw_scan_tlist</structfield>, which describes the tuples returned by
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
the FDW for this plan node. For simple foreign table scans this can be
|
2017-10-09 03:44:17 +02:00
|
|
|
set to <literal>NIL</literal>, implying that the returned tuples have the
|
2015-09-11 03:22:21 +02:00
|
|
|
row type declared for the foreign table. A non-<symbol>NIL</symbol> value must be a
|
2017-10-09 03:44:17 +02:00
|
|
|
target list (list of <structname>TargetEntry</structname>s) containing Vars and/or
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
expressions representing the returned columns. This might be used, for
|
|
|
|
example, to show that the FDW has omitted some columns that it noticed
|
|
|
|
won't be needed for the query. Also, if the FDW can compute expressions
|
|
|
|
used by the query more cheaply than can be done locally, it could add
|
2017-10-09 03:44:17 +02:00
|
|
|
those expressions to <structfield>fdw_scan_tlist</structfield>. Note that join
|
|
|
|
plans (created from paths made by <function>GetForeignJoinPaths</function>) must
|
|
|
|
always supply <structfield>fdw_scan_tlist</structfield> to describe the set of
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
columns they will return.
|
|
|
|
</para>
|
|
|
|
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
<para>
|
|
|
|
The FDW should always construct at least one path that depends only on
|
|
|
|
the table's restriction clauses. In join queries, it might also choose
|
|
|
|
to construct path(s) that depend on join clauses, for example
|
2017-10-09 03:44:17 +02:00
|
|
|
<replaceable>foreign_variable</replaceable> <literal>=</literal>
|
|
|
|
<replaceable>local_variable</replaceable>. Such clauses will not be found in
|
|
|
|
<literal>baserel->baserestrictinfo</literal> but must be sought in the
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
relation's join lists. A path using such a clause is called a
|
2017-10-09 03:44:17 +02:00
|
|
|
<quote>parameterized path</quote>. It must identify the other relations
|
2012-10-10 19:54:38 +02:00
|
|
|
used in the selected join clause(s) with a suitable value of
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>param_info</literal>; use <function>get_baserel_parampathinfo</function>
|
|
|
|
to compute that value. In <function>GetForeignPlan</function>, the
|
|
|
|
<replaceable>local_variable</replaceable> portion of the join clause would be added
|
|
|
|
to <structfield>fdw_exprs</structfield>, and then at run time the case works the
|
Revise FDW planning API, again.
Further reflection shows that a single callback isn't very workable if we
desire to let FDWs generate multiple Paths, because that forces the FDW to
do all work necessary to generate a valid Plan node for each Path. Instead
split the former PlanForeignScan API into three steps: GetForeignRelSize,
GetForeignPaths, GetForeignPlan. We had already bit the bullet of breaking
the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
and it's substantially more flexible for complex FDWs.
Add an fdw_private field to RelOptInfo so that the new functions can save
state there rather than possibly having to recalculate information two or
three times.
In addition, we'd not thought through what would be needed to allow an FDW
to set up subexpressions of its choice for runtime execution. We could
treat ForeignScan.fdw_private as an executable expression but that seems
likely to break existing FDWs unnecessarily (in particular, it would
restrict the set of node types allowable in fdw_private to those supported
by expression_tree_walker). Instead, invent a separate field fdw_exprs
which will receive the postprocessing appropriate for expression trees.
(One field is enough since it can be a list of expressions; also, we assume
the corresponding expression state tree(s) will be held within fdw_state,
so we don't need to add anything to ForeignScanState.)
Per review of Hanada Shigeru's pgsql_fdw patch. We may need to tweak this
further as we continue to work on that patch, but to me it feels a lot
closer to being right now.
2012-03-09 18:48:48 +01:00
|
|
|
same as for an ordinary restriction clause.
|
|
|
|
</para>
|
|
|
|
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
If an FDW supports remote joins, <function>GetForeignJoinPaths</function> should
|
|
|
|
produce <structname>ForeignPath</structname>s for potential remote joins in much
|
|
|
|
the same way as <function>GetForeignPaths</function> works for base tables.
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
Information about the intended join can be passed forward
|
2017-10-09 03:44:17 +02:00
|
|
|
to <function>GetForeignPlan</function> in the same ways described above.
|
|
|
|
However, <structfield>baserestrictinfo</structfield> is not relevant for join
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
relations; instead, the relevant join clauses for a particular join are
|
2017-10-09 03:44:17 +02:00
|
|
|
passed to <function>GetForeignJoinPaths</function> as a separate parameter
|
|
|
|
(<literal>extra->restrictlist</literal>).
|
Code review for foreign/custom join pushdown patch.
Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design
decisions that seem pretty questionable to me, and there was quite a lot
of stuff not to like about the documentation and comments. Clean up
as follows:
* Consider foreign joins only between foreign tables on the same server,
rather than between any two foreign tables with the same underlying FDW
handler function. In most if not all cases, the FDW would simply have had
to apply the same-server restriction itself (far more expensively, both for
lack of caching and because it would be repeated for each combination of
input sub-joins), or else risk nasty bugs. Anyone who's really intent on
doing something outside this restriction can always use the
set_join_pathlist_hook.
* Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
to better reflect what they're for, and allow these custom scan tlists
to be used even for base relations.
* Change make_foreignscan() API to include passing the fdw_scan_tlist
value, since the FDW is required to set that. Backwards compatibility
doesn't seem like an adequate reason to expect FDWs to set it in some
ad-hoc extra step, and anyway existing FDWs can just pass NIL.
* Change the API of path-generating subroutines of add_paths_to_joinrel,
and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
so that various less-used parameters are passed in a struct rather than
as separate parameter-list entries. The objective here is to reduce the
probability that future additions to those parameter lists will result in
source-level API breaks for users of these hooks. It's possible that this
is even a small win for the core code, since most CPU architectures can't
pass more than half a dozen parameters efficiently anyway. I kept root,
joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
code churn in joinpath.c --- in particular, putting jointype into the
struct would have been problematic because of the subroutines' habit of
changing their local copies of that variable.
* Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all
right for it to know about IndexOnlyScan, but if the list is to grow
we should refactor the knowledge out to the callers.
* Restore nodeForeignscan.c's previous use of the relcache to avoid
extra GetFdwRoutine lookups for base-relation scans.
* Lots of cleanup of documentation and missed comments. Re-order some
code additions into more logical places.
2015-05-10 20:36:30 +02:00
|
|
|
</para>
|
|
|
|
|
Make the upper part of the planner work by generating and comparing Paths.
I've been saying we needed to do this for more than five years, and here it
finally is. This patch removes the ever-growing tangle of spaghetti logic
that grouping_planner() used to use to try to identify the best plan for
post-scan/join query steps. Now, there is (nearly) independent
consideration of each execution step, and entirely separate construction of
Paths to represent each of the possible ways to do that step. We choose
the best Path or set of Paths using the same add_path() logic that's been
used inside query_planner() for years.
In addition, this patch removes the old restriction that subquery_planner()
could return only a single Plan. It now returns a RelOptInfo containing a
set of Paths, just as query_planner() does, and the parent query level can
use each of those Paths as the basis of a SubqueryScanPath at its level.
This allows finding some optimizations that we missed before, wherein a
subquery was capable of returning presorted data and thereby avoiding a
sort in the parent level, making the overall cost cheaper even though
delivering sorted output was not the cheapest plan for the subquery in
isolation. (A couple of regression test outputs change in consequence of
that. However, there is very little change in visible planner behavior
overall, because the point of this patch is not to get immediate planning
benefits but to create the infrastructure for future improvements.)
There is a great deal left to do here. This patch unblocks a lot of
planner work that was basically impractical in the old code structure,
such as allowing FDWs to implement remote aggregation, or rewriting
plan_set_operations() to allow consideration of multiple implementation
orders for set operations. (The latter will likely require a full
rewrite of plan_set_operations(); what I've done here is only to fix it
to return Paths not Plans.) I have also left unfinished some localized
refactoring in createplan.c and planner.c, because it was not necessary
to get this patch to a working state.
Thanks to Robert Haas, David Rowley, and Amit Kapila for review.
2016-03-07 21:58:22 +01:00
|
|
|
<para>
|
|
|
|
An FDW might additionally support direct execution of some plan actions
|
|
|
|
that are above the level of scans and joins, such as grouping or
|
2016-07-01 19:12:34 +02:00
|
|
|
aggregation. To offer such options, the FDW should generate paths and
|
2017-10-09 03:44:17 +02:00
|
|
|
insert them into the appropriate <firstterm>upper relation</firstterm>. For
|
2016-07-01 19:12:34 +02:00
|
|
|
example, a path representing remote aggregation should be inserted into
|
2017-10-09 03:44:17 +02:00
|
|
|
the <literal>UPPERREL_GROUP_AGG</literal> relation, using <function>add_path</function>.
|
2016-07-01 19:12:34 +02:00
|
|
|
This path will be compared on a cost basis with local aggregation
|
|
|
|
performed by reading a simple scan path for the foreign relation (note
|
|
|
|
that such a path must also be supplied, else there will be an error at
|
|
|
|
plan time). If the remote-aggregation path wins, which it usually would,
|
|
|
|
it will be converted into a plan in the usual way, by
|
2017-10-09 03:44:17 +02:00
|
|
|
calling <function>GetForeignPlan</function>. The recommended place to generate
|
|
|
|
such paths is in the <function>GetForeignUpperPaths</function>
|
2016-07-01 19:12:34 +02:00
|
|
|
callback function, which is called for each upper relation (i.e., each
|
|
|
|
post-scan/join processing step), if all the base relations of the query
|
|
|
|
come from the same FDW.
|
Make the upper part of the planner work by generating and comparing Paths.
I've been saying we needed to do this for more than five years, and here it
finally is. This patch removes the ever-growing tangle of spaghetti logic
that grouping_planner() used to use to try to identify the best plan for
post-scan/join query steps. Now, there is (nearly) independent
consideration of each execution step, and entirely separate construction of
Paths to represent each of the possible ways to do that step. We choose
the best Path or set of Paths using the same add_path() logic that's been
used inside query_planner() for years.
In addition, this patch removes the old restriction that subquery_planner()
could return only a single Plan. It now returns a RelOptInfo containing a
set of Paths, just as query_planner() does, and the parent query level can
use each of those Paths as the basis of a SubqueryScanPath at its level.
This allows finding some optimizations that we missed before, wherein a
subquery was capable of returning presorted data and thereby avoiding a
sort in the parent level, making the overall cost cheaper even though
delivering sorted output was not the cheapest plan for the subquery in
isolation. (A couple of regression test outputs change in consequence of
that. However, there is very little change in visible planner behavior
overall, because the point of this patch is not to get immediate planning
benefits but to create the infrastructure for future improvements.)
There is a great deal left to do here. This patch unblocks a lot of
planner work that was basically impractical in the old code structure,
such as allowing FDWs to implement remote aggregation, or rewriting
plan_set_operations() to allow consideration of multiple implementation
orders for set operations. (The latter will likely require a full
rewrite of plan_set_operations(); what I've done here is only to fix it
to return Paths not Plans.) I have also left unfinished some localized
refactoring in createplan.c and planner.c, because it was not necessary
to get this patch to a working state.
Thanks to Robert Haas, David Rowley, and Amit Kapila for review.
2016-03-07 21:58:22 +01:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
<function>PlanForeignModify</function> and the other callbacks described in
|
2017-11-23 15:39:47 +01:00
|
|
|
<xref linkend="fdw-callbacks-update"/> are designed around the assumption
|
Make the upper part of the planner work by generating and comparing Paths.
I've been saying we needed to do this for more than five years, and here it
finally is. This patch removes the ever-growing tangle of spaghetti logic
that grouping_planner() used to use to try to identify the best plan for
post-scan/join query steps. Now, there is (nearly) independent
consideration of each execution step, and entirely separate construction of
Paths to represent each of the possible ways to do that step. We choose
the best Path or set of Paths using the same add_path() logic that's been
used inside query_planner() for years.
In addition, this patch removes the old restriction that subquery_planner()
could return only a single Plan. It now returns a RelOptInfo containing a
set of Paths, just as query_planner() does, and the parent query level can
use each of those Paths as the basis of a SubqueryScanPath at its level.
This allows finding some optimizations that we missed before, wherein a
subquery was capable of returning presorted data and thereby avoiding a
sort in the parent level, making the overall cost cheaper even though
delivering sorted output was not the cheapest plan for the subquery in
isolation. (A couple of regression test outputs change in consequence of
that. However, there is very little change in visible planner behavior
overall, because the point of this patch is not to get immediate planning
benefits but to create the infrastructure for future improvements.)
There is a great deal left to do here. This patch unblocks a lot of
planner work that was basically impractical in the old code structure,
such as allowing FDWs to implement remote aggregation, or rewriting
plan_set_operations() to allow consideration of multiple implementation
orders for set operations. (The latter will likely require a full
rewrite of plan_set_operations(); what I've done here is only to fix it
to return Paths not Plans.) I have also left unfinished some localized
refactoring in createplan.c and planner.c, because it was not necessary
to get this patch to a working state.
Thanks to Robert Haas, David Rowley, and Amit Kapila for review.
2016-03-07 21:58:22 +01:00
|
|
|
that the foreign relation will be scanned in the usual way and then
|
2017-10-09 03:44:17 +02:00
|
|
|
individual row updates will be driven by a local <literal>ModifyTable</literal>
|
Make the upper part of the planner work by generating and comparing Paths.
I've been saying we needed to do this for more than five years, and here it
finally is. This patch removes the ever-growing tangle of spaghetti logic
that grouping_planner() used to use to try to identify the best plan for
post-scan/join query steps. Now, there is (nearly) independent
consideration of each execution step, and entirely separate construction of
Paths to represent each of the possible ways to do that step. We choose
the best Path or set of Paths using the same add_path() logic that's been
used inside query_planner() for years.
In addition, this patch removes the old restriction that subquery_planner()
could return only a single Plan. It now returns a RelOptInfo containing a
set of Paths, just as query_planner() does, and the parent query level can
use each of those Paths as the basis of a SubqueryScanPath at its level.
This allows finding some optimizations that we missed before, wherein a
subquery was capable of returning presorted data and thereby avoiding a
sort in the parent level, making the overall cost cheaper even though
delivering sorted output was not the cheapest plan for the subquery in
isolation. (A couple of regression test outputs change in consequence of
that. However, there is very little change in visible planner behavior
overall, because the point of this patch is not to get immediate planning
benefits but to create the infrastructure for future improvements.)
There is a great deal left to do here. This patch unblocks a lot of
planner work that was basically impractical in the old code structure,
such as allowing FDWs to implement remote aggregation, or rewriting
plan_set_operations() to allow consideration of multiple implementation
orders for set operations. (The latter will likely require a full
rewrite of plan_set_operations(); what I've done here is only to fix it
to return Paths not Plans.) I have also left unfinished some localized
refactoring in createplan.c and planner.c, because it was not necessary
to get this patch to a working state.
Thanks to Robert Haas, David Rowley, and Amit Kapila for review.
2016-03-07 21:58:22 +01:00
|
|
|
plan node. This approach is necessary for the general case where an
|
|
|
|
update requires reading local tables as well as foreign tables.
|
|
|
|
However, if the operation could be executed entirely by the foreign
|
|
|
|
server, the FDW could generate a path representing that and insert it
|
2017-10-09 03:44:17 +02:00
|
|
|
into the <literal>UPPERREL_FINAL</literal> upper relation, where it would
|
|
|
|
compete against the <literal>ModifyTable</literal> approach. This approach
|
|
|
|
could also be used to implement remote <literal>SELECT FOR UPDATE</literal>,
|
Make the upper part of the planner work by generating and comparing Paths.
I've been saying we needed to do this for more than five years, and here it
finally is. This patch removes the ever-growing tangle of spaghetti logic
that grouping_planner() used to use to try to identify the best plan for
post-scan/join query steps. Now, there is (nearly) independent
consideration of each execution step, and entirely separate construction of
Paths to represent each of the possible ways to do that step. We choose
the best Path or set of Paths using the same add_path() logic that's been
used inside query_planner() for years.
In addition, this patch removes the old restriction that subquery_planner()
could return only a single Plan. It now returns a RelOptInfo containing a
set of Paths, just as query_planner() does, and the parent query level can
use each of those Paths as the basis of a SubqueryScanPath at its level.
This allows finding some optimizations that we missed before, wherein a
subquery was capable of returning presorted data and thereby avoiding a
sort in the parent level, making the overall cost cheaper even though
delivering sorted output was not the cheapest plan for the subquery in
isolation. (A couple of regression test outputs change in consequence of
that. However, there is very little change in visible planner behavior
overall, because the point of this patch is not to get immediate planning
benefits but to create the infrastructure for future improvements.)
There is a great deal left to do here. This patch unblocks a lot of
planner work that was basically impractical in the old code structure,
such as allowing FDWs to implement remote aggregation, or rewriting
plan_set_operations() to allow consideration of multiple implementation
orders for set operations. (The latter will likely require a full
rewrite of plan_set_operations(); what I've done here is only to fix it
to return Paths not Plans.) I have also left unfinished some localized
refactoring in createplan.c and planner.c, because it was not necessary
to get this patch to a working state.
Thanks to Robert Haas, David Rowley, and Amit Kapila for review.
2016-03-07 21:58:22 +01:00
|
|
|
rather than using the row locking callbacks described in
|
2017-11-23 15:39:47 +01:00
|
|
|
<xref linkend="fdw-callbacks-row-locking"/>. Keep in mind that a path
|
2017-10-09 03:44:17 +02:00
|
|
|
inserted into <literal>UPPERREL_FINAL</literal> is responsible for
|
|
|
|
implementing <emphasis>all</emphasis> behavior of the query.
|
Make the upper part of the planner work by generating and comparing Paths.
I've been saying we needed to do this for more than five years, and here it
finally is. This patch removes the ever-growing tangle of spaghetti logic
that grouping_planner() used to use to try to identify the best plan for
post-scan/join query steps. Now, there is (nearly) independent
consideration of each execution step, and entirely separate construction of
Paths to represent each of the possible ways to do that step. We choose
the best Path or set of Paths using the same add_path() logic that's been
used inside query_planner() for years.
In addition, this patch removes the old restriction that subquery_planner()
could return only a single Plan. It now returns a RelOptInfo containing a
set of Paths, just as query_planner() does, and the parent query level can
use each of those Paths as the basis of a SubqueryScanPath at its level.
This allows finding some optimizations that we missed before, wherein a
subquery was capable of returning presorted data and thereby avoiding a
sort in the parent level, making the overall cost cheaper even though
delivering sorted output was not the cheapest plan for the subquery in
isolation. (A couple of regression test outputs change in consequence of
that. However, there is very little change in visible planner behavior
overall, because the point of this patch is not to get immediate planning
benefits but to create the infrastructure for future improvements.)
There is a great deal left to do here. This patch unblocks a lot of
planner work that was basically impractical in the old code structure,
such as allowing FDWs to implement remote aggregation, or rewriting
plan_set_operations() to allow consideration of multiple implementation
orders for set operations. (The latter will likely require a full
rewrite of plan_set_operations(); what I've done here is only to fix it
to return Paths not Plans.) I have also left unfinished some localized
refactoring in createplan.c and planner.c, because it was not necessary
to get this patch to a working state.
Thanks to Robert Haas, David Rowley, and Amit Kapila for review.
2016-03-07 21:58:22 +01:00
|
|
|
</para>
|
|
|
|
|
2013-03-10 19:14:53 +01:00
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
When planning an <command>UPDATE</command> or <command>DELETE</command>,
|
|
|
|
<function>PlanForeignModify</function> and <function>PlanDirectModify</function>
|
|
|
|
can look up the <structname>RelOptInfo</structname>
|
2013-03-10 19:14:53 +01:00
|
|
|
struct for the foreign table and make use of the
|
2017-10-09 03:44:17 +02:00
|
|
|
<literal>baserel->fdw_private</literal> data previously created by the
|
|
|
|
scan-planning functions. However, in <command>INSERT</command> the target
|
|
|
|
table is not scanned so there is no <structname>RelOptInfo</structname> for it.
|
|
|
|
The <structname>List</structname> returned by <function>PlanForeignModify</function> has
|
|
|
|
the same restrictions as the <structfield>fdw_private</structfield> list of a
|
|
|
|
<structname>ForeignScan</structname> plan node, that is it must contain only
|
|
|
|
structures that <function>copyObject</function> knows how to copy.
|
2013-03-10 19:14:53 +01:00
|
|
|
</para>
|
|
|
|
|
Add support for INSERT ... ON CONFLICT DO NOTHING/UPDATE.
The newly added ON CONFLICT clause allows to specify an alternative to
raising a unique or exclusion constraint violation error when inserting.
ON CONFLICT refers to constraints that can either be specified using a
inference clause (by specifying the columns of a unique constraint) or
by naming a unique or exclusion constraint. DO NOTHING avoids the
constraint violation, without touching the pre-existing row. DO UPDATE
SET ... [WHERE ...] updates the pre-existing tuple, and has access to
both the tuple proposed for insertion and the existing tuple; the
optional WHERE clause can be used to prevent an update from being
executed. The UPDATE SET and WHERE clauses have access to the tuple
proposed for insertion using the "magic" EXCLUDED alias, and to the
pre-existing tuple using the table name or its alias.
This feature is often referred to as upsert.
This is implemented using a new infrastructure called "speculative
insertion". It is an optimistic variant of regular insertion that first
does a pre-check for existing tuples and then attempts an insert. If a
violating tuple was inserted concurrently, the speculatively inserted
tuple is deleted and a new attempt is made. If the pre-check finds a
matching tuple the alternative DO NOTHING or DO UPDATE action is taken.
If the insertion succeeds without detecting a conflict, the tuple is
deemed inserted.
To handle the possible ambiguity between the excluded alias and a table
named excluded, and for convenience with long relation names, INSERT
INTO now can alias its target table.
Bumps catversion as stored rules change.
Author: Peter Geoghegan, with significant contributions from Heikki
Linnakangas and Andres Freund. Testing infrastructure by Jeff Janes.
Reviewed-By: Heikki Linnakangas, Andres Freund, Robert Haas, Simon Riggs,
Dean Rasheed, Stephen Frost and many others.
2015-05-08 05:31:36 +02:00
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
<command>INSERT</command> with an <literal>ON CONFLICT</literal> clause does not
|
2015-10-02 22:55:47 +02:00
|
|
|
support specifying the conflict target, as unique constraints or
|
|
|
|
exclusion constraints on remote tables are not locally known. This
|
2017-10-09 03:44:17 +02:00
|
|
|
in turn implies that <literal>ON CONFLICT DO UPDATE</literal> is not supported,
|
2015-10-02 22:55:47 +02:00
|
|
|
since the specification is mandatory there.
|
Add support for INSERT ... ON CONFLICT DO NOTHING/UPDATE.
The newly added ON CONFLICT clause allows to specify an alternative to
raising a unique or exclusion constraint violation error when inserting.
ON CONFLICT refers to constraints that can either be specified using a
inference clause (by specifying the columns of a unique constraint) or
by naming a unique or exclusion constraint. DO NOTHING avoids the
constraint violation, without touching the pre-existing row. DO UPDATE
SET ... [WHERE ...] updates the pre-existing tuple, and has access to
both the tuple proposed for insertion and the existing tuple; the
optional WHERE clause can be used to prevent an update from being
executed. The UPDATE SET and WHERE clauses have access to the tuple
proposed for insertion using the "magic" EXCLUDED alias, and to the
pre-existing tuple using the table name or its alias.
This feature is often referred to as upsert.
This is implemented using a new infrastructure called "speculative
insertion". It is an optimistic variant of regular insertion that first
does a pre-check for existing tuples and then attempts an insert. If a
violating tuple was inserted concurrently, the speculatively inserted
tuple is deleted and a new attempt is made. If the pre-check finds a
matching tuple the alternative DO NOTHING or DO UPDATE action is taken.
If the insertion succeeds without detecting a conflict, the tuple is
deemed inserted.
To handle the possible ambiguity between the excluded alias and a table
named excluded, and for convenience with long relation names, INSERT
INTO now can alias its target table.
Bumps catversion as stored rules change.
Author: Peter Geoghegan, with significant contributions from Heikki
Linnakangas and Andres Freund. Testing infrastructure by Jeff Janes.
Reviewed-By: Heikki Linnakangas, Andres Freund, Robert Haas, Simon Riggs,
Dean Rasheed, Stephen Frost and many others.
2015-05-08 05:31:36 +02:00
|
|
|
</para>
|
|
|
|
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
</sect1>
|
|
|
|
|
|
|
|
<sect1 id="fdw-row-locking">
|
|
|
|
<title>Row Locking in Foreign Data Wrappers</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
If an FDW's underlying storage mechanism has a concept of locking
|
|
|
|
individual rows to prevent concurrent updates of those rows, it is
|
|
|
|
usually worthwhile for the FDW to perform row-level locking with as
|
|
|
|
close an approximation as practical to the semantics used in
|
2017-10-09 03:44:17 +02:00
|
|
|
ordinary <productname>PostgreSQL</productname> tables. There are multiple
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
considerations involved in this.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
One key decision to be made is whether to perform <firstterm>early
|
2017-10-09 03:44:17 +02:00
|
|
|
locking</firstterm> or <firstterm>late locking</firstterm>. In early locking, a row is
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
locked when it is first retrieved from the underlying store, while in
|
|
|
|
late locking, the row is locked only when it is known that it needs to
|
|
|
|
be locked. (The difference arises because some rows may be discarded by
|
|
|
|
locally-checked restriction or join conditions.) Early locking is much
|
|
|
|
simpler and avoids extra round trips to a remote store, but it can cause
|
|
|
|
locking of rows that need not have been locked, resulting in reduced
|
|
|
|
concurrency or even unexpected deadlocks. Also, late locking is only
|
|
|
|
possible if the row to be locked can be uniquely re-identified later.
|
|
|
|
Preferably the row identifier should identify a specific version of the
|
2017-10-09 03:44:17 +02:00
|
|
|
row, as <productname>PostgreSQL</productname> TIDs do.
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
By default, <productname>PostgreSQL</productname> ignores locking considerations
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
when interfacing to FDWs, but an FDW can perform early locking without
|
|
|
|
any explicit support from the core code. The API functions described
|
2017-11-23 15:39:47 +01:00
|
|
|
in <xref linkend="fdw-callbacks-row-locking"/>, which were added
|
2017-10-09 03:44:17 +02:00
|
|
|
in <productname>PostgreSQL</productname> 9.5, allow an FDW to use late locking if
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
it wishes.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
An additional consideration is that in <literal>READ COMMITTED</literal>
|
|
|
|
isolation mode, <productname>PostgreSQL</productname> may need to re-check
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
restriction and join conditions against an updated version of some
|
|
|
|
target tuple. Rechecking join conditions requires re-obtaining copies
|
|
|
|
of the non-target rows that were previously joined to the target tuple.
|
2017-10-09 03:44:17 +02:00
|
|
|
When working with standard <productname>PostgreSQL</productname> tables, this is
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
done by including the TIDs of the non-target tables in the column list
|
|
|
|
projected through the join, and then re-fetching non-target rows when
|
|
|
|
required. This approach keeps the join data set compact, but it
|
|
|
|
requires inexpensive re-fetch capability, as well as a TID that can
|
|
|
|
uniquely identify the row version to be re-fetched. By default,
|
|
|
|
therefore, the approach used with foreign tables is to include a copy of
|
|
|
|
the entire row fetched from a foreign table in the column list projected
|
|
|
|
through the join. This puts no special demands on the FDW but can
|
|
|
|
result in reduced performance of merge and hash joins. An FDW that is
|
|
|
|
capable of meeting the re-fetch requirements can choose to do it the
|
|
|
|
first way.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
For an <command>UPDATE</command> or <command>DELETE</command> on a foreign table, it
|
|
|
|
is recommended that the <literal>ForeignScan</literal> operation on the target
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
table perform early locking on the rows that it fetches, perhaps via the
|
2017-10-09 03:44:17 +02:00
|
|
|
equivalent of <command>SELECT FOR UPDATE</command>. An FDW can detect whether
|
|
|
|
a table is an <command>UPDATE</command>/<command>DELETE</command> target at plan time
|
|
|
|
by comparing its relid to <literal>root->parse->resultRelation</literal>,
|
|
|
|
or at execution time by using <function>ExecRelationIsTargetRelation()</function>.
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
An alternative possibility is to perform late locking within the
|
2017-10-09 03:44:17 +02:00
|
|
|
<function>ExecForeignUpdate</function> or <function>ExecForeignDelete</function>
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
callback, but no special support is provided for this.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
For foreign tables that are specified to be locked by a <command>SELECT
|
2017-10-09 03:44:17 +02:00
|
|
|
FOR UPDATE/SHARE</command> command, the <literal>ForeignScan</literal> operation can
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
again perform early locking by fetching tuples with the equivalent
|
2017-10-09 03:44:17 +02:00
|
|
|
of <command>SELECT FOR UPDATE/SHARE</command>. To perform late locking
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
instead, provide the callback functions defined
|
2017-11-23 15:39:47 +01:00
|
|
|
in <xref linkend="fdw-callbacks-row-locking"/>.
|
2017-10-09 03:44:17 +02:00
|
|
|
In <function>GetForeignRowMarkType</function>, select rowmark option
|
|
|
|
<literal>ROW_MARK_EXCLUSIVE</literal>, <literal>ROW_MARK_NOKEYEXCLUSIVE</literal>,
|
|
|
|
<literal>ROW_MARK_SHARE</literal>, or <literal>ROW_MARK_KEYSHARE</literal> depending
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
on the requested lock strength. (The core code will act the same
|
|
|
|
regardless of which of these four options you choose.)
|
|
|
|
Elsewhere, you can detect whether a foreign table was specified to be
|
2017-10-09 03:44:17 +02:00
|
|
|
locked by this type of command by using <function>get_plan_rowmark</function> at
|
|
|
|
plan time, or <function>ExecFindRowMark</function> at execution time; you must
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
check not only whether a non-null rowmark struct is returned, but that
|
2017-10-09 03:44:17 +02:00
|
|
|
its <structfield>strength</structfield> field is not <literal>LCS_NONE</literal>.
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
Lastly, for foreign tables that are used in an <command>UPDATE</command>,
|
|
|
|
<command>DELETE</command> or <command>SELECT FOR UPDATE/SHARE</command> command but
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
are not specified to be row-locked, you can override the default choice
|
2017-10-09 03:44:17 +02:00
|
|
|
to copy entire rows by having <function>GetForeignRowMarkType</function> select
|
|
|
|
option <literal>ROW_MARK_REFERENCE</literal> when it sees lock strength
|
|
|
|
<literal>LCS_NONE</literal>. This will cause <function>RefetchForeignRow</function> to
|
|
|
|
be called with that value for <structfield>markType</structfield>; it should then
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
re-fetch the row without acquiring any new lock. (If you have
|
2017-10-09 03:44:17 +02:00
|
|
|
a <function>GetForeignRowMarkType</function> function but don't wish to re-fetch
|
|
|
|
unlocked rows, select option <literal>ROW_MARK_COPY</literal>
|
|
|
|
for <literal>LCS_NONE</literal>.)
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2017-10-09 03:44:17 +02:00
|
|
|
See <filename>src/include/nodes/lockoptions.h</filename>, the comments
|
|
|
|
for <type>RowMarkType</type> and <type>PlanRowMark</type>
|
|
|
|
in <filename>src/include/nodes/plannodes.h</filename>, and the comments for
|
|
|
|
<type>ExecRowMark</type> in <filename>src/include/nodes/execnodes.h</filename> for
|
Add support for doing late row locking in FDWs.
Previously, FDWs could only do "early row locking", that is lock a row as
soon as it's fetched, even though local restriction/join conditions might
discard the row later. This patch adds callbacks that allow FDWs to do
late locking in the same way that it's done for regular tables.
To make use of this feature, an FDW must support the "ctid" column as a
unique row identifier. Currently, since ctid has to be of type TID,
the feature is of limited use, though in principle it could be used by
postgres_fdw. We may eventually allow FDWs to specify another data type
for ctid, which would make it possible for more FDWs to use this feature.
This commit does not modify postgres_fdw to use late locking. We've
tested some prototype code for that, but it's not in committable shape,
and besides it's quite unclear whether it actually makes sense to do late
locking against a remote server. The extra round trips required are likely
to outweigh any benefit from improved concurrency.
Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me
2015-05-12 20:10:10 +02:00
|
|
|
additional information.
|
|
|
|
</para>
|
|
|
|
|
2012-03-08 00:20:58 +01:00
|
|
|
</sect1>
|
|
|
|
|
2011-02-20 06:17:18 +01:00
|
|
|
</chapter>
|