Writing A Foreign Data Wrapper
foreign data wrapper
handler for
All operations on a foreign table are handled through its foreign data
wrapper, which consists of a set of functions that the core server
calls. The foreign data wrapper is responsible for fetching
data from the remote data source and returning it to the
PostgreSQL executor. If updating foreign
tables is to be supported, the wrapper must handle that, too.
This chapter outlines how to write a new foreign data wrapper.
The foreign data wrappers included in the standard distribution are good
references when trying to write your own. Look into the
contrib> subdirectory of the source tree.
The reference page also has
some useful details.
The SQL standard specifies an interface for writing foreign data wrappers.
However, PostgreSQL does not implement that API, because the effort to
accommodate it into PostgreSQL would be large, and the standard API hasn't
gained wide adoption anyway.
Foreign Data Wrapper Functions
The FDW author needs to implement a handler function, and optionally
a validator function. Both functions must be written in a compiled
language such as C, using the version-1 interface.
For details on C language calling conventions and dynamic loading,
see .
The handler function simply returns a struct of function pointers to
callback functions that will be called by the planner, executor, and
various maintenance commands.
Most of the effort in writing an FDW is in implementing these callback
functions.
The handler function must be registered with
PostgreSQL as taking no arguments and
returning the special pseudo-type fdw_handler. The
callback functions are plain C functions and are not visible or
callable at the SQL level. The callback functions are described in
.
The validator function is responsible for validating options given in
CREATE and ALTER commands for its
foreign data wrapper, as well as foreign servers, user mappings, and
foreign tables using the wrapper.
The validator function must be registered as taking two arguments, a
text array containing the options to be validated, and an OID
representing the type of object the options are associated with (in
the form of the OID of the system catalog the object would be stored
in, either
ForeignDataWrapperRelationId>,
ForeignServerRelationId>,
UserMappingRelationId>,
or ForeignTableRelationId>).
If no validator function is supplied, options are not checked at object
creation time or object alteration time.
Foreign Data Wrapper Callback Routines
The FDW handler function returns a palloc'd FdwRoutine>
struct containing pointers to the callback functions described below.
The scan-related functions are required, the rest are optional.
The FdwRoutine> struct type is declared in
src/include/foreign/fdwapi.h>, which see for additional
details.
FDW Routines For Scanning Foreign Tables
void
GetForeignRelSize (PlannerInfo *root,
RelOptInfo *baserel,
Oid foreigntableid);
Obtain relation size estimates for a foreign table. This is called
at the beginning of planning for a query that scans a foreign table.
root> is the planner's global information about the query;
baserel> is the planner's information about this table; and
foreigntableid> is the pg_class> OID of the
foreign table. (foreigntableid> could be obtained from the
planner data structures, but it's passed explicitly to save effort.)
This function should update baserel->rows> to be the
expected number of rows returned by the table scan, after accounting for
the filtering done by the restriction quals. The initial value of
baserel->rows> is just a constant default estimate, which
should be replaced if at all possible. The function may also choose to
update baserel->width> if it can compute a better estimate
of the average result row width.
See for additional information.
void
GetForeignPaths (PlannerInfo *root,
RelOptInfo *baserel,
Oid foreigntableid);
Create possible access paths for a scan on a foreign table.
This is called during query planning.
The parameters are the same as for GetForeignRelSize>,
which has already been called.
This function must generate at least one access path
(ForeignPath> node) for a scan on the foreign table and
must call add_path> to add each such path to
baserel->pathlist>. It's recommended to use
create_foreignscan_path> to build the
ForeignPath> nodes. The function can generate multiple
access paths, e.g., a path which has valid pathkeys> to
represent a pre-sorted result. Each access path must contain cost
estimates, and can contain any FDW-private information that is needed to
identify the specific scan method intended.
See for additional information.
ForeignScan *
GetForeignPlan (PlannerInfo *root,
RelOptInfo *baserel,
Oid foreigntableid,
ForeignPath *best_path,
List *tlist,
List *scan_clauses,
Plan *outer_plan);
Create a ForeignScan> plan node from the selected foreign
access path. This is called at the end of query planning.
The parameters are as for GetForeignRelSize>, plus
the selected ForeignPath> (previously produced by
GetForeignPaths>, GetForeignJoinPaths>,
or GetForeignUpperPaths>),
the target list to be emitted by the plan node,
the restriction clauses to be enforced by the plan node,
and the outer subplan of the ForeignScan>,
which is used for rechecks performed by RecheckForeignScan>.
(If the path is for a join rather than a base
relation, foreigntableid> is InvalidOid>.)
This function must create and return a ForeignScan> plan
node; it's recommended to use make_foreignscan> to build the
ForeignScan> node.
See for additional information.
void
BeginForeignScan (ForeignScanState *node,
int eflags);
Begin executing a foreign scan. This is called during executor startup.
It should perform any initialization needed before the scan can start,
but not start executing the actual scan (that should be done upon the
first call to IterateForeignScan>).
The ForeignScanState> node has already been created, but
its fdw_state> field is still NULL. Information about
the table to scan is accessible through the
ForeignScanState> node (in particular, from the underlying
ForeignScan> plan node, which contains any FDW-private
information provided by GetForeignPlan>).
eflags> contains flag bits describing the executor's
operating mode for this plan node.
Note that when (eflags & EXEC_FLAG_EXPLAIN_ONLY)> is
true, this function should not perform any externally-visible actions;
it should only do the minimum required to make the node state valid
for ExplainForeignScan> and EndForeignScan>.
TupleTableSlot *
IterateForeignScan (ForeignScanState *node);
Fetch one row from the foreign source, returning it in a tuple table slot
(the node's ScanTupleSlot> should be used for this
purpose). Return NULL if no more rows are available. The tuple table
slot infrastructure allows either a physical or virtual tuple to be
returned; in most cases the latter choice is preferable from a
performance standpoint. Note that this is called in a short-lived memory
context that will be reset between invocations. Create a memory context
in BeginForeignScan> if you need longer-lived storage, or use
the es_query_cxt> of the node's EState>.
The rows returned must match the fdw_scan_tlist> target
list if one was supplied, otherwise they must match the row type of the
foreign table being scanned. If you choose to optimize away fetching
columns that are not needed, you should insert nulls in those column
positions, or else generate a fdw_scan_tlist> list with
those columns omitted.
Note that PostgreSQL's executor doesn't care
whether the rows returned violate any constraints that were defined on
the foreign table — but the planner does care, and may optimize
queries incorrectly if there are rows visible in the foreign table that
do not satisfy a declared constraint. If a constraint is violated when
the user has declared that the constraint should hold true, it may be
appropriate to raise an error (just as you would need to do in the case
of a data type mismatch).
void
ReScanForeignScan (ForeignScanState *node);
Restart the scan from the beginning. Note that any parameters the
scan depends on may have changed value, so the new scan does not
necessarily return exactly the same rows.
void
EndForeignScan (ForeignScanState *node);
End the scan and release resources. It is normally not important
to release palloc'd memory, but for example open files and connections
to remote servers should be cleaned up.
FDW Routines For Scanning Foreign Joins
If an FDW supports performing foreign joins remotely (rather than
by fetching both tables' data and doing the join locally), it should
provide this callback function:
void
GetForeignJoinPaths (PlannerInfo *root,
RelOptInfo *joinrel,
RelOptInfo *outerrel,
RelOptInfo *innerrel,
JoinType jointype,
JoinPathExtraData *extra);
Create possible access paths for a join of two (or more) foreign tables
that all belong to the same foreign server. This optional
function is called during query planning. As
with GetForeignPaths>, this function should
generate ForeignPath> path(s) for the
supplied joinrel>, and call add_path> to add these
paths to the set of paths considered for the join. But unlike
GetForeignPaths>, it is not necessary that this function
succeed in creating at least one path, since paths involving local
joining are always possible.
Note that this function will be invoked repeatedly for the same join
relation, with different combinations of inner and outer relations; it is
the responsibility of the FDW to minimize duplicated work.
If a ForeignPath> path is chosen for the join, it will
represent the entire join process; paths generated for the component
tables and subsidiary joins will not be used. Subsequent processing of
the join path proceeds much as it does for a path scanning a single
foreign table. One difference is that the scanrelid> of
the resulting ForeignScan> plan node should be set to zero,
since there is no single relation that it represents; instead,
the fs_relids> field of the ForeignScan>
node represents the set of relations that were joined. (The latter field
is set up automatically by the core planner code, and need not be filled
by the FDW.) Another difference is that, because the column list for a
remote join cannot be found from the system catalogs, the FDW must
fill fdw_scan_tlist> with an appropriate list
of TargetEntry> nodes, representing the set of columns
it will supply at run time in the tuples it returns.
See for additional information.
FDW Routines For Planning Post-Scan/Join Processing
If an FDW supports performing remote post-scan/join processing, such as
remote aggregation, it should provide this callback function:
void
GetForeignUpperPaths (PlannerInfo *root,
RelOptInfo *scan_join_rel);
Create possible access paths for upper relation> processing,
which is the planner's term for all post-scan/join query processing, such
as aggregation, window functions, sorting, and table updates. This
optional function is called during query planning. Currently, it is
called only if all base relation(s) involved in the query belong to the
same FDW. This function should generate ForeignPath>
path(s) for the steps that the FDW knows how to perform remotely, and
call add_path> to add these paths to the appropriate upper
relation. As with GetForeignJoinPaths>, it is not necessary
that this function succeed in creating any paths, since paths involving
local processing are always possible.
See for additional information.
FDW Routines For Updating Foreign Tables
If an FDW supports writable foreign tables, it should provide
some or all of the following callback functions depending on
the needs and capabilities of the FDW:
void
AddForeignUpdateTargets (Query *parsetree,
RangeTblEntry *target_rte,
Relation target_relation);
UPDATE> and DELETE> operations are performed
against rows previously fetched by the table-scanning functions. The
FDW may need extra information, such as a row ID or the values of
primary-key columns, to ensure that it can identify the exact row to
update or delete. To support that, this function can add extra hidden,
or junk>, target columns to the list of columns that are to be
retrieved from the foreign table during an UPDATE> or
DELETE>.
To do that, add TargetEntry> items to
parsetree->targetList>, containing expressions for the
extra values to be fetched. Each such entry must be marked
resjunk> = true>, and must have a distinct
resname> that will identify it at execution time.
Avoid using names matching ctidN>,
wholerow, or
wholerowN>, as the core system can
generate junk columns of these names.
This function is called in the rewriter, not the planner, so the
information available is a bit different from that available to the
planning routines.
parsetree> is the parse tree for the UPDATE> or
DELETE> command, while target_rte> and
target_relation> describe the target foreign table.
If the AddForeignUpdateTargets> pointer is set to
NULL>, no extra target expressions are added.
(This will make it impossible to implement DELETE>
operations, though UPDATE> may still be feasible if the FDW
relies on an unchanging primary key to identify rows.)
List *
PlanForeignModify (PlannerInfo *root,
ModifyTable *plan,
Index resultRelation,
int subplan_index);
Perform any additional planning actions needed for an insert, update, or
delete on a foreign table. This function generates the FDW-private
information that will be attached to the ModifyTable> plan
node that performs the update action. This private information must
have the form of a List>, and will be delivered to
BeginForeignModify> during the execution stage.
root> is the planner's global information about the query.
plan> is the ModifyTable> plan node, which is
complete except for the fdwPrivLists> field.
resultRelation> identifies the target foreign table by its
range table index. subplan_index> identifies which target of
the ModifyTable> plan node this is, counting from zero;
use this if you want to index into plan->plans> or other
substructure of the plan> node.
See for additional information.
If the PlanForeignModify> pointer is set to
NULL>, no additional plan-time actions are taken, and the
fdw_private> list delivered to
BeginForeignModify> will be NIL.
void
BeginForeignModify (ModifyTableState *mtstate,
ResultRelInfo *rinfo,
List *fdw_private,
int subplan_index,
int eflags);
Begin executing a foreign table modification operation. This routine is
called during executor startup. It should perform any initialization
needed prior to the actual table modifications. Subsequently,
ExecForeignInsert>, ExecForeignUpdate> or
ExecForeignDelete> will be called for each tuple to be
inserted, updated, or deleted.
mtstate> is the overall state of the
ModifyTable> plan node being executed; global data about
the plan and execution state is available via this structure.
rinfo> is the ResultRelInfo> struct describing
the target foreign table. (The ri_FdwState> field of
ResultRelInfo> is available for the FDW to store any
private state it needs for this operation.)
fdw_private> contains the private data generated by
PlanForeignModify>, if any.
subplan_index> identifies which target of
the ModifyTable> plan node this is.
eflags> contains flag bits describing the executor's
operating mode for this plan node.
Note that when (eflags & EXEC_FLAG_EXPLAIN_ONLY)> is
true, this function should not perform any externally-visible actions;
it should only do the minimum required to make the node state valid
for ExplainForeignModify> and EndForeignModify>.
If the BeginForeignModify> pointer is set to
NULL>, no action is taken during executor startup.
TupleTableSlot *
ExecForeignInsert (EState *estate,
ResultRelInfo *rinfo,
TupleTableSlot *slot,
TupleTableSlot *planSlot);
Insert one tuple into the foreign table.
estate> is global execution state for the query.
rinfo> is the ResultRelInfo> struct describing
the target foreign table.
slot> contains the tuple to be inserted; it will match the
row-type definition of the foreign table.
planSlot> contains the tuple that was generated by the
ModifyTable> plan node's subplan; it differs from
slot> in possibly containing additional junk>
columns. (The planSlot> is typically of little interest
for INSERT> cases, but is provided for completeness.)
The return value is either a slot containing the data that was actually
inserted (this might differ from the data supplied, for example as a
result of trigger actions), or NULL if no row was actually inserted
(again, typically as a result of triggers). The passed-in
slot> can be re-used for this purpose.
The data in the returned slot is used only if the INSERT>
query has a RETURNING> clause or the foreign table has
an AFTER ROW> trigger. Triggers require all columns, but the
FDW could choose to optimize away returning some or all columns depending
on the contents of the RETURNING> clause. Regardless, some
slot must be returned to indicate success, or the query's reported row
count will be wrong.
If the ExecForeignInsert> pointer is set to
NULL>, attempts to insert into the foreign table will fail
with an error message.
TupleTableSlot *
ExecForeignUpdate (EState *estate,
ResultRelInfo *rinfo,
TupleTableSlot *slot,
TupleTableSlot *planSlot);
Update one tuple in the foreign table.
estate> is global execution state for the query.
rinfo> is the ResultRelInfo> struct describing
the target foreign table.
slot> contains the new data for the tuple; it will match the
row-type definition of the foreign table.
planSlot> contains the tuple that was generated by the
ModifyTable> plan node's subplan; it differs from
slot> in possibly containing additional junk>
columns. In particular, any junk columns that were requested by
AddForeignUpdateTargets> will be available from this slot.
The return value is either a slot containing the row as it was actually
updated (this might differ from the data supplied, for example as a
result of trigger actions), or NULL if no row was actually updated
(again, typically as a result of triggers). The passed-in
slot> can be re-used for this purpose.
The data in the returned slot is used only if the UPDATE>
query has a RETURNING> clause or the foreign table has
an AFTER ROW> trigger. Triggers require all columns, but the
FDW could choose to optimize away returning some or all columns depending
on the contents of the RETURNING> clause. Regardless, some
slot must be returned to indicate success, or the query's reported row
count will be wrong.
If the ExecForeignUpdate> pointer is set to
NULL>, attempts to update the foreign table will fail
with an error message.
TupleTableSlot *
ExecForeignDelete (EState *estate,
ResultRelInfo *rinfo,
TupleTableSlot *slot,
TupleTableSlot *planSlot);
Delete one tuple from the foreign table.
estate> is global execution state for the query.
rinfo> is the ResultRelInfo> struct describing
the target foreign table.
slot> contains nothing useful upon call, but can be used to
hold the returned tuple.
planSlot> contains the tuple that was generated by the
ModifyTable> plan node's subplan; in particular, it will
carry any junk columns that were requested by
AddForeignUpdateTargets>. The junk column(s) must be used
to identify the tuple to be deleted.
The return value is either a slot containing the row that was deleted,
or NULL if no row was deleted (typically as a result of triggers). The
passed-in slot> can be used to hold the tuple to be returned.
The data in the returned slot is used only if the DELETE>
query has a RETURNING> clause or the foreign table has
an AFTER ROW> trigger. Triggers require all columns, but the
FDW could choose to optimize away returning some or all columns depending
on the contents of the RETURNING> clause. Regardless, some
slot must be returned to indicate success, or the query's reported row
count will be wrong.
If the ExecForeignDelete> pointer is set to
NULL>, attempts to delete from the foreign table will fail
with an error message.
void
EndForeignModify (EState *estate,
ResultRelInfo *rinfo);
End the table update and release resources. It is normally not important
to release palloc'd memory, but for example open files and connections
to remote servers should be cleaned up.
If the EndForeignModify> pointer is set to
NULL>, no action is taken during executor shutdown.
int
IsForeignRelUpdatable (Relation rel);
Report which update operations the specified foreign table supports.
The return value should be a bit mask of rule event numbers indicating
which operations are supported by the foreign table, using the
CmdType> enumeration; that is,
(1 << CMD_UPDATE) = 4> for UPDATE>,
(1 << CMD_INSERT) = 8> for INSERT>, and
(1 << CMD_DELETE) = 16> for DELETE>.
If the IsForeignRelUpdatable> pointer is set to
NULL>, foreign tables are assumed to be insertable, updatable,
or deletable if the FDW provides ExecForeignInsert>,
ExecForeignUpdate>, or ExecForeignDelete>
respectively. This function is only needed if the FDW supports some
tables that are updatable and some that are not. (Even then, it's
permissible to throw an error in the execution routine instead of
checking in this function. However, this function is used to determine
updatability for display in the information_schema> views.)
FDW Routines For Row Locking
If an FDW wishes to support late row locking> (as described
in ), it must provide the following
callback functions:
RowMarkType
GetForeignRowMarkType (RangeTblEntry *rte,
LockClauseStrength strength);
Report which row-marking option to use for a foreign table.
rte> is the RangeTblEntry> node for the table
and strength> describes the lock strength requested by the
relevant FOR UPDATE/SHARE> clause, if any. The result must be
a member of the RowMarkType> enum type.
This function is called during query planning for each foreign table that
appears in an UPDATE>, DELETE>, or SELECT
FOR UPDATE/SHARE> query and is not the target of UPDATE>
or DELETE>.
If the GetForeignRowMarkType> pointer is set to
NULL>, the ROW_MARK_COPY> option is always used.
(This implies that RefetchForeignRow> will never be called,
so it need not be provided either.)
See for more information.
HeapTuple
RefetchForeignRow (EState *estate,
ExecRowMark *erm,
Datum rowid,
bool *updated);
Re-fetch one tuple from the foreign table, after locking it if required.
estate> is global execution state for the query.
erm> is the ExecRowMark> struct describing
the target foreign table and the row lock type (if any) to acquire.
rowid> identifies the tuple to be fetched.
updated> is an output parameter.
This function should return a palloc'ed copy of the fetched tuple,
or NULL> if the row lock couldn't be obtained. The row lock
type to acquire is defined by erm->markType>, which is the
value previously returned by GetForeignRowMarkType>.
(ROW_MARK_REFERENCE> means to just re-fetch the tuple without
acquiring any lock, and ROW_MARK_COPY> will never be seen by
this routine.)
In addition, *updated> should be set to true>
if what was fetched was an updated version of the tuple rather than
the same version previously obtained. (If the FDW cannot be sure about
this, always returning true> is recommended.)
Note that by default, failure to acquire a row lock should result in
raising an error; a NULL> return is only appropriate if
the SKIP LOCKED> option is specified
by erm->waitPolicy>.
The rowid> is the ctid> value previously read
for the row to be re-fetched. Although the rowid> value is
passed as a Datum>, it can currently only be a tid>. The
function API is chosen in hopes that it may be possible to allow other
data types for row IDs in future.
If the RefetchForeignRow> pointer is set to
NULL>, attempts to re-fetch rows will fail
with an error message.
See for more information.
bool
RecheckForeignScan (ForeignScanState *node, TupleTableSlot *slot);
Recheck that a previously-returned tuple still matches the relevant
scan and join qualifiers, and possibly provide a modified version of
the tuple. For foreign data wrappers which do not perform join pushdown,
it will typically be more convenient to set this to NULL> and
instead set fdw_recheck_quals appropriately.
When outer joins are pushed down, however, it isn't sufficient to
reapply the checks relevant to all the base tables to the result tuple,
even if all needed attributes are present, because failure to match some
qualifier might result in some attributes going to NULL, rather than in
no tuple being returned. RecheckForeignScan> can recheck
qualifiers and return true if they are still satisfied and false
otherwise, but it can also store a replacement tuple into the supplied
slot.
To implement join pushdown, a foreign data wrapper will typically
construct an alternative local join plan which is used only for
rechecks; this will become the outer subplan of the
ForeignScan>. When a recheck is required, this subplan
can be executed and the resulting tuple can be stored in the slot.
This plan need not be efficient since no base table will return more
than one row; for example, it may implement all joins as nested loops.
The function GetExistingLocalJoinPath> may be used to search
existing paths for a suitable local join path, which can be used as the
alternative local join plan. GetExistingLocalJoinPath>
searches for an unparameterized path in the path list of the specified
join relation. (If it does not find such a path, it returns NULL, in
which case a foreign data wrapper may build the local path by itself or
may choose not to create access paths for that join.)
FDW Routines for EXPLAIN>
void
ExplainForeignScan (ForeignScanState *node,
ExplainState *es);
Print additional EXPLAIN> output for a foreign table scan.
This function can call ExplainPropertyText> and
related functions to add fields to the EXPLAIN> output.
The flag fields in es> can be used to determine what to
print, and the state of the ForeignScanState> node
can be inspected to provide run-time statistics in the EXPLAIN
ANALYZE> case.
If the ExplainForeignScan> pointer is set to
NULL>, no additional information is printed during
EXPLAIN>.
void
ExplainForeignModify (ModifyTableState *mtstate,
ResultRelInfo *rinfo,
List *fdw_private,
int subplan_index,
struct ExplainState *es);
Print additional EXPLAIN> output for a foreign table update.
This function can call ExplainPropertyText> and
related functions to add fields to the EXPLAIN> output.
The flag fields in es> can be used to determine what to
print, and the state of the ModifyTableState> node
can be inspected to provide run-time statistics in the EXPLAIN
ANALYZE> case. The first four arguments are the same as for
BeginForeignModify>.
If the ExplainForeignModify> pointer is set to
NULL>, no additional information is printed during
EXPLAIN>.
FDW Routines for ANALYZE>
bool
AnalyzeForeignTable (Relation relation,
AcquireSampleRowsFunc *func,
BlockNumber *totalpages);
This function is called when is executed on
a foreign table. If the FDW can collect statistics for this
foreign table, it should return true>, and provide a pointer
to a function that will collect sample rows from the table in
func>, plus the estimated size of the table in pages in
totalpages>. Otherwise, return false>.
If the FDW does not support collecting statistics for any tables, the
AnalyzeForeignTable> pointer can be set to NULL>.
If provided, the sample collection function must have the signature
int
AcquireSampleRowsFunc (Relation relation, int elevel,
HeapTuple *rows, int targrows,
double *totalrows,
double *totaldeadrows);
A random sample of up to targrows> rows should be collected
from the table and stored into the caller-provided rows>
array. The actual number of rows collected must be returned. In
addition, store estimates of the total numbers of live and dead rows in
the table into the output parameters totalrows> and
totaldeadrows>. (Set totaldeadrows> to zero
if the FDW does not have any concept of dead rows.)
FDW Routines For IMPORT FOREIGN SCHEMA>
List *
ImportForeignSchema (ImportForeignSchemaStmt *stmt, Oid serverOid);
Obtain a list of foreign table creation commands. This function is
called when executing , and is
passed the parse tree for that statement, as well as the OID of the
foreign server to use. It should return a list of C strings, each of
which must contain a command.
These strings will be parsed and executed by the core server.
Within the ImportForeignSchemaStmt> struct,
remote_schema> is the name of the remote schema from
which tables are to be imported.
list_type> identifies how to filter table names:
FDW_IMPORT_SCHEMA_ALL> means that all tables in the remote
schema should be imported (in this case table_list> is
empty), FDW_IMPORT_SCHEMA_LIMIT_TO> means to include only
tables listed in table_list>,
and FDW_IMPORT_SCHEMA_EXCEPT> means to exclude the tables
listed in table_list>.
options> is a list of options used for the import process.
The meanings of the options are up to the FDW.
For example, an FDW could use an option to define whether the
NOT NULL> attributes of columns should be imported.
These options need not have anything to do with those supported by the
FDW as database object options.
The FDW may ignore the local_schema> field of
the ImportForeignSchemaStmt>, because the core server
will automatically insert that name into the parsed CREATE
FOREIGN TABLE> commands.
The FDW does not have to concern itself with implementing the filtering
specified by list_type> and table_list>,
either, as the core server will automatically skip any returned commands
for tables excluded according to those options. However, it's often
useful to avoid the work of creating commands for excluded tables in the
first place. The function IsImportableForeignTable()> may be
useful to test whether a given foreign-table name will pass the filter.
If the FDW does not support importing table definitions, the
ImportForeignSchema> pointer can be set to NULL>.
FDW Routines for Parallel Execution
A ForeignScan> node can, optionally, support parallel
execution. A parallel ForeignScan> will be executed
in multiple processes and should return each row only once across
all cooperating processes. To do this, processes can coordinate through
fixed size chunks of dynamic shared memory. This shared memory is not
guaranteed to be mapped at the same address in every process, so pointers
may not be used. The following callbacks are all optional in general,
but required if parallel execution is to be supported.
Size
IsForeignScanParallelSafe(PlannerInfo *root, RelOptInfo *rel,
RangeTblEntry *rte);
Test whether a scan can be performed within a parallel worker. This
function will only be called when the planner believes that a parallel
plan might be possible, and should return true if it is safe for that scan
to run within a parallel worker. This will generally not be the case if
the remote data source has transaction semantics, unless the worker's
connection to the data can somehow be made to share the same transaction
context as the leader.
If this callback is not defined, it is assumed that the scan must take
place within the parallel leader. Note that returning true does not mean
that the scan itself can be done in parallel, only that the scan can be
performed within a parallel worker. Therefore, it can be useful to define
this method even when parallel execution is not supported.
Size
EstimateDSMForeignScan(ForeignScanState *node, ParallelContext *pcxt);
Estimate the amount of dynamic shared memory that will be required
for parallel operation. This may be higher than the amount that will
actually be used, but it must not be lower. The return value is in bytes.
void
InitializeDSMForeignScan(ForeignScanState *node, ParallelContext *pcxt,
void *coordinate);
Initialize the dynamic shared memory that will be required for parallel
operation; coordinate> points to an amount of allocated space
equal to the return value of EstimateDSMForeignScan>.
void
InitializeWorkerForeignScan(ForeignScanState *node, shm_toc *toc,
void *coordinate);
Initialize a parallel worker's custom state based on the shared state
set up in the leader by InitializeDSMForeignScan>.
This callback is optional, and needs only be supplied if this
custom path supports parallel execution.
Foreign Data Wrapper Helper Functions
Several helper functions are exported from the core server so that
authors of foreign data wrappers can get easy access to attributes of
FDW-related objects, such as FDW options.
To use any of these functions, you need to include the header file
foreign/foreign.h in your source file.
That header also defines the struct types that are returned by
these functions.
ForeignDataWrapper *
GetForeignDataWrapper(Oid fdwid);
This function returns a ForeignDataWrapper
object for the foreign-data wrapper with the given OID. A
ForeignDataWrapper object contains properties
of the FDW (see foreign/foreign.h for details).
ForeignServer *
GetForeignServer(Oid serverid);
This function returns a ForeignServer object
for the foreign server with the given OID. A
ForeignServer object contains properties
of the server (see foreign/foreign.h for details).
UserMapping *
GetUserMapping(Oid userid, Oid serverid);
This function returns a UserMapping object for
the user mapping of the given role on the given server. (If there is no
mapping for the specific user, it will return the mapping for
PUBLIC>, or throw error if there is none.) A
UserMapping object contains properties of the
user mapping (see foreign/foreign.h for details).
ForeignTable *
GetForeignTable(Oid relid);
This function returns a ForeignTable object for
the foreign table with the given OID. A
ForeignTable object contains properties of the
foreign table (see foreign/foreign.h for details).
UserMapping *
GetUserMappingById(Oid umid);
This function returns the UserMapping object for
the given user mapping OID. The OID of a user mapping for a foreign scan
is available in the RelOptInfo.
If there is no mapping for the OID, this function will throw an error.
A UserMapping object contains properties of the
user mapping (see foreign/foreign.h for details).
List *
GetForeignColumnOptions(Oid relid, AttrNumber attnum);
This function returns the per-column FDW options for the column with the
given foreign table OID and attribute number, in the form of a list of
DefElem. NIL is returned if the column has no
options.
Some object types have name-based lookup functions in addition to the
OID-based ones:
ForeignDataWrapper *
GetForeignDataWrapperByName(const char *name, bool missing_ok);
This function returns a ForeignDataWrapper
object for the foreign-data wrapper with the given name. If the wrapper
is not found, return NULL if missing_ok is true, otherwise raise an
error.
ForeignServer *
GetForeignServerByName(const char *name, bool missing_ok);
This function returns a ForeignServer object
for the foreign server with the given name. If the server is not found,
return NULL if missing_ok is true, otherwise raise an error.
Foreign Data Wrapper Query Planning
The FDW callback functions GetForeignRelSize>,
GetForeignPaths>, GetForeignPlan>,
PlanForeignModify>, GetForeignJoinPaths>,
and GetForeignUpperPaths>
must fit into the workings of the PostgreSQL> planner.
Here are some notes about what they must do.
The information in root> and baserel> can be used
to reduce the amount of information that has to be fetched from the
foreign table (and therefore reduce the cost).
baserel->baserestrictinfo> is particularly interesting, as
it contains restriction quals (WHERE> clauses) that should be
used to filter the rows to be fetched. (The FDW itself is not required
to enforce these quals, as the core executor can check them instead.)
baserel->reltarget->exprs> can be used to determine which
columns need to be fetched; but note that it only lists columns that
have to be emitted by the ForeignScan> plan node, not
columns that are used in qual evaluation but not output by the query.
Various private fields are available for the FDW planning functions to
keep information in. Generally, whatever you store in FDW private fields
should be palloc'd, so that it will be reclaimed at the end of planning.
baserel->fdw_private> is a void> pointer that is
available for FDW planning functions to store information relevant to
the particular foreign table. The core planner does not touch it except
to initialize it to NULL when the RelOptInfo> node is created.
It is useful for passing information forward from
GetForeignRelSize> to GetForeignPaths> and/or
GetForeignPaths> to GetForeignPlan>, thereby
avoiding recalculation.
GetForeignPaths> can identify the meaning of different
access paths by storing private information in the
fdw_private> field of ForeignPath> nodes.
fdw_private> is declared as a List> pointer, but
could actually contain anything since the core planner does not touch
it. However, best practice is to use a representation that's dumpable
by nodeToString>, for use with debugging support available
in the backend.
GetForeignPlan> can examine the fdw_private>
field of the selected ForeignPath> node, and can generate
fdw_exprs> and fdw_private> lists to be
placed in the ForeignScan> plan node, where they will be
available at execution time. Both of these lists must be
represented in a form that copyObject> knows how to copy.
The fdw_private> list has no other restrictions and is
not interpreted by the core backend in any way. The
fdw_exprs> list, if not NIL, is expected to contain
expression trees that are intended to be executed at run time. These
trees will undergo post-processing by the planner to make them fully
executable.
In GetForeignPlan>, generally the passed-in target list can
be copied into the plan node as-is. The passed scan_clauses> list
contains the same clauses as baserel->baserestrictinfo>,
but may be re-ordered for better execution efficiency. In simple cases
the FDW can just strip RestrictInfo> nodes from the
scan_clauses> list (using extract_actual_clauses>) and put
all the clauses into the plan node's qual list, which means that all the
clauses will be checked by the executor at run time. More complex FDWs
may be able to check some of the clauses internally, in which case those
clauses can be removed from the plan node's qual list so that the
executor doesn't waste time rechecking them.
As an example, the FDW might identify some restriction clauses of the
form foreign_variable> =>
sub_expression>, which it determines can be executed on
the remote server given the locally-evaluated value of the
sub_expression>. The actual identification of such a
clause should happen during GetForeignPaths>, since it would
affect the cost estimate for the path. The path's
fdw_private> field would probably include a pointer to
the identified clause's RestrictInfo> node. Then
GetForeignPlan> would remove that clause from scan_clauses>,
but add the sub_expression> to fdw_exprs>
to ensure that it gets massaged into executable form. It would probably
also put control information into the plan node's
fdw_private> field to tell the execution functions what
to do at run time. The query transmitted to the remote server would
involve something like WHERE foreign_variable> =
$1, with the parameter value obtained at run time from
evaluation of the fdw_exprs> expression tree.
Any clauses removed from the plan node's qual list must instead be added
to fdw_recheck_quals> or rechecked by
RecheckForeignScan> in order to ensure correct behavior
at the READ COMMITTED> isolation level. When a concurrent
update occurs for some other table involved in the query, the executor
may need to verify that all of the original quals are still satisfied for
the tuple, possibly against a different set of parameter values. Using
fdw_recheck_quals> is typically easier than implementing checks
inside RecheckForeignScan>, but this method will be
insufficient when outer joins have been pushed down, since the join tuples
in that case might have some fields go to NULL without rejecting the
tuple entirely.
Another ForeignScan> field that can be filled by FDWs
is fdw_scan_tlist>, which describes the tuples returned by
the FDW for this plan node. For simple foreign table scans this can be
set to NIL>, implying that the returned tuples have the
row type declared for the foreign table. A non-NIL value must be a
target list (list of TargetEntry>s) containing Vars and/or
expressions representing the returned columns. This might be used, for
example, to show that the FDW has omitted some columns that it noticed
won't be needed for the query. Also, if the FDW can compute expressions
used by the query more cheaply than can be done locally, it could add
those expressions to fdw_scan_tlist>. Note that join
plans (created from paths made by GetForeignJoinPaths>) must
always supply fdw_scan_tlist> to describe the set of
columns they will return.
The FDW should always construct at least one path that depends only on
the table's restriction clauses. In join queries, it might also choose
to construct path(s) that depend on join clauses, for example
foreign_variable> =>
local_variable>. Such clauses will not be found in
baserel->baserestrictinfo> but must be sought in the
relation's join lists. A path using such a clause is called a
parameterized path>. It must identify the other relations
used in the selected join clause(s) with a suitable value of
param_info>; use get_baserel_parampathinfo>
to compute that value. In GetForeignPlan>, the
local_variable> portion of the join clause would be added
to fdw_exprs>, and then at run time the case works the
same as for an ordinary restriction clause.
If an FDW supports remote joins, GetForeignJoinPaths> should
produce ForeignPath>s for potential remote joins in much
the same way as GetForeignPaths> works for base tables.
Information about the intended join can be passed forward
to GetForeignPlan> in the same ways described above.
However, baserestrictinfo> is not relevant for join
relations; instead, the relevant join clauses for a particular join are
passed to GetForeignJoinPaths> as a separate parameter
(extra->restrictlist>).
An FDW might additionally support direct execution of some plan actions
that are above the level of scans and joins, such as grouping or
aggregation. To offer such options, the FDW should generate paths
and insert them into the
appropriate upper relation>. For example, a path
representing remote aggregation should be inserted into the relation
obtained from fetch_upper_rel(root, UPPERREL_GROUP_AGG,
NULL)>, using add_path>. This path will be compared on a
cost basis with local aggregation performed by reading a simple scan path
for the foreign relation (note that such a path must also be supplied,
else there will be an error at plan time). If the remote-aggregation
path wins, which it usually would, it will be converted into a plan in
the usual way, by calling GetForeignPlan>.
Usually the most convenient place to generate such paths is in
the GetForeignUpperPaths> callback function, although
it can be done earlier if that seems appropriate.
PlanForeignModify> and the other callbacks described in
are designed around the assumption
that the foreign relation will be scanned in the usual way and then
individual row updates will be driven by a local ModifyTable>
plan node. This approach is necessary for the general case where an
update requires reading local tables as well as foreign tables.
However, if the operation could be executed entirely by the foreign
server, the FDW could generate a path representing that and insert it
into the UPPERREL_FINAL> upper relation, where it would
compete against the ModifyTable> approach. This approach
could also be used to implement remote SELECT FOR UPDATE>,
rather than using the row locking callbacks described in
. Keep in mind that a path
inserted into UPPERREL_FINAL> is responsible for
implementing all> behavior of the query.
When planning an UPDATE> or DELETE>,
PlanForeignModify> can look up the RelOptInfo>
struct for the foreign table and make use of the
baserel->fdw_private> data previously created by the
scan-planning functions. However, in INSERT> the target
table is not scanned so there is no RelOptInfo> for it.
The List> returned by PlanForeignModify> has
the same restrictions as the fdw_private> list of a
ForeignScan> plan node, that is it must contain only
structures that copyObject> knows how to copy.
INSERT> with an ON CONFLICT> clause does not
support specifying the conflict target, as unique constraints or
exclusion constraints on remote tables are not locally known. This
in turn implies that ON CONFLICT DO UPDATE> is not supported,
since the specification is mandatory there.
Row Locking in Foreign Data Wrappers
If an FDW's underlying storage mechanism has a concept of locking
individual rows to prevent concurrent updates of those rows, it is
usually worthwhile for the FDW to perform row-level locking with as
close an approximation as practical to the semantics used in
ordinary PostgreSQL> tables. There are multiple
considerations involved in this.
One key decision to be made is whether to perform early
locking> or late locking>. In early locking, a row is
locked when it is first retrieved from the underlying store, while in
late locking, the row is locked only when it is known that it needs to
be locked. (The difference arises because some rows may be discarded by
locally-checked restriction or join conditions.) Early locking is much
simpler and avoids extra round trips to a remote store, but it can cause
locking of rows that need not have been locked, resulting in reduced
concurrency or even unexpected deadlocks. Also, late locking is only
possible if the row to be locked can be uniquely re-identified later.
Preferably the row identifier should identify a specific version of the
row, as PostgreSQL> TIDs do.
By default, PostgreSQL> ignores locking considerations
when interfacing to FDWs, but an FDW can perform early locking without
any explicit support from the core code. The API functions described
in , which were added
in PostgreSQL> 9.5, allow an FDW to use late locking if
it wishes.
An additional consideration is that in READ COMMITTED>
isolation mode, PostgreSQL> may need to re-check
restriction and join conditions against an updated version of some
target tuple. Rechecking join conditions requires re-obtaining copies
of the non-target rows that were previously joined to the target tuple.
When working with standard PostgreSQL> tables, this is
done by including the TIDs of the non-target tables in the column list
projected through the join, and then re-fetching non-target rows when
required. This approach keeps the join data set compact, but it
requires inexpensive re-fetch capability, as well as a TID that can
uniquely identify the row version to be re-fetched. By default,
therefore, the approach used with foreign tables is to include a copy of
the entire row fetched from a foreign table in the column list projected
through the join. This puts no special demands on the FDW but can
result in reduced performance of merge and hash joins. An FDW that is
capable of meeting the re-fetch requirements can choose to do it the
first way.
For an UPDATE> or DELETE> on a foreign table, it
is recommended that the ForeignScan> operation on the target
table perform early locking on the rows that it fetches, perhaps via the
equivalent of SELECT FOR UPDATE>. An FDW can detect whether
a table is an UPDATE>/DELETE> target at plan time
by comparing its relid to root->parse->resultRelation>,
or at execution time by using ExecRelationIsTargetRelation()>.
An alternative possibility is to perform late locking within the
ExecForeignUpdate> or ExecForeignDelete>
callback, but no special support is provided for this.
For foreign tables that are specified to be locked by a SELECT
FOR UPDATE/SHARE> command, the ForeignScan> operation can
again perform early locking by fetching tuples with the equivalent
of SELECT FOR UPDATE/SHARE>. To perform late locking
instead, provide the callback functions defined
in .
In GetForeignRowMarkType>, select rowmark option
ROW_MARK_EXCLUSIVE>, ROW_MARK_NOKEYEXCLUSIVE>,
ROW_MARK_SHARE>, or ROW_MARK_KEYSHARE> depending
on the requested lock strength. (The core code will act the same
regardless of which of these four options you choose.)
Elsewhere, you can detect whether a foreign table was specified to be
locked by this type of command by using get_plan_rowmark> at
plan time, or ExecFindRowMark> at execution time; you must
check not only whether a non-null rowmark struct is returned, but that
its strength> field is not LCS_NONE>.
Lastly, for foreign tables that are used in an UPDATE>,
DELETE> or SELECT FOR UPDATE/SHARE> command but
are not specified to be row-locked, you can override the default choice
to copy entire rows by having GetForeignRowMarkType> select
option ROW_MARK_REFERENCE> when it sees lock strength
LCS_NONE>. This will cause RefetchForeignRow> to
be called with that value for markType>; it should then
re-fetch the row without acquiring any new lock. (If you have
a GetForeignRowMarkType> function but don't wish to re-fetch
unlocked rows, select option ROW_MARK_COPY>
for LCS_NONE>.)
See src/include/nodes/lockoptions.h>, the comments
for RowMarkType> and PlanRowMark>
in src/include/nodes/plannodes.h>, and the comments for
ExecRowMark> in src/include/nodes/execnodes.h> for
additional information.