# Writing a Custom Scan Provider

PostgreSQL supports a set of experimental facilities which are intended to allow extension modules to add new scan types to the system. Unlike a foreign data wrapper, which is only responsible for knowing how to scan its own foreign tables, a custom scan provider can provide an alternative method of scanning any relation in the system. Typically, the motivation for writing a custom scan provider will be to allow the use of some optimization not supported by the core system, such as caching or some form of hardware acceleration. This chapter outlines how to write a new custom scan provider.

Implementing a new type of custom scan is a three-step process. First, during planning, it is necessary to generate access paths representing a scan using the proposed strategy. Second, if one of those access paths is selected by the planner as the optimal strategy for scanning a particular relation, the access path must be converted to a plan. Finally, it must be possible to execute the plan and generate the same results that would have been generated for any other access path targeting the same relation.

## Creating Custom Scan Paths

A custom scan provider will typically add paths for a base relation by setting the following hook, which is called after the core code has generated all the access paths it can for the relation (except for Gather and Gather Merge paths, which are made after this call so that they can use partial paths added by the hook):

    typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,
                                                RelOptInfo *rel,
                                                Index rti,
                                                RangeTblEntry *rte);
    extern PGDLLIMPORT set_rel_pathlist_hook_type set_rel_pathlist_hook;

Although this hook function can be used to examine, modify, or remove paths generated by the core system, a custom scan provider will typically confine itself to generating CustomPath objects and adding them to `rel` using `add_path`, or `add_partial_path` if they are partial paths. The custom scan provider is responsible for initializing the CustomPath object, which is declared like this:

    typedef struct CustomPath
    {
        Path      path;
        uint32    flags;
        List     *custom_paths;
        List     *custom_restrictinfo;
        List     *custom_private;
        const CustomPathMethods *methods;
    } CustomPath;

path must be initialized as for any other path, including the row-count estimate, start and total cost, and sort ordering provided by this path. flags is a bit mask, which specifies whether the scan provider can support certain optional capabilities. flags should include `CUSTOMPATH_SUPPORT_BACKWARD_SCAN` if the custom path can support a backward scan, `CUSTOMPATH_SUPPORT_MARK_RESTORE` if it can support mark and restore, and `CUSTOMPATH_SUPPORT_PROJECTION` if it can perform projections. (If `CUSTOMPATH_SUPPORT_PROJECTION` is not set, the scan node will only be asked to produce Vars of the scanned relation; while if that flag is set, the scan node must be able to evaluate scalar expressions over these Vars.) An optional custom_paths is a list of Path nodes used by this custom-path node; these will be transformed into Plan nodes by planner. As described below, custom paths can be created for join relations as well. In such a case, custom_restrictinfo should be used to store the set of join clauses to apply to the join the custom path replaces. Otherwise it should be NIL. custom_private can be used to store the custom path's private data. Private data should be stored in a form that can be handled by `nodeToString`, so that debugging routines that attempt to print the custom path will work as designed. methods must point to a (usually statically allocated) object implementing the required custom path methods, which are further detailed below.

A custom scan provider can also provide join paths. Just as for base relations, such a path must produce the same output as would normally be produced by the join it replaces. To do this, the join provider should set the following hook, and then within the hook function, create CustomPath path(s) for the join relation.

    typedef void (*set_join_pathlist_hook_type) (PlannerInfo *root,
                                                 RelOptInfo *joinrel,
                                                 RelOptInfo *outerrel,
                                                 RelOptInfo *innerrel,
                                                 JoinType jointype,
                                                 JoinPathExtraData *extra);
    extern PGDLLIMPORT set_join_pathlist_hook_type set_join_pathlist_hook;

This hook will be invoked repeatedly for the same join relation, with different combinations of inner and outer relations; it is the responsibility of the hook to minimize duplicated work.

Note also that the set of join clauses to apply to the join, which is passed as `extra->restrictlist`, varies depending on the combination of inner and outer relations. A CustomPath path generated for the `joinrel` must contain the set of join clauses it uses, which will be used by the planner to convert the CustomPath path into a plan, if it is selected by the planner as the best path for the `joinrel`.

### Custom Scan Path Callbacks

    Plan *(*PlanCustomPath) (PlannerInfo *root,
                             RelOptInfo *rel,
                             CustomPath *best_path,
                             List *tlist,
                             List *clauses,
                             List *custom_plans);

Convert a custom path to a finished plan. The return value will generally be a `CustomScan` object, which the callback must allocate and initialize. See custom-scan-plan for more details.

    List *(*ReparameterizeCustomPathByChild) (PlannerInfo *root,
                                              List *custom_private,
                                              RelOptInfo *child_rel);

This callback is called while converting a path parameterized by the top-most parent of the given child relation `child_rel` to be parameterized by the child relation. The callback is used to reparameterize any paths or translate any expression nodes saved in the given `custom_private` member of a CustomPath. The callback may use `reparameterize_path_by_child`, `adjust_appendrel_attrs` or `adjust_appendrel_attrs_multilevel` as required.

## Creating Custom Scan Plans

A custom scan is represented in a finished plan tree using the following structure:

    typedef struct CustomScan
    {
        Scan      scan;
        uint32    flags;
        List     *custom_plans;
        List     *custom_exprs;
        List     *custom_private;
        List     *custom_scan_tlist;
        Bitmapset *custom_relids;
        const CustomScanMethods *methods;
    } CustomScan;

scan must be initialized as for any other scan, including estimated costs, target lists, qualifications, and so on. flags is a bit mask with the same meaning as in CustomPath. custom_plans can be used to store child Plan nodes. custom_exprs should be used to store expression trees that will need to be fixed up by `setrefs.c` and `subselect.c`, while custom_private should be used to store other private data that is only used by the custom scan provider itself. custom_scan_tlist can be NIL when scanning a base relation, indicating that the custom scan returns scan tuples that match the base relation's row type. Otherwise it is a target list describing the actual scan tuples. custom_scan_tlist must be provided for joins, and could be provided for scans if the custom scan provider can compute some non-Var expressions. custom_relids is set by the core code to the set of relations (range table indexes) that this scan node handles; except when this scan is replacing a join, it will have only one member. methods must point to a (usually statically allocated) object implementing the required custom scan methods, which are further detailed below.

When a CustomScan scans a single relation, scan.scanrelid must be the range table index of the table to be scanned. When it replaces a join, scan.scanrelid should be zero.

Plan trees must be able to be duplicated using `copyObject`, so all the data stored within the custom fields must consist of nodes that that function can handle. Furthermore, custom scan providers cannot substitute a larger structure that embeds a CustomScan for the structure itself, as would be possible for a CustomPath or CustomScanState.

### Custom Scan Plan Callbacks

    Node *(*CreateCustomScanState) (CustomScan *cscan);

Allocate a CustomScanState for this CustomScan. The actual allocation will often be larger than required for an ordinary CustomScanState, because many providers will wish to embed that as the first field of a larger structure. The value returned must have the node tag and methods set appropriately, but other fields should be left as zeroes at this stage; after `ExecInitCustomScan` performs basic initialization, the `BeginCustomScan` callback will be invoked to give the custom scan provider a chance to do whatever else is needed.

## Executing Custom Scans

When a CustomScan is executed, its execution state is represented by a CustomScanState, which is declared as follows:

    typedef struct CustomScanState
    {
        ScanState ss;
        uint32    flags;
        const CustomExecMethods *methods;
    } CustomScanState;

ss is initialized as for any other scan state, except that if the scan is for a join rather than a base relation, `ss.ss_currentRelation` is left NULL. flags is a bit mask with the same meaning as in CustomPath and CustomScan. methods must point to a (usually statically allocated) object implementing the required custom scan state methods, which are further detailed below. Typically, a CustomScanState, which need not support `copyObject`, will actually be a larger structure embedding the above as its first member.

### Custom Scan Execution Callbacks

    void (*BeginCustomScan) (CustomScanState *node,
                             EState *estate,
                             int eflags);

Complete initialization of the supplied CustomScanState. Standard fields have been initialized by `ExecInitCustomScan`, but any private fields should be initialized here.

    TupleTableSlot *(*ExecCustomScan) (CustomScanState *node);

Fetch the next scan tuple. If any tuples remain, it should fill `ps_ResultTupleSlot` with the next tuple in the current scan direction, and then return the tuple slot. If not, `NULL` or an empty slot should be returned.

    void (*EndCustomScan) (CustomScanState *node);

Clean up any private data associated with the `CustomScanState`. This method is required, but it does not need to do anything if there is no associated data or it will be cleaned up automatically.

    void (*ReScanCustomScan) (CustomScanState *node);

Rewind the current scan to the beginning and prepare to rescan the relation.

    void (*MarkPosCustomScan) (CustomScanState *node);

Save the current scan position so that it can subsequently be restored by the `RestrPosCustomScan` callback. This callback is optional, and need only be supplied if the `CUSTOMPATH_SUPPORT_MARK_RESTORE` flag is set.

    void (*RestrPosCustomScan) (CustomScanState *node);

Restore the previous scan position as saved by the `MarkPosCustomScan` callback. This callback is optional, and need only be supplied if the `CUSTOMPATH_SUPPORT_MARK_RESTORE` flag is set.

    Size (*EstimateDSMCustomScan) (CustomScanState *node,
                                   ParallelContext *pcxt);

Estimate the amount of dynamic shared memory that will be required for parallel operation. This may be higher than the amount that will actually be used, but it must not be lower. The return value is in bytes. This callback is optional, and need only be supplied if this custom scan provider supports parallel execution.

    void (*InitializeDSMCustomScan) (CustomScanState *node,
                                     ParallelContext *pcxt,
                                     void *coordinate);

Initialize the dynamic shared memory that will be required for parallel operation. `coordinate` points to a shared memory area of size equal to the return value of `EstimateDSMCustomScan`. This callback is optional, and need only be supplied if this custom scan provider supports parallel execution.

    void (*ReInitializeDSMCustomScan) (CustomScanState *node,
                                       ParallelContext *pcxt,
                                       void *coordinate);

Re-initialize the dynamic shared memory required for parallel operation when the custom-scan plan node is about to be re-scanned. This callback is optional, and need only be supplied if this custom scan provider supports parallel execution. Recommended practice is that this callback reset only shared state, while the `ReScanCustomScan` callback resets only local state. Currently, this callback will be called before `ReScanCustomScan`, but it's best not to rely on that ordering.

    void (*InitializeWorkerCustomScan) (CustomScanState *node,
                                        shm_toc *toc,
                                        void *coordinate);

Initialize a parallel worker's local state based on the shared state set up by the leader during `InitializeDSMCustomScan`. This callback is optional, and need only be supplied if this custom scan provider supports parallel execution.

    void (*ShutdownCustomScan) (CustomScanState *node);

Release resources when it is anticipated the node will not be executed to completion. This is not called in all cases; sometimes, `EndCustomScan` may be called without this function having been called first. Since the DSM segment used by parallel query is destroyed just after this callback is invoked, custom scan providers that wish to take some action before the DSM segment goes away should implement this method.

    void (*ExplainCustomScan) (CustomScanState *node,
                               List *ancestors,
                               ExplainState *es);

Output additional information for `EXPLAIN` of a custom-scan plan node. This callback is optional. Common data stored in the ScanState, such as the target list and scan relation, will be shown even without this callback, but the callback allows the display of additional, private state.