Working with Data Records

Pipeline Pilot protocols function by flowing data records through a network of components. A data record can contain individual data properties in a flat property list or can contain a structured hierarchy of properties. Pipeline Pilot components operate on the data records that flow through it, and can analyze properties, edit or remove them, and create new properties before the data is passed on to subsequent components.

Referencing a Property

The details of working with data records and their properties depend upon the way in which a component is constructed. Language specific SDKs exist with appropriate syntax for each supported programming language or scripting language. You can access and update data record properties using the following:

Data Property Lists

A data record contains one or more lists of name/value property pairs. Such a list is referred to as a "property list", and can also be considered a property hash. When there is more than a single property list in a data record, the data record is structured as a number of hierarchical nodes, with each node being the owner of a property list. The case of a single property list is a simplified version of the hierarchical format, except that there is a single property level with no child nodes.

Data Properties

A data property is the basic unit of data storage within a property list. Note the following:

Hierarchical Data

As discussed previously, a data record can hold multiple property lists in a hierarchical structure. Strictly speaking, a record can be a hierarchical tree of nodes that descend from a single root node that is owned by the data record, as shown here:

Data Record Tree Structure

Each non-root node has a single parent node and each node can have zero, one, or more child nodes. Each node has an associated property list. A property that exists on a property list below the root node is often referred to as a "deep property".

When using a simple name for a data record property, such as “maximum”, you are implying that the property is in the default property list for the data record. The default property list is the property list associated with the root node of the hierarchy.

In many cases, a data record only has the root node with a single property list. You can treat such a record as a flat list of properties and values. Indeed, you can treat any record in this manner, if you ignore any deep property lists elsewhere in the data tree and you only require access to the default property list.

To access deep properties in a data record tree, most contexts support a special path syntax for naming properties that navigates the levels of the hierarchy to locate the property list that you specify.

Deep Property Path Names

The name of a deep property begins with the path to the node that contains the property list where the property exists. The node path is followed by a forward slash and then the name of the property itself. Therefore a node path and a deep property path share the same syntax. The only distinction is the context in which they are used. This follows the analogy of a file system object, where a deep property corresponds to a file name, written as the leaf element of a file path. While deep property names can be used in a number of language-specific component development environments, the following examples refer to PilotScript, where a hierarchical property name's value is always written using the %'...' form, since the forward slash has special meaning in PilotScript (arithmetic division):

isMine := %'/annotations/complex/history/submitter';
%'/annotations/complex/history/submission_date' := '2005/06/30';

Or, identically:

property('/annotations/complex/history/submission_date') := '2005/06/30';

To identify a hierarchical property (for example, to find out if it is defined), use the hierarchical path name in single quotes as you would with a non-hierarchical property name. For example:

IF %'/annotations/complex/history/amendment_date' IS DEFINED THEN

The PilotScript documentation covers additional details, including the use of XPath to access hierarchical data values. See the Help Center > Users > Pilotscript.

Viewing Data Records

The structure of data records flowing in a protocol can be viewed in a data watch window when you are using Design Mode or checkpoints. You can also add the Data Record Tree Viewer component to a protocol to analyze the data flowing on a pipe.


Global Data

Global data settings exist outside data records and are associated at run time with a protocol or subprotocol. The global settings persist throughout the execution of the protocol. The logic in components can work with global data settings and in some cases can actually make changes to global data.

Examples of global data are:

All but the last example can be considered read-only settings, and this the role for most global properties.

Global Data Structure

Global data is implemented using the same data structures as in data records as described previously (stored as a hierarchy of property lists). Many of the global settings are defined by application packages and the hierarchical nature of the data allows a package to define global nodes to hold its settings without danger of namespace collision.

Global Data Subprotocol Scope

In a hierarchy of subprotocols, each subprotocol typically adds to the global settings accessible within the subprotocol since additional global nodes and properties can be defined at each level. Such additions may also be designed to override settings defined at a higher scope.

A global setting defined at the top protocol level is visible to all components within all subprotocols within the protocol. However, when a subprotocol defines an additional global setting, it is only visible within that subprotocol and its descendent subprotocols. For this reason, some global data can be described as being "scoped" to a subprotocol.

Viewing Global Data

You can add the Global Data Tree Viewer component to a protocol as a diagnostic tool to view the global data within the scope of a protocol or subprotocol.

See Also:  Pipelines and Data Flow