Data Cache

This article applies to version 4.17 and later of IFS OI Explorer. 

A data cache is a local database that stores data from one or more external datasources, creating a local copy of this data.  This can be used to improve fetch performance on datasources with lower access speeds or high network latency.

The Data Cache can be connected to local datasource and also to and Edge Hosted datasources.

Data Cache also allows caching of values from sources which do not themselves retain history, such as a DA-only OPC source. In this scenario, we retain snapshots of values at the configured intervals so Explorer will show the captured history in displays (such as the trend), even though it's only possible to fetch the current value from the original source.

 

How Data Cache Works

With Data Cache enabled, the system will collect the configured data from the external source systems, and store them into the configured Cache Database.  InfluxDB and SQL Server databases are supported to store this data.

Once the cache is enabled, when a user asks for that piece of data, the system will recognize that the tag is being added to the cache.  It will then automatically collect that tag from the cache when the system asks for data.

For end users of the data, this means that there is a seamless change from the tag being collected from the original source, to being retrieved from the Data Cache instead.

Prerequisites and Installation

Prerequisites

• If you are using InfluxDB as the Cache database:
  • InfluxDB version 2.7.6
• Network:
  • Server port 443 (HTTPS; for configuration)
  • Server port 8883
• SSL certificate

• If using Edge Host:
  • Same Adaptor must be installed on both sides
  • Server port 8883 must be firewall accessible

Installation

The Explorer Data Cache is installed as part of the standard Explorer installer.

In the ServerConfig.xml configuration file, add DataCache to the list of enabled features in the Web configuration group.

<ConfigGroup Name="Web">
   <CollectionParam Key="EnabledFeatures">
     <Value>DataCache</Value>
...

The section below shows the tuning options that are available.  Most of these are for tuning the cache on limited bandwidth connections where running it at full speed may use the majority of the connection bandwidth.

Configuration File Settings

Locate these settings under the <ConfigGroup Name="DataCache"> node in ServerConfig.xml.

DataCacheDirectory	The path on disk the data ingestor should persist data batches. Default: %programdata%\P2 Energy Solutions\DataCache
DataIngestorMetricsDatasourceName	The name of the Datasource which is used to write data cache metrics data to.
DataCacheMaxDataPointsToWrite	Maximum batch size of data points to write to data cache. Note: setting too high a value may cause writes to be ignored. Default: 15000
DataCacheMaxParallelWrites	Maximum Data Cache Parallel writes Default: 16
DataCacheWriteTimeout	Timeout (secs) for waiting until a Data Cache write is completed. Default: 60
DataCacheWritePrecision	Timestamp precision for datums written to a Data Cache. Options are: Seconds, Milliseconds. Default: Seconds
DataIngestorTransferThrottleMBperMinute	Maximum compressed MB per minute of data to write to the Data Cache. Default: 30
DataIngestorHistoryFetchDays	Number of days in the past to start caching historical data. Changing this setting without service restarts will cause Scan groups to restart history fetching if history recovery is enabled. Default: 30.
DataIngestorRecentFetchDays	Number of days in the past to start caching recent data. Default: 1
DataIngestorDelayOffset	Delay offset (in seconds) for reading data for ingestion. Setting to -1 will use the Scan group period as the offset. Default: -1
DataIngestorDataItemsPerRequest	Maximum number of datums to return in ingestion batch read. NOTE: Increasing this value may impact performance and/or cause timeouts. Default: 50000
DataIngestorMaxEntitiesPerRequest	Maximum number of entities to return in ingestion batch read. NOTE: Increasing this value may impact performance and/or cause timeouts. Default: 200
DataIngestorDisable	Disable the Data Ingestion. Default: false

---

Creating a Data Cache

In IFS OI Server Management, adding a data cache is performed through the Connections menu option.

1. Open IFS OI Server Management, and click the Connections icon.

2. Select the Data Cache option from the middle panel.

3. Click the Create New Data Cache button. If you are updating a datasource, click the name of the data cache instead.

4. Fill in the datasource parameters and adaptor parameters (see below), and then click Save.

After you have saved, you can then create Scan Groups.

Datasource Parameters

The configuration parameters are different depending on the type of datasource you are configuring. For an InfluxDB Cache, the parameters are as follows.

Parameter	Description and example
Name	The name that this datasource will be known by, to be used in other applications such as IFS OI Explorer. In the example below, the list of datasources appears in IFS OI Explorer when adding a dataset to a page.
Description	Detailed information on the purpose of the data cache. A good description helps to address any ambiguities when there are several data caches with similar names.
Adaptor	The type of connection that is used by the datasource to retrieve the data. This is, by default, InfluxDB Cache.  SQL Server can also be used for caching smaller numbers of tags.
Data Cache Pool Size	The adaptor pool hosts a number of adaptors (datasource connections), which can be called upon to retrieve data. You need to specify the minimum and maximum adaptor pool size by moving the sliders left and right. If your external system only allows one active connection at a time, set both minimum and maximum to 1. Default: Min=2, Max=8. For more information on adaptor pool sizes, see Datasources and Adaptors.
Data Cache Write Pool Size	The number of adaptor instances that are used to write data from the original datasource to the cache.
Initialisation Timeout	Applies to the initialisation of an adaptor, when it first connects to a datasource and data cache. This is the time limit allowed for the adaptor to initially set up and connect to the datasource and data cache before timing out.
Request Timeout	Applies to all other operations against the datasource, such as tag fetch. It refers to the time limit applied to all non-initialisation operations against the cache and the original datasource. The precise behaviour when a datasource operation is aborted may vary by adaptor.

Adaptor Parameters

The properties of the InfluxDB Cache adaptor that will allow it to correctly store and retrieve data. The following table lists the adaptor's parameters, along with the name and type.

Parameter	Description and example	Name	Type
Server	The name of the server on which the data cache is hosted.	Server	String
Port	The port to use when connecting to the data cache.	Port	Integer
Database	The name of the database where this data cache is stored.	Database	String
Fetch Mode	Specifies how the source data is fetched.	FetchMode	String
Max Parallel Reads	The maximum number of parallel tag fetches.	MaxParallelReads	Integer
Use Secure Transport	Connect to InfluxDB using HTTPS	UseSecureTransport	Boolean
Username	The user name to use to connect to the data cache.	Username	String
Password	The password to use to connect to the data cache.	Password	EncryptedString

---

Creating a Scan Group

A Scan Group is a scheduled collection of tags from a single Datasource. Multiple Scan Groups can be configured to a single Data Cache.

From a Data Cache page:

1. Click the Create button. If you are updating a scan group, click the name of the scan group instead.

2. Fill in the Scan Group Details (see below), and then click Save.

After you have saved, you can then create more scan groups.

Scan Group Details

The configuration parameters are different depending on the type of datasource you are configuring. For a tag datasource, the parameters are as follows.

Parameter	Description
Name	The name of the scan group, must be unique within this data cache and must not use reserved characters.
Tag Type	The type of tags that can be assigned to this scan group. Datasource: Tags that are associated with a timeseries datasource. Dataset datasources are not available for caching. Calculation: Tags that are calculations.
Datasource	For Datasource tag type, specify the datasource this scan group is fetching from. You can only have one data cache per datasource, therefore Datasources which have already been assigned to other data caches are not available. The current Data Ingestion Pool Size of the datasource is displayed, and you can adjust it by clicking the link provided. This opens the Datasource page in a new tab. Scroll to the parameter and adjust as needed.
Scan Period	The scheduling frequency for reading from the original datasource.
Fetch Type	The type of fetch being made to the datasource. Raw Range: All data is fetched from the Datasource for the entire fetch range. Raw Spot: A single data point is fetched at the triggered time. The recorded timestamp is maintained from the original source. Note: History Fetch is not available for Raw Spot. Last Known Value Spot: A single data point is fetched at the triggered time. The recorded timestamp is the time of the fetch.
History Recovery	For Raw Range and Last Known Spot Value fetch types, you have the option of also  collecting historical data. The time period for the historical data is specified in the Server Configuration file. Selecting ‘Collect historical data’ tells the data cache to go back and fetch all data from a point in time in the past. When enabled, the data fetching is split into two parts – Historical and Recent. Each group is fetched independently. Recent (the last day by default) will always keep recent data up-to-date. Historical will keep back-filling from the start of the history period up to when the recent data started fetching. Once the historical data has been fetched, history recovery is disabled and only recent data will be fetched. If any data in the past is modified after the history has been fetched, changes will not be copied/updated by the cache. History Fetch Days: In days, specify the number of days to fetch history for. If left empty, the system default value will be used.
Delay Offset	Delay offset, in seconds, for reading data for data fetch. The offset will be earlier than the run time and is intended to account for any delay in recent data appearing in the historian. If left empty, the system default value will be used.
Data Items Per Request	Restricts the number of data items per data fetch. If left empty, the system default value will be used.
Max Entities Per Request	Restricts the number of entities per data fetch. If left empty, the system default value will be used.
Tag Source Type	The default source tags are fetched from. Use the “src” parameter to overwrite the default behavior. Default: Cached values will be fetched by default. Origin: Non-cached, original values will be fetched by default.
Scan Group Type	Determines how tags are assigned to the scan group. All: Assigns all tags from the associated datasource to the scan group. By Filter: Use naming expressions to assign matching tags to the scan group. Click the Preview button to view the list of matching tags. All matching tags will be assigned to the scan group on Save. Manual: Manually assign tags to the scan group. Click the Assign button to add specific tags to the scan group.

---

Notes and Troubleshooting

• If the cache fetches a value with low confidence, the confidence will be cached along with the datum values and it won’t re-fetch the data.

Current Status

Server Management displays statuses for both the data cache and scan groups.

Data Cache

A Data Cache can have the following statuses:

• Started: The data cache is active and in use.
• Updating:  The data cache is in the process of fetching data.
• Service Down: The data cache cannot connect to the server.
• No Status Available: The data cache cannot determine a status.

Scan Group

A Scan Group can have the following statuses:

• No Status Available: No scan groups are available.
• Waiting:  The scan group is waiting to fetch data.
• Scheduling: The scan group is scheduling a data fetch.
• Ready: The scan group is ready to use.
• Running: The scan group is fetching data.
• Stopping: The scheduled fetch is stopping.
• Shutdown: The scan group is not available.

When a scan group is in a Ready state, it also displays the Last fetch time, if available.

Overriding Cached Tags

• It is possible to override cached tags to read data from the original data source by using the following syntax in place of the tagname: {Tagname, src=”origin”}
• InfluxDB versions 2.0 to 2.7.5 are known incompatible.
• Relevant changes to the Data Dictionary are detected and Scan Groups will update themselves without the need for Service restarts.
• If tags exists in multiple Scan Groups, the Scan Group with the lowest scan period will fetch the tag data.
• A Datasource can only be assigned to one Data Cache via a Scan group.

Data Cache Buffering

• Data which has been fetched for a Scan Group is compressed and stored on the local hard drive.  These are sent to the P2 Server Service which then sends it to InfluxDB or MS SQL Server via the Cache Adaptor.  Even if the Data Cache database is unavailable, the data will be stored on the file system until the Cache database is back online.
• For low network bandwidth scenarios, the data sent can be throttled using the DataIngestorTransferThrottleMBperMinute setting in ServerConfig.xml. 
• Note: Throttling too low can cause the files to back up on the local system if they are being collected faster than they are being pushed to the Cache, potentially running out of hard drive space.

Data Ingestor System Metrics

• For diagnostics, the Data Ingestor can create system metrics tags via the Influx Adaptor which can be trended in Explorer.
• Tags names are prefixed with DataCache.
• Data is written once per minute.
• Edge Hosts which are ingesting will create unique tags by appending the Edge host name.

Edge Host Notes

• Edge Hosts can be configured to remotely ingest data. Set ‘DataIngestorDisable’ to ‘false’ in xml.
• Scan group scheduling and fetching for remote Datasources happens in the Edge Host process at the Edge Node.
• Buffered data is stored locally on the Edge Host server and sent to the P2 Server Service for storage in InfluxDB.
• It is possible to enable remote logging from an Edge Host. ISS Logger messages are sent from the Edge Host to the main Explorer machine where they are written to ISS Logger.  It is recommended to enable this for short periods while debugging to reduce the network overhead that it adds when enabled.