Sync Process

This section describes the architecture and settings of the Field Operator syncing system. This system was designed to accommodate low bandwidth remote devices and tax the ProCount database as little as possible.

Goals of the Syncing System

Several goals were considered while developing the syncing system. Some of which include:

• Ability to alert a user to new data: A key component of P2’s Route Surveillance package is the ability to monitor SCADA feeds and raise alerts and cases using Sentinel monitors. When a monitor raises a case an Explorer user validates the case and might assign task to a field user to remediate a problem. To get this information to the remote user as soon as possible, the Field Operator remote device automatically polls for new records on a regular basis. Instead of the user clicking a button to check for new data, the sync process runs automatically in the background (while the device is connected).
• Syncing data to the remote device within a relatively short connectivity window: Experience has shown that many areas of the oil field still do not have very good mobile coverage. Many field locations are far away from large cities, so they are not high on communication companies’ priority list for coverage such as 5G. The sync system is designed to quickly communicate data when a device has connectivity. When the sync happens, the device uploads files, downloads files and disconnects. Processing of the files is handled after the connection is complete. This keeps the connection time short. Also, if the user’s connection is disrupted, the server will still complete processing of uploaded files.
• Avoiding unnecessary data loads from the ProCount database: In many cases, users will re-download the same data over and over throughout the day. Often a user who has several routes on their device will re-download all routes 8-10 times during the day. If the system re-reads every record from the database every time such a user re-downloads, the ProCount database can be taxed unnecessarily to re-read records that have not changed. The Field Operator sync process is designed to only re-load changed records, and the user’s device automatically downloads changes, eliminating the need to re-read all data from the database throughout the day.

Data download processes

Downloading data to the remote device is split into two independently running processes.

• On the server the data sources (ProCount and Explorer), are polled for changes and the latest version of the data is stored in the download folders ready for the remote devices.
• On the client the process simply downloads any files that have changed since the last poll. The files are compressed and encrypted, downloaded, and then decompressed and decrypted.

The downloads folders are located on the Field Operator Server in the {Field Operator Home}\Field Operator Data\{Instance Name}\MobileClients folder.

Data Upload Process

Uploading data from the devices into ProCount and Explorer is accomplished through two independently running processes.

• When the user publishes readings in the client application the client-side sync process uploads the changes to the Uploads folder. After the file is transmitted, the user can disconnect.
• The server process is watching the Uploads folder and processes the records. Under normal circumstances, the records are saved, and the file is moved to the Processed folder where it will remain for 21 days. If an error occurs, a new file is created with only the errored records in the Errors folder. Often, data is also posted back to the Downloads folder at the end of this process.

The uploads folders are located on the Field Operator Server in the {Field Operator Home}\Field Operator Data\{Instance Name}\MobileClients folder.

There is also a temporary ‘Working’ folder that holds files that are currently being processed by the server. If the Field Operator Server exe were to be terminated (for example through task manager or a power outage), the next time the server starts up, files orphaned in the working folder are moved back to the uploads folder for re-processing.

Files in the Processed folder could be reprocessed if necessary. For example, if the ProCount database were to become corrupted at 2:00 pm and the only available database backup was taken at 10:00 am, after the database is restored, files from between 10:00 am and 2:00 pm can be moved back to the uploads folder and the records will be re-processed. Customers also might find these files helpful in test environments. Processed files can be copied from the production server to the uploads folder of a test server to simulate the users pushing data into both systems.

The name of an upload file contains information about the device from which the data was sent and the time stamp when the data was sent.

After they have been decompressed and decrypted into the uploads folders, the files can be read using any json reading tool. Within the file the columns of data being uploaded and the UserTb.MerrickID can be easily interpreted.

Settings Controlling the Sync Processes

In the eVIN Metadata configuration utility on the Field Operator tab, there are a couple of settings that control the process. The main piece to configure is the ProCount Changes watcher. This process on the Field Operator Server polls the ProCount database for any changes and updates the download folders when records have changed.

Records in ProCount contain UserDateStamp and UserTimeStamp columns which indicate when each record was changed. The ProCount Changes watcher periodically checks the database for changes to know which files need to be updated. The queries are similar (but not identical) to this query:

select Distinct rsi.RouteStopMerrickID, d.MerrickID
from MeterDailyTb d with (NOLOCK)
inner join RouteStopItemTb rsi with (NOLOCK) on d.MerrickID = rsi.ObjectMerrickID and rsi.ObjectMerrickType = 2
inner join FieldDataSourcePollTb p with (NOLOCK) on rsi.RouteStopMerrickID = p.ItemID and p.ItemType = 'Location'
where d.RecordDate > '20210317'
and((Convert(date, d.UserDateStamp) > p.ProcessStartDateStamp OR (Convert(date, d.UserDateStamp) = p.ProcessStartDateStamp and d.UserTimeStamp > p.ProcessStartTimeStamp))
or (Convert(date, d.BLogicDateStamp) > p.ProcessStartDateStamp OR (Convert(date, d.BLogicDateStamp) = p.ProcessStartDateStamp and d.BLogicTimeStamp > p.ProcessStartTimeStamp)))
and d.DataSourceCode <> 15
order by rsi.RouteStopMerrickID, d.MerrickID

This process is used to find which records have been changed in the ProCount database through the ProCount application or by import processes since the last time the download folder was updated.

Note: Changes uploaded from Field Operator are automatically loaded into the downloads folder. This means that if Field Operator user 1 uploaded a change and Field Operator user 2 had the same location on his device, the change would flow to Field Operator user 2 without the ProCount Changes watcher executing.

Administrators can control how often the ProCount Changes watcher looks for changes, how hard it works when changes are found and a time of day to suspend activities. These settings are found on the Field Operator Options Setup screen, P2 Merrick tab.

Poll ProCount for Changes every X seconds

This setting determines how often to check the ProCount database for changes that need to be added to the downloads folder. The default value is 300 seconds (5 minutes). Changes sent up from Field Operator users automatically are saved to the downloads folder, so this process is watching for changes from eVIN Laptop users, ProCount users or SCADA imports.

In P2’s test environment with a 3,000 well ProCount database, the ProCount changes watcher took between 400 and 800 milliseconds to look for changes. In a production environment it could take 2-3 seconds to check for changes. If the process takes more than 3 seconds, it will be noted in the logs. If you have a large installation where checking for changes takes a long time, consider implementing multiple instances of the Field Operator Server and lengthening the time between polls for changes.

Devices should poll for changes every X seconds

This setting is passed down to the Field Operator devices and instructs those devices how often to poll for changes. The default is 120 seconds (2 minutes), Every time the device checks for changes it communicates with the server to check what has changed in the download folders since the last download. Even if there is nothing to download, this process must pass a security token to validate the request and make a check. This is a fairly light weight process, but in a very large implementation, if the Field Operator Server’s CPU or network capacity is being taxed, consider lengthening this interval.

Suspend the ProCount changes watcher for nightly processing

Most Merrick implementations rely on a nightly processing routine to run the day’s allocations, import SCADA data, and run the Carte Loader. If Field Operator users likely are not downloading data at night, consider suspending the changes watcher while other processes are utilizing the database server’s capacity. Enter the time to suspend and the time to resume in 24 hour format. For example, a suspend time of 22:30 and a resume time of 05:15 will turn off the ProCount changes watcher between 10:30 pm and 5:15 am. Suspend and resume times will not be exact, but within X seconds of the set time where X equals the Poll ProCount for Changes every X seconds setting.

Be aware that if SCADA is loaded into the database at 5:00 am, but the ProCount changes watcher doesn’t resume until 6:00 am then the SCADA data will not be available to the Field users until sometime after 6:00am. At 6:00 am in this setup, the Field Operator Server will begin updating the download folders from the ProCount database and that process can take some time. On the other hand, if the ProCount changes watcher is not suspended, it will be reading data as the SCADA data is being written which could slow the SCADA import process.

Reload all stops when processing resumes

At some Merrick implementations, users have been trained to ‘Start over with new data’ every morning in eVIN and early versions of Field Operator. Although this probably isn’t necessary, it ensures that the user starts with exactly the data as is in the ProCount database. The “Reload all stops when processing resumes” option is the Field Operator version of eVIN’s ‘Start over with new data’. When the process resumes in the morning, all of the data will be re-read from the database into the download folders as if every object had changed. Although not necessary in most situations, this process ensures that the latest data from the database gets to the user even if the UserDateStamp, UserTimeStamp fields were not correctly updated. If during the night SCADA records have been pushed to the database for almost all of the objects, it may make sense to turn this option on because reloading all data might take about the same processing time as checking for what’s changed and reloading almost all of the data.

Every environment will reload the Field data in a different amount of time, so it’s difficult to provide metrics for how long the process will take in the morning. In one example, a test environment had about 11,000 active entities (completions + meters + tanks + equipment) and the process took about 18 minutes to reload all download folders. This works out to roughly 0.1 seconds per entity. Although a very rough guide, this might give the reader an idea as to how long before user’s start seeking data in the morning the ProCount changes watcher should be resumed.

Wait to resume until this query returns one or more records

It might be difficult to tell the right time in the morning to resume watching the ProCount database for changes to update the downloads folder. If this option is selected, you should provide a query which will be executed against the ProCount database to determine if the batch job that you’re waiting on is completed. It is your responsibility to ensure that the query is valid and returns at least one record when the Field Operator Server should resume. When this option is set, at ‘Time to resume watching for changes’, the Field Operator Server will begin executing your query every X seconds (where X is the setting in Poll ProCount for Changes every X seconds.) When the query returns a result, then the downloads folder update process will resume.

Even if the query does not return a record, resume watching for changes at

It is highly recommended that if you set the Wait to resume until this query returns one or more records setting that you also set a time to resume even if the query does not return results. If allocations normally finish at 5:00 for example, but it’s still running one day at 6:00, you may want to let the Field Operator Server begin preparing downloads folders even if allocations is still running. If the ProCount changes watcher is suspended, the users will get no readings for today when they sync.

Number of stops to load concurrently (threads) when loading data from ProCount

Like many Merrick processes, the Field Operator Server’s load on the database can be controlled by this thread count setting. More threads will use more CPU and memory on the Field Operator Server computer and will cause more load on the database server. But more threads will update the download folders quicker. The default is 3, but since every environment is different, it’s the responsibility of the administrator to experiment some with increasing and decreasing the threads. The goal is to keep the memory and CPU usage of the server machine at about 90% during the load and keep the Field Operator Server from dominating the database’s resources such that other processes are adversely affected.

Pad timestamp queries by X second(s)

Normally, leave this setting to zero seconds, but if you experience SCADA data being imported into the ProCount database but not arriving to the Field Operator devices, you may considering adjusting this setting.

When checking for records that have changed, the ProCount changes watcher issues a query like:

Select x,y,z from table where UserDateStamp > a and UserTimeStamp > b

...where a and b are the last time that the watcher executed the same query.

If there is a situation where a record arrives in the database after it’s UserTimeStamp, it’s possible that the query could miss a record causing them to not get downloaded to the devices. For example, a process is importing 1,000 records and it uses the current time as the UserTimeStamp that it’s setting on all 1,000 records. If the inserts take more than a second to execute and if the ProCount changes watcher was querying at that exact same time, then the last records inserted might be skipped.

The Pad timestamp query allows you to add a second or two to be sure that all records are found. The query will now be something like (this is simplified for illustration):

Select x,y,z from table where UserDateStamp > a and UserTimeStamp > (b – setting seconds)

The drawback is that it’s possible that the ProCount changes watcher might load a record twice if the last query was exactly at the time of an insert from SCADA. Don’t set this value two high as more records could get loaded twice. Recommended setting is 5 seconds or less.