Background + problem
When PI Connector tries to save data for a tag that does not already exist (e.g. data for a new piece of equipment), it will create that tag automatically. This is very convenient, but it sometimes led to issues where data that you never intended to be saved would be caught and saved and increase the number of tags being used.
Seemingly in response to this, AVEVA Adapters separate the discovery of new data streams from the querying of data to archive for the selected data streams. That is, using a data discovery, you first look at the new streams of data that are available, and then you choose whose which streams of data you actually want to save, and if you want all new streams to be added automatically, then you enable auto-select in the data discovery.
However, there are several problems with this approach. While new streams can be automatically selected from a data discovery, data discoveries themselves cannot be run automatically without scheduling a script to do so. Also, the data source gets processed/queried separately for data discovery and for the querying of data to archive, and both have separate query configurations, and this is all excessive in some cases.
Even if you use a scheduled script to perform data discoveries that have auto-select enabled, the data discoveries must still be performed before the query for data to archive so that the data selection configuration for the new streams already exists by the time the query is run. However, if a new data stream becomes available at the data source after the data discovery is run but before the next query is run, then any data for the new streams in the query will not be saved to the new streams, which causes data loss. The workarounds to this issue are:
Minimize the time gap between running the data discoveries and running the query, but this only reduces the chances of the issue occurring and does not recover any data that was missed when the issue does occur
Ensure that new streams do not become available during this gap, which is not always possible to ensure
Process/query the data source for the same time range twice, which is inefficient and unreasonable, especially for the AVEVA Adapter For Structured Data Files
Perform history recovery during this time, which not every AVEVA Adapter supports, and it is either very manual or requires quite a bit of setup to become automatic
AVEVA Adapters cannot currently treat new data streams with the seamlessness that PI Connectors offer.
Solution
AVEVA Adapters should have the option to behave more similarly to PI Connectors for new data streams. When this option is used, the AVEVA Adapter can process/query the data source just once, and with the results of that processing/query, the AVEVA Adapter can discover new data, auto-select it, and save the data that it just retrieved to the stream that it just created, similar to PI Connectors. This option would allow new data streams to be created automatically as needed rather than ahead of time with a lot of extra overhead. As for the issue that PI Connectors had where unintended data streams would be created, this can be solved with more careful configuration of the data discovery/selection.
The approaches that PI Connectors and AVEVA Adapters currently use to handle new data streams are useful in different scenarios, but it would be nice if AVEVA Adapters could handle both methods since PI Connectors are no longer being actively developed. That is, let us choose the approach through a setting rather than through the choice of program.
