Currently, the procedure to recover buffer queue files consists of parsing the queue file into a temporary archive and then merging this archive with the registered archive files that overlap with the impacted period.
This final merge requires to take the registered archive offline, which risks dataloss if new data arrives for the period of this archive. Alternatively, the archive subsystem can be stopped to prevent this dataloss, but this will impact users and calculations that need archive data, which in turn leads to incorrect data or inconsistencies.
Additionally, in the case of collectives, this needs to be repeated for all collective members, risking differences in data between the members while the merge process is done.
The only available option to do this without risking dataloss or inconsistencies is setting up a temporary server with a backup and merge the queue/archive to this temporary server, which then allows you to extract the data with standard tools. This is not the easiest or most practical solution.
There should (at least) be an option to extract events with their point ID from buffer queue files or offline archive files to a CSV format, so that the events can be manually processed with the SDK without the need to unregister archives or stop the archive subsystem.
Even better would be the possibility to merge a buffer queue or offline archive file with a registered archive file (as an extension to the current online reprocessing).
Understandably, this is much more complex, so a minimum viable solution is a way to extract the events to a CSV format.
The ability to review a buffer queue file before processing it into a PI system is critical. If the data is in fact valid, it would be nice to be able to push the data back through the buffer subsystem via UFL or powershell.
Furthermore, we can create pipelines to process buffer queue files automatically when they are generated via the process below:
Buffer queue file gets generated
script copies the queue file to a new directory and converts it to CSV
UFL/powershell ingests file and writes the data to the PI system via buffer subsystem.
leveraging NoReplace write mode to only Add event(s) that do not exist.
https://customers.osisoft.com/s/knowledgearticle?knowledgeArticleUrl=000040388