Optimize the storage and retrieval of digital tags

My other suggestions related to the archive file format: https://feedback.osisoft.com/forums/555148-pi-server/suggestions/18401782-64-bit-unix-time https://feedback.osisoft.com/forums/555148-pi-server/suggestions/40343131-optimize-and-compress-non-primary-historic-archive

The built in swinging door compression and by the user configuring the compression settings properly, digital tags are already fairly optimized for storage efficiency. Any further optimization may incur a penalty for decompression by the server itself or by the client. In addition, there are complexities associated with server side calculations of summaries, search functions and calculation functions. For example, FindLE, TagAvg, etc.

Hi Stephen, Can you elaborate on what you mean by "configuring the compression setting properly"? I know that you can get great compression if you set excmax and compmax to be extremely high so then a value is recorded only when it changes, but a user might not want to do that so then they have events confirming that the value did indeed stay constant and that the PI Interface/PI Connector wasn't just turned off. When there are no events for a long time, you can't tell if they were dropped by exception and compression or not read at all. I was thinking that digital values could perhaps be stored as a list of timestamps for each possible value rather than storing events as pairs of timestamp and value, or something similar. Nothing too hardcore. If digital tags are already well optimized for storage, then why do you suspect that disk compression shrinks the archive files so much? The digital tags in the archives that I used use an excmax of 18 hours and have compression turned off.

Kenneth, Due to the structure of the archive files and what you have done with your PI Points (such as deleting a PI Point), the archive files may have unused space. You can contact tech support and they can help you understand if re-processing your archive files can reclaim some space. BTW, I'm not disputing that compression would decrease file size. In terms of digital tags, if the value doesn't change and you have Compression turned on, we don't store additional values unless CompMax has been reached. Imagine that you have a digital tag that hasn't changed for 24 hours and you have Compression turned on, and a CompMax of 8 hours, we would only store 4 events for the 24 hours of duration including the start and end of the 24 hours period. One at start, one at +8h, one at +16h, then lastly one at +24h.

Hi Stephen, I actually already reprocessed all of our archive files in an attempt to save space. The % Full decreased by a few %, but since they are fixed-size archives, the size on disk didn't change. However, when I use disk compression on the files, most of the archives shrink to less than half of their original size. As I mentioned, our tags use an excmax of 18 hours, which should lead to pretty decent lossy compression, and yet even with that in place, the archives still compress very well losslessly. I'll take your advice and contact tech support. Right now I'm pointing the finger at digital tags with little evidence.

@Kenneth - I want to make sure I understand what you're describing. You're saying you Have ExcMax of 18 hours? If so, that's a setting on the interface (exception max) and not on the server. I don't think UserVoice is a proper forum to discuss what your particular PI Server configurations are. Perhaps in your tech support case, you can have a detailed discussion with our support personnel.

As it turns out, the archives that I've been testing compress well mostly because of a high number of unused records: https://feedback.osisoft.com/forums/555148-pi-server/suggestions/33078940-reduce-archive-file-size-by-eliminating-unused-rec