Provide ability to specify ID (Guid) when creating elements or attributes

Hi Sean, you can use WebIDs based only on path info. It's a feature introduced in PI Web API 2017 R2 (1.10). We've got some details on how to build them without needing to do the lookup call first. Check out this PI Square article... https://pisquare.osisoft.com/community/developers-club/blog/2018/01/26/pi-web-api-using-web-id-20-to-optimize-your-applications

Although your question was around the AF SDK, have you considered using the XML import/export option to preserve unique ID's?

@Frank: Correct, we want to build queries via Guid the way that article describes, but we cannot do that unless we can inject our own GUID which is mastered outside of AF. @Stephen: Interesting suggestion but the data updates will be coming from both a stream (live updates) and from scheduled replication job. Both of these will be written against the AF SDK. PI Web API would be even better, but XML import would not be appropriate.

It's also important to note that we do plan on using templates so we can create element relative displays in PI Vision. The XML import documentation contains a disclaimer that Guids cannot be preserved when importing attributes that are based on templates. That is concerning because we need to be able to specify these Guids when we create elements based on a template. If they cannot be specified during the create, then we will need a way to update them afterwards.

In response to Sean Jost, "It's also important to note that we do p..." The IDs for attributes created from a template will be preserved as long as their template was also imported with their IDs preserved.

Thanks, David. In our case the template might already be defined. We need to stream updates from another data source where the guids are already defined. For instance, an element template will be created one day, then the next day we will need to create an element based on that template.

In response to Sean Jost, "Thanks, David.  In our case the template..." For the attribute ID to match, both the template and element IDs must match because storage for the attribute created from a template does not exist in the SQL database.   Will the option suggested by Frank using WebIDs based upon path work for you instead?

The applications that will be querying web API need store state which is associated to attributes in AF. We can't use path or a web ID based on path because path is subject to change. We need to be able to store GUIDs which can be easily translated into web IDs for web API queries. I am not quite understanding the first part of your response. Let me see if I have this right conceptually. Say for example I have a template called square, with an attribute called size and is mapped to a PI tag. Then I create an element called SmallSquare based on that template where size is 2, and another one called LargeSquare where size is 20. SmallSquare|Size and LargeSquare|Size will each have their own GUID, and I should be able to use those GUIDs to formulate WebIDs to query the sizes, correct? In this example, our applications need to store the GUIDs for SmallSquare|Size and LargeSquare|Size in their own DBs, so they can be related to domain specific data. If SmallSqure|Size were to be renamed to "Small Square|Size" or we rename the "Size" attribute Template to "Length", our integrity of our application's data should remain in tact because they are using GUIDs and not path.

In response to Sean Jost, "The applications that will be querying w..." In your example, the ID for the attribute SmallSquare|Size is generated by combining the ID from the template and owning AFElement. Therefore, if either of the IDs for the template or element do not match, then the ID for the attribute will not match between the two systems.   About the only alternative is to store the external ID in an extended property and then you can use that extended property as a search key.

Thanks, David, now I understand. The attribute ID is a deterministic derivation of the attribute template ID and the element ID. With that settled, can you add the ability to specify the IDs when creating the elements, attributes and attribute templates so we don't have to go through the XML import process for streaming incremental updates?

In response to Sean Jost, "Thanks, David, now I understand.  The at..." The feature was added to XML import to allow systems to be cloned between environments which can be done in a controlled manner from the XML file to ensure everything is consistently using the IDs correctly. When not importing the entire database, there are still ways of it not keeping the specified IDs. There are so many ways that it would not work as expected (e.g. IDs for attributes created from a template) that it was not made available directly in the SDK. 

In response to David Hearn, "The feature was added to XML import to a..." Thanks, David.  We will attempt the XML import method via AF SDK and hopefully that will be sufficient to use a long term option.  I have to say this has been a great experience, both in terms of quality and timeliness of response.  We are going to highlight this within the company as a success story.

In response to David Hearn, "The feature was added to XML import to a..." So we used AF SDK to export an attribute (not based on template), then updated the name and id with new values and imported the attribute back to the same element using the option to preserve unique IDs.  The attribute was added, but not with the ID that I specified.  I am using 2018 SP2 of AF SDK and AF Server.  Is there some trick I might be missing?

In response to Sean Jost, "So we used AF SDK to export an attribute..." Sean, are you able to provide additional information on what your use case(s) are with concrete examples?  I am guessing here a bit, but do you have assets like a pump, that you want to added to the AF hierarchy and you want to use ID from a foreign system as a unique way to track this asset through it's lifetime?   While we're on this, David previously mentioned the fact that elements can have extended property which can hold key-value pairs that you can search on.  Perhaps that does what you need?

In response to Stephen Kwan, "Sean, are you able to provide additional..." We are building middleware that will serve many different applications, so providing one or two use cases will under-represent what we are trying to do.  Your example is on the right track though.  We need the GUIDs to be mastered by another system that has been built for our end users to maintain hierarchy for their respective regions.   We need to be able to delete elements and/or attributes in the AF DB and then re-create them with the same ID.  This way users can make changes in our master system which get propagated to AF DB without breaking our applications that have stored state associated to those IDs.  You can think of our applications like PI Vision, which also stores state (displays), but it was apparently stored against path and not ID, which is why it cannot handle elements or attributes being moved.  We want to avoid that problem by storing state against GUIDs in app DBs.   We ran into another problem though, apparently WebIDs (in ID-only form) for attributes are composed with both attribute and element ID, so we cannot just store a single GUID and use that to compose the WebID.  This means our apps will have to store WebIDs instead of GUIDs which will be a costly change for apps that have historical data already associated with GUIDs.   We have had some luck in importing IDs with PSE, but it's not working in code for some reason.  We're suspecting it has to do with importing at the element vs DB as the target, importing from flat XML and importing the whole element vs just an attribute.  We are going to keep playing with it.  If you have any rules documented on this, that would be very helpful.   I do not believe extended properties will help us because: It is only at the element level, we need a globally unique identifier for attributes too.It does not seem that PI Web API can lookup based on extended properties.  At least not without relying on the crawler search index.Forces all apps to perform lookups before they can run any query.  We want PI Web API to be really easy to use.

In response to Stephen Kwan, "Sean, are you able to provide additional..." We have found that importing IDs is working great so long as we only specify the ExportMode attribute on the root node (AF).  We tested both StrongReferences mode while importing a single attribute directly against a descendant, and Flat mode while importing elements and attributes together.