Multiple instances for Asset Analytics

we are also an Enterprise customer and we have been facing a lot of challenges with single instance of Analytics Service & Notification service. for a small scale system, analysis service works well but when system grows with thousands of assets (35K assets in our case) and analyses (140K Analyses in our case) the system starts to show signs of performance deterioration. all techsupport calls ends up providing gigs of logs files and service statistics and in the end we just hear the feedback of fine tuning our calculations. we have some very critical 10-20K analyses which we can't afford to loose for even 1 minute, but if analysis service goes down, it takes all 140K analyses with it including super critical ones as well.

Aveva/OSIsoft should definitely look into creating multi-instances of PI Analysis Services like PI-PI Interface so that we can group all critical analyses to 1 instance and do some load distribution. We have seen similar setup for PI Integrator for business analytics where we can configure up to 4 worker node services on 1 server machine.

Regards,

Gaurav A

Agree and Supported by EA Customer in Australia: User Voice on Behalf of EA customer: 1 AF server to 1 Analytics box gets 'way overloaded' even with allot of CPU thrown at it, why can’t we split analytics boxes to AF servers. Case alignment: closed case 710771 100% agree with this and voting on behalf of customer

In our case this is also critical. We have a database with 600 assets right now and each asset have around 14 rollup calculations. With this our server is using 30GB of memory and takes 1,5 hour to start and 7 hours to finish the first recalculation (after it starts). With the help of Rory (from OSIsoft) we reduced that to 30 minutes and 1,5 hours respectively using the Excluded functionality to remove some of the Rollup triggers (as there's no way to select individual triggers for Rollups) but this was just a test server. In our production server we will have 20000 assets, which will make the usage of Analysis for anything else very difficult. So, In our case ideally we would want to load balance all of those Rollup to split the load on different servers (this could be set by category or when you create the analysis) and one server to Event frames, another one for regular analysis, etc.

Wilson, We need to take a closer look at your system. It's not necessarily a good idea to just throw hardware and brute strength to solve every problem. Looking at your description, it should not take 1.5 hours to start. Which version of software are you running? We fixed a couple of issues that dramatically reduced start time with recent release(s). Do you have a lot of missing PI Points in your rollups? How often are your rollups being triggered? Are you in a situation where you're triggering more often that you think due to the rollups having so many inputs? You're welcome to email me directly, especially if you have a current tech support case.

In addition to resource usage, there are possible legal (international trade) or contractual (multi-tenant) concerns where having separate services with access to selected AF databases could make it easier to prove to the lawyers that there was no way for someone to have calculated data from pi server 'y' (on the same AF / Analytic server) and stored data on server 'x'

Stephen, to answer your questions: Which version of software are you running? A) We are running the latest version of everything (PI AF 2017R2 and Analysis 2017R2)   We fixed a couple of issues that dramatically reduced start time with recent release(s). A) Yeah, we know, the previous version wasn't usable. We noticed a lot of improvement on PI System Explorer as well as before it would take too long to even load an Analysis on it giving the amount of attributes we have.   Do you have a lot of missing PI Points in your rollups? A) No, we make sure we create all the points. But we had a lot of attributes that would not get a value (its optional manual data), so, we started using the Excluded functionality to remove those and it improved the overall performance as I mentioned above.     How often are your rollups being triggered? A) Our data flow is pretty slow. Around 2000 assets will have one data point each 10 minutes on average, but the other 18000 usually comes once in a day. And we would like to be able to subscribe to just some of the attributes to generate a rollup, but that is not possible in the current version, Rollups are set to monitor all in event based or you need to change to period, which is not ok to us.   Are you in a situation where you're triggering more often that you think due to the rollups having so many inputs? A) We didn't identify any problem with the data flow. As soon as the service start, it kind of behave ok, but we are still in the test phase and the data will start to flow in groups of assets, so, we may have a problem later, but as of now we didn't notice anything. But as stated before, if we could subscribe to just three or four attributes, in our case, that would be actually a better solution as we expect the data to come at the same time.

Stephen, to answer your questions: Which version of software are you running? A) We are running the latest version of everything (PI AF 2017R2 and Analysis 2017R2)   We fixed a couple of issues that dramatically reduced start time with recent release(s). A) Yeah, we know, the previous version wasn't usable. We noticed a lot of improvement on PI System Explorer as well as before it would take too long to even load an Analysis on it giving the amount of attributes we have.   Do you have a lot of missing PI Points in your rollups? A) No, we make sure we create all the points. But we had a lot of attributes that would not get a value (its optional manual data), so, we started using the Excluded functionality to remove those and it improved the overall performance as I mentioned above.     How often are your rollups being triggered? A) Our data flow is pretty slow. Around 2000 assets will have one data point each 10 minutes on average, but the other 18000 usually comes once in a day. And we would like to be able to subscribe to just some of the attributes to generate a rollup, but that is not possible in the current version, Rollups are set to monitor all in event based or you need to change to period, which is not ok to us.   Are you in a situation where you're triggering more often that you think due to the rollups having so many inputs? A) We didn't identify any problem with the data flow. As soon as the service start, it kind of behave ok, but we are still in the test phase and the data will start to flow in groups of assets, so, we may have a problem later, but as of now we didn't notice anything. But as stated before, if we could subscribe to just three or four attributes, in our case, that would be actually a better solution as we expect the data to come at the same time.

Stephen, to answer your questions: Which version of software are you running? A) We are running the latest version of everything (PI AF 2017R2 and Analysis 2017R2)   We fixed a couple of issues that dramatically reduced start time with recent release(s). A) Yeah, we know, the previous version wasn't usable. We noticed a lot of improvement on PI System Explorer as well as before it would take too long to even load an Analysis on it giving the amount of attributes we have.   Do you have a lot of missing PI Points in your rollups? A) No, we make sure we create all the points. But we had a lot of attributes that would not get a value (its optional manual data), so, we started using the Excluded functionality to remove those and it improved the overall performance as I mentioned above.     How often are your rollups being triggered? A) Our data flow is pretty slow. Around 2000 assets will have one data point each 10 minutes on average, but the other 18000 usually comes once in a day. And we would like to be able to subscribe to just some of the attributes to generate a rollup, but that is not possible in the current version, Rollups are set to monitor all in event based or you need to change to period, which is not ok to us.   Are you in a situation where you're triggering more often that you think due to the rollups having so many inputs? A) We didn't identify any problem with the data flow. As soon as the service start, it kind of behave ok, but we are still in the test phase and the data will start to flow in groups of assets, so, we may have a problem later, but as of now we didn't notice anything. But as stated before, if we could subscribe to just three or four attributes, in our case, that would be actually a better solution as we expect the data to come at the same time.

Stephen, to answer your questions: Which version of software are you running? A) We are running the latest version of everything (PI AF 2017R2 and Analysis 2017R2)   We fixed a couple of issues that dramatically reduced start time with recent release(s). A) Yeah, we know, the previous version wasn't usable. We noticed a lot of improvement on PI System Explorer as well as before it would take too long to even load an Analysis on it giving the amount of attributes we have.   Do you have a lot of missing PI Points in your rollups? A) No, we make sure we create all the points. But we had a lot of attributes that would not get a value (its optional manual data), so, we started using the Excluded functionality to remove those and it improved the overall performance as I mentioned above.     How often are your rollups being triggered? A) Our data flow is pretty slow. Around 2000 assets will have one data point each 10 minutes on average, but the other 18000 usually comes once in a day. And we would like to be able to subscribe to just some of the attributes to generate a rollup, but that is not possible in the current version, Rollups are set to monitor all in event based or you need to change to period, which is not ok to us.   Are you in a situation where you're triggering more often that you think due to the rollups having so many inputs? A) We didn't identify any problem with the data flow. As soon as the service start, it kind of behave ok, but we are still in the test phase and the data will start to flow in groups of assets, so, we may have a problem later, but as of now we didn't notice anything. But as stated before, if we could subscribe to just three or four attributes, in our case, that would be actually a better solution as we expect the data to come at the same time.

Stephen, to answer your questions: Which version of software are you running? A) We are running the latest version of everything (PI AF 2017R2 and Analysis 2017R2)   We fixed a couple of issues that dramatically reduced start time with recent release(s). A) Yeah, we know, the previous version wasn't usable. We noticed a lot of improvement on PI System Explorer as well as before it would take too long to even load an Analysis on it giving the amount of attributes we have.   Do you have a lot of missing PI Points in your rollups? A) No, we make sure we create all the points. But we had a lot of attributes that would not get a value (its optional manual data), so, we started using the Excluded functionality to remove those and it improved the overall performance as I mentioned above.     How often are your rollups being triggered? A) Our data flow is pretty slow. Around 2000 assets will have one data point each 10 minutes on average, but the other 18000 usually comes once in a day. And we would like to be able to subscribe to just some of the attributes to generate a rollup, but that is not possible in the current version, Rollups are set to monitor all in event based or you need to change to period, which is not ok to us.   Are you in a situation where you're triggering more often that you think due to the rollups having so many inputs? A) We didn't identify any problem with the data flow. As soon as the service start, it kind of behave ok, but we are still in the test phase and the data will start to flow in groups of assets, so, we may have a problem later, but as of now we didn't notice anything. But as stated before, if we could subscribe to just three or four attributes, in our case, that would be actually a better solution as we expect the data to come at the same time.

Stephen, to answer your questions: Which version of software are you running? A) We are running the latest version of everything (PI AF 2017R2 and Analysis 2017R2)   We fixed a couple of issues that dramatically reduced start time with recent release(s). A) Yeah, we know, the previous version wasn't usable. We noticed a lot of improvement on PI System Explorer as well as before it would take too long to even load an Analysis on it giving the amount of attributes we have.   Do you have a lot of missing PI Points in your rollups? A) No, we make sure we create all the points. But we had a lot of attributes that would not get a value (its optional manual data), so, we started using the Excluded functionality to remove those and it improved the overall performance as I mentioned above.     How often are your rollups being triggered? A) Our data flow is pretty slow. Around 2000 assets will have one data point each 10 minutes on average, but the other 18000 usually comes once in a day. And we would like to be able to subscribe to just some of the attributes to generate a rollup, but that is not possible in the current version, Rollups are set to monitor all in event based or you need to change to period, which is not ok to us.   Are you in a situation where you're triggering more often that you think due to the rollups having so many inputs? A) We didn't identify any problem with the data flow. As soon as the service start, it kind of behave ok, but we are still in the test phase and the data will start to flow in groups of assets, so, we may have a problem later, but as of now we didn't notice anything. But as stated before, if we could subscribe to just three or four attributes, in our case, that would be actually a better solution as we expect the data to come at the same time.

Stephen, to answer your questions: Which version of software are you running? A) We are running the latest version of everything (PI AF 2017R2 and Analysis 2017R2)   We fixed a couple of issues that dramatically reduced start time with recent release(s). A) Yeah, we know, the previous version wasn't usable. We noticed a lot of improvement on PI System Explorer as well as before it would take too long to even load an Analysis on it giving the amount of attributes we have.   Do you have a lot of missing PI Points in your rollups? A) No, we make sure we create all the points. But we had a lot of attributes that would not get a value (its optional manual data), so, we started using the Excluded functionality to remove those and it improved the overall performance as I mentioned above.     How often are your rollups being triggered? A) Our data flow is pretty slow. Around 2000 assets will have one data point each 10 minutes on average, but the other 18000 usually comes once in a day. And we would like to be able to subscribe to just some of the attributes to generate a rollup, but that is not possible in the current version, Rollups are set to monitor all in event based or you need to change to period, which is not ok to us.   Are you in a situation where you're triggering more often that you think due to the rollups having so many inputs? A) We didn't identify any problem with the data flow. As soon as the service start, it kind of behave ok, but we are still in the test phase and the data will start to flow in groups of assets, so, we may have a problem later, but as of now we didn't notice anything. But as stated before, if we could subscribe to just three or four attributes, in our case, that would be actually a better solution as we expect the data to come at the same time.

Analytics High Availability is implemented via Windows Server Failover Clustering. When one PI Analysis Service fails, WSFC reroutes to the other PI Analysis Service. Currently there can only be one PI Analysis Service instance per AF Server.

Stephen, I am currently working with a customer who would like to see this functionality.  The High Availability isn't the same as what they are looking for.  I would love to discuss further with you.  I am currently on a long term engagement with them to make their existing calculations more efficient, but they will likely still need more than one analytics server in the end.

I completely agree with Wilson. Bringing a first of our 40 plants on our AF Server we already fought a lot of performance issues with PI Analysis. I have no good feeling about setting 39 more customers up on the same AF Server. Also for such a Setup - with just one analytics Service, performance issues of one customer will have Impact on calculations of other plants and managebility gets worse with every new plant. How are you supposed to know which calculations may cause errors when 80 different people create calculations in different AF databases.

This is most definitely needed. I have a few customers here where this type of load balancing would make a significant difference to analysis performance.   As a variation to this idea, what about the option of Analysis pools, where a group of Analyses could be configured to run in a higher or lower priority pool, or even on a separate Analysis server? Even being able to target a specific AF database on a given Analysis server - when you have a need to restart the Analysis service for the calculations in a given database, you don't necessarily want analyses in other AF databases to be affected.

This is most definitely needed. I have a few customers here where this type of load balancing would make a significant difference to analysis performance.   As a variation to this idea, what about the option of Analysis pools, where a group of Analyses could be configured to run in a higher or lower priority pool, or even on a separate Analysis server? Even being able to target a specific AF database on a given Analysis server - when you have a need to restart the Analysis service for the calculations in a given database, you don't necessarily want analyses in other AF databases to be affected.

In response to John Messinger, "This is most definitely needed. I have a..." I do think John's idea is great. In fact, we were expecting it as a first step before a true HA setup. Meaning, the ability to separate the analysis in different servers. I haven't thought on the group idea before, but I do think that it would be great, nowadays we kind of do that using Categories, but with the number of analysis we have it gets really slow overall.   And I understand that OSIsoft implemented HA with Windows Cluster, but that is really not a viable solution for us as our experience with it (in other cases) was not satisfactory as it would change the nodes unexpectedly and that would make our system be unresponsive for a while (given the time it takes to restart the services).

I agree here. We need to have load balancing on the services level.

We need this asap