Some more information on this issue. We were unsuccessful in catching an instance where an update was missed to a subscribed tag which was being routed through the OPC Analyzer (note that we were only monitoring 10 tags of the ~ 5000). We also have to consider that the act of running a subscription through the OPC Analyzer prevented the problem from occurring.

In the mean time, the customer was getting very fed up as they could not longer trust data from the system and also believed that the frequency of missed updates was increasing (they would find a tag with an incorrect value at least once a day, but the frequency was probably greater as there is a good chance missed updates would not be noticed).

Given the severity of the situation we purchased the ClientAce .NET OPC Driver from Kepware and integrated it into our system. The customer has reported that since that change they have not noticed any missed updates. We also wrote an "incorrect value detector" which, every 10 seconds, does a read of all tags from the OPC Server (via the "ReadMultipleItems" method in EasyDAClient or the "Read" method in DaServerMgt). For each tag we compare the value vs. the last value received via the subscription and if we see 5 reads in a row with incorrect values (and no change in the subscribed value) we raise an alarm. Note that we specify a max age of 5000ms which should always get the cached value from the OPC Server given that all tags are subscribed with a rate of 1000ms or less). We have two systems running the ClientAce driver for the last week and have not reported any incorrect values (consistent with the customers') finding. We have another server (less critical) which has been running the OPCLabs driver and last right reported a missed update:

Subscribed: (False) Direct: (True) Subscribed Timestamp: 2016-02-19T06:52:42.2993453Z Direct Timestamp: 2016-02-19T07:57:42.9820322Z
Subscribed: (False) Direct: (True) Subscribed Timestamp: 2016-02-19T06:52:42.2993453Z Direct Timestamp: 2016-02-19T07:57:52.9664878Z
Subscribed: (False) Direct: (True) Subscribed Timestamp: 2016-02-19T06:52:42.2993453Z Direct Timestamp: 2016-02-19T07:58:02.9509489Z
Subscribed: (False) Direct: (True) Subscribed Timestamp: 2016-02-19T06:52:42.2993453Z Direct Timestamp: 2016-02-19T07:58:12.9822813Z
Subscribed: (False) Direct: (True) Subscribed Timestamp: 2016-02-19T06:52:42.2993453Z Direct Timestamp: 2016-02-19T07:58:22.9823650Z

You can see that the value changed from False to True in the PLC but the "MultipleItemsChanged" event was not raised (note that we have tried running the system with listening to the "ItemChanged" event also).

With this data we can confidently say the problem is either with the OPCLabs driver or in the code we use to interface with it, but I do not believe it is the latter as we are simply adding item changed updates to a queue and then processing the queue on a different thread (to minimize time spent in the callback). Appropriate locking is used.

We are willing to share the source code we use to interface with the OPCLabs driver as well as the source code we use to interface with the ClientAce driver if you are interested. If you have a different version of the OPCLabs driver you would like us to try we are willing to run tests on the less critical system.

I suspect we will run a subset of the tags (perhaps 500) through the OPC Analyzer to minimize the performance impact. We have 88 terabytes of hard drive space attached to the system (8 terabytes of which is solid state), so log file size will not be an issue. Looking at OPCAnalyzer 1.03.1014, I don't see a way to write trace data to a file automatically (ideally, it would create a new log at regular intervals). Am I missing something or will we have to manually save the log file to disk every so often? Do you have any other suggestions for running this experiment?

The last time the customer notified me of this issue was April 16th, 2014, but the gentleman who would typically deal with it (by restarting our software) is out until next week so we will have to wait until he is back before determining whether it happens more frequently.

This instance of the issue has unfortunately escalated to upper management who is very concerned that they can now not trust the data from our system, so it is very important to us to determine which component is at fault and fix it.

We have an application that subscribes to ~5000 tags from an RSLinx OPC Server (Quick OPC 5.31 Build 1331.1). I was alerted today of a situation where the most recent update for a tag indicated true when in actual fact it was false in the PLC (implying that a change was missed).

I started an instance of the EasyOPC.NET Demo Application and performed a "Read item" of the tag and received the correct value of false. I started a subscription and the value remained false. I then went back to our application and unsubscribed/resubscribed to the tag and received the correct value of false.

Can you think of a way to determine which of the 3 forces at work (RSLinx, EasyOPC, or our application) is responsible for the error? While I obviously can't say conclusively, the chances of it being our application are small as we are simply updating a dictionary in the tag update callback.

Would running OPC analyzer be useful in this case?