Professional OPC
Development Tools

logos

Online Forums

Technical support is provided through Support Forums below. Anybody can view them; you need to Register/Login to our site (see links in upper right corner) in order to Post questions. You do not have to be a licensed user of our product.

Please read Rules for forum posts before reporting your issue or asking a question. OPC Labs team is actively monitoring the forums, and replies as soon as possible. Various technical information can also be found in our Knowledge Base. For your convenience, we have also assembled a Frequently Asked Questions page.

Do not use the Contact page for technical issues.

Best practice for subscribing to large numbers of tags

More
04 Feb 2021 04:22 #9391 by krito
Hi,

Yes, we would be willing to try out a prerelease version. We're currently using the nuget packages.

Best regards

Please Log in or Create an account to join the conversation.

More
03 Feb 2021 20:00 #9390 by support
Regarding the error message ""The maximum message size quota for incoming messages (65536) has been exceeded. To increase the quota, use the MaxReceivedMessageSize property on the appropriate binding element. [...]":

In the new version we are working on (2021.1), we have made following changes:

1. Increased the initial limit settings so that with default chunk size (1000 for subscriptions), and with our test server, the limits are not exceeded.
2. Made it possible to configure the various parameters of BasicHttpBinding through a configuration file.

If you we willing to work with a prerelease version, I can provide it to you. If so, let me know whether you use the "Setup" installer, or just the NuGet packages, or both.

Best regards

Please Log in or Create an account to join the conversation.

More
03 Feb 2021 14:05 #9386 by krito
Thank you very much for the detailed response, it's very informative!

Based on your suggestions, we're going to make the necessary changes to the code and we'll see how the application performs. We'll start with removing the parallelization. As you pointed out, using multiple EasDAClient instances in Isolated mode will not be a viable option as we cannot put a strain on the server. So we'll start by performing the browse/subscribe for each asset in a serial manner.

I also agree with you about caching the browse results, rather than always interrogating the server -- this will give me some good arguments for doing some client-side caching.

Best regards

Please Log in or Create an account to join the conversation.

More
31 Jan 2021 18:11 #9379 by support
Hello.
Some foreword:

A. Regarding the links to other forums posts you have provided in your first post: Do not rely on information from these posts (without further verifying them with us), because a) some of it is obsolete, and b) most of it applies only to OPC DA (COM-based); the internal implementation of OPC XML-DA is different, and many of the settings mentioned in these posts have no effect.

B. We have not aimed at speed or larger volumes with our OPC XML-DA support. The intent was to complement the existing OPC DA functionality, keeping the high-level API.

To the parallelism/concurrency:

There is no point in making parallel calls to a single EasyDAClient instance. The instance maintains a single connection to each target server (one, in your case), and there is a single request/response at any point in time on that connection. Consequently, all these parallel calls will get serialized on the client side.

If you truly wanted to make calls concurrently, you would have to use separate EasyDAClient instances, AND set their Isolated property (right after creation) to True. However:
- Doing so would create parallel calls/connection to the server, so one needs to consider how many the server would be able to handle.
- We generally do not recommend having too many EasyDAClient instances - they are a bit heavy on the client side too.
- Some servers might serialize the calls, so with such server there will not be any gain.
- With most real servers, the bottleneck is in their communication link to the device. And this is a physical thing, so the server *has* to serialize access to the device. Calls like Read, if made in parallel, will end up serialized with practically any real server, so parallelization would bring no gain. Yes, with calls like Browse there is better chance that it would help, because the address space is usually (but not always) loaded in server's memory already - many servers do not need any device access is needed. Calls like Subscribe are somewhere in the middle - they may or may not need serialization on the server side; it depends.

You certainly need to be prepared for the fact that effects of parallelization might differ significantly between a simulation server (or a real server in simulation mode) and the real server.

My understanding is that for Browse, you do not get any errors. So I would just measure which approach (parallel or not) works better, and use that. But it should be the real parallelization - with separate EasyDAClient-s set to Isolated. If that does not help, you might be facing a real block with OPC XML-DA, which is in the fact that the Browse request is for a single starting item only. In other words, with a hierarchical address space where each of you assets has its own "root node" and then a structure beneath it, even if you know multiple nodes you would like to browse into (such as all the assets nodes, after the first level browse), OPC XML-DA cannot browse them in one request (this is fixed in OPC UA). So either you do it one by one, or try to parallelize, but there is nothing better than that - it's not a client a problem, it's OPC XML-DA problem. Depending on the application, caching the browse results on the client side may be a solution.

With Subscribe, I do not expect the parallelization to be of real help, *plus* it only makes sense to think about after having dealt with the error ("The maximum message size quota for incoming messages (65536) has been exceeded… ).

The error indeed has to do with the underlying communication parameters/limits. And there are two possible approaches:
a) Increase the limits. Unfortunately we do not have/expose a way to do so currently. I will ask the developer who worked on this part to assess how difficult it would be to provide something in this area to help you out.

b) Chunk the Subscribe-s. You can do it yourself, but you do not have to - there is a built-in mechanism that already does it. And, for Subscribe, it is set to make chunks of maximum size of 1000 items by default. Apparently that is still too high.

Instructions on how to change the chunk size are here: opclabs.doc-that.com/files/onlinedocs/QuickOpc/Latest/User%2...html#OPC%20DA%20Optimizer.html

If you do not get good results by yourself with reasonable effort, but were willing to help us set up a repro, we might then be able to analyze things on our side.

Best regards

Please Log in or Create an account to join the conversation.

More
29 Jan 2021 15:24 #9378 by krito
Hi,

Thanks for the quick response.

1. Yes, this is strictly about OPC XML-DA only.

2. Yes, just one OPC XML-DA server.

3. Due to security constraints related to accessing the live production OPC server, I can only use a simulator during this stage of development.

4. The network is our corporate LAN for IT and software development, and it's fast. The simulators I'm using are a Matrikon OPC DA simulator together with Kassl dOPCXGate as OPC XML-DA gateway. I'm also using an Advosol OPC XML-DA SDK server. I'm using the CSharpSimple simulator example included with their Evaluation kit, and only made some minor changes to create a larger set of simulated tags. The Matrikon/XGate setup is running on one of our development servers within the same network I'm on, and I'm running the Advosol server locally on my development PC.

5. Yes, I'm using the Advosol XML-DA test client against both simulator server instances. In both cases I can setup subscriptions to poll the server for updates. Due to the GUI refresh of the Advosol client I have to poll at a slower refresh rate than the desired rate of 1 sec.

The problem I initially wanted to solve was how to code our application to initialize the subscriptions as quickly as possible on application startup. Regardless of whether I use a parallel approach or not, I can usually get the application to launch and setup subscriptions for up to 1000 tags, and the updates are received just fine. It's just that increasing the tags to the required number will generally result in one or both of the following happening:
- increased delays in browsing/subscribing leading to excessive timeouts and an excruciatingly long application startup routine
- or errors from the EasyDAClient depending on how many tags were added in a single SubscribeMultiple() call

Please Log in or Create an account to join the conversation.

More
29 Jan 2021 14:09 #9377 by support
Hello,
can you please provide some additional information, before I answer your questions?

1. Although this appears just once, mentioned "by the way" in your post, please confirm that we are talking strictly about about OPC XML-DA only, and not OPC Data Access (COM-based).

2. [we have] "100 assets": Please confirm that you are, in f=effect, communicating to just one target OPC server. Or are you connecting to multiple servers?

3. What is the vendor/model of the OPC server you are connecting to? Does it have a public demo available, or documentation/info pages?

4. What is the nature of network between your client, and the server? Is it fast or slow?

5. Do you have any way to assess whether your results are related to what the client side is doing? I mean, do you have some other client app that is doing similar things - from the same computer to the same server - and if so, what results are you seeing with it?

Thank you

Please Log in or Create an account to join the conversation.

More
29 Jan 2021 12:55 #9376 by krito
Hi,

I've looked at the following posts for some guidance:
Error after 100,001 tags are added to Data Client
Subscribing to a large number of tags, in parallel
Subscribing and reading large number of tags
Design Question / Rely on automatic subscriptions or subscribe manually?

However I'm still unclear what is the best approach for subscribing to a large number of tags.

I'm using:
- QuickOPC nuget package version 5.59.492
- Windows 10 version 1809 build 17763.1697
- our application is targeting .NET Core 3.1 running as a windows service.

In our test environment, we have around 100 assets (nodes) having 50 tags each. We need to subscribe to approximately 5000 tags in total with 1 second refresh rate, and we'd like the application to start up and subscribe to these tags as quickly as possible so we're currently trying to do it in parallel.

pseudocode
var assetNodes = BrowseNodes(root) // get all the assets beneath the root node
Parallel.ForEach(var node in assetNodes)
{
  var tags = BrowseNodesRecursive(node) //get all the asset's child tags recursively
  EasyDAClient.SubscribeMultiple(tags)
}

The code uses a single instance of EasyDAClient and it's not isolated.

What I'm observing is that the approach shown above takes many minutes to complete and causes numerous timeouts on the server. It's rare that I can successfully launch our application with just 50 asset nodes (~2500 tags). Most of the time the Browse() and Subscribe() connections timeout and throw exceptions. I've tried to increase the various browsing related EastDAClientTimeout values, but that just means it takes longer and in most cases it still times out. So I've tried to do it differently by doing the following:

pseudocode
var bigListOfTags = new List<string>();
 
var assetNodes = BrowseNodes(root); // get all the assets beneath the root node
Parallel.ForEach(var node in assetNodes)
{
  var tags = BrowseNodesRecursive(node); //get all the asset's child tags recursively
  bigListOfTags = bigListOfTags.Concatenate(tags);
}
 
EasyDAClient.SubscribeMultiple(bigListOfTags )

The browsing is reasonably faster (presumably because there's less contention from Subscribe() calls), and the subscription is complete. But if we launch the app to subscribe to 50 asset nodes (~2500 tags) I begin to get errors from the subscription handler about the received message size in what appears to be the underlying WCF client:

"The maximum message size quota for incoming messages (65536) has been exceeded. To increase the quota, use the MaxReceivedMessageSize property on the appropriate binding element. [...]"


The stacktrace for the above error is attached.

I've also tried to just do everything synchronously and just walk the server hierarchy by getting all the nodes and tags in one long operation (no parallelism), and then subscribe to all 5000 tags in a single call to SubscribeMultiple(). I still get the same error as above.

So my questions are:
  1. Is there any guidance on how many simultaneous operations an OPC XML DA server can be expected to handle?

  2. Is there a difference in terms of overhead between Browse() and Subscribe() calls? Is one more expensive or resource intensive than the other?

  3. Can you recommend a 'chunking' approach to setting up a subscription for 5000 tags?

  4. Is it better to execute a single Browse() call and "walk the tree" rather than execute multiple concurrent Browse() calls?

  5. Is there a recommended size to the number of tags used in a call to SubscribeMultiple() and are there parameters that I should be aware of such as ClientParameters, EngineParameters or any settings related to subscription size?
Attachments:

Please Log in or Create an account to join the conversation.

Moderators: support
Time to create page: 0.080 seconds