Hello,
and thank you for all the details.

For clarification from my side, the two PeriodicWorker.Process threads that you are seeing look precisely as expected.
I am somewhat surprised by the threads near the bottom of your picture (Managed Ids 15, 16, 17), as they seem to have something to do Visual Studio design-time mode - I have not seen them before. But I suppose you know what you are doing/what is happening there.

I think it would definitely be worth testing .NET Framework 4.8.4110.0 or 4.8.3928.0 (those from the good machines) on the bad machine.

I also suggest to make absolutely sure that the problem is not actually related to the instrumentation (which is in part put in place to troubleshoot it, but probably it was there before as well). I have seen a situation where a lock somewhere inside the .NET tracing infrastructure caused mysterious problems. Note that we have some tracing calls - although the traces normally do not normally end up anywhere - in our code as well - so there can be contention. Try to remove all existing external-facing instrumentation except for some simple test of the occurrence of the problem, and check if the problem persists.

The ideas above are just that - ideas, wild guesses. But I think they are quote important to test, before going any further. Plus, it is not clear to me how to proceed afterwards. If I had a reproducible scenario here, I could perhaps turn on/off parts of the code that gets executed as a consequence of "new EasyUAClient()", and search for the part that triggers the issue. That would be very tedious, but in principle doable. But I cannot imagine doing that remotely. In addition, it is possible that the outcome of such troubleshooting process would be that the problem is triggered by some to a lower-level library - a call that we need to make, but cannot influence what is happening inside.

Unfortunately, Windows/.NET do not provide real-time execution guarantees, so what you are observing still falls under "normal behavior", seen from that perspective. Normally it is managed by making the necessary intermediate queues long enough to cope with situations encountered in the wild. I am not sure whether that is a possibility in your case.

Hello.

If it is not related to garbage collection, and is not something we can reproduce locally, it will be difficult to diagnose. So, I will have some more questions and perhaps some suggestions, but it just to obtain some more clarity and perhaps trigger an idea that could lead us in the right direction.

1. Why precisely is the issue titled "Threading/Periodic Worker Processes"? In the original report, there is only one sentence that appears related to this, saying "Ultimately, we are looking to get information on the possible threading/periodic worker processes that could be having an effect on this.". We have a class (that runs a thread in its instance) called PeriodicWorker. Are you referring to this, i.e. do you see some (or many) threads named "OpcLabs.PeriodicWorker.Process"? If so, how does it look in "normal" situation, and how does it look when the problem occurs (if it can be determined at that time)?

2. Is the general CPU usage on the computer, and CPU usage by the application, "good" (low)?

3. The architecture ("block diagram") of the whole system is not fully clear to me. But I assume that whatever dequeues the entries from the "event status queue" in the .NET application must be running on some thread. How is this thread created? Is it with "new Thread()"? Or is a thread from a .NET thread pool, perhaps a Task? Or something else?

4. Can you obtain the actual full .NET Framework version being used by the application on the "good", and "bad" machines? Suggestion is to read and capture System.Runtime.InteropServices.RuntimeInformation.FrameworkDescription from inside the app.

5. Have you tried to break into the debugger when the problem occurs, and look for suspicious symptoms? Mainly, does the number and "structure" of managed threads significantly differ, when the problem occurs? (this is more or less repeating a part of question #1, but is more general here)

6. Does the issue possibly appear multiple times during a single run of the application, if you let it run long enough- or just once? (If it happens once, it may indicate an issue with .NET JIT compilation)

7. In the original report, you wrote "When they removed the call to client = new EasyUAClient(), the problems stopped occurring.". Does this mean that adding "client = new EasyUAClient()", *without any further operation on that 'client' object* , is enough to introduce the problem? Or did you actually mean that besides "client = new EasyUAClient()", a whole set of new functionality has also been enabled inside the app, consequently?

Regards

Hello.

There can be various reasons, but the main suspect is .NET Garbage Collector (GC). In its default behavior, it kicks in at unpredictable times, and it stops all other threads in the running process, performs memory cleanup, and then resumes the normal execution of the program. Pulling in our library (well, any library, or adding any code) can change the memory usage patterns and therefore the GC invocations - which is nobody's fault, but can stull have negative effect.

Please determine which .NET or .NET Framework version are they running the program in (because the GC behavior depends on that).
It is important to know the version is truly running, and not just the version they are targeting. Example: They may be targeting .NET Framework 4.7.2, but run under .NET Framework 4.8. Or, they can be targeting .NET 6, but run under .NET 7.

In order to determine whether the above suspicion is correct, please ask the customer to correlate the occurrences of the problem with the times when .NET GC runs. There are tools such as PerfMon (and others) that can help with that. Microsoft has all the docs. Based on the outcome, we can decide on further steps. If the culprit is truly the GC, there are some (limited) ways to influence its behavior (and they depend greatly on the .NET Runtime version).

Best regards

Hello,
I am currently travelling, please allow some time before I can answer.

Best regards