PCO Connection to KepWare Session Closes

[Former Member]

Hi Experts,

I have KepServerEX v.5 and PCO 2.3, both on same server.  I can connect successfully and read an OPC tag into MII via PCO. 

HOWEVER:  I am experiencing the following behavior:

• The first request for a tag fails after ~15 seconds.  Error message is meaningless.
• Subsequent requests (of any tag) succeed … until I stop making requests for 60 seconds, then they succeed again after the initial time-out.
• ODD PART:  If I open an OPC Test Client on on the OPC / PCO server and configure a tag (and don’t even poll it) then my issue above disappear

I see a KepWare configuration setting set to close connections/sessions if no activity for 60 seconds.  (I can change the duration but not disable it.)

What I expect is happening is that the first PCO request creates the session (this takes a several seconds, even with the OPC Test Client).  Subsequent PCO requests within 60 seconds use the same connection/session.

What I seek is that the PCO agent starts, it opens the session and keeps it open, like the OPC Test Client does.  I don't see any obvious settings in PCO to configure it this way.  Any advice or direction?

Regards,

Dave

[former_member196557]

Hi David,

what is the current SP and Patch level of PCo 2.3?

What is the full version of Kepware?

What is Kepware connecting to (Channels/Devices)?

Is you Agent Instance only for MII Query?

Do you have any Agent Instances that are subscribed to tags and have notifications?

Regards, Steve

[Former Member]

Steve,

what is the current SP and Patch level of PCo 2.3?

2.304.1930.2112

What is the full version of Kepware?

KepWare 5.14.493.0

What is Kepware connecting to (Channels/Devices)?

Devices I think.  A tag on an A-B ControlLogix.

Is you Agent Instance only for MII Query?

Yes

Do you have any Agent Instances that are subscribed to tags and have notifications?

No

To clarify, in my original post I referred to "OPC Test Client", I meant "OPC Quick Client" that comes bundled with KepServerEX

Dave

[former_member196557]

Hi Dave,

Can you provide screen shots of the Kepware ControlLogix Device Configuration for Timing, Auto-Demotion, and Logix Options tabs?

Also, run tests again and observe the Status bar along the bottom of the KepServerEX Configuration app for the Clients and Active Tags values. When PCo Agent Instance starts you should see the Clients count increment, and NOT change until either you stop the Agent Instance or start/stop another OPC Client.  For a Query-only Agent instance, Active Tags value will be 0 of X, where X is the number of different tags you have queried from MII. ONce you connect with the Quick Client, or an Agent Instance that has Subscriptions and active Notifications, the Active Tags value will be something like X of Y.

FYI, ControlLogix connections can take a significant amount of time to initiate from KepServerEX due to Controller Synchronization in any of the Logical Addressing Modes. I have seen some ControlLogix PLC's with 150K tags take as long as 3 minutes to connect to Kepware after a restart or disconnect.

I would also recommend that you update PCo to the latest Patch 11 for SP4.

Regards, Steve

[Former Member]

Thanks for the direction Steve,

I've attached the screenshots of the KepWare configuration, as requested.  Noticed here that the connection timeout that I see is 9 sec. (= 3 sec. x 3 tries).

I see what you describe.  When I start the Agent, the Client Count increments by 1 (or to 1 when starting from scratch).  When I issue a request from MII the Active Tags increment by 1 to 0 of 1, 0 of 2, etc.  The Agent is query only as you identified.

You suggested creating a PCO Subscription Item against the Agent (which might solve my problem by keeping the connection open?).  I added 1 tag and restart the Agent.  This DIDN'T do what you suggested; still said 0 of X.  That suggests that the subscribed tag isn't getting scanned maybe?  Am I missing something in the PCO configuration?  Could simply adding a dummy Subscription item solve my problem?

Dave

[former_member196557]

Hi Dave,

Does the Client count decrement after 60 seconds of no activity?

I have PCo Agent Instances that do not have Subscriptions or Notifications and are used only for MII Queries, and they do not exhibit the behavior you are describing when connected to Kepware.  Is it possible that something is closing the connection on the PLC or Network side? if the PLC is disconnected from Kepware there should be a Kepware Event Log entry indicating such.

Please try a few things:

• Provide a screen shot of the Kepware Project Properties OPC DA tab.
• Reset the Kepware Event Log and then set it to log all the events (right click on the Event Log area to access the settings and reset option).

• Set the PCo Source System Reliable Connections to 9999 Max. No of Retries, and 30 seconds interval.

• Set the PCo Agent Instance Log level to Verbose and restart the Agent Instance.

• Execute a Query from MII, then wait 60+ seconds to see if there are any errors or warnings listed in either the Pco Log or the Kepware log.
• Disable the Auto-Device Demotion in the Kepware Device settings, restart PCo Agent Instance and run queries and wait 60+ sec to see if that changes anything.

You can export the PCo and kepware logs and post here for review if you want.

Can you update PCo to the latest SP Patch? Kepware also has Version 5.15.585.0 out now as well. I would update both to the latest versions and try your tests again.

You can set up a single tag in the Subscription for the Agent instance, set an On True trigger and an expression for that tag  that never evaluates to True, Assign the tag an expression on the Output tab, and set a destination to the Simulation Destination.  When the agent is started, PCo will set up an Active subscription group with Kepware that will force Kepware to poll the tag for a value at the Update Rate defined in the Source System setting -- this is effectively what the OPC Quick Client does when you connect it.

Regards, Steve

[Former Member]

Hi Steve,

I skipped over the diagnostics to your last suggestion and it works!  This is great and thanks for your help.

HOWEVER:  Is this a valid and encouraged in-production setup?  It seems to rely on the notification firing in addition to PCO service and connecting.  What if KepWare is restarted, does the notification pick back up again?  (This is difficult for me to test immediately, OPC server in my case is the in production.)

To clarify, here's a log of my tests/results:

1. All off: 

- Data Clients:  0

2. Turn on OPC Test Client, configure 1 group and 1 tag in group:

- Data Clients:  1

- Active Tags:  1 of 1

3. Turn on PCO Agent:

- Data Clients:  2

- Active Tags:  1 of 1 (still...)

4. Query same tag from step 2 from MII:

- SUCCESS, NO DELAY

- Data Clients:  2

- Active Tags:  1 of 1 (still...)

5. Query new tag from MII:

- SUCCESS, NO DELAY

- Data Clients:  2

- Active Tags:  1 of 2

6. Turn off OPC Test Client

- Data Clients:  1

- Active Tags:  0 of 2 (doh!)

7. Wait 60+ sec, Query tag from MII:

- First try:  DELAY, ERROR

- Immediate second try:  SUCCESS, NO DELAY

- Data Clients:  1

- Active Tags:  0 of 2

8. Create Subscription Item of 1 DUMMY tag against Source System.  Create DUMMY PCO notification against the Agent (trigger expression will likely never evaluate, did not need to setup a destination) then start Agent

- Data Clients:  1

- Active Tags:  1 of 1 (hopeful!...)

9. Query new tag from MII:

- SUCCESS, NO DELAY (yay!)

- Data Clients:  1

- Active Tags:  1 of 2

[former_member196557]

Hi David,

OK, good news! But:

You should not have to invoke a "dummy" notification in PCo to maintain the PLC connection to Kepware. You need to investigate this using the steps I provided to get more details.

In the event of a Kepware restart, PCo will detect the OPC Server stop and restart with the Reliable Messaging configuration for the Source Agent. When the OPC Server restarts PCo will automatically reconnect to it and re-initiate the notification processing.

You can install and run Kepware without a license for 2 hours at a time if you need to set up a non-production test environment.

Regards, Steve

[Former Member]

Steve,

OK, understood.  Disabled the notification (though continued to work uninterrupted for 12 hours).

Some Results:

• Provide a screen shot of the Kepware Project Properties OPC DA tab.
  • I don't know where to find this(?)

• Reset the Kepware Event Log and then set it to log all the events (right click on the Event Log area to access the settings and reset option).
  • Event Log isn't logging many events as far as I can see, despite all event types being turned on.  Doesn't tell me anything regardless.

• Set the PCo Source System Reliable Connections to 9999 Max. No of Retries, and 30 seconds interval.
  • Tried, no difference

• Set the PCo Agent Instance Log level to Verbose and restart the Agent Instance.
• Execute a Query from MII, then wait 60+ seconds to see if there are any errors or warnings listed in either the Pco Log or the Kepware log.
  • Did this and have attached the log result below (sorry for format).  Log will show turning the Agent on (1 item subscribed, for what it is worth) then requesting 1 tag.  Notice the ~23 sec. delay between receiving the query and retrieval.

• Disable the Auto-Device Demotion in the Kepware Device settings, restart PCo Agent Instance and run queries and wait 60+ sec to see if that changes anything.
  • Tried, no difference

I haven't upgraded PCO or KepWare yet.

From what I can tell, the direct cause is that the subscription items in the PCO agent aren't getting activated.  When I add 2 items using the OPC Quick Client they get added to the "Active Items" list as "2 of 2".  When I add the same to items to the PCO Agent I get "0 of 0".  How do I activate the PCO subscription items?

Regards,

Dave

9/16/2014 9:41:12 AM 403    Verbose    .    10    11600    KEP30    OpcDaAgent    OPC DA Server status (Vender Info: Kepware, Server Version: 5.14.493) returned the following state: Running   

9/16/2014 9:41:12 AM 403    Verbose    .    10    11600    KEP30    AgentBase    IsConnected returned TRUE   

9/16/2014 9:41:12 AM 403    Verbose    .    10    11600    KEP30    AgentBase    connection check   

9/16/2014 9:40:42 AM 403    Verbose    .    16    11600    KEP30    AgentBase    IsConnected returned TRUE   

9/16/2014 9:40:42 AM 403    Verbose    .    16    11600    KEP30    OpcDaAgent    OPC DA Server status (Vender Info: Kepware, Server Version: 5.14.493) returned the following state: Running   

9/16/2014 9:40:42 AM 403    Verbose    .    16    11600    KEP30    AgentBase    connection check   

9/16/2014 9:40:39 AM 340    Information    .    6    11600    KEP30    PCoQueryRequestHandler    Query Processing Time: 23483.9241 milliseconds   

9/16/2014 9:40:39 AM 310    Verbose    .    6    11600    KEP30    OpcDaAgent    Retrieve for Tag FRM_PMON/CLx_PMON/DLS_WUCS_PO_0 [FRM_PMON.CLx_PMON.DLS_WUCS_PO_0] succeeded.   

9/16/2014 9:40:39 AM 280    Verbose    .    6    11600    KEP30    OpcDaAgent    Retrieve Tag Query recieved for 1 items.   

9/16/2014 9:40:32 AM 187    Verbose    .    6    11600    KEP30    OpcDaAgent    Secondary query for Tag FRM_PMON/CLx_PMON/DLS_WUCS_PO_0 [FRM_PMON.CLx_PMON.DLS_WUCS_PO_0] recieved.   

9/16/2014 9:40:16 AM 060    Verbose    .    6    11600    KEP30    OpcDaAgent    Metadata query for Tag FRM_PMON/CLx_PMON/DLS_WUCS_PO_0 [FRM_PMON.CLx_PMON.DLS_WUCS_PO_0] recieved.   

9/16/2014 9:40:16 AM 030    Verbose    .    6    11600    KEP30    OpcDaAgent    Native Tag Resolve Query recieved for 1 items.   

9/16/2014 9:40:15 AM 920    Information    .    6    11600    KEP30    AgentBase    Tag Query:  RETRIEVE 'FRM_PMON/CLx_PMON/DLS_WUCS_PO_0' WHERE RowCount = 60, StartDate = '2014-09-16T08:40:15-0400', EndDate = '2014-09-16T09:40:15-0400';   

9/16/2014 9:40:15 AM 920    Information    .    6    11600    KEP30    AgentBase    Query:  <pco:tag xmlns:pco="uri:sap-pco-request"><![CDATA[RETRIEVE 'FRM_PMON&#x2f;CLx_PMON&#x2f;DLS_WUCS_PO_0' WHERE RowCount = 60, StartDate = '2014-09-16T08:40:15-0400', EndDate = '2014-09-16T09:40:15-0400';]]></pco:tag>   

9/16/2014 9:40:15 AM 857    Verbose    .    6    11600    KEP30    PCoQueryRequestHandler    Decoded request: <pco:request xmlns:pco="uri:sap-pco-request" pco:version="1.0"><pco:tag><![CDATA[RETRIEVE 'FRM_PMON&#x2f;CLx_PMON&#x2f;DLS_WUCS_PO_0' WHERE RowCount = 60, StartDate = '2014-09-16T08:40:15-0400', EndDate = '2014-09-16T09:40:15-0400';]]   

9/16/2014 9:40:15 AM 857    Verbose    .    6    11600    KEP30    PCoQueryRequestHandler    Raw request string: [MII | <?xml version="1.0" encoding="UTF-8"?><pco:request xmlns:pco="uri:sap-pco-request" pco:version="1.0"><pco:tag><![CDATA[RETRIEVE 'FRM_PMON&#x2f;CLx_PMON&#x2f;DLS_WUCS_PO_0' WHERE RowCount = 60, StartDate = '2014-09-16T08:40:15-04   

9/16/2014 9:40:12 AM 483    Information    .    11    11600    KEP30    AgentBase    Start ignored.  Agent state = Started   

9/16/2014 9:40:12 AM 420    Information    .    8    11600    KEP30    Host    WCF service is started.   

9/16/2014 9:40:12 AM 420    Verbose    .    11    11600    KEP30    AgentBase    started   

9/16/2014 9:40:12 AM 390    Verbose    .    11    11600    KEP30    AgentBase    Namespace cache cleared.   

9/16/2014 9:40:12 AM 390    Information    .    11    11600    KEP30    AgentBase    Namespace Cache Loaded.   

9/16/2014 9:40:12 AM 390    Verbose    .    11    11600    KEP30    AgentBase    Namespace cache constructor mode=Demand   

9/16/2014 9:40:12 AM 390    Information    .    11    11600    KEP30    AgentBase    Retry Model = Active. Creating Connection Heartbeat to poll agent every 30 (secs) for connection status...   

9/16/2014 9:40:12 AM 390    Information    .    11    11600    KEP30    AgentBase    initializing Connection Monitor...   

9/16/2014 9:40:12 AM 390    Information    .    11    11600    KEP30    AgentBase    Agent RetryModel is: Active   

9/16/2014 9:40:12 AM 390    Verbose    .    11    11600    KEP30    OpcDaAgent    The OPC DA agent has completed startup.   

9/16/2014 9:40:12 AM 390    Verbose    .    11    11600    KEP30    OpcDaAgent    All items added.   

9/16/2014 9:40:12 AM 373    Verbose    .    11    11600    KEP30    OpcDaAgent    Adding items to the OPC DA group.   

9/16/2014 9:40:12 AM 373    Verbose    .    11    11600    KEP30    OpcDaAgent    Tag Group created.   

9/16/2014 9:40:12 AM 373    Information    .    11    11600    KEP30    OpcDaAgent    The Update Rate of the Tag Query Group was set to 1000 ms   

9/16/2014 9:40:12 AM 343    Verbose    .    11    11600    KEP30    OpcDaAgent    Creating the OPC DA group for Tag Query.   

9/16/2014 9:40:12 AM 343    Verbose    .    11    11600    KEP30    OpcDaAgent    Connection established.   

9/16/2014 9:40:12 AM 297    Verbose    .    11    11600    KEP30    OpcDaAgent    Connecting to the OPC DA source.   

9/16/2014 9:40:11 AM 530    Verbose    .    11    11600    KEP30    OpcDaAgent    Using connection string : localhost:Kepware.KEPServerEX.V5[DA 3.00]   

9/16/2014 9:40:11 AM 530    Verbose    .    11    11600    KEP30    OpcDaAgent    Establising the connection.   

9/16/2014 9:40:10 AM 047    Verbose    .    11    11600    KEP30    AgentBase    starting...   

9/16/2014 9:40:09 AM 217    Information    .    8    11600    KEP30    Host    implicitly calling start on the hosted IConnectivityAgent service contract   

9/16/2014 9:40:08 AM 763    Information    .    8    11600    KEP30    Host    using default NetTcpBinding endpoint for internal IConnectivityAgent proxy   

9/16/2014 9:40:08 AM 763    Verbose    .    8    11600    KEP30    Host    Service endpoints:   

9/16/2014 9:40:08 AM 763    Verbose    .    8    11600    KEP30    Host    SAP.Manufacturing.Connectivity.Agents.Opc.Da.OpcDaAgent is running with the following endpoints:   

9/16/2014 9:40:08 AM 717    Information    .    8    11600    KEP30    Host    WCF service is starting.   

9/16/2014 9:40:08 AM 670    Verbose    .    1    11600    KEP30    OpcDaAgent    Creating the OPC DA agent.   

9/16/2014 9:40:08 AM 670    Verbose    .    1    11600    KEP30    BrowserSessionManager    Created browse manager.   

9/16/2014 9:40:08 AM 560    Verbose    .    1    11600    KEP30    OpcDaAgent    Creating the OPC DA agent.   

9/16/2014 9:40:08 AM 560    Verbose    .    1    11600    KEP30    BrowserSessionManager    Created browse manager.   

9/16/2014 9:40:08 AM 357    Verbose    .    1    11600    KEP30    Host    Running [KEP30] as a windows service...   

[former_member196557]

David,

OK, lets back up.

I am convinced the issue is not in PCo or you would be seeing OPC Server disconnect and reconnect messages in the PCo logs.

The 15-25sec delay in the initial query after 60+ seconds of inactivity sounds like Kepware is disconnecting the ControlLogix network connection in the Channel or Device.  when that is re-established, it does the Controller synchronization to update the Logical Tags addresses.

Project Properties: from Kepware Configuration, select File, Project Properties

Where did you find  a KepWare configuration setting set to close connections/sessions if no activity for 60 seconds? Please send a screen shot.

Regards, Steve

[Former Member]

Steve,

Below are screenshots of the Session Management, and OPC DA Project Properties.

Regards,

Dave

[former_member196557]

Hi David,

OK. The settings on the Confuguration Tab are for connectivity to the the server_runtime service for the Kepware Configuration tool, not for OPC server connections.

Try this:

• Stop the PCo agent
• Disconnect th Quick Client
• wait 60+ seconds
• Connect the Quick Client and open the Items view to see the tag data -- how long does it take the tag data to appear?
• Are there any messages in the Kepware Event log related to the ControlLogix  Channel/Device connection (not necessarily errors)?

Next step: Install KepServerEX 5.15.585.0 and re-test...

Regards, Steve

[Former Member]

Steve,

Did what you suggested.  Results in OPC Quick Client:

• When adding the first ITEM (with default scan rates) to the GROUP, it takes 20-25 seconds for data to appear (similar to performance via PCO)
• When adding the second ITEM to the group, it takes < 1 second for data to appear
• After waiting 60+ seconds, can re-query any tag in < 1 second
• No errors in the KepWare event log

In the Quick Client, I can ACTIVATE or DEACTIVATE any item.  When I deactivate all items then the connection times out after 60 seconds of inactivity and a request takes 20-25 seconds again.  Quick Client items are ACTIVE by default.

What I'm deducing is that the PCO needs ACTIVE subscription items.  PCO subscription items are INACTIVE by default.  Creating a "dummy" or any PCO Agent notification seems to "Activate" the subscription items and ultimately keeps a session open.  Is there a way to activate PCO subscription items (am I missing that step)?

Next Step of upgrading KepWare will take a few days...  But have requested the relevant parties to do this.  Am not giving up that this can't be solved with PCO alone.

Regards,

Dave

[Former Member]

Steve,

Upgraded to latest version of KepWare with no differences on my original issue.

Eventual conclusion was (per our discussion):  update the Protocol Type to “Symbolic” then the initialization is not required and the tag retrieves immediately.  (In the default mode, it would scan all CLX tags to initialize every 60 sec. or so.  There are pros and cons of each mode, but in our case of low-volume this is the best option.)  PCO configuration remains unchanged and all settings default.

Dave

[former_member196557]

Hi Dave,

I recommend also that you raise a support ticket with Kepware on this issue as well. There is something in the landscape that is causing Kepware to determine that it needs to re-initialize the tags after the 60+ second period of inactivity. Recommend you include the version of Kepware and the models/firmware versions of the PLC to which you are connected.  Please update this thread with any resolution.

Regards, Steve