NWDI | Requests getting into queue

[Former Member]

Hi All,

We have NWDI system with 4 tracks.

Everytime the deployments are triggered they are getting into queue and then get processed.

With in TCS deployer, if we try to check the logs from below options,

we see logs as below first(Bad request screen) and then the second(try to connect to deploy tool) after some time. If we wait for more time, deployments get processed and completely new log gets updated.

If we manually trigger the deployments from CMS --> deployments, then the requests get processed at a little faster rate.

Is anybody in the forum facing such issue. Kindly assist/share how this issue is fixed.

Regards,

Ravi

[ErvinSzolke]

Hello Ravi,

Deployment in DEV and CONS is triggered in asynchronous way, because the TCS has to wait up the build and can deploy only afterwards. In TEST PROD this is done in a synchronous way since there the SCA already generated, i.e. no need to build anything, no need to wait to anyone and anything, and so the deployment can be triggered immediately.

I'd first recommend to check whether your RTS settings are fine, you can make a very quick test as per: (NWDI)(CMS)Q0002 - Java Development - SCN Wiki

Also, what is the Release version of the RTSs (the j2ee engine where you depoy) ?
What is the Release version of the Build plugins (dependent SCs) in the problematic track(s) ?

I mean under Release version numbers like 640, 700, 730, etc.

Thanks and Best Regards,

Ervin

[Former Member]

Hi Ervin,

Thanks for response. will check the quick test as per link suggested.

what is the Release version of the RTSs (the j2ee engine where you depoy) ?

-->SAP NW 7.31 sps12

What is the Release version of the Build plugins (dependent SCs) in the problematic track(s) ?

--> I hope this is about the actual NWDI system you are refering to and its SAP NW 7.31 sps10

Regards,

Ravi

[ErvinSzolke]

Hi,

it is not really about the version of the NWDI itself but the version of the SCs of the track we are talking about. That has to match to the version of the RTS. I just wanted to rule out with this that you are not trying to deploy say a 700 content to a 731 j2ee engine.

More details about this here: 

Let me know the outcome of the RTS tests.

I hope this helps.

Regards,

Ervin

[Former Member]

Hi Ervin,

Went through your blog. Thanks a lot. It gave me basic idea on how NWDI works &  which  areas to be used to troubleshooting.

We have 2 development teams. Say X & Y

X is using 7.3 sp12 PAT002

Y is using 7.3 sp8 PAT002

Runtime systems are on : NW 7.31 sps12

NOTE: I hope runtime systems are those configured under tracks where deployments happen.

Track version: Hope Below screenshot from Generate Landscape Graphic help in telling the same. Please correct if this is not the right way to know track version.

Help in knowing how to get same please if am wrong.

Connectivity test:

we have configured with option "Deploy controller" for all target systems. Access details are maintained and access details are same across the tracks/systems.

When we try to expand "View/Edit Deployment Substitution Parameters" below is our observation from each track.

X Release track: Dev opens easily, test & production takes long time and gives below screen

X Maintenance track: Dev opens easily, test & production takes long time and gives below screen

Y Release track: Dev opens easily, test & production takes long time and gives below screen

Y Maintenance track: Dev opens easily, test & production takes long time and gives below screen

We are not sure if this is an issue. Can you please guide us.

Regards,

Ravi

[former_member189220]

Hi,

I think this might help (I faced similar problems in the past).

Deployment stuck in TSC deployer queue. When triggered manually it continues. Sporadic problem(usually)

The problem might be caused by the fact that the servers are running on multiple IP addresses.

So do the involved servers have more than 1 multiple IP addresses?

If yes, are all these IP addresses seen by the other server? (does x1 see all IPs of x2 and does x2 see all IPs of x1?)

Most probably the P4 connection is unstable because of your multiple IP configuration. 

Would you please revise the dev_icm_sec_xx trace file whether there is a similar message:

...

Error: Protocol error (-21), client with this banner already exists: 1:10.50.23.56:50004 [p4_plg_mt.c 3548]

CONNECTION (id=17/26333):

used: 1, type: multiplexed, role: Server(1), stateful: 0

nihdl: 263, ssl: (nil), protocol: P4(9)

local host: 10.48.130.66:50004 ()

remote host: 10.50.23.56:45432 ()

...

Usually implementation of note

1158626 - P4 connection to ICM on machine with multiple IPs fails

helps to adjust the ICM configuration to properly work with the network configuration.

If this does not help, in order to troubleshoot this deployment case, you need to trace down the messages from these logs:

1. /usr/sap/<SID>/SYS/global/TCS/LOG/*

2. defaultTrace (as of 710 since there is no SDM any more)

3. /usr/sap/ <SID>/J<Instance>/work

the files dev_icm and dev_icm.old from deploy_target server in directory /usr/sap/ <SID>/J<Instance>/work for all instances

4. So you will have to mark the time of the starting of the deployment and when it finishes.

Then you have to refer to these logs and collect the messages in this time frame.

Analyse and eventually find the solution. I believe it is a kind of network issue.

Good luck!

Regards

[Former Member]

Thanks Milen for suggestion.

We do see  "client with this banner already exists" entry in  dev_icm_sec_xx

We had analyzed the note "1158626 - P4 connection to ICM on machine with multiple IPs fails"before.

But this is applicable for  SAP NetWeaver AS Java 7.1 SP5 and higher versions and

the problem has been fixed since 7.10 SP11.

We are already on SAP NW 7.31 sps12. Hence we ignored this note.

Was this same case with you as well. Systems at higher level & note was still applicable in your case.

--------------------------------------------------

Yes we do have 2 IP's for each of the systems.

Eg In Dev system currently we have entry as below. 

icm/server_port_2 = PROT=P4, PORT=54504

So as per note we need to make it as below.

icm/server_port_2 = PROT=P4, PORT=54504, HOST=<1 IP of the host>

In same way, we need to update for all respective systems involved in track.

Can you please correct me if am wrong.

Regards,

Ravi

[former_member189220]

Hi,

My point was, you to narrow-down the related messages in the above mentioned traces (for the time of deployment) and in these messages to cross-check whether you do have such error. (sorry for not being clear enough)

Only if there is such message, in this time-frame, then you might apply the note (& mainly for test reason, because the multiple IP addresses issue should have been solved). And you are right, this should be the settings. But as I wrote, this will be valid if we have this error in the time-frame of the deployment.

Provided, you do not have such messages in the related time-frame, then the issue is different one.

So far we haven't seen (or at least me) the related messages from

TCS log,

defaultTrace,

dev_icm.

The error indicates that the connection is not established.

It would be helpful to know if connectively fails completely to P4 at the time of the incident.

Capture the netstat -an | grep 54704 output directly on the target RTS to see status of any connections between NWDI and RTS.

In addition, it would help to increase the trace level of RunTime System(RTS)'s ICM to level 3 and get a new trace while reproducing.

Further, have you tried connecting to the P4 port directly right after the issue occurs?

You could do a quick check with 'telnet msplsa103.corp.medtronic.com 50004' from the DN1 system as well as directly on DMA system.

Regards