Gateway - connection to partner (sapgw00) broken

[tobias_ptz]

Hi,

my problem is a bit tricky, but I hope someone can help me ..

Server:

SAP ERP 6.03

Oracle 10g

Windows 2k3 SR2 SP2

Client:

Windows 7

SAP GUI 730  PL 5

3. Party Tool: CDI (Cenit) (Java Tool (RFC))

Connection:

Client > CDI > SAPRouter > SAP > SAPRouter > SAPGUI/CDI > SAPRouter > SAP > Contentserver > SAPGUI/CDI

The problem:

The client uses the CDI tool to do a selection and to connect to the SAP Server. At one point, we get the following gateway error:

ERROR connection to partner SAPSYSTEM:sapgw00 broken

The annoying thing is, the termination is like a 50:50 chance .. sometimes the connection breaks and sometime not.

I've traced the SMGW in level 3, and found the spot where the gateway send back a return code to the cpic partner, but not which! (LINE 39.) And a bit later, the CONVID was deleted. (LINE 59. / 81.)

********** GWRD LOOP **********
I'm in GwReceive
SiSelNSelect: start select (timeout=501)
SiSelNNext: sock 1352 selected (revt=r--)
NiSelIListInsert: add hdl 57 [7] to sel-list (0) of set0
NiSelISelectInt: 1 handles selected (0 buffered)
main: NiSelNext (ni_hdl=57, read=1, write=0, connect=0, hdl_info=0000000002E82FE8)
request from R3_WORKPROCESS
NiIRead: hdl 57 received data (rcd=28,pac=2,MESG_IO)
GwReadSapWpRq: request from sap wp 4
-IN- tid         35            uid  1429         mode  0
-IN- appc_ca_blk 30            len  80           rq_id 2047
-IN- wait_f_answ 0             cmd  0 
DpGetCpicCommIdx: found index 6 via appc_hdr
DpGetCpicCommIdx: found comm entry 6 (tid/uid/mode/conv_id/a_r: 35/1429/0/25987468/0)
GwSendToRemGw: request for myself (len=80)
GwGetMemory: allocated 000000000FB9F0B0 (len=312)
GwGetMemory: act_overflow_size = 312 (+ 312)
REL APPC ca_blk 30
GwSendToRemGw: comp req = adr/total/alloc/hdr/param_hdr/data 000000000FB9F0B0/312/312/80/0/0
GwSearchConn3: found conn 2 without search
{root-id=35323742353235443532374235323544}_{conn-id=002264FBFE481ED391F298197F23AE06}_1
IN:
---
ReqType     : CPIC         SysIdx  : 0      ReqLen  : 80     ReqId   : 2580
CommIdx     : 6            ConnIdx : 2      ReqVer  : 6      Trace   : 0
RqType      : DEALLOCATE   Prot    : INT    UID     : 1429   Mode    : 0
User        :              SapRc   : 0      AppcRc  : 0      ConvId  : 25987468
Info2       : GW_IMMEDIATE
GwIHandleRq: decrement conv_no of client 0: 7
GwIHandleRq: decrement conv_no of client 0: 6
GwIHandleRq: own_index: 0
GwIHandleRq: set act rq type DEALLOCATE
R3DEAL(25987468, cpic_vector=0, ..)
I am the client (0)
R3DEAL: state of conn/index 2/0: RESET
R3DEAL: send return code to cpic partner
GwISendRc: send appc_rc/sap_rc/status 18/0/0 to conn/target/client 2/1/0
GwGetMemory: allocated 000000000FBB10B0 (len=1236)
GwGetMemory: act_overflow_size = 1548 (+ 1236)
GwSetCPICState: state of conn/index 2/1: RESET
GwSendToClient: send data to conn/target/client 2/1/0
OUT:
----
ReqType     : CPIC         SysIdx  : 0      ReqLen  : 80     ReqId   : 0
CommIdx     : 3            ConnIdx : 2      ReqVer  : 6      Trace   : 0
RqType      : SAP_SEND     Prot    : INT    UID     : 1431   Mode    : 0
User        :              SapRc   : 0      AppcRc  : 18     ConvId  : 25987468
Vector      :
Info        : SYNC_CPIC_FUNCTION
GwRqDpSendTo: (SAPSERVER / sapgw00, gw_id=0, appc_ca_blk=-1, len=80, comm_index=-1)
GwRqDpSendTo: (SAPSERVER / sapgw00, sys_idx=0, appc_ca_blk=-1, len=80, comm_index=-1)
DpLockAppcBlk: lock appc ca_blk 5
DpGetCpicCommIdx: found index 3 via appc_hdr
DpGetCpicCommIdx: found comm entry 3 (tid/uid/mode/conv_id/a_r: 29/1431/0/25987468/0)
GwFiSearchConvId: deleted 25987468 local, conn=2
GwRqDpSendTo: send close (bad rc or deallocate)
GwInitLocCommAdmEntry: init gw_comm_adm entry 3
GwListRemove: elem 3 not in comm_wait_list
GwListRemove: elem 3 not in comm_lookup_list
make DISP owner of appc_ca_blk 5
DpRqIPutIntoQ: put elem in queue NOWP, elems=1
DpRqPutIntoQueue: put request into queue (reqtype 0, prio LOW, rq_id 17820)
GwDispWakeUp: send wakeup with ni handle 17
NiIWrite: hdl 17 sent data (wrt=1,pac=1,MESG_IO)
-OUT- sender_id APPC_SERVER       tid  29    wp_ca_blk   -1      wp_id -1
-OUT- action    SEND_TO_WP        uid  1431   appc_ca_blk 5       type  NOWP
-OUT- new_stat  NO_CHANGE         mode 0     len         80      rq_id 17820
GwFreeMemory: free 000000000FBB10B0 (len=1236)
GwFreeMemory: act_overflow_size = 312 (- 1236)
GwFreeMemory: free 000000000FB9F0B0 (len=312)
GwFreeMemory: act_overflow_size = 0 (- 312)
GwIDelR3Conn: idx/conv = ((0/RESET) | (0/RESET)), delete conv 2 from conv-table
GwClearConn: conv_no/tcp_conv_no: 8/8
GwClearConn: free r3 conv info
GwClearConn: free buffer info
GwClearConn: free remsh_usr info
GwFiSearchConvId: not found 25987468 (remove=1)
GwListInsert: insert elem 2 into conn_free_list (at begin)
GwListInsert: 492 elems in conn_free_list
GwListRemove: elem 2 not in conn_search_list
GwListRemove: elem 2 not in conn_write_list
GwListRemove: remove elem 2 from conn_inuse_list
GwListRemove: 8 elems in conn_inuse_list
DpDelCpicCommEntry: T35/M0 with in_use field , don't change
DpCommDelEntryFromIndex: del comm entry 6, prev=-1, next=-1, listFirst=-1, elems=0
DpDelCpicCommEntry: comm entry 6 (T35/U1429/M0) deleted (actEntries=10)
GwInitLocCommAdmEntry: init gw_comm_adm entry 6
GwListRemove: elem 6 not in comm_wait_list
GwListRemove: elem 6 not in comm_lookup_list

My problem is, I don't find any return code, so I don't know what kind of error I have (http://help.sap.com/saphelp_nw04/helpdata/en/d2/296d4353d446758671cb065d6f3b6a/content.htm)

In Line 37. the gateway switched to the state client, is that mean that the server (the real client) broke the connection?

What can I do to find the error?

Thanks for any advise!

Best regards,

Tobias

[tobias_ptz · Accepted solution]

It was the firewall itself, see

[tobias_ptz]

Hi,

Anyone who knows on which point I have an eye on?

After a few more tests, I just get in the tracefile every time just the message, that the convid was deleted, but I don't see is that because of an error, or if the deletion was correct.

Thank you ..

Tobias

[Former Member]

Hi,

I can only think about the bytes transferred.If it is the size of the file and network bandwidth might cause issues

GwReadFromRemGw: received 338 bytes

Thanks

Rishi Abrol

[tobias_ptz]

Hi guys,

I'm back with you error messages.

After a few tests with with other errors but no results, I don't found any error in the gateway log, we've done yesterday a new test with the following result.

Two ConvID's were created during the EnCo tool communicates with the SAP server.

ConvID 49108147 - create 14:07:44 - delete 14:30:59

ConvID 49109272 - create 14:07:45 - delete 14:27:59

And one ConvID for every file that the client upload to the content server (GUI Upload to Contentserver, GUI communicates with SAP that a files was uplaoded, SAP ask Contenserver if the file was really uploaded)

ConvID 50396758 - create 14:28:38 - delete 14:30:59

ConvID 50420249 - create 14:29:01 - delete 14:29:11

Some more Informations:

192.168.1.1 = Default Network Gateway

192.168.1.2 = SAP Server (SAP.domain.de) (SAP Gateway)

CLIENTXY = Client PC of the EnCo User

ConvID = 49108147

Don't know if the first LOOP is interesting, but the second is.

What I don't get is, the connection to the partner 192.168.1.1 was broken, so the connection to the default network gateway.

Does that mean in this test I've got a network error?

Does anyone else find some more details from the logs?

********** GWRD LOOP **********
I'm in GwReceive
SiSelNSelect: start select (timeout=20000)
Thu Nov 28 14:30:59 2013
SiSelNNext: sock 1156 selected (revt=r--)
NiSelIListInsert: add hdl 250 [34] to sel-list (0) of set0
NiSelISelectInt: 1 handles selected (0 buffered)
main: NiSelNext (ni_hdl=250, read=1, write=0, connect=0, hdl_info=0000000002E839A8)
NiIRead: hdl 250 received data (rcd=136,pac=2,MESG_IO)
GwHdlPendingRead: read 136 bytes (rc=ok) (MONITOR_NEXT_CMD)
GwFreeHdlInfoMemory: free 0000000002961660 (data)
GwIExecMonCmd: received cmd (READ_CONN_TBL2/-1)
GwFoundSysAddr: found addr
GwSendConnTbl: 5 active connection(s)
NiIWrite: hdl 250 sent data (wrt=2720,pac=1,MESG_IO)
GwPrepareExtHdlInfo: 0000000002961660 for data (136 bytes)
GwSelSet: set R-- (0000000002E839A8) for hdl 250
SiSelNSet: set events of sock 1156 to: rp-
********** GWRD LOOP **********
I'm in GwReceive
SiSelNSelect: start select (timeout=20000)
SiSelNNext: sock 1212 selected (revt=r--)
NiSelIListInsert: add hdl 102 [12] to sel-list (0) of set0
NiSelISelectInt: 1 handles selected (0 buffered)
main: NiSelNext (ni_hdl=102, read=1, write=0, connect=0, hdl_info=0000000002E831F0)
request from CLIENT
GwRequest: request from client 1021
GwGetMemory: allocated 000000001040C640 (len=34704)
NiHLGetHostName: found address 192.168.1.1 in cache
NiIRead: SiRecv failed for hdl 102/sock 1212
    (SI_ECONN_BROKEN/0; I4; ST; P=192.168.1.1:63420; L=192.168.1.2:3300)GwSaveErrInfo2: save err info (258)
***LOG S23=> GwIDisconnectClient, client disconnected (1021) [gwxxrd.c     10480]
***LOG S74=> GwIDisconnectClient, client disconnected ( CLIENTXY) [gwxxrd.c     10491]
***LOG S0R=> GwIDisconnectClient, client disconnected () [gwxxrd.c     10508]
***LOG S0I=> GwIDisconnectClient, client disconnected ( java) [gwxxrd.c     10521]
*****************************************************************************
*
*  LOCATION    SAP-Gateway on host SAP.domain.de / sapgw00
*  ERROR       connection to partner '192.168.1.1:63420' broken
*
*  TIME        Thu Nov 28 14:30:59 2013
*  RELEASE     720
*  COMPONENT   NI (network interface)
*  VERSION     40
*  RC          -6
*  MODULE      nixxi.cpp
*  LINE        4850
*  DETAIL      NiIRead: P=192.168.1.1:63420; L=192.168.1.2:3300
*  SYSTEM CALL recv
*  COUNTER     92520
*
*****************************************************************************
GwSaveErrInfo: save err info (258)
GwSysErr: reset frag write for conn 4 (49108147)
GwSysErr: delete conn 4 (49108147)
GwIDelR3Conn: send error message to server
GwSendRc3: send (appc_rc=CM_PRODUCT_SPECIFIC_ERROR / sap_rc=NI_READ_FAILED) to client 217.16.165.115 / sapdp00 (24)
GwSendRc3: don't export trace level
NiIWrite: hdl 209 sent data (wrt=338,pac=1,MESG_IO)
GwIDelR3Conn: decrement conv_no of client 24: 0
GwIDelR3Conn: idx/conv = ((-1/SEND) | (-1/RESET)), delete conv 4 from conv-table
GwClearConn: conv_no/tcp_conv_no: 4/4
GwClearConn: free err info
GwClearConn: free r3 conv info
GwClearConn: free buffer info
GwClearConn: free remsh_usr info
GwFiSearchConvId: deleted 49108147 local, conn=4
GwListInsert: insert elem 4 into conn_free_list (at begin)
GwListInsert: 4092 elems in conn_free_list
GwListRemove: elem 4 not in conn_search_list
GwListRemove: elem 4 not in conn_write_list
GwListRemove: remove elem 4 from conn_inuse_list
GwListRemove: 4 elems in conn_inuse_list
GwSelClear: clear RWC for hdl 102
SiSelNClear: set events of sock 1212 to: ---
NiSelIListRemove: remove hdl 102 [12] from sel-list (1) of set0
SiSelNRemove: removed sock 1212 (pos=12)
SiSelNRemove: removed sock 1212
NiSelIRemove: removed hdl 102
NiICloseHandle: shutdown and close hdl 102/sock 1212
GwIFreeMemForLU_TP: freed memory for sys 1021
GwListRemove: elem 1021 not in sys_r3_list
GwListRemove: elem 1021 not in sys_reg_list
GwListRemove: remove elem 1021 from sys_inuse_list
GwListRemove: 10 elems in sys_inuse_list
GwUpdateClient: act_sys: 10
GwIDisconnectClient: client 1021 disconnected
GwFreeMemory: free 000000001040C640 (len=34704)
********** GWRD LOOP **********
I'm in GwReceive
SiSelNSelect: start select (timeout=20000)
SiSelNNext: sock 1028 selected (revt=r--)
NiSelIListInsert: add hdl 330 [41] to sel-list (0) of set0
SiSelNNext: sock 36 selected (revt=r--)
NiSelIListInsert: add hdl 217 [27] to sel-list (1) of set0
NiSelISelectInt: 2 handles selected (0 buffered)
main: NiSelNext (ni_hdl=217, read=1, write=0, connect=0, hdl_info=0000000002E83808)
request from REM_GW
GwRemGwHandle: handle request from gateway (SAP.domain.de / sapdp00 idx=4)
DpLockAppcBlk: lock appc ca_blk 5
NiIRead: hdl 217 received data (rcd=338,pac=2,MESG_IO)
GwReadFromRemGw: received 338 bytes from gw SAP.domain.de / sapdp00 idx=4 after 2 timeouts
GwReadFromRemGw: gw SAP.domain.de / sapdp00 idx=4 has 1 active connections
GwFiInfoConvId: not found 49108147
GwRemGwHandle: received data for conv_ID 49108147
Adresse   Offset  Appc-Data from Gateway
------------------------------------------------------------------------
000000000FB35C90  000000  06cb0000 2f8e0000 01020000 00000000 |..../...........|
000000000FB35CA0  000016  01000000 00000000 00000000 00001100 |................|
000000000FB35CB0  000032  00000014 000000df 34393130 38313437 |........49108147|
000000000FB35CC0  000048  00000000 00000000 00000000 00000000 |................|
000000000FB35CD0  000064  00000000 00313130 30000000 000b0000 |.....1100.......|
000000000FB35CE0  000080  2a455252 2a003100 636f6e6e 65637469 |*ERR*.1.connecti|
000000000FB35CF0  000096  6f6e2074 6f207061 72746e65 72202731 |on to partner '1|
000000000FB35D00  000112  39322e31 36382e34 312e313a 36333432 |92.168.1.1:6342|
000000000FB35D10  000128  30272062 726f6b65 6e002d36 004e4920 |0' broken.-6.NI |
000000000FB35D20  000144  286e6574 776f726b 20696e74 65726661 |(network interfa|
000000000FB35D30  000160  63652900 37323000 3430006e 69787869 |ce).720.40.nixxi|
000000000FB35D40  000176  2e637070 00343835 30004e69 49526561 |.cpp.4850.NiIRea|
000000000FB35D50  000192  643a2050 3d313932 2e313638 2e34312e |d: P=192.168.1.|
000000000FB35D60  000208  313a3633 3432303b 204c3d31 39322e31 |1:63420; L=192.1|
000000000FB35D70  000224  36382e34 312e3131 323a3333 30300054 |68.1.12:3300.T|
000000000FB35D80  000240  6875204e 6f762032 38203134 3a33303a |hu Nov 28 14:30:|
000000000FB35D90  000256  35392032 30313300 72656376 00000039 |59 2013.recv...9|
000000000FB35DA0  000272  32353231 00534150 2d476174 65776179 |2521.SAP-Gateway|
000000000FB35DB0  000288  206f6e20 686f7374 20475349 5453542e | on host SAP.|
000000000FB35DC0  000304  6773692e 6974632d 727a2e64 65202f20 |domain.de / |
000000000FB35DD0  000320  73617067 77303000 00000000 2a455252 |sapgw00.....*ERR|
000000000FB35DE0  000336  2a00                                |*.              |
------------------------------------------------------------------------
IN:
---
ReqType     : CPIC         SysIdx  : -1     ReqLen  : 338    ReqId   : -1
CommIdx     : 11           ConnIdx : 0      ReqVer  : 6      Trace   : 0
RqType      : SAP_SEND     Prot    : 0      UID     : 12174  Mode    : 0
User        :              SapRc   : 223    AppcRc  : 20     ConvId  : 49108147
Vector      :
Info        : SYNC_CPIC_FUNCTION WITH_ERR_INFO
Info3       : GW_WITH_CODE_PAGE
GwFiSearchConvId: not found 49108147 (remove=1)
GwRemGwHandle: remote conversation id 49108147 removed (CM_PRODUCT_SPECIFIC_ERROR)
GwRqDpSendTo: (SAP.domain.de / sapdp00, gw_id=4, appc_ca_blk=5, len=338, comm_index=-1)
DpGetCpicCommIdx: found index 11 via appc_hdr
DpGetCpicCommIdx: found comm entry 11 (tid/uid/mode/conv_id/a_r: 30/12174/0/49108147/0)
GwFiSearchConvId: not found 49108147 (remove=1)
GwRqDpSendTo: send close (bad rc or deallocate)
GwInitLocCommAdmEntry: init gw_comm_adm entry 11
GwListRemove: elem 11 not in comm_wait_list
GwListRemove: elem 11 not in comm_lookup_list
make DISP owner of appc_ca_blk 5
DpRqIPutIntoQ: put elem in queue NOWP, elems=1
DpRqPutIntoQueue: put request into queue (reqtype 0, prio LOW, rq_id 63404)
GwDispWakeUp: send wakeup with ni handle 17
NiIWrite: hdl 17 sent data (wrt=1,pac=1,MESG_IO)
-OUT- sender_id APPC_SERVER       tid  30    wp_ca_blk   -1      wp_id -1
-OUT- action    SEND_TO_WP        uid  12174   appc_ca_blk 5       type  NOWP
-OUT- new_stat  NO_CHANGE         mode 0     len         338     rq_id 63404
main: NiSelNext (ni_hdl=330, read=1, write=0, connect=0, hdl_info=0000000002E83DB8)
request from CLIENT
GwRequest: request from client 1165
GwGetMemory: allocated 000000001040C640 (len=34704)
NiHLGetHostName: found address 192.168.1.1 in cache
NiIRead: SiRecv failed for hdl 330/sock 1028
    (SI_ECONN_BROKEN/0; I4; ST; P=192.168.1.1:63483; L=192.168.1.2:3300)GwSaveErrInfo2: save err info (258)
***LOG S23=> GwIDisconnectClient, client disconnected (1165) [gwxxrd.c     10480]
***LOG S74=> GwIDisconnectClient, client disconnected ( CLIENTXY) [gwxxrd.c     10491]
***LOG S0R=> GwIDisconnectClient, client disconnected () [gwxxrd.c     10508]
***LOG S0I=> GwIDisconnectClient, client disconnected ( java) [gwxxrd.c     10521]
*****************************************************************************
*
*  LOCATION    SAP-Gateway on host SAP.domain.de / sapgw00
*  ERROR       connection to partner '192.168.1.1:63483' broken
*
*  TIME        Thu Nov 28 14:30:59 2013
*  RELEASE     720
*  COMPONENT   NI (network interface)
*  VERSION     40
*  RC          -6
*  MODULE      nixxi.cpp
*  LINE        4850
*  DETAIL      NiIRead: P=192.168.1.1:63483; L=192.168.1.2:3300
*  SYSTEM CALL recv
*  COUNTER     92523
*
*****************************************************************************

Thanks for any advise!

Best regards,

Tobias

[Former Member]

yes, Pls try..

BR,

Prabhakar

[tobias_ptz]

FYI

Tests are still running .. I'll update this thread asap!

[Former Member]

Hi,

Do you have an entry for "localhost" in your /etc/hosts file?

Also if possible just "rebooted" the machine once.

BR,

Prabhakar

[tobias_ptz]

Hi Prabhakar,

we have, but it is not active:

# 127.0.0.1       localhost

Should I activate this entry?

Thanks.

Best regards,
Tobias

[Former Member]

Hi,

Not an expert in this as not worked on CDI. But does this happened when large dataflows or you try to move large files to content server. If yes you should involve Network team in it.

Second thing try to increase the value of this parameter and see if it helps.

gw/max_conn

Thanks

Rishi Abrol

[tobias_ptz]

Hi Rishi,

thank you for the hint, but we still set it up to 1000.

I forgot to say, that we increase many SMGW parameters, but with that was not the solution.

Of cause we thought about to involve the network team, but I first wanna know, why I don't get a return code ..

Thanks!

Best regards,

Tobias

[Former Member]

Hi,

can you try an increase the value more .

Second thing have you tied to identify if the issue is with large file or all files.

Thanks

Rishi abrol

[tobias_ptz]

Hi,

Thanks for your reply.

I'll increase the parameter and restart the system tonight.

That's a bit tricky, because I cannot test this by my self.

Hopefully some of the customer can help me out with this question.

Thanks.

Tobias