DB2 HADR Connectivity Problems between Primary and Aux Standby

[Former Member]

Experts-

I am trying to set up HADR for our ERP system with 3 nodes (Primary, Principal Standby and Aux Standby). The Primary and Principal Standby are present in AWS Virginia region and Auxiliary database node will be hosted in Oregon.

I am using NEARSYNC for Primary and Principal Standby replication.
i am using SUPERASYNC for Primary and Aux Standby replication.

Problem- As soon as I start the Aux Standby, it always remains in REMOTE CATCHUP PENDING with db2pd showing the connection to primary as DISCONNECTED.

I have opened all the relevant firewall ports between the two nodes but the HADR state never changes from REMOTE CATCHUP PENDING to REMOTE CATCHUP.

I can see following logs in Primary and Aux Standby-

Primary-
2014-12-15-01.43.16.140111-300 I31286928E459 LEVEL: Info
PID : 2669 TID : 140115892496128 PROC : db2sysc 0
INSTANCE: db2sr1 NODE : 000 DB : SR1 
HOSTNAME: sr1dbnode1
EDUID : 53 EDUNAME: db2hadrp.0.2 (SR1) 0
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrSendHsMsgNoDefer, probe:30539
DATA #1 : <preformatted>
A HDR_MSG_REDIRECT message was sent to <Aux IP>:55003

2014-12-15-01.43.37.125181-300 I31287388E459 LEVEL: Info
PID : 2669 TID : 140115892496128 PROC : db2sysc 0
INSTANCE: db2sr1 NODE : 000 DB : SR1 
HOSTNAME: sr1dbnode1
EDUID : 53 EDUNAME: db2hadrp.0.2 (SR1) 0
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrSendHsMsgNoDefer, probe:30539
DATA #1 : <preformatted>
A HDR_MSG_REDIRECT message was sent to <AUX IP>:55003

Aux Standby-
2014-12-15-01.41.30.994781-300 I22260318E458 LEVEL: Warning
PID : 2613 TID : 139655295002368 PROC : db2sysc 0
INSTANCE: db2sr1 NODE : 000 DB : SR1 
HOSTNAME: sr1dbnode3
EDUID : 52 EDUNAME: db2hadrs.0.0 (SR1) 0
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrHandleRemoteConn, probe:30160
MESSAGE : TCP socket connection accepted. Remote Host: <Primary IP> Port: 3005

2014-12-15-01.41.51.979192-300 I22260777E458 LEVEL: Warning
PID : 2613 TID : 139655295002368 PROC : db2sysc 0
INSTANCE: db2sr1 NODE : 000 DB : SR1 
HOSTNAME: sr1dbnode3
EDUID : 52 EDUNAME: db2hadrs.0.0 (SR1) 0
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrHandleRemoteConn, probe:30160
MESSAGE : TCP socket connection accepted. Remote Host: <Primary IP> Port: 3031

2014-12-15-01.42.12.964815-300 I22261236E458 LEVEL: Warning
PID : 2613 TID : 139655295002368 PROC : db2sysc 0
INSTANCE: db2sr1 NODE : 000 DB : SR1 
HOSTNAME: sr1dbnode3
EDUID : 52 EDUNAME: db2hadrs.0.0 (SR1) 0
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrHandleRemoteConn, probe:30160
MESSAGE : TCP socket connection accepted. Remote Host: <Primary IP> Port: 3056

2014-12-15-01.42.34.050819-300 I22261695E458 LEVEL: Warning
PID : 2613 TID : 139655295002368 PROC : db2sysc 0
INSTANCE: db2sr1 NODE : 000 DB : SR1 
HOSTNAME: sr1dbnode3
EDUID : 52 EDUNAME: db2hadrs.0.0 (SR1) 0
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrHandleRemoteConn, probe:30160
MESSAGE : TCP socket connection accepted. Remote Host: <Primary IP> Port: 3082

Secondary Logs-

2014-12-18-14.39.24.310896-300 E22562601E650         LEVEL: Error

PID     : 12537                TID : 140457434670848 PROC : db2sysc 0

INSTANCE: <IntsanceID>               NODE : 000            DB   : <DBSID>    

HOSTNAME: <Aux-Host>

EDUID   : 110                  EDUNAME: db2hadrs.0.0 (SR1) 0

FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrEduAcceptEvent, probe:20200

MESSAGE : Did not receive anything through HADR connection for the duration of

          HADR_TIMEOUT. Closing connection.

DATA #1 : String, 30 bytes

hdrCurrentTime/hdrLastRecvTime

DATA #2 : unsigned integer, 4 bytes

1418931564

DATA #3 : unsigned integer, 4 bytes

1418931443

P.S - All this while I face no problems b/w primary and principal standby. They are connected and are in "PEER" status.

Any help here will be greatly appreciated.

-Amit

[Former Member]

We were able to fix the error.

Thanks

Amit