HANA Replication - Alert "Connection between systems in system replication setup" (id 78)

[Former Member]

Hi

I'm getting this email alert from the HANA tenant database:

snapshot   : 2015-08-18 16:42:52.0000000

Alert Name : Connection between systems in system replication setup

Rating     : Error

Details    : At 2015-08-18 18:35:45.0000000 on hostp01:30103; Site 2: Communication channel closed

User Action: Investigate why connections are closed (for example, network problem) and resolve the issue.

I see more alerts in the trace files, don't know if they are related:

Tenant's database trace files:

[178728]{419183}[119/-1] 2015-08-18 20:56:11.225670 e cePlanExec cePlanExecutor.cpp(07183) : Error during Plan execution of model _SYS_STATISTICS:_SYS_SS_CE_1402084_140190768844608_4_INS (-1), reason: executor: plan operation failed;CalculationNode ($$_SYS_SS2_RESULT$$) -> operation (CustomLOp):Compilation failed; OpenChannelException at network layer:

  message: an error occured while opening the channel

  info:    connection refused

  param:   127.0.0.1:30110

Replication database trace files:

nameserver_alert_hostp01.trc

[42096]{-1}[-1/-1] 2015-08-18 18:45:18.355758 e TrexNet          EndPoint.cpp(00260) : ERROR: failed to open channel 127.0.0.1:30107! reason: (connection refused)

Please assist.

[Former Member · Accepted solution]

This seems to be communication problem between your primary and secondary. Check if your secondary system is up and running.

Mostly the above error will cause due to un avaibality of secondary system.

Manually start the secondary system if it is down and see if the error ocurs again. 

[Former Member · Accepted solution]

Hi

Thanks, the secondary database is up and running, and in the primary's HANA studio on the System Replication TAB the status is ACTIVE, and the LAST_LOG_POSTITION_TIME is up to date.

Unfortunately I still continue to receive the alert above every 5 min.

What else should I check?

Kind Regards

Willem

[Former Member · Accepted solution]

Strange !! can you ask your network team if there is any break in packt loss ?

If no, will you be able to attach the complete trace file

[Former Member · Accepted solution]

Hi Pavan

I have attached the trace files, please see attached.

As per the Network team, there is no firewall or packet loss between the 2 servers.

Kind Regards

Willem

trace_secondary.zip.txt.zip

trace_primary.zip.txt.zip

[Former Member · Accepted solution]

I am not able to open the log files

Please attach the Nameserver and index server log files completely.

[Former Member · Accepted solution]

Hi Pavan

Please rename the below to a .zip file, then extract with winzip.

I have also done the same with the nameserver trace for the primary, please see attached if you are able to open.

There is not a index server trace as it is embedded.

Regards

Willem

nameserver_jhbtfrlpsha01.30101.902.zip.txt.zip

[Former Member · Accepted solution]

No Luck!! File are not opening

[Former Member · Accepted solution]

Hi Pavan

Please remove the ".txt" from the file name, then open the file with winzip, please let me know if it works.

Thanks

Willem

[Former Member · Accepted solution]

Can you please set the below parameter keep_old_style_alert = false (default=true) . I belive you are still getting old fashion alerts.

See if it resolves the issue. If the issue still exists . Please execute below commands and paste the output

hdbcons -e hdbindexserver "replication info"

lsof –n –p

Regards,

Pavan Gunda

[Former Member · Accepted solution]

Hi Pavan

After I changed the parameter keep_old_style_alert , I no longer received this alerts

Thanks and Regards

Willem

[Former Member · Accepted solution]

I unfortunately notice this alert is still occurring.

I have attached the outputs of above.

Kind Regards

output.zip.txt.zip

[Former Member · Accepted solution]

See if you have maintained correctly in the tenant DB/instance and SYSTEMDB. I don't have much idea on multi tenant DB. And not sure, I am not able to open the files. Paste the output in reply

[Former Member]

Hi Willem,

Do you see the communication channel closed errors on the SYSTEM Replication TAB of the SYSTEMDB?

Do you get this error after SYSTEM Replication has completed the Full Sync of data or at the very first stage of System replication?

The common cause of the communication channel closed errors is a loss of connectivity or the unavailability of the secondary tier of system replication, please provide more details to understand your errors

Sunil