HANA Nameserver not responding FAIL: process hdbdaemon HDB Daemon not running

[Former Member]

Hi All,

HANA server restarted due to power outage. When we try to start HDB once the server is UP we are facing name server issues.

When i check in Trace , i found below issue. Please help if anyone faced this issue. Thanks in Advance.

6144]{-1}[-1/-1] 2016-09-06 16:49:24.148687 e TNS	TNSClient.cpp(00756) : nameserver hprdn2:30201 not responding. retry in 5 sec...
[6144]{-1}[-1/-1] 2016-09-06 16:49:29.149197 e Stream	NetworkChannel.cpp(02672) : NetworkChannel::connectBlocking peer closed
[6144]{-1}[-1/-1] 2016-09-06 16:49:29.149246 e TrexNet	EndPoint.cpp(00329) : ERROR: failed to open channel 127.0.0.1:30201! reason: (connection refused)
[6144]{-1}[-1/-1] 2016-09-06 16:49:29.149256 e TrexNet	EndPoint.cpp(00329) : details:
[6144]{-1}[-1/-1] 2016-09-06 16:49:29.149494 e TNS	TNSClient.cpp(00821) : sendRequest setstarting to hprdn2:30201 failed with NetException. data=(S)databaseid=2|host=hprdn2|port=30207|(I)type=6|(B)watchdog=0|(N)node=|pid=6144|activated_at=2016-09-06 16:48:44.140|...|
[6144]{-1}[-1/-1] 2016-09-06 16:49:29.149508 e TNS	TNSClient.cpp(00775) : nameserver hprdn2:30201 not responding after 10 retries. giving up.
[6144]{-1}[-1/-1] 2016-09-06 16:49:29.149829 f indexserver	TREXIndexServer.cpp(00844) : Nameserver setStarting call failed with errorcode=5521 msg=nameserver communication timeout;
[6144]{-1}[-1/-1] 2016-09-06 16:49:29.150329 e assign	TREXIndexServer.cpp(00961) : Exception: exception  1: no.70005521  (TREXIndexServer2/TREXIndexServer.cpp:845)
	nameserver communication timeout; $message$=nameserver communication timeout;

exception throw location:

1: 0x00007f540ad2b76b in TRexAPI::TREXIndexServer::started()+0x6e7 at TREXIndexServer.cpp:845 (hdbnameserver)

2: 0x00007f540ad8db16 in nlsui_main+0x17b2 at TrexService.cpp:769 (hdbnameserver)

3: 0x00007f53e9e26194 in System::mainWrapper(int, char**, char**)+0x20 at IsInMain.cpp:302 (libhdbbasis.so)

4: 0x00007f53e57adc36 in __libc_start_main+0xe2 (libc.so.6)

[6144]{-1}[-1/-1] 2016-09-06 16:49:29.169061 f Service

TrexService.cpp(00772) : FATAL: set service started failed

[6144]{-1}[-1/-1] 2016-09-06 16:49:29.169365 i Service_Shutdown TrexService.cpp(00803) : Preparing for shutting service down

[9623]{-1}[-1/-1] 2016-09-06 16:49:29.169883 i Service_Shutdown TREXIndexServer.cpp(04325) : Preparing to shutdown

[9623]{-1}[-1/-1] 2016-09-06 16:49:29.170071 e Stream	NetworkChannel.cpp(02672) : NetworkChannel::connectBlocking peer closed
[9623]{-1}[-1/-1] 2016-09-06 16:49:29.170238 e TNS	TNSClient.cpp(00821) : sendRequest setstopping to hprdn2:30201 failed with NetException. data=(S)databaseid=2|host=hprdn2|port=30207|(I)type=6|(B)watchdog=0|
[9623]{-1}[-1/-1] 2016-09-06 16:49:29.170248 e TNS	TNSClient.cpp(00769) : process is terminating -> aborting nameserver retries

[former_member205280]

Hi Shree,

Can you post the most recent nameserver trace file?

You can find them at

/usr/sap/<SID>/HDB<instance number>/<hostname>/trace/nameserver_<hostname>.xxxxx.xxx.trc

Reards,

Michael

[Former Member]

Hi Michael,

when i am starting HDB its not creating name server trace. Its generating xsengine and daemon trace files.

I found below error in daemon and xsengine trace. When i try to start name server alone as well facing same issue.

6144]{-1}[-1/-1] 2016-09-06 16:49:24.148687 e TNS         

	TNSClient.cpp(00756) : nameserver hprdn2:30201 not responding. retry in 5 sec...
[6144]{-1}[-1/-1] 2016-09-06 16:49:29.149197 e Stream	NetworkChannel.cpp(02672) : NetworkChannel::connectBlocking peer closed
[6144]{-1}[-1/-1] 2016-09-06 16:49:29.149246 e TrexNet	EndPoint.cpp(00329) : ERROR: failed to open channel 127.0.0.1:30201! reason: (connection refused)
[6144]{-1}[-1/-1] 2016-09-06 16:49:29.149256 e TrexNet	EndPoint.cpp(00329) : details:
[6144]{-1}[-1/-1] 2016-09-06 16:49:29.149494 e TNS	TNSClient.cpp(00821) : sendRequest setstarting to hprdn2:30201 failed with NetException. data=(S)databaseid=2|host=hprdn2|port=30207|(I)type=6|(B)watchdog=0|(N)node=|pid=6144|activated_at=2016-09-06 16:48:44.140|...|
[6144]{-1}[-1/-1] 2016-09-06 16:49:29.149508 e TNS	TNSClient.cpp(00775) : nameserver hprdn2:30201 not responding after 10 retries. giving up.
[6144]{-1}[-1/-1] 2016-09-06 16:49:29.149829 f indexserver	TREXIndexServer.cpp(00844) : Nameserver setStarting call failed with errorcode=5521 msg=nameserver communication timeout;
[6144]{-1}[-1/-1] 2016-09-06 16:49:29.150329 e assign	TREXIndexServer.cpp(00961) : Exception: exception  1: no.70005521  (TREXIndexServer2/TREXIndexServer.cpp:845)
	nameserver communication timeout; $message$=nameserver communication timeout;

exception throw location:

1: 0x00007f540ad2b76b in TRexAPI::TREXIndexServer::started()+0x6e7 at TREXIndexServer.cpp:845 (hdbnameserver)

2: 0x00007f540ad8db16 in nlsui_main+0x17b2 at TrexService.cpp:769 (hdbnameserver)

3: 0x00007f53e9e26194 in System::mainWrapper(int, char**, char**)+0x20 at IsInMain.cpp:302 (libhdbbasis.so)

4: 0x00007f53e57adc36 in __libc_start_main+0xe2 (libc.so.6)

[6144]{-1}[-1/-1] 2016-09-06 16:49:29.169061 f Service

TrexService.cpp(00772) : FATAL: set service started failed

[6144]{-1}[-1/-1] 2016-09-06 16:49:29.169365 i Service_Shutdown TrexService.cpp(00803) : Preparing for shutting service down

[9623]{-1}[-1/-1] 2016-09-06 16:49:29.169883 i Service_Shutdown TREXIndexServer.cpp(04325) : Preparing to shutdown

[9623]{-1}[-1/-1] 2016-09-06 16:49:29.170071 e Stream	NetworkChannel.cpp(02672) : NetworkChannel::connectBlocking peer closed
[9623]{-1}[-1/-1] 2016-09-06 16:49:29.170238 e TNS	TNSClient.cpp(00821) : sendRequest setstopping to hprdn2:30201 failed with NetException. data=(S)databaseid=2|host=hprdn2|port=30207|(I)type=6|(B)watchdog=0|
[9623]{-1}[-1/-1] 2016-09-06 16:49:29.170248 e TNS	TNSClient.cpp(00769) : process is terminating -> aborting nameserver retries

[former_member205280]

Hi Shree,

In that case, you may have to check if there's any process running on port 30201 with the following command

netstat -anpl | grep :30201

Sometime delete the .sap* file in /tmp directory and restart the server may be helpful.

And one question, are you using scale-out landscape?

Regards,

Michael

[Former Member]

Hi Michael,

We have already restarted server twice and checked for port 30201 in netstat. But no luck

[former_member205280]

Hi,

Can you post the all available trace log files, if you don't mind you can send them to my email I'll try my best to help!

hu.michael@live.com

Regards,

Michael

[Former Member]

Hi Michael,

I have sent mail.

Regards,

Shree