Can not start service nameserver

[Former Member]

Dear All,

During system copy i've got error message at process recovery database:

recoverStart.trc

============================

2015-08-29T02:15:27+00:00  P045764      14f773cef55 ERROR   RECOVERY RECOVER DATA finished with error: [110092] Recovery failed in nameserver startup: check services against topology failed: mismatch: 0 statisticsservers configured in source topology, 1 in destination system

=============================

after that i run command HDB start and 've got this error:

nameserver_hanaprd.30001.000.trc

=============================

[44170]{-1}[-1/-1] 2015-09-01 04:13:50.113837 f NameServer       TREXNameServer.cpp(02861) : Can not start the service 'nameserver' at 'hanaprd:30001' responsible for the volume '1' because an error during recovery before -> stopping instance .

===================================

now sap hana not running, please advices.

Thanks & regards,

Deby Afandi

(Sorry my bad english)

[Former Member]

Hi Deby,

can you please tell me

what is HANA version of source ?

what is the HANA version of target ?

Regards,

Pavan Gunda

[Former Member]

Dear Pavan,

source hana version

# HDB version

HDB version info:

  version:             1.00.80.00.391861

  branch:              NewDB100_REL

  git hash:            not set

  git merge time:      not set

  weekstone:           0000.00.0

  compile date:        2014-05-23 12:00:36

  compile host:        ld7272.wdf.sap.corp

  compile type:        opt

target hana version

# HDB version

HDB version info:

  version:             1.00.80.00.391861

  branch:              NewDB100_REL

  git hash:            not set

  git merge time:      not set

  weekstone:           0000.00.0

  compile date:        2014-05-23 12:00:36

  compile host:        ld7272.wdf.sap.corp

  compile type:        opt

Thanks & Regards,

Deby Afandi

[Former Member]

Deby,

Could you please check following SAP note:

Literally we dried of all options

1820553 - Incompatibility of data backup and DB configuration

Regards

Anand.

[Former Member]

Dear Anand,

This sap note for recovery error number [448]:

"RECOVER DATA finished with error: [448] recovery could not be completed, volume 1, reached log position 0, [110044] The service 'nameserver' at '<HOST>:<PORT>' responsible for the volume '1' does crash".

my recovery error number [110092]:

"2015-08-29T02:15:27+00:00  P045764      14f773cef55 ERROR   RECOVERY RECOVER DATA finished with error: [110092] Recovery failed in nameserver startup: check services against topology failed: mismatch: 0 statisticsservers configured in source topology, 1 in destination system"

i have tried to search on google with key "sap hana recovery 110092" and found sites

Recovery failed | SCN

Recovery of SAP HANA database fails | HANA Notes | ERP Champs

probably due to the log, as in the db2 database after the restore process must Rollforward

Regards,

Deby Afandi

[Former Member]

Hi Deby,

Taking the log backup.....This was suggested in our initial replies itself.

You could try and let me know.. if this helps.

Regards

Anand

[Former Member]

Dear Anand,

You mean I have to delete the log backups on the target and fill it with log backups from the source?

Thanks & Regards,

Deby Afandi

[Former Member]

Dear Anand,

During recovery logbackup is running, below log from backup.trc:

2015-08-28T12:23:53+00:00  P012820      14f7443a34b INFO    LOGBCKUP progress of service: nameserver, hanaprdkti:30001, volume: 1, 0% 0/98304

2015-08-28T12:23:53+00:00  P012820      14f7443a34b INFO    LOGBCKUP progress of service: nameserver, hanaprdkti:30001, volume: 1, 100% 98304/98304

2015-08-28T12:23:53+00:00  P012820      14f7443a367 INFO    LOGBCKUP state of service: nameserver, hanaprdkti:30001, volume: 0, BackupExecuteCatalogBackupInProgress

2015-08-28T12:23:54+00:00  P012820      14f7443a367 INFO    LOGBCKUP state of service: nameserver, hanaprdkti:30001, volume: 0, BackupExecuteCatalogBackupFinished

2015-08-28T12:23:54+00:00  P012820      14f7443a34b INFO    LOGBCKUP state of service: nameserver, hanaprdkti:30001, volume: 1, BackupLogSuccess

2015-08-28T12:24:11+00:00  P006438      14f7443e795 INFO    RECOVERY RECOVER DATA started

2015-08-28T12:24:11+00:00  P006438      14f7443e795 INFO    RECOVERY command: RECOVER DATA ALL USING FILE ('/sapmnt/hana_backup/PRD/25') CLEAR LOG

2015-08-28T12:24:11+00:00  P006438      14f7443e795 INFO    RECOVERY state of service: nameserver, hanaprdkti:30001, volume: 0, RecoveryExecuteCatalogRecoveryInProgress

2015-08-28T12:24:11+00:00  P006438      14f7443e795 INFO    RECOVERY state of service: nameserver, hanaprdkti:30001, volume: 0, RecoveryExecuteCatalogRecoveryFinished

2015-08-28T12:24:11+00:00  P006438      14f7443e795 INFO    RECOVERY using backup catalog 1440764633959 from file /usr/sap/PRD/HDB00/backup/log/log_backup_0_0_0_0.1440764633959

2015-08-28T12:24:11+00:00  P006438      14f7443e795 INFO    RECOVERY state of service: nameserver, hanaprdkti:30001, volume: 1, RecoveryExecuteTopologyRecoveryInProgress

2015-08-28T12:24:12+00:00  P006438      14f7443e795 INFO    RECOVERY state of service: nameserver, hanaprdkti:30001, volume: 1, RecoveryExecuteTopologyAndSSFSRecoveryFinished

2015-08-28T12:24:12+00:00  P006438      14f7443e795 ERROR   RECOVERY RECOVER DATA finished with error: [110092] Recovery failed in nameserver startup: check services against topology failed: mismatch: 0 statisticsservers configured in source topology, 1 in destination system

2015-08-29T02:15:26+00:00  P045764      14f773cef55 INFO    RECOVERY RECOVER DATA started

2015-08-29T02:15:26+00:00  P045764      14f773cef55 INFO    RECOVERY command: RECOVER DATA  USING FILE ('/sapmnt/hana_backup/PRD/25')  CLEAR LOG

2015-08-29T02:15:26+00:00  P045764      14f773cef55 INFO    RECOVERY state of service: nameserver, hanaprdkti:30001, volume: 0, RecoveryExecuteCatalogRecoveryInProgress

2015-08-29T02:15:26+00:00  P045764      14f773cef55 INFO    RECOVERY state of service: nameserver, hanaprdkti:30001, volume: 0, RecoveryExecuteCatalogRecoveryFinished

2015-08-29T02:15:26+00:00  P045764      14f773cef55 INFO    RECOVERY using backup catalog 1440764633959 from file /usr/sap/PRD/HDB00/backup/log/log_backup_0_0_0_0.1440764633959

2015-08-29T02:15:26+00:00  P045764      14f773cef55 INFO    RECOVERY state of service: nameserver, hanaprdkti:30001, volume: 1, RecoveryExecuteTopologyRecoveryInProgress

2015-08-29T02:15:27+00:00  P045764      14f773cef55 INFO    RECOVERY state of service: nameserver, hanaprdkti:30001, volume: 1, RecoveryExecuteTopologyAndSSFSRecoveryFinished

2015-08-29T02:15:27+00:00  P045764      14f773cef55 ERROR   RECOVERY RECOVER DATA finished with error: [110092] Recovery failed in nameserver startup: check services against topology failed: mismatch: 0 statisticsservers configured in source topology, 1 in destination system

Regards,

Deby Afandi

[Former Member]

Deby,

During recovery log backup are written in file as shown in above location.

Ex:-The file system probably has a limit of 10GB.when it is full. it might not have space to write additional log.

When a Backup of Logs is triggered it will save in different location as set and delete the existing logs from the file.freeing up the log file.

Log Backups - SAP HANA Administration Guide - SAP Library

&amp;raquo; SAP HANA Tutorial

some of the FAQs are answered in SAP note :- 1642148

Regards

Anand

[Former Member]

Hello Deby,

Please check following SAP note:-

1905462 - HANA scale-out landscape : topology mismatch leads to recovery failure

I hope this helps you in addressing the issue.

Regards

Anand

[Former Member]

Dear Anand,

I've removed daemon.ini from directory :/usr/sap/PRD/HDB00/hanaprd and run HDB start got same error :

[50252]{-1}[-1/-1] 2015-09-01 05:09:29.676725 i Basis            TraceStream.cpp(00396) : ==== Starting hdbnameserver, version 1.00.80.00.391861 (NewDB100_REL), build linuxx86_64 not set 2014-05-23 12:00:36 ld7272.wdf.sap.corp gcc (SAP release 20130806, based on SUSE gcc47-4.7.2_20130108-0.15.45) 4.7.2 20130108 [gcc-4_7-branch revision 195014]

[50252]{-1}[-1/-1] 2015-09-01 05:09:29.676827 i Basis            TraceStream.cpp(00401) : MaxOpenFiles: 1048576

[50252]{-1}[-1/-1] 2015-09-01 05:09:29.676856 i Memory           MallocProxy.cpp(01181) : Installed malloc hooks

[50252]{-1}[-1/-1] 2015-09-01 05:09:29.676858 i Basis            Timer.cpp(00584) : Using RDTSC for HR timer

[50252]{-1}[-1/-1] 2015-09-01 05:09:29.676859 i Memory           AllocatorImpl.cpp(01219) : Allocators activated

[50252]{-1}[-1/-1] 2015-09-01 05:09:29.676859 i Memory           AllocatorImpl.cpp(01235) : Using big block segment size 268435456

[50252]{-1}[-1/-1] 2015-09-01 05:09:29.676861 i Basis            ProcessorInfo.cpp(00746) : Using GDT segment limit to determine current CPU ID

[50252]{-1}[-1/-1] 2015-09-01 05:09:29.676862 w Environment      Environment.cpp(00286) : Changing environment set IMSLERRPATH=/usr/sap/PRD/HDB00/exe//

[50252]{-1}[-1/-1] 2015-09-01 05:09:29.676863 w Environment      Environment.cpp(00286) : Changing environment set IMSLSERRPATH=/usr/sap/PRD/HDB00/exe//

[50252]{-1}[-1/-1] 2015-09-01 05:09:29.691445 w Environment      Environment.cpp(00286) : Changing environment set SSL_WITH_OPENSSL=0

[50252]{-1}[-1/-1] 2015-09-01 05:09:29.893657 e tns_ddl          TNSClient.cpp(00917) : setServiceType no topology databaseId use service default:  2

[50265]{-1}[-1/-1] 2015-09-01 05:09:29.918926 i assign           TREXNameServer.cpp(01502) : assign as master nameserver. assign to volume 1 started

[50265]{-1}[-1/-1] 2015-09-01 05:09:29.919770 i FileIO           FileStatistics.cpp(00287) : FileFactoryConfiguration::initial(path="/usr/sap/PRD/SYS/global/hdb/backint/", AsyncWriteSubmitActive=auto,AsyncWriteSubmitBlocks=new,AsynReadSubmit=off,#SubmitQueues=1,#CompletionQueues=1)

[50265]{-1}[-1/-1] 2015-09-01 05:09:29.921149 i FileIO           FileStatistics.cpp(00287) : FileFactoryConfiguration::initial(path="/usr/sap/PRD/SYS/global/hdb/backint/", AsyncWriteSubmitActive=auto,AsyncWriteSubmitBlocks=new,AsynReadSubmit=off,#SubmitQueues=1,#CompletionQueues=1)

[50265]{-1}[-1/-1] 2015-09-01 05:09:29.923715 e Backup           Backup_Recover.cpp(00095) : exception: exception  1: no.110043  (Backup/impl/Backup_Recover.cpp:86)

    Can not start the service 'nameserver' at 'hanaprdkti:30001' responsible for the volume '1' because an error during recovery before

exception throw location:

1: 0x00007f2682c79694 in Backup::Backup_Recover::lookForRecoveryRequest(Backup::BackupCbc_Parser const&, Backup::BackupCbc_Nameserver*, Backup::Backup_ExtendedRecoveryInformation&, ltt_adp::basic_stringstream<char, ltt::char_traits<char> >&)+0xd0 at Backup_Recover.cpp:86 (libhdbbackup.so)

2: 0x00007f2688def7bc in NameServer::TREXNameServer::assign(NameServer::ServiceStartInfo&)+0x3158 at TREXNameServer.cpp:1514 (libhdbns.so)

3: 0x00007f2688e1ebb7 in NameServer::SelfAssignThread::run(void*)+0x13 at TREXNameServer.cpp:302 (libhdbns.so)

4: 0x00007f26877b2132 in TrexThreads::PoolThread::run()+0x850 at PoolThread.cpp:265 (libhdbbasement.so)

5: 0x00007f26877b3c98 in TrexThreads::PoolThread::run(void*&)+0x14 at PoolThread.cpp:124 (libhdbbasement.so)

6: 0x00007f2685e541ef in Execution::Thread::staticMainImp(void**)+0x98b at Thread.cpp:475 (libhdbbasis.so)

7: 0x00007f2685e5473d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:545 (libhdbbasis.so)

[50265]{-1}[-1/-1] 2015-09-01 05:09:29.939786 f NameServer       TREXNameServer.cpp(02861) : Can not start the service 'nameserver' at 'hanaprdkti:30001' responsible for the volume '1' because an error during recovery before -> stopping instance ...

[50252]{-1}[-1/-1] 2015-09-01 05:09:29.941324 i Service_Shutdown TrexService.cpp(05738) : Preparing for shutting service down

[50265]{-1}[-1/-1] 2015-09-01 05:09:29.945792 e Basis            TREXNameServer.cpp(02891) : Process exited due to an error via explicit exit call with exit

Thanks & regards,

Deby Afandi

[Former Member]

Hello Deby,

Can you manually start nameserver and try again.

Regards

Anand

[Former Member]

Dear Anand,

hanaprdkti:/usr/sap/PRD/HDB00/exe> ./hdbnameserver

unclean shutdown of service instance with pid 50252.

service startup...

accepting requests at 127.0.0.1:30001; 127.0.0.2:30001; 192.168.20.101:30001

service started

searching for master nameserver hanaprdkti:30001 ...

assign as master nameserver. assign to volume 1 started

Can not start the service 'nameserver' at 'hanaprdkti:30001' responsible for the volume '1' because an error during recovery before -> stopping instance ...

cannot send signal. (2, No such file or directory)

prepare for shutting service down...

stop ClockMonitor thread...

setInactive(nameserver@hanaprdkti:30001)

Thanks & Regards,

Deby Afandi

[Former Member]

Deby,

Have taken log backup

/backup/HDB/log

I believe during system recovery process you might not have disabled log archive option.

Please take log backup and try again.

Check following SAP note too:-

1944052 - Recovery failed due to hdbdaemon process not able to stop

I hope this helps.

Regards

Anand

[Former Member]

Dear Anand,

Thanks for your replies, i'm not disable log archive option before recovery process. I'm run recovery from SWPM - System Copy.

I've been following this note sap note http://service.sap.com/sap/support/notes/1944052

#cat hanaprdkti/trace/recoverStart.trc

own pid: 44847

recoverSys started: 2015-08-29 02:15:23

testing master: hanaprdkti

hanaprdkti is master

creating file recoverInstance.sql

restart database

restart master nameserver: 2015-08-29 02:15:23

start system: hanaprdkti

start system: hanaprdkti failed: start of nameserver failed

recoverSys failed: 2015-08-29 02:15:28

2015-08-29T02:15:27+00:00  P045764      14f773cef55 INFO    RECOVERY state of service: nameserver, hanaprdkti:30001, volume: 1, RecoveryExecuteTopologyAndSSFSRecoveryFinished

2015-08-29T02:15:27+00:00  P045764      14f773cef55 ERROR   RECOVERY RECOVER DATA finished with error: [110092] Recovery failed in nameserver startup: check services against topology failed: mismatch: 0 statisticsservers configured in source topology, 1 in destination system

#HDB proc

hanaprdkti:/usr/sap/PRD/HDB00> HDB proc

USER       PID  PPID %CPU    VSZ   RSS COMMAND

prdadm    5167  3420  0.0  12912  1720  \_ /bin/sh /usr/sap/PRD/HDB00/HDB proc

prdadm    5192  5167  0.0  12912   564      \_ /bin/sh /usr/sap/PRD/HDB00/HDB proc

prdadm   42901     1  0.0  13352   632 /bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session

prdadm   31838     1  0.0 221576 75744 /usr/sap/PRD/HDB00/exe/sapstartsrv pf=/usr/sap/PRD/SYS/profile/PRD_HDB00_hanaprdkti -D -u prdadm

Regards,

Deby afandi

[Former Member]

Hello Deby,

2088228 - Recovery from backup fails on a newly re-installed HANA system.

Regards

Anand

[Former Member]

Dear Anand,

This sap note

topology failed: mismatch: 0 statisticsservers configured in source topology, 1 in destination system

Check daemon.ini file at source:

# cat hanaprd/daemon.ini

[sapwebdisp]

instances = 1

[xsengine]

instances = 1

check daemon.ini file at target:

#cat /usr/local/src/daemon.ini.backup

[statisticsserver]

instances=1

[xsengine]

instances=1

[sapwebdisp]

instances=1

i've been remove line

[statisticsserver]

instances=1

and try to start, got same error

Check Myhana db version SPS08

# HDB version

HDB version info:

  version:             1.00.80.00.391861

  branch:              NewDB100_REL

  git hash:            not set

  git merge time:      not set

  weekstone:           0000.00.0

  compile date:        2014-05-23 12:00:36

  compile host:        ld7272.wdf.sap.corp

  compile type:        opt

Do I need to create a new hana instance?

Thanks

Deby afandi

[Former Member]

Deby,

We might have followed similar steps.

Seems we are missing something small.

If you have time...then please create SAP ticket with high priority.

Regards

Anand

[Former Member]

Dear Anand,

Many thanks for your advices

Regards,

Deby Afandi