How to start sapstartsrv and sapstart?

[Former Member]

Hello experts,

I have a problem regarding start sap after I changed the IP's for HANA Instance and ECC APP server Instance.

First of all I checked firewalls to be disabled and I maintained the new IPs to /etc/host . The PING works between HANA OS and ECC OS and I can to connect to HDB with HANA Studio.

But when I run with <SID>adm the command startsap all pf=/usr/sap/HD1/profile/HD1_DVEBMGS00_awssap1 I receive the following message:

Database db must be started first

Log on to remote server and start database

How should I start sap in this case?

Best regards,

Teodor

This is the trans.log file

awssap1:hd1adm 21> cat trans.log

4 ETW000 R3trans version 6.25 (release 742 - 11.02.15 - 14:06:42).

4 ETW000 unicode enabled version

4 ETW000 ===============================================

4 ETW000

4 ETW000 date&time   : 05.07.2016 - 07:29:46

4 ETW000 control file: <no ctrlfile>

4 ETW000 R3trans was called as follows: R3trans -d

4 ETW000  trace at level 2 opened for a given file pointer

4 ETW000  [     dev trc,00000]  Tue Jul  5 07:29:46 2016                                                  36  0.000036

4 ETW000  [     dev trc,00000]  db_con_init called                                                        10  0.000046

4 ETW000  [     dev trc,00000]  set_use_ext_con_info(): usage of ssfs switched off (rsdb/ssfs_connect=0)

4 ETW000                                                                                                  13  0.000059

4 ETW000  [     dev trc,00000]  determine_block_commit: no con_hdl found as blocked for con_name = R/3

4 ETW000                                                                                                   7  0.000066

4 ETW000  [     dev trc,00000]  create_con (con_name=R/3)                                                  7  0.000073

4 ETW000  [     dev trc,00000]  Loading DB library '/usr/sap/HD1/SYS/exe/run/dbhdbslib.so' ...            16  0.000089

4 ETW000  [     dev trc,00000]  DlLoadLib() success: dlopen("/usr/sap/HD1/SYS/exe/run/dbhdbslib.so"), hdl 0, count 1, addr 1de1530

4 ETW000                                                                                                3672  0.003761

4 ETW000  [     dev trc,00000]  Library '/usr/sap/HD1/SYS/exe/run/dbhdbslib.so' loaded                    11  0.003772

4 ETW000  [     dev trc,00000]  function DbSlExpFuns loaded from library /usr/sap/HD1/SYS/exe/run/dbhdbslib.so

4 ETW000                                                                                                  10  0.003782

4 ETW000  [     dev trc,00000]  Version of '/usr/sap/HD1/SYS/exe/run/dbhdbslib.so' is "742.06", patchlevel (0.38)

4 ETW000                                                                                                  42  0.003824

4 ETW000  [     dev trc,00000]  function dsql_db_init loaded from library /usr/sap/HD1/SYS/exe/run/dbhdbslib.so

4 ETW000                                                                                                   9  0.003833

4 ETW000  [     dev trc,00000]  function dbdd_exp_funs loaded from library /usr/sap/HD1/SYS/exe/run/dbhdbslib.so

4 ETW000                                                                                                  16  0.003849

4 ETW000  [     dev trc,00000]  } DbSlHDBControl(rc=0)                                                    15  0.003864

4 ETW000  [     dev trc,00000]  { DbSlHDBControl(con_hdl=-1,command=39,arg_p=(nil))                        7  0.003871

4 ETW000  [     dev trc,00000]  } DbSlHDBControl(rc=0)                                                     6  0.003877

4 ETW000  [     dev trc,00000]  { DbSlHDBControl(con_hdl=-1,command=10,arg_p=7fff37dc0500)                 7  0.003884

4 ETW000  [     dev trc,00000]  } DbSlHDBControl(rc=0)                                                     5  0.003889

4 ETW000  [     dev trc,00000]  New connection 0 created                                                   5  0.003894

4 ETW000  [     dev trc,00000]  0: name = R/3, con_id = -000000001, state = DISCONNECTED, tx = NO , bc = NO , oc = 000, hc = NO , perm = YES, reco = NO , info = NO , timeout = 000, con_max = 255, con_opt = 255, occ = NO , prog =

4 ETW000                                                                                                  19  0.003913

4 ETW000  [     dev trc,00000]  { DbSlHDBControl(con_hdl=-1,command=10,arg_p=1896960)                     11  0.003924

4 ETW000  [     dev trc,00000]  } DbSlHDBControl(rc=0)                                                     5  0.003929

4 ETW000  [     dev trc,00000]  db_con_connect (con_name=R/3)                                              5  0.003934

4 ETW000  [     dev trc,00000]  determine_block_commit: no con_hdl found as blocked for con_name = R/3

4 ETW000                                                                                                  16  0.003950

4 ETW000  [     dev trc,00000]  find_con_by_name found the following connection:                           5  0.003955

4 ETW000  [     dev trc,00000]  0: name = R/3, con_id = 000000000, state = DISCONNECTED, tx = NO , bc = NO , oc = 000, hc = NO , perm = YES, reco = NO , info = NO , timeout = 000, con_max = 255, con_opt = 255, occ = NO , prog =

4 ETW000                                                                                                  11  0.003966

4 ETW000  [     dev trc,00000]  { DbSlHDBConnect(con_info_p=(nil))                                        21  0.003987

4 ETW000  [     dev trc,00000]  DBHDBSLIB : version 742.06, patch 0.038 (Make PL 0.101)                   12  0.003999

4 ETW000  [     dev trc,00000]  HDB shared library (dbhdbslib) patchlevels (last 10)                       8  0.004007

4 ETW000  [     dev trc,00000]    (0.038) Close all result sets (note 2124493)                             8  0.004015

4 ETW000  [     dev trc,00000]    (0.038) Initialization of deferred LOB writing for ZDO changed (note 2106186)

4 ETW000                                                                                                  10  0.004025

4 ETW000  [     dev trc,00000]    (0.032) DBSL supports deferred lob writing with HANA SP9 (note 2106186)

4 ETW000                                                                                                  11  0.004036

4 ETW000  [     dev trc,00000]    (0.031) Read last SQL message after the connection is closed (note 2104804)

4 ETW000                                                                                                  11  0.004047

4 ETW000  [     dev trc,00000]    (0.031) Check Enable_Select_Into_Scalar_UDF before the ALTER call (note 2105480)

4 ETW000                                                                                                   9  0.004056

4 ETW000  [     dev trc,00000]    (0.029) Certain SAP applications hang for several minutes (note 888312)

4 ETW000                                                                                                   9  0.004065

4 ETW000  [     dev trc,00000]    (0.029) Structure DBSL_CA initialized (note 2094457)                     8  0.004073

4 ETW000  [     dev trc,00000]    (0.027) Data aging for the service connection (note 2090613)             7  0.004080

4 ETW000  [     dev trc,00000]    (0.026) Use prepared statements for R3szchk (note 1952609)               7  0.004087

4 ETW000  [     dev trc,00000]    (0.025) Use bulk fetch on a result set of a procedure call (note 2082661)

4 ETW000                                                                                                   9  0.004096

4 ETW000  [     dev trc,00000]                                                                             4  0.004100

4 ETW000  [     dev trc,00000]   -> init()                                                                 8  0.004108

4 ETW000  [     dev trc,00000]    STATEMENT_CACHE_SIZE = 1000                                             28  0.004136

4 ETW000  [     dev trc,00000]   -> init()                                                               389  0.004525

4 ETW000  [     dev trc,00000]   -> loadClientRuntime()                                                    8  0.004533

4 ETW000  [     dev trc,00000]  Loading SQLDBC client runtime ...                                          6  0.004539

4 ETW000  [     dev trc,00000]  SQLDBC Module  : /usr/sap/HD1/hdbclient/libSQLDBCHDB.so                 2465  0.007004

4 ETW000  [     dev trc,00000]  SQLDBC Runtime : libSQLDBCHDB 1.00.097.00 Build 1434028111-1530           31  0.007035

4 ETW000  [     dev trc,00000]  SQLDBC client runtime is 1.00.097.00.1434028111                           13  0.007048

4 ETW000  [     dev trc,00000]   -> quiesceConnection()                                                    7  0.007055

4 ETW000  [     dev trc,00000]   -> getNewConnection()                                                     7  0.007062

4 ETW000  [     dev trc,00000]   <- getNewConnection(con_hdl=0)                                           37  0.007099

4 ETW000  [     dev trc,00000]   -> checkEnvironment(con_hdl=0)                                           11  0.007110

4 ETW000  [     dev trc,00000]   -> connect(con_info_p=(nil))                                             12  0.007122

4 ETW000  [     dev trc,00000]  Try to connect via secure store (DEFAULT) on connection 0 ...             23  0.007145

4 ETW000  [     dev trc,00000]  Tue Jul  5 07:29:49 2016                                             3003044  3.010189

4 ETW000  [dbhdbsql.cpp,00000]  *** ERROR => Connect to database failed, rc=1, rcSQL=-10709               25  3.010214

4 ETW000  [     dev trc,00000]  SQLCODE    : -10709                                                       11  3.010225

4 ETW000  [     dev trc,00000]  SQLERRTEXT : Connection failed (RTE:[89006] System call 'connect' failed, rc=113:No route to host)

4 ETW000                                                                                                  10  3.010235

4 ETW000  [     dev trc,00000]   -> SetSdbDbslCA(errcode=-10709)                                           8  3.010243

4 ETW000  [     dev trc,00000]   -> freeConnection(con_hdl=0)                                             15  3.010258

4 ETW000  [     dev trc,00000]     close all opened locators of connection 0                               8  3.010266

4 ETW000  [     dev trc,00000]  } DbSlHDBConnect(rc=99)                                                   26  3.010292

4 ETW000  [    dblink  ,00000]  ***LOG BY2=>sql error -10709 performing CON                               28  3.010320

4 ETW000  [    dblink  ,00000]  ***LOG BY0=>Connection failed (RTE:[89006] System call 'connect' failed, rc=113:No route to host)

4 ETW000                                                                                                  11  3.010331

4 ETW000  [     dev trc,00000]  { DbSlHDBControl(con_hdl=0,command=41,arg_p=d774d8)                       20  3.010351

4 ETW000  [     dev trc,00000]  } DbSlHDBControl(rc=0)                                                    11  3.010362

4 ETW000  [     dev trc,00000]  { DbSlHDBControl(con_hdl=0,command=14,arg_p=7fff37db6ef0)                 10  3.010372

4 ETW000  [     dev trc,00000]  } DbSlHDBControl(rc=0)                                                     6  3.010378

4 ETW000  [     dev trc,00000]  { DbSlHDBControl(con_hdl=0,command=50,arg_p=7fff37db6f08)                  6  3.010384

4 ETW000  [     dev trc,00000]  } DbSlHDBControl(rc=0)                                                    13  3.010397

4 ETW000  [     dev trc,00000]  { DbSlHDBControl(con_hdl=0,command=13,arg_p=7fff37db6e60)                  9  3.010406

4 ETW000  [     dev trc,00000]  } DbSlHDBControl(rc=0)                                                     7  3.010413

4 ETW000  [     dev trc,00000]  { DbSlHDBControl(con_hdl=0,command=52,arg_p=7fff37db6eb0)                  8  3.010421

4 ETW000  [     dev trc,00000]  } DbSlHDBControl(rc=0)                                                     9  3.010430

2EETW169 no connect possible: "DBMS = HDB                              ---  SERVER = '' PORT = ''"

4 ETW000  [     dev trc,00000]  release memory of the SIBU buffers                                        61  3.010491

4 ETW000  [     dev trc,00000]  release memory of the STATEMENT CACHE                                     11  3.010502

awssap1:hd1adm 22> startsap all pf=/usr/sap/HD1/profile/HD1_DVEBMGS00_awssap1

Database db must be started first

Log on to remote server and start database

[Former Member · Accepted solution]

Looks like a wrong entry in hdbuserstore.

Make sure you get the correct output of this command:

hdbuserstore list

[Former Member · Accepted solution]

Hi Josef,

Thank you for your answer. After I managed the new IP on the hdbuserstore I run again  startsap all <without pf parameter> and it started but still I am not able to connect to SAP.

But when I run startsap all pf=/usr/sap/HD1/profile/HD1_DVEBMGS00_awssap1  I received

Database db must be started first

Log on to remote server and start database

Regards,

Teodor

startsap all

Checking db Database

Database is running

-------------------------------------------

Starting Startup Agent sapstartsrv

OK

Instance Service on host awssap1 started

-------------------------------------------

starting SAP Instance ASCS01

Startup-Log is written to /home/hd1adm/startsap                                                                                         _ASCS01.log

-------------------------------------------

/usr/sap/HD1/ASCS01/exe/sapcontrol -prot NI_HTT                                                                                         P -nr 01 -function Start

Instance on host awssap1 started

Starting Startup Agent sapstartsrv

OK

Instance Service on host awssap1 started

-------------------------------------------

starting SAP Instance DVEBMGS00

Startup-Log is written to /home/hd1adm/startsap_DVEBMGS00.log

-------------------------------------------

/usr/sap/HD1/DVEBMGS00/exe/sapcontrol -prot NI_HTTP -nr 00 -function Start

Instance on host awssap1 started

[Former Member · Accepted solution]

After you changed hdbuserstore, what is in trans.log now?

Is it still 'rc=113:No route to host' ?

[Sriram2009 · Accepted solution]

Hi

1. After change the IP, Are you getting DNS resolution from IP & Host name? In all the SAP systems (DB, CI & App Servers add the local Host enters)

2. Refer the SAP KBA for the errror message "ERROR => Connect to database failed, rc=1, rcSQL=-10709"

1828028 - DBCON connection to remote HANA database cannot be established

2213725 - How-To: Troubleshooting of -10709 errors

BR

SS

[Former Member · Accepted solution]

The trans.log ends like this:

4 ETW000  [     dev trc,00000]    EXECUTE CURSOR C_0007 on connection 0                                    6  0.048036

4 ETW000  [     dev trc,00000]    CURSOR C_0007 SET InputSize=1                                            6  0.048042

4 ETW000  [     dev trc,00000]    CURSOR C_0007 EXECUTE on connection 0                                    6  0.048048

4 ETW000  [     dev trc,00000]    execute() of C_0007, #rec=1, rcSQL=0, rc=0                             512  0.048560

4 ETW000  [     dev trc,00000]    CURSOR C_0007, rc=0,#rec=1,#dbcount=1                                   10  0.048570

4 ETW000  [     dev trc,00000]   -> deactivate_stmt(sc_hdl=7f9944d05448,da_p=17d9f90,rc=0)                 6  0.048576

4 ETW000  [     dev trc,00000]   -> StmtCacheFree(DBSL:C_0007)                                             7  0.048583

4 ETW000  [     dev trc,00000]  } DbSlHDBModify(rc=0)                                                      6  0.048589

4 ETW000  [     dev trc,00000]  } db_rtab( fcode = 'RT_DELETE', retcode = 0 )                              6  0.048595

4 ETW690 COMMIT "0" "0"

4 ETW000  [     dev trc,00000]  db_con_commit (con_da={R/3,0,0},th_commit=1,tx=1,hold_cursor=0)           19  0.048614

4 ETW000  [     dev trc,00000]  { DbSlHDBCommit(con_hdl=0,forced=1,hold=0)                                 9  0.048623

4 ETW000  [     dev trc,00000]   -> dsql_sapdb_free(con_hdl=0,hold=0)                                      6  0.048629

4 ETW000  [     dev trc,00000]   -> commit(con_hdl=0)                                                      6  0.048635

4 ETW000  [     dev trc,00000]     close all opened locators of connection 0                              12  0.048647

4 ETW000  [     dev trc,00000]   -> quiesceConnection()                                                 1068  0.049715

4 ETW000  [     dev trc,00000]  } DbSlHDBCommit(rc=0)                                                      9  0.049724

4 ETW000  [     dev trc,00000]  db_con_check_dirty: db_con_dirty = NO                                      8  0.049732

4 ETW000  [     dev trc,00000]   -> dsql_hdb_sync()                                                        9  0.049741

4 ETW000  [     dev trc,00000]  { DbSlHDBControl(con_hdl=0,command=12,arg_p=189ef00)                      17  0.049758

4 ETW000  [     dev trc,00000]  } DbSlHDBControl(rc=0)                                                     8  0.049766

4 ETW000  trace to file pointer closed

4 ETW000  trace at level 1 opened for a given file pointer

4 ETW000

4 ETW000 ================== STEP 1 =====================

4 ETW000 date&time        : 05.07.2016 - 09:11:09

4 ETW000 function         : CONNECT

4 ETW000 buffersync       : YES

4 ETW000 clients          : default

4 ETW000 purgedocu        : YES

4 ETW000 commit           : 100000

4 ETW000 table cache      : dynamic

4 ETW000

4 ETW000  [     dev trc,00000]  Disconnecting from ALL connections:                                       80  0.000080

4 ETW000  [     dev trc,00000]  Disconnecting from connection 0 ...                                     2298  0.002378

4 ETW000  [     dev trc,00000]  Now I'm disconnected from HDB                                            263  0.002641

4 ETW000  [     dev trc,00000]  Disconnected from connection 0                                            19  0.002660

4 ETW000  [     dev trc,00000]  statistics db_con_commit (com_total=1, com_tx=1)                           8  0.002668

4 ETW000  [     dev trc,00000]  statistics db_con_rollback (roll_total=0, roll_tx=0)                       6  0.002674

4 ETW000 Disconnected from database.

4 ETW000 End of Transport (0000).

4 ETW000 date&time: 05.07.2016 - 09:11:09

[Former Member · Accepted solution]

The end alone doesn't help much.

Please run 'R3trans -d' and look for errors in trans.log.

Looks like it was successful last time?

[former_member185239 · Accepted solution]

Hi Teodor,

It means that the problem does not exist on database side as the return code is 0000.

With Regards

Ashutosh Chaturvedi

[Johan_sapbasis · Accepted solution]

Hi,

What do you see now in startsap.log file ?

Also make sure to stop remote application server first. Then stop database server.

cleanipc <instancenr> remove

then startsap all

Kind Regards,

[Former Member · Accepted solution]

yes, R3Trans returns 0  

awssap1:hd1adm 52> R3trans -d

This is R3trans version 6.25 (release 742 - 11.02.15 - 14:06:42).

unicode enabled version

R3trans finished (0000).

Actually I have two cases.

In the 1st case when I run startsap all <without provide the path for my SAP's Instance Profile> it starts bun I am not able to connect to the  SAP.

In the second case when I run startsap all with profile provided I receive that message that "Database db must be started first"

In the second hand in /usr/sap/<sid>/profile I have the following profile files: HD1_ASCS01_awssap1, HD1_DVEBMGS00_awssap1, DEFAULT.PFL.   It by default starts with HD1_ASCS01_awssap1, but I want to start the HD1_DVEBMGS00_awssap1 profile and when I provide the path for my wished profile it says "Database db must be started first"  otherwise it starts but I can't connet to the SAP

[Johan_sapbasis · Accepted solution]

Hi,

Just to clarify is your DB and Application Server running on the same physical server or seperate hosts?

My understanding from your previous posts were that you had the db server and a seperate application server.

If it is one server it should startup if your R3trans -d connect successfully.

If it is two servers with DB started on DB server logon to application server and then do a R3trans -d from there?

Please give clarity on this.

Regards,

[Former Member · Accepted solution]

Hi Nicolass,

To clarify my DB runs on a virtual server and Application Server runs on a second virtual server both are in Amazon. DB starts and work properly and when I run R3trans -d it returns code 0 but still, I ma not able to connect to SAP. And if I run lsof /usr/sap/ I don't have any output.

[Johan_sapbasis · Accepted solution]

Hi,

When you changed the IP of the DB server did you update /etc/hosts on both servers? If so see if you can do nslookup on new IP address on both servers, if this is successfull, hosts file were update both sides. Important to make sure here that both hosts can see the new IP address/DNS resolution and each other.

Now log onto the Application Server and from there you run a R3trans -d what is status?

On this host application server:

stopsap R3

cleanipc <instnr> remove

Now log onto your DB host R3trans -d results what is status?

stopsap

cleanipc <instnr> remove

Before you try to run startsap from DB host , make sure on both hosts that the sapstartsrv processes are killed after the stopping of SAP and DB, this is to release possible holds on /tmp/.sapstream* files.

Now startsap cleanly and then do checks again on application server host as well.

Kind Regards,

[former_member230159]

Hi Teodor,

If the issue still persists after performing as per Nicolas steps, please paste here:

+ startsap.log

+sapstartsrv.log

+stderr* traces

+stopdb.log

Best regards,

Manjunath

[Former Member]

Hi Manjunath,

I checked the log files and I found that the system was out of virtual memory.

Thank you for the hint.

Best Regards,

Teodor

[former_member230159]

Hi Teodor,

Cool !!

Hope the issue is resolved now

Cheers!

Manjunath

[Former Member]

Hi all,

After I scanned the public IP with an online scan tool I found that only port 3201 is opened, I tried to connect to instance 01 but the connection was refused. I checked in /tmp/ and I didn't found the .sapstream3200 ... I have a lot of .sapstream's on other ports. Initially the instance was 00 and I didn't changed anything on profile files.

Regards,

Teodor

[Johan_sapbasis]

Hi Teodor,

Instance 3201 is your ASCS instance dispatcher port. I would expect the the 3301 for gateway to be open on ASCS. Also 3601 for Message Server on ASCS

3300 on Application server for gateway.

Ports of SAP NetWeaver Application Server ABAP - SAP NetWeaver Application Server ABAP Security Guid...

Stop the SAP instances, kill the sapstartsrv processes. Delete the sapstream files this will have to be done as root perhaps I cannot recall the ownership of these files.

Kind Regards,

PS: just check on both servers that you can see the others DNS and reverse DNS correctly.

[former_member185239]

Hi Teodar,

Mantain the environment variable

SAPDBHOST = <hana database server>

Then run the command startsap all.

Since the IP of the system has changed , So you have to maintained the entries in /etc/hosts.

When you are starting the SAP system with startsap only then your sap system is coming up but you are not able to login to the system. The reason is that it is not able to find the message server host / it is not able to resolve the hostname where message server is running as a result it is going to loop.

With Regards

Ashutosh Chaturvedi

[Former Member]

After reading this thread again, a few more thoughts:

Did it run this way before IP change? Meanwhile I doubt that.

Instances HD1_ASCS01_awssap1, HD1_DVEBMGS00_awssap1: Actually you will have to start both of them.

startsap all pf=/usr/sap/HD1/profile/HD1_DVEBMGS00_awssap1

That's not how it works. You can't start all instances with just one profile. Each instance will need its own.

... it starts bun I am not able to connect to the  SAP.

If SAP processes are running, I would check the configuration of your SAPGUI hosts. Might it be trying to connect to the old IP?

hope this helps

[Former Member]

I checked the SAPGUI host and it is the new IP

[former_member185239]

Hi Teodar,

Have you checked the file hosts  in /etc .

With Regards

Ashutosh Chaturvedi

[Former Member]

Hi Ashutosh,

Yes, first time I mananged the IPs in /etc/hosts and I also checked hosts.allow hosts.deny.

Regards,

Teodor

[Johan_sapbasis]

Hi,

This to me seems to be  a network problem.You missed some config somewhere.

4 ETW000  [     dev trc,00000]  Tue Jul  5 07:29:49 2016                                             3003044  3.010189

4 ETW000  [dbhdbsql.cpp,00000]  *** ERROR => Connect to database failed, rc=1, rcSQL=-10709               25  3.010214

4 ETW000  [     dev trc,00000]  SQLCODE    : -10709                                                       11  3.010225

4 ETW000  [     dev trc,00000]  SQLERRTEXT : Connection failed (RTE:[89006] System call 'connect' failed, rc=113:No route to host)

Especially since you have changed the IP's what do you see in netstat -in ?

Check your dev_rd gateway file also reading also host entry probably.

Since you cant connect to db check your network related configurations after change.

You would have had to correct the IP and network configuration on both hosts as both will have /etc/hosts entries and lan configurations pointing to each other and DNS.

Kind Regards,