URGENT HELP NEEDED - SAP J2EE is not starting !!!

[symon_braunbaer]

Hello,

I don't know what was going on and who did what, as I am out of office.

I was just called, that a J2EE system can't connect to the DB.

When I attempt to start it, I am getting this:

Database <SID> must be started on remote server

Logon to remote server and start database

The database is running, of course, the LISTENER is running as well !!

No log files are being created, nothing...

I am able to start the config tool without any errors, looked at the secure

store, looks fine! I have also tried to login in SQLPLUS with the SAPSR3DB

user and it works fine !!

System: NetWeaver AS Java 7.3 EHP1, no ABAP stack

DB: Oracle 11g (11.2.0.3)

OS: Solaris 10

PLEAE URGENTLY HELP ME !!! Thank you so much !!!

[Reagan · Accepted solution]

Hello Symon

Try these:

Restart the Oracle Listener

Try starting the J2EE stack using the command startsap j2ee

If it fails then restart the server where SAP is installed and make sure all the file systems are mounted correctly especially the Oracle client.

Cheers

RB

[symon_braunbaer · Accepted solution]

Hi Reagan,

I have tried all this without any luck 😕 The oracle client is available...

[manumohandas82 · Accepted solution]

Hi Symon ,

check whether the profile directories are in place  ie /sapmnt/<SID/profile/ ? if present check whether the hostname  is the same as  in the profile

Check whether you can start with startsap <hostname> ?

tnsping to the oracle server successfully ?

Please check the status using JSMON / jcmon ?

Thanks ,

Manu

[symon_braunbaer · Accepted solution]

Hello,

Profiles were in place. The hostname was fine. I didn't try tnsping, but I suppose it would have

worked. The system wasn't writing any fresh logs, which I found kinda stunning !! And there

was no dev_dispatcher present at all !!

In the end a restore from backup had been performed and now the system is up and running.

Although it solved the problem, I am very sad, as probably I will never find out how this could

be fixed.

[manumohandas82 · Accepted solution]

Hi Symon ,

Not sure how the restore helped you unless the issue was with the database itself .

Check in alert log  for any issues  .

All well that ends well any way

Thanks ,

Manu

[symon_braunbaer · Accepted solution]

Hello Manu,

it must have been in the DB, when a restore helped to solve it...

I will ask the unix guys if it is possible to clone the entire system and try to find out the root cause for that, because it is very very strange !!

[divyanshu_srivastava3]

Hi Symon,

Please provide me a fresh std_server0.out and dev_dispatcher with default trace of server0 once.

Goto DB2 server - and start DB

Then goto SAP apps - and then start SAP using startsap script.

Regards,

[hemanth2]

Hi,
 

Hope you are doing good.

Nice to hear from you again. To rule out any connection issues, run the JdbcCheckup connection tool as mentioned in SAP Note No. 509719 so that we can ascertain the connection between the DB and the J2ee server nodes.

Let us know the result.

Hope this helps.

_ _ _ _ _ _ _ _ _

Kind Regards,

Hemanth
SAP AGS
_ _ _ _ _ _ _ _ _
 

[symon_braunbaer]

Hello Hemanth and many thanks for replying!

Nice to hear from me, but it's not nice, because this is a very big problem and it seems to be a very hard nut to crack, so I really everyone's help here...

I have checked note 509719 in detail. Unfortunately I don't have the samples subdirectory, hence I really don't how to perform this check. I have opened the URL to the oracle website, provided in the note, but I didn't see a possibility to download them.

I am really wondering how Java could not see the DB, when everything in the secure store of configtool is fine. I am really thinking of restoring everything to the point before this happened!

And it is especially disturbing me, that there are no recent logs written !!

[manumohandas82]

Hi  Symon ,

Please supply the log files mentioned by Yuksel in the above post .

The files will be available in the directory /usr/sap/<SID>/JC<ins>/work

Thanks ,

Manu

[yakcinar]

Hello Symon,

I understood that your system is distributed.

Could there be a firewall activated between app server and db them?

Or network connection is broken?

Can you telnet DB port from app server?

Can you attach developer traces dev_ms, dev_disp, dev_server*, stderr*

Regards,

Yuksel AKCINAR

[symon_braunbaer]

Hello Yuksel and thanks for your reply!

No, there is definitely no firewall. I don't know what exactly do you mean by "DB port",

but I have tried telnet db-serv 1521 (which is the listener port) and this connection works!

I didn't find many of the log files mentioned by you, there is just dev_server* and stderr*,

I am attaching those, but I hardly doubt that you'll find something valuable in them.

I am attaching whatever I found... Please remove the .txt addition, without which it wouldn't

let me upload the attachment... Thanks !!!

j2ee_logs.zip.txt.zip

[yakcinar]

Hello Symon,

I am not sure whether this is your main problem but in the dev_server0 trace file there exist below errors;

F [Thr 24] Wed May  6 12:49:23 2015

F  [Thr 24] *** LOG => SfCJavaVm: exit hook is called. (rc = 335)

F  ********************************************************************************

F  *** ERROR => Java node 'server0' terminated with exit code 335.

F  ***

F  *** Please see section 'Java program exit codes'

F  *** in SAP Note 1316652 for additional information and trouble shooting advice.

F  ********************************************************************************

Did you check the recommended note.

You can check below wiki also.

(JSTSG)(Kernel)Cluster-MSConnectionProblems - Java Server Troubleshooting Guide - SCN Wiki

You can use Technology Troubleshooting Guide for this kind or errors.

Regards,

Yuksel AKCINAR