SQL 2012 upgrade and Solman Java stack start error

[Former Member]

Hello

I upgraded my MSSQL 2008 R2 on Solution Manager 7.1 SP8 to MS SQL 2012 SP1 CU8 but my Java stack is not starting up. ABAP side is connecting without any problems but Java is offline.

I read following notes and did upgrade according to those.

1676665 - Setting up Microsoft SQL Server 2012

683447 - SAP Tools for MS SQL Server

Also I followed Upgrade and Installation Guide - Upgrade to and Installation of SQL Server 2012 in an SAP Environment document.

I upgraded my kernel to 7.20 / 500, installed latest version of libdsl, tp, r3trans and r3load but Java is still not coming up. I even restarted whole server but nothing changed.

As for logs

dev_bootstrap

[Thr 4808] JHVM_RegisterNatives: com.sap.bc.krn.perf.PerfTimes

[Thr 6096] JLaunchIExitJava: exit hook is called (rc = 66)

[Thr 6096] **********************************************************************

[Thr 6096] *** ERROR => The Java VM terminated with a non-zero exit code.

[Thr 6096] *** Please see SAP Note 943602 , section 'J2EE Engine exit codes'

[Thr 6096] *** for additional information and trouble shooting.

[Thr 6096] **********************************************************************

[Thr 6096] JLaunchCloseProgram: good bye (exitcode = 66)

jvm_bootstrap

Exception occurred:

com.sap.engine.frame.core.configuration.NameNotFoundException: A configuration with the path "cluster_data/server/ID4070350" does not exist.

  at com.sap.engine.core.configuration.impl.ConfigurationDataCacheAllImpl.getSubConfiguration(ConfigurationDataCacheAllImpl.java:107)

  at com.sap.engine.core.configuration.impl.ReadAccessConfiguration.getSubConfiguration(ReadAccessConfiguration.java:715)

  at com.sap.engine.bootstrap.Synchronizer.getConfiguration(Synchronizer.java:573)

  at com.sap.engine.bootstrap.Synchronizer.getElementType(Synchronizer.java:93)

  at com.sap.engine.bootstrap.Bootstrap.initDatabaseConnection(Bootstrap.java:475)

  at com.sap.engine.bootstrap.Bootstrap.<init>(Bootstrap.java:156)

  at com.sap.engine.bootstrap.Bootstrap.main(Bootstrap.java:946)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

  at java.lang.reflect.Method.invoke(Method.java:331)

  at com.sap.engine.offline.OfflineToolStart.main(OfflineToolStart.java:81)

Exception occurred:

com.sap.engine.bootstrap.SynchronizationException: Database initialization failed! Check database properties!

  at com.sap.engine.bootstrap.Bootstrap.initDatabaseConnection(Bootstrap.java:477)

  at com.sap.engine.bootstrap.Bootstrap.<init>(Bootstrap.java:156)

  at com.sap.engine.bootstrap.Bootstrap.main(Bootstrap.java:946)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

  at java.lang.reflect.Method.invoke(Method.java:331)

  at com.sap.engine.offline.OfflineToolStart.main(OfflineToolStart.java:81)

----------==[ Caused by: ]==----------

com.sap.engine.bootstrap.SynchronizationException: No such Dialog Instance (ID4070350) in the database! Check database consistency or local Bootstrap properties!

  at com.sap.engine.bootstrap.Synchronizer.getElementType(Synchronizer.java:96)

  at com.sap.engine.bootstrap.Bootstrap.initDatabaseConnection(Bootstrap.java:475)

  at com.sap.engine.bootstrap.Bootstrap.<init>(Bootstrap.java:156)

  at com.sap.engine.bootstrap.Bootstrap.main(Bootstrap.java:946)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

  at java.lang.reflect.Method.invoke(Method.java:331)

  at com.sap.engine.offline.OfflineToolStart.main(OfflineToolStart.java:81)

[Bootstrap module]> Problem occurred while performing synchronization.

I can connect to SQL database without any problems via SQL Management Console. R3trans -d and R3trans -x results are RC=0. I can connect to DB via configtool but admintool fails to connect.

For upgrade I did following;

-Took a full backup

-Shutdown SAP server

-Executed setup from SQL 2012 installation media and did upgrade mssqlserver instance

-Restarted server and applied SP1

-Restarted again and applied CU

-Restarted whole server and run SAP Tools for MSSQL, executed Database Upgrade Completion

-Checked SQL ports (they are default 1433) and saw they active.

-Started SAP

What am I missing?

Edit

I did not uninstall SQL 2008 R2 on that server, and also compatibility level is 110 (SQL 2012).

Edit 2

Changing compatibility level didn't solve my problem.

[Former Member · Accepted solution]

Well finally I managed it working.

Here is what I from start;

I restored my old Vmware clone to another server with different hostname.

I detached SQL DB and reattached it (maybe it was useless but I did it anyways)

I changed all hostnames to new one on ConfigurationManager and LockingManager tabs via Config Tool

I changed hostnames on config.properties / instance.properties / localsdmgui.properties / sdmrepository.sdc

I changed Secure Store passwords to default ones

I changed all hostnames in SAP profiles

After that I started new SAP system and all were working.

Then I took export of Java via SWPM

On production system what I did is like below;

I took full backup on SAP DB

Edited Java Export's source.properties file and changed hostname into new server.

I first upgraded SQL to 2012 and then SP1 and then CU8

I used SWPM10 > Generic Installation Options > MSSQL > Database upgrade completion

Started SWPM and then MSSQL System Copy > Target System Installation > Single Instance

Selected Homogenous Copy

Started system copy but it failed on Configure Java Parameters

Then I went back to new test system and executed Offline Config Tool.

I exported /cluster_data/Propertysheet.instance.properties.IDxxx / /cluster_data/serverIDxxx / /cluster_data/dispatcher/IDxxx to local files and then imported them to production system.

After that I changed hostnames again in configuration manager and locking manager

Restarted SWPM and then it finished.

Now all is working.

Thanks for your thoughts, I hope this will help others

[Former Member]

when you restored from Backup, did you also restore the master/system databases. The user mapping of instance vs DB will get wiped out if just reinstalled the Instance.

You can also use the STM tools to restore the DB permissions. This will take care of fixing all users and permissions. I believe with the latest SWPM, STM tool is integrated. Let me know if this helps.

You can also use an old J2EE export from the same system if that was taken after applying SPS8.

J2EE export should be part of all Backups procedures for dual stack systems, also backing up system databases. Only NW7.3 onwards Java export is no longer required.

regards

Yogesh

[Former Member]

Hello

I didn't take master and system db backups, I only took SAP DB backup. I already used schema4sap script but nothing changed. Script cleary sees my ABAP + Java stack schemas and creates query for it but my problem still resides.

I have a Vmware snapshot which was taken after upgrade. Can I export Java from it and then import it to production system?

[Former Member]

You were having issues since after the upgrade, not sure if the VMware snapshot will work for you to be able to take a Java export.

schema4sap or the STM tools will only help to setup permission, not with any inconsistency with the Database. You can try with the STM tool to be sure. Once done, can you paste the bootstrap log after you try restarting the system.

Did SAP help you yet.

regards

Yogesh

[Former Member]

Hello

They are still inspecting the system. So far they said my instance and dispatcher nodes deleted prior to upgrade and now they are checking my databases.

My vmware image was taken after I did SP8 upgrade which was few weeks ago. I am planning to use it but don't know if I manage to take export and make import of Java in a dual stack system, never done that before.

Thanks

[Former Member]

You can surely use it if that has the DB and Filesystem. But to be able to start the system and take a java export, you will need to restore it over your current host.

if it only has the DB, it may also work but maybe not depending on if your FS has got messed up or not.  

To do that, you need to backup your current DB and FS and then overwrite it. When able to take an export, you need to restore it back and do a system copy using the java export. It may or may not work depending on if your DB is corrupt o not.

IF it does not work, another thing I would like to mention was that if SAP gives up and if you have the FS and DB backups of after the upgrade, you may want to restore it and see if that brings up the system. I am not sure what you use your solution manager system for but if it is only monitoring EWA and MOPZ, going 3 weeks back may not be a major loss. Better than having to build a new solution manager.

But first I would suggest you to wait for SAP support and see what they have to say. .

[Former Member]

My previous vmware image has both db and fs from weeks ago. My customer is using Solman system for project management and timesheet entries which is being used by consultants and employees everyday. If it was ewa and mopz only, I would install Solman from scratch.

On worst case, I will adapt old system parameters into new changed hostname (where vmware image will be restored), then I will close Solman, take an export of system, then replace Java export with old system's export and then import it to production system.

Still waiting SAP to make a conclusion on my case

[Former Member]

i am not sure changing hostname would get it working, usually a system copy is required, especially with Java. I have tried a few times, never got it to work.

At this time, best bet would be to wait for SAP. Since this is your production system, you need to check with them the options you have to confirm that that option will be supported.

regards

Yogesh

[Former Member]

Compare the JDK folder to a folder on a working JAVA server.

Even try to re-install the JDK on the java stack server.

I had a similar issue in the past and the problem was missing folders and files in the JDK installation itself.

It was a mystery why those lib files were missing. The java process threw exceptions about DB synchronization and missing nodes.

I compared the entire c:\j2sdk1.4.2_17-x64 folder to another - and realized that many folders\files were missing. Re-installing fixed it.

Give it a try.

[Former Member]

I have an update on my case.

SAP forwarded my case to development department and while they are working on I did several things.

I created another virtual server and restored a Vmware snapshot which was taken a month ago. I detached and attached my DB, I changed hostnames on profile files and Java properties files and then started SAP. It worked and SAP is running on test system without any problem.

I took Java export on that test system and then did a system copy on production system. Only hostnames were different, but it failed on Configure Java Parameters phase.

I can see my instance in Prod system via configtool but server and dispatcher is missing.

Logs

jswitch.log

ERROR: 2014-02-24 20:02:51 com.sap.inst.jswitch.config.SwitchTask run

Fatal exception during execution of JSwitch.

com.sap.engine.frame.core.configuration.NameNotFoundException: A configuration with the path "cluster_data/dispatcher/ID4070300" does not exist.

jswitchconsole.log

com.sap.engine.frame.core.configuration.NameNotFoundException: A configuration with the path "cluster_data/dispatcher/ID4070300" does not exist.

jdkswitch.log

com.sap.engine.frame.core.configuration.NameNotFoundException: A configuration with the path "cluster_data/dispatcher/ID4070300" does not exist.

In theory I shouldn't be getting those errors because test system was created from production system's clone. Everything is same except for hostname and IP.

Best regards

[Former Member]

Try to restart system and upload most recent log files.

[Former Member]

I already restarted whole server many times and I always receive same error.

[Former Member]

Can you attach most recent log files (preferably as attachments)?

[Former Member]

Hello

Here are the combines logs, if you need more I can provide whatever you need.

Thanks

jvmlog.txt.zip

devlog.txt.zip

[Sriram2009]

Hi Kaan

Check this SAP Note 1924336 - Java node 'Instance_bootstrap' terminated

Regards

[Former Member]

There are no instances in my offline config tool.

[Sriram2009]

In MS SQL MMC you can find the ISP database ?

[Former Member]

Yes, ISP is a dual stack Solution Manager system. As I said before, ABAP side is working without any problems but Java is not starting. After getting "synchronizing binaries" phase it shuts down itself and then restart itself and then it loops few times, but few minutes later it stops.

[Sriram2009]

Just follow the steps try

1. Stop the application server.

2. Start SQL Server Management Studio, open a query window, and execute:

"update statistics <SID>.SAP<SID>DB.UME_STRINGS WITH FULLSCAN" (where <SID> is the SAP system ID).

3. Start the application server.

[Former Member]

Hello

I executed update statistics ISP.SAPISPDB.UME_STRINGS WITH FULLSCAN and restarted server but nothing changed.

Thanks

[Sriram2009]

Hi

Kindly check the SAP Note to repair the Schema 1294762 - SCHEMA4SAP.VBS

Regards

[Former Member]

Already used it but nothing changed

[Sriram2009]

Hi

Better you can raise the OSS Ticket to SAP

Regards

[Former Member]

Hello

I already have a "very high" priority ticket on OSS but they still didn't come up with a proper analysis and solution.

Thanks

[Former Member]

-Hope you took a Java export and a SQL Backup before you started the upgrade. if yes, try restore it in a test system and compare the settings.

-Did you check SQL Server Logs, are you getting any errors. Usually SQL upgrade is non intrusive if your SAP release support it. It should not alter the DB contents. The user permissions may get altered. If that is true, you will see alerts in the SQL error log.

- Make sure your kernel support SQL 2012.

regards

Yogesh

[Former Member]

Hello

There is nothing seems wrong in SQL logs and also I have latest version of 720 UC kernel.

Unfortunately I have only full SQL backup but not Java export.

Best regards

[Former Member]

I uninstalled SQL 2012 and previous SQL 2008 R2 installations. Restarted server and installed SQL 2012 via sql4sap.vbs . After that I restored my backup and then executed SWPM MSSQL Database Upgrade but still nothing changed. ABAP is opening but Java is not. I cannot even see my Java instance in Config Tool.

To make sure I copied instance.properties file from another Solman server, adjusted IDs and server names and restarted again but still nothing.

I even resetted Secure Store passwords via Config Tool but no changes to Java situation.

Unless SAP provide a solution, my only option seems to restore whole server from a Vmware snapshot which was taken days ago and redo all of project management entries again.

Edit.

I uninstalled SQL 2012 and installed SQL 2008 R2 and restored database but Java didn't open, again.

[Sriram2009]

Hi Kaan

I thing you JAVA schema was not correct for that reason java stack not connecting to MS Sql DB

Check this SAP Note

1950926 - System cannot start due to login SQLServerException

1086375 - User ID/Schema ID inconsistency in SQL database

Regards

Ram

[Former Member]

Hello

I changed connection pool and secure store passwords then restarted server but nothing changed. I didn't change my schemas, I only did upgrade my DB. Also at the moment I am running my SAP on SQL 2008R2 which was original system but still Java is not working. I took my backup before upgrading system and restored it into both SQL 2012 and SQL 2008R2 but Java insists to ignore my instances.

Thanks

[former_member185239]

Hi Kaan,

Please follow the below sap note

1795975 - "cluster_data/dispatcher/ID5001" does not exist. "cluster_data/server0/ID5001" does not exist.

To Resolve For explanation purpose's :ERROR(

com.sap.engine.frame.core.configuration.NameNotFoundException: A configuration with the path "cluster_data/server/ID4070350" does not exist.)

instance_IDXXXXX: instance_ID40703

dispatcher_IDXXXXXXX: dispatcher _ID4070300

server_IDXXXXXXX: server_ID4070350

Next step is to fix the incorrect values.

In the configtool navigate to:

1.instance_ID40703

2.dispatcher_ID4070300/server_ID4070350

3.managers

4.ClusterManager

5.Local properties

6.Then set the following properties

element.clusterId to 4070300 or 4070350 depenging if is a dispatcher or server0 node.

element.groupId to 40703

Save and then restart the J2ee system for the changes to come into affect.

[Former Member]

Hello

There are no instance or server IDs on configtool, as seen in the screenshot but it had to be there..

Thanks

Edit:

Also in Cluster Manager section in configtool there are no local properties

[former_member185239]

Hi Kaan,

It has to be there. Raise a message with SAP and follow them.

If you have a export of java , then remort it through SWPM.

With Regards

Ashutosh Chaturvedi

[Former Member]

I have only a full backup before SQL upgrade. Creating an OSS message now.

[former_member185239]

Hi Kaan,

Please keep us update on SAP reply.

Somehow the propertysheet is missing.

Regards

Ashutosh Chaturvedi