SAP Installation on 3 nodes cluster

Hello,

I have to implement 3 nodes Linux cluster for SAP Failover ( Oracle DB + SAP Instance). We are using two SAP Systems ECP and EPP. We have shared Storage and the OS is SLES 11 and the Filesystem is OCFS2 and one Machine is the failover of Both of the production machine. Now question is How Can I install the SAP to work properly in the cluster environment. For eg. I have the following shared (SAN) filesystem.

/oracle/<SID> /oracle/<SID> For both the Databases ( ECP+EPP)

/usr/sap<SID> For E.g ECP

/usr/sap<SID> For E,g EPP

/sapmnt/<SID> ECP

/sapmnt/<SID> EPP

All the sharedfilesystems are OCFS2 and are mounted on all the cluster nodes... Is my above configuration are correct? Or I have to change my filesystem layout.

Please help me.

Regards

Athar

Edited by: ather_36 on Jul 12, 2010 9:29 PM

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Accepted Solutions (0)

Answers (1)

How are you configuring on the 3 nodes? db+ci on one node? db on a separate node and ci on a separate node? application servers?

Are you going to mount the database storage on all cluster nodes at the same time? This way you can create a split-brain situation.. So I wouldn't do that if I where you...

Kind regards,

Mark

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

As I mention this is a three node cluster and there are two SAP systems ( ECC+EPP). So ECC database and its instance running on one nodeA and EPP database and its instance running on different nodeB and nodeC is the failover of both NodeA and NodeB.

For storage we are using OCFS2 and it mounts the filesystems at same time on all the cluster nodes. For avoiding Split brain situation we are using SLES HAE Stonith.

If you any other FIlesystem layout please help me and advise me.

I'm not familiar with Stonith devices, so I cannot advice you on that..

On other thing: make sure that you install your instances on different system numbers, cause they can not run together on one instance after a fail-over.

Kind regards,

Mark

Yes you are right, we are installing SAP Instance with the different system number and their no issue with SAP Instance. Both the SAP Instances are starting thorugh the cluster service. But the problem is with the Database. When both the databases are installed, cluster not starting the Database Instance and logs file shows database terminating with the error code 12. My Database filesystem are mounted parallelly on all the cluster nodes and when I create the cluster resource for SAP Database, its show the failoed status.

Please help.

Hello Mark,

R u using SLES 11 HAE for SAP HA? If yes, How many nodes in your cluster?

Hi,

I'm working for different customers, so I see many scenario's. Having three nodes in your HA scenario should be possible without any problem.

Can you post the exact error that you get when starting the database?

Kind regards,

Mark

Thanks for your kind help. Please download my /var/log/messages file from megaupload. Just 250 KB file. If you furhter information please let me know.

http://www.megaupload.com/?d=TF0KAK6F

Thanks and Regards

Athar

Hi Athar,

Can you please try to start the database manually. If the database won't start, please post the last 100 lines of the Oracle alert log.

Kind regards,

Mark

Hello Mark,

Thanks for your reply.

There is no issue with manual database start/stop. The only problem when cluster starting the database. As you see in logs, 1st cluster starts the database and then it stop.

Please help.

Well, in that case it is obvious that the problem is in the cluster scripts.. Let the Linux guys debug what's going wrong.. Perhaps the trace level can be increased to see exactly what is going wrong.

Kind regards,

Mark

When we were using one SAP system then there was no issue with the cluster, Cluster starting the SAP Database Service and also providing the Failover. But when we install the 2nd SAP System then we are facing the mentioned problem. Both the database are not starting through the cluster.

Now as per our Linux Vendor recommendation we updated the SLES 11 to SLES 11 SP1 and also installing the SAP.

Below are the list of current my cluster ocfs2 filesystem mounted on my all 3 nodes.

Filesystem Mounted on

/dev/mapper/mpathd /oracleECP

/dev/mapper/mpathc /oracleEPP

/dev/mapper/mpathe /usr/sap

/dev/mapper/mpathf /sapmnt/ECP

/dev/mapper/mpathg /sapmntEPP

Please help. These filesystem are OK for the two sap systems.

But what happens if you only fail-over the filesystems of both instances to the third node, and start both databases and SAP systems manually. Is that working fine?

If so, the setup of your filesystems shouldn't be the problem.. The only thing I can think of, is the Oracle filesystem, which isn't having a / in front of the SID. But if you are saying that it is being mounted the same way on the primary node in the same way, than this shouldn't be a problem either..

Are you able to increase the log level of the cluster, so you can see exactly what's going on?

Kind regards,

Mark

Well there is no need of filesystem failover. As I mention you this is the cluster filesystem and mounted paraelly on all the cluster nodes. The filesystem output was the as I mentioned in my last post was the output of all three nodes and are mounted at the same time on all nodes. What I do for manually start the database on the failover node is that I move the Virtual IP to the failover node and start the database with the sqlplus commands.

Acutally the filesystem which is /oracleECP is the mount point and the SID is inside in the filesystem which is /oracleECP/ECP.

But when we are using single system then there is no issue with the cluster amd failover.

Hope it better understand to you.

Whats your advise for the 3 nodes cluster running two sap system. Whats your advise for the filesystem layout.

We also logged this case to novell they replied the following:-

As already said, in the logs I can only see that the startdb program ofSAP terminates with error code 12: Jul 6 14:42:08 sapep01 lrmd: [5319]: info: RA output: (EPP_DB:start:stderr) 2010/07/06_14:42:08 ERROR: SAP database EPP startfailed: Trying to start EPP database ... Log file: /home/eppadm/startdb.log /usr/sap/EPP/SYS/exe/run/startdb: Terminatingwith error code 12 This is the reason why the resource agent fails: Jul 6 14:42:08 sapep01 crmd: [5322]: info: process_lrm_event: LRM operation EPP_DB_start_0 (call=30, rc=1, cib-update=48, confirmed=true)complete unknown error Would it be possible for you to open a call to SAP support and ask whaterror code 12 from SAP means?

I have no doubts about the filesystem layout. Also the usage of a three node cluster shouldn't be any problem.

The cluster script is probably just executing the startsap db which is failing with return code 12. In the log I see that also the startup of the listener is failing. Are both listeners using port 1527?

Can you post the startdb and startsap log, which are located in the home directory of the sidadm user?

Just to be sure: with a manual start of the databases, you are able to start both instances on the fail over node, correct?

Kind regards,

Mark

Thankyou.

No. We are using two different Port for Oracle Listener. 1527 and 1528. Also the name of the Listener are different one is "LISTENER" and the other is " LISTENER1".

Sure I will post you the Startdb and Startsap log but not today. As I mention you that we are installing the SAP again after the UPgradation of OS.

I will also confirm you by starting the both the database on the failover node.

Thanks for your help.

Athar

Hello,

Now my one database ECP has been starting through cluster. But now I am facing problem with the 2nd database which EPP.

My EPP database starting with the oracle user. But when I tried to start with the SAP user it giving me the following error.

Trying to start EPP database ...

Log file: /home/eppadm/startdb.log

No SQL*Net V2 connect to EPP available.

Check that the listener is running: "lsnrctl status".

Start the listener as user oraepp: "lsnrctl start".

/usr/sap/EPP/SYS/exe/run/startdb: Terminating with error code 14.

My Listener name is LISTENER1

Fri Jul 16 12:12:42 PKT 2010

LOGFILE FOR STARTING ORACLE

Trying to start EPP database ...

Fri Jul 16 12:12:42 PKT 2010

checking required environment variables

ORACLE_HOME is >/oracleEPP/EPP/102_64<

ORACLE_SID is >EPP<

Fri Jul 16 12:12:42 PKT 2010

check initora

Fri Jul 16 12:12:42 PKT 2010

check initora

Fri Jul 16 12:12:42 PKT 2010

checking V2 connect

TNS Ping Utility for Linux: Version 10.2.0.4.0 - Production on 16-JUL-2010 12:12:42

Used parameter files:

/usr/sap/EPP/SYS/profile/oracle/sqlnet.ora

Used TNSNAMES adapter to resolve the alias

Attempting to contact (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (COMMUNITY = SAP.WORLD) (PROTOCOL = TCP) (HOST = sapep) (PORT = 1528))) (CONNECT_DATA = (SID = EPP) (GLOBAL_NAME = EPP.WORLD)))

TNS-12541: TNS:no listener

V2 connect is not available. Terminating

/usr/sap/EPP/SYS/exe/run/startdb: Terminating with error code 14

In addition to my last post,

When I run lsnrctl status/start it gives me the following messages:-

sapep01:eppadm 45> lsnrctl status

LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 16-JUL-2010 12:16:08

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))

TNS-12541: TNS:no listener

TNS-12560: TNS:protocol adapter error

TNS-00511: No listener

Linux Error: 111: Connection refused

sapep01:eppadm 46> lsnrctl start

LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 16-JUL-2010 12:16:18

Starting /oracleEPP/EPP/102_64/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 10.2.0.4.0 - Production

NL-00280: error creating log stream /oracleEPP/EPP/102_64/network/log/listener.log

NL-00278: cannot open log file

SNL-00016: snlfohd: error opening file

Linux Error: 13: Permission denied

Listener failed to start. See the error message(s) above...

Regards

I am also giving the link of listener.log.

http://www.megaupload.com/?d=RMC6K053

What are the permissions of that file? This is most likely that your system isn't starting...

Just change the permissions of /oracleEPP/EPP/102_64/network/log/listener.log and start the listener again.

Kind regards,

Mark

Edited by: M. Dijsselbloem on Jul 16, 2010 9:30 AM

Permission : -rw-r----- 1 oraepp dba 553189 Jul 16 12:06 listener1.log

But another system is running with the oracle user permission on listener.log

In the log is mentioned /oracleEPP/EPP/102_64/network/log/listener.log you are providing the permissions of listener1.log

Does listener.log exist? If not, what are the permissions of the directory /oracleEPP/EPP/102_64/network/log

Your other listener is running on the other Oracle account. It could be that there are some differences in the rights of the directories, though it shouldn't be...

Kind regards,

Mark

listener.log is the log file of other listener which is running with the cluster and it is located in /oracleECP/ECP/102_64/ and listener1.log is the log file of problematic listener which is not starting manually with startdb command and it is located in /oracleEPP/EPP/102_64.

Well, the log file is clearly mentioning the following: NL-00280: error creating log stream /oracleEPP/EPP/102_64/network/log/listener.log

So I would say: change the permissions of this file (and maybe the log directory as well) and try to restart the listener.

Kind regards,

Mark

Ask a Question