cancel
Showing results for 
Search instead for 
Did you mean: 

SAP BO hangs off continuosly, terrible performance

Former Member
0 Kudos

Hi all,

We are experiencing what's becoming a critical issue for us in our company.

We have started experiencing performance issues in our Business Objects since two weeks. Last week we could still work with, and users run some reports, even if it took ages to open any webi; after webi was open, it run without problems. But since the last days it's almost impossible to do anything, as BO hangs quite often.

We are working with our Basis team and they have restarted the BO server several times. As soon as they do it, BO works fine... during 5 - 10 minutes. After that, we have the same problem, and performance is so slow that it's impossible to do anything and to work with.

We have webi reports, and Crystal Reports, and none are the issue as we have done several tests and verifications. We can also confirm that problem doesn't come from interfaces, RFCs, or any kind of connection to other system, and there is no critical report that may cause this incident, specially because there are no users now running any reports.

We have followed the following:   We deleted 21 Gb of old logs and traces, and issue persists.  We have no clue what to do, besides that opening an OSS Note with SAP (which actually is already done).

Our system: BO 4.1 SP3 Patch 7

I don't know if some of you has experienced something similar. If so, any suggestion will be more than welcome.

Thanks!

Accepted Solutions (1)

Accepted Solutions (1)

Former Member

Hi all,

First of all, thank you very much for your questions and suggestions. We were hard working till yesterday evening with an external consultant, and SAP was checking it as well as it was a Very High issue for us.

We could solve it, and today it's working as usual. We applied the following 4 improvements:

1.Disabling logs:

-Go to CMC,navigate to the servers and then webi processing server.

-Right click and go to properties.

-Set log level as "None"

-Do the same for all webi processing server.

-Same step you have to follow for rest of the servers.

-For only CMS keep log level as "Unspecified".


2.Delete logs:

-Go to Installing Directory of BusinessObjetcs\SAP BusinessObjects\SAPBusinessObjects Enterprise XI 4.0\logging directory.

-Delete all unwanted logs(.glf files).


3.Clear Cache:

-Go to Installing Directory of BusinessObjetcs\SAP BusinessObjects\SAPBusinessObjects Enterprise XI 4.0\Data directory.

-Take back up of all the content inside it and delete everything.


4.Clear Tomcat cache:

-Stop Tomcat.

-Go to Installing Directory of BusinessObjetcs\SAPBusinessObjects\Tomcat\work\Catalina\localhost.

-Take back up of all the content inside it and delete everything.

-Start Tomcat.


First thing was deleting logs, as I mentioned in my original post, we deleted 21Gb!! Then we applied the other 3 options, and we cannot ensure which one of those definitely solved the issue, but now it's ok.


Thanks again.

Former Member
0 Kudos

Actually this was done on our ssytem. At the first view everything was good. After a while problem repeated.

4.Clear Tomcat cache:

-Stop Tomcat.

-Go to Installing Directory of BusinessObjetcs\SAPBusinessObjects\Tomcat\work\Catalina\localhost.

-Take back up of all the content inside it and delete everything.

-Start Tomcat.

Answers (3)

Answers (3)

Former Member
0 Kudos

We have similar problem. Our System is BI 4.1 SP6. On windows with Web server Tomcat. System hangs after a while. Still can not find anything about system hang. Paralely trying to configure BO servers, but this is not a solution for us.

Tomcat is working properly. After a system logon system starts hanging. And after a while starts working normally and then problem again occur.

Henry_Banks
Product and Topic Expert
Product and Topic Expert
0 Kudos

Are you running on VMWare?

Former Member
0 Kudos

What is your cms db?

Do you have auditing and cmsdb schema present in same db server?

Also what is the auditing status?

Former Member
0 Kudos

Hi Alberto,

Few things in order to narrow down the issue:

- Is the BO server on Windows Server or linux? Any patches installed on BO server recently?

- When you remotely log onto BO server and open WRC or BI Launchpad, is there a lag?

- Is BO deployed on Tomcat? if yes, opening the home page takes time or comes up quickly when you start facing the issue.

    For example: If BILaunchpad URL is http://<servername>:8080/BOE/BI simply put http://<servername>:8080/ and check.

   Please check this BO server remotely as well.

- Are there any changes done in firewalls on client or server machines?

- When do you face performance issue- Opening launchpad or client tool? after entering credentials and pressing enter? browsing the reports in folders? opening or refreshing reports?

- Can you put traces on CMS server and Webi server and share the logs?

Regards,

Yuvraj

Former Member
0 Kudos

Hi Yuvraj,

It's in a Windows Server. I'm not sure if there has been some improvement or upgrade on it, I should ask Basis team.

I'm intrigated about your question about Firewall, does it affect? I think we have had some Firewall modification but it shouldn't affect us as the problem is either via LauncPad (web), either via RichClient (BO client software), besides that, when we are connected at work we use a corporate internet connection.

Regarding when we have the incident, is after puting the credentials, and it could happen anywhere: executing a report,browsing folders...

Regards

Former Member
0 Kudos

Hello Alberto,

You said this issue is observed since last 2 weeks. Was there any change in the environment before that like upgrade, bulk migration, change in database, etc.

If auditing and platform search is enabled then disable them and verify the behavior.

Is there any CPU spike on the BI server or the CMS database server?

Regards,
Swapnil

Former Member
0 Kudos

Hi Yuvraj,

Here some useful information:

1- Operating System. (on the BO server)

--> Windows Server 2008 R2

2- Exact version of BusinessObjects (along with any SP or FP)

--> BO 4.1 SP3 Patch 7

3- Deployment (Clustered / Standalone)

--> Deployment: Standalone

4- CMS database (Oracle / MS SQL server...)

--> Deployment: Standalone

5- Web Application Server (Tomcat / WebLogic...)

--> Tomcat

6- Is the issue reproducible on the BO server machine itself? (http://localhost:8080/BOE/BI)

--> Yes, it's reproducible on the BO Server

7- When the issue occurs, is it the same when accessing the Web Application Server home page (tomcat home page for example http://servername:8080)

--> No, the issue doesn't occur

8- Is the issue related to the load on the system? (does it depend on the number of sessions created / number of users connected..)

--> No, it doesn't depend on the number of sessions nor users connected.

9- Please indicate the CPU / RAM consumption when the issue occurs.

--> CPU: 25-30% RAM: 12Gb (Total 32Gb)

Regards

Former Member
0 Kudos

Hi Swapnil,

No, there hasn't been any upgrade/installation/improvement, besides that the one related with Firewalls that Basis team is checking but that probably is not the reason.

Regards

Former Member
0 Kudos

Hi Alberto,


Firewall can be checked when there is no issue on BO server but there is lag in client server communication.

As the issue is reproducible on BO server itself using (http://localhost:8080/BOE/BI) then it shows that the issue does not look relevant to firewalls.

Try this on BO server in order to investigate further:

- When you see performance issue, check which process in Task Manager is consuming maximum CPU and RAM.

- There are some diagnostic tools like procmon (process monitor) & regmon (registry monitor) which can be used. At time of hanging, we can check which files are visible in procmon.

- Check if there are any errors getting logged inside CMS or WebiServer log files.

- Try increasing the heap size of Tomcat to 4G and MaxThreads to 900 and check if it helps.

- Is your Windows Server 32 bits or 64 bits? As far as I know, in case of 32 bits, an EXE can only go till 2GB size and will crash after that. I am not 100% sure on this.

As you have already raised an OSS with SAP, have they been able to reproduce the issue at their end? Serdar has mentioned that he is facing similar issue. In case it is reproducible and a bug then we can probably wait for a fix.

Regards,

Yuvraj

Henry_Banks
Product and Topic Expert
Product and Topic Expert
0 Kudos

Are you running on VMWare?

Former Member
0 Kudos

yes we are on VM.

Henry_Banks
Product and Topic Expert
Product and Topic Expert
0 Kudos

Hi,

I don't know how much control you have over your VMWare-layer, but in my experience 'unexplainable slowness' can often be attributed to over-utilization of resources. That is to say, perhaps Memory, CPU and/or Disk (physically coming from the VMWare infrastructure) is being over-committed to the landscapes  that are hosted on it (like your BO server).

For example, suppose your VM infrastructure has 4 (physical) CPUs available in total. Next, 2 projects spool-up servers  each requesting 4 (virtual) CPU ... the system won't say 'no' ...   it will just re-assign resources dynamically "in the hope" that no one will notice a degradation in performance.  As a BO Admin, you'll never have visibility of these fluctuations.

Sound likely or possible?

Regards,
H

Former Member
0 Kudos

I can connect to the server and check its performance. I do not see any performance problem at there. Most of the time server is idle.

Now I check the logs for CMS and see this error.

We are getting this error. In every 3-5 minutes. All night and day. There is no user connected to the system. Most probably our problem is because of this error. I have search for this error but can not find anything.

Location: .\apsfactory.cpp:582:-: TraceLog message 262

Guid: 65de5699-76c1-e084-58ca-8eef3f686190

Time: 2016 04 08 09:24:33:812

Tzone: +0300

Trace: Error

Severity: Error

DeviceName: cms_ETI00BOP01.CentralManagementServer

ProcessID: 5292

ThreadID: 7152

MajorTick: 21

MinorTick: 0

MajorDepth: 1

MinorDepth: 0

RootName: BIPSDK.InfoStore:query

RootID: ETI00BOP01:11272:36.98775:1

CallerName: BIPSDK.InfoStore:query

CallerID: ETI00BOP01:11272:36.98775:1

CalleeName: cms_ETI00BOP01.CentralManagementServer.newService

Henry_Banks
Product and Topic Expert
Product and Topic Expert
0 Kudos

Hi,

that's precisely the point - if your BO system is idle, then the virtualised 'system resources' will be reallocated to another tenant on the VMWare. (you won't have visibility of that)

Next, when your BO needs to do some work - uh oh - the resources are being used elsewhere .. so your users could then experience slowness.

that error message isn't familiar to me (from the screenshot) although a search of the knowledge base seems to suggest it may be related to Information Steward (and fixed in 4.2 SP2) - see Note 2021116

regards,

H

Former Member
0 Kudos

I asked system guys how the environment structured. If it is a shared system or not.

We dont have Information steward.

Also on adaptive processing server side, there are many communication error. I have put in detail only last one. We dont have a clustered system.

Guid: 3567111C5C7B4F7FBBAF9E6237766BA7cd7

Time: 2016 04 08 14:46:34:537

Tzone: +0300

Trace: Error

Importance: ==

Severity: Error

DeviceName: aps_ETI00BOP01.AdaptiveProcessingServer1

ProcessID: 5276

ThreadID: 221

ThreadName: Monitoring Service Thread: 30

Unit: com.businessobjects.monitoring.trendingmanager.TrendingManager

Text: [com.businessobjects.monitoring.trendingmanager.TrendingManager$MetricDataCommitTask:invoke] unable to access CrystalReportsServices MBean.Continuing with next managed entity

javax.management.InstanceNotFoundException: Topology:name=null,type=ServiceCategories

        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1092)

        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:832)

        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:771)

        at com.businessobjects.monitoring.trendingmanager.TrendingManager$MetricDataCommitTask.invoke(TrendingManager.java:437)

        at com.businessobjects.monitoring.common.MonitoringTask.run(MonitoringTask.java:53)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)

        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

        at java.util.concurrent.FutureTask.run(FutureTask.java:138)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)

        at java.lang.Thread.run(Thread.java:763)

Former Member
0 Kudos

We have check the system configuration on VM. All the system resources (CPU and RAM) on the server are dedicated resources. Not a shared system.

Former Member
0 Kudos

Finally we have solved the problem. Our Solman system and BI instalations are on different networks. And between these networks these is a firewall terminating connection which they are idle more than 30 minutes.

We have moved BI system to the other network and problem solved. I never think SOLMAN - BO connection was that much important.