Connection issues while executing test with 100 portal users

[christian_gnther3]

Hello All,

I have an HCM system ECC 6 with NW 7 - SP 14 for ABAP and Java and Enhancement Package 2. I am in the process of implementing ESS on this system using the web dynpro for java iviews. My scenarios are all set and work fine.

I have a central instance and two dialog instances on different hosts - all Solaris 10 - and my DB is DB2.

My 2 dialog instances each have 2 Java server processes and 6 dialog work processes and all are configured to use 2GB roll area and 2 GB heap - according to SAP notes. I have setup a web dispatcher to do load balancing between the central- and the 2 dialog instance - in fact the web dispatcher is configured to connect users ONLY to the two dialog instances so the central instance with database is freed of the http load.

Overall system performance is good and the errors only happen in case 100 people access my system simultaneously.

My problem is, whenever I schedule a test with say 100 people I receive errors in my portal. The users see al sort of errors but they all conclude to connections beeing turned down or stating that no backend connection was possible. In my defaulttrace I can see errors like this:

Exception com.sap.engine.services.connector.exceptions.BaseResourceException:  Cannot get connection for 60 seconds. 
Possible reasons: 
1) Connections are cached within SystemThread (can be any server service or any code invoked within SystemThread in the SAP J2EE Engine), 
2) The pool size of adapter "SAPFactory" is not enough according to the current load of the system or 
3) The specified time to wait for connection is not enough according to the pool size and current load of the system. 
In case 1) the solution is to check for cached connections using the Connector Service list-conns command, 
in case 2) to increase the size of the pool 
and in case 3) to increase the time to wait for connection property. In case of application thread, there is an automatic mechanism which detects unclosed connections and unfinished transactions.

or this one:

The Timelimit for this iView was exceedet and no data from backend was received...

I tried to change nearly all parameter that I know of have something to do with number of connections to solve this issue:

• I set

> rdisp/appc_ca_blk_no = 2000

> rdisp/wp_ca_blk_no = 2000

> rdisp/max_arq and rdisp/max_comm_entries = 2000

> gw/max_conn = 2000

> gw/max_sys = 2000

• I increased the number of JCo connections in the system connection from 100 to 1000

• I increased the maximum number of JDBC connections from 50 to 100

• I increased the value of icm/HTTP/j2ee_0 in system profile from 500 connections to 1000

• Finally I increased the number of concurrent JMS Connections (with Visual Admin) from 100 to 200.

But now I receive a new error from the portal stating that the iView could not be displayed because the backend did not respond

I am stuck - can anyone point me to a direction which parameter to check or where to look for more info?

Thanks in advance,

Christian

[Former Member · Accepted solution]

Hi Christian

Congrats to this post, finally somebody takes some time to outline his situation. I am assuming that the backend is the ERP Abap part of the installation. I have two things as a start:

- If the timeouts occur after 60 seconds, then have a look at icm/keep_alive_timeout, the default is 60 seconds. I remember a case where setting it to 900 seconds solved a similar issue (make sure you don't override it in the icm/server_port_X settings)

- Even if the increase of the timeout above does help, the users might/should not have to wait over 60s to get a response, so you will need to investigate the situation in the backend system. Please have a look a the the work process occupation (6 processes might not be enough) and the cpu consumption. Do not forget to check the load on the CI/DB server as well. You should monitor the processing time as well, it might be necessary to tune the application in the end.

If you think you don't lack ressources on the CI-DB server, you can also try to add the CI (only for test purposes) to the web dispatcher logon group, and verify if this helps or not.

To help you further we will need more information on: what are the users doing (iview / transaction), where the bottleneck seems to be (cpu / io / database / dialog instance).

Regards Michael

[christian_gnther3]

I don't really understand this, but the error occurs again!

[christian_gnther3]

We transferred the iView with the Visual Composer model in it (remeber, it used a BAPI call to receive data about a user from a function module) to our productive system - ECC 6 HCM with SP14 on SAP NetWeaver WebAS 7.00 with SP 14.

The moment about 100+ users try to logon to our portal and make use of the model, I can see the following entries in my defaultTrace:

Cannot get connection for 60 seconds. Possible reasons: 
1) Connections are cached within SystemThread(can be any server service or any code invoked within SystemThread in the SAP J2EE Engine), 
2) The pool size of adapter "SAPFactory" is not enough according to the current load of the system or 
3) The specified time to wait for connection is not enough according to the pool size and current load of the system. 
In case 1) the solution is to check for cached connections using the Connector Service list-conns command, 
in case 2) to increase the size of the pool and 
in case 3) to increase the time to wait for connection property. In case of application thread, there is an automatic mechanism which detects unclosed connections and unfinished transactions.

I thought we hit the point with our adaptions above, but it seems that we did not.

Could someone have a look at this and give me a hint?

Thanks in advance,

Christian

[Former Member]

Sorry to hear that you still have issues. I suggest you have a look at these things (they are garbage collector related):

- Check the top running processes on the os during the test. Is the server0 (should appear as jlaunch) process consuming lots of cpu?

- Check the dev_server0 and std_server0.out in the work directory, you can do a tail -f and lookout for garbage collection information. They look like this:

121795.818: [GC 121795.818: [DefNew: 173443K->57507K(262656K), 0.3298834 secs] 7
32956K->617020K(1223168K), 0.3301172 secs]

Best regards - Michael

Edit: hobbla, I didn't see your 2nd post regarding the SAPFactory, cannot help you there, but you might still check on the garbage collection issue...

[Former Member]

Hi Christian,

We experienced the same issue today - thank you for your helpful hints. After we made some of the settings recommended (here are ours):

gw/keepalive = 300

rdisp/rfc_min_wait_dia_wp=5

rdisp/wp_ca_blk_no=1000

rdisp/appc_ca_blk_no=2000

gw/max_shm_req=400

gw/max_overflow_size=25000000

gw/max_sys=4000

gw/max_conn=4000

rdisp/max_arq=4000

rdisp/max_comm_entries=4000

rdisp/tm_max_no=2000

We restarted both Portal and in our case, BI, in order to clear the issue. After backend restart SMGW still showed lots of hanging sessions so that's why we restarted Portal as well. If it remains an issue I suggest checking that Portal is correctly closing the backend sessions created at runtime.

We found also that there were a high number of Portal connections to the backend and are still looking at that issue. We had a few unnecessary JCO connections in the VA which we have removed.

I trust this helps,

Marc

[Former Member]

Hi,

We are also getting the same errors during the load test with just 4 concurrent users. Did you fix the problem? If so, can you please share with us.

Thanks!

[christian_gnther3]

Ok to all of you. Here is what we did to fix the issue:

1. setup Dialog instances with 1 java server node each and routed all http-users through to these - we use a web dispatcher for this. So no user is accessing the central instance via http directly anymore.

2. set the following parameter:

a) increased the number of concurrent jdbc connections from 50 to 100

Visual Admin -> Global Configuration -> Server -> Properties -> sysDS.maximumConnections = 100

b) increase number of jms connections to 200

Visual Admin -> Server -> Services -> JMS Connector -> Properties -> maxConnections = 200

c) in the profile set the following params

rdisp/appc_ca_blk_no = 2000

rdisp/wp_ca_blk_no = 2000

rdisp/max_arq = same as rdisp/max_comm_entries => 2000

gw/max_conn = 2000

gw/max_sys = 2000

em/global_area_MB = 1500

icm/HTTP/j2ee_0 = PREFIX=/, HOST=<hostname>, CONN=0-1000, PORT=5$$00

d) On os level set the environment variables (remember this is Solaris 10 with zoning)

CPIC_MAX_CONV = 2000

We also set the following Java parameter for all our server and dispatcher nodes:

Server:

-Djava.security.policy=./java.policy

-Djava.security.egd=file:/dev/urandom

-Dorg.omg.CORBA.ORBClass=com.sap.engine.system.ORBProxy

-Dorg.omg.CORBA.ORBSingletonClass=com.sap.engine.system.ORBSingletonProxy

-Djavax.rmi.CORBA.PortableRemoteObjectClass=com.sap.engine.system.PortableRemoteObjectProxy

-Dorg.xml.sax.driver=com.sap.engine.lib.xml.parser.SAXParser

-Dsun.io.useCanonCaches=false

-Djava.awt.headless=true

-Djco.jarm=1

-Xms2048m

-XX:PermSize=512m

-XX:MaxPermSize=512m

-XX:NewSize=280m

-XX:MaxNewSize=280m

-XX:SurvivorRatio=5

-XX:TargetSurvivorRatio=90

-XX:SoftRefLRUPolicyMSPerMB=1

-verbose:gc

-XX:+PrintGCTimeStamps

-XX:+UseParNewGC

-XX:+DisableExplicitGC

-XX:+HandlePromotionFailure

-Dorg.omg.PortableInterceptor.ORBInitializerClass.com.sap.engine.services.ts.jts.ots.PortableInterceptor.JTSInitializer

Dispatcher:

-Djava.security.policy=./java.policy

-Djava.security.egd=file:/dev/urandom

-Djco.jarm=1

-Xms256m

-verbose:gc

-XX:+UseParNewGC

-XX:+PrintGCTimeStamps

-XX:+DisableExplicitGC

This last line is appended on the dialog instances for dispatcher and server

-Drdbms.driverLocation=/db2/db2<sid>/sqllib/java/db2jcc.jar:/db2/db2<sid>/sqllib/java/db2jcc_license_cu.jar

I think that was all the tuning we did so far.

Hope it is helpful,

Christian