SAP .NET Connector Threading issues with ASP.NET 1.1

[Former Member]

Hi,

I am running a Web application with the SAP connector for .NET and am experiencing what i think is a threading issue with the SAP .NE Connector's Thread Pool.

- Windows 2003 Enterprise Server

- IIS 6, ASP.NET 1.1 (machine.cfg changed to set threads to ~100 or so ...)

If I use only 1 process for the application pool, I seem to get bogged down with a throughput of about 6-7 requests per second. My App/Web server is only about 10% or less CPU%. If I increase the number of concurrent client connections from 10 to 50 or 100, the throughput remains the same, and response time increases proportionally. This indicates a client-side limit on the number of threads servicing requests (or a SAP server side performance problem - which is not the case as seen by further tests). Response time is about 300ms for a single connection, and grows as the number of concurrent connections/requests grows.

If I increase the number of processes servicing the IIS6 .NET 1.1 application pool from 1 to 10, my throughput goes up to above 40 requests per second, 2 seconds response time avg. and my web/app server's CPU is now the bottleneck. This indicates the SAP server and network link etc, are all up to the task.

I set the connector parameters in a shared function as shown below. I initially used AutoPooling but then changed to use

oSAPProxy.Connection = Connection.GetConnectionFromPool(oSAPDestination.ConnectionString)

and

Connection.ReturnConnection(oSAPProxy.Connection)

Public Shared Sub init()

Config.Instance.ConnectionPool.CleanupInterval = 300

Config.Instance.ConnectionPool.MaxCapacity = 200

Config.Instance.ConnectionPool.MaxIdleTime = 300

Config.Instance.ConnectionPool.MaxOpenConnections = 200

'Config.Instance.ConnectionPool.UseAutoPooling = True

End Sub

I feel the shared sub is not setting the SAP connection parameters properly. Is there a way to set these in a config file (that the SAP connector uses to initialize itself with) or something?

I don't have a place (such as a startup method etc.) where I can put such code. I dynamically load an assembly (part of a framework) which in turn references the SAP Connector assemblies.

If I look at the number of concurrent connections using TCPView, with a 1 process App pool, I get about 8-10 TCP connections to the SAP server. With the 10 process app pool, I get above 50 (hard to count etc.) connections.

My framework handles the 50-100 concurrent connections in separate threads OK. I set up an adaptor with about 300ms delay etc, and throughput & response times all look OK.

Any ideas?

Are the connector settings taking effect the way I coded them up? Is there a way to set them in a config file that the connector will use (i.e. override the defaults in config rather than in code?).

Does the .MaxOpenConnections and .MaxCapacity refer to the size of the thread pool?

If I look at task manager etc. I can see my w3wp.exe has 100 or so threads running.

[reiner_hille-doering]

Here some answers:

1. You init sub is ok and will configure NCo accordingly.

2. Instead of coding the config, you can declare it in your web.config. How is explained in the reference documentation of Config class ( ms-help://MS.VSCC.2003/SAP.NCO/NCOV2Docu/reference/SAP.Connector.Config.html )

3. The .NET thread pool and the NCo Connection pool don't have a lot in common. The only thing in NCo that uses the Thread pool is the Cleanup thread fro the connection pool. So setting the thread pool to 100 shouln't have an effect on NCo.

4. The performance you can achieve is usually limited by the SAP server(s).

5. In some cases Connection Pooling can reduce the performance: Assume you have many application servers and you you a connection string with load balancing. If now for some external reason the "GetConnection - Call - ReturnConnection" blocks are serialized, you would only use a single backend connection, independently how you configure the pool. This happens because GetConnection would allways take the connection just returned from ReturnConnection before. This would bring high load on one SAP server and keep all others idle.

As consequece you need to make sure that all you "use blocks" are really executed in parallel. Of causes this will happen more likely if the call(s) in the middle are non-trivial and take some time.

[Former Member]

1 - Thanks

2 - Great thanks. I'll try that...but need to get to customer site again! Unfortunately this means next stage of the project...

3 - I've set both pools to 100. Seems to have no effect.

4 - In this case, I get 6-7 rq/s & 7 second response time, with a 1 process app pool, and 10% local CPU utilization. And I get 40+ rq/s & 2 second response time with a 10 process app pool & 100% local CPU utilization. With the 6-7 rq/s & 7 sec. resp. time , the SAP Server is obviously not the bottleneck in this case. This is a serious performance bottleneck for us. 40+ rq/s is not really an issue, but in that case, the local PC is the bottleneck. I'm not concerned with the latter.

5 - I first tried without pooling, got similar results ~4-6 rq/s or so (can't remember response time, but I suspect similar...)

We don't have a load balancing setup as far as I know. The SAP connection string was setup with single server parameters etc.

With the 1 process app pool, I have 100 threads calling into SAP in parallel, though only a small percentage of those are actually getting done in parallel (if any!).

==>If now for some external reason the "GetConnection - Call - ReturnConnection" blocks are serialized

My app is not serializing calls as far as I can tell. I've done other tests (such as put a delay in one of my components & measure throughput etc.), and they all exhibit parallel execution behavior.

The SAP Connector for some reason is not doing this.

Is there anything I can do to test? Can I do anything to test concurrency & calls going into the connector without a SAP server by any chance? I'd need to go onsite to do these tests with a SAP server, and it's a bit of an ordeal if done outside of the project plan.

I'd like to tinker & get it working nicely in my own time so that I know it's all working nicely.

There's no demo SAP server or anything that you guys expose by any chance is there?

Thanks for the prompt reply. At least I've got definitive answers for 1 & 2 (although I'm a little skeptical about 1 though...I'll definitely put 2 in next time I'm there.

[reiner_hille-doering]

Unfortunately I can't you tell you in detail what happens in you app and what the limiting factor is. My suggestion is that you instrument your app and write a trace file so that you can see where your threads are. You can also turn on NCo tracing to get some hints.

There is a number of possible reasons and you have to understnd how the NCo Connection Pooling works, it's really trmendously simple: ReturnConnection would put an open connection into an array so that it can be re-used by the next GetConnectionFromPool.

Some typical mis-uses:

- All connection strings are different. Each GetConnectionFromPool would create a new connection. These connections would stay in the pool until it overflows or until they exceed the idle time.

- You have a lot of parallel incoming requests and the the GetConnection-Call-ReturnConnection block is long-running. If you have set the MaximumOpenConnections and PoolSize too high, each incoming call would create a new Connection (because there are no unused connection in pool). Later you would have a high number of connections in the pool that are only partly used.

It could be also the case that something else in your client code limits the performance, again you can only find this out by intrumenting your app.

[Former Member]

Thanks ...

I have 1 unique request, 1 connection string,

See code below:

Apart from this bit of code, my framework is multi-threaded & everything runs in parallel.

I'll see if I can turn on tracing on the connector.

==>You have a lot of parallel incoming requests and the the GetConnection-Call-ReturnConnection block is long-running.

...This is my setup (long running ~ 0.3 - 0.5 seconds or so...)

If you have set the MaximumOpenConnections and PoolSize too high, each incoming call would create a new Connection (because there are no unused connection in pool).

==>This does not seem to happen. My 100 ASP.NET threads seem to be waiting...

The SAP Connector only opens up about 10 connections.

I have 40 other threads waiting.

If I replace the function below with something else that has a similar processing delay/latency, I get about 60-80 requests per second, with a response time closely related to the delay/latency in that bit of code.

Can you see any immediately obvious issues with the function below (GetPOItemsFromSAP())?

-

---

50 concurrent users

-

---

Throughput = 6.3 requests per second

Response time = 7.3 seconds

TCPView - 10 SAP connections open

CPU ~10%

Later you would have a high number of connections in the pool that are only partly used.

Public Function GetPOItemsFromSAP()

Dim sAcctasscat, s2, s3, s4, s5, s6, s7, s8, s9, s10, s11, s12, s13, s14, sPurchaseOrder, s16, s17, s18, s19, s20 As String

Dim s21, s22, s23, s24, s25, s26, s27, s28, s29, s30, s31, s32, s33, s34, s35, s36, s37, s38, s39, s40 As String

Dim x As New BAPIEKKOLTable

Dim y As New BAPIEKPOCTable

Dim z As New BAPIRETURNTable

Dim oSAPProxy As New SAPProxy1

Dim oSAPDestination As New Destination

oSAPDestination.AppServerHost = System.Configuration.ConfigurationSettings.AppSettings("Destination1.AppServerHost")

oSAPDestination.Client = System.Configuration.ConfigurationSettings.AppSettings("Destination1.Client")

oSAPDestination.Password = System.Configuration.ConfigurationSettings.AppSettings("Destination1.Password")

oSAPDestination.SystemNumber = System.Configuration.ConfigurationSettings.AppSettings("Destination1.SystemNumber")

oSAPDestination.Username = System.Configuration.ConfigurationSettings.AppSettings("Destination1.Username")

Try

oSAPProxy.Connection = Connection.GetConnectionFromPool(oSAPDestination.ConnectionString)

sPurchaseOrder = m_oPlgRq.PONumber

oSAPProxy.Bapi_Po_Getitems(sAcctasscat, s2, s3, s4, s5, s6, s7, s8, s9, s10, _

s11, s12, s13, s14, sPurchaseOrder, s16, s17, s18, s19, s20, _

x, y, z)

If z.Count > 0 Then Throw New Exception(z(0).Message())

Dim oPOItem As PurchaseOrderSummaryRs.POItem

oPOItem = m_oPlgRs.POItemCollection.Add()

oPOItem.MaterialDescription = y(0).Material

oPOItem.Plant = y(0).Plant

oPOItem.ProposalQuantityToBeGRNed = y(0).Disp_Quan

oPOItem.QuantityInPO = y(0).Po_Unit_Iso

Catch ex As Exception

Throw ex

Finally

Connection.ReturnConnection(oSAPProxy.Connection)

End Try

End Function

Thanks again.

[reiner_hille-doering]

Indeed I see some issues in your code:

>> Dim oSAPProxy As New SAPProxy1

This call is extremly expensive. This is because Microsoft's Soap Client that we use as base class uses the XMLSerializer that creates a temporary assembly and loads it. Create it earlier and reuse it. NCo ensures that the many calls with the same proxy are syncronized. If your proxy is not stateless (has proxy fields), you could have a pool of proxies.

Dim oSAPDestination As New Destination

oSAPDestination.AppServerHost = System.Configuration.ConfigurationSettings.AppSettings("Destination1.AppServerHost")

oSAPDestination.Client = System.Configuration.ConfigurationSettings.AppSettings("Destination1.Client")

oSAPDestination.Password = System.Configuration.ConfigurationSettings.AppSettings("Destination1.Password")

oSAPDestination.SystemNumber = System.Configuration.ConfigurationSettings.AppSettings("Destination1.SystemNumber")

oSAPDestination.Username = System.Configuration.ConfigurationSettings.AppSettings("Destination1.Username")

This is also not very fast, better do this once and pass the Destination in. It's anyway allways the same.

>>oSAPProxy.Connection = Connection.GetConnectionFromPool(oSAPDestination.ConnectionString)

You could also pool over the Destination directly. This is slidely faster.

[Former Member]

RE:

Dim oSAPProxy As New SAPProxy1

This call is extremly expensive. This is because Microsoft's Soap Client that we use as base class uses the XMLSerializer that creates a temporary assembly and loads it. Create it earlier and reuse it. NCo ensures that the many calls with the same proxy are syncronized. If your proxy is not stateless (has proxy fields), you could have a pool of proxies.

1)Thanks. I expected it'd be a little expensive, but unfortunately, the architecture & framework I have is completely stateless and does not initialise an adaptor.

(So far, this has not been a problem for other systems I integrate to, although I can see how it'd be useful sometimes. Problem is that stuff tends to hang around that way, and somewhere along the line, someone will manage to hold onto state they should not.

Although it's still possible to do this with this infrastructure, you have to do it all yourself & be conscious of it. Anyway, I digress...)

Would this be causing single threading issues? If so, I'd have to build some sort of state into my SAP adaptor that calls into the SAP connector.

Note that I haven't experienced performance issues on my side in terms of local CPU or disk I/O (not that I've noticed anyway). All I've noticed is the apparent serialization of calls to SAP.

I do not want to serialize any calls to SAP - at all.

I'm trying to get them all to run in parallel. That's my problem. The more I look at the numbers, the more it seems that I can only get 1 parallel thread (to talk to SAP) or so to run at a time per process..., even though my ASP.NET worker process has 100 threads running...

Stuck!

Same deal with the other bits of code.

I know I'm reading stuff multiple times. In the production code, I'll use a quicker cache to store the values so I don't have to use the std. config classes etc.

What I wanted to run past you was if there's anything in this bit of code that would cause single-threading.

That's my issue.

Unfortunately, if I can't sort that out, I have to run multiple processes to get around it - not very nice at all. Very heavy...

[reiner_hille-doering]

I don't know what happens in the constructors. Maybe that MS synchronizes them somehow, definitely it's slow.

Even if you don't have an Init function, you can still have the proxy being a member variable that is initalized if needed (I write in C# because I know this better):

lock(this)

{

if (this.myProxy == null)

this.myProxy = new MySAPProxy();

if (this.myDest == null)

{

myDest = new Destination();

...

}

}

Let the GC do it's job to clean up later...

About serialization of calls: Yes, NCo does sometimes synchronice calls that they are executed sequentially, but ONLY if the use the same connection. But this is no problem, because GetConnectionFromPool will give your thread an exclusive connection.

I can guarantee this behaviour at least for the new version 2.0.1 that came out yesterday. Therere has been some change in ths area, so it can be that older version behave a little bit different.