Parrallel Processing in xMII

[Former Member]

Hello All.

I am designing a large scale Real Time Reporting solution using regional xMII servers to source data that is then fed into NetWeaver BI Virtual InfoProviders for reporting.

My current problem is that I can't figure out how to make xMII do two tasks in parallel. Let me explain why I want to do this.

Suppose I need data for a report from multiple source applications that are connected to a regional xMII server. I can have Net Weaver BI make a call to the xMII server for the data, and then xMII needs to collect the data from each of the sources. At the moment, xMII makes a request to the first source, waits for the response, then makes a request to the next source and waits for the response, and then the next...etc

Ideally, I need to have xMII initiate each of the requests from the source systems and run the queries in parallel. When the last query has finished, then post the entire result set back to Netweaver BI.

How do I spawn Multiple queries in parallel in xMII?

Geoff.

[Former Member]

Thanks for all your help.

We will request the modification to xMII as a functional enhancement, and in the interim, investigate using an intermidiate service to kick off requests.

Geoff.

[Former Member]

Hi, Geoffrey.

I suspect that there is significant potential performance that can be gained from "intermediate content caching". There is probably some overlap between subsequent user requests that could share a previous resultset.

xMII's business logic service engine was designed such that multiple actions within a sequence could run in parallel (I designed it. ) , but this capability was not enabled for a number of reasons related to scaleability when the product ran on IIS (version 11.5 and older).

I would suggest pushing this is an enhancement request - the infrastructure to do this would not be too substantial and would be useful if used carefully.

Alternatively, it would be rather easy to write a standalone app/servlet that invoked xMII transactions in parallel (via calls the the Runner servlet) to make the data requests (and persist the results to disk temporarily), then invoked the aggregrator/calculation transaction. Not a perfect solution, but would work.

Best regards,

Rick

[Former Member]

what if you put all of the queries in the same sequence block in bls?

[Former Member]

Even if the queries are in the same sequence block, they are still actioned sequentially. I.e. Start one action, wait for it to complete, start the next action.

We have tested this and proven this to be the case.

[Former Member]

What are you trying to do? What is the importance of parellelism in your task?

[Former Member]

I am using Netweaver BI for Real time reporting off MES systems. To do this, I am using the Virtual Info Providers in Netweaver BI to connect to xMII Transactions

xMII then sources data from various MES Source systems and returns the union of the result set.

Because this needs to happen on a Request / Response model (I.e. User Clicks Generate Report in Netweaver BI, and then waits for the result set to be displayed on the screen) speed is off the upmost importance.

I need to be able to spawn multiple request in parrallel to optimise the speed of response to BI.

If I have many data sources, and request data from each source sequentially, the response time will be unacceptable, but If I can spawn mutiple requests, with the response time only equal to the longest single request, rather than the summ of all requests, this will be acceptable.

[Former Member]

Geofffrey,

In BLS, multiple action blocks in a sequence will execute in parallel. However, the next sequence will not be executed until both actions have completed. If you are only looking to query data from a datasource, then use the respective action blocks in a single sequence. If you must perform additional shaping of data, then I would suggest a transaction for each data source, which includes the additional conditioning. In a parent transaction, place two transaction call action blocks in a single sequence to fire off each child transaction. This way, all data retrieval is handled independently. You can call this transaction via a url call.

Regards,

Michael Teti

[salvatore_castro]

All,

For clarification <b>everything</b> in the BLS environment occurs in serial...no exceptions. When multiple actions are placed in the same sequence the are performed in a serial fashion from left to right. However, with that said there is nothing that says in a future release that this wont change.

Geoffrey,

If you would like to achieve parallel processing of data then have BI call three separate transactions in parallel to return data. The JDBC driver uses connection pooling so you will not have multiple connections opened and closed on the JDBC side. This will give you the best performance from a parallel processing standpoint but you may be overdesigning your solution, you're better off making all the calls from a single transaction rather than maintaining three separate integration points into the xMII system.

Hope that this helps.

-Sam

[Former Member]

Thanks Sam.

You have hit on my current dilema. Like you suggest, I can make this work by forcing BI to call each individual request, but doing this creates a large overhead on the BI side as I effectivly need to have a seperate virtual Info provider for each request.

I was hoping to have a simpler structure in BI where I used a single, generic Info Provider to call a single generic transaction in xMII, and then have xMII make the various requests, union the result sets and return a single response back to BI, hence the need for parrallelism im xMII.

There are several other options I am considering.

Write a custom Web Service outside of xMII to spawn mutiple requests into xMII.

Use XI between BI and xMII spawn multiple requests in xMII

I have been told that it should be possible to write a custom action block in xMII that can create parrallel requests.

Regards

Geoff Nunan

[Former Member]

Any chance of caching the queries whether through xMII caching capabilities, or pumping summary data from BI to a SQL database or something on a routine basis?

How fast are the queries coming out of BI (executing from BI)?

[salvatore_castro]

Geoffrey,

I understand your dilemma and recommond to you to use XI or some other middle layer wrapper to perform the multiple requests in parallel. Yes it is possible to create an action block that will spawn multiple execution threads but this will not be a reliable solution going forward as it will not be Cluster Safe and therefore can cause additional problems in the Netweaver environment of v12 for xMII.

Additionally, I have experimented a bit with the multi-threaded transaction call idea already but there's no way to wait for the threads to finish and process their responses. The call can be made but that's where the control ends.

Hope this helps to answer your question on how to proceed.

-Sam