Advice needed on large volume of data coming from aRFC.

[Former Member]

Hi Experts,

Using NWDS 7.0.18, EP 7.00 SPS 18

I am calling an rfc to bring back a table structure to the front end and then I am building a tree hierarchy out of it. Everything was fine with small amounts of records (100-200).

But in real life, we will have approx 300,000 records coming in. As a test, we tried it with just 50,000 but the application wasnt able to cope. I just got the web dynpro spinning wheel for almost an hour and then nothing!

Can anyone provide me with some advice on how this can be resolved? Is there a way to preload the data before the user accesses the app? Can I bulk load small sections of data at a time?

Thanks in advance.

Marshall.

[Former Member · Accepted solution]

Hi,

We also faced same kind of issue in my project

If you have more records please check with the backend team wether they can do any changes to increase the performance of the aRFC.

But from portal side make sure that you are not using for loop inside a for loop.

Say first for loop has m records and second for loop has n records then if you are using a for loop inside a for loop it will loop as m*n times.

so please maintain a single for loop from portal side and avoid nested looping.

Regards,

Saleem

[Former Member · Accepted solution]

Hello All,

Thanks alot for all the answers.

I am using a recursive function call to load all the data into the tree structure, the reason I do this is because so I can use Expand All function from first load.

I also do have ONE nested for loop...which determines what level each record from the table belongs to in the Hierachy.

So, from what I gather from what everyone is saying, is that the problem isnt actually retrieving the records from the BAPI, the problem is actually how the front end processes these records??

Is that true?

Any tips how I can do this with using recursive function call to load all the data into the tree? I have to do this because the user might click Expand All as soon as s/he starts the application.

Thanks in advance

Marshall.

[Former Member · Accepted solution]

Hi ,

Yes Marshall you have to avoid the nested looping from front end process prespective.

And try to maintain the levels of a table rows in a list object and check inside the first for loop.

Say if your first loop contains m records then say your list contains n records your loop should run for m+n times rather than m*n times. Change your logic in this fashion..

Hope this might help you..

Regards,

Saleem.

[Former Member · Accepted solution]

Hi,

As per my guess,

Most of the time is taking at recursive call and inner for loop at UI side.

Just put these statements and calcuate the time taking as below for both BAPI call and logic at WD side.

long before=System.currentTimeMillis();

long after=System.currentTimeMillis();

long totalTimeTaken=System.currentTimeMillis();

For example your method is

getDatafromBackend()
{

//Code for model execution
long before=System.currentTimeMillis();
--------------------------------------------------
-------------------------------------------------
long after=System.currentTimeMillis();

long totalTimeTaken=System.currentTimeMillis();

}

Similarly for your recursive logic also.

Using these checks you can findout..where is the issue.

Note if you are facing issue for 50,000 records..then check for 40000 first.

As per my analysis if you are calling the method recursively for 50,000 records.

*Then the for loop will execute 50,000 * 50,000 times. This is very minimum. There are many chances like one parent can have many childs. Those childs can have again childs. So this number will increate drastically based on the parent and child relationships in the data.*

Better not to use recursive for this much of data records. So dont populate the records at the beginning. Populate when user expands the parent.

Regards,

Charan

[Former Member · Accepted solution]

Hi Charan,

Thanks for the response....I just figured out, the problem is not with the BAPI call, the RFC gets 50,000 records within a couple of seconds.

The problem is definitely with the recursive call to load the tree.

The thing is, can you provide me with suggestions on how to use expand all functionality without loading the tree first? Initially I was loading the children when user expands parent. But then you provided me with a solution for expand all by loading the entire tree first!! Now I'm getting stuck with large volumes of data!!

Regards,

Marshall.

[Former Member · Accepted solution]

Hi,

As per my knowledge there is no possiblity of loading the entire tree structure at first time itself without using recursive calls.

Note: Expand All functionality works only after the entire tree is loaded.

So as per my knowledge I suggest that as the data is very very huge its better not to load the entire tree at one time. Load the childern when user expands the parent node.

You have to remove the Expand All functionality.

Yes I have provided the expand all functionality. But at that time I have no idea that your runtime data will be this much huge. And you have only asked me to provide the solution for expand all...:)

Regards,

Charan

[Former Member · Accepted solution]

Charan,

At that time even I had no idea my runtme data will be this huge!! But I cannot thank you enough for all the help that you have provided.

I guess I will have to drop the Expand/Collapse All functionality.

Thanks

Marshall

[Former Member]

Decided to load just 50 records at a time.

[Former Member]

Follow up questions related to this thread.

It seems like Web Dynpro is really not capable of handling large volumes of real life data coming from SAP Backend...I have removed practically all recursive calls, nested for loops, and sacrificed functionality such as Expand All and Collapse All. Still I am not able to efficiently load, and manipulate even just 500-1000 records quickly enough. Response time is slow. Is/Has anyone else faced similar issues? In production we will be working with almost 300,000 records coming from BW/BI. But yet just a 1000 is causing huge slow times.

I have re-opened this thread because I still could use any advice out there that anyone can offer? Will Web Dynpro abap be a better solution?

Please Help!!!

Edited by: Marshall Mathers on Jun 16, 2009 11:43 PM

[Former Member]

Hi,

The problem here is building tree structure for 50,000 to 300,000 records.

I guess here the problem is with the tree structure:

How are you building the tree strutue?

Are you loading entire tree at the first time of screen loading?

If yes, to achieve this you have to use recursive funtion. Recursive in the sence for loop inside the for loop. When there is a chance of getting huge amount of runtime data recursive (nothing but for loop inside for loop) is allways strongly not recommended.

Are you loading the childern when parent is selected?

I think here we dont use the recursive function. So in this case your application should work with good performance.

Note: But make sure you dont have for loop inside another for loop / recursive calls in your code.

Hope this information helps.

Regards,

Charan

[Former Member]

hi

First w.r.t your application of getting huge data , what is the requirement you are

trying to .

i had the same scenario where we need to get 30,000 records but most of the filtering

was done in the backend and some from my side .

filtering as removing the duplicated copies of data from the table strucure and the storng

in the other table , this would reduce the amount of records you need to fetch .