Is there a max size for a blob?

[Former Member]

PB Classic 12.5.1 build 4015

Is there a max size for a blob?  Documentation says "unbounded", but I'm seeing problems.

Before starting, I will mention that I have already sent the email about why we have so much data in a datawindow anyway and there are probably better ways to accomplish what the users want, but I'm still curious.

We use GetFullState/SetFullState to move a report between objects in different threads.  I don't know the exact limit I'm running up against, but the GetFullState command succeeds when the resulting blob is 30Mg and under, but hangs when I estimate the result should be about 150Mg.

A GenerateResultSet succeeds on the same datawindow and I tried to modfiy our system to use that instead, but it doesn't get all the data from nested reports (unless I'm just not using it right).

The machine I'm testing with has 16Gg memory but less than 4Gb used while running this.

[Former Member · Accepted solution]

Hi Tim;

  Sorry for the memory switch digression but the thought there was that maybe your Blob size limitation (way before the 2G limit) was because your PB Application was running out of usable memory when running as a 32 bit application. Especially,  since you mentioned "threads" - so I assume that you are running a multi-threaded application.

  If that is truly the case and you are running up against the 2G application memory space limitation then the following things might help you work around this issue:

1) You can try either mechanism that Bruce or I mentioned to raise the usable memory to 3G - but, in my experience doing this, many applications can become really unstable when running in this mode. However, it might be worth a try and see what happens in your case.

2) You could consider upgrading to PB 12.6 where Classic application .EXE's can now be compiled to 64bits.

3) The key with my applications is that I release large storage ASAP. In the case of your application using Blobs, I would code the following (if you already haven't) ... for example:

Blob   lblb_null

Blob   lblb_data

Long  ll_len

SetNull (lblb_null)

ll_len = FileReadEx (li_FileNo, lblb_data)

<blob processing>

lblb_data  = lblb_null

4) Make sure that you destroy any Classes ASAP that you no longer need.

5) Never use Auto Instantiated Classes as do not release their memory until the application is closing.

6) Try using the GarbageCollect ( ) commands after each processing loop

7) Always clean-up your DW Controls or DS's as follows to also release DWO's memory.

DS.Reset ()

DS.DataObject = ""

Destroy DS

HTH

Regards ... Chris

[Former Member · Accepted solution]

As for destroying objects, unfortunately, my processing is quite simple (after stripping out error checking and tracing cmds) the code I'm looking at is only this.

ll_return = lds_ds.retrieve( <args> )

ll_rc = lds_ds.GetFullState(lbl_blob)

The retrieve succeeds.  There is nothing to destroy until after the GerFullState completes, which it does not.  But we do destroy objects, even local ones like lds_ds above, when it succeeds.  This code has been running for years at multiple locations and this is the first time we're seeing it hang up.

I tried getting more info from Task Manager.  The above code runs in an nvo instatiated with SharedObjectRegister, which is why I used the term threads.  However, while I see two entries on the Applications tab, right-clicking and selecting Go To Process on either one takes me to the same exe in the Processes tab.  That may be how it is supposed to work, but I was expecting 2 separate processes.  I've never used multiple threads personally and didn't write this process either, just maintaining it.  So that might very well be how it's supposed to look.

Anyway, while waiting on the database, the memory used by the exe on the Processes tab is 8,612K.  Once the data starts coming back for the retrieve, the size increases steadily to 312,196K over a 30 second period.  Then, over the span of 3 seconds, the size increases to 710,080K, and at that point it just hangs.

The memory column I used is "Memory (Private Working Set)".  I added Peak Working Set and noticed that QweryBuilder, which I'm pretty sure is also 32bit PB, has been up to 830,396K.  So it doesn't look to me like I've hit any limits yet.

I Ran another test looking at Peak for my app and it topped out at 773,860K.

Thanks to both of you for the tips and insights, but I think I'm just going to have to throw my hands up on this one and look for a better way to accomplish the users goal (which I still don't know what that is).  I don't see much sense in debugging problems with an 8,000 page report.  Unless you have any ideas on how to get GenerateResultSet to work with composite and nested reports.  I can make that change easily, but I don't feel like looping through nested reports and returning multiple result sets, just to fix this one problem either.

Tim

[Former Member · Accepted solution]

Hi Tim;

  Ouch .. I agree!

   IMHO .. I think your only easy short-term solution get around this  - what looks like a memory limitation - might be to try PB 12.6 (the 30 day Eval is free to download). Using the 64 bit compile option might/should allow you to break free of the Blob memory limitations of  PB 12.5.1. That is, if your PB shop is willing to entertain a migration to PB 12.6.

Good luck!

Regards ... Chris

[former_member190719 · Accepted solution]

You might try running the 4GB batch tool on your PowerBuilder executable (the not IDE, but on the executable you generate from PB).  It turns on the LARGEADDRESSAWARE flag so your 32 bit app can see 4GB on a 64 bit system:

Bruce Armstrong's blog: Nice tool for Increasing memory for 32 bit apps on 64 bit systems

[Former Member · Accepted solution]

Thanks for the thought.   I tried it, and verified in Task Manager that I was in fact running the modified exe, but no difference.  My memory allocation on the app peaked at 712,952K.

Scratch that.  I don't think I was running the new exe after all.

UPDATE!!!

Holy crap, it worked!

Is there a way to set this so all builds will use it?  Or will I have to run this batch patch on it after each build?

[former_member190719 · Accepted solution]

I don't think the tool has a command line mode.  If it did, you could use OrcaScript to do your compile and then call it.  All I can suggest is contacting the author to see if they might a version of it that would support a command line option.

[Former Member]

Hi Tim;

  Since PB 12.5.x is 32bit - the Blob upper limit is 2G. Also, since its a 32bit application .EXE your running .. The maximum application  memory it can utilize is 4G as well. Remember though that that 4G is 2G for your App.EXE + O/S over head.

  FWIW: I use Blobs all the time that are easily 100M - 1G in size with no issues.

HTH

Regards ... Chris

[former_member190719]

I'm not sure where you got any of those numbers from.

A 32 windows executable is limited to 2GB of memory unless it has the LARGEADDRESSAWARE flag on, in which case it can access up to 3GB of memory.  By default, 32 bit applications have the flag off.

https://msdn.microsoft.com/en-us/library/wz223b1z.aspx

Since a 32 bit application (without the LARGEADDRESSAWARE flag on) would need to reserve room for itself, it wouldn't be able to handle a 2GB blob.

[Former Member]

If you read my response carefully, you'll see I said 2G for the EXE and 4G for an application address space in total. Of which, the Windows O/S uses the remaining part of the 4G space for its own runtime work area. Yes, older 32bit MS-Windows limited you to 2G but you could use the /3G switch. Newer 64 bit O/S versions will give you the 4G space when running a 32bit application using WOW.

[former_member190719]

If you read my response carefully, you'll see I said 2G for the EXE 

I did read it correctly.  You said:

Since PB 12.5.x is 32bit - the Blob upper limit is 2G. 

You also say:

Newer 64 bit O/S versions will give you the 4G space when running a 32bit application using WOW.

Once again, that's only if the LARGEADDRESSAWARE flag is set.  If it isn't, a 32 bit application is still limited to 2GB, and by default the flag is not set:

https://msdn.microsoft.com/en-us/library/windows/desktop/aa366778(v=vs.85).aspx#memory_limits

Memory type	Limit on X86	Limit in 64-bit Windows
User-mode virtual address space for each 32-bit process	2 GB Up to 3 GB withIMAGE_FILE_LARGE_ADDRESS_AWARE and 4GT	2 GB withIMAGE_FILE_LARGE_ADDRESS_AWAREcleared (default) 4 GB withIMAGE_FILE_LARGE_ADDRESS_AWAREset

[Former Member]

That would be true for VS applications. However, for PB applications - I use the "/3GB" switch.

FYI:  /3GB - Windows 10 hardware dev

[former_member190719]

You can use that option all you want, it won't have any effect.

To take advantage of the 3 GB available to user-mode programs, the program must be linked with the/LARGEADDRESSAWARE option.

That's a quote from the page you referenced.

[Former Member]

I still use it OK but either way, we're saying the same thing to Tim. 

Unless you condition the address space via one of these mechanisms, the application will only run in 2G of memory within a 4G managed space by the O/S.