Error: could not create async IO thread [8](-903)

[Former Member]

Hello,

we're experiencing problems with the MaxDB backup mechanism(Log and Full-Backup).

It started last week, when we discovered a full log volume,

because the autoLog was deactivated by a failed log backup.

Since then our automatic full_backup script,

wich runs at 00:25 each night, is not able to do a full backup anymore.

The error we found in the backup history was:

Error: could not create async IO thread [8](-903)

In the diagnosis files the error looks like:

2009-02-27 00:25:00 0xE88 ERR 18008 TASKING Could not create thread: 'ASYNCi', rc = 8

2009-02-27 00:25:00 0x584 ERR 52011 SAVE devspace: surprising blk cnt:

2009-02-27 00:25:00 0x584 ERR 52012 SAVE error occured, basis_err 3700

2009-02-27 08:58:39 0xE88 ERR 18008 TASKING Could not create thread: 'ASYNCi', rc = 8

2009-02-27 08:58:39 0x930 ERR 52039 AUTOSAVE terminated, because of: 3700

(at 00:25 it's a failed full_backup, at 8:58 a failed autolog backup)

If the autolog was turned off because of this error,

it was not possible to do a manual log-backup (stated the same error as in the logs)

We could "solve" this by restarting the X-server.

After this a manual log-backup was possible and we could turn on

autolog again.

As today, the database was not accessible at all and we had to reboot

the database server. The kerneldumps of these crashes are in the attachment.

Additionally, you find some log files there.

Our environment:

Windows Server 2003 R2 - SP2

4GB Ram

MaxDB 7.5.00.44

Any help to find out, what's going wrong here and to have a stable backup again would be nice.

Dominic

.... how can I upload attachments btw?

Edited by: Transporeon on Mar 2, 2009 4:36 PM

[lbreddemann]

Hi there!

> The error we found in the backup history was:

> Error: could not create async IO thread [8](-903)

>

> In the diagnosis files the error looks like:

> 2009-02-27 00:25:00 0xE88 ERR 18008 TASKING Could not create thread: 'ASYNCi', rc = 8

>

> We could "solve" this by restarting the X-server.

> After this a manual log-backup was possible and we could turn on

> autolog again.

Basically what you're facing is - like Markus correctly assumed - a shortage of Windows system resources.

Check the returncode meaning with the xsysrc tool which is delivered with every MaxDB installation on Windows:

commandline> xsysrc 8

Errortext for errorcode 8:
'Not enough storage is available to process this command.'

This means: Windows cannot process the system call (create a new thread in this case), because of a shortage of internal resources.

Since the resource management of Windows is not really open that much to outside (as in 'outside of the windows kernel development') analysis, very often you cannot do much about this, but reboot.

Since MaxDB is dynamically creating the threads when they are needed for backup (and removing them afterwards), the problem itself may not be solved completely.

Anyhow, there are several steps you can make to make it less likely to happen:

1. Use the /3GB switch for Windows

2. Use the "USE_FIBERS" parameter to have MaxDB use lightweight threads (aka FIBERS).

3. Use the "PREALLOCATE_IOWORKER" parameter to disable the dynamic creation of threads.

MaxDB will then instead create all thread at instance startup - so backups won't fail anymore due to missing resources for new threads.

and as usual: use the current version of MaxDB instead that old 7.5 Build 44.

At least install the current 7.5 patch.

> Our environment:

> Windows Server 2003 R2 - SP2

> 4GB Ram

> MaxDB 7.5.00.44

> .... how can I upload attachments btw?

There is no upload facility in the forums. You still may provide a link to your webspace if you want to provide logfiles.

regards,

Lars

[markus_doehr2]

> 2009-02-27 08:58:39 0x930 ERR 52039 AUTOSAVE terminated, because of: 3700

This sounds to me like you were reaching operating system imposed limits...

Is this a 32bit system? If yes, how big is your CACHE_SIZE?

Markus

[Former Member]

Yes, it's a 32-Bit system.

Parameter CACHE_SIZE is set to 200000

[markus_doehr2]

This is pretty high for a 32bit system. Not more than 400 MB will need to be allocated and you will run into the problem you see. There are other relevant parameters as well (MAXUSERS etc.) which will allocate memory. You may have reached the magic 2 GB border.

Markus

[Former Member]

thanks a lot for the quick reply...

we check, if the reducing of the cache_size

solves our problems...

btw which unit has the cache_size parameter?

so is 200000 = 2 GB ?

[lbreddemann]

> btw which unit has the cache_size parameter?

> so is 200000 = 2 GB ?

Check the documentation...

It's "pages", that is 8KB pages.

So CACHE_SIZE = 200000 means 200000*8 = 1600000 KB /1024 = 1562,5 MB

regards,

Lars

[federico_rossi]

Hi, we are experiencing the same problem,

I mean sometimes we get the error "0xE88 ERR 18008 TASKING Could not create thread: 'ASYNCi', rc = 8" while offline backup is running.

Could you please tell me the relationship btw parameter CACHE_SIZE and the above problem (# threads ??)?

I understood that Transporeon was able to resolve the issue using a lower value of CACHE_SIZE.

We are running MAXDB 7.6 on Win32 machine with 4GB of ram, CACHE_SIZE = 300000.

Thanks in advance

[markus_doehr2]

> I mean sometimes we get the error "0xE88 ERR 18008 TASKING Could not create thread: 'ASYNCi', rc = 8" while offline backup is running.

What do you mean with "offline backup"?

> Could you please tell me the relationship btw parameter CACHE_SIZE and the above problem (# threads ??)?

the problem is, that a too big cache size exceeds the max. process memory of 2 GB. If all buffers together (CACHE_SIZE plus other buffers) are bigger than 2 GB the kernel can't allocate more memory for thread local storage and hence you get this error.

> I understood that Transporeon was able to resolve the issue using a lower value of CACHE_SIZE.

> We are running MAXDB 7.6 on Win32 machine with 4GB of ram, CACHE_SIZE = 300000.

300,000 x 8 kb = 2,400,000 - so you are already at the limit.

Markus

[federico_rossi]

hello Markus

regarding the backup I did a mistake, sorry... our backup strategy is the following: weekly full backup and daily incremental backup;

as soon as a backup fails (due to the above error) then all the following ones fail as well, the only option we have to solve the issue is to reboot the whole node

concerning the memory, we have 4GB ram (greather than 2,3), and 10GB of swap space;

in this situation my guess is that we have 4+10=14GB of available memory space to allocate threads, and cache_size plus other buffers dont reach that limit in our configuration;

besides, in our boot.ini file we have /3GB and /PAE options enabled

do you have any further suggestion?

thanks

[markus_doehr2]

> concerning the memory, we have 4GB ram (greather than 2,3), and 10GB of swap space;

Yes - but the amount of memory you have is not relevant. 32bit operating systems have the limitation of only being able to allocate 2^{32 bits of memory in one process - which is roughly 2 GB (that's why they are call '32bit'). MaxDB is a single process database with lots of threads inside so the max. amount of memory you can allocate is that - 2}32. There's nothing to circumvent or workaround that issue, it's the CPU and the operating system limiting, not the database.

> in this situation my guess is that we have 4+10=14GB of available memory space to allocate threads, and cache_size plus other buffers dont reach that limit in our configuration;

> besides, in our boot.ini file we have /3GB and /PAE options enabled

AWE/PAE is not supported for MaxDB. It was for older versions (7.5) for the Livecache heap but support stopped with 7.6.

Markus

[federico_rossi]

Thanks again Markus

your comments are really appreciated and useful.

I understood that we cannot do anything in order to make a better use of our OS and hardware..

In the end, what would you do in your situation?

[markus_doehr2]

> I understood that we cannot do anything in order to make a better use of our OS and hardware..

> In the end, what would you do in your situation?

As a short term solution I'd decrease CACHE_SIZE to 260,000 and see if you still get the problem.

You may also check if your CPU is maybe already 64bit enabled (almost all servers produced in the last two years are).

Markus

[federico_rossi]

We'll follow your suggestion and we'll decrease CACHE_SIZE value to 260000.

We'll let you know as soon as possible about the result.

Regarding processors, we have 4 x 2,4GHz AMD Dual-Core Opteron 280, and they should be 64bit enabled. Does it mean that we have additional options in addressing memory?

Edited by: Federico Rossi on Sep 16, 2009 11:53 AM

[markus_doehr2]

> We'll follow your suggestion and we'll decrease CACHE_SIZE value to 260000.

> We'll let you know as soon as possible about the result.

OK

> Regarding processors, we have 4 x 2,4GHz AMD Dual-Core Opteron 280, and they should be 64bit enabled. Does it mean that we have additional options in addressing memory?

Yes - this CPU is 64bit capable!

Install Windows x86_64 (instead of Windows 32bit) and use the 64bit MaxDB. Then your memory adressing problems will be gone.

Makrus