SAP HANA not starting - statisticsserver Out of memory occured

[Former Member]

Hi,

I cannot start my SAP HANA database:

HDB version info:

  version:            1.00.090.00.1416514886

___________________________________________________________________________________________

:~> HDB start

StartService

Impromptu CCC initialization by 'rscpCInit'.

  See SAP note 1266393.

OK

OK

Starting instance using: /usr/sap/HDB/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function StartWait 2700 2

06.06.2015 12:31:54

Start

OK

06.06.2015 12:32:24

StartWait

FAIL: process hdbstatisticsserver HDB Statisticsserver not running

_____________________________________________________________________________________________

When I start the statisticsserver, I get this error:

:~> hdbstatisticsserver

service startup...

accepting requests at 127.0.0.1:30005; 127.0.0.2:30005

assigning to volume 2 ...

run as transaction slave

Out of memory occured. Writing runtime dump!

Finished writing oom runtime dump!

assign failed with persistence startup error. exception  1: no.1000002  (ltt/impl/memory.cpp:134)

________________________________________________________________________________________________

The only thing I think that changed on this box is that Netbackup was installed

trace files - rename to .zipx .txt.zip

PUTTY SESSION.txt.zip

[Former Member · Accepted solution]

Probably you temporarily have to set

statisticsserver.ini -> [memorymanager] -> allocationlimit

to a higher value (e.g. "20%"). In general please have a look at SAP Note 2147247 ("How can the memory requirements of the statistics server be minimized?") in order to minimize the memory requirements of the SAP HANA statistics server.

[Former Member · Accepted solution]

Hi,

The allocationlimit of statisticsserver.ini was still at 5% default.  I increased it to 20%, then to 50%, but still exacly the same issue on startup.

I cannot execute the "Memory requirements Minimized" steps while the service is down.

May I try to change to Embedded statistics server while the statisticsserver service is unable to start?

• nameserver.ini -> [statisticsserver]->active=true (steps in notes 1917938,2092033)

[Former Member · Accepted solution]

If the statistics server doesn't start, you won't be able to migrate to ESS. The original OOM dump (with 5% allocation limit, so 6 GB) showed that

Pool/PersistenceManager/LogRecovery

is mainly responsible for the OOM. So it looks like you had a crash before?

Can you provide the new OOM dump with 50% allocation limit? You might have to set oom_dump_time_delta as described in SAP Note 1999997 in order to make sure that a new OOM dump is triggered after less than 24 hours.

[Former Member · Accepted solution]

Hi,

I'm trying to bring the system up from a crash, so there was a crash before.

hdbstatisticsserver started again, please see trace location and dump attached

😆 hdbstatisticsserver

service startup...

accepting requests at 127.0.0.1:30005; 127.0.0.2:30005

assigning to volume 2 ...

run as transaction slave

Out of memory occured. Writing runtime dump!

Finished writing oom runtime dump!

assign failed with persistence startup error. exception  1: no.1000002  (ltt/impl/memory.cpp:134)

    Allocation failed ; $size$=4294967296; $name$=LogRecovery; $type$=pool; $inuse_count$=2; $allocated_size$=544; $alignment$=4096

Thanks

trace location (2) - rename to .zipx.txt.zip

[Former Member · Accepted solution]

That's not a typical OOM - it points to the problem described in SAP Note 2133638. Can you check if the solution described there works?

In any case I would strongly recommend to avoid using SAP HANA Revision 90 to 94 due to several potentially critical issues.

[Former Member · Accepted solution]

One more thing: What is the exact result of the following query?

SELECT * FROM M_INIFILE_CONTENTS WHERE KEY = 'allocationlimit'

[Former Member · Accepted solution]

Hi,

Output is the following:

SELECT * FROM M_INIFILE_CONTENTS WHERE KEY = 'allocationlimit'

FILE_NAME,LAYER_NAME,TENANT_NAME,HOST,SECTION,KEY,VALUE

"statisticsserver.ini","DEFAULT","","","memorymanager","allocationlimit","5%"

"statisticsserver.ini","SYSTEM","","","memorymanager","allocationlimit","50"

"global.ini","DEFAULT","","","memorymanager","allocationlimit",""

I tried to change the:

global.ini [memorymanager] allocationlimit to 117760 (value from oom dump)

but get a error:

[129]: transaction rolled back by an internal error: exception 1000002: Allocation failed; $size$=1048567; $name$=QueryCompilation; $type$=pool; $inuse_count$=2; $allocated_size$=352, $allignment$=16

Should I also include a <global_allocation_limit> in the indexserver.ini file?  What value, from oom dump is 32650 the correct value?

Thanks

[Former Member · Accepted solution]

You have set the allocationlimit to "50" which corresponds to 50 MB. As instructed please set it to a percentage - e.g. "20%". Three characters, '2', '0' and '%', without any blank in between.

[Former Member · Accepted solution]

The originally critical allocator Pool/PersistenceManager/LogRecovery is used to store 4 log segments in memory during recovery. A size of 4 GB indicates that you have increased the default statistics server log segment size of 64 MB to 1 GB. Is this correct? What is the output of the following command?

SELECT * FROM M_INIFILE_CONTENTS WHERE KEY = 'log_segment_size_mb'

[Former Member · Accepted solution]

Hi,

Apologies, I tried to fix the allocationlimit to "20%", and noticed hdbstudio had disconnected from the database [same 129 as above].  When I tried reconnecting it asked for the <sid>adm username and password, so I bounced HANA.

Now I cannot start HANA and is getting this error:

😆 HDB start

StartService

Impromptu CCC initialization by 'rscpCInit'.

  See SAP note 1266393.

OK

OK

Starting instance using: /usr/sap/HDB/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function StartWait 2700 2

08.06.2015 16:48:24

Start

OK

08.06.2015 16:48:34

StartWait

FAIL: process hdbdaemon HDB Daemon not running

😆 hdbdaemon

Environment variable IQDIR16 must be set before using HDB configured with DT

Environment variable STREAMING_HOME  must be set before using HDB configured with Streaming Server

Yes, I did change the log_segment_size_mb to 64 mb previously - tried to get the log backups to run only every 30 min.

I notice error in oom:

[MEMORY_OOM]  Information about current out of memory situation: (2015-06-08 16:41:29 169 Local)

OUT OF MEMORY occurred.

Host: S4ZAJ1SYLX16

Executable: hdbnameserver

PID: 14119

Failed to allocate 33024 byte.

3rd trace.zipx .txt.zip

[Former Member · Accepted solution]

Please make sure that you revert the changes that you implemented for SAP Note 2133638. If the problem still exists, you can check if rebooting the complete host helps.

[Former Member · Accepted solution]

Hi,

I reverted the changes from SAP Note 2133638, and also the allocationlimit to "50%" in the INI files, and the system started  :):):)

I will be looking to upgrade to revision 96 as soos as possible.

Thanks a mill!!!

[Mofizur]

Hi,

I believe applying to latest revision(96) can help you.

Thanks,

Mofizur