Environment PRD froze

[Former Member]

Hi,

Two days ago (08/06/2012), the environment PRD froze. I have checked the OS and there was high CPU spikes. I had to run stop / start in SAP and returned to normal operation. In SAP only found dumps (attachments), but I could not be sure the real reason for this freeze. Any ideas?

Windows Server 2003 En.terprise x64 Sp 2

SQL Server 2008

64GB Ram

SWAp conf       free                  swap sise

107.383.544    32.226.980       41.926.656

ztta/roll_area   = 6500000

ztta/roll_extension   = 2000000000

abap/heap_area_total = 38654705664

abap/heap_area_dia = 2000000000

abap/heap_area_nondia = 0

em/initial_size_MB= 36864

abap/heaplimit = 40000000

tks

TSV_TNEW_PAGE_ALLOC_FAILED1.TXT.zip

SYSTEM_NO_TASK_STORAGE1.txt.zip

DBIF_RSQL_NO_MEMORY1.txt.zip

[paul_power]

Hi Emerson,

It seems you have a 64 bit system which is good but from the short dumps your memory config needs to be addjusted. I see you have heap non-dia set to 0 which is not as per

note 146289.

You can also tune the memory usage to decrease the swap occupation, so
please check and tune the memory parameter following the note:

146289 - Parameter Recommendations for 64-Bit SAP Kernel

especially increase the following parameters

abap/heap_area_dia      2000000000 (2GB)

abap/heap_area_nondia   2000000000 (2GB)

abap/heap_area_total    2000000000 (2GB)

Check SAP Note 

88416 - Zero administration memory management as of 4.0A/Windows

Check what is set for PHYS_MEMSIZE and how much pyhsical RAM is on the machine.

Please, evaluate according to the note if that PHYS_MEMSIZE value is convenient for your
system (this value is the key for the application of note 88416).

   110172 - Windows: Transactions with large memory requirements
   552209 - Maximum memory utilization for processes on NT/Win2000
   313347 - Windows editions and memory support

Hope this helps.

Regards,

Paul

[Former Member]

Hello Paul

I checked note 146289, 88416. The note 88416 states that the  parameter abap / heap_area_nondia is 0 when using the parameter PHYS_MEMSIZE (Current PHYS_MEMSIZE = 36864).

Tks

[Former Member]

Hi,

What else did you observe when the PRD was in frozen state ?

When the CPU was spiking did you check SM66/SM50 what was it looking like ? Also did you check killing some of the spiking processes from SM50 as  a workaround to stop/start SAP to restore normalcy ?

You can as well check SM21 and dev trace files for any errors there.

Regards,

Samik

[nicholas_chang]

Hi,

In addition, you can check the time before PRD freeze on event viewer, dev_W0.old (provided it hasnt overwrite) , DB log and see is there any suspicious behavior.

Cheers,

Nicholas Chang

[Former Member]

Hi Samik,

I had no access to the environment PRD why he was frozen. Only sapmmc showed some programs with Roll in action.

tks

[Former Member]

I checked the file dev_w0.old, but I could not understand the real cause of the problem:

Mon Aug 06 15:15:45 2012

A  ***SUBPOOL*** generating subroutine pool %_T02A9Q for user CALCARFE (226b0302).

A                MainProg=SAPLSBAL_DISPLAY_BASE, Incl=LSBAL_DISPLAY_BASEF01, Line=316

M 

M Mon Aug 06 15:17:33 2012

M 

M  *****************************************************************************

M  *

M  *  LOCATION    SAP-Gateway on host ERPTB706P.erpbtg.com / sapgw00

M  *  ERROR       internal error

M  *

M  *  TIME        Mon Aug 06 15:17:33 2012

M  *  RELEASE     701

M  *  COMPONENT   NI (network interface)

M  *  VERSION     38

M  *  RC          -1

M  *  MODULE      ninti.c

M  *  LINE        742

M  *  DETAIL      NiPGetHostByAddr3: '???' not found

M  *  SYSTEM CALL getnameinfo

M  *  ERRNO       10022

M  *  ERRNO TEXT  WSAEINVAL: Invalid argument

M  *  COUNTER     983632

M  *

M  *****************************************************************************

M 

A  RFC 2322  CONVID 86716184

A   * CMRC=20 DATA=0 STATUS=0 SAPRC=236 ThSAPCMINIT

A  RFC> ABAP Programm: /BDL/SAPLBDL11 (Transaction: )

A  RFC> User: BASIS (Client: 300)

A  RFC> Destination: SM_SMPCLNT100_BACK (handle: 3, , {F8F2DFE1-D475-F10C-BB82-00265533596C})

A  RFC> Called function module: RFC_GET_FUNCTION_INTERFACE_P

A  RFC SERVER> RFC Server Session (handle: 1, 86690075, {EFF2DFE1-5974-F193-BB82-00265533596C})

A  RFC SERVER> Caller host:

A  RFC SERVER> Caller transaction code:  (Caller Program: /BDL/TASK_SCHEDULER)

A  RFC SERVER> Called function module: /BDL/_SESSION_REFRESH_TASK

A  TH VERBOSE LEVEL FULL

A  ** RABAX: end RX_GET_MESSAGE

A 

A Mon Aug 06 15:22:08 2012

A  ***SUBPOOL*** generating subroutine pool %_T02AA3 for user AFONSOFE (24d40001).

A                MainProg=SAPLSBAL_DISPLAY_BASE, Incl=LSBAL_DISPLAY_BASEF01, Line=316

A 

A Mon Aug 06 15:23:17 2012

A  ***SUBPOOL*** generating subroutine pool %_T02AA6 for user FERREIFA (24320202).

A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

A  ***SUBPOOL*** generating subroutine pool %_T02AA7 for user FERREIFA (24320202).

A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

A  ***SUBPOOL*** generating subroutine pool %_T02AA8 for user FERREIFA (24320202).

A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

A  ***SUBPOOL*** generating subroutine pool %_T02AA9 for user FERREIFA (24320202).

A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

A  ***SUBPOOL*** generating subroutine pool %_T02AAA for user FERREIFA (24320202).

A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

A 

A Mon Aug 06 15:23:51 2012

A  ***SUBPOOL*** generating subroutine pool %_T02AAB for user RODRIGHA (20f90101).

A                MainProg=SAPLSBAL_DISPLAY_BASE, Incl=LSBAL_DISPLAY_BASEF01, Line=316

A 

A Mon Aug 06 15:26:36 2012

A  ***SUBPOOL*** generating subroutine pool %_T02AAH for user GRECHIGI (22b50001).

A                MainProg=SAPLSBAL_DISPLAY_BASE, Incl=LSBAL_DISPLAY_BASEF01, Line=316

A 

A Mon Aug 06 15:27:42 2012

A  ***SUBPOOL*** generating subroutine pool %_T02AAI for user CAETANDA (29b20102).

A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

A  ***SUBPOOL*** generating subroutine pool %_T02AAJ for user CAETANDA (29b20102).

A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

A  ***SUBPOOL*** generating subroutine pool %_T02AAK for user CAETANDA (29b20102).

A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

A  ***SUBPOOL*** generating subroutine pool %_T02AAL for user CAETANDA (29b20102).

A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

A  ***SUBPOOL*** generating subroutine pool %_T02AAM for user CAETANDA (29b20102).

A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

C 

C Mon Aug 06 15:33:56 2012

C  Thread ID:14396

C  Thank You for using the SLODBC-interface

C  Using dynamic link library 'F:\usr\sap\PRD\DVEBMGS00\exe\dbmssslib.dll'

C  dbmssslib.dll patch info

C    SAP patchlevel  0

C    SAP patchno  137

C    Last MSSQL DBSL patchlevel 0

C    Last MSSQL DBSL patchno         134

C    Last MSSQL DBSL patchcomment UPSERT retry corrected (1566676)

C  lpc:(local) connection used on ERPTB706P

C  lpc:(local) connection used on ERPTB706P

C  Driver: sqlncli10.dll Driver release: 10.00.2531

C  GetDbRelease: 10.00.2531

C  GetDbRelease: Got DB release numbers (10,0,2531)

B  Connection 4 opened (DBSL handle 2)

A 

A Mon Aug 06 15:34:32 2012

A  TH VERBOSE LEVEL FULL

A  ** RABAX: level LEV_RX_PXA_RELEASE_MTX entered.

A  ** RABAX: level LEV_RX_PXA_RELEASE_MTX completed.

A  ** RABAX: level LEV_RX_COVERAGE_ANALYSER entered.

A  ** RABAX: level LEV_RX_COVERAGE_ANALYSER completed.

A  ** RABAX: level LEV_RX_SAVE_SHMLOCKS entered.

A  ** RABAX: level LEV_RX_SAVE_SHMLOCKS completed.

A  ** RABAX: level LEV_RX_RESET_SHMLOCKS entered.

A  ** RABAX: level LEV_RX_RESET_SHMLOCKS completed.

A  ** RABAX: level LEV_RX_ROLLBACK entered.

A  ** RABAX: level LEV_RX_ROLLBACK completed.

A  ** RABAX: level LEV_RX_DB_ALIVE entered.

A  ** RABAX: level LEV_RX_DB_ALIVE completed.

A  ** RABAX: level LEV_RX_HOOKS entered.

A  ** RABAX: level LEV_RX_HOOKS completed.

A  ** RABAX: level LEV_RX_STANDARD entered.

A 

A Mon Aug 06 15:34:33 2012

A  ** RABAX: level LEV_RX_STANDARD completed.

A  ** RABAX: level LEV_RX_STOR_VALUES entered.

A  ** RABAX: level LEV_RX_STOR_VALUES completed.

A  ** RABAX: level LEV_RX_C_STACK entered.

A  ** RABAX: level LEV_RX_C_STACK completed.

A  ** RABAX: level LEV_RX_MEMO_CHECK entered.

A  ** RABAX: level LEV_RX_MEMO_CHECK completed.

A  ** RABAX: level LEV_RX_AFTER_MEMO_CHECK entered.

A  ** RABAX: level LEV_RX_AFTER_MEMO_CHECK completed.

A  ** RABAX: level LEV_RX_INTERFACES entered.

A  ** RABAX: level LEV_RX_INTERFACES completed.

A  ** RABAX: level LEV_RX_GET_MESS entered.

A  ** RABAX: level LEV_RX_GET_MESS completed.

A  ** RABAX: level LEV_RX_INIT_SNAP entered.

A  ** RABAX: level LEV_RX_INIT_SNAP completed.

A  ** RABAX: level LEV_RX_WRITE_SYSLOG entered.

A  ** RABAX: level LEV_RX_WRITE_SYSLOG completed.

A  ** RABAX: level LEV_RX_WRITE_SNAP_BEG entered.

A  ** RABAX: level LEV_RX_WRITE_SNAP_BEG completed.

A  ** RABAX: level LEV_RX_WRITE_SNAP entered.

A 

A Mon Aug 06 15:34:38 2012

A  ** RABAX: level LEV_SN_END completed.

A  ** RABAX: level LEV_RX_WRITE_SNAP_END entered.

A  ** RABAX: level LEV_RX_WRITE_SNAP_END completed.

A  ** RABAX: level LEV_RX_SET_ALERT entered.

A  ** RABAX: level LEV_RX_SET_ALERT completed.

A  ** RABAX: level LEV_RX_COMMIT entered.

A  ** RABAX: level LEV_RX_COMMIT completed.

A  ** RABAX: level LEV_RX_SNAP_SYSLOG entered.

A  ** RABAX: level LEV_RX_SNAP_SYSLOG completed.

A  ** RABAX: level LEV_RX_RESET_PROGS entered.

A  ** RABAX: level LEV_RX_RESET_PROGS completed.

A  ** RABAX: level LEV_RX_STDERR entered.

A  Mon Aug 06 15:34:38 2012

A 

A  ABAP Program SAPLKMS0                                .

A  Source LKMS0F03                                 Line 480.

A  Error Code SYSTEM_NO_TASK_STORAGE.

A  Module  $Id: //bas/701_REL/src/krn/runt/abtsv2.c#4 $ SAP.

A  Function ArraySort Line 1318.

A  ** RABAX: level LEV_RX_STDERR completed.

A  ** RABAX: level LEV_RX_RFC_ERROR entered.

A  ** RABAX: level LEV_RX_RFC_ERROR completed.

A  ** RABAX: level LEV_RX_RFC_CLOSE entered.

A  ** RABAX: level LEV_RX_RFC_CLOSE completed.

A  ** RABAX: level LEV_RX_IMC_ERROR entered.

A  ** RABAX: level LEV_RX_IMC_ERROR completed.

A  ** RABAX: level LEV_RX_DATASET_CLOSE entered.

A  ** RABAX: level LEV_RX_DATASET_CLOSE completed.

A  ** RABAX: level LEV_RX_ERROR_SAVE entered.

A  ** RABAX: level LEV_RX_ERROR_SAVE completed.

A  ** RABAX: level LEV_RX_ERROR_TPDA entered.

A  ** RABAX: level LEV_RX_ERROR_TPDA completed.

A  ** RABAX: level LEV_RX_PXA_RELEASE_RUDI entered.

A  ** RABAX: level LEV_RX_PXA_RELEASE_RUDI completed.

A  ** RABAX: level LEV_RX_LIVE_CACHE_CLEANUP entered.

A  ** RABAX: level LEV_RX_LIVE_CACHE_CLEANUP completed.

A  ** RABAX: level LEV_RX_END entered.

A  ** RABAX: level LEV_RX_END completed.

A  ** RABAX: end no http/smtp

A  ** RABAX: end RX_GOTO_RABAX

A  Unable to fulfil request for 7883584 bytes of storage space..

A 

A 

A Mon Aug 06 15:36:16 2012

A  ***SUBPOOL*** generating subroutine pool %_T02AAY for user RODRIGHA (20f90202).

A                MainProg=SAPLSBAL_DISPLAY_BASE, Incl=LSBAL_DISPLAY_BASEF01, Line=316

M 

M Mon Aug 06 15:52:41 2012

M  in_ThErrHandle: 1

M  ThIErrHandle: new stat of W0 is WP_SHUTDOWN   

M  ThIErrHandle: I'm during shutdown

M  PfStatDisconnect: disconnect statistics

M  Entering ThSetStatError

C  SlBreak

B  db_sqlbreak() = 15

M  *** WARNING => sql break failed (16388)

M  ThIErrHandle: don't try rollback again

M  ThShutDownServer: shutdown server

M  PfWriteIntoFile: shared buffer is empty.

M  ThIErrHandle: do not call ThrCoreInfo (no_core_info=0, in_dynp_env=0)

M  Entering ThReadDetachMode

M  call ThrShutDown (1)...

B  Disconnecting from ALL connections:

B  Wp  Hdl ConName                        ConId     ConState     TX  HC  PRM RCT FRC TIM MAX OPT Date     Time   DBHost                       

B  000 000 R/3                            000000000 INACTIVE     NO  NO  YES NO  NO  000 255 255 20120804 171703 ERPTB706P                    

B  000 001 ++DBO++0020                    000084952 INACTIVE     NO  NO  NO  NO  NO  004 255 255 20120804 172237 ERPTB706P                    

B  000 002 R/3*INACT_PACK                 000084907 DISCONNECTED NO  NO  NO  NO  NO  000 255 255 20120806 141743 ERPTB706P                    

B  000 003 ++DBO++0020                    000084562 DISCONNECTED NO  NO  NO  NO  NO  000 255 255 20120806 044242 ERPTB706P                    

B  000 004 R/3*EXPRESSPOPUP               000084954 INACTIVE     NO  NO  NO  NO  NO  004 255 255 20120806 153356 ERPTB706P                    

B  Disconnected from connection 0

B  Disconnected from connection 1

B 

B Mon Aug 06 15:52:42 2012

B  Disconnected from connection 4

B  statistics db_con_commit (com_total=5918337, com_tx=46876)

B  statistics db_con_rollback (roll_total=8049, roll_tx=3)

M  ***LOG Q02=> wp_halt, WPStop (Workproc 0 14636) [dpnttool.c   334]