Internal Kernel Error in EHP5 "SYSTEM_CORE_DUMPED"

[Former Member]

Hi experts,

duing background job run run time error accure as below.

Category               Internal Kernel Error

Runtime Errors         SYSTEM_CORE_DUMPED

is will accure on sap performane monitor job also.

ECC 6 EHP5 oracle 10g on RHEL5

i know this error can be avoide after kernal upgrad. so please give me why this accure or how to reslove?

thanks in advance.

Please do the needful.

Regards,

vivek

[Former Member]

Vivek,

Can you provide the work process trace file that this dump is occurring in?  Also, are you using a non-ext kernel release due to the fact that you're not running Oracle 11?

Regards,

Josh

[Former Member]

we are getting below error .

    PXA

PXA INITIALIZATION

PXA: max_map_count = 1000000: ok

System page size: 4kb, total admin_size: 98128kb, dir_size: 32104kb.

Attached to PXA (address 2af3cf5d0000, size 1176000K, 3 fragments of 359292K )

PXA allocated (address 2af3cf5d0000, size 1176000K)

*** WARNING: INTERNAL_KERNEL_VERSION of message server

              unknown yet.

abap/pxa = shared protect gen_remote

PXA: checking structure sizes: 752 264 8

Regards,

[Former Member]

Vivek,

Can you post the entire log file?

Regards,

Josh

[Former Member]

LOG**************

---------------------------------------------------

trc file: "dev_w12", trc level: 1, release: "720"

---------------------------------------------------

*

*  ACTIVE TRACE LEVEL           1

*  ACTIVE TRACE COMPONENTS      all, MJ

*

M sysno      00

M sid        DEV

M systemid   390 (AMD/Intel x86_64 with Linux)

M relno      7200

M patchlevel 0

M patchno    105

M intno      20020600

M make       single threaded, Unicode, 64 bit, optimized

M profile    /usr/sap/DEV/SYS/profile/DEV_DVEBMGS00_prdbasis

M pid        22021

M

M

M Sun Oct  7 15:35:42 2012

M  kernel runs with dp version 131000(ext=118000) (@(#) DPLIB-INT-VERSION-131000-UC)

M  length of sys_adm_ext is 588 bytes

M  ThStart: taskhandler started

M  ThStart: prv_action of w12 = 0x4

M  ThInit: initializing BTC work process w12

M  ThInit: running on host prdbasis

M  calling db_connect ...

B  Loading DB library '/usr/sap/DEV/DVEBMGS00/exe/dboraslib.so' ...

B

B Sun Oct  7 15:35:43 2012

B  Library '/usr/sap/DEV/DVEBMGS00/exe/dboraslib.so' loaded

B  Version of '/usr/sap/DEV/DVEBMGS00/exe/dboraslib.so' is "720.00", patchlevel (0.102)

C  Register application info.

C  Oracle client version: 10.2.0.4.0, V5, default build, (dbsl 720 260811, UNICODE[2])

C  Installed Oracle client patches:

C  # Patch  9254968      : Created on 15 Jan 2010, 09:30:31 hrs PST8PDT

C  # Patch  8732366      : Created on 27 Jul 2009, 15:49:18 hrs CEST

C  # Patch  6944036      : Created on  3 Jun 2008, 23:35:37 hrs PST8PDT

C  # Patch  6826661      : Created on 20 May 2008, 00:23:58 hrs PST8PDT

C  # Patch  6775231      : Created on  2 Dec 2008, 21:45:40 hrs PST8PDT

C  # Patch  6670871      : Created on 13 Sep 2010, 23:47:52 hrs PST8PDT

C  Client NLS setting (by OCINlsGetInfo(con=0)): 'AMERICAN_AMERICA.UTF16'

C  Logon as OPS$-user to get SAPSR3's password

C  Connecting as /@DEV on connection 0 (nls 0) ... (dbsl 720 260811, UNICODE[2])

C  Attaching to database server DEV (con=0, svchp=1150e588, srvhp=11511018)

C  Starting user session: OCISessionBegin(con=0, usr='/', svc=1150e588, srv=11511018, usr=1156b210)

C  Now '/@DEV' is connected: con=0, nls=0, session_id=2645.

C  Got SAPSR3's password from OPS$-user

C  Disconnecting from connection 0 ...

C  Closing user session (con=0, svc=1150e588, usr=1156b210)

C  Disconnected (con=0) from ORACLE.

C  Connecting as SAPSR3/<pwd>@DEV on connection 0 (nls 0) ... (dbsl 720 260811, UNICODE[2])

C  Starting user session: OCISessionBegin(con=0, usr='SAPSR3', svc=1150e588, srv=11511018, usr=1156b210)

C  Now 'SAPSR3/<pwd>@DEV' is connected: con=0, nls=0, session_id=2645.

C  con=0, V$NLS_PARAMETERS: NLS_LANG=AMERICAN_AMERICA.UTF8, NLS_NCHAR=UTF8

C  Nls CharacterSet                      NationalCharSet                   EnvHp            ErrHp            ErrBt

C    0 UTF16                             AL16UTF16                         114f6400         114fdf00         1150e658

C  DB instance DEV002 is running on eul1506 with ORACLE version 11.2.0.2.0 since OCT 07, 2012, 06:08:44.

B  Connection 0 opened (DBSL handle 0)

M  ThInit: db_connect o.k.

M  ICT: exclude compression: *.zip,*.rar,*.arj,*.z,*.gz,*.tar,*.lzh,*.cab,*.hqx,*.ace,*.jar,*.ear,*.war,*.css,*.pdf,*.gzip,*.uue,*.b

I  MtxInit: 44 0 0

M  SHM_PRES_BUF   (addr: 2af17c0c6000, size: 34000000)

M  SHM_ROLL_AREA  (addr: 2af3ac83e000, size: 268435456)

M  SHM_PAGING_AREA  (addr: 2af3bc83e000, size: 268435456)

M  SHM_ROLL_ADM   (addr: 2af17e134000, size: 11142696)

M  SHM_PAGING_ADM  (addr: 2af3cc83e000, size: 3277856)

M  ThCreateNoBuffer  allocated 564152 bytes for 1000 entries at 2af3ccb61000

M  ThCreateNoBuffer  index size: 3000 elems

M  ThCreateVBAdm  allocated 30976 bytes (50 server) at 2af3d5c55000

X  EmInit: MmSetImplementation( 2 ).

X  MM global diagnostic options set: 0

X  <ES> client 44 initializing ....

X  Using implementation std

X  Linux: Kernel supports shared memory disclaiming

X  Linux: using madvise(<pointer>, <size>, 9).

X  Linux: disclaiming for shared memory enabled

X  ES initialized.

X  mm.dump: set maximum dump mem to 96 MB

I  *** INFO  SHM 45 in POOL 40     6131 KB estimated      4438 KB real (   -1693 KB    -28 %)

B  db_con_shm_ini:  WP_ID = 44, WP_CNT = 61, CON_ID = -1

B  dbtbxbuf: Buffer TABL  (addr: 2af3d8737100, size: 672000000, end: 2af400815900)

B  dbtbxbuf: Buffer TABLP (addr: 2af400816100, size: 122880000, end: 2af407d46100)

B  dbsync[db_syinit]: successfully attached to shared memory, sync_adm_p = 2af3d1021000

B  dbsync[db_syinit]: Buffer synchronisation started with

B    sync_concept      = SEQ_NR

B    sendon            = 1

B    bufreftime        = 120

B    del_ddlog         = 1

B    last_counter      = -2146479929

B    time_of_last_sync = 20121007153421

B    MySysId           = 'prdbasis             01'

B  dbexpbuf: Buffer EIBUF (addr: 2af407d46108, size: 249692160, end: 2af416b66108)

B  dbexpbuf: Buffer ESM   (addr: 2af416b67108, size: 4194304, end: 2af416f67108)

B  dbexpbuf: Buffer CUA   (addr: 2af3d1023108, size: 20480000, end: 2af3d23ab108)

B  dbexpbuf: Buffer OTR   (addr: 2af416f68108, size: 4194304, end: 2af417368108)

I  MPI: dynamic quotas disabled.

I  MPI init: pipes=4000 buffers=1279 reserved=383 quota=10%

M  rdisp/thsend_mode : -1 -> 0

M  rdisp/setactfields_late : -1 -> 0

M  CCMS uses Shared Memory Key 73 for monitoring.

S  *** init spool environment

S  TSPEVJOB updates outside critical section: event_update_nocsec = 1

S  initialize debug system

T  Stack direction is downwards.

T  debug control: prepare exclude for printer trace

T  new memory block 115e9410

S  spool kernel/ddic check: Ok

S  using table TSP02FX for frontend printing

S  0 spool work process(es) found

S  frontend print via spool service enabled

S  printer list size is 150

S  printer type list size is 50

S  queue size (profile)   = 300

S  hostspool list size = 3000

S  option list size is 30

I  *** INFO  SHM 49 in POOL 40     2898 KB estimated      1632 KB real (   -1266 KB    -44 %)

S      found processing queue enabled

S  found spool memory service RSPO-RCLOCKS at 2af3d23ad070

S  doing lock recovery

S  setting server cache root

S  found spool memory service RSPO-SERVERCACHE at 2af3d23ae5c0

S    using messages for server info

S  size of spec char cache entry: 297032 bytes (timeout 100 sec)

S  size of open spool request entry: 2512 bytes

S  immediate print option for implicitely closed spool requests is disabled

A  ***GENER* Trace switched on ***

A

A  ---PXA-------------------------------------------

A  PXA INITIALIZATION

A  PXA: max_map_count = 1000000: ok

A  System page size: 4kb, total admin_size: 98128kb, dir_size: 32104kb.

A  Attached to PXA (address 2af35ec7c000, size 1176000K, 3 fragments of 359292K )

A  PXA allocated (address 2af35ec7c000, size 1176000K)

A  *** WARNING: INTERNAL_KERNEL_VERSION of message server

A               unknown yet.

A  abap/pxa = shared protect gen_remote

A  PXA: checking structure sizes: 752|264|8

A  PXA INITIALIZATION FINISHED

A  ---PXA-------------------------------------------

A

A  ATRA: pfclock execution time = 0

A  ABAP ShmAdm attached (addr=2af19666a000 leng=20955136 end=2af197a66000)

A  >> Shm MMADM area (addr=2af196b6d0b0 leng=241472 end=2af196ba7ff0)

A  >> Shm MMDAT area (addr=2af196ba9000 leng=15454208 end=2af197a66000)

A  RFC rfc/signon_error_log = -1

A  RFC rfc/dump_connection_info = 0

A  RFC rfc/dump_client_info = 0

A  RFC rfc/cp_convert/ignore_error = 1

A  RFC rfc/cp_convert/conversion_char = 23

A  RFC rfc/wan_compress/threshold = 251

A  RFC rfc/recorder_pcs not set, use defaule value: 2

A  RFC rfc/delta_trc_level not set, use default value: 0

A  RFC rfc/no_uuid_check not set, use default value: 0

H  HTTP> Parameter icf/ssocookie_mandatory set to 0

B  dbtran INFO (init_connection '<DEFAULT>' [ORACLE:720.00]):

B   max_blocking_factor       =   5,  min_blocking_factor         =   5,

B   max_in_blocking_factor    =   5,  min_in_blocking_factor      =   5,

B   max_union_blocking_factor =   5,  min_union_blocking_factor   =   5,

B   prefer_union_all          =   0,  prefer_join                 =   0,

B   prefer_fix_blocking       =   0,  prefer_in_itab_opt          =   1,

B   convert AVG               =   0,  alias table FUPD            =   0,

B   escape_as_literal         =   1,

B   select *                  =0x0f,  character encoding          = STD / <none>:-,

B   use_hints                 = abap->1, dbif->0x1, upto->4294967295

M  ThrCreateShObjects  allocated 126790 bytes at 2af3a68ec000

N  SsfSapSecin: getenv(SECUDIR)=="/usr/sap/DEV/DVEBMGS00/sec"

N

N  =================================================

N  === SSF INITIALIZATION:

N  ===...SSF Security Toolkit name SAPSECULIB .

N  ===...SSF library is /usr/sap/DEV/DVEBMGS00/exe/libsapcrypto.so .

N  ===...SSF default hash algorithm is SHA1 .

N  ===...SSF default symmetric encryption algorithm is DES-CBC .

N  ===...SECUDIR="/usr/sap/DEV/DVEBMGS00/sec"

N  ===...loading of Security Toolkit successfully completed.

N  ===   SAPCRYPTOLIB  5.5.5C pl32  (Apr  2 2011) MT-safe

N  =================================================

N

N Sun Oct  7 15:35:44 2012

N  MskiInitLogonTicketCacheHandle: Logon Ticket cache pointer retrieved from shared memory.

N  MskiInitLogonTicketCacheHandle: Workprocess runs with Logon Ticket cache.

M  JrfcVmcRegisterNativesDriver o.k.

W  =================================================

W  === ipl_Init() called

W    ipl_Init(): wp task type is 4:BTC, skip initialization

W  === ipl_Init() returns 0, ITSPE_OK: OK

W  =================================================

N  SignInit: successfully obtained handle for Security Context cache

N  VSI: WP init in ABAP VM completed with rc=0

E  Enqueue Info: enque/use_pfclock2 = FALSE

E  Enqueue Info: row condense enabled

E  EnqId_SuppressIpc: local EnqId initialization o.k.

E  EnqCcInitialize: local enqueue client init o.k.

A

A Sun Oct  7 15:35:57 2012

A  UpdateProfile (new settings): off,krn_impl,(no dyn check),(no logging),(no stat err),(local generation),(no commit)

A  UpdateProfile (new exceptions): (tmp_err),(home_err),(local_err)

A  UpdateProfile (new version): 0

A  ***GENER* Trace switched off ***

M  SecAudit(rsauinit): WP attached to existing shared memory.

M  SecAudit(RsauShmInit): addr of SHM for Audit.. = 2af417be0000

M  SecAudit(RsauShmInit): addr of RSAUSHM........ = 2af417be1000

M  SecAudit(RsauShmInit): addr of RSAUSLOTINFO... = 2af417be1610

M  SecAudit(RsauShmInit): addr of RSAUSLOTS...... = 2af417be1620

M  extStatHook: This process does not report performance statistics

S  handle memory type is RSTSPROMMM

B

B Mon Oct  8 17:30:44 2012

B  dbmyclu : info : my major identification is 172696986, minor one 4401.

B  dbmyclu : info : Time Reference is 1.12.2001 00:00:00h GMT.

B  dbmyclu : info : my initial uuid is F28A71503D850556E10000000A4B259A.

B  dbmyclu : info : current optimistic cluster level: 0

B  dbmyclu : info : pessimistic reads set to 2.

B

B Mon Oct  8 17:30:45 2012

B  dbdynpdb2: no VERSION column found in table DYNPSOURCE

Y  *DYLOAD* SAPMF02D 0101: waited 200 ms, load is under construction

B

B Mon Oct  8 18:50:23 2012

B  RECONNECT: rsdb/reco_trials: 3

B  RECONNECT: rsdb/reco_sleep_time: 5

B  RECONNECT: rsdb/reco_sync_all_server: OFF

B  ***LOG BV4=> reconnect state is set for the work process [dblink       1985]

B  ***LOG BYY=> work process left reconnect status [dblink       1986]

M  ThHdlReconnect: reconnect o.k.

L

L Mon Oct  8 23:02:45 2012

L  SXPG: ------------- start -------------------

L  SXPG: 2012/10/08 21:02:45 708973

L  SXPG:   Start status of external program: external program successfully started

L  SXPG:   Id of external process: 21.039

L  SXPG:   Conversation ID: 33285675

L  SXPG:   Contents of protocol:

L  SXPG:     No protocol messages reported

L  SXPG: ------------- start -------------------

L  SXPG: -------------- end --------------------

L  SXPG: 2012/10/08 21:02:45 716223

L  SXPG:   rc= 0

L  SXPG:   Exit status of external program: no errors reported

L  SXPG:   Exit code: 0

L  SXPG:   Contents of protocol:

L  SXPG:     No protocol messages reported

L  SXPG: -------------- end --------------------

L  SXPG: end of SXPG_STEP_XPG_START

L  SXPG: 2012/10/08 21:02:45 716416