SAP 4.6C hangs when Brtools run after oracle upgrade 10.2.0.4

[Former Member]

Hi All,

We have Upgraded the Oracle 9i to 10.2.0.4 for sap 4.6C production machine. After the upgrade the sap was succesfully able to start. even i can see the version was reflected in the sap system status. in the Opatch lsinventory all the bug fixes are covered. However when ever the brtools starts like taking the backup, or dbcheck, or archivelog backup. we are getting terminated with errors and the system system hangs. here is the initSID.log as follows.

**************************************************************************************************

Wed Nov 18 12:52:02 2009

Errors in file i:\oracle\cpa\saptrace\usertrace\cpa_ora_5888.trc:

ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [_kghstack_err+101] [PC:0x3FB10E9] [ADDR:0x4E202024] [UNABLE_TO_READ] []

ORA-00607: Internal error occurred while making a change to a data block

ORA-00602: internal programming exception

ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [_kghstack_err+101] [PC:0x3FB10E9] [ADDR:0x4E202024] [UNABLE_TO_READ] []

ORA-00607: Internal error occurred while making a change to a data block

ORA-00602: internal programming exception

ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [_kghstack_err+101] [PC:0x3FB10E9] [ADDR:0x4E202024] [UNABLE_TO_READ] []

ORA-00607: Internal error occurred while making a change to a data block

ORA-00602: internal programming exception

ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [_kghstack_err+101] [PC:0x3FB10E9] [ADDR:0x4E202024] [UNABLE_TO_READ] []

ORA-00607: Internal error occurred while making a change to a data block

ORA-00602: internal programming exception

ORA-07445: excepti

Wed Nov 18 12:52:04 2009

Doing block recovery for file 28 block 68766

Block recovery from logseq 13424, block 2157 to scn 186123675

Wed Nov 18 12:52:04 2009

Recovery of Online Redo Log: Thread 1 Group 11 Seq 13424 Reading mem 0

Mem# 0: M:\ORACLE\CPA\ORIGLOGA\LOG_G11M1.DBF

Mem# 1: M:\ORACLE\CPA\MIRRLOGA\LOG_G11M2.DBF

Block recovery completed at rba 13424.2423.16, scn 0.186123678

Doing block recovery for file 27 block 1547

Doing block recovery for file 27 block 1547

Block recovery from logseq 13424, block 2157 to scn 186123679

Wed Nov 18 12:52:04 2009

Recovery of Online Redo Log: Thread 1 Group 11 Seq 13424 Reading mem 0

Mem# 0: M:\ORACLE\CPA\ORIGLOGA\LOG_G11M1.DBF

Mem# 1: M:\ORACLE\CPA\MIRRLOGA\LOG_G11M2.DBF

Block recovery stopped at EOT rba 13424.2426.16

Block recovery completed at rba 13424.2426.16, scn 0.186123679

Wed Nov 18 13:35:44 2009

Starting ORACLE instance (normal)

Wed Nov 18 13:59:16 2009

Thread 1 advanced to log sequence 13425 (LGWR switch)

Current log# 13 seq# 13425 mem# 0: M:\ORACLE\CPA\ORIGLOGA\LOG_G13M1.DBF

Current log# 13 seq# 13425 mem# 1: M:\ORACLE\CPA\MIRRLOGA\LOG_G13M2.DBF

Wed Nov 18 15:45:52 2009

Shutting down instance: further logons disabled

Wed Nov 18 15:45:54 2009

Stopping background process QMNC

Wed Nov 18 15:45:55 2009

Stopping background process CJQ0

Wed Nov 18 15:45:56 2009

Stopping background process MMNL

Wed Nov 18 15:45:57 2009

Stopping background process MMON

Wed Nov 18 15:45:58 2009

Shutting down instance (immediate)

License high water mark = 27

Wed Nov 18 15:45:58 2009

Stopping Job queue slave processes, flags = 7

Wed Nov 18 15:45:58 2009

Job queue slave processes stopped

Wed Nov 18 15:46:01 2009

ALTER DATABASE CLOSE NORMAL

Wed Nov 18 15:46:01 2009

SMON: disabling tx recovery

SMON: disabling cache recovery

Wed Nov 18 15:46:01 2009

Shutting down archive processes

Archiving is disabled

Wed Nov 18 15:46:06 2009

ARCH shutting down

ARC1: Archival stopped

Wed Nov 18 15:46:11 2009

ARCH shutting down

ARC0: Archival stopped

Wed Nov 18 15:46:12 2009

Thread 1 closed at log sequence 13425

Successful close of redo thread 1

Wed Nov 18 15:46:12 2009

Completed: ALTER DATABASE CLOSE NORMAL

Wed Nov 18 15:47:10 2009

ALTER DATABASE DISMOUNT

Wed Nov 18 15:47:10 2009

Completed: ALTER DATABASE DISMOUNT

ARCH: Archival disabled due to shutdown: 1089

Shutting down archive processes

Archiving is disabled

Archive process shutdown avoided: 0 active

ARCH: Archival disabled due to shutdown: 1089

Shutting down archive processes

Archiving is disabled

Archive process shutdown avoided: 0 active

Wed Nov 18 15:51:52 2009

Starting ORACLE instance (normal)

Wed Nov 18 15:51:52 2009

System parameter db_block_buffers enabled without VLM on.

LICENSE_MAX_SESSION = 0

LICENSE_SESSIONS_WARNING = 0

Picked latch-free SCN scheme 2

Autotune of undo retention is turned on.

IMODE=BR

ILAT =12

LICENSE_MAX_USERS = 0

SYS auditing is disabled

ksdpec: called for event 13740 prior to event group initialization

Starting up ORACLE RDBMS Version: 10.2.0.4.0.

System parameters with non-default values:

processes = 100

sessions = 115

timed_statistics = TRUE

event = 10411 trace name context forever, level 1

shared_pool_size = 377487360

large_pool_size = 8388608

java_pool_size = 67108864

streams_pool_size = 50331648

shared_pool_reserved_size= 37614518

firstspare_parameter = 1

control_files = M:\ORACLE\CPA\SAPDATA1\CNTRL\CNTRLCPA.DBF, M:\ORACLE\CPA\MIRRLOGA\CNTRL\CNTRLCPA.DBF, M:\ORACLE\CPA\ORIGLOGA\CNTRL\CNTRLCPA.DBF

control_file_record_keep_time= 30

db_block_buffers = 68874

db_block_size = 8192

compatible = 10.2.0

log_archive_dest = H:\oracle\CPA\saparch\CPAarch

log_buffer = 2850816

log_checkpoint_interval = 0

log_checkpoint_timeout = 0

db_files = 254

db_file_multiblock_read_count= 8

undo_management = AUTO

undo_tablespace = PSAPUNDO

undo_retention = 43200

recyclebin = off

remote_login_passwordfile= EXCLUSIVE

local_listener = LISTENER_CPA

cursor_space_for_time = FALSE

job_queue_processes = 1

create_bitmap_area_size = 33554432

bitmap_merge_area_size = 33554432

parallel_max_servers = 4

parallel_execution_message_size= 8192

hash_area_size = 4194304

background_dump_dest = I:\ORACLE\CPA\SAPTRACE\BACKGROUND

user_dump_dest = I:\ORACLE\CPA\SAPTRACE\USERTRACE

max_dump_file_size = 20000

session_max_open_files = 20

optimizer_features_enable= 9.2.0

sort_area_size = 2097152

sort_area_retained_size = 0

db_name = CPA

open_cursors = 800

optimizer_mode = CHOOSE

star_transformation_enabled= true

optimizer_index_cost_adj = 10

pga_aggregate_target = 25165824

PMON started with pid=2, OS id=2880

PSP0 started with pid=3, OS id=5268

MMAN started with pid=4, OS id=5768

DBW0 started with pid=5, OS id=5076

LGWR started with pid=6, OS id=6600

CKPT started with pid=7, OS id=812

SMON started with pid=8, OS id=3052

RECO started with pid=9, OS id=6132

CJQ0 started with pid=10, OS id=3648

MMON started with pid=11, OS id=4872

MMNL started with pid=12, OS id=3656

Wed Nov 18 15:51:53 2009

ALTER DATABASE MOUNT

Wed Nov 18 15:51:57 2009

Setting recovery target incarnation to 1

Wed Nov 18 15:51:57 2009

Successful mount of redo thread 1, with mount id 4087130713

Wed Nov 18 15:51:57 2009

Database mounted in Exclusive Mode

Completed: ALTER DATABASE MOUNT

Wed Nov 18 15:51:58 2009

ALTER DATABASE OPEN

Wed Nov 18 15:51:58 2009

LGWR: STARTING ARCH PROCESSES

ARC0 started with pid=14, OS id=5620

Wed Nov 18 15:51:58 2009

ARC0: Archival started

Wed Nov 18 15:51:58 2009

ARC1: Archival started

LGWR: STARTING ARCH PROCESSES COMPLETE

Thread 1 opened at log sequence 13425

Current log# 13 seq# 13425 mem# 0: M:\ORACLE\CPA\ORIGLOGA\LOG_G13M1.DBF

Current log# 13 seq# 13425 mem# 1: M:\ORACLE\CPA\MIRRLOGA\LOG_G13M2.DBF

Successful open of redo thread 1

Wed Nov 18 15:51:58 2009

MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set

ARC1 started with pid=15, OS id=3344

Wed Nov 18 15:51:59 2009

ARC0: Becoming the 'no FAL' ARCH

ARC0: Becoming the 'no SRL' ARCH

Wed Nov 18 15:51:59 2009

ARC1: Becoming the heartbeat ARCH

Wed Nov 18 15:51:59 2009

SMON: enabling cache recovery

Wed Nov 18 15:52:00 2009

Successfully onlined Undo Tablespace 31.

Wed Nov 18 15:52:00 2009

SMON: enabling tx recovery

Wed Nov 18 15:52:00 2009

Database Characterset is US7ASCII

Opening with internal Resource Manager plan

where NUMA PG = 1, CPUs = 4

replication_dependency_tracking turned off (no async multimaster replication found)

Starting background process QMNC

QMNC started with pid=16, OS id=5172

Wed Nov 18 15:52:04 2009

Completed: ALTER DATABASE OPEN

Wed Nov 18 15:53:00 2009

Thread 1 advanced to log sequence 13426 (LGWR switch)

Current log# 14 seq# 13426 mem# 0: M:\ORACLE\CPA\ORIGLOGB\LOG_G14M1.DBF

Current log# 14 seq# 13426 mem# 1: M:\ORACLE\CPA\MIRRLOGB\LOG_G14M2.DBF

Wed Nov 18 16:02:30 2009

Errors in file i:\oracle\cpa\saptrace\usertrace\cpa_ora_6540.trc:

ORA-00600: internal error code, arguments: [qertbFetchByRowID], [], [], [], [], [], [], []

Wed Nov 18 16:07:23 2009

Hex dump of (file 15, block 157170) in trace file i:\oracle\cpa\saptrace\background\cpa_dbw0_5076.trc

Corrupt block relative dba: 0x03c265f2 (file 15, block 157170)

Bad header found during preparing block for write

Data in bad block:

type: 48 format: 0 rdba: 0x30303030

last change scn: 0x5043.50415330 seq: 0x41 flg: 0x41

spare1: 0x30 spare2: 0x30 spare3: 0x5a31

consistency value in tail: 0x8b15a015

check value in block header: 0x4730

block checksum disabled

Wed Nov 18 16:07:24 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_dbw0_5076.trc:

ORA-00600: internal error code, arguments: [kcbzpbuf_1], [4], [1], [], [], [], [], []

Wed Nov 18 16:07:26 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_dbw0_5076.trc:

ORA-00600: internal error code, arguments: [kcbzpbuf_1], [4], [1], [], [], [], [], []

DBW0: terminating instance due to error 471

Wed Nov 18 16:07:26 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_pmon_2880.trc:

ORA-00471: DBWR process terminated with error

Wed Nov 18 16:07:26 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_ckpt_812.trc:

ORA-00471: DBWR process terminated with error

Wed Nov 18 16:07:28 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_q001_1980.trc:

ORA-00471: DBWR process terminated with error

Wed Nov 18 16:07:28 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_j000_5552.trc:

ORA-00471: DBWR process terminated with error

Wed Nov 18 16:07:29 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_lgwr_6600.trc:

ORA-00471: DBWR process terminated with error

Wed Nov 18 16:12:29 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_mman_5768.trc:

ORA-00471: DBWR process terminated with error

Wed Nov 18 16:12:29 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_psp0_5268.trc:

ORA-00471: DBWR process terminated with error

Wed Nov 18 16:12:34 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_dbw0_5076.trc:

ORA-27300: OS system dependent operation:spdde:9262:4210 failed with status: 0

ORA-27301: OS failure message: The operation completed successfully.

ORA-27302: failure occurred at: skgpkill4

ORA-27144: attempt to kill process failed

ORA-00600: internal error code, arguments: [kcbzpbuf_1], [4], [1], [], [], [], [], []

Wed Nov 18 16:12:39 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_smon_3052.trc:

ORA-00471: DBWR process terminated with error

***********************************************************************************************************************

can any one please help me on this . as this is production we are facing this problem from past 3 weeks. no one was able to analyze whats giong wrong. so because of this we even stopped the backup's of archivelog and online backup. Please some one help me asap. Your help is really appreciated.

Hemanth

[stefan_koehler · Accepted solution]

Hello Hemanth,

this looks like a block corruption .. i would highly recommend to open a SAP call for that issue.

We would need much more information from you to help you, so the fastest way is to open a sap call.

You can also check metalink .. there are several notes (especially after applying patchset 10.2.0.4).

Regards

Stefan

[Former Member · Accepted solution]

Hi Stefan,

I have already raised an sap call with very high priority. Sap has also verified and given me a few commands to verify if there any data block corruptions by using the dbverify and export of database to null. The export log and the db verify log shows no error. all the database if fine without any data block corruptions. is it like if patchset is not done well then will be get these type of problems? But i can see the Opatch lsinventory log shows all the patchsets and the bugs covered as per the patchset we applied. here is the Opatch lsinventory logfile for your reference. finally sap told that it is due to fault memory chip. we also did a memory test to see is it really a fault memory chip. but the memory test shows no errors. what could be the error. please help!

-

---

Installed Top-level Products (3):

Oracle Database 10g 10.2.0.1.0

Oracle Database 10g Release 2 Patch Set 1 10.2.0.2.0

Oracle Database 10g Release 2 Patch Set 3 10.2.0.4.0

There are 3 products installed in this Oracle Home.

Interim patches (1) :

Patch 8880857 : applied on Sat Nov 14 12:32:58 CET 2009

Unique Patch ID: 11799402

Created on 18 Sep 2009, 00:36:22 hrs PST8PDT

Bugs fixed:

8880857, 7270434, 7597354, 6945157, 4966512, 6827260, 8426816, 7172752

8702276, 7196863, 6857917, 8217011, 8433026, 8409848, 8341623, 5727166

8199266, 6687381, 6271590, 6749617, 5630796, 5457450, 8301952, 7442260

8628970, 6460304, 6919709, 6282944, 8205205, 6765855, 4627274, 8236723

7315521, 8232056, 8762356, 7452373, 8220734, 8537544, 8599814, 4598439

6808012, 7370410, 7585472, 8247017, 8357689, 6642751, 8355120, 8366255

8780444, 8660243, 6394861, 7298987, 6126439, 6005525, 7556778, 6009358

8512015, 5345978, 5598333, 7440973, 8339055, 8750358, 8537075, 5926510

8753157, 5258914, 8640813, 6165874, 7670135, 7268438, 5743318, 6784671

6034609, 8242879, 8708077, 8308278, 6730567, 8244734, 7041059, 8556340

6521934, 7185460, 8268348, 8506458, 8542772, 7189984, 8547175, 5452113

8295094, 6825866, 6915767, 5933656, 7207654, 7662491, 6988971, 5224794

8566266, 6134664, 7487294, 7382225, 6004127, 7434148, 5726433, 8348028

8420551, 6932504, 6643194, 7145531, 8586236, 6755419, 8609346, 7658224

6029786, 7126504, 8423950, 7225204, 7551539, 5643036, 6997042, 6814502

5399699, 5231155, 8209514, 5702433, 8309791, 6338068, 7695070, 8526100

8467800, 8348514, 8318020, 7891471, 7657126, 6399168, 5394888, 8284633

8284438, 7716219, 8471715, 8256539, 7291157, 8407546, 6401347, 8234504

7698349, 7520091, 7411265, 7524768, 7480997, 8569254, 6996030, 8510218

6655588, 4740049, 4085593, 6340297, 6023472, 6940487, 6964283, 6726958

6954407, 8559466, 7033630, 7039896, 7763473, 8453040, 7446153, 4871892

8247215, 6954722, 6053134, 8198906, 7591985, 6975521, 8274994, 7615454

7521164, 6745682, 7598781, 7639914, 6153930, 6408317, 8541774, 8373035

7659217, 7258065, 6035809, 6162731, 5852921, 7434973, 5745084, 5247029

6155919, 5872943, 5939230, 6073325, 7722575, 8273910, 7423771, 6628122

8216754, 7836441, 8282743, 7704827, 8347139, 8439220, 7023147, 8303388

7695742, 6512811, 6021687, 5847453, 6522654, 8281791, 7207921, 3320140

6960874, 6392076, 7475294, 7381308, 8373286, 6800507, 6858062, 7330909

7477934, 7661251, 7835247, 8247855, 8342923, 8299095, 6741425, 7272039

7441785, 7692248, 6526370, 7116357, 7463138, 6331062, 7830001, 8331807

8392340, 7688581, 6824129, 5259835, 8199533, 7639602, 5705385, 5764671

6052226, 4693355, 7710538, 7575925, 5868598, 7686186, 7219752, 8267348

6919819, 7257770, 8230457, 7257461, 7263842, 7332423, 7307821, 7514969

7581964, 7032684, 5879114, 8299199, 6404447, 7527908, 6014513, 7378661

6145177, 7552067, 6799205, 6352003, 7502837, 6069409, 6371909, 8227091

6952701, 7298688, 7396409, 8227106, 6343150, 6403091, 6610218, 6681695

6724797, 6802650, 6804746, 6817593, 6833965, 6960489, 7136866, 7173005

7238230, 7457766, 7600026, 7599944, 8251247, 8268054, 8324577, 8331466

8287504, 6711853, 6976005, 7460818, 7276960, 5126719, 7044551, 6882739

7568556, 7159505, 6840740, 7593835, 6941717, 7569205, 6955744, 5404871

7013124, 6345573, 6070225, 7432601, 6490140, 6074620, 7140204, 7136489

7494333, 5476236, 6980597, 7643632, 7197445, 6469211, 8287155, 8304589

8283650, 3934160, 8292378, 7331867, 6599920, 7036453, 7363767, 6265559

7341598, 6987790, 6926448, 6181488, 6615740, 7125408, 6596564, 6903051

6120004, 6988017, 6851438, 7527650, 7432514, 7553884, 7499353, 7175513

6452375, 7393804, 7523787, 7189645, 8239142, 7043989, 6268409, 7693128

7345904, 7149004, 6683178, 6413089, 6650256, 8340379, 8328954, 6378112

7710551, 6778714, 6919764, 7254987, 6752765, 6418420, 6733655, 7247217

7691766, 7552042, 4637902, 6219529, 8347704, 7552082, 7424804, 6658484

8340383, 7378735, 6509115, 5863926, 8340387, 7528105, 6673834, 7243184

7291029, 8223467, 7237687, 7665893, 7519531, 7293244, 5262526, 8282992

8307237, 8208765, 8208643, 7319691, 7505535, 7489862, 7688543, 5945060

7238163, 4459936, 7172531, 5147386, 6500033, 6908967, 7459192, 7694979

4736319, 8210889, 8223481, 5893331, 7704811, 6143420, 7193068, 6367360

8214576, 7162704, 6869712, 6999955, 5705196, 6642692, 8362683, 6638558

7340448, 8258839, 5399610, 6370967, 7585970, 7378401, 7043173, 7159451

6122696, 5697354, 6981690, 5256703, 8216864, 7697802, 7523475, 6838714

7439465, 5730603, 8225197, 7592450, 5099019, 6319761, 7664560, 7509689

7229351, 7278117, 7654407, 7690331, 7828479, 7003718, 6491175, 6737765

6991626, 7634610, 7519687, 6896371, 5686407, 6379441, 6313035, 5849687

6894671, 7170213, 8201796, 6656309, 6155146, 7516867, 7296258, 7715044

7486155, 7450460, 6113018, 7447345, 7305703, 7513673, 6640411, 7558379

7597059, 7372337, 7484102, 6730125, 6936368, 7648406, 6047490, 6270137

6452035, 4902697, 7677780, 6661393, 7501180, 7592168, 7007924, 7295298

7361423, 7453155, 6282093, 5701723, 7318276, 6652767, 7458976, 7241059

7385253, 6759910, 7375077, 7272297, 7008262, 6870047, 6896554, 7263061

7395972, 6939835, 7515898, 6454634, 7314091, 7439689, 6811031, 5229627

5586604, 7196532, 7573151, 7254367, 7606362, 6083292, 6994490, 5718815

7207932, 6013981, 7592319, 7415038, 6144426, 6145687, 7509964, 6629890

7444320, 7579079, 6350579, 7584866, 7522608, 5648287, 5709135, 6430500

6833602, 7211965, 7236148, 7257160, 7325597, 7429070, 7430474, 7439957

7454131, 7522909, 7515828, 7462072, 6904068, 7145872, 7190477, 7498506

6519198, 6770417, 6395755, 6782437, 6452766, 6775231, 7189447, 7330258

7144872, 7395472, 6216149, 7477691, 5951091, 7448543, 7522473, 6967923

6742924, 4695511, 7413928, 6054929, 7129643, 5879865, 7430745, 6923450

6685261, 6415531, 6359490, 7196894, 6924458, 7151667, 6627593, 6766884

6766956, 6791019, 7494876, 7336032, 6368000, 7037203, 6955040, 5945647

5573896, 7246117, 7433792, 6670740, 7388606, 7118478, 6768289, 7002207

5883691, 7251909, 7366749, 7280289, 7273832, 7155655, 7138405, 6845871

7331569, 7188610, 7480785, 7359366, 6874522, 5343559, 7356443, 7144891

6375150, 5059447, 7038750, 6877038, 6193945, 7309458, 6854919, 5490816

6605106, 6472286, 6606749, 6067825, 6932206, 6455412, 6674196, 7362653

6746196, 6774830, 6451626, 7364935, 4561087, 4440589, 7315642, 7411865

5348308, 6865378, 7331929, 7191744, 7342082, 6506818, 5944239, 7271619

6619922, 5926711, 5996457, 7326738, 7386320, 5131180, 7046751, 6672737

7237571, 7269450, 6782276, 7113299, 7190270, 4235212, 7348412, 5712781

7357454, 6978876, 7287289, 7162648, 6635214, 6844739, 7046187, 6962622

6797677, 7154415, 7142215, 7300599, 6399597, 7013768, 6875111, 6959964

6133008, 7123643, 6122097, 6678845, 7210296, 7216535, 6594005, 6527074

6262792, 7340474, 7313129, 6504899, 5095023, 6134368, 6725634, 7210921

6870937, 7014646, 7017637, 6805009, 6876196, 6972343, 6897966, 6809093

6736159, 7244238, 7127618, 7195088, 6607486, 6822672, 6687129, 6655657

6355282, 6489332, 6789923, 6454468, 7005671, 7140644, 7039080, 6695247

6689334, 6936964, 6445329, 7253058, 6862248, 6082292, 7004914, 5690152

6374297, 6653934, 5208177, 6119017, 6679303, 6600051, 7228971, 6988848

7149751, 6221403, 6440977, 6972291, 7188932, 6344497, 6724838, 7218676

7036589, 4033868, 5883585, 6944036, 6327692, 6163785, 6913090, 7027551

6130317, 5868257, 5190885, 6798910, 6934892, 7022905, 6795880, 6935367

6667800, 5386874, 6925163, 6076414, 6432837, 6956212, 6714608, 6336234

6471770, 6771400, 6511850, 7003151, 6952800, 6656630, 6898054, 5688060

7154241, 7150609, 6660648, 6954829, 6207468, 6455659, 5970301, 6530171

6607505, 5213488, 5944121, 5843814, 6163771, 7117709, 6896701, 6833159

6759032, 5900004, 4393920, 6645719, 5251842, 7019835, 6826661, 6883418

7011807, 6241222, 6956461, 6193802, 7006942, 6986658, 5410059, 6626018

5704108, 5890312, 5934363, 6760697, 6121357, 6110752, 6725855, 5923486

6705635, 5234947, 5902962, 6143047, 5895190, 6917874, 6151380, 6456530

5368296, 5386204, 6369463, 6768114, 6200820, 6972843, 5880921, 6506617

6994160, 6345197, 6506816, 6670579, 6688740, 5961436, 6823287, 6604020

6372848, 6414616, 6195891, 6822995, 6595426, 5747462, 6120483, 6981215

6082832, 5623467, 4551675, 6329318, 6324944, 6376928, 6783426, 6051177

6024730, 5935935, 6778404, 4169479, 5756769, 5714944, 6670551, 6607676

6084232, 6815733

-

---

Hemanth

[Former Member · Accepted solution]

Hi,

What's in in the alert log....

and it could be a coruption in the Rollback.

Have you also gone through this note as you have parameters set which you do not need.

https://websmp230.sap-ag.de/sap(bD1lbiZjPTAwMQ==)/bc/bsp/spn/sapnotes/index2.htm?numm=830576

Mark

[Former Member · Accepted solution]

Hello Mark,

I have already set the parameters as suggestd by the sap and sap also had a look over the parameters and said there is no problem with the parameter configurations. But i am really surprised when ever the brtools are run we are facing this problem. if not the system is stable. all are users are able to logon to system and work. we get the following error when ever run teh brtools. we are getting lot of errors.

he frequent error we see is

BR0301W: sql error -3113 at location BRdblogwrite -10, SQL statement: 'INSERT INTO SAP_SDBAD (BEG, FUNCT, POS, LINE) VALUES ('20091120120000', 'svd', '0029', CS0134452009111917513220091120120100 0000000000000SAPCPAA0G28UAPJ90001H:\ORACLE\CPA\SAPARCH\CPAARCHARC13445_0564246247.001')'

ORA-03113: end-of-file on communication channel

BR0325W: Writing to database log failed

and after this the db suddenly hangs and get disconnects all the time. this is the major problem. At this time we can also see ,most of the times that the work process all get jammed , i.e get into run mode. all most all the dialog's and only a spool process get into run mode. if we try to terminate the work process they will not get terminate and they will just hang like that.

some time without workprocess getting hanged they will be in waiting mode only all in wait status and the db will get suddenly disconnected. even the db is down we see in the mmc that sap is green and work process is in wait mode and we will not be able to connect.

if we dont excute the brtools the sap will work fine. that is the issue.

[Former Member · Accepted solution]

What is version and patch level of your brtools?

[Former Member · Accepted solution]

Hi,

As for the parameters where's the log picking this up as it should not set as of 10G?

 optimizer_features_enable= 9.2.0

The H:\ destination is this a local drive or mapped ?

Regards

Mark

[Former Member · Accepted solution]

Hi ,

SAP Version is sap 4.6C. the brtools are with version 700 and 33 is patch level. here are the iniSID.ora file of my system. please suggest. i dont think that is a problem with the parameter configuration. Can u please suggest me one more thing that if the patches are not installed correctly do we get this type of errors? But i see the patches installed and bugs fixed in the Opatch lsinventory.

########################################################################

1. This init.ora profile is a transparent copy of Oracle spfile. #

2. It was created by BRARCHIVE at 2009-11-18 22.00.00. #

3. Please do not change this file! Maintain only the spfile. #

4. You can use BRSPACE for the altering of database parameters: #

5. brspace -f dbparam #

########################################################################

*._first_spare_parameter=1

*._fix_control='5099019:ON','5705630:ON','5765456:3','6221403:ON','6329318:ON','6399597:ON','6430500:ON','6440977:ON','6626018:ON','6670551:ON','6972291:ON','7325597:ON','7692248:ON','7891471:ON'

*.background_dump_dest='I:\oracle\CPA\saptrace\background'

*.bitmap_merge_area_size=33554432

*.compatible='10.2.0'

*.control_file_record_keep_time=30

*.control_files='M:\oracle\CPA\sapdata1\cntrl\cntrlCPA.dbf','M:\oracle\CPA\mirrloga\cntrl\cntrlCPA.dbf','M:\oracle\CPA\origloga\cntrl\cntrlCPA.dbf'

*.create_bitmap_area_size=33554432

*.cursor_space_for_time=FALSE

*.db_block_buffers=102400

*.db_files=254

*.db_name='CPA'

*.event='10027 trace name context forever, level 1','10028 trace name context forever, level 1','10142 trace name context forever, level 1','10183 trace name context forever, level 1','10191 trace name context forever, level 1','10411 trace name context forever, level 1','10629 trace name context forever, level 32','10753 trace name context forever, level 2','14532 trace name context forever, level 1','38068 trace name context forever, level 100','38085 trace name context forever, level 1','38087 trace name context forever, level 1','44951 trace name context forever, level 1024','10049 trace name context forever, level 2'

*.hash_area_size=4194304

*.java_pool_size=67108864

*.job_queue_processes=1

*.large_pool_size=8388608

*.local_listener='LISTENER_CPA'

*.log_buffer=1048576

*.log_checkpoint_interval=0

*.log_checkpoint_timeout=0

*.max_dump_file_size='20000'

*.open_cursors=800

*.optimizer_index_cost_adj=20

*.parallel_execution_message_size=16384

*.parallel_max_servers=4

*.processes=100

*.recyclebin='off'

*.remote_login_passwordfile='EXCLUSIVE'

*.session_max_open_files=20

*.sessions=200

*.shared_pool_reserved_size=37614518

*.shared_pool_size=524288000

*.sort_area_retained_size=0

*.sort_area_size=2097152

*.star_transformation_enabled='true'

*.streams_pool_size=50331648

*.timed_statistics=TRUE

*.undo_management='AUTO'

*.undo_retention=43200

*.undo_tablespace='PSAPUNDO'

*.user_dump_dest='I:\oracle\CPA\saptrace\usertrace'

*.core_dump_dest='E:\ORACLE\CPA\102\RDBMS\TRACE'

*._sort_elimination_cost_ratio=10

*._optimizer_mjc_enabled='FALSE'

*._optimizer_better_inlist_costing='OFF'

*._optim_peek_user_binds='FALSE'

*.filesystemio_options=SETALL

*.log_archive_dest='H:\oracle\CPA\saparch\CPAarch'

*.log_archive_format=%t_%s_%r.dbf

*.log_checkpoints_to_alert='TRUE'

*.parallel_threads_per_cpu=1

*.remote_os_authent='TRUE'

*._cursor_features_enabled=10

*.db_writer_processes=1

*.query_rewrite_enabled='FALSE'

*.PGA_AGGREGATE_TARGET=314572800

*.USE_INDIRECT_DATA_BUFFERS=TRUE

[Former Member · Accepted solution]

Hi,

Can you also paste the spfile.ora

mark

[Former Member · Accepted solution]

Hi Mark,

As we did lot of chnges in the Pfile as per the sap recommendations we made the spfile to spfile.old and currently the system is running on pfile itself. still the spfile is not yet generated.

Hemanth

[Former Member · Accepted solution]

Hi,

I am assuming you have restarted the system since renaming the spfile and running the pfile.

looking at you config file It does seem strange you are running with no db_cache or sga_max_size ?

Is the redo log drive a local disk to the machine ?

Regards

Mark

[Former Member · Accepted solution]

Hi,

It is a local Drive.

Hemanth

[Former Member · Accepted solution]

Hi Mark,

Yes we renamed the spfile and restarted the db and sap and running the system currently on pfile. The redolog drive is a local drive.

Hemanth

[Former Member · Accepted solution]

Hi,

BR0301W: sql error -3113 at location BRdblogwrite -10

Intemates losing connection to oracle, hence write failure on the Drive

Now this can mean a host of things....What oracle client are you running ?

Mark

[Former Member · Accepted solution]

I copied the oracle instant client 10.2.0.2 to E;\usr\sap\SID\sys\exe\run. so i am using the oracle instant client 10.2.0.2.

Hemanth

[Former Member · Accepted solution]

Hi,

when you did the upgrade what what patch level was you on of 9i and did you have any funnies when upgrading.

The database does start up normally e.g startup.

Mark

[Former Member · Accepted solution]

Hi I did the upgrade 3 weeks before. During the upgrade the upgrade went smoothly. I only got one error that upgrade.xml is not a valid file. as per the sap note suggested by sap there was workaround for that. i checked the psaptem is a dictionary based tablespace. so dropped the table space and recreated as suggested in the sap note. i upgraded the oracle to 10.2.0.2. after the upgrade i took the backup. even i see the backup was done successfull but went a bit longer time then the expected time. i dont see any errors in the backup. after that the other day i did the oracle upgrade 10.2.0.4 patchset upgrade and pathes relevant to that. during that period i see some magics as u said. during the upgrade i see the result of sql statemets are not shown properly. like if i execute an sql statement most of the times it is not showing like pl sql procedure completed successfuly. some times it showed that the statement comppleted successfully. when i ran the patchset upgrade script the last few lines like the script completed successfully were not shown properly.

But the opatches and other installations were successfully and were ok. even opatch lsinventory was also done successfully. I am no getting what could be wrong. is it any HDD problem? any memory areas in hdd are not corrupted? what could be the problem?

Hemanth

[Former Member · Accepted solution]

Hi,

So the database was fine under releases 9i ?

It worked fine when you went to 10.2.0.2

It started running funny when you went to 10.2.0.4 ?

HDD you could temporally point the archive_dest to another file drive for a test if you have space

Have you a spare server you could clone this system to for testing purposes ?

Mark

[Former Member · Accepted solution]

Yes The system is ok with 9i and even ok with 10.2.0.2. However there is no much gap between 10.2.0.2 and 0.4. it is hardly 1 day gap before the 10.2.0.4 upgrade. Yes you are right. when the system went to 10.2.0.4 it started funny. yes we do have a spare hardware for cloning. if we clone it what are the tests we can conduct on the test machine. Now i have a few questions in ming.

1. is there any problem with the patchset upgrade. is the system behaving in a different way because of this?

2. if so can we reexecute the patchset upgrade once again? if so can u please tell me how can we do that and what are the steps we can take to make it succcessful and reexecute the patchset upgrade properly.

3. Can we revert back the system to 9i? if so how can we do that?

4. how can we find the exact error ? what could be the reason for this problem?

5. As this is the production machine. what would be the other options for us right now? can u please suggest that?

Hemanth

[Former Member · Accepted solution]

Hi,

just noticed this have look.

https://websmp230.sap-ag.de/sap(bD1lbiZjPTAwMQ==)/bc/bsp/spn/sapnotes/index2.htm?numm=1273790

.

My point is did you have these errors on 10.2.0.2 when you ran it for a day....By your remarks No

So there is an issue with patch level 0.4 ==.> have a look at the upgrade notes I think there's something about regressing ?

You could test on your clone.

Clone:

You could clone the database at level 9i test functionality backup redo's etc

upgrade to 10.2.0.2 do the same test

patch to 0.4 test see what happens.

Mark

[Former Member · Accepted solution]

Hi,

The sap note u provided seems to be intresting. so if we apply this note it will cover if there are any errors in the previous patchset right?. does it mean that?

Yes for cloning and doing the testing i think we need a customer approval. so i will inform you as soon as we do this test in the sandbox. in the mean while can we go ahead with the patchset given in the sap note provide by you above. will it be ok?

any other quick checks or work arounds ? can u please suggest ?

Hemanth

[Former Member · Accepted solution]

Hi Mark,

can we reexecute the catupgrd.sql once again and recreate the oracle dictionary for a check. will it be ok. can u please suggest.

Hemanth

[Former Member · Accepted solution]

Hi,

All I can say is the sap note has some relevance to your errors. so it's worth putting it in.

Will it fix it who knows...I've done quite a few upgrades and patching but your's seems unique.

I tend to when I can clone production then practice the upgrade and functions to be sure before I go into prod.

Strange it did not show itself in QA....

Incidentally you should not rule out HDD hardware failure. What is your disk array manufacturer, as it's production issue I'm assuming you have a support contract. Can you get them to look over your disk array for any strange messages etc.

Regards

mark

[Former Member · Accepted solution]

Hi Mark,

All you help is highly appreciated. i will definetly check with the respective team for HDD check. will let u know the status soon and update you the status on this when ever it is ...

Hemanth

[Former Member · Accepted solution]

Hi Mark,

One more question i can ask is can i run the catupgrd.sql at this moment to recreate the dictionary? will it be ok if i run it now? if we run it now will it not mess up the db at this moment?

ALso please suggest if we can run any other sql scripts which doesnt harm and may be useful to have a check.

Hemanth

[Former Member · Accepted solution]

Hi,

Can you please suggest on this?

Hemanth

[Former Member · Accepted solution]

Hi All,

can any one suggest me on this.

can i run the catupgrd.sql at this moment to recreate the dictionary? will it be ok if i run it now? if we run it now will it not mess

up the db at this moment?

Also can u please suggest what scripts we can run at this moment to have a check and which will not mess up the db.

Hemanth

[Former Member · Accepted solution]

Hi,

There might be the corrupt block in the SDBAD. please have a check and try to eliminate the corrupt block. and then check the dbcheck and backups. It should work fine.

[Former Member · Accepted solution]

Hi ,

Yes you are right. There was a corrupt block in the SDBAD table and i identified the corrupt block and removed that. Now the backups and db checks are all running fine. thanks a million.