on 07-23-2011 6:11 AM
Dear Experts,
I am in a process to upgrade oracle 10.2.0.4 to 11G Database upgrade. During the Mid of the upgrade from 10.2.0.4 to 11G the upgrade immediately got terminated. DBUA is in step upgrading oracle server. It was almost all 50% completed. suddenly got terminated unknowingly. I checked the alertSID.log and all other relavant log files. rather i didnt find any root cause for the cause of the termination.
I suspect it might have got terminated due to inactive session security levied by our unix team.
I tried to restart the DBUA once again. I am getting the error ora-00600. I checked the alertSID.log. here is the log file for your reference.
I checked the trace.log. The last few lines for the trace.log, i will post it in the following thread.
I also checked oracleserver.log.
I will post the logfile followed by this thread.
However the current situation is
I tried to recover the oracle binaries with the option FLASH BACK.
1. startup nomount
2. FLASHBACK DATABASE TO RESTORE POINT BEFORE_11G_UPGRADE;
while i give that command I am getting the error end of file of communication channel. when i check the alertSID.log i see ora-600 error. the log files is as follows.
First i checked alertSID.log
Successful mount of redo thread 1, with mount id 3602317500
Fri Jul 22 16:54:24 2011
Allocated 3981120 bytes in shared pool for flashback generation buffer
Starting background process RVWR
RVWR started with pid=13, OS id=7594040
Fri Jul 22 16:54:25 2011
Database mounted in Exclusive Mode
Completed: ALTER DATABASE MOUNT
Fri Jul 22 16:54:29 2011
FLASHBACK DATABASE TO RESTORE POINT BEFORE_11G_UPGRADE
Flashback Restore Start
Fri Jul 22 16:54:35 2011
Errors in file /oracle/HR2/saptrace/background/hr2_dbw0_7782512.trc:
ORA-00600: internal error code, arguments: [999], [0x70000003D7B9FBA], [], [], [], [], [], []
Fri Jul 22 16:54:35 2011
Errors in file /oracle/HR2/saptrace/background/hr2_dbw0_7782512.trc:
ORA-00600: internal error code, arguments: [999], [0x70000003D7B9FBA], [], [], [], [], [], []
Fri Jul 22 16:54:35 2011
DBW0: terminating instance due to error 471
Instance terminated by DBW0, pid = 7782512
I have also checked the file Errors in file /oracle/HR2/saptrace/background/hr2_dbw0_7782512.trc: Here is the logfile data which gives the error.
Dump file /oracle/HR2/saptrace/background/hr2_dbw0_7782512.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Data Mining and Real Application Testing options
ORACLE_HOME = /oracle/HR2/102_64
System name: AIX
Node name: kftdeedcbx12
Release: 3
Version: 5
Machine: 00C95BEE4C00
Instance name: HR2
Redo thread mounted by this instance: 1
Oracle process number: 5
Unix process pid: 7782512, image: oracle@kftdeedcbx12 (DBW0)
2011-07-22 16:54:35.137
SERVICE NAME:() 2011-07-22 16:54:35.135
SESSION ID:(167.1) 2011-07-22 16:54:35.135
Block Checking: DBA = 402665978, Block Type = Undo data block
ERROR: Undo Block Corrupted. Error Code = 2005
ktu4ubck: Undo Record addr did not aligned to boundary
record number=1, addr=3d7b9fba, aligned addr=3d7b9fbc
UNDO BLK HEADER:
xid: 0x0000.01c.000001fa seq: 0x176 cnt: 0x6a irb: 0x6a icl: 0x0 flg: 0x0000
Rec Offset | Rec Offset | Rec Offset | Rec Offset | Rec Offset
-
0x00 0x1fe8 | 0x01 0x1fa6 | 0x02 0x1f54 | 0x03 0x1f12 | 0x04 0x1ec0
0x05 0x1e7e | 0x06 0x1e2c | 0x07 0x1dea | 0x08 0x1d98 | 0x09 0x1d56
0x0a 0x1d04 | 0x0b 0x1cc2 | 0x0c 0x1c70 | 0x0d 0x1c2e | 0x0e 0x1bdc
0x0f 0x1b9a | 0x10 0x1b48 | 0x11 0x1b06 | 0x12 0x1ab4 | 0x13 0x1a72
What can we do at this current situation. I am not that much expert in ORACLE DBA. However know a few things how to manage. Can any of the experts guide me how can we resolve the error at this current sitaution and move forward with the upgrade. What i need to know now is
1. How to revert back to the healthy 10.2.0.4 oracle now?
2. how to restart and complete the upgrade successfully at this situation.
Edited by: Hemanth Kumar on Jul 23, 2011 7:12 AM
Edited by: Hemanth Kumar on Jul 23, 2011 7:16 AM
Hello Hemanth,
well you did the wrong things at the wrong time.
Depending on the error and the situation you have to use this procedure:
-> [http://download.oracle.com/docs/cd/E11882_01/server.112/e10897/software.htm#CHDDBDHC]
<div class="jive-quote">If you stop the upgrade, but do not restore the database, then you should not restart DBUA until you start the database instance in UPGRADE mode using the Oracle Database 11g release 2 (11.2) software. You cannot go back to the original software version unless you restore your database.
</div>
1. How to revert back to the healthy 10.2.0.4 oracle now?
-> Restore a backup of your 10.2.0.4 database with your oracle 10.2.0.4 binaries and recover to point-in-time before upgrade.
2. how to restart and complete the upgrade successfully at this situation.
-> No way - need to revert to the start and try again.
Regards
Stefan
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Thank you so much Stefan, Now i have a question.
1. I have a valid backup before i performed my oracle upgrade to 11.2.0.2, But i dont have the oracle binaries backup. Now my
question is if i just restore my datafiles will it be ok?
2. Do i need to perform any other post steps after the restore?
3. Will the upgrade modify any oracle binaries in the oracle 10.2.0.4 ( /oracle/SID/102_64)
Hemanth
Edited by: Hemanth Kumar on Jul 23, 2011 2:24 PM
Edited by: Hemanth Kumar on Jul 23, 2011 2:26 PM
Hello Hemanth,
I have a valid backup before i performed my oracle upgrade to 11.2.0.2, But i dont have the oracle binaries backup. Now my question is if i just restore my datafiles will it be ok?
-> This depends on your backup type. If you have performed an offline backup just by copying all the data-,redo- and controlfiles to another location - then yes it is sufficient. If you have performed an online backup with an archivelog backup afterwards (with RMAN or BR*Tools) - then no.
Do i need to perform any other post steps after the restore?
-> This also depends on your kind of backup.
3. Will the upgrade modify any oracle binaries in the oracle 10.2.0.4 ( /oracle/SID/102_64)
-> No. The database upgrade "only" modifies the oracle data dictionary and its content. The old ORACLE_HOME will not be modified.
Regards
Stefan
Thank you so much stefan,
So i understood the procedure as below to perform a successful revert back to oracle 10.2.0.4 database. I have checked my backup log file. it is an online backup. I checked the log file and found only one archivelog file listed in my backup file. So what i understood now is, i need to do as follows.
1. make sure there is no ora_ process and any other related process related to SID.
2. delete all the existing data files and redo log files.
3. schedule the restore using tivoli as we use tivoli backup and restore in our landscape.
4. after the restore is done successful will check wether all the files are restored properly or not.
5. start database,Restore the one archive file which is shown in the database backup log file and check R3trans -d and make
sure the connectivity is successful.
6. startsap
Can you please confirm all the above steps are ok so that i can proceed further. Please let me know if i am missing something above.
To perform the upgrade to 11G once again i need to do the following before i start the upgrade once again.
As said in the following link
http://download.oracle.com/docs/cd/E11882_01/server.112/e10897/software.htm#CHDDBDHC
If you restore your database manually (not using DBUA), then remove the following file from the Oracle Database 11g release 2 (11.2) home directory before starting DBUA:
$ Oracle_home/cfgtoollogs/dbua/logs/Welcome_SID.txt
The presence of this file indicates to DBUA that this operation is a reattempt of a previous operation.
so i delete the above said file and perform the necessary environment variables and start the DBUA.
Can you please confirm all the above steps are ok so that i can proceed further. Please let me know if i am missing something above.
Hemanth
Edited by: Hemanth Kumar on Jul 23, 2011 6:13 PM
Hello Hemanth,
5. start database,Restore the one archive file which is shown in the database backup log file and check R3trans -d and make sure the connectivity is successful.
You must restore and apply the one archive log file that was backuped after the online backup. The restore and recover steps need to be performed with the Oracle 10.2.0.4 binaries.
Apart from that the procedure is correct.
Regards
Stefan
Hi Stefan,
Do we need to recreate any pfile and spfile?
1. Hope pfile will not be changed during the upgrade
2. however spfile will be changed with following commands , so i think we need to change the spfile.
Activate Flashback database
Set the following Oracle parameters:
sqlplus "/as sysdba"
alter system set db_recovery_file_dest = "/oracle/stage/oraflash" scope=spfile;
alter system set db_recovery_file_dest_size = 5G scope=spfile;
CREATE RESTORE POINT before_11g_upgrade GUARANTEE FLASHBACK DATABASE;
so before i start the database i might need to create spfile from pfile while the database is down and then start the db and proceed with remaining steps like removing the following line from listener.ora ( ADR_BASE_LISTENER = /oracle/<SID>/saptrace )
and then start the listener, check the R3trans connectivity, if it is 0000 and then start the sap.
am i correct?
Regarding the recovery of archive log file can i use the following procedure?
1. restore the archive log file from the backup by using the command
brrestore u2013a <log number> ( as it is a single archive log file )
2. Identify the time when the archive log file was backedup and give the following command with that time.
sqlplus> startup mount;
sqlplus> recover database using backup controlfile until time u2018yyyy-mm-dd:hh:mm:ssu2019;
u201Cmedia recovery completed.u201D
3. sqlplus> alter database open resetlogs;
sqlplus> alter system switch logfile; (for each online redo logfile)
4. Stop / restart Oracle database
Please advise if i am missing anything.
Hemanth
Please find the last few lines of Oracle server.log.
create or replace public synonym v$rman_backup_job_details for v_$rman_backup_job_details;
Synonym created.
grant select on v_$rman_backup_job_details to select_catalog_role;
Grant succeeded.
create or replace view v_$backup_set_details as select * from v$backup_set_details;
View created.
create or replace public synonym v$backup_set_details for v_$backup_set_details;
Synonym created.
grant select on v_$backup_set_details to select_catalog_role;
Grant succeeded.
create or replace view v_$backup_piece_details as select * from v$backup_piece_details;
View created.
create or replace public synonym v$backup_piece_details for v_$backup_piece_details;
Synonym created.
grant select on v_$backup_piece_details to select_catalog_role;
Grant succeeded.
create or replace view v_$backup_copy_details as select * from v$backup_copy_details;
View created.
create or replace public synonym v$backup_copy_details for v_$backup_copy_details;
Synonym created.
grant select on v_$backup_copy_details to select_catalog_role;
Grant succeeded.
create or replace view v_$proxy_copy_details as select * from v$proxy_copy_details;
View created.
create or replace public synonym v$proxy_copy_details for v_$proxy_copy_details;
Synonym created.
grant select on v_$proxy_copy_details to select_catalog_role;
Grant succeeded.
create or replace view v_$proxy_archivelog_details as select * from v$proxy_archivelog_details;
View created.
create or replace public synonym v$proxy_archivelog_details for v_$proxy_archivelog_details;
Synonym created.
grant select on v_$proxy_archivelog_details to select_catalog_role;
Grant succeeded.
create or replace view v_$backup_datafile_details as select * from v$backup_datafile_details;
View created.
create or replace public synonym v$backup_datafile_details for v_$backup_datafile_details;
Synonym created.
grant select on v_$backup_datafile_details to select_catalog_role;
Grant succeeded.
create or replace view v_$backup_controlfile_details as select * from v$backup_controlfile_details;
View created.
create or replace public synonym v$backup_controlfile_details for v_$backup_controlfile_details;
Synonym created.
grant select on v_$backup_controlfile_details to select_catalog_role;
Grant succeeded.
create or replace view v_$backup_archivelog_details as select * from v$backup_archivelog_details;
View created.
create or replace public synonym v$backup_archivelog_details for v_$backup_archivelog_details;
Synonym created.
grant select on v_$backup_archivelog_details to select_catalog_role;
Grant succeeded.
create or replace view v_$backup_spfile_details as select * from v$backup_spfile_details;
View created.
create or replace public synonym v$backup_spfile_details for v_$backup_spfile_details;
Synonym created.
grant select on v_$backup_spfile_details to select_catalog_role;
Grant succeeded.
create or replace view v_$backup_set_summary as select * from v$backup_set_summary;
View created.
create or replace public synonym v$backup_set_summary for v_$backup_set_summary;
Synonym created.
grant select on v_$backup_set_summary to select_catalog_role;
Grant succeeded.
create or replace view v_$backup_datafile_summary as select * from v$backup_datafile_summary;
View created.
create or replace public synonym v$backup_datafile_summary for v_$backup_datafile_summary;
Synonym created.
grant select on v_$backup_datafile_summary to select_catalog_role;
Grant succeeded.
create or replace view v_$backup_controlfile_summary as select * from v$backup_controlfile_summary;
View created.
create or replace public synonym v$backup_controlfile_summary for v_$backup_controlfile_summary;
Synonym created.
grant select on v_$backup_controlfile_summary to select_catalog_role;
Grant succeeded.
create or replace view v_$backup_archivelog_summary as select * from v$backup_archivelog_summary;
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Please find last few lines of the trace.log file for youre reference.
[Thread-65] [ 2011-07-22 10:43:19.474 CEST ] [CompManager.isModeSet:9689] DB Upgrade Mode Setting := 1
[Thread-65] [ 2011-07-22 10:43:19.474 CEST ] [CompManager.isModeSet:9690] Mode := 1 is set
[Thread-65] [ 2011-07-22 10:43:19.474 CEST ] [CompManager.isModeSet:9689] DB Upgrade Mode Setting := 1
[Thread-65] [ 2011-07-22 10:43:19.474 CEST ] [CompManager.isModeSet:9690] Mode := 4 is not set
[Thread-65] [ 2011-07-22 10:43:19.475 CEST ] [ScriptDirective.processDirective:161] Executing Script:=rdbms/admin/catupgrd.sql
[Thread-65] [ 2011-07-22 10:43:19.475 CEST ] [SQLPlusEngine.executeScript:328] Spool before script run = /oracle/cfgtoollogs/dbua/HR2/upgrade1/Oracle_Server.log
[Thread-65] [ 2011-07-22 10:43:19.475 CEST ] [SQLEngine.done:2167] Done called
[Thread-65] [ 2011-07-22 10:43:19.475 CEST ] [SQLEngine.spoolOff:2013] Setting spool off = /oracle/cfgtoollogs/dbua/HR2/upgrade1/Oracle_Server.log
[Thread-65] [ 2011-07-22 10:43:19.482 CEST ] [SQLEngine.setSpool:1945] old Spool = null
[Thread-65] [ 2011-07-22 10:43:19.482 CEST ] [SQLEngine.setSpool:1946] Setting Spool = /oracle/cfgtoollogs/dbua/HR2/upgrade1/Oracle_Server.log
[Thread-65] [ 2011-07-22 10:43:19.482 CEST ] [SQLEngine.setSpool:1947] Is spool appendable? --> true
[Thread-65] [ 2011-07-22 10:43:19.482 CEST ] [SQLEngine.reInitialize:735] Reinitializing SQLEngine...
[Thread-65] [ 2011-07-22 10:43:19.482 CEST ] [OracleHome.getVersion:885] OracleHome.getVersion called. Current Version: 11.2.0.2.0
[Thread-65] [ 2011-07-22 10:43:19.482 CEST ] [OracleHome.getVersion:953] Current Version From Inventory: 11.2.0.2.0
[Thread-65] [ 2011-07-22 10:43:19.483 CEST ] [OracleHome.getVersion:885] OracleHome.getVersion called. Current Version: 11.2.0.2.0
[Thread-65] [ 2011-07-22 10:43:19.483 CEST ] [OracleHome.getVersion:953] Current Version From Inventory: 11.2.0.2.0
[Thread-65] [ 2011-07-22 10:43:19.483 CEST ] [SQLEngine.getEnvParams:562] NLS_LANG: AMERICAN_AMERICA.WE8DEC
[Thread-65] [ 2011-07-22 10:43:19.484 CEST ] [SQLEngine.initialize:322] Execing SQLPLUS/SVRMGR process...
[Thread-65] [ 2011-07-22 10:43:19.578 CEST ] [SQLEngine.initialize:359] m_bReaderStarted: false
[Thread-65] [ 2011-07-22 10:43:19.579 CEST ] [SQLEngine.initialize:363] Starting Reader Thread...
[Thread-65] [ 2011-07-22 10:43:19.581 CEST ] [SQLEngine.initialize:412] Waiting for m_bReaderStarted to be true
[Thread-65] [ 2011-07-22 10:43:19.595 CEST ] [SQLEngine.initialize:412] Waiting for m_bReaderStarted to be true
[Thread-65] [ 2011-07-22 11:29:26.817 CEST ] [SQLPlusEngine.executeScript:345] Spool after script run = /oracle/cfgtoollogs/dbua/HR2/upgrade1/Oracle_Server.log
[Thread-65] [ 2011-07-22 11:29:26.817 CEST ] [SQLEngine.done:2167] Done called
[Thread-65] [ 2011-07-22 11:29:26.817 CEST ] [SQLEngine.spoolOff:2013] Setting spool off = /oracle/cfgtoollogs/dbua/HR2/upgrade1/Oracle_Server.log
[Thread-65] [ 2011-07-22 11:29:26.818 CEST ] [SQLEngine.setSpool:1945] old Spool = null
[Thread-65] [ 2011-07-22 11:29:26.818 CEST ] [SQLEngine.setSpool:1946] Setting Spool = /oracle/cfgtoollogs/dbua/HR2/upgrade1/Oracle_Server.log
[Thread-65] [ 2011-07-22 11:29:26.818 CEST ] [SQLEngine.setSpool:1947] Is spool appendable? --> true
[Thread-65] [ 2011-07-22 11:29:26.818 CEST ] [SQLEngine.reInitialize:735] Reinitializing SQLEngine...
[Thread-65] [ 2011-07-22 11:29:26.818 CEST ] [OracleHome.getVersion:885] OracleHome.getVersion called. Current Version: 11.2.0.2.0
[Thread-65] [ 2011-07-22 11:29:26.819 CEST ] [OracleHome.getVersion:953] Current Version From Inventory: 11.2.0.2.0
[Thread-65] [ 2011-07-22 11:29:26.819 CEST ] [OracleHome.getVersion:885] OracleHome.getVersion called. Current Version: 11.2.0.2.0
[Thread-65] [ 2011-07-22 11:29:26.819 CEST ] [OracleHome.getVersion:953] Current Version From Inventory: 11.2.0.2.0
[Thread-65] [ 2011-07-22 11:29:26.819 CEST ] [OracleHome.getVersion:885] OracleHome.getVersion called. Current Version: 11.2.0.2.0
[Thread-65] [ 2011-07-22 11:29:26.819 CEST ] [OracleHome.getVersion:953] Current Version From Inventory: 11.2.0.2.0
[Thread-65] [ 2011-07-22 11:29:26.819 CEST ] [SQLPlusEngine.getCmmdParams:222] m_home 11.2.0.2.0
[Thread-65] [ 2011-07-22 11:29:26.819 CEST ] [SQLPlusEngine.getCmmdParams:223] version > 112 true
[Thread-65] [ 2011-07-22 11:29:26.820 CEST ] [SQLEngine.getEnvParams:562] NLS_LANG: AMERICAN_AMERICA.WE8DEC
[Thread-65] [ 2011-07-22 11:29:26.820 CEST ] [SQLEngine.initialize:322] Execing SQLPLUS/SVRMGR process...
[Thread-65] [ 2011-07-22 11:29:26.943 CEST ] [SQLEngine.initialize:359] m_bReaderStarted: false
[Thread-65] [ 2011-07-22 11:29:26.943 CEST ] [SQLEngine.initialize:363] Starting Reader Thread...
[Thread-65] [ 2011-07-22 11:29:26.951 CEST ] [SQLEngine.initialize:412] Waiting for m_bReaderStarted to be true
User | Count |
---|---|
84 | |
10 | |
10 | |
10 | |
7 | |
6 | |
6 | |
5 | |
4 | |
4 |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.