cancel
Showing results for 
Search instead for 
Did you mean: 

How to revert back if Oracle Upgrade from 10.2.0.4 to 11G failed.

Former Member
0 Kudos

Dear Experts,

I am in a process to upgrade oracle 10.2.0.4 to 11G Database upgrade. During the Mid of the upgrade from 10.2.0.4 to 11G the upgrade immediately got terminated. DBUA is in step upgrading oracle server. It was almost all 50% completed. suddenly got terminated unknowingly. I checked the alertSID.log and all other relavant log files. rather i didnt find any root cause for the cause of the termination.

I suspect it might have got terminated due to inactive session security levied by our unix team.

I tried to restart the DBUA once again. I am getting the error ora-00600. I checked the alertSID.log. here is the log file for your reference.

I checked the trace.log. The last few lines for the trace.log, i will post it in the following thread.

I also checked oracleserver.log.

I will post the logfile followed by this thread.

However the current situation is

I tried to recover the oracle binaries with the option FLASH BACK.

1. startup nomount

2. FLASHBACK DATABASE TO RESTORE POINT BEFORE_11G_UPGRADE;

while i give that command I am getting the error end of file of communication channel. when i check the alertSID.log i see ora-600 error. the log files is as follows.

First i checked alertSID.log

Successful mount of redo thread 1, with mount id 3602317500

Fri Jul 22 16:54:24 2011

Allocated 3981120 bytes in shared pool for flashback generation buffer

Starting background process RVWR

RVWR started with pid=13, OS id=7594040

Fri Jul 22 16:54:25 2011

Database mounted in Exclusive Mode

Completed: ALTER DATABASE MOUNT

Fri Jul 22 16:54:29 2011

FLASHBACK DATABASE TO RESTORE POINT BEFORE_11G_UPGRADE

Flashback Restore Start

Fri Jul 22 16:54:35 2011

Errors in file /oracle/HR2/saptrace/background/hr2_dbw0_7782512.trc:

ORA-00600: internal error code, arguments: [999], [0x70000003D7B9FBA], [], [], [], [], [], []

Fri Jul 22 16:54:35 2011

Errors in file /oracle/HR2/saptrace/background/hr2_dbw0_7782512.trc:

ORA-00600: internal error code, arguments: [999], [0x70000003D7B9FBA], [], [], [], [], [], []

Fri Jul 22 16:54:35 2011

DBW0: terminating instance due to error 471

Instance terminated by DBW0, pid = 7782512

I have also checked the file Errors in file /oracle/HR2/saptrace/background/hr2_dbw0_7782512.trc: Here is the logfile data which gives the error.

Dump file /oracle/HR2/saptrace/background/hr2_dbw0_7782512.trc

Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production

With the Partitioning, Data Mining and Real Application Testing options

ORACLE_HOME = /oracle/HR2/102_64

System name: AIX

Node name: kftdeedcbx12

Release: 3

Version: 5

Machine: 00C95BEE4C00

Instance name: HR2

Redo thread mounted by this instance: 1

Oracle process number: 5

Unix process pid: 7782512, image: oracle@kftdeedcbx12 (DBW0)

      • 2011-07-22 16:54:35.137

      • SERVICE NAME:() 2011-07-22 16:54:35.135

      • SESSION ID:(167.1) 2011-07-22 16:54:35.135

Block Checking: DBA = 402665978, Block Type = Undo data block

ERROR: Undo Block Corrupted. Error Code = 2005

ktu4ubck: Undo Record addr did not aligned to boundary

record number=1, addr=3d7b9fba, aligned addr=3d7b9fbc

UNDO BLK HEADER:

xid: 0x0000.01c.000001fa seq: 0x176 cnt: 0x6a irb: 0x6a icl: 0x0 flg: 0x0000

Rec Offset | Rec Offset | Rec Offset | Rec Offset | Rec Offset

-


0x00 0x1fe8 | 0x01 0x1fa6 | 0x02 0x1f54 | 0x03 0x1f12 | 0x04 0x1ec0

0x05 0x1e7e | 0x06 0x1e2c | 0x07 0x1dea | 0x08 0x1d98 | 0x09 0x1d56

0x0a 0x1d04 | 0x0b 0x1cc2 | 0x0c 0x1c70 | 0x0d 0x1c2e | 0x0e 0x1bdc

0x0f 0x1b9a | 0x10 0x1b48 | 0x11 0x1b06 | 0x12 0x1ab4 | 0x13 0x1a72

What can we do at this current situation. I am not that much expert in ORACLE DBA. However know a few things how to manage. Can any of the experts guide me how can we resolve the error at this current sitaution and move forward with the upgrade. What i need to know now is

1. How to revert back to the healthy 10.2.0.4 oracle now?

2. how to restart and complete the upgrade successfully at this situation.

Edited by: Hemanth Kumar on Jul 23, 2011 7:12 AM

Edited by: Hemanth Kumar on Jul 23, 2011 7:16 AM

Accepted Solutions (0)

Answers (2)

Answers (2)

stefan_koehler
Active Contributor
0 Kudos

Hello Hemanth,

well you did the wrong things at the wrong time.

Depending on the error and the situation you have to use this procedure:

-> [http://download.oracle.com/docs/cd/E11882_01/server.112/e10897/software.htm#CHDDBDHC]

<div class="jive-quote">If you stop the upgrade, but do not restore the database, then you should not restart DBUA until you start the database instance in UPGRADE mode using the Oracle Database 11g release 2 (11.2) software. You cannot go back to the original software version unless you restore your database.

</div>

1. How to revert back to the healthy 10.2.0.4 oracle now?

-&gt; Restore a backup of your 10.2.0.4 database with your oracle 10.2.0.4 binaries and recover to point-in-time before upgrade.

2. how to restart and complete the upgrade successfully at this situation.

-&gt; No way - need to revert to the start and try again.

Regards

Stefan

Former Member
0 Kudos

Thank you so much Stefan, Now i have a question.

1. I have a valid backup before i performed my oracle upgrade to 11.2.0.2, But i dont have the oracle binaries backup. Now my

question is if i just restore my datafiles will it be ok?

2. Do i need to perform any other post steps after the restore?

3. Will the upgrade modify any oracle binaries in the oracle 10.2.0.4 ( /oracle/SID/102_64)

Hemanth

Edited by: Hemanth Kumar on Jul 23, 2011 2:24 PM

Edited by: Hemanth Kumar on Jul 23, 2011 2:26 PM

stefan_koehler
Active Contributor
0 Kudos

Hello Hemanth,

I have a valid backup before i performed my oracle upgrade to 11.2.0.2, But i dont have the oracle binaries backup. Now my question is if i just restore my datafiles will it be ok?

-> This depends on your backup type. If you have performed an offline backup just by copying all the data-,redo- and controlfiles to another location - then yes it is sufficient. If you have performed an online backup with an archivelog backup afterwards (with RMAN or BR*Tools) - then no.

Do i need to perform any other post steps after the restore?

-> This also depends on your kind of backup.

3. Will the upgrade modify any oracle binaries in the oracle 10.2.0.4 ( /oracle/SID/102_64)

-> No. The database upgrade "only" modifies the oracle data dictionary and its content. The old ORACLE_HOME will not be modified.

Regards

Stefan

Former Member
0 Kudos

Thank you so much stefan,

So i understood the procedure as below to perform a successful revert back to oracle 10.2.0.4 database. I have checked my backup log file. it is an online backup. I checked the log file and found only one archivelog file listed in my backup file. So what i understood now is, i need to do as follows.

1. make sure there is no ora_ process and any other related process related to SID.

2. delete all the existing data files and redo log files.

3. schedule the restore using tivoli as we use tivoli backup and restore in our landscape.

4. after the restore is done successful will check wether all the files are restored properly or not.

5. start database,Restore the one archive file which is shown in the database backup log file and check R3trans -d and make

sure the connectivity is successful.

6. startsap

Can you please confirm all the above steps are ok so that i can proceed further. Please let me know if i am missing something above.

To perform the upgrade to 11G once again i need to do the following before i start the upgrade once again.

As said in the following link

http://download.oracle.com/docs/cd/E11882_01/server.112/e10897/software.htm#CHDDBDHC

If you restore your database manually (not using DBUA), then remove the following file from the Oracle Database 11g release 2 (11.2) home directory before starting DBUA:

$ Oracle_home/cfgtoollogs/dbua/logs/Welcome_SID.txt

The presence of this file indicates to DBUA that this operation is a reattempt of a previous operation.

so i delete the above said file and perform the necessary environment variables and start the DBUA.

Can you please confirm all the above steps are ok so that i can proceed further. Please let me know if i am missing something above.

Hemanth

Edited by: Hemanth Kumar on Jul 23, 2011 6:13 PM

stefan_koehler
Active Contributor
0 Kudos

Hello Hemanth,

5. start database,Restore the one archive file which is shown in the database backup log file and check R3trans -d and make sure the connectivity is successful.

You must restore and apply the one archive log file that was backuped after the online backup. The restore and recover steps need to be performed with the Oracle 10.2.0.4 binaries.

Apart from that the procedure is correct.

Regards

Stefan

Former Member
0 Kudos

Hi Stefan,

Do we need to recreate any pfile and spfile?

1. Hope pfile will not be changed during the upgrade

2. however spfile will be changed with following commands , so i think we need to change the spfile.

Activate Flashback database

Set the following Oracle parameters:

sqlplus "/as sysdba"

alter system set db_recovery_file_dest = "/oracle/stage/oraflash" scope=spfile;

alter system set db_recovery_file_dest_size = 5G scope=spfile;

CREATE RESTORE POINT before_11g_upgrade GUARANTEE FLASHBACK DATABASE;

so before i start the database i might need to create spfile from pfile while the database is down and then start the db and proceed with remaining steps like removing the following line from listener.ora ( ADR_BASE_LISTENER = /oracle/<SID>/saptrace )

and then start the listener, check the R3trans connectivity, if it is 0000 and then start the sap.

am i correct?

Regarding the recovery of archive log file can i use the following procedure?

1. restore the archive log file from the backup by using the command

brrestore u2013a <log number> ( as it is a single archive log file )

2. Identify the time when the archive log file was backedup and give the following command with that time.

sqlplus> startup mount;

sqlplus> recover database using backup controlfile until time u2018yyyy-mm-dd:hh:mm:ssu2019;

u201Cmedia recovery completed.u201D

3. sqlplus> alter database open resetlogs;

sqlplus> alter system switch logfile; (for each online redo logfile)

4. Stop / restart Oracle database

Please advise if i am missing anything.

Hemanth

Former Member
0 Kudos

Hi Stefan,

Thank you so much for your greate help. I was successfully able to restore back and bring my database with 10.2.0.4 again. Now i just want to find out what i would have did to bring the dbua successfully and complete the upgrade soon after the upgrade is complete.

Hemanth

Former Member
0 Kudos

Please find the last few lines of Oracle server.log.

create or replace public synonym v$rman_backup_job_details for v_$rman_backup_job_details;

Synonym created.

grant select on v_$rman_backup_job_details to select_catalog_role;

Grant succeeded.

create or replace view v_$backup_set_details as select * from v$backup_set_details;

View created.

create or replace public synonym v$backup_set_details for v_$backup_set_details;

Synonym created.

grant select on v_$backup_set_details to select_catalog_role;

Grant succeeded.

create or replace view v_$backup_piece_details as select * from v$backup_piece_details;

View created.

create or replace public synonym v$backup_piece_details for v_$backup_piece_details;

Synonym created.

grant select on v_$backup_piece_details to select_catalog_role;

Grant succeeded.

create or replace view v_$backup_copy_details as select * from v$backup_copy_details;

View created.

create or replace public synonym v$backup_copy_details for v_$backup_copy_details;

Synonym created.

grant select on v_$backup_copy_details to select_catalog_role;

Grant succeeded.

create or replace view v_$proxy_copy_details as select * from v$proxy_copy_details;

View created.

create or replace public synonym v$proxy_copy_details for v_$proxy_copy_details;

Synonym created.

grant select on v_$proxy_copy_details to select_catalog_role;

Grant succeeded.

create or replace view v_$proxy_archivelog_details as select * from v$proxy_archivelog_details;

View created.

create or replace public synonym v$proxy_archivelog_details for v_$proxy_archivelog_details;

Synonym created.

grant select on v_$proxy_archivelog_details to select_catalog_role;

Grant succeeded.

create or replace view v_$backup_datafile_details as select * from v$backup_datafile_details;

View created.

create or replace public synonym v$backup_datafile_details for v_$backup_datafile_details;

Synonym created.

grant select on v_$backup_datafile_details to select_catalog_role;

Grant succeeded.

create or replace view v_$backup_controlfile_details as select * from v$backup_controlfile_details;

View created.

create or replace public synonym v$backup_controlfile_details for v_$backup_controlfile_details;

Synonym created.

grant select on v_$backup_controlfile_details to select_catalog_role;

Grant succeeded.

create or replace view v_$backup_archivelog_details as select * from v$backup_archivelog_details;

View created.

create or replace public synonym v$backup_archivelog_details for v_$backup_archivelog_details;

Synonym created.

grant select on v_$backup_archivelog_details to select_catalog_role;

Grant succeeded.

create or replace view v_$backup_spfile_details as select * from v$backup_spfile_details;

View created.

create or replace public synonym v$backup_spfile_details for v_$backup_spfile_details;

Synonym created.

grant select on v_$backup_spfile_details to select_catalog_role;

Grant succeeded.

create or replace view v_$backup_set_summary as select * from v$backup_set_summary;

View created.

create or replace public synonym v$backup_set_summary for v_$backup_set_summary;

Synonym created.

grant select on v_$backup_set_summary to select_catalog_role;

Grant succeeded.

create or replace view v_$backup_datafile_summary as select * from v$backup_datafile_summary;

View created.

create or replace public synonym v$backup_datafile_summary for v_$backup_datafile_summary;

Synonym created.

grant select on v_$backup_datafile_summary to select_catalog_role;

Grant succeeded.

create or replace view v_$backup_controlfile_summary as select * from v$backup_controlfile_summary;

View created.

create or replace public synonym v$backup_controlfile_summary for v_$backup_controlfile_summary;

Synonym created.

grant select on v_$backup_controlfile_summary to select_catalog_role;

Grant succeeded.

create or replace view v_$backup_archivelog_summary as select * from v$backup_archivelog_summary;

Former Member
0 Kudos

Please find last few lines of the trace.log file for youre reference.

[Thread-65] [ 2011-07-22 10:43:19.474 CEST ] [CompManager.isModeSet:9689] DB Upgrade Mode Setting := 1

[Thread-65] [ 2011-07-22 10:43:19.474 CEST ] [CompManager.isModeSet:9690] Mode := 1 is set

[Thread-65] [ 2011-07-22 10:43:19.474 CEST ] [CompManager.isModeSet:9689] DB Upgrade Mode Setting := 1

[Thread-65] [ 2011-07-22 10:43:19.474 CEST ] [CompManager.isModeSet:9690] Mode := 4 is not set

[Thread-65] [ 2011-07-22 10:43:19.475 CEST ] [ScriptDirective.processDirective:161] Executing Script:=rdbms/admin/catupgrd.sql

[Thread-65] [ 2011-07-22 10:43:19.475 CEST ] [SQLPlusEngine.executeScript:328] Spool before script run = /oracle/cfgtoollogs/dbua/HR2/upgrade1/Oracle_Server.log

[Thread-65] [ 2011-07-22 10:43:19.475 CEST ] [SQLEngine.done:2167] Done called

[Thread-65] [ 2011-07-22 10:43:19.475 CEST ] [SQLEngine.spoolOff:2013] Setting spool off = /oracle/cfgtoollogs/dbua/HR2/upgrade1/Oracle_Server.log

[Thread-65] [ 2011-07-22 10:43:19.482 CEST ] [SQLEngine.setSpool:1945] old Spool = null

[Thread-65] [ 2011-07-22 10:43:19.482 CEST ] [SQLEngine.setSpool:1946] Setting Spool = /oracle/cfgtoollogs/dbua/HR2/upgrade1/Oracle_Server.log

[Thread-65] [ 2011-07-22 10:43:19.482 CEST ] [SQLEngine.setSpool:1947] Is spool appendable? --> true

[Thread-65] [ 2011-07-22 10:43:19.482 CEST ] [SQLEngine.reInitialize:735] Reinitializing SQLEngine...

[Thread-65] [ 2011-07-22 10:43:19.482 CEST ] [OracleHome.getVersion:885] OracleHome.getVersion called. Current Version: 11.2.0.2.0

[Thread-65] [ 2011-07-22 10:43:19.482 CEST ] [OracleHome.getVersion:953] Current Version From Inventory: 11.2.0.2.0

[Thread-65] [ 2011-07-22 10:43:19.483 CEST ] [OracleHome.getVersion:885] OracleHome.getVersion called. Current Version: 11.2.0.2.0

[Thread-65] [ 2011-07-22 10:43:19.483 CEST ] [OracleHome.getVersion:953] Current Version From Inventory: 11.2.0.2.0

[Thread-65] [ 2011-07-22 10:43:19.483 CEST ] [SQLEngine.getEnvParams:562] NLS_LANG: AMERICAN_AMERICA.WE8DEC

[Thread-65] [ 2011-07-22 10:43:19.484 CEST ] [SQLEngine.initialize:322] Execing SQLPLUS/SVRMGR process...

[Thread-65] [ 2011-07-22 10:43:19.578 CEST ] [SQLEngine.initialize:359] m_bReaderStarted: false

[Thread-65] [ 2011-07-22 10:43:19.579 CEST ] [SQLEngine.initialize:363] Starting Reader Thread...

[Thread-65] [ 2011-07-22 10:43:19.581 CEST ] [SQLEngine.initialize:412] Waiting for m_bReaderStarted to be true

[Thread-65] [ 2011-07-22 10:43:19.595 CEST ] [SQLEngine.initialize:412] Waiting for m_bReaderStarted to be true

[Thread-65] [ 2011-07-22 11:29:26.817 CEST ] [SQLPlusEngine.executeScript:345] Spool after script run = /oracle/cfgtoollogs/dbua/HR2/upgrade1/Oracle_Server.log

[Thread-65] [ 2011-07-22 11:29:26.817 CEST ] [SQLEngine.done:2167] Done called

[Thread-65] [ 2011-07-22 11:29:26.817 CEST ] [SQLEngine.spoolOff:2013] Setting spool off = /oracle/cfgtoollogs/dbua/HR2/upgrade1/Oracle_Server.log

[Thread-65] [ 2011-07-22 11:29:26.818 CEST ] [SQLEngine.setSpool:1945] old Spool = null

[Thread-65] [ 2011-07-22 11:29:26.818 CEST ] [SQLEngine.setSpool:1946] Setting Spool = /oracle/cfgtoollogs/dbua/HR2/upgrade1/Oracle_Server.log

[Thread-65] [ 2011-07-22 11:29:26.818 CEST ] [SQLEngine.setSpool:1947] Is spool appendable? --> true

[Thread-65] [ 2011-07-22 11:29:26.818 CEST ] [SQLEngine.reInitialize:735] Reinitializing SQLEngine...

[Thread-65] [ 2011-07-22 11:29:26.818 CEST ] [OracleHome.getVersion:885] OracleHome.getVersion called. Current Version: 11.2.0.2.0

[Thread-65] [ 2011-07-22 11:29:26.819 CEST ] [OracleHome.getVersion:953] Current Version From Inventory: 11.2.0.2.0

[Thread-65] [ 2011-07-22 11:29:26.819 CEST ] [OracleHome.getVersion:885] OracleHome.getVersion called. Current Version: 11.2.0.2.0

[Thread-65] [ 2011-07-22 11:29:26.819 CEST ] [OracleHome.getVersion:953] Current Version From Inventory: 11.2.0.2.0

[Thread-65] [ 2011-07-22 11:29:26.819 CEST ] [OracleHome.getVersion:885] OracleHome.getVersion called. Current Version: 11.2.0.2.0

[Thread-65] [ 2011-07-22 11:29:26.819 CEST ] [OracleHome.getVersion:953] Current Version From Inventory: 11.2.0.2.0

[Thread-65] [ 2011-07-22 11:29:26.819 CEST ] [SQLPlusEngine.getCmmdParams:222] m_home 11.2.0.2.0

[Thread-65] [ 2011-07-22 11:29:26.819 CEST ] [SQLPlusEngine.getCmmdParams:223] version > 112 true

[Thread-65] [ 2011-07-22 11:29:26.820 CEST ] [SQLEngine.getEnvParams:562] NLS_LANG: AMERICAN_AMERICA.WE8DEC

[Thread-65] [ 2011-07-22 11:29:26.820 CEST ] [SQLEngine.initialize:322] Execing SQLPLUS/SVRMGR process...

[Thread-65] [ 2011-07-22 11:29:26.943 CEST ] [SQLEngine.initialize:359] m_bReaderStarted: false

[Thread-65] [ 2011-07-22 11:29:26.943 CEST ] [SQLEngine.initialize:363] Starting Reader Thread...

[Thread-65] [ 2011-07-22 11:29:26.951 CEST ] [SQLEngine.initialize:412] Waiting for m_bReaderStarted to be true