cancel
Showing results for 
Search instead for 
Did you mean: 

Deleted /oracle/<sid> files

Former Member
0 Kudos

Hi Folks, on the unix shell in a rush I gave the bloody command

find . -type f -mtime +15 -exec rm -f {} \;

I Obviously deleted some files, now that I have done that.

Environment: UNIX, Oracle 10.2, PI server (ABAP+JAVA) stack.

I have backup from 2 months back, I need help in what to do now.

Any help is appreciated

Thanks

Accepted Solutions (0)

Answers (19)

Answers (19)

Former Member
0 Kudos

HI Folks, as requested I am attaching the last 300 lines of the alert_SID.log file for you to see:

---------------------------------------

Output of the alert_<SID>.log File

SMON started with pid=8, OS id=17649
RECO started with pid=9, OS id=17651
CJQ0 started with pid=10, OS id=17653
MMON started with pid=11, OS id=17655
MMNL started with pid=12, OS id=17657
Wed Apr 09 17:53:53 IST 2014
ALTER DATABASE   MOUNT
Wed Apr 09 17:53:57 IST 2014
Setting recovery target incarnation to 1
Wed Apr 09 17:53:57 IST 2014
Successful mount of redo thread 1, with mount id 76606833
Wed Apr 09 17:53:57 IST 2014
Database mounted in Exclusive Mode
Completed: ALTER DATABASE   MOUNT
Wed Apr 09 17:53:57 IST 2014
ALTER DATABASE OPEN
Wed Apr 09 17:53:57 IST 2014
Errors in file /oracle/<SID>/saptrace/background/<SID>_dbw0_17643.trc:
ORA-01157: cannot identify/lock data file 20 - see DBWR trace file
ORA-01110: data file 20: '/oracle/<SID>/sapdata2/sr3711_7/sr3711.data7'
ORA-27046: file size is not a multiple of logical block size
Additional information: 1
ORA-1157 signalled during: ALTER DATABASE OPEN...
Wed Apr 09 18:02:57 IST 2014
Shutting down instance (immediate)
Wed Apr 09 18:02:57 IST 2014
Shutting down instance: further logons disabled
Wed Apr 09 18:02:57 IST 2014
Stopping background process MMNL
Wed Apr 09 18:02:57 IST 2014
Stopping background process CJQ0
Wed Apr 09 18:02:58 IST 2014
Stopping background process MMON
License high water mark = 1
Wed Apr 09 18:02:59 IST 2014
Job queue slave processes stopped
Wed Apr 09 18:02:59 IST 2014
ALTER DATABASE CLOSE NORMAL
ORA-1109 signalled during: ALTER DATABASE CLOSE NORMAL...
Wed Apr 09 18:02:59 IST 2014
ALTER DATABASE DISMOUNT
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Wed Apr 09 18:03:01 IST 2014
Starting ORACLE instance (normal)
Wed Apr 09 18:03:01 IST 2014
Specified value of sga_max_size is too small, bumping to 1795162112
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.5.0.
System parameters with non-default values:
  processes                = 80
  sessions                 = 160
  event                    = 10191 trace name context forever, level 1
  sga_max_size             = 1795162112
  shared_pool_size         = 872415232
  shared_pool_reserved_size= 86888153
  filesystemio_options     = setall
  control_files            = /oracle/<SID>/origlogA/cntrl/cntrl<SID>.dbf, /oracle/<SID>/origlogB/cntrl/cntrl<SID>.dbf, /oracle/<SID>/sapdata1/cntrl/cntrl<SID>.dbf
  control_file_record_keep_time= 30
  db_block_size            = 8192
  db_cache_size            = 872415232
  compatible               = 10.2.0
  log_archive_dest         = /oracle/<SID>/oraarch/<SID>arch
  log_buffer               = 14477312
  log_checkpoints_to_alert = TRUE
  dml_locks                = 4000
  replication_dependency_tracking= FALSE
  undo_management          = AUTO
  undo_tablespace          = PSAPUNDO
  _in_memory_undo          = FALSE
  undo_retention           = 43200
  recyclebin               = off
  remote_os_authent        = TRUE
  remote_login_passwordfile= EXCLUSIVE
  job_queue_processes      = 1
  parallel_execution_message_size= 16384
  _table_lookup_prefetch_size= 0
  background_dump_dest     = /oracle/<SID>/saptrace/background
  user_dump_dest           = /oracle/<SID>/saptrace/usertrace
  core_dump_dest           = /oracle/<SID>/saptrace/background
  db_name                  = <SID>
  open_cursors             = 2000
  _sort_elimination_cost_ratio= 10
  _b_tree_bitmap_plans     = FALSE
  star_transformation_enabled= true
  query_rewrite_enabled    = false
  _index_join_enabled      = FALSE
  _optim_peek_user_binds   = FALSE
  pga_aggregate_target     = 1158508707
  _optimizer_mjc_enabled   = FALSE
PMON started with pid=2, OS id=18357
PSP0 started with pid=3, OS id=18359
MMAN started with pid=4, OS id=18361
DBW0 started with pid=5, OS id=18363
LGWR started with pid=6, OS id=18365
CKPT started with pid=7, OS id=18367
SMON started with pid=8, OS id=18369
RECO started with pid=9, OS id=18371
CJQ0 started with pid=10, OS id=18373
MMON started with pid=11, OS id=18375
MMNL started with pid=12, OS id=18377
Wed Apr 09 18:03:02 IST 2014
ALTER DATABASE   MOUNT
Wed Apr 09 18:03:06 IST 2014
Setting recovery target incarnation to 1
Wed Apr 09 18:03:06 IST 2014
Successful mount of redo thread 1, with mount id 76548758
Wed Apr 09 18:03:06 IST 2014
Database mounted in Exclusive Mode
Completed: ALTER DATABASE   MOUNT
Wed Apr 09 18:03:06 IST 2014
ALTER DATABASE OPEN
Wed Apr 09 18:03:06 IST 2014
Errors in file /oracle/<SID>/saptrace/background/<SID>_dbw0_18363.trc:
ORA-01157: cannot identify/lock data file 20 - see DBWR trace file
ORA-01110: data file 20: '/oracle/<SID>/sapdata2/sr3711_7/sr3711.data7'
ORA-27046: file size is not a multiple of logical block size
Additional information: 1
ORA-1157 signalled during: ALTER DATABASE OPEN...
Wed Apr 09 18:04:30 IST 2014
Shutting down instance (immediate)
Wed Apr 09 18:04:30 IST 2014
Shutting down instance: further logons disabled
Wed Apr 09 18:04:30 IST 2014
Stopping background process MMNL
Wed Apr 09 18:04:30 IST 2014
Stopping background process CJQ0
Wed Apr 09 18:04:31 IST 2014
Stopping background process MMON
License high water mark = 1
Wed Apr 09 18:04:33 IST 2014
Job queue slave processes stopped
Wed Apr 09 18:04:33 IST 2014
ALTER DATABASE CLOSE NORMAL
ORA-1109 signalled during: ALTER DATABASE CLOSE NORMAL...
Wed Apr 09 18:04:33 IST 2014
ALTER DATABASE DISMOUNT
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Wed Apr 09 18:04:40 IST 2014
Starting ORACLE instance (normal)
Wed Apr 09 18:04:40 IST 2014
Specified value of sga_max_size is too small, bumping to 1795162112
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.5.0.
System parameters with non-default values:
  processes                = 80
  sessions                 = 160
  event                    = 10191 trace name context forever, level 1
  sga_max_size             = 1795162112
  shared_pool_size         = 872415232
  shared_pool_reserved_size= 86888153
  filesystemio_options     = setall
  control_files            = /oracle/<SID>/origlogA/cntrl/cntrl<SID>.dbf, /oracle/<SID>/origlogB/cntrl/cntrl<SID>.dbf, /oracle/<SID>/sapdata1/cntrl/cntrl<SID>.dbf
  control_file_record_keep_time= 30
  db_block_size            = 8192
  db_cache_size            = 872415232
  compatible               = 10.2.0
  log_archive_dest         = /oracle/<SID>/oraarch/<SID>arch
  log_buffer               = 14477312
  log_checkpoints_to_alert = TRUE
  dml_locks                = 4000
  replication_dependency_tracking= FALSE
  undo_management          = AUTO
  undo_tablespace          = PSAPUNDO
  _in_memory_undo          = FALSE
  undo_retention           = 43200
  recyclebin               = off
  remote_os_authent        = TRUE
  remote_login_passwordfile= EXCLUSIVE
  job_queue_processes      = 1
  parallel_execution_message_size= 16384
  _table_lookup_prefetch_size= 0
  background_dump_dest     = /oracle/<SID>/saptrace/background
  user_dump_dest           = /oracle/<SID>/saptrace/usertrace
  core_dump_dest           = /oracle/<SID>/saptrace/background
  db_name                  = <SID>
  open_cursors             = 2000
  _sort_elimination_cost_ratio= 10
  _b_tree_bitmap_plans     = FALSE
  star_transformation_enabled= true
  query_rewrite_enabled    = false
  _index_join_enabled      = FALSE
  _optim_peek_user_binds   = FALSE
  pga_aggregate_target     = 1158508707
  _optimizer_mjc_enabled   = FALSE
PMON started with pid=2, OS id=18599
PSP0 started with pid=3, OS id=18601
MMAN started with pid=4, OS id=18603
DBW0 started with pid=5, OS id=18605
LGWR started with pid=6, OS id=18607
CKPT started with pid=7, OS id=18609
SMON started with pid=8, OS id=18611
RECO started with pid=9, OS id=18613
CJQ0 started with pid=10, OS id=18615
MMON started with pid=11, OS id=18617
MMNL started with pid=12, OS id=18619
Wed Apr 09 18:04:41 IST 2014
ALTER DATABASE   MOUNT
Wed Apr 09 18:04:45 IST 2014
Setting recovery target incarnation to 1
Wed Apr 09 18:04:45 IST 2014
Successful mount of redo thread 1, with mount id 76579065
Wed Apr 09 18:04:45 IST 2014
Database mounted in Exclusive Mode
Completed: ALTER DATABASE   MOUNT
Wed Apr 09 18:05:13 IST 2014
alter database archivelog
Wed Apr 09 18:05:13 IST 2014
Completed: alter database archivelog
Wed Apr 09 18:05:20 IST 2014
alter database open
Wed Apr 09 18:05:20 IST 2014
Errors in file /oracle/<SID>/saptrace/background/<SID>_dbw0_18605.trc:
ORA-01157: cannot identify/lock data file 20 - see DBWR trace file
ORA-01110: data file 20: '/oracle/<SID>/sapdata2/sr3711_7/sr3711.data7'
ORA-27046: file size is not a multiple of logical block size
Additional information: 1
ORA-1157 signalled during: alter database open...

I think the database is getting mounted but not opening because I tried to turn on archive log manually using sqlplus, everything worked fine till the last command of "alter database open"

Thanks

Aps

former_member182657
Active Contributor
0 Kudos

Hi Aps,


ORA-27046: file size is not a multiple of logical block size

This error indicates file size mismatch,as it could be by usage of current controlfile to start database restored with 2 months back data.(Mismatch b/w controlfile & data files size).Here i could be wrong but it may be the possibility.

I suggest you try to compare the size for faulty file sr3711_7/sr3711.data7 with the correct one.If any difference restore this single file in place and follow the previous procedure to open database. Also suggesting a workaround for your last ref. ORA-27046 file size is not a multiple of logical block size | The Gruff DBA

Regards,

Gaurav

former_member182657
Active Contributor
0 Kudos

Hi Aps,

Could you please share alert_SID.log file also.

Regards,

Gaurav

Former Member
0 Kudos

Hi RB

as you suggested now the Error has changed, with the oracle intialisation in progress

trans.log

-------------------------

R3trans -d

This is R3trans version 6.19 (release 711 - 26.10.11 - 13:00:00).

unicode enabled version

2EETW169 no connect possible: "DBMS = ORACLE                           --- dbs_ora_tnsname = '<SID>'"

R3trans finished (0012).

<hostname>:<SID>adm 22> more trans.log

4 ETW000 R3trans version 6.19 (release 711 - 26.10.11 - 13:00:00).

4 ETW000 unicode enabled version

4 ETW000 ===============================================

4 ETW000

4 ETW000 date&time   : 09.04.2014 - 16:57:13

4 ETW000 control file: <no ctrlfile>

4 ETW000 R3trans was called as follows: R3trans -d

4 ETW000  trace at level 2 opened for a given file pointer

4 ETW000  [     dev trc,00000]  Wed Apr  9 16:57:13 2014                                                 573  0.000573

4 ETW000  [     dev trc,00000]  db_con_init called                                                        55  0.000628

4 ETW000  [     dev trc,00000]  determine_block_commit: no con_hdl found as blocked for con_name = R/3

4 ETW000                                                                                                 168  0.000796

4 ETW000  [     dev trc,00000]  create_con (con_name=R/3)                                                 71  0.000867

4 ETW000  [     dev trc,00000]  Loading DB library '/usr/sap/<SID>/SYS/exe/uc/hpia64/dboraslib.so' ...

4 ETW000                                                                                                 156  0.001023

4 ETW000  [     dev trc,00000]  DlLoadLib() success: dlopen("/usr/sap/<SID>/SYS/exe/uc/hpia64/dboraslib.so"), hdl 0

4 ETW000                                                                                               47825  0.048848

4 ETW000  [     dev trc,00000]  Library '/usr/sap/<SID>/SYS/exe/uc/hpia64/dboraslib.so' loaded              66  0.048914

4 ETW000  [     dev trc,00000]  function DbSlExpFuns loaded from library /usr/sap/<SID>/SYS/exe/uc/hpia64/dboraslib.so

4 ETW000                                                                                                 173  0.049087

4 ETW000  [     dev trc,00000]  Version of '/usr/sap/<SID>/SYS/exe/uc/hpia64/dboraslib.so' is "710.21", patchlevel (0.146)

4 ETW000                                                                                                 492  0.049579

4 ETW000  [     dev trc,00000]  function dsql_db_init loaded from library /usr/sap/<SID>/SYS/exe/uc/hpia64/dboraslib.so

4 ETW000                                                                                                  89  0.049668

4 ETW000  [     dev trc,00000]  function dbdd_exp_funs loaded from library /usr/sap/<SID>/SYS/exe/uc/hpia64/dboraslib.so

4 ETW000                                                                                                 291  0.049959

4 ETW000  [     dev trc,00000]  New connection 0 created                                                 131  0.050090

4 ETW000  [     dev trc,00000]  0: name = R/3, con_id = -000000001, state = DISCONNECTED, tx = NO , bc = NO , hc = NO , perm = YES, reco = NO , frco = NO , timeout = 00

0, con_max = 255, con_opt = 255, occ = NO , prog =

4 ETW000                                                                                                  99  0.050189

4 ETW000  [     dev trc,00000]  db_con_connect (con_name=R/3)                                             47  0.050236

4 ETW000  [     dev trc,00000]  determine_block_commit: no con_hdl found as blocked for con_name = R/3

4 ETW000                                                                                                  69  0.050305

4 ETW000  [     dev trc,00000]  find_con_by_name found the following connection:                          44  0.050349

4 ETW000  [     dev trc,00000]  0: name = R/3, con_id = 000000000, state = DISCONNECTED, tx = NO , bc = NO , hc = NO , perm = YES, reco = NO , frco = NO , timeout = 000

, con_max = 255, con_opt = 255, occ = NO , prog =

4 ETW000                                                                                                 254  0.050603

4 ETW000  [     dev trc,00000]  -->DbSlConnect(vers=-1, type=0, env='[default env]', userid='[default usr]')

4 ETW000                                                                                                 294  0.050897

4 ETW000  [     dev trc,00000]  Register application info.                                               763  0.051660

4 ETW000  [     dev trc,00000]  Oracle client version: 10.2.0.1.0, (dbsl 711 131011, UNICODE[2])         181  0.051841

4 ETW000  [     dev trc,00000]    Default connection: tnsname ='<SID>'                                     101  0.051942

4 ETW000  [     dev trc,00000]  -->oci_initialize(con=0, char='UTF16', nchar='AL16UTF16', nls=-1),uc_ln=2

4 ETW000                                                                                                 122  0.052064

4 ETW000  [     dev trc,00000]  OCIEnvCreate(mode=16384=OCI_UTF16) returned 0 (for default NLS handle)

4 ETW000                                                                                               14429  0.066493

4 ETW000  [     dev trc,00000]     OCIHandleAlloc(OCI_HTYPE_ERROR, con=0, ErrHp=60000000022455b0)        242  0.066735

4 ETW000  [     dev trc,00000]     OCIHandleAlloc(OCI_HTYPE_ERROR, con=0, ErrBt=6000000002256168)         64  0.066799

4 ETW000  [     dev trc,00000]  NLS env. settings: lang=0041004D00450052004900430041004E                 707  0.067506

4 ETW000  [     dev trc,00000]                     terr=0041004D00450052004900430041                      77  0.067583

4 ETW000  [     dev trc,00000]                     char=00550054004600310036                              64  0.067647

4 ETW000  [     dev trc,00000]  Client NLS setting (by OCINlsGetInfo(con=0)): 'AMERICAN_AMERICA.UTF16'

4 ETW000                                                                                                  76  0.067723

4 ETW000  [     dev trc,00000]  charset='UTF16', ncharset='AL16UTF16', UNI_ASC=FALSE                      52  0.067775

4 ETW000  [     dev trc,00000]  Logon as OPS$-user to get SAPSR3's password                              144  0.067919

4 ETW000  [     dev trc,00000]  Connecting as /@<SID> on connection 0 (nls 0) ... (dbsl 711 131011, UNICODE[2])

4 ETW000                                                                                                  84  0.068003

4 ETW000  [     dev trc,00000]     OCIHandleAlloc(OCI_HTYPE_SVCCTX, con=0, svchp=6000000002256098)       135  0.068138

4 ETW000  [     dev trc,00000]  Allocating server context handle                                          56  0.068194

4 ETW000  [     dev trc,00000]  Attaching to database server <SID> (con=0, svchp=6000000002256098, srvhp=60000000022598e8)

4 ETW000                                                                                                  90  0.068284

4 ETW000  [     dev trc,00000]  Assigning server context 60000000022598e8 to service context 6000000002256098

4 ETW000                                                                                               83670  0.151954

4 ETW000  [     dev trc,00000]     OCIHandleAlloc(OCI_HTYPE_SESSION, con=0, usrhp=6000000002245dc8)      311  0.152265

4 ETW000  [     dev trc,00000]  Starting user session: OCISessionBegin(con=0, usr='/', svc=6000000002256098, srv=60000000022598e8, usr=6000000002245dc8)

4 ETW000                                                                                                  86  0.152351

4 ETW000  [     dev trc,00000]     OCISessionBegin(OCI_DEFAULT) failed with -1=OCI_ERROR               14410  0.166761

4 ETW000  [     dev trc,00000]     OCISessionBegin(OCI_DEFAULT) failed with SQL error 1033:              389  0.167150

4 ETW000  [     dev trc,00000]     ORA-01033: ORACLE initialization or shutdown in progress

4 ETW000                                                                                                 289  0.167439

4 ETW000  [     dev trc,00000]  server_detach(con=0,stale=1,srvhp=60000000022598e8)                       57  0.167496

4 ETW000  [     dev trc,00000]  Server handle (con=0,svchp=6000000002256098,srvhp=60000000022598e8) detached

4 ETW000                                                                                                5009  0.172505

4 ETW000  [     dev trc,00000]  Free server handle 60000000022598e8 of con=0                              54  0.172559

4 ETW000  [    dbsloci.,00000]  *** ERROR => CONNECT failed with sql error '1033'                        181  0.172740

4 ETW000  [     dev trc,00000]     set_ocica() -> SQL error code 1033                                    164  0.172904

4 ETW000  [     dev trc,00000]  -->oci_get_errmsg (con=0, rc=1033)                                        58  0.172962

4 ETW000  [     dev trc,00000]     OCIErrorGet -> SQL error code: 1033                                   161  0.173123

4 ETW000  [     dev trc,00000]     ORA-01033: ORACLE initialization or shutdown in progress

4 ETW000                                                                                                  75  0.173198

4 ETW000  [     dev trc,00000]  SAPUSER error or connect error 99=DBSL_ERR_DB, oerr=1033.                 54  0.173252

4 ETW000  [     dev trc,00000]     DbSlConnect(con=0) -> orc=0, 99=DBSL_ERR_DB                            55  0.173307

4 ETW000  [    dblink  ,00000]  ***LOG BY2=>sql error 1033   performing CON                              404  0.173711

4 ETW000  [    dblink  ,00000]  ***LOG BY0=>ORA-01033: ORACLE initialization or shutdown in progress

4 ETW000                                                                                                  84  0.173795

2EETW169 no connect possible: "DBMS = ORACLE                           --- dbs_ora_tnsname = '<SID>'"

-------------------------

Reagan
Product and Topic Expert
Product and Topic Expert
0 Kudos

4 ETW000  [     dev trc,00000]     OCISessionBegin(OCI_DEFAULT) failed with -1=OCI_ERROR               14410  0.166761

4 ETW000  [     dev trc,00000]     OCISessionBegin(OCI_DEFAULT) failed with SQL error 1033:              389  0.167150

4 ETW000  [     dev trc,00000]     ORA-01033: ORACLE initialization or shutdown in progress

Now the error is not related to the listener.

su - orasid

sqlplus "/as sysdba"

select name, open_mode from v$database;

If the database is down then start the database.

Regards

RB

Former Member
0 Kudos

Hi RB

I did what u said but this is what I get when I run "startup"

-------------------

SQL> startup
ORACLE instance started.

Total System Global Area 1795162112 bytes
Fixed Size                  2068648 bytes
Variable Size             905973592 bytes
Database Buffers          872415232 bytes
Redo Buffers               14704640 bytes
Database mounted.
ORA-01157: cannot identify/lock data file 20 - see DBWR trace file
ORA-01110: data file 20: '/oracle/<SID>/sapdata2/sr3711_7/sr3711.data7'

----------------------------------------------

former_member182307
Contributor
0 Kudos

Hello,

This means the datafile /oracle/<SID>/sapdata2/sr3711_7/sr3711.data7 does not exist, or the RDBMS user does not have the neede ACL to access it.

It should be a file whose proprietary is orasid:sapsys.

You can set the ACL to 664 on the datafiles.

Can you :

- check you restored the datafiles in the reight location.

- check the access rights on the files ?

Best regards,

Steve.

Reagan
Product and Topic Expert
Product and Topic Expert
0 Kudos

ORA-01157: cannot identify/lock data file 20 - see DBWR trace file

ORA-01110: data file 20: '/oracle/<SID>/sapdata2/sr3711_7/sr3711.data7'

Check the Oracle alert log to find out more information.

Is this datafile present at the OS level ?

Regards

RB

Former Member
0 Kudos

HI RB

checked the file and found out that the owner of this file sr3711_6  was root so I changed the owner and the group but the error still persistes

Aps

Reagan
Product and Topic Expert
Product and Topic Expert
0 Kudos

Change the ownership of all the datafiles to orasid:dba

chown -R orasid:dba /oracle/<SID>/sapdata*

Once done restart the database.

I am sure the error is not the same if you have changed the ownsership of the datafile

Regards

RB

Former Member
0 Kudos

HI Deepak, Gaurav,

No the control files are there in all the 3 locations, and I did stop the listener and start again the listener, but no luck.

The same error.

The control files are from todays date, and they are present.

Thanks


former_member182657
Active Contributor
0 Kudos

Hi Aps,

What about controlfile ?

Former Member
0 Kudos

HI Folks, my DC team has restored the /oracle/SID/102_64 mount point and the sapdata files are from the old back up from 2 month old.

I can connect to sqlplus as sysdba which I was not able to do before.

I have started the listener using listener start but I cannot startsap using the startsap command. Folks I am attaching the contents of the trans.log file which says that the listener is not up. help is apppreciated.

----------------------------

more trans.log

4 ETW000 R3trans version 6.19 (release 711 - 26.10.11 - 13:00:00).

4 ETW000 unicode enabled version

4 ETW000 ===============================================

4 ETW000

4 ETW000 date&time   : 08.04.2014 - 15:05:27

4 ETW000 control file: <no ctrlfile>

4 ETW000 R3trans was called as follows: R3trans -d

4 ETW000  trace at level 2 opened for a given file pointer

4 ETW000  [     dev trc,00000]  Tue Apr  8 15:05:27 2014                                                 528  0.000528

4 ETW000  [     dev trc,00000]  db_con_init called                                                        51  0.000579

4 ETW000  [     dev trc,00000]  determine_block_commit: no con_hdl found as blocked for con_name = R/3

4 ETW000                                                                                                 143  0.000722

4 ETW000  [     dev trc,00000]  create_con (con_name=R/3)                                                 74  0.000796

4 ETW000  [     dev trc,00000]  Loading DB library '/usr/sap/<SID>/SYS/exe/uc/hpia64/dboraslib.so' ...

4 ETW000                                                                                                 158  0.000954

4 ETW000  [     dev trc,00000]  DlLoadLib() success: dlopen("/usr/sap/<SID>/SYS/exe/uc/hpia64/dboraslib.so"), hdl 0

4 ETW000                                                                                               68452  0.069406

4 ETW000  [     dev trc,00000]  Library '/usr/sap/<SID>/SYS/exe/uc/hpia64/dboraslib.so' loaded              78  0.069484

4 ETW000  [     dev trc,00000]  function DbSlExpFuns loaded from library /usr/sap/<SID>/SYS/exe/uc/hpia64/dboraslib.so

4 ETW000                                                                                                 177  0.069661

4 ETW000  [     dev trc,00000]  Version of '/usr/sap/<SID>/SYS/exe/uc/hpia64/dboraslib.so' is "710.21", patchlevel (0.146)

4 ETW000                                                                                                 497  0.070158

4 ETW000  [     dev trc,00000]  function dsql_db_init loaded from library /usr/sap/<SID>/SYS/exe/uc/hpia64/dboraslib.so

4 ETW000                                                                                                  83  0.070241

4 ETW000  [     dev trc,00000]  function dbdd_exp_funs loaded from library /usr/sap/<SID>/SYS/exe/uc/hpia64/dboraslib.so

4 ETW000                                                                                                 319  0.070560

4 ETW000  [     dev trc,00000]  New connection 0 created                                                 118  0.070678

4 ETW000  [     dev trc,00000]  0: name = R/3, con_id = -000000001, state = DISCONNECTED, tx = NO , bc = NO , hc = NO , perm = YES, reco = NO , frco = NO , timeout = 00

0, con_max = 255, con_opt = 255, occ = NO , prog =

4 ETW000                                                                                                  97  0.070775

4 ETW000  [     dev trc,00000]  db_con_connect (con_name=R/3)                                             46  0.070821

4 ETW000  [     dev trc,00000]  determine_block_commit: no con_hdl found as blocked for con_name = R/3

4 ETW000                                                                                                  65  0.070886

4 ETW000  [     dev trc,00000]  find_con_by_name found the following connection:                          43  0.070929

4 ETW000  [     dev trc,00000]  0: name = R/3, con_id = 000000000, state = DISCONNECTED, tx = NO , bc = NO , hc = NO , perm = YES, reco = NO , frco = NO , timeout = 000

, con_max = 255, con_opt = 255, occ = NO , prog =

4 ETW000                                                                                                 253  0.071182

4 ETW000  [     dev trc,00000]  -->DbSlConnect(vers=-1, type=0, env='[default env]', userid='[default usr]')

4 ETW000                                                                                                 283  0.071465

4 ETW000  [     dev trc,00000]  Register application info.                                               735  0.072200

4 ETW000  [     dev trc,00000]  Oracle client version: 10.2.0.1.0, (dbsl 711 131011, UNICODE[2])         192  0.072392

4 ETW000  [     dev trc,00000]    Default connection: tnsname ='<SID>'                                      97  0.072489

4 ETW000  [     dev trc,00000]  -->oci_initialize(con=0, char='UTF16', nchar='AL16UTF16', nls=-1),uc_ln=2

4 ETW000                                                                                                 107  0.072596

4 ETW000  [     dev trc,00000]  OCIEnvCreate(mode=16384=OCI_UTF16) returned 0 (for default NLS handle)

4 ETW000                                                                                               15088  0.087684

4 ETW000  [     dev trc,00000]     OCIHandleAlloc(OCI_HTYPE_ERROR, con=0, ErrHp=60000000022455b0)        238  0.087922

4 ETW000  [     dev trc,00000]     OCIHandleAlloc(OCI_HTYPE_ERROR, con=0, ErrBt=6000000002256168)         64  0.087986

4 ETW000  [     dev trc,00000]  NLS env. settings: lang=0041004D00450052004900430041004E                 752  0.088738

4 ETW000  [     dev trc,00000]                     terr=0041004D00450052004900430041                      80  0.088818

4 ETW000  [     dev trc,00000]                     char=00550054004600310036                              64  0.088882

4 ETW000  [     dev trc,00000]  Client NLS setting (by OCINlsGetInfo(con=0)): 'AMERICAN_AMERICA.UTF16'

4 ETW000                                                                                                  76  0.088958

4 ETW000  [     dev trc,00000]  charset='UTF16', ncharset='AL16UTF16', UNI_ASC=FALSE                      53  0.089011

4 ETW000  [     dev trc,00000]  Logon as OPS$-user to get SAPSR3's password                              138  0.089149

4 ETW000  [     dev trc,00000]  Connecting as /@<SID> on connection 0 (nls 0) ... (dbsl 711 131011, UNICODE[2])

4 ETW000                                                                                                  84  0.089233

4 ETW000  [     dev trc,00000]     OCIHandleAlloc(OCI_HTYPE_SVCCTX, con=0, svchp=6000000002256098)       129  0.089362

4 ETW000  [     dev trc,00000]  Allocating server context handle                                          56  0.089418

4 ETW000  [     dev trc,00000]  Attaching to database server <SID> (con=0, svchp=6000000002256098, srvhp=60000000022598e8)

4 ETW000                                                                                                  86  0.089504

4 ETW000  [     dev trc,00000]     OCIServerAttach(OCI_DEFAULT) failed with -1=OCI_ERROR               17668  0.107172

4 ETW000  [     dev trc,00000]     OCIServerAttach(OCI_DEFAULT) failed with SQL error 12541:             417  0.107589

4 ETW000  [     dev trc,00000]     ORA-12541: TNS:no listener

4 ETW000                                                                                                 104  0.107693

4 ETW000  [    dbsloci.,00000]  *** ERROR => CONNECT failed with sql error '12541'                        93  0.107786

4 ETW000  [     dev trc,00000]     set_ocica() -> SQL error code 12541                                    51  0.107837

4 ETW000  [     dev trc,00000]  -->oci_get_errmsg (con=0, rc=12541)                                       58  0.107895

4 ETW000  [     dev trc,00000]     OCIErrorGet -> SQL error code: 12541                                  155  0.108050

4 ETW000  [     dev trc,00000]     ORA-12541: TNS:no listener

4 ETW000                                                                                                  87  0.108137

4 ETW000  [     dev trc,00000]  SAPUSER error or connect error 99=DBSL_ERR_DB, oerr=12541.                55  0.108192

4 ETW000  [     dev trc,00000]     DbSlConnect(con=0) -> orc=0, 99=DBSL_ERR_DB                            55  0.108247

4 ETW000  [    dblink  ,00000]  ***LOG BY2=>sql error 12541  performing CON                              375  0.108622

4 ETW000  [    dblink  ,00000]  ***LOG BY0=>ORA-12541: TNS:no listener                                    79  0.108701

2EETW169 no connect possible: "DBMS = ORACLE                           --- dbs_ora_tnsname = '<SID>'"

-----------------------

former_member188883
Active Contributor
0 Kudos

Hi Aps,

Just stop listener again.

check whether any other listener process is running . If yes , kill it.

Then start listener.

Did you perform any change in listner file in last 2 months

Hope this helps.

Regards,

Deepak Kori

Reagan
Product and Topic Expert
Product and Topic Expert
0 Kudos

4 ETW000  [     dev trc,00000]     OCIServerAttach(OCI_DEFAULT) failed with -1=OCI_ERROR               17668  0.107172

4 ETW000  [     dev trc,00000]     OCIServerAttach(OCI_DEFAULT) failed with SQL error 12541:             417  0.107589

4 ETW000  [     dev trc,00000]     ORA-12541: TNS:no listener

su - orasid

lsnrctl stop

lsnrctl start

exit

su - sidadm

R3trans -d

and now provide the trans.log

Regards

RB

Former Member
0 Kudos

Thanks to you all

@RB - I know I have asked a lot of senseless questions but I think I just freaked out more than required.

@Nick Loy - thanks right now DC team restoring back up 2 months old on to the /oracle/SID/102_64. and then I will create the control file stuff. Let you know after that.

I will update the blog after this is resolved, thanks to everybody else who provided the information I really appreciate this.

Thanks


Former Member
0 Kudos

Hi Gaurav,

At this point I do not care what data is there on the server I just want it  running.

If I do a system refresh will I be able to restore the Server or no?

What I want to know is that will it work or no, thanks man.

Regards,

Aps

former_member182657
Active Contributor
0 Kudos

Hi Aps,

Unfortunately it's no (If go with refresh activity with DEV or PRD). But after refresh you can restore the system with your previous 2 months backup data with you.Follow System Copy guides for this activity.

In refresh activity you need to install oracle 11g (Procedure is like you are going to install a fresh system but using data from other system) .I suggest from the existing system just backup required files like complete trans directory. .

Regards,

Gaurav

Former Member
0 Kudos

Hi Aps,

If data is not important as you have mentioned, you can restore dev or prd, however install oracle first.

Prd because it can be useful if you want to test on almost real time data.

Follow the System refresh procedure, like backup and restore etc to complete the refresh.

Before starting anything now, first take the backup of the file system.

For the oracle issues, re-install the oracle software. Apply the latest SBP too.

1525025 - Oracle 11.2.0: Oracle Software Deinstallation

Regards, Jatin

Former Member
0 Kudos

Hi Reagan, Gaurav and Deepak

I have the control files but I think they are from 2 month old backup.

I did run the commands as reagan sugggested on the Dev and the Qua server.I coudnot count the files on both the servers but definately the files on the DEV server are more in count.

What the hell am I suppose to do. Shall I install oracle from a fresh if yes then how do I go about it.

Shall I fiirst deintall Oracle 10g and SAP ?

Shall I deintall oracle 10g only ?

Shall I start intalling oracle on the system with the old one present as well and the new oracle is going to verwrite the old Oracle.?

I have done Oracle Upgrade recently using OUI from 10g to 11g on both windows and UNIX but how do I go about fresh installation.

I need help folks

Regards,

Aps

former_member182657
Active Contributor
0 Kudos

Hi Aps,

First what is your approach now,are you going with old backup restore or going with system refresh either by PRD or by DEV.Please confirm.For your query


Shall I fiirst deintall Oracle 10g and SAP ?

If you are going for fresh installation try to clear oracle & sap.Also you mentioned that you recent upgraded the oracle 10g to 11g.So go with 11g installation instead of 10g for fresh installation.

Regards,

Gaurav

Former Member
0 Kudos

Hi Gaurav,

If I do a system refresh will I be able to restore the Server or no?

If I go for a fresh Installtion can you provide me with some notes or I just follow the installation guide for Installing SAP PI ?

Regards

aps

former_member182657
Active Contributor
0 Kudos

Hi Aps,


If I do a system refresh will I be able to restore the Server or no?

If you go with system refresh you would not be able to restore your existing one as it'll refresh this system either by PRD or by DEV data.


If I go for a fresh Installtion can you provide me with some notes or I just follow the installation guide for Installing SAP PI ?

Better to follow standard,general SAP PI installation guides.

Regards,

Gaurav

Reagan
Product and Topic Expert
Product and Topic Expert
0 Kudos

What the hell am I suppose to do. Shall I install oracle from a fresh if yes then how do I go about it.

Read Oracle installation guide.


Shall I fiirst deintall Oracle 10g and SAP ?

Shall I deintall oracle 10g only ?

Shall I start intalling oracle on the system with the old one present as well and the new oracle is going to verwrite the old Oracle.?

I have done Oracle Upgrade recently using OUI from 10g to 11g on both windows and UNIX but how do I go about fresh installation.

Too many irrelevant and baseless questions. Why do you want to uninstall SAP ?

As this is an Unix system you don't need to uninstall the Oracle or SAP. Just delete the files under $ORACLE_HOME and start the Oracle installation.

You don't need to upgrade the database if you are installing the database software. Install the database version which was used before and apply the latest SBP.

Adapt the environment variables if required and start SAP.

Regards

RB

Reagan
Product and Topic Expert
Product and Topic Expert
0 Kudos

if the command was executed under /oracle/SID then yo might have lost the environment variable files, and the files under /oracle/SID/112_64 including the parameter files. The database datafiles and redologs will be there. If one of the control files were deleted then copy from the other location (if possible). You probably need to re-install the Oracle software to fix this.

I would first try to copy the missing files from another server in the landscape.

You can try to identify the missing files by modifying your command by replacing the rm condition with the listing condition and then execute the command on both the servers and find the missing ones.

Regards

RB

Former Member
0 Kudos

Hi Josef and Gaurav,

@josef I was logged in as /oracle/SID when I eneterd the command

@ Gaurav I do not mind doing the complete restore, but I think I have control files from the old back up and also I do have the sapdata files.

My question to you all is that If I do a complete Restore will the oracle binaries be restored or not and do I need to follow some procedures that I need to do after this is complete.

Regards

former_member188883
Active Contributor
0 Kudos

Hi Aps,

Oracle binaries are not backed up as a part of oracle database backup. You need to backup them separately using file system backup.

Worst case scenario.

Take a copy of oracle home from PRD or DEV and paste it into QAS.

Adjust the listener.ora, tnsnames.ora and sqlnet.ora files.

Post this restore the fresh backup from production and recover it.

Start the database and then application.

Hope this helps.

Regards,

Deepak Kori

former_member182657
Active Contributor
0 Kudos

Hi Aps,


My question to you all is that If I do a complete Restore will the oracle binaries be restored or not and do I need to follow some procedures that I need to do after this is complete.

If you do a complete restore oracle binaries will not be restored from the backup (Usually we not take backup of oracle binaries,just confirm it with your database team),for this you need to re install oracle 10g software as said earlier.

Also please clear for sapdata files,(these files are from latest or also from the old)


I think I have control files from the old back up and also I do have the sapdata files.

Regards,

Gaurav

Former Member
0 Kudos

find . -type f -mtime +15 -exec rm -f {} \;

Good news is:
Data files should still be there, because they should be younger than 15 days.
No need to restore database backup imho.

You may have lost control files though. Please first check if at least one ot them is left!

And you still haven't told us:
What was your current directory when you typed the command above?


PS

And abou the 'disk full' messages:

Check if you lost mount points or filesystems!

PPS:

And if you already startet restore, my advice from above may be too late. Sorry for that.

Former Member
0 Kudos

Hi,

Since you have datafiles, you can recover your database as below:

1) rsync the Oracle home between your production and Quality (/oracle/SID/102_64)

2) Create controlfile and recover/open database

Other easiest option I would follow:

1) rsync the Oracle home same as above

2) Restore Production database into Quality (It will be your Quality refresh)

This option will give me the consistent system with latest production data

*All other folders can be checked/corrected manually (for sap*)

Regards,

Nick Loy

Former Member
0 Kudos

Hi Deepak and Gaurav,

I have complete offline backup on Tape, I can use that. But how do I check the santity of the DB (Oracle).

I do have enough space on the orarch, only saptrace and sapdata2 are full.

Guys I did tell the DC team to restore the Quality server from the old backup but the Restore failed on the following drives i.e.

1) /oracle/sid/oraarch - disk full

2) /oracle/sid/saptrace - disk full

3) /oracle/sid/102_64 - disk full

4) /oracle/sid/sapdata2 - disk full

After the delete debacle I saw that there was no sqlplus in the /oracle/xqa so I thought of I restore but that did not go well as you can see.

So what do I do now, shall I tell them to do it again anything specific that I need to tell them.

Regards.


former_member182657
Active Contributor
0 Kudos

Hi Aps,

If you have valid backup then no worries,also as you mentioned you're not able to connect database via command sqlplus "/as sysdba" (As per me during deletion some database executable's also get  removed from the system.Try to find out first what things left & what are gone completely.

If we're going for complete restore we can go for installation up to database load phase & here we can restore our old backup from tapes.

Regards,

Gaurav

former_member182657
Active Contributor
0 Kudos

Hi Aps,


Error: Controlfile was changed externally while mounted

       Please check if another Oracle database is running

       and accessing the same controlfile

Mon Apr 07 12:00:36 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26202.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

ARC1 started with pid=35, OS id=26204

After issue occurred have you restored some data files ? Something changed / misplaced (Which is already with your environment.)

Try to restore it from valid backup.

Regards,

Former Member
0 Kudos

Hi Gaurav following are the logs:

last 300 lines of alert_SID.log File contents are

-------------------------------

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26200.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:36 IST 2014

ARCH: Detected ARCH process failure

ARCH: Detected ARCH process failure

ARCH: STARTING ARCH PROCESSES

ARC0 started with pid=14, OS id=26202

Mon Apr 07 12:00:36 IST 2014

ARC0: Archival started

ARC1: Archival started

ARCH: STARTING ARCH PROCESSES COMPLETE

Mon Apr 07 12:00:36 IST 2014

Error: Controlfile was changed externally while mounted

       Please check if another Oracle database is running

       and accessing the same controlfile

Mon Apr 07 12:00:36 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26202.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

ARC1 started with pid=35, OS id=26204

Error: Controlfile was changed externally while mounted

       Please check if another Oracle database is running

       and accessing the same controlfile

Mon Apr 07 12:00:36 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26204.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:41 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26202.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:41 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26202.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:41 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26204.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:41 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26204.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:43 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26202.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:43 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26202.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:43 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26202.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:43 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26204.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:43 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26204.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:43 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26204.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:45 IST 2014

ARCH: Detected ARCH process failure

ARCH: Detected ARCH process failure

ARCH: STARTING ARCH PROCESSES

ARC0 started with pid=14, OS id=26206

Mon Apr 07 12:00:45 IST 2014

ARC0: Archival started

ARC1: Archival started

ARCH: STARTING ARCH PROCESSES COMPLETE

Mon Apr 07 12:00:45 IST 2014

Error: Controlfile was changed externally while mounted

       Please check if another Oracle database is running

       and accessing the same controlfile

Mon Apr 07 12:00:45 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26206.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

ARC1 started with pid=36, OS id=26208

Error: Controlfile was changed externally while mounted

       Please check if another Oracle database is running

       and accessing the same controlfile

Mon Apr 07 12:00:45 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26208.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:50 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26208.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:50 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26208.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:51 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26206.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:51 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26206.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:52 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26208.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:52 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26208.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:52 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26208.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:53 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26206.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:53 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26206.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:53 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26206.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:54 IST 2014

ARCH: Detected ARCH process failure

ARCH: Detected ARCH process failure

ARCH: STARTING ARCH PROCESSES

ARC0 started with pid=35, OS id=26211

Mon Apr 07 12:00:55 IST 2014

ARC0: Archival started

ARC1: Archival started

ARCH: STARTING ARCH PROCESSES COMPLETE

Mon Apr 07 12:00:55 IST 2014

Error: Controlfile was changed externally while mounted

       Please check if another Oracle database is running

       and accessing the same controlfile

ARC1 started with pid=36, OS id=26213

Mon Apr 07 12:00:55 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26211.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Error: Controlfile was changed externally while mounted

       Please check if another Oracle database is running

       and accessing the same controlfile

Mon Apr 07 12:00:55 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26213.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:59 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26213.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:00:59 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26213.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:00 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26211.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:00 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26211.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:01 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26213.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:01 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26213.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:01 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26213.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:01 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26211.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:01 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26211.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:01 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26211.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:04 IST 2014

ARCH: Detected ARCH process failure

ARCH: Detected ARCH process failure

ARCH: STARTING ARCH PROCESSES

ARC0 started with pid=14, OS id=26215

Mon Apr 07 12:01:04 IST 2014

ARC0: Archival started

ARC1: Archival started

ARCH: STARTING ARCH PROCESSES COMPLETE

ARC1 started with pid=37, OS id=26217

Mon Apr 07 12:01:04 IST 2014

Error: Controlfile was changed externally while mounted

       Please check if another Oracle database is running

       and accessing the same controlfile

Mon Apr 07 12:01:04 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26215.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:04 IST 2014

Error: Controlfile was changed externally while mounted

       Please check if another Oracle database is running

       and accessing the same controlfile

Mon Apr 07 12:01:04 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26217.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:09 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26215.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:09 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26215.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:09 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26217.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:09 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26217.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:10 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26217.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:10 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26217.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:10 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26217.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:11 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26215.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:11 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26215.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:11 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26215.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:13 IST 2014

ARCH: Detected ARCH process failure

ARCH: Detected ARCH process failure

ARCH: STARTING ARCH PROCESSES

ARC0 started with pid=35, OS id=26219

Mon Apr 07 12:01:13 IST 2014

ARC0: Archival started

ARC1: Archival started

ARCH: STARTING ARCH PROCESSES COMPLETE

Mon Apr 07 12:01:13 IST 2014

Error: Controlfile was changed externally while mounted

       Please check if another Oracle database is running

       and accessing the same controlfile

Mon Apr 07 12:01:13 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26219.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

ARC1 started with pid=36, OS id=26221

Error: Controlfile was changed externally while mounted

       Please check if another Oracle database is running

       and accessing the same controlfile

Mon Apr 07 12:01:13 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc1_26221.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:20 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26219.trc:

ORA-00600: internal error code, arguments: [2141], [67556390], [60878963], [], [], [], [], []

Mon Apr 07 12:01:20 IST 2014

Errors in file /oracle/<SID>/saptrace/background/<SID>_arc0_26219.trc:

ORA-00600: internal error code,

Contents of trans.log

---------------------------------

more trans.log

4 ETW000 R3trans version 6.19 (release 711 - 26.10.11 - 13:00:00).

4 ETW000 unicode enabled version

4 ETW000 ===============================================

4 ETW000

4 ETW000 date&time   : 07.04.2014 - 15:47:54

4 ETW000 control file: <no ctrlfile>

4 ETW000 R3trans was called as follows: R3trans -d

4 ETW000  trace at level 2 opened for a given file pointer

4 ETW000  [     dev trc,00000]  Mon Apr  7 15:47:54 2014                                                 671  0.000671

4 ETW000  [     dev trc,00000]  db_con_init called                                                        48  0.000719

4 ETW000  [     dev trc,00000]  determine_block_commit: no con_hdl found as blocked for con_name = R/3

4 ETW000                                                                                                 173  0.000892

4 ETW000  [     dev trc,00000]  create_con (con_name=R/3)                                                 86  0.000978

4 ETW000  [     dev trc,00000]  Loading DB library '/usr/sap/SID/SYS/exe/uc/hpia64/dboraslib.so' ...

4 ETW000                                                                                                 217  0.001195

4 ETW000  [     dev trc,00000]  Mon Apr  7 15:47:55 2014                                               46903  0.048098

4 ETW000  [     dev trc,00000]  DlLoadLib() success: dlopen("/usr/sap/SID/SYS/exe/uc/hpia64/dboraslib.so"), hdl 0

4 ETW000                                                                                                 841  0.048939

4 ETW000  [     dev trc,00000]  Library '/usr/sap/SID/SYS/exe/uc/hpia64/dboraslib.so' loaded              66  0.049005

4 ETW000  [     dev trc,00000]  function DbSlExpFuns loaded from library /usr/sap/SID/SYS/exe/uc/hpia64/dboraslib.so

4 ETW000                                                                                                 183  0.049188

4 ETW000  [     dev trc,00000]  Version of '/usr/sap/SID/SYS/exe/uc/hpia64/dboraslib.so' is "710.21", patchlevel (0.146)

4 ETW000                                                                                                 569  0.049757

4 ETW000  [     dev trc,00000]  function dsql_db_init loaded from library /usr/sap/SID/SYS/exe/uc/hpia64/dboraslib.so

4 ETW000                                                                                                  92  0.049849

4 ETW000  [     dev trc,00000]  function dbdd_exp_funs loaded from library /usr/sap/SID/SYS/exe/uc/hpia64/dboraslib.so

4 ETW000                                                                                                 350  0.050199

4 ETW000  [     dev trc,00000]  New connection 0 created                                                 150  0.050349

4 ETW000  [     dev trc,00000]  0: name = R/3, con_id = -000000001, state = DISCONNECTED, tx = NO , bc = NO , hc = NO , perm = YES, reco = NO , frco = NO , timeout = 00

0, con_max = 255, con_opt = 255, occ = NO , prog =

4 ETW000                                                                                                 102  0.050451

4 ETW000  [     dev trc,00000]  db_con_connect (con_name=R/3)                                             47  0.050498

4 ETW000  [     dev trc,00000]  determine_block_commit: no con_hdl found as blocked for con_name = R/3

4 ETW000                                                                                                  72  0.050570

4 ETW000  [     dev trc,00000]  find_con_by_name found the following connection:                          44  0.050614

4 ETW000  [     dev trc,00000]  0: name = R/3, con_id = 000000000, state = DISCONNECTED, tx = NO , bc = NO , hc = NO , perm = YES, reco = NO , frco = NO , timeout = 000

, con_max = 255, con_opt = 255, occ = NO , prog =

4 ETW000                                                                                                 262  0.050876

4 ETW000  [     dev trc,00000]  -->DbSlConnect(vers=-1, type=0, env='[default env]', userid='[default usr]')

4 ETW000                                                                                                 294  0.051170

4 ETW000  [     dev trc,00000]  Register application info.                                               803  0.051973

4 ETW000  [     dev trc,00000]  Oracle client version: 10.2.0.1.0, (dbsl 711 131011, UNICODE[2])         197  0.052170

4 ETW000  [     dev trc,00000]    Default connection: tnsname ='SID'                                      61  0.052231

4 ETW000  [     dev trc,00000]  -->oci_initialize(con=0, char='UTF16', nchar='AL16UTF16', nls=-1),uc_ln=2

4 ETW000                                                                                                 120  0.052351

4 ETW000  [     dev trc,00000]  OCIEnvCreate(mode=16384=OCI_UTF16) returned 0 (for default NLS handle)

4 ETW000                                                                                                8016  0.060367

4 ETW000  [     dev trc,00000]     OCIHandleAlloc(OCI_HTYPE_ERROR, con=0, ErrHp=60000000022455b0)        207  0.060574

4 ETW000  [     dev trc,00000]     OCIHandleAlloc(OCI_HTYPE_ERROR, con=0, ErrBt=6000000002256168)         65  0.060639

4 ETW000  [     dev trc,00000]  NLS env. settings: lang=0041004D00450052004900430041004E                 366  0.061005

4 ETW000  [     dev trc,00000]                     terr=0041004D00450052004900430041                      78  0.061083

4 ETW000  [     dev trc,00000]                     char=00550054004600310036                              86  0.061169

4 ETW000  [     dev trc,00000]  Client NLS setting (by OCINlsGetInfo(con=0)): 'AMERICAN_AMERICA.UTF16'

4 ETW000                                                                                                  75  0.061244

4 ETW000  [     dev trc,00000]  charset='UTF16', ncharset='AL16UTF16', UNI_ASC=FALSE                      53  0.061297

4 ETW000  [     dev trc,00000]  Logon as OPS$-user to get SAPSR3's password                              153  0.061450

4 ETW000  [     dev trc,00000]  Connecting as /@SID on connection 0 (nls 0) ... (dbsl 711 131011, UNICODE[2])

4 ETW000                                                                                                  85  0.061535

4 ETW000  [     dev trc,00000]     OCIHandleAlloc(OCI_HTYPE_SVCCTX, con=0, svchp=6000000002256098)       174  0.061709

4 ETW000  [     dev trc,00000]  Allocating server context handle                                          66  0.061775

4 ETW000  [     dev trc,00000]  Attaching to database server SID (con=0, svchp=6000000002256098, srvhp=60000000022598e8)

4 ETW000                                                                                                  93  0.061868

4 ETW000  [     dev trc,00000]  Assigning server context 60000000022598e8 to service context 6000000002256098

4 ETW000                                                                                              983608  1.045476

4 ETW000  [     dev trc,00000]     OCIHandleAlloc(OCI_HTYPE_SESSION, con=0, usrhp=6000000002245dc8)      192  1.045668

4 ETW000  [     dev trc,00000]  Starting user session: OCISessionBegin(con=0, usr='/', svc=6000000002256098, srv=60000000022598e8, usr=6000000002245dc8)

4 ETW000                                                                                                  87  1.045755

4 ETW000  [     dev trc,00000]  Mon Apr  7 15:47:56 2014                                               35897  1.081652

4 ETW000  [     dev trc,00000]     OCISessionBegin(OCI_DEFAULT) failed with -1=OCI_ERROR                 138  1.081790

4 ETW000  [     dev trc,00000]     OCISessionBegin(OCI_DEFAULT) failed with SQL error 257:               404  1.082194

4 ETW000  [     dev trc,00000]     ORA-00257: archiver error. Connect internal only, until freed.

4 ETW000                                                                                                 100  1.082294

4 ETW000  [     dev trc,00000]  server_detach(con=0,stale=1,srvhp=60000000022598e8)                       65  1.082359

4 ETW000  [     dev trc,00000]  Server handle (con=0,svchp=6000000002256098,srvhp=60000000022598e8) detached

4 ETW000                                                                                                1122  1.083481

4 ETW000  [     dev trc,00000]  Free server handle 60000000022598e8 of con=0                             194  1.083675

4 ETW000  [    dbsloci.,00000]  *** ERROR => CONNECT failed with sql error '257'                         186  1.083861

4 ETW000  [     dev trc,00000]     set_ocica() -> SQL error code 257                                      88  1.083949

4 ETW000  [     dev trc,00000]  -->oci_get_errmsg (con=0, rc=257)                                         64  1.084013

4 ETW000  [     dev trc,00000]     OCIErrorGet -> SQL error code: 257                                    160  1.084173

4 ETW000  [     dev trc,00000]     ORA-00257: archiver error. Connect internal only, until freed.

4 ETW000                                                                                                  87  1.084260

4 ETW000  [     dev trc,00000]  SAPUSER error or connect error 99=DBSL_ERR_DB, oerr=257.                  63  1.084323

4 ETW000  [     dev trc,00000]     DbSlConnect(con=0) -> orc=0, 99=DBSL_ERR_DB                            57  1.084380

4 ETW000  [    dblink  ,00000]  ***LOG BY2=>sql error 257    performing CON                              557  1.084937

4 ETW000  [    dblink  ,00000]  ***LOG BY0=>ORA-00257: archiver error. Connect internal only, until freed.

4 ETW000                                                                                                  88  1.085025

2EETW169 no connect possible: "DBMS = ORACLE                           --- dbs_ora_tnsname = 'SID'"

former_member188883
Active Contributor
0 Kudos

Hi Aps,

1) Check orarach has enough space before you try to start the database.

2) Do you have any file system backup available ?

Regards,

Deepak Kori

Former Member
0 Kudos

Hi Deepak and Gaurav,

The Database is not working properly when I try to login as sqlplus"/as sysdba"

it is asking me for a username and password which is not how it should be working and somehow the /oracle/sid/saptrace mount point is getting 100% no matter what I do.

also R3trans -d is giving a code 12 which is obvious since the DB is not working. Listeneer is up though but I cannot start the DB since I cannot loginto sqlpus.

I cannot also change the password for any users since brconnect and BRTOOLS is failing as well.

If you want I can post some logs like the alert_SID.log and trans.log

Regards.

former_member182657
Active Contributor
0 Kudos

Hi Aps,

Please share both the logs.

Thanks,

Former Member
0 Kudos

Hi Deepak and Gaurav,

when I had given the command I within maybe 30 seconds pressed control+c and exited out but the damage was already done.

Now I have the complete offline backup from 2 months ago and It is by the grace of god not Production Environment, but it is Quality Server.

Yes Gaurav they use HP DATA Protector to do backup and restores.

So please what should I tell my DC team that what should I do, also I can even tell them to do a restore from the Dev Backup.

I want you to please tell me the process to Restore the System and get the PI environment working again.

Thanks


former_member182657
Active Contributor
0 Kudos

Hi Aps,

Thank to GOD that it's not your PRD system,Now you can go with previous restore & other option is to make a system copy of QTY from Production System.( It's on you from which system you gonna make system copy,practically it would be PRD.)

Tell your database team to perform restore activities as per standard methods of HP Data Protector.

Thanks,

Gaurav

former_member188883
Active Contributor
0 Kudos

Hi,

Here only the database backup will not help as we do not know what all folders got deleted.

Can you first check whether your oracle home is consistent.

Compare the file count and folder count with that from Production .

In case the files are consistent, you may try to connect to database. If the database connection works fine, you may proceed with restoration of data from backup.

Hope this helps.

Regards,

Deepak Kori

former_member182657
Active Contributor
0 Kudos

Hi Aps,

I think It would be a horrible situation at your end after deletion of complete files under directory /oracle/<SID> .As per me it's difficult to retrieve the structure if you are not with a valid backup (Hope it's not a Production environment).

For deleted files recovery you need to search out for a third party solution,not sure it'll retrieve the complete data files.Try to search any recent backup and if not,then last is to restore the system with previous backup & provide the redo files for complete recovery up till now.

Thanks,

Gaurav 

former_member188883
Active Contributor
0 Kudos

Hi ,

1) Have you delete everything under /oracle/SID or what is the current state of the system ?

2) What kind of backup is available for the system and time period of that backup.

3) Do you have logs available from the last backup until today ?

Regards,

Deepak Kori