data file at the time copy its error I/O

[Former Member]

Hi Experts,

We found the error of the subject issue. There is data file I/O error. Even we are unable to copy from cp command. While copying the data through cp it is giving I/O error Also I am giving the /var/adm/messages file of solaris

May 13 10:07:35 bsldb scsi: [ID 107833 kern.notice] Sense Key: Media Error

May 13 10:07:35 bsldb scsi: [ID 107833 kern.notice] ASC: 0x11 (unrecovered read error), ASCQ: 0x0, FRU: 0x0

May 13 10:07:35 bsldb scsi_vhci: [ID 734749 kern.warning] WARNING: vhci_scsi_reset 0x1

May 13 10:07:35 bsldb scsi: [ID 243001 kern.warning] WARNING: /pci@9,600000/SUNW,qlc@1/fp@0,0 (fcp1):

May 13 10:07:35 bsldb FCP: WWN 0x500601683ce050fb reset successfully

May 13 10:07:35 bsldb scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/ssd@g600601600bc025004a1ed78a8290df11 (ssd46):

May 13 10:07:35 bsldb Error for Command: read(10) Error Level: Retryable

May 13 10:07:35 bsldb scsi: [ID 107833 kern.notice] Requested Block: 164889584 Error Block: 164889584

May 13 10:07:35 bsldb scsi: [ID 107833 kern.notice] Vendor: DGC Serial Number: 1700008DE1CL

May 13 10:07:35 bsldb scsi: [ID 107833 kern.notice] Sense Key: Media Error

May 13 10:07:35 bsldb scsi: [ID 107833 kern.notice] ASC: 0x11 (unrecovered read error), ASCQ: 0x0, FRU: 0x0

May 13 10:07:35 bsldb scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/ssd@g600601600bc02500968551c78190df11 (ssd47):

May 13 10:07:35 bsldb Error for Command: read(10) Error Level: Retryable

May 13 10:07:35 bsldb scsi: [ID 107833 kern.notice] Requested Block: 573848752 Error Block: 573848752

May 13 10:07:35 bsldb scsi: [ID 107833 kern.notice] Vendor: DGC Serial Number: 160000822ACL

May 13 10:07:35 bsldb scsi: [ID 107833 kern.notice] Sense Key: Unit Attention

May 13 10:07:35 bsldb scsi: [ID 107833 kern.notice] ASC: 0x29 (power on, reset, or bus reset occurred), ASCQ: 0x0, FRU: 0x0

May 13 10:07:35 bsldb scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/ssd@g600601600bc0250004521bdc190fdf11 (ssd24):

May 13 10:07:35 bsldb Error for Command: write(10) Error Level: Retryable

May 13 10:07:35 bsldb scsi: [ID 107833 kern.notice] Requested Block: 86546 Error Block: 86546

May 13 10:07:35 bsldb scsi: [ID 107833 kern.notice] Vendor: DGC Serial Number: 2300007A8ACL

May 13 10:07:35 bsldb scsi: [ID 107833 kern.notice] Sense Key: Unit Attention

May 13 10:07:35 bsldb scsi: [ID 107833 kern.notice] ASC: 0x29 (power on, reset, or bus reset occurred), ASCQ: 0x0, FRU: 0x0

May 13 10:07:37 bsldb scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/ssd@g600601600bc0250002521bdc190fdf11 (ssd26):

May 13 10:07:37 bsldb Error for Command: write(10) Error Level: Retryable

May 13 10:07:37 bsldb scsi: [ID 107833 kern.notice] Requested Block: 10848 Error Block: 10848

May 13 10:07:37 bsldb scsi: [ID 107833 kern.notice] Vendor: DGC Serial Number: 2100007A84CL

May 13 10:07:37 bsldb scsi: [ID 107833 kern.notice] Sense Key: Unit Attention

May 13 10:07:37 bsldb scsi: [ID 107833 kern.notice] ASC: 0x29 (power on, reset, or bus reset occurred), ASCQ: 0x0, FRU: 0x0

May 13 10:07:37 bsldb scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/ssd@g600601600bc02500e8eec5c17f90df11 (ssd49):

May 13 10:07:37 bsldb Error for Command: write(10) Error Level: Retryable

May 13 10:07:37 bsldb scsi: [ID 107833 kern.notice] Requested Block: 126416 Error Block: 126416

May 13 10:07:37 bsldb scsi: [ID 107833 kern.notice] Vendor: DGC Serial Number: 1400006327CL

May 13 10:07:37 bsldb scsi: [ID 107833 kern.notice] Sense Key: Unit Attention

May 13 10:07:37 bsldb scsi: [ID 107833 kern.notice] ASC: 0x29 (power on, reset, or bus reset occurred), ASCQ: 0x0, FRU: 0x0

May 13 10:07:38 bsldb scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/ssd@g600601600bc025004a1ed78a8290df11 (ssd46):

May 13 10:07:38 bsldb Error for Command: read(10) Error Level: Retryable

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] Requested Block: 243254560 Error Block: 243254560

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] Vendor: DGC Serial Number: 1700008DE1CL

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] Sense Key: Unit Attention

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] ASC: 0x29 (power on, reset, or bus reset occurred), ASCQ: 0x0, FRU: 0x0

May 13 10:07:38 bsldb scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/ssd@g600601600bc0250005521bdc190fdf11 (ssd23):

May 13 10:07:38 bsldb Error for Command: write(10) Error Level: Retryable

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] Requested Block: 26930 Error Block: 26930

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] Vendor: DGC Serial Number: 2400007A8DCL

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] Sense Key: Unit Attention

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] ASC: 0x29 (power on, reset, or bus reset occurred), ASCQ: 0x0, FRU: 0x0

May 13 10:07:38 bsldb scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/ssd@g600601600bc0250078a892eb8090df11 (ssd48):

May 13 10:07:38 bsldb Error for Command: read(10) Error Level: Retryable

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] Requested Block: 13984624 Error Block: 13984624

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] Vendor: DGC Serial Number: 15000074FFCL

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] Sense Key: Unit Attention

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] ASC: 0x29 (power on, reset, or bus reset occurred), ASCQ: 0x0, FRU: 0x0

May 13 10:07:38 bsldb scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/ssd@g600601600bc025004a1ed78a8290df11 (ssd46):

May 13 10:07:38 bsldb Error for Command: read(10) Error Level: Retryable

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] Requested Block: 164889584 Error Block: 164889584

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] Vendor: DGC Serial Number: 1700008DE1CL

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] Sense Key: Media Error

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] ASC: 0x11 (unrecovered read error), ASCQ: 0x0, FRU: 0x0

May 13 10:07:38 bsldb scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/ssd@g600601600bc025004a1ed78a8290df11 (ssd46):

May 13 10:07:38 bsldb Error for Command: read(10) Error Level: Fatal

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] Requested Block: 164889584 Error Block: 164889584

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] Vendor: DGC Serial Number: 1700008DE1CL

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] Sense Key: Media Error

May 13 10:07:38 bsldb scsi: [ID 107833 kern.notice] ASC: 0x11 (unrecovered read error), ASCQ: 0x0, FRU: 0x0

May 13 10:07:38 bsldb md_stripe: [ID 641072 kern.warning] WARNING: md: dbset/d5: read error on /dev/did/dsk/d5s0

May 13 10:08:35 bsldb scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0):

May 13 10:08:35 bsldb /scsi_vhci/ssd@g600601600bc02500968551c78190df11 (ssd47): Command Timeout on path /pci@9,600000/SUNW,emlxs@2,1/fp@0,0 (fp6)

Kindly suggest to resolve this issue. We have not taken backup for last 11 days.

Regards,

Jitendra

[Former Member]

Good support from the team specialy from markus.

[markus_doehr2]

> May 13 10:07:38 bsldb md_stripe: [ID 641072 kern.warning] WARNING: md: dbset/d5: read error on /dev/did/dsk/d5s0

> May 13 10:08:35 bsldb scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0):

> May 13 10:08:35 bsldb /scsi_vhci/ssd@g600601600bc02500968551c78190df11 (ssd47): Command Timeout on path /pci@9,600000/SUNW,emlxs@2,1/fp@0,0 (fp6)

>

>

> Kindly suggest to resolve this issue. We have not taken backup for last 11 days.

Your SFS FC driver is giving your those errors, you have a hardware corruption.

Execute

metastat -a

and post the results.

I'd open an call with Sun/Oracle also.

Markus

[Former Member]

hi Markus,

Please find the output of metastat -a

root@bsldb # metastat -a

d120: Mirror

Submirror 0: d121

State: Okay

Submirror 1: d122

State: Okay

Pass: 1

Read option: roundrobin (default)

Write option: parallel (default)

Size: 1221120 blocks (596 MB)

d121: Submirror of d120

State: Okay

Size: 1221120 blocks (596 MB)

Stripe 0:

Device Start Block Dbase State Reloc Hot Spare

c1t0d0s6 0 No Okay Yes

d122: Submirror of d120

State: Okay

Size: 1221120 blocks (596 MB)

Stripe 0:

Device Start Block Dbase State Reloc Hot Spare

c1t1d0s6 0 No Okay Yes

d140: Mirror

Submirror 0: d141

State: Okay

Submirror 1: d142

State: Okay

Pass: 1

Read option: roundrobin (default)

Write option: parallel (default)

Size: 10501632 blocks (5.0 GB)

d141: Submirror of d140

State: Okay

Size: 10501632 blocks (5.0 GB)

Stripe 0:

Device Start Block Dbase State Reloc Hot Spare

c1t0d0s4 0 No Okay Yes

d142: Submirror of d140

State: Okay

Size: 10501632 blocks (5.0 GB)

Stripe 0:

Device Start Block Dbase State Reloc Hot Spare

c1t1d0s4 0 No Okay Yes

d130: Mirror

Submirror 0: d131

State: Okay

Submirror 1: d132

State: Okay

Pass: 1

Read option: roundrobin (default)

Write option: parallel (default)

Size: 20982912 blocks (10 GB)

d131: Submirror of d130

State: Okay

Size: 20982912 blocks (10 GB)

Stripe 0:

Device Start Block Dbase State Reloc Hot Spare

c1t0d0s3 0 No Okay Yes

d132: Submirror of d130

State: Okay

Size: 20982912 blocks (10 GB)

Stripe 0:

Device Start Block Dbase State Reloc Hot Spare

c1t1d0s3 0 No Okay Yes

d110: Mirror

Submirror 0: d111

State: Okay

Submirror 1: d112

State: Okay

Pass: 1

Read option: roundrobin (default)

Write option: parallel (default)

Size: 205534848 blocks (98 GB)

d111: Submirror of d110

State: Okay

Size: 205534848 blocks (98 GB)

Stripe 0:

Device Start Block Dbase State Reloc Hot Spare

c1t0d0s1 0 No Okay Yes

d112: Submirror of d110

State: Okay

Size: 205534848 blocks (98 GB)

Stripe 0:

Device Start Block Dbase State Reloc Hot Spare

c1t1d0s1 0 No Okay Yes

d100: Mirror

Submirror 0: d101

State: Okay

Submirror 1: d102

State: Okay

Pass: 1

Read option: roundrobin (default)

Write option: parallel (default)

Size: 41945472 blocks (20 GB)

d101: Submirror of d100

State: Okay

Size: 41945472 blocks (20 GB)

Stripe 0:

Device Start Block Dbase State Reloc Hot Spare

c1t0d0s0 0 No Okay Yes

d102: Submirror of d100

State: Okay

Size: 41945472 blocks (20 GB)

Stripe 0:

Device Start Block Dbase State Reloc Hot Spare

c1t1d0s0 0 No Okay Yes

d123: Concat/Stripe

Size: 1221120 blocks (596 MB)

Stripe 0:

Device Start Block Dbase Reloc

c1t2d0s6 0 No Yes

d143: Concat/Stripe

Size: 10501632 blocks (5.0 GB)

Stripe 0:

Device Start Block Dbase Reloc

c1t2d0s4 0 No Yes

d133: Concat/Stripe

Size: 20982912 blocks (10 GB)

Stripe 0:

Device Start Block Dbase Reloc

c1t2d0s3 0 No Yes

d103: Concat/Stripe

Size: 41945472 blocks (20 GB)

Stripe 0:

Device Start Block Dbase Reloc

c1t2d0s0 0 No Yes

d113: Concat/Stripe

Size: 205534848 blocks (98 GB)

Stripe 0:

Device Start Block Dbase Reloc

c1t2d0s1 0 No Yes

Device Relocation Information:

Device Reloc Device ID

c1t2d0 Yes id1,ssd@n500000e1149f9f20

c1t1d0 Yes id1,ssd@n500000e114fd40e0

c1t0d0 Yes id1,ssd@n500000e01f87e1f0

dbset/d36: Concat/Stripe

Size: 115332096 blocks (54 GB)

Stripe 0:

Device Start Block Dbase Reloc

d36s0 0 No No

dbset/d29: Concat/Stripe

Size: 23058816 blocks (10 GB)

Stripe 0:

Device Start Block Dbase Reloc

d29s0 0 No No

dbset/d35: Concat/Stripe

Size: 136302848 blocks (64 GB)

Stripe 0:

Device Start Block Dbase Reloc

d35s0 0 No No

dbset/d28: Concat/Stripe

Size: 6282816 blocks (3.0 GB)

Stripe 0:

Device Start Block Dbase Reloc

d28s0 0 No No

dbset/d27: Concat/Stripe

Size: 10476800 blocks (5.0 GB)

Stripe 0:

Device Start Block Dbase Reloc

d27s0 0 No No

dbset/d26: Concat/Stripe

Size: 8379648 blocks (4.0 GB)

Stripe 0:

Device Start Block Dbase Reloc

d26s0 0 No No

dbset/d25: Concat/Stripe

Size: 6282816 blocks (3.0 GB)

Stripe 0:

Device Start Block Dbase Reloc

d25s0 0 No No

dbset/d21: Concat/Stripe

Size: 8379648 blocks (4.0 GB)

Stripe 0:

Device Start Block Dbase Reloc

d21s0 0 No No

dbset/d20: Concat/Stripe

Size: 2088768 blocks (1019 MB)

Stripe 0:

Device Start Block Dbase Reloc

d20s0 0 No No

dbset/d18: Concat/Stripe

Size: 2088768 blocks (1019 MB)

Stripe 0:

Device Start Block Dbase Reloc

d18s0 0 No No

dbset/d19: Concat/Stripe

Size: 2088768 blocks (1019 MB)

Stripe 0:

Device Start Block Dbase Reloc

d19s0 0 No No

dbset/d17: Concat/Stripe

Size: 2088768 blocks (1019 MB)

Stripe 0:

Device Start Block Dbase Reloc

d17s0 0 No No

dbset/d16: Concat/Stripe

Size: 167751680 blocks (79 GB)

Stripe 0:

Device Start Block Dbase Reloc

d16s0 0 No No

dbset/d15: Concat/Stripe

Size: 4185728 blocks (2.0 GB)

Stripe 0:

Device Start Block Dbase Reloc

d15s0 0 No No

dbset/d14: Concat/Stripe

Size: 1048529664 blocks (499 GB)

Stripe 0:

Device Start Block Dbase Reloc

d14s0 0 No No

dbset/d5: Concat/Stripe

Size: 1363079424 blocks (649 GB)

Stripe 0:

Device Start Block Dbase Reloc

d5s0 0 No No

Device Relocation Information:

Device Reloc Device ID

d36 No -

d29 No -

d35 No -

d28 No -

d27 No -

d26 No -

d25 No -

d21 No -

d20 No -

d18 No -

d19 No -

d17 No -

d16 No -

d15 No -

d14 No -

d5 No -

appset/d24: Concat/Stripe

Size: 20961920 blocks (10.0 GB)

Stripe 0:

Device Start Block Dbase Reloc

d24s0 0 No No

appset/d23: Concat/Stripe

Size: 25155840 blocks (11 GB)

Stripe 0:

Device Start Block Dbase Reloc

d23s0 0 No No

appset/d22: Concat/Stripe

Size: 65000448 blocks (30 GB)

Stripe 0:

Device Start Block Dbase Reloc

d22s0 0 No No

Device Relocation Information:

Device Reloc Device ID

d24 No -

d23 No -

d22 No -

[Former Member]

Hi Markus,

yes problem is related to the lun issue. Now we have unbind and rebind the lun. and restore the lun data again. Now issue has closed.

Regards,

Jitendra

[Former Member]

Hi Expers,

I have 11 days old backup alongwith all archive files but 2 days before I have added 2 more datafile so can I restore this old backup alongiwth the added 2 datafile.

Kindly help. It will be highly appreciable for this help.

Awaiting your reply.

Regards,

Jitendra

[volker_borowski2]

Hi,

most important: keep the current binary controlfiles!

In no case, create new controlfile now!

Be more safe and copy them to a safe location when the instance is stopped, before you do anything else.

I think current brtools are fully capable in handling this situation.

So a "repair" should work.

In case you have to go manually:

Restore the backup without the controlfiles

- brrestore -m all (NOT full) ...

Restore the redologs

- brrestore -a ....

Rename / move the possible defective datafiles, that do not belong

to the backup (the two ones you created later) to another position.

"startup mount" the DB

Issue a

create datafile '/oracle/SID/sapdata....';

for the both files missing in the backup. Since the binary controlfile is still aware of these two,

they will be created with all neccessary information.

After this simply do a

recover database

alter database open

Proceeding with brtools is more safe, because it does some additional checks (for nologging stuff and more).

But as long as you have the backup and the logs, the DB can be recovered, even if you loose the controlfiles,

allthough it is a bit more complicate in that case.

Good luck

Volker

[volker_borowski2]

> 
> Kindly suggest to resolve this issue. We have not taken backup for last 11 days.
> 
> Regards,
> 
> Jitendra

Hello,

1) Replace defective hardware

2) repair filesystems

3) Restore your 11 days old backup

4) Recover with hopefully available Redologs of the last 11 days

5) open database

6) mission completed

Volker

[Former Member]

Hi Volker,

I have 11 days old backup alongwith all archive files but 2 days before I have added 2 more datafile so can I restore this old backup alongiwth the added 2 datafile.

Kindly help. It will be highly appreciable for this help.

Awaiting your reply.

Regards,

Jitendra

[Former Member]

Hi ,

This is not a problem related to SAP application or with oracle database. Please ask your storage team to look into it as this is error related to LPAR located on /dev/did/dsk/d5s0 .

Your storage (OS) guys will help you for same.

Thanks..

Mohit