SAP+Oracle on ASM

[Former Member]

Hello SCN members. I want to perform some research regarding ASM technology in Oracle.
Do you use ASM in your SAP installations without RAC ?
If yes , how it works in terms of stability , have you faced some ASM related issues or it works like a conventional FS without any major problems ?

Thanks.

[stefan_koehler · Accepted solution]

Hi Sergo,

i asked the same question some time ago regarding the usage of ASM and single instances. Please check this thread as Fidel replied:

> If yes , how it works in terms of stability , have you faced some ASM related issues or it works like a conventional FS without any major problems ?

However as i already mentioned, it is state-of-the-art in Oracle RAC and the code does not differ rather it is much easier as you have not to handle concurrency, voting disks, etc. Yes it is very stable, but it does not work like a conventional FS (disregarding ACFS for now). But in my opinion it does only make sense for single instances, if you have a highly consolidated environment (e.g. all your SAP Oracle databases on one central cluster / server or separated by DEV / QA / PRD) otherwise the overhead of the GI stack (for single instances) is too heavy and you can not make use of most of the ASM features.

Regards

Stefan

[Former Member · Accepted solution]

Hi Stefan ,
      thank you for the link provided.
We have approx 10 test systems moved to ASM (single instance 1host=1GRID).
Disks management and operations with datafiles look more convenient with ASM then with

FS, but we can't measure and evaluate performance (need to have additional test there , had no time yet for  this) but it looks no worse in general.
On one system , one of the admins destroyed database,
FS was created on top of ASM disk , this looks as really problem in such configurations. 

Our management want to move Prod systems as well on ASM , so now we researching information
about successful installations (or not successful ).

[stefan_koehler · Accepted solution]

Hi Sergo,

> We have approx 10 test systems moved to ASM (single instance 1host=1GRID)

Wow, then i guess you will have a lot of fun patching each GI and RDBMS separately

> but we can't measure and evaluate performance (need to have additional test there , had no time yet for  this) but it looks no worse in general.

In environments with higher I/O load (not very typical for plain test systems), you may have to care about stripe sizes (templates), etc. However this depends on your storage and environment of course.

> On one system , one of the admins destroyed database, FS was created on top of ASM disk , this looks as really problem in such configurations.

A classic one However you may want to have a look at Oracle Automatic Storage Management Filter Driver (ASMFD) to avoid such issues: Oracle Automatic Storage Management Filter Driver (Oracle ASMFD)

Regards

Stefan

[Former Member · Accepted solution]

Hi Stefan ,

yes , i have played with AFD on one system , but it only prevents from incorrect calls from applications, it can't stop tough Admin guys from UNIX team.

About "stripe sizes" you mean AU size ? Yes , this is not completely clear to which value set this parameter ....

[stefan_koehler · Accepted solution]

Hi Sergo,

> i have played with AFD on one system , but it only prevents from incorrect calls from applications, it can't stop tough Admin guys from UNIX team.

Sorry, i don't get this as ASMFD also prevents even root from doing any modifications on/to the disc. Please verify section "Verify that when filter is enabled it does not allow OS commands to alter the contents of ASM disks" in the previous mentioned blog post.

> About "stripe sizes" you mean AU size ? Yes , this is not completely clear to which value set this parameter ....

No, i mean strip size and the corresponding ASM templates. AU size also matters in case of large I/O requests (e.g. FTS in BI systems), but that is a different topic.

Regards

Stefan

[Former Member · Accepted solution]

Hi Stefan ,
i was confused by information from several blogs , there was stated that AFD can't help from root

commands. I have just tried and here my findings :

/home/oracle> asmcmd afd_lsdsk

-----------------------------------------------------

Label                          Filtering       Path

=================================

DATA0000                 ENABLED   /dev/sdp1

RECO0000                ENABLED   /dev/sdq1

OCR                           ENABLED   /dev/sdr1

/root # pvcreate /dev/sdq

Device /dev/sdq not found (or ignored by filtering).

root # dd if=/dev/zero of=/dev/sdq bs=8192  count=10000

10000+0 records in

10000+0 records out

81920000 bytes (82 MB) copied, 0.266769 s, 307 MB/s

kernel: [42030759.191525] F 59323018.604/150617131208 flush-65:0[32441]  afd_mkrequest_fn: write IO on ASM managed device (major=65/minor=1)  not supported i=1 start=159984 seccnt=8  pstart=2048  pend=419428352

So looks like it really helps , good feature.

About  "strip size and the corresponding ASM templates" , i don't know what you mean  .  Yet.

Will read documentation

[stefan_koehler · Accepted solution]

Hi Sergo,

to make it a little bit easier

1) Silently changed stripe size in 11g R2

• Blog: https://community.emc.com/people/DarrylBSmith/blog/2012/01/06/where-did-that-fine-grained-striping-g...
• This is also my experience with various storage systems, but you have to test it as it depends on the environment

2) AU size - I/O request

• Physical IO on Linux | Frits Hoogland Weblog
• This depends on your I/O load profile of the system(s)

3) ASM templates

• Administering Oracle ASM Files, Directories, and Templates

Regards

Stefan

[Former Member · Accepted solution]

Thank you Stefan as usual for your time and assistance.

I have performed some test with ASM Filter Driver , and still have some doubts.

Finally after tests which i showed above , after reboot of server,  partition where ASM(AFD)

disk was located disappeared. 

I was able to restore partition , and data on it was not destroyed (at least everything was accessible).
So additional test was done , using parted existing partition was destroyed again "parted /dev/sdq" and  rm 1. After reboot the same situation , partition has been lost.

As you know "oracleasm createdisk" worked only on top of some partition , so if we will migrate from ASMLIB to ASMFD , root still can destroy partitions.
I have backed up database , and dropped all ASM diskgroups following creation new ones
directly on devices (without partitions, they were dropped as well)
asmcmd afd_label DATA0000 /dev/sdk .

Database was restored and recovered without any issue, and the same tests were performed again.

/root # parted /dev/sdk

GNU Parted 2.3

Using /dev/sdk

Welcome to GNU Parted! Type 'help' to view a list of commands.

(parted) mklabel

New disk label type? GPT

Error: Input/output error during write on /dev/sdk

Retry/Ignore/Cancel? c

(parted) q

Jun 23 04:46:39 kernel: [37657.419180] F 17218400.338/150623084639 parted[29406] afd_mkrequest_fn: write IO on ASM managed device (major=8/minor=160)  not supported i=0 start=0 seccnt=8  pstart=0  pend=44040192

/root # fdisk /dev/sdk

WARNING: GPT (GUID Partition Table) detected on '/dev/sdk'! The util fdisk doesn't support GPT. Use GNU Parted.

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel

Building a new DOS disklabel with disk identifier 0xdbbdd492.

Changes will remain in memory only, until you decide to write them.

After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): o

Building a new DOS disklabel with disk identifier 0x648291e9.

Changes will remain in memory only, until you decide to write them.

After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Jun 23 04:48:21  kernel: [37758.618779] F 17218501.809/150623084821 flush-8:160[29818]  afd_mkrequest_fn: write IO on ASM managed device (major=8/minor=160)  not supported i=0 start=0 seccnt=8  pstart=0  pend=44040192

Jun 23 04:48:21  kernel: [37758.618800] quiet_error: 6 callbacks suppressed

Jun 23 04:48:21  kernel: [37758.618804] Buffer I/O error on device sdk, logical block 0

Jun 23 04:48:21  kernel: [37758.618807] lost page write due to I/O error on sdk

Jun 23 04:48:21  kernel: [37759.122873] sd 0:0:11:0: [sdk] Cache data unavailable

Jun 23 04:48:21  kernel: [37759.122878] sd 0:0:11:0: [sdk] Assuming drive cache: write through

Jun 23 04:48:21  kernel: [37759.123913]  sdk: unknown partition table

root # pvcreate /dev/sdk

  /dev/sdk: write failed after 0 of 4096 at 4096: Input/output error

  Failed to wipe new metadata area

  /dev/sdk: Format-specific setup of physical volume failed.

  Failed to setup physical volume "/dev/sdk"

root # dd if=/dev/zero of=/dev/sdk bs=1024 count=100000

100000+0 records in

100000+0 records out

102400000 bytes (102 MB) copied, 10.0883 s, 10.2 MB/s

After reboot , everything is ok now.

I couldn't find any information regarding the creation of AFD disks on devices directly , is this supported and should be done in this way ?

Thanks , Sergo.

[stefan_koehler · Accepted solution]

Hi Sergo,

to be honest most of my clients are not using ASMLIB, they (and myself) prefer the udev way. So in consequence i can not make a wide statement about ASMLIB and the needed partitions. By the way you only have this issue with "oracleasm createdisk", but if you would use "asmtool" instead you even can create the ASMLIB header without partitions (e.g. ASMLIB createdisk problem on emcpower devices).

I also cross-checked the official documentation and MOS and could not find any limitation with ASMFD, so i guess they also removed this restriction in the official command-set. But this is only an educated guess for ASMFD. An Oracle SR would be the best option in this case i guess.

Regards

Stefan

[Former Member · Accepted solution]

Hi Stefan ,

what do you mean > By the way you only have this issue with "oracleasm createdisk" ?
In link you have provided , they can't create oracleasm disk even if they have partition -->
"Marking disk "/dev/emcpowera1" as an ASM disk:"
asmtool: Device "/dev/emcpowera1" is not a partition

In Oracle documentation and in any other sources everywhere used partitions for "oracleasm createdisk"

and moreover only one single partition preferable.

1. Note:Oracle recommends that you create a single whole-disk partition on each disk to use.
2. Use either fdisk or parted to create a single whole-disk partition on the disk devices to use.

[stefan_koehler · Accepted solution]

Hi Sergo,

i have no ASMLIB installation at hand, but just try the "asmtool" (with option "-a force=yes") command on a disk without a partition. I am pretty sure, that you are able to use this disk with ASMLIB afterwards as well.

I just wanted to say, that you were also able to create/use a non-partitioned disk with ASMLIB in the past (with the asmtool work-around) and it seems like that this (default) restriction has been removed with ASMFD. However Oracle SR is the only option for confirmation about ASMFD

Regards

Stefan

[Former Member · Accepted solution]

Hi Stefan, just for information, answer from oracle about ASMFD on top of the disk -->

Yes, it is possible.

Regards,

xxxxxx

OCS