SETUP ASM&ASMLib on
Multipath Disks
1. Install Correct
Packages
We must be careful while downloading
correct rpms. Before download the rpms, check your kernel version and
processor(32 bit or 64 bit):
oracleasm-(kernel) --- must be same version with the kernel
oracleasmlib ---
can be the latest version of correct 32 or 64 bit version
oracleasm-support --- can be the latest version of
correct 32 or 64 bit version
[root@racnode1 ~]# uname -a
Linux racnode1.msb.bak 2.6.18-194.el5 #1
SMP Tue Mar 16 21:52:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
[root@racnode1 ~]# rpm -qa | grep asm
oracleasm-support-2.1.8-1.el5
oracleasmlib-2.0.4-1.el5
oracleasm-2.6.18-194.el5-2.0.5-1.el5
[root@racnode1 ~]#
if you downloaded and installed wrong
“oracleasm-(kernel)” rpm, you can get the following errors:
[root@racnode1 bin]# /etc/init.d/oracleasm
start
Initializing the Oracle ASMLib driver:
[FAILED]
[root@racnode1 bin]# more
/var/log/oracleasm
"/var/log/oracleasm"
"/var/log/oracleasm" 145L,
5999CCreating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm":
failed
Unable to load module "oracleasm"
Creating /dev/oracleasm mount point:
/dev/oracleasm
Loading module "oracleasm":
failed
Unable to load module "oracleasm"
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
oracleasm-read-label: Unable to open device
"/dev/sdb1": No such file or directory
oracleasm-read-label: Unable to open device
"/dev/dm-0": No such file or directory
oracleasm-read-label: Unable to open device
"/dev/dm-0": No such file or directory
oracleasm-read-label: Unable to open device
"/dev/dm-0": No such file or directory
oracleasm-read-label: Unable to open device
"/dev/dm-0": No such file or directory
oracleasm-read-label: Unable to open device
"/dev/dm-1": No such file or directory
oracleasm-read-label: Unable to open device
"/dev/dm-1": No such file or directory
oracleasm-read-label: Unable to open device
"/dev/dm-1": No such file or directory
oracleasm-read-label: Unable to open device
"/dev/dm-1": No such file or directory
2. Check your
disks
[oracle@racnode1 ~]$ ls -lrt
/dev/mapper/
total 0
brw-rw---- 1 root disk 253, 8 May 15 12:47 oradata2p1
brw-rw---- 1 root disk 253, 9 May 15 12:47 oradata3p1
brw-rw---- 1 root disk 253, 10 May
15 12:47 oradata1p1
3. Create ASM
disks
[root@racnode1 ~]#
/etc/init.d/oracleasm createdisk ASM1 /dev/mapper/oradata1p1
[root@racnode1 ~]# /etc/init.d/oracleasm
createdisk ASM2 /dev/mapper/oradata2p1
4. For RAC
systems, run the command for every node
[root@racnode1 ~]#
/etc/init.d/oracleasm scandisks
Scanning the system for Oracle
ASMLib disks: [ OK ]
[root@racnode1 ~]#
/etc/init.d/oracleasm listdisks
ASM1
5. Configure
ASMLib for multipath (do the following operations for every node)
Modify /etc/sysconfig/oracleasm
ORACLEASM_SCANORDER=”dm”
ORACLEASM_SCANEXCLUDE=”sd”
Restart asmlib
/etc/init.d/oracleasm stop
/etc/init.d/oracleasm start
6. Check disk
configuration
ASMLib checks the
/proc/partitions to detect the disks:
[root@racnode1 ~]# cat
/proc/partitions | grep dm-
253
0 90963968 dm-0
253
1 51478528 dm-1
253
2 536870912 dm-2
253
3 536870912 dm-3
253
4 536870912 dm-4
253
5 536870912 dm-5
253
6 104857600 dm-6
253
7 536868171 dm-7
253 8
536868171 dm-8
253
9 536868171 dm-9
253
10 536868171 dm-10
253
11 104856223 dm-11
7. ASM disks are
mounted under the /dev/oracleasm/disks
root@racnode1 ~]# ls -la
/dev/oracleasm/disks
brw-rw---- 1 oracle oinstall 253,
10 May 15 12:48 ASM1
brw-rw---- 1 oracle oinstall
253, 8 May 15 12:48 ASM2
8. Major and Minor
numbers of the disks have to be same in both /proc/partitions/ and /dev/oracleasm/disks/
/proc/partitions 253
10 536868171 dm-10
brw-rw---- 1 oracle oinstall 253,
10 May 15 12:48 ASM1
[root@racnode1 ~]# /etc/init.d/oracleasm
querydisk -d ASM1
Disk "ASM1" is a valid ASM disk
on device /dev/dm-10[253,10]
Hi Emre ;
YanıtlaSilI woluld like to add some experince about this subject.
Once our os guys changed disk acces to/from multipath witout rebooting , asmlib has some socket files or something as a latch in database :) , causing failures.
------------- How to discard module libasm.so -----------
Running "/etc/init.d/oracleasm scandisks" command is not good enough after changing ORACLEASM_SCANORDER="dm" from /etc/sysconfig/oracleasm file. ASMlib should be restarted using the following way.
1. Shutdown all databases.
2. Shutdown ASM instance
3. Make sure that there is no process loading libasm.so and no user is in /dev/oracleasm directory before restarting ASMLib.
# lsof | grep libasm.so | grep -v grep | awk -F ' ' '{print $2}' | xargs ps
# lsof | grep /dev/oracleasm | grep -v grep | awk -F ' ' '{print $2}' | xargs ps
Otherwise, ASMLib restart will fail with the following message.
# /sbin/service oracleasm restart
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [FAILED]
4. Restart ASMLib
As root,
# /sbin/service oracleasm restart
5. Check
5-1. v$asm_disk.header_status column shows proper status - PROVISIONED/MEMBER/FORMER etc. except "UNKNOWN".
5-2. All ASM devices under /dev/oracleasm/disks/* shows major# as "253" or the one for the multipath devices seen from the output of "cat /proc/partitions".
$ ls -l /dev/oracleasm/disks/*
brw-rw---- 1 oracle oinstall 253, 1 Apr 23 22:53 /dev/oracleasm/disks/CRS1
brw-rw---- 1 oracle oinstall 253, 2 Apr 23 22:53 /dev/oracleasm/disks/DATA1
5-3. Check whether GRID OUI shows all ASMLib devices with "ORCL:*' discovery string.
thanks for ur extra sharing about the topic.
YanıtlaSil