In 11gR2 Grid Infrastructure and RAC
After the previous demonstration of adding a new DiskGroup, I now demonstrate migrating the DiskGroup to a new pair of disks.
First, I create a table in the Tablespace on that DiskGroup.
Next, I verify that the DiskGroup is currently on disk asmdisk.7 and that the two new disks that I plan to migrate the DiskGroup to are available as asmdisk.8 and asmdisk.9 (yes, unfortunately, they are on /fra, instead of /data1 or /data2 because I have run out of disk space in /data1 and /data2).
This I do from node1 :
Note how the ownership and permissions are set for the two new disks (see my previous post).
I now add the two new disks.
With ASM_POWERLIMIT set to 1, Oracle ASM automatically starts a REBALANCE operation. However, since I did *not* drop the existing asmdisk.7, Oracle will still continue to use it.
After a while, I confirm that the REBALANCE has completed. I can now drop asmdisk.7. Unfortunately, this will trigger a new REBALANCE !
NOTE : Note how I must specify the Disk NAME (not the PATH) for the DROP. When I added disks asmdisk.8 and asmdisk.9, I could have given then meaningful names as well. Oracle has automatically named them.
Ideally, what I should have done is to use the ADD and DROP command together. That way, I would have a single-pass REBALANCE required.
After a while, I run my validation queries on node2.
I have now accessed the table, tablespace, diskgroup and disks from node2 successfully. Disk asmdisk.7 is no longer part of the DiskGroup.
I can physically remove disk asmdisk7 from the storage.
The disk asmdisk.7 is no longer part of the storage. (Remember : All my disks here are on NFS).
.
.
.
After the previous demonstration of adding a new DiskGroup, I now demonstrate migrating the DiskGroup to a new pair of disks.
First, I create a table in the Tablespace on that DiskGroup.
[root@node1 ~]# su - oracle
-sh-3.2$ sqlplus
SQL*Plus: Release 11.2.0.2.0 Production on Sun Aug 24 22:17:28 2014
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Enter user-name: hemant/hemant
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> create table new_tbs_tbl
2 tablespace new_tbs
3 as select * from dba_objects
4 /
Table created.
SQL> select segment_name, bytes/1048576
2 from user_segments
3 where tablespace_name = 'NEW_TBS'
4 /
SEGMENT_NAME
--------------------------------------------------------------------------------
BYTES/1048576
-------------
NEW_TBS_TBL
9
SQL> select file_name, bytes/1048576
2 from dba_data_files
3 where tablespace_name = 'NEW_TBS'
4 /
FILE_NAME
--------------------------------------------------------------------------------
BYTES/1048576
-------------
+DATA3/racdb/datafile/new_tbs.256.855792859
100
SQL>
Next, I verify that the DiskGroup is currently on disk asmdisk.7 and that the two new disks that I plan to migrate the DiskGroup to are available as asmdisk.8 and asmdisk.9 (yes, unfortunately, they are on /fra, instead of /data1 or /data2 because I have run out of disk space in /data1 and /data2).
This I do from node1 :
SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options -sh-3.2$ exit logout [root@node1 ~]# [root@node1 ~]# su - grid -sh-3.2$ sqlplus SQL*Plus: Release 11.2.0.2.0 Production on Sun Aug 24 22:22:32 2014 Copyright (c) 1982, 2010, Oracle. All rights reserved. Enter user-name: / as sysasm Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - Production With the Real Application Clusters and Automatic Storage Management options SQL> l 1 select d.name, d.path 2 from v$asm_disk d, v$asm_diskgroup g 3 where d.group_number=g.group_number 4* and g.name = 'DATA3' SQL> / NAME ------------------------------ PATH -------------------------------------------------------------------------------- DATA3_0000 /data1/asmdisk.7 SQL> SQL> !sh sh-3.2$ ls -l /fra/asmdisk* -rwxrwxr-x 1 grid oinstall 1024000000 Aug 24 22:06 /fra/asmdisk.8 -rwxrwxr-x 1 grid oinstall 1024000000 Aug 24 22:07 /fra/asmdisk.9 sh-3.2$
Note how the ownership and permissions are set for the two new disks (see my previous post).
I now add the two new disks.
sh-3.2$ exit
exit
SQL> show parameter power
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_power_limit integer 1
SQL> alter diskgroup data3 add disk '/fra/asmdisk.8', '/fra/asmdisk.9';
Diskgroup altered.
SQL>
SQL> select * from v$asm_operation;
GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE
------------ ----- ---- ---------- ---------- ---------- ---------- ----------
EST_MINUTES ERROR_CODE
----------- --------------------------------------------
3 REBAL RUN 1 1 1 101 60
1
SQL>
With ASM_POWERLIMIT set to 1, Oracle ASM automatically starts a REBALANCE operation. However, since I did *not* drop the existing asmdisk.7, Oracle will still continue to use it.
After a while, I confirm that the REBALANCE has completed. I can now drop asmdisk.7. Unfortunately, this will trigger a new REBALANCE !
SQL> l
1* select * from v$asm_operation
SQL> /
no rows selected
SQL>
SQL> l
1 select d.name, d.path
2 from v$asm_disk d, v$asm_diskgroup g
3 where d.group_number=g.group_number
4* and g.name = 'DATA3'
SQL> /
NAME
------------------------------
PATH
--------------------------------------------------------------------------------
DATA3_0000
/data1/asmdisk.7
DATA3_0002
/fra/asmdisk.9
DATA3_0001
/fra/asmdisk.8
SQL>
SQL> alter diskgroup data3 drop disk '/data1/asmdisk.7';
alter diskgroup data3 drop disk '/data1/asmdisk.7'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15054: disk "/DATA1/ASMDISK.7" does not exist in diskgroup "DATA3"
SQL> alter diskgroup data3 drop disk 'DATA3_0000';
Diskgroup altered.
SQL>
SQL> select * from v$asm_operation;
GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE
------------ ----- ---- ---------- ---------- ---------- ---------- ----------
EST_MINUTES ERROR_CODE
----------- --------------------------------------------
3 REBAL RUN 1 1 2 102 120
0
SQL>
SQL> l
1* select * from v$asm_operation
SQL>
SQL> /
GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE
------------ ----- ---- ---------- ---------- ---------- ---------- ----------
EST_MINUTES ERROR_CODE
----------- --------------------------------------------
3 REBAL RUN 1 1 47 101 95
0
SQL> /
no rows selected
SQL>
NOTE : Note how I must specify the Disk NAME (not the PATH) for the DROP. When I added disks asmdisk.8 and asmdisk.9, I could have given then meaningful names as well. Oracle has automatically named them.
Ideally, what I should have done is to use the ADD and DROP command together. That way, I would have a single-pass REBALANCE required.
After a while, I run my validation queries on node2.
[root@node2 ~]# su - grid
-sh-3.2$ sqlplus
SQL*Plus: Release 11.2.0.2.0 Production on Sun Aug 24 22:42:39 2014
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Enter user-name: / as sysasm
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - Production
With the Real Application Clusters and Automatic Storage Management options
SQL> select d.name, d.path
from v$asm_disk d, v$asm_diskgroup g
where d.group_number=g.group_number
and g.name = 'DATA3' 2 3 4
5
SQL> l
1 select d.name, d.path
2 from v$asm_disk d, v$asm_diskgroup g
3 where d.group_number=g.group_number
4* and g.name = 'DATA3'
SQL> /
NAME
------------------------------
PATH
--------------------------------------------------------------------------------
DATA3_0002
/fra/asmdisk.9
DATA3_0001
/fra/asmdisk.8
SQL>
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - Production
With the Real Application Clusters and Automatic Storage Management options
-sh-3.2$ exit
logout
[root@node2 ~]# su - oracle
-sh-3.2$
-sh-3.2$ sqlplus
SQL*Plus: Release 11.2.0.2.0 Production on Sun Aug 24 22:44:10 2014
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Enter user-name: hemant/hemant
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> select count(*) from new_tbs_tbl;
COUNT(*)
----------
72460
SQL>
I have now accessed the table, tablespace, diskgroup and disks from node2 successfully. Disk asmdisk.7 is no longer part of the DiskGroup.
I can physically remove disk asmdisk7 from the storage.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - Production
With the Real Application Clusters and Automatic Storage Management options
-sh-3.2$ exit
logout
[root@node1 ~]# cd /data1
[root@node1 data1]# ls -l asmdisk.7
-rwxrwxr-x 1 grid oinstall 2048000000 Aug 24 22:39 asmdisk.7
[root@node1 data1]# rm asmdisk.7
rm: remove regular file `asmdisk.7'? y
[root@node1 data1]#
[root@node1 data1]# su - grid
-sh-3.2$ sqlplus
SQL*Plus: Release 11.2.0.2.0 Production on Sun Aug 24 22:50:18 2014
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Enter user-name: / as sysasm
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - Production
With the Real Application Clusters and Automatic Storage Management options
SQL> set pages 60
SQL> col name format a15
SQL> col path format a20
SQL> select group_number, name, path
2 from v$asm_disk
3 order by 1,2;
GROUP_NUMBER NAME PATH
------------ --------------- --------------------
0 /crs/voting.disk
0 /data1/votedisk.1
0 /data2/votedisk.2
0 /fra/votedisk.3
1 DATA1_0000 /data1/asmdisk.1
1 DATA1_0001 /data2/asmdisk.4
2 DATA2_0000 /data1/asmdisk.2
2 DATA2_0001 /data2/asmdisk.5
2 DATA2_0002 /data2/asmdisk.6
3 DATA3_0001 /fra/asmdisk.8
3 DATA3_0002 /fra/asmdisk.9
4 DATA_0000 /crs/ocr.configurati
on
5 FRA_0000 /fra/fradisk.3
5 FRA_0001 /fra/fradisk.2
5 FRA_0002 /fra/fradisk.1
5 FRA_0003 /fra/fradisk.4
16 rows selected.
SQL>
The disk asmdisk.7 is no longer part of the storage. (Remember : All my disks here are on NFS).
.
.
.