Contents
CONTENTS 1
1.1 硬件 1
1.2 软件环境 1
1.3 虚拟机配置 1
1.4 共享存储配置 1
1.5 ORACLE RAC 数据库环境 2
1.6 网络配置 2
1.7 物理结构图 2
1.8 逻辑结构图 2
1.9 创建第一个虚拟机 3
1.10 增加两个虚拟网卡 3
1.11 添加仲裁盘 3
1.12 安装操作系统 4
1.13 安装 VMWARE TOOLS 8
1.14 时钟同步 8
1.15 虚机克隆 8
1.16 配置仲裁盘 8
1.17 使用FDISK格式化仲裁盘： 9
介绍
本文通过一个案例简要介绍 Oracle Real Application Clusters (RAC)在VMware vSphere®平台上的安装过程,帮助Oracle 数据库管理员在VMware 平台上规划和实施Oracle RAC环境.在VMware环境下,数据库管理员通常需要和VMware的管理员以及存储厂商的管理员合作,以顺利部署一个Oracle RAC环境.
在虚拟化平台上部署Oracle RAC在许多方面与纯物理环境上的部署很类似,比如:
在虚拟机创建完成之后,安装操作系统和Oracle RAC软件的过程与物理环境下基本一致
存储的规划,包括用于数据存放的LUN数量的规划和多路径的设计与物理环境也基本一致.
差别在于我们可以充分依靠虚拟化软件, 简化许多工作, 从而加快安装进度,减少错误的发生,比如:
利用事先准备好的模板, 多次安装同版本的系统和数据库,同时加强标准化
利用快照技术, 很方便的进行测试和回退
利用虚拟机复制技术,简化多个RAC节点的系统安装、打补丁、配置工作
由于安装过程的细节与软件版本（尤其是Linux版本和Oracle版本）有较强的关联性，本文只能尽量把安装步骤描述清楚.读者在自己进行安装的时候，还需要参考相应版本的官方安装文档.
先将本次安装中用到的主要软件和版本罗列如下：
VMware ESXi™5.1
VMware vCenter Server™5.1
vSphere Client 5.1
Oracle Database 11g 11.2.0.3.
Oracle Enterprise Linux x86_64 6.3.
软硬件环境
1.1 硬件
组件 数量 配置
ESXiHosts 2 Cisco UCS B440 4x 10-core CPUs, 256GB RAM, and 4 NICs
Storage 1 NetApp F3240双路控制器
Ethernet Switches 2 Cisco FI6248
FC Switches 1 Cisco Nexus 7000
Host Based Adapters 4 每个刀片两个，与网卡共享流量
1.2 软件环境
组件 版本 作用
vSphere 5.1 Hypervisor hosting all virtual machines
VMware vCenter™ 5.1 管理 vSphere
Oracle Enterprise Linux x86_64 6.3 Guest operatingsystemforvirtual machines running Oracle Database server
Oracle Database 11g (with Oracle RAC and Oracle Grid Infrastructure) Enterprise Edition 11.2.0.3
Microsoft Windows Server 2008 R2 用于 vCenter Server
1.3 虚拟机配置
虚拟机用途 数量 配置
Oracle RAC nodes 2 20vCPU, 120GB RAM,Oracle Enterprise Linux x86_64 6.3
vCenter 1 2 vCPUs, 8GB RAM, Windows Server 2008 R2
1.4 共享存储配置
Datastore Name Virtual Disk on ESX Linux路径 虚拟路径 驱动类型 VMDK Size(GB) 用途
VMDATASTORE VMDK – Hard disk 1 /dev/sda SCSI 0:0 LSI Logic 150 Oracle Enterprise Linux 5.5 OS and Oracle binaries
CRS1 VMDK – Hard disk 2 Shared disk
/dev/sdb1 SCSI 1:0 Paravirtual 20 CRS and voting disk
REDO1 VMDK – Hard disk 3 Shared disk
/dev/sdc1 SCSI 1:1 Paravirtual 900 REDO and Archieve
DATA01 VMDK – Hard disk 4 Shared disk
/dev/sdd1 SCSI 2:0 Paravirtual 1000 RAC database DATA
DATA02 VMDK – Hard disk 5 Shared disk
/dev/sde1 SCSI 2:1 Paravirtual 1000 RAC database DATA
DATA03 VMDK – Hard disk 6 Shared disk
/dev/sdf1 SCSI 3:0 Paravirtual 1000 RAC database DATA
DATA04 VMDK – Hard disk 7 Shared disk
/dev/sdg1 SCSI 3:1 Paravirtual 1000 RAC database DATA
1.5 Oracle RAC 数据库环境
Node Name ASM Instance RAC Instance Database ASM Storage
RAC1 +ASM1 RAC1 RAC ASM file system
RAC2 +ASM2 RAC2 RAC ASM file system
1.6 网络配置
按照Oracle RAC的实施惯例，每个RAC节点都提供一个Public网口用于承载业务，并在其上捆绑虚拟IP地址 (VIP)用于HA切换…
各RAC节点之间还需要一个私有网络进行节点之间的通信。
在物理环境中，通常需要考虑链路的冗余以消除单点鼓掌。而在虚拟化环境中，冗余问题由底层虚拟化软件处理，所以对于每个RAC节点，只是简单地需要两个网口就可以工作，大大简化了配置和安装工作.
我们这次测试的网络配置如下三:
• UCS底层软件配置出三个物理网口（均为10Gb）– vmnic0和 vmnic1用于Oracle Public and Oracle Private网络, vmnic2用于vSphere®vMotion®工作.
• 两个分布式虚拟交换机– Oracle Public (VM Net) and Oracle Private (Oracle HB).
• 每个RAC节点配置3个IP地址– 以节点1 (RAC1)为例:
o Onepublic static IP address (eth0): 162.16.3.101
o Oneprivate static IP address (eth1): 192.168.2.30
o Onevirtual static IP address(VIP): 162.16.3.111（由CRS软件配置和管理）
还有三个虚拟IP用于节点间负载均衡(SCAN): 162.16.3.200,162.16.3.201,162.16.3.202（由CRS配置和管理）
物理网口 vmnic0 连接到分布式交换机 VM Net,vmnic1连接到分布式交换机Oracle HB， vMotion使用标准交换机vSwitch3.这样区分概念上比较清晰，也可以只建一个分布式网络，用不同的端口组区分数据网络和心跳网。
Figure 1.Network Configuration

1.7 物理结构图
Figure 2. Physical Architecture of Oracle RAC on vSphere

1.8 逻辑结构图
Figure 3. Logical Architecture of Two-Node Oracle RAC on vSphere

部署流程概览
部署流程大体按照下面的流程图进行.
Figure 4. Deployment Steps for Four-Node Oracle RAC on vSphere with VMFS

为第一个RAC节点准备虚拟机环境
本节安装步骤简介如下:
• 使用vSphere Client创建第一个虚拟机。添加公网和心跳网卡，挑选适当的LUN创建仲裁盘CRS （VMDK模式）。
• 安装Redhat Linux操作系统，安装VMware tools，按照Oracle RAC要求安装相应补丁.
• 在Linux中配置NTP 客户端.
1.9 创建第一个虚拟机

1. 使用vSphere Client登录vCenter.

2. 选中一台主机,在Getting Started页面单击Create a new virtual machine.

3. In the Create New Virtual Machinewizard, select Typical for the Configuration, and click Next.

4. On the Name and Location page, enter RAC1 for the Name, and click Next.

5. On the Datastore page, select VMDATASTORE, and click Next.

6. On the Guest Operating System page, select Linux from the Guest Operating System list. For the Version, select Oracle Linux 6 (64-bit) from the drop-down menu. Click Next.

7. On the Create a Disk page, set the Virtual Disk Size to 20GB, and click Next.

8. On the Ready to Complete page, select Edit the virtual machine settings before completion. Click Continue.

9. On the Virtual Machine Properties page for theRAC1 virtual machine, select 8CPUs and 128GBMemory.

10. Click Finish.
    1.10 增加两个虚拟网卡
    现在需要增加两块网卡. 一块用于公共数据网络, 另一块用于心跳网络.添加这两块网卡的过程基本相同,差别仅在于添加公网网卡时选择标签:VM Net而添加心跳网卡时选择Oracle HB.
    添加公网网卡的步骤：

11. Select a virtual machine.

12. Click Edit settings to display the Virtual Machine Properties pane.

13. Click Add.

14. Select Ethernet Adaptor and click Next to display the Add Hardware wizard.

15. On the Network connection page, select VMXNET 3 from the Adapter Type drop-down menu.

16. In the Network Connection section of the Network connection page, select Named network with specified label.

17. In the Named network with specified label drop-down menu, select VM Net.

18. In the Device Status section of the Network connection page, select Connect at power on.

19. On the Ready to Complete page, click Finish, and then click OK.
    添加心跳网卡的步骤：

20. Repeat steps 1 through 6 from the preceding procedure.

21. In the Named network with specified label drop-down menu, select Oracle HB.

22. Continue with steps 8 and 9 from the preceding procedure.
    1.11 添加仲裁盘
    Oracle RAC需要至少一个共享的仲裁盘存放clusterware信息和节点投票信息。存储管理员需要事先为仲裁盘分配好单独的LUN，然后我们需要创建单独的SCSI控制器并设为Paravirtual驱动.SCSI 总线sharing policy 设为none.
    下面描述创建共享存储的步骤.
    为虚拟机创建10GB datastore：

23. In vSphere Client, select a virtual machine.

24. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.

25. Click Addto display the Add Hardware wizard.

26. In the Add Hardware wizard select Hard Disk for the Device Type, and click Next.

27. In the Disk section of the Select a Disk page, select Create a new virtual disk. Set the disk size to 10GB, select the Support clustering features such as Fault Tolerance check box, and select CRS1 to specify a datastore. Also, for vSphere 5, in the Disk Provisioning section select Thick Provision Eager Zeroed. Click Next.

28. On the Compatibility Mode page, click Next.

29. On the Advanced Options page, in the Virtual Device Node section, select SCSI (1:0). In theMode section, selectIndependent. Click Next.

30. On the Ready to Complete page, click Finish.

31. Click OK.
    Add configuration parameters for each Oracle RAC virtual machine as described in Disabling simultaneous write protection provided by VMFS using the multi-writer flag(http://kb.vmware.com/kb/1034165).
    手工修改虚拟机参数

32. In vSphere Client, select a virtual machine.

33. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.

34. Click the Options tab, click Advanced, and click General.

35. Click Configuration Parameters.

36. On the Configuration Parameters page, add rows and entries for the shared disks in the virtual machine’s configuration parameters as shown in the following screenshot.

37. Repeat steps 1 through 5 for each of the remaining virtual machines.
    
    You can view the virtual machine propertiesfor the RAC1 virtual machinein vCenteras in the following screenshot.
    
    web client 显示如下图：
    
    1.12 安装操作系统
    将OEL6.4安装介质ISO文件上传到datastore，然后进行安装。
    Note: This document is specific to Linux guest operating system. Oracle RAC 11.2.0.2 and above versions can be deployed and supported on the list ofguest operation systems usingVMware multi-writer flag(http://kb.vmware.com/kb/1008027).
    在虚拟机上安装Linux操作系统

38. In vSphere Client, select a virtual machine.

39. Click Edit settings.

40. In the Hardware tab of the Virtual Machine Properties pane, selectCD/DVD Drive 1.

41. In the Device Status section of this pane, select the Connect at power on check box.

42. In the Device Type section of this pane, select Datastore ISO File and enter the path to your Linux ISO image file.

43. In the Virtual Device Node section of this pane, select CD/DVD Drive 1 from the drop-down menu.

44. Click OK.

45. Power on the virtual machine.

46. Install Linux with the following settings.
    Custom partitioning scheme:
    /dev/sda1 ext3 100GB mounted on /
    Installed packages – use the default package selection, plus the following:
    • kernel-uek.x86_64
    • kmod-oracleasm
    • oracleasm-2.6.18-194.el5-2.0.5-1.el5.x86_64 (download from
    http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html.)
    • oracleasm-support-2.1.3-1.el5.x86_64
    • systat-7.0.2-3.el5.x86_64

47. Use below script to install remaining packages:
    yum install binutils
    compat-libcap1
    compat-libstdc+±33
    compat-libstdc+±33.i686
    gcc.x86_64
    gcc-c++.x86_64
    glibc.x86_64
    glibc.i686
    glibc-devel.x86_64
    glibc-devel.i686
    unixODBC.i686
    unixODBC.x86_64
    unixODBC-devel.x86_64
    unixODBC-devel.i686
    libgcc.x86_64
    libgcc.i686
    libstdc++.x86_64
    libstdc++.i686
    libstdc+±devel.x86_64
    libstdc+±devel.i686
    compat-libstdc+±33.x86_64
    compat-libstdc+±33.i686
    libaio.x86_64
    libaio.i686
    libaio-devel.x86_64
    libaio-devel.i686
    sysstat.x86_64
    ksh
    libXtst-devel.x86_64
    libXtst.i686
    compat-db.x86_64
    xorg-x11-utils.x86_64
    xorg-x11-xinit.x86_64
    make

48. View the installed packages with this command:
    run command “rpm –q

49. Confirm that the following packages have been installed:
    • binutils-2.20.51.0.2-5.11.el6 (x86_64)
    • compat-libcap1-1.10-1 (x86_64)
    • compat-libstdc+±33-3.2.3-69.el6 (x86_64)
    • compat-libstdc+±33-3.2.3-69.el6.i686
    • gcc-4.4.4-13.el6 (x86_64)
    • gcc-c+±4.4.4-13.el6 (x86_64)
    • glibc-2.12-1.7.el6 (i686)
    • glibc-2.12-1.7.el6 (x86_64)
    • glibc-devel-2.12-1.7.el6 (x86_64)
    • glibc-devel-2.12-1.7.el6.i686
    • ksh
    • libgcc-4.4.4-13.el6 (i686)
    • libgcc-4.4.4-13.el6 (x86_64)
    • libstdc+±4.4.4-13.el6 (x86_64)
    • libstdc+±4.4.4-13.el6.i686
    • libstdc+±devel-4.4.4-13.el6 (x86_64)
    • libstdc+±devel-4.4.4-13.el6.i686 • libaio-0.3.107-10.el6 (x86_64)
    • libaio-0.3.107-10.el6.i686
    • libaio-devel-0.3.107-10.el6 (x86_64)
    • libaio-devel-0.3.107-10.el6.i686
    • make-3.81-19.el6
    • sysstat-9.0.4-11.el6 (x86_64)
    • unixODBC-2.2.14-11.el6 (x86_64) or later
    • unixODBC-2.2.14-11.el6.i686 or later
    • unixODBC-devel-2.2.14-11.el6 (x86_64) or later
    • unixODBC-devel-2.2.14-11.el6.i686 or later

50. 最后安装下面两个包:
    • oracle-rdbms-server-11gR2-preinstall (从YUM安装，这个包会配置Oracle RAC所需系统参数和环境变量)
    • cvuqdisk (Oracle安装介质中提供这个包，用于安装前检查)

51. 检查配置文件
    /etc/sysctl.conf
    kernel.shmmni = 4096
    kernel.sem = 250 32000 100 128
    fs.file-max = 6815744 (512 x processes for example 6815744 for 13312 processes)
    net.ipv4.ip_local_port_range = 9000 65500
    net.core.rmem_default = 262144
    net.core.rmem_max = 4194304
    net.core.wmem_default = 262144
    net.core.wmem_max = 1048576
    kernel.shmall = physical RAM size / pagesize For most systems, this will be the value 2097152. See Note: 301830.1 for more information.
    eg:
    kernel.shmall=4160749568(62GB)
    kernel.shnmax= 34359738368(32GB)
    kernel.shmmax = RAM times 0.5 (or higher at customer’s discretion - seeNote:567506.1)

/etc/profile
ulimit –u 16384
ulimit –n 65535
15. 依照oracle传统，创建clusterware主用户 “grid”:
[root@racnode1 ~]# groupadd -g 1000 oinstall
[root@racnode1 ~]# groupadd -g 1200 asmadmin
[root@racnode1 ~]# groupadd -g 1201 asmdba
[root@racnode1 ~]# groupadd -g 1202 asmoper
[root@racnode1 ~]# useradd -m -u 1100 -g oinstall -G asmadmin,asmdba,asmoper -d
/home/grid -s /bin/bash -c “Grid Infrastructure Owner” grid
[root@racnode1 ~]# id grid uid=1100(grid) gid=1000(oinstall)
groups=1000(oinstall), 1200(asmadmin), 1201(asmdba),1202(asmoper)
设置 grid 帐户的口令：
[root@racnode1 ~]# passwd grid
Changing password for user grid.
New UNIX password:xxxxxxxxxxx
Retype new UNIX password:xxxxxxxxxxx
passwd: all authentication tokens updated successfully.
为grid用户创建初始化脚本(/home/grid/.bash_profile):

---------------------------------------------------

.bash_profile

---------------------------------------------------

OS User: grid

Application: Oracle Grid Infrastructure

Version: Oracle 11g release 2

---------------------------------------------------

Get the aliases and functions

if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

alias ls=“ls -FA”

---------------------------------------------------

ORACLE_SID

---------------------------------------------------

Specifies the Oracle system identifier (SID)

for the Automatic Storage Management (ASM)instance

running on this node.

Each RAC node must have a unique ORACLE_SID.

(i.e. +ASM1, +ASM2,…)

---------------------------------------------------

ORACLE_SID=+ASM1; export ORACLE_SID(注意节点2这里应改为 +ASM2)

---------------------------------------------------

JAVA_HOME

JAVA_HOME=/usr/local/java; export JAVA_HOME

ORACLE_BASE=/u01/app/grid; export ORACLE_BASE

---------------------------------------------------

ORACLE_HOME

---------------------------------------------------

Specifies the directory containing the Oracle

Grid Infrastructure software. For grid

infrastructure for a cluster installations, the Grid

home must not be placed under one of the Oracle base

directories, or under Oracle home directories of

Oracle Database installation owners, or in the home

directory of an installation owner. During

installation, ownership of the path to the Grid

home is changed to root. This change causes

permission errors for other installations.

---------------------------------------------------

ORACLE_HOME=/u01/app/11.2.0/grid; export ORACLE_HOME

---------------------------------------------------

ORACLE_PATH

---------------------------------------------------

Specifies the search path for files used by Oracle

applications such as SQL*Plus. If the full path to

the file is not specified, or if the file is not

in the current directory, the Oracle application

uses ORACLE_PATH to locate the file.

This variable is used by SQL*Plus, Forms and Menu.

---------------------------------------------------

ORACLE_PATH=/u01/app/oracle/common/oracle/sql; export ORACLE_PATH

---------------------------------------------------

SQLPATH

---------------------------------------------------

Specifies the directory or list of directories that

SQL*Plus searches for a login.sql file.

---------------------------------------------------

SQLPATH=/u01/app/common/oracle/sql; export SQLPATH

---------------------------------------------------

ORACLE_TERM

---------------------------------------------------

Defines a terminal definition. If not set, it

defaults to the value of your TERM environment

variable. Used by all character mode products.

---------------------------------------------------

ORACLE_TERM=xterm; export ORACLE_TERM

---------------------------------------------------

NLS_DATE_FORMAT

---------------------------------------------------

Specifies the default date format to use with the

TO_CHAR and TO_DATE functions. The default value of

this parameter is determined by NLS_TERRITORY. The

value of this parameter can be any valid date

format mask, and the value must be surrounded by

double quotation marks. For example:

# NLS_DATE_FORMAT = “MM/DD/YYYY”

# ---------------------------------------------------

NLS_DATE_FORMAT=“DD-MON-YYYY HH24:MI:SS”; export NLS_DATE_FORMAT

---------------------------------------------------

TNS_ADMIN

---------------------------------------------------

Specifies the directory containing the Oracle Net

Services configuration files like listener.ora,

tnsnames.ora, and sqlnet.ora.

---------------------------------------------------

TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN

---------------------------------------------------

ORA_NLS11

---------------------------------------------------

Specifies the directory where the language,

territory, character set, and linguistic definition

files are stored.

---------------------------------------------------

ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11

---------------------------------------------------

PATH

---------------------------------------------------

Used by the shell to locate executable programs;

must include the $ORACLE_HOME/bin directory.

---------------------------------------------------

PATH=.:$JAV{A}_{H}OME/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=$PATH:/usr/bin:/bin:/usr/bin/X11:/usr/local/binPATH=${PATH}:/u01/app/common/oracle/bin
export PATH

---------------------------------------------------

LD_LIBRARY_PATH

---------------------------------------------------

Specifies the list of directories that the shared

library loader searches to locate shared object

libraries at runtime.

---------------------------------------------------

LD_LIBRARY_PATH=$ORACL{E}_{H}OME/libL{D}_{L}IBRAR{Y}_{P}ATH=${LD_LIBRARY_PATH}:$ORACL{E}_{H}OME/oracm/libL{D}_{L}IBRAR{Y}_{P}ATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH

---------------------------------------------------

CLASSPATH

---------------------------------------------------

Specifies the directory or list of directories that

contain compiled Java classes.

---------------------------------------------------

CLASSPATH=$ORACL{E}_{H}OME/JRECLASSPATH=${CLASSPATH}:$ORACL{E}_{H}OME/jlibCLASSPATH=${CLASSPATH}:$ORACL{E}_{H}OME/rdbms/jlibCLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH

---------------------------------------------------

THREADS_FLAG

---------------------------------------------------

All the tools in the JDK use green threads as a

default. To specify that native threads should be

used, set the THREADS_FLAG environment variable to

“native”. You can revert to the use of green

threads by setting THREADS_FLAG to the value

“green”.

---------------------------------------------------

THREADS_FLAG=native; export THREADS_FLAG

---------------------------------------------------

TEMP, TMP, and TMPDIR

---------------------------------------------------

Specify the default directories for temporary

files; if set, tools that create temporary files

create them in one of these directories.

---------------------------------------------------

export TEMP=/tmp
export TMPDIR=/tmp

---------------------------------------------------

UMASK

---------------------------------------------------

Set the default file mode creation mask

(umask) to 022 to ensure that the user performing

the Oracle software installation creates files

with 644 permissions.

---------------------------------------------------

umask 022
16. 创建oracle账号 “oracle”:
[root@racnode1 ~]# groupadd -g 1300 dba
[root@racnode1 ~]# groupadd -g 1301 oper
[root@racnode1 ~]# useradd -m -u 1101 -g oinstall -G dba,oper,asmdba -d
/home/oracle -s /bin/bash -c “Oracle Software Owner” oracle

[root@racnode1 ~]# id oracle
uid=1101(oracle)
gid=1000(oinstall)
groups=1000(oinstall),1201(asmdba),1300(dba),1301(oper)
然后为oracle用户创建登录环境(/home/oracle/.bash_profile)

---------------------------------------------------

.bash_profile

---------------------------------------------------

OS User: oracle

Application: Oracle Database Software Owner

Version: Oracle 11g release 2

---------------------------------------------------

Get the aliases and functions

if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

alias ls=“ls -FA”

---------------------------------------------------

ORACLE_SID

---------------------------------------------------

Specifies the Oracle system identifier (SID) for

the Oracle instance running on this node.

Each RAC node must have a unique ORACLE_SID.

(i.e. racdb1, racdb2,…)

---------------------------------------------------

ORACLE_SID=racdb1; export ORACLE_SID(注意第二个节点改为racdb2)

---------------------------------------------------

ORACLE_UNQNAME

---------------------------------------------------

In previous releases of Oracle Database, you were

required to set environment variables for

ORACLE_HOME and ORACLE_SID to start, stop, and

check the status of Enterprise Manager. With

Oracle Database 11g release 2 (11.2) and later, you

need to set the environment variables ORACLE_HOME

and ORACLE_UNQNAME to use Enterprise Manager.

Set ORACLE_UNQNAME equal to the database unique

name.

---------------------------------------------------

ORACLE_UNQNAME=racdb; export ORACLE_UNQNAME

---------------------------------------------------

JAVA_HOME

---------------------------------------------------

Specifies the directory of the Java SDK and Runtime

Environment.

---------------------------------------------------

JAVA_HOME=/usr/local/java; export JAVA_HOME

---------------------------------------------------

ORACLE_BASE

---------------------------------------------------

Specifies the base of the Oracle directory structure

for Optimal Flexible Architecture (OFA) compliant

database software installations.

---------------------------------------------------

ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE

---------------------------------------------------

ORACLE_HOME

---------------------------------------------------

Specifies the directory containing the Oracle

Database software.

---------------------------------------------------

ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1; export ORACLE_HOME

---------------------------------------------------

ORACLE_PATH

---------------------------------------------------

Specifies the search path for files used by Oracle

applications such as SQL*Plus. If the full path to

the file is not specified, or if the file is not

in the current directory, the Oracle application

uses ORACLE_PATH to locate the file.

This variable is used by SQL*Plus, Forms and Menu.

---------------------------------------------------

ORACLE_PATH=/u01/app/common/oracle/sql; export ORACLE_PATH

---------------------------------------------------

SQLPATH

---------------------------------------------------

Specifies the directory or list of directories that

SQL*Plus searches for a login.sql file.

---------------------------------------------------

SQLPATH=/u01/app/common/oracle/sql; export SQLPATH

---------------------------------------------------

ORACLE_TERM

---------------------------------------------------

Defines a terminal definition. If not set, it

defaults to the value of your TERM environment

variable. Used by all character mode products.

---------------------------------------------------

ORACLE_TERM=xterm; export ORACLE_TERM

---------------------------------------------------

NLS_DATE_FORMAT

---------------------------------------------------

Specifies the default date format to use with the

TO_CHAR and TO_DATE functions. The default value of

this parameter is determined by NLS_TERRITORY. The

value of this parameter can be any valid date

format mask, and the value must be surrounded by

double quotation marks. For example:

# NLS_DATE_FORMAT = “MM/DD/YYYY”

# ---------------------------------------------------

NLS_DATE_FORMAT=“DD-MON-YYYY HH24:MI:SS”; export NLS_DATE_FORMAT

---------------------------------------------------

TNS_ADMIN

---------------------------------------------------

Specifies the directory containing the Oracle Net

Services configuration files like listener.ora,

tnsnames.ora, and sqlnet.ora.

---------------------------------------------------

TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN

---------------------------------------------------

ORA_NLS11

---------------------------------------------------

Specifies the directory where the language,

territory, character set, and linguistic definition

files are stored.

---------------------------------------------------

ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11

---------------------------------------------------

PATH

---------------------------------------------------

Used by the shell to locate executable programs;

must include the $ORACLE_HOME/bin directory.

---------------------------------------------------

PATH=.:$JAV{A}_{H}OME/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=$PATH:/usr/bin:/bin:/usr/bin/X11:/usr/local/binPATH=${PATH}:/u01/app/common/oracle/bin
export PATH

---------------------------------------------------

LD_LIBRARY_PATH

---------------------------------------------------

Specifies the list of directories that the shared

library loader searches to locate shared object

libraries at runtime.

---------------------------------------------------

LD_LIBRARY_PATH=$ORACL{E}_{H}OME/libL{D}_{L}IBRAR{Y}_{P}ATH=${LD_LIBRARY_PATH}:$ORACL{E}_{H}OME/oracm/libL{D}_{L}IBRAR{Y}_{P}ATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH

---------------------------------------------------

CLASSPATH

---------------------------------------------------

Specifies the directory or list of directories that

contain compiled Java classes.

---------------------------------------------------

CLASSPATH=$ORACL{E}_{H}OME/JRECLASSPATH=${CLASSPATH}:$ORACL{E}_{H}OME/jlibCLASSPATH=${CLASSPATH}:$ORACL{E}_{H}OME/rdbms/jlibCLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH

---------------------------------------------------

THREADS_FLAG

---------------------------------------------------

All the tools in the JDK use green threads as a

default. To specify that native threads should be

used, set the THREADS_FLAG environment variable to

“native”. You can revert to the use of green

threads by setting THREADS_FLAG to the value

“green”.

---------------------------------------------------

THREADS_FLAG=native; export THREADS_FLAG

---------------------------------------------------

TEMP, TMP, and TMPDIR

---------------------------------------------------

Specify the default directories for temporary

files; if set, tools that create temporary files

create them in one of these directories.

---------------------------------------------------

export TEMP=/tmp
export TMPDIR=/tmp

---------------------------------------------------

UMASK

---------------------------------------------------

Set the default file mode creation mask

(umask) to 022 to ensure that the user performing

the Oracle software installation creates files

with 644 permissions.

---------------------------------------------------

umask 022
17. 创建RAC安装所需目录结构:

mkdir -p /u01/app/11.2.0/grid

chown grid:oinstall /u01/app/11.2.0/grid

chmod -R 775 /u01/app/11.2.0/grid

mkdir -p /u01/app/oracle

chown -R oracle:oinstall /u01/app/oracle

chmod -R 775 /u01/app/oracle

mkdir /u01/app/oraInventory

chwon grid:oinstall /u01/app/oraInventory

chmod –R 775 /u01/app/oraInventory

1.13 安装 VMware Tools
VMware Tools 是VMware为虚拟机操作系统提供的工具包，用于提升虚拟机性能和方便对虚拟机的监控和管理。VMware Tools 必须在操作系统内安装。其安装步骤可以参考以下知识库文档：Installing VMware Tools in a Linux virtual machine using Red Hat Package Manager (RPM)(http://kb.vmware.com/kb/1018392).
To install VMware Tools

1. Confirm that the Linux virtual machine is powered on.
2. In vSphere Client, right-click the virtual machine and selectGuest>Install VMware Tools.
3. In the Linux guest OS,create a mount point and run the following command:
   mkdir /mnt/cdrom
4. Mount the CD-ROM with the following command:
   mount /dev/cdrom /mnt/cdrom
5. Install VMware Tools using RPM, with the following command:
   rpm -ivh /mnt/cdrom/VMwareTools-8.3.2-257589.rpm
   As an example, “8.3.2-257589” is the version of VMware Tools used here.
6. Configure VMware Tools, by running the following command:
   /usr/bin/vmware-config-tools.pl
7. Unmount the CD-ROM with the following command:
   umount /mnt/cdrom
8. Click VM in the virtual machine menu, and click Guest>End VMware Tools Install.
   1.14 时钟同步
   按照VMware官方文档Timekeeping best practices for Linux guests(http://kb.vmware.com/kb/1006427) 来配置在Linux上配置NTP，同时停掉虚拟机中VMware tools与ESX主机的时钟同步.
   有两种方法停止VMware tools的时钟同步进程：
   • 在虚拟机配置文件中设置选项：tools.syncTime = “False”
   • 在Linux操作系统中执行命令：vmware-guestd --cmd “vmx.set_option synctime 1 0”
   编辑/etc/sysconfig/ntpd 增加 -x 选项，如下所示:

Drop root to id ‘ntp:ntp’ by default.

OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"
/etc/ntp.conf 示例:
tinker panic 0

driftfile /var/lib/ntp/drift
statistics loopstats
statsdir /var/log/ntp/
logfile /var/log/ntp/ntpd.log
filegen peerstats file peers type day link enable
filegen loopstats file loops type day link enable

server xxx.xxx.xxx.xxx (这里设置环境可访问的时钟服务器地址)

准备其他虚拟机和RAC节点
现在第一个RAC节点（RAC1）所用的虚拟机就创建好了，现在可以用这个虚拟机克隆出其他RAC节点所用的虚拟机，如果前面的步骤都执行正确，现在就可以把这个虚拟机制成模板，留作以后使用。大概步骤如下：
• 在vCenter里利用RAC1复制新的虚拟机.
• 在新创建的虚拟机里删除RAC1所带的共享磁盘。
• 重新添加仲裁盘，仲裁盘同样需要开启multi-writer开关。
• 修改客户机IP地址.
1.15 虚机克隆

1. 配置文件/etc/udev/rules.d/70-persistent-net.rules 做备份然后删除
2. RAC1停机
3. In vSphere Client, select the virtual machine, RAC1.
4. Right-click the name of the virtual machine and select Clone.
5. In the Clone Virtual Machine wizardon the Name and Location page, enter RAC2 for the Name. Click Next.
6. On the Host / Cluster page, select a host or cluster. Click Next.
7. On the Specific Host page, select a unique ESX host (that is, one not running a RAC node virtual machine). Click Next.
8. On the Datastore page, select a VMFS datastore. Click Next.
9. On the Disk Format page, select Same format as source. Click Next.
10. On the Guest Customization page, select customize using the customization wizard.
11. config IP address for eth0 and eth1
    
12. On the Ready to Complete page, click Finish.
    1.16 配置仲裁盘
    虚拟机克隆之后，指向仲裁盘的路径会失效，所以需要先在虚机配置里删除这个硬盘重建，同时把SCSI控制器类型改为paravirtual.
    按照下述步骤配置RAC2:
    删除共享盘
13. In vSphere Client, select a virtual machine.
14. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.
15. Select hard disk 2 and click Remove. In the Removal Options section, select Remove from virtual machine and click OK.
    再新建一次，注意要找到RAC1所用的共享磁盘
16. In vSphere Client, select a virtual machine.
17. Right-click the name of a virtual machine and select Edit Settings to display the Virtual Machine Properties pane.
18. Click Addto display the Add Hardware wizard.
19. In the Add Hardware wizard select Hard Disk for the Device Type, and click Next.
20. In the Disk section of the Select a Disk page, select Use an existing virtual disk and click Next.
21. On the Select Existing Disk page, enter the path to the first VMDK of the virtual machine (selected in step 1) used for the CRS and voting datastores. Click Next.
22. On the Advanced Options page, in the Virtual Device Node section, select SCSI (1:0). In theMode section, select Independent and Persistent. Click Next.
23. On the Ready to Complete page, click Finish.
24. Click OK.
    修改SCSI控制器类型为 paravirtual
25. In vSphere Client, select the virtual machine that was selected in the preceding step 1.
26. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.
27. In the device list, select SCSI controller 1.
28. In the SCSI Controller Type section, click Change Type.
29. Select VMware Paravirtual.
30. Click OK, and click OK again.
    1.17 使用fdisk格式化仲裁盘：
    必须在一个节点上对共享盘进行分区和格式化。在Linux上可以使用系统自带的fdisk工具进行 。步骤如下：
    仲裁盘分区：
31. 以root用户登录RAC1节点.
32. Enter the command:
    fdisk /dev/sdb
33. Enter n to create a new partition.
34. Enter p to create a primary partition.
35. Enter 1 to create the first partition.
36. Press Enter to take the default value of 1 for the first cylinder.
37. Press Enter to take the default value of 20480 for the last cylinder.
38. Enter w to write the new partition table to disk and exit fdisk.
    fdisk命令的执行过程如下图所示：:
    root@vmorarac1 ~# fdisk /dev/sdb

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won’t be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-20480, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-20480, default 20480):
Using default value 20480
Command (m for help): w

The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
安装Oracle Grid软件
现在我们有两个虚机，可以开始安装grid环境了.
安装Oracle Grid Infrastructure

1. 用oracleasm在/dev/sdb1创建griddisk CRS1.
2. 将Oracle RAC安装包（11.2.0.3）上传到RAC1并解压
3. 切换到grid用户启动Oracle Grid安装程序 ./runInstaller
4. 在第一页安装选项页，选择Install and Configure Infrastructure for a Cluster. Click Next.
5. On the Installation Type page, select Advanced Installation and click Next.
6. On the Product Languages page, select English and click Next.
7. On the Grid Plug and Play page, enter rac for the Cluster Name,
   rac-scan.localdomainfor the SCAN Name and 1521 for the SCAN Port. Click Next.
8. On the Cluster Node Information page, click Add to enter aHostname and Virtual IP Name. Repeat the process for each of the four hosts, as follows:
   Hostname Virtual IP Name
   RAC1 RAC1-vip
   RAC2 RAC2-vip
9. Click Next.
10. On the Network Interface Usage page, enter the following information for the Interface Name, Subnet, and Interface Type:
    Interface Name Subnet Interface Type
    eth0 162.16.3.0 Public
    eth1 192.168.2.0 Private
11. Click Next.
12. On the Storage Option page, select Automatic Storage Management (ASM), and click Next.
13. On the Create ASM Disk Group page, enter CRS for the Disk Group Name, and select Normal for Redundancy.
14. In the Add Disks section of the Create ASM Disk Group page, select Candidate Disks. Make the following entries:
    Disk Path Size (in MB) Status
    ORCL:CRSDISK1 20479 Candidate
15. Select the check box to the left of each of these entries, and click Next.
16. On the ASM Password page, select Use same passwords for these accounts and enter the password “oracle” in the Specify Password and Confirm Password fields. Click Next.
17. On the Failure Isolation page, select Do not use Intelligent Platform Management Interface (IPMI). Click Next.
18. On the Operating System Groups page, make the following entries:
    Field Selection
    ASM Database Administrator (OSDBA) Group asmdba
    ASM Instance Administration Operator (OSOPER) Group asmoper
    ASM Instance Administrator (OSASM) Group asmadmin
19. Click Next.
20. On the Installation Location page, enter /u01/app/grid for the Oracle Base, and /u01/app/11.2.0/grid for the Software Location. Click Next
21. On the Create Inventory page, enter /u01/app/orainventory for the Inventory Directory. Click Next.
22. Wait while the setup application conducts the prerequisite checks and installs Grid on each node. After setup is complete, execute the following configuration scripts on the four nodes VMORARAC1,VMORARAC2. These scripts must be executed in a terminal window as root.
    /u01/app/orainventory/orinstRoot.sh
    /u01/app/11.2.0/grid/root.sh
    Note: don’t run root.sh scripts on 2 nodes at the same time, or else
    It will fail.
23. Click OK, and click Finish.
    验证Grid安装的正确性
    下面几幅图演示了怎样使用Oracle命令验证clusterware是否已经正常工作：
    Figure 5. Check the CRS Status on the First Node as Grid User

Figure 6.Check for Cluster Nodes

Figure 7. Check for Cluster Registry (OCR)

Figure 8. Check Voting Disk

增加数据盘和REDO盘
在这一节，我们为RAC节点添加数据盘和重做日志（redo log）盘：
• 现在预先分配好的LUN上创建datastore，然后用RAC1创建对应的VMDK，SCSI控制器类型设为paravirtual.
• 把数据盘和重做日志盘添加到RAC2.
• 在RAC1使用fdisk格式化新添加的磁盘
• 为每个共享盘设置multi-writer参数.
下面描述的RAC1的配置过程，RAC2类似.
添加数据盘和重做日志盘:

1. In vSphere Client, select the virtual machine.
2. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.
3. Click Addto display the Add Hardware wizard.
4. In the Add Hardware wizard select Hard Disk for the Device Type, and click Next.
5. In the Disk section of the Select a Disk page, select Create a new virtual disk. Set the disk size to 1000GB for DATA or 900GB for REDO, and select the Support clustering features such as Fault Tolerance check box. Also, for vSphere 5, in the Disk Provisioning section, select Thick Provision Eager Zeroed. Select DATA01 for DATA, or REDO01 for REDO to specify a datastore. Click Next.
6. On the Compatibility Mode page, click Next.
7. On the Advanced Options page, in the Virtual Device Node section, select SCSI (1:1) for REDO , or SCSI(2:0) for DATA.Set the Mode to Independent. Click Next
8. On the Ready to Complete page, click Finish.
9. Click OK.
10. Repeat steps 1 through 9 for all the DATA datastores.Change step 7 to use SCSI (2:1)/SCSI (3:0)/SCSI (3:1)for the other DATA datastores.
    设置SCSI控制器类型为 paravirtual:
11. Select the virtual machine that was selected in the preceding step 1.
12. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.
13. In the device list, select SCSI controller 1.
14. In the SCSI Bus Sharing section, select None, keep the default selection.
15. In the SCSI Controller Type section, click Change Type.
16. Select VMware Paravirtual.
17. Click OK, and click OK again.
    参照5.3小节，使用fdisk命令格式化所有共享盘。
    为每个RAC节点的共享盘配置multi-writer参数
    细节请参考官方文档Disabling simultaneous write protection provided by VMFS using the multi-writer flag (http://kb.vmware.com/kb/1034165).
18. In vSphere Client, select a virtual machine.
19. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.
20. Click the Options tab, click Advanced, and click General.
21. Click Configuration Parameters.
22. On the Configuration Parameters page, add rows and entries for the shared disks in the virtual machine’s configuration parameters as shown in the following screenshot.
    
    虚拟机硬件配置如下图所示：
    
    以grid用户创建DATA和REDO磁盘组
    为安装数据库做准备
23. 使用oracleasm创建4个数据盘DATA1到DATA4，使用的设备为/dev/sdd1到/dev/sdg1。用设备/dev/sdc1创建REDO磁盘REDO1.
24. 以grid用户登录节点1(RAC1),启动ASM配置助手（asmca），创建两个磁盘组DATADG (选择所有DATA标签盘DATA1到DATA4) 和REDODG(选择磁盘 REDO1).
25. In the ASM Configuration Assistant, on the Create Disk Group page, enter DATADG for the Disk Group Name. In the Redundancy section, select External (None). Click OK, and click OK again.
26. On the Create Disk Group page, in the Select Member Disks section, select ORCL:DATA1, ORCL:DATA2, ORCL:DATA3, and ORCL:DATA4. Set the size of each to 1000GB. Click OK.
27. On the Create Disk Group page, enter REDODG for the Disk Group Name. In the Redundancy section, select External (None). Click OK, and click OK again.
28. On the Create Disk Group page, in the Select Member Disks section, select ORCL:REDO1, ,. Set the size of each to 900GB. Click OK.
    安装 RAC数据库
    安装并创建数据库
29. Login into RAC1 as oracle user and launch runInstallerfromthe Oracle software location to install the Oracle binaries.
30. In the Oracle Installer wizard, on the Configure Security Updates page, leave the Email field blank, and do not select I wish to receive security updates via My Oracle Support. Click Next.
31. On the Installation Option page, select Install database software only.Click Next.
32. On the Grid Options page, select Real Application Clusters database installation. Select the four nodes RAC1, RAC2.Click Next.
33. On the Product Languages page, select English.Click Next.
34. On the Database Edition page, select Enterprise Edition (4.29GB).Click Next.
35. On the Installation Location page, enter /u01/app/oracle for Oracle Base, and /u01/app/oracle/product/11.2.0/dbhome_1 for Software Location.Click Next.
36. On the Operating System Groups page, select dba for the Database Administrator (OSDBA) Group and oinstall for the Database Operator (OSOPER) Group. Click Next.
37. Review the configuration on the Summary page, and click Finish.
38. After the installation is complete, open a terminal window, log in as root, and run the following script:
    /u01/app/oracle/product/11.2.0/dbhome_1/root.sh
    Repeat for each of the four nodes, and click OK.
39. On the Finish page, click Close.
40. Create the Oracle RAC database using Oracle Database Configuration Assistant (DBCA).
    参考资料
    The following are resources and references for Oracle and VMware vSphere.
    vSphere Installation and Setup
    http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-installation-setup-guide.pdf
    vSphere Networking
    http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-networking-guide.pdf

vSphere Storage
http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-storage-guide.pdf
Guide to configure NTP on ESX servers
http://kb.vmware.com/kb/1003063
Timekeeping in VMware Virtual Machines
http://www.vmware.com/files/pdf/Timekeeping-In-VirtualMachines.pdf
Installing VMware Tools in a Linux virtual machine using Red Hat Package Manager (RPM)
http://kb.vmware.com/kb/1018392
Oracle ASMLib
http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html
Oracle Database Oracle Clusterware and Oracle Real Application Clusters Installation Guide
10g Release 2 (10.2) for Linux
http://download.oracle.com/docs/cd/B19306_01/install.102/b14203/toc.htm
Oracle Clusterware Installation Guide11g Release 1 (11.1) for Linux
http://www.oracle.com/pls/db111/to_toc?pathname=install.111/b28263/toc.htm
Oracle Real Application Clusters Installation Guide11g Release 1 (11.1) for Linux and UNIX
http://www.oracle.com/pls/db111/to_toc?pathname=install.111/b28264/toc.htm