如何在云环境下部署oracle18c RAC(京东云公有云)

理论篇安装oracle RAC　主要需要几个条件私有网络和公共网络隔离（不在同一网段）共享存储网络需要支持mulitcast我们可以利用京东云提供的基础设施解决以上问题通过创建"创建无预设CIDR私有网络"的vpc进行网络隔离，实现public和private网段隔离通过创建"多点挂载存储"实现ASM所需的共享存储通过n2n点对点vpn实现multicast

来留去送

2181人浏览 · 2018-11-13 18:59:13

来留去送 · 2018-11-13 18:59:13 发布

理论篇

安装oracle RAC　主要需要几个条件

私有网络和公共网络隔离（不在同一网段）
共享存储
网络需要支持mulitcast

我们可以利用京东云提供的基础设施解决以上问题

通过创建"创建无预设CIDR私有网络"的vpc进行网络隔离，实现public和private网段隔离
通过创建"多点挂载存储"实现ASM所需的共享存储
通过n2n点对点vpn实现multicast支持

实施篇

网络配置

创建“创建无预设CIDR私有网络”vpc
控制台=>网络=>私有网络(vpc)=>创建=>选择"创建无预设CIDR私有网络"
创建子网
分别创建两个不同网段的子网
10.0.1.0/24
192.168.1.0/24

主机创建

创建共享磁盘

控制台=>弹性计算=>云硬盘=>创建=>选择"多点挂载"（目前仅华东有此选项）

创建弹性网卡

192.168.1.10 rac1private
192.168.1.20 rac2private

修改hosts文件

vi /etc/hosts

#oracle rac public ip
10.0.0.10 rac1
10.0.0.20 rac2

#oracle rac vip
10.0.0.11 rac1vip
10.0.0.21 rac2vip

#oracle rac scanip
10.0.0.30 scan-rac

#oracle rac private ip
192.168.２.10 rac1private
192.168.２.20 rac2private

这里可以利用弹性网卡来绑定多个ip地址来实现public ip和vip

创建用户

https://docs.oracle.com/en/database/oracle/oracle-database/18/cwlin/identifying-an-oracle-software-owner-user-account.html#GUID-0A95F4B1-1045-455D-9897-A23012E4E27F

用户和组：
/usr/sbin/groupadd -g 54321 oinstall
/usr/sbin/groupadd -g 54322 dba
/usr/sbin/groupadd -g 54323 oper
/usr/sbin/groupadd -g 54324 backupdba
/usr/sbin/groupadd -g 54325 dgdba
/usr/sbin/groupadd -g 54326 kmdba
/usr/sbin/groupadd -g 54327 asmdba
/usr/sbin/groupadd -g 54328 asmoper
/usr/sbin/groupadd -g 54329 asmadmin
/usr/sbin/groupadd -g 54330 racdba

/usr/sbin/useradd -u 54321 -g oinstall -G dba,asmdba,backupdba,dgdba,kmdba,racdba,oper oracle
/usr/sbin/useradd -u 54331 -g oinstall -G dba,asmdba,asmoper,asmadmin grid

echo oracle | passwd --stdin oracle
echo oracle | passwd --stdin grid

11、目录：
mkdir -p /u01/app/18.3.0/grid
mkdir -p /u01/app/grid
mkdir -p /u01/app/oracle
chown -R grid:oinstall /u01
chown oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/

12、环境变量

节点1：
--grid
export ORACLE_SID=+ASM1
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/18.3.0/grid
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
#export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

--oracle用户
export ORACLE_SID=db1
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=/u01/app/oracle/product/18.3.0/db_1
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
#export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib


节点2：
--gird
export ORACLE_SID=+ASM2
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/18.3.0/grid
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
#export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

--oracle用户
export ORACLE_SID=db2
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=/u01/app/oracle/product/18.3.0/db_1
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
#export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

验证用户组

$ grep "oinstall" /etc/group
oinstall:x:54321:grid,oracle
You can then use the ID command to verify that the Oracle installation owners you intend to use have the Oracle Inventory group as their primary group. For example:

$ id oracle
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba), 
54323(oper),54324(backupdba),54325(dgdba),54326(kmdba),54327(asmdba),54330(racdba)

$ id grid
uid=54331(grid) gid=54321(oinstall) groups=54321(oinstall),54322(dba),
54327(asmdba),54328(asmoper),54329(asmadmin),54330(racdba)

操作系统参数

vi /etc/sysctl.conf
sysctl -p /etc/sysctl.d/*.conf

fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 67108864
kernel.shmmax = 274877906944
kernel.shmmni = 4096
#kernel.sem = 250 32000 100 128
kernel.sem = 5010 641280 5010 128
net.ipv4.ip_local_port_range = 9000 65500
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 16384 4194304
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

限制文件：vi /etc/security/limits.conf

grid soft nproc 2047 
grid hard nproc 16384 
grid soft nofile 1024 
grid hard nofile 65536 
grid soft stack 10240 
oracle soft nproc 2047 
oracle hard nproc 16384 
oracle soft nofile 1024 
oracle hard nofile 65536 
oracle soft stack 10240
* soft memlock 524288000
* hard memlock 524288000

vi /etc/pam.d/login 文件
session required pam_limits.so
vi /etc/fstab
tmpfs /dev/shm tmpfs size=5000m 0 0
/etc/sysconfig/network
NOZEROCONF=yes
vi /etc/profile

if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
    if [ $SHELL = "/bin/ksh" ]; then
        ulimit -p 16384
        ulimit -n 65536
    else
        ulimit -u 16384 -n 65536
    fi
    umask 022
fi

安装必要包

yum install -y gcc-c++
yum install -y bc
yum install -y binutils
yum install -y compat-libcap1
yum install -y compat-libstdc++-33.i686    
yum install -y compat-libstdc++-33.x86_64
yum install -y glibc-2.17-36.el7.i686
yum install -y glibc-2.17-36.el7.x86_64
yum install -y glibc-devel.i686
yum install -y glibc-devel.x86_64
yum install -y ksh
yum install -y libaio.i686
yum install -y libaio.x86_64
yum install -y libaio-devel.i686
yum install -y libaio-devel.x86_64
yum install -y libX11.i686
yum install -y libX11.x86_64
yum install -y libXau.i686
yum install -y libXau.x86_64
yum install -y libXi.i686
yum install -y libXi.x86_64
yum install -y libXtst.i686
yum install -y libXtst.el7.x86_64
yum install -y libgcc.i686
yum install -y libgcc.x86_64
yum install -y libstdc++.i686
yum install -y libstdc++
yum install -y libstdc++-devel.i686
yum install -y libstdc++-devel.x86_64
yum install -y libxcb.i686
yum install -y libxcb.x86_64
yum install -y make
yum install -y nfs-utils 
yum install -y net-tools
yum install -y python
yum install -y python-configshell
yum install -y python-rtslib
yum install -y python-six
yum install -y smartmontools
yum install -y sysstat
yum install -y targetcli

为保证xwindow正常显示的必要安装

yum groupinstall "X Window System" "Fonts" -y

准备asm磁盘组（所有节点操作）

asmlib方式
参考文档：https://docs.microsoft.com/zh-tw/azure/virtual-machines/workloads/oracle/configure-oracle-asm

从 https://www.oracle.com/technetwork/server-storage/linux/asmlib/rhel7-2773795.html下载rpm
安装
rpm -Uvh oracleasm-support-2.1.11-2.el7.x86_64.rpm
yum -y install kmod-oracleasm.x86_64
rpm -ivh oracleasmlib-2.0.12-1.el7.x86_64.rpm
配置asm

/usr/sbin/oracleasm configure -i

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done

此处需要reboot

创建asm磁盘组

oracleasm createdisk ocr /dev/vdb1
oracleasm createdisk data /dev/vdc1
oracleasm scandisks
oracleasm listdisks

清除磁盘数据

dd if=/dev/zero of=/dev/vdb1 bs=1M count=1000

n2n安装及配置

安装

yum install -y gcc
yum install -y openssl-devel

git clone https://github.com/ntop/n2n.git
cd n2n
git checkout 2.4-stable

make
make PREFIX=/opt/n2n install

安装完毕后启动supernode服务

nohup /opt/n2n/sbin/supernode -l 65530 &

在RAC节点安装ntop软件（计划加入vpn网络的节点都需要安装ntop）

scp -r /opt/n2n 10.0.0.10:/opt
scp -r /opt/n2n 10.0.0.20:/opt

节点1

/opt/n2n/sbin/edge -d edge1 -a 192.168.2.10 -s 255.255.255.0 -c dtstack -k  dtstack -l 192.168.1.40:65530 -E -r

节点2

/opt/n2n/sbin/edge -d edge1 -a 192.168.2.20 -s 255.255.255.0 -c dtstack -k  dtstack -l 192.168.1.40:65530 -E -r

测试multicast(非必要，主要是验证multicast无异常)
```
./mcasttest.pl -n rac1,rac2 -i edge1
```

ssh互信

分别在grid和oracle用户，每个节点上执行。
- 创建ssh目录
```
mkdir .ssh
chmod 700 ~/.ssh
cd ~/.ssh
```
- 生成秘钥
```
ssh-keygen -t rsa
ssh-keygen -t dsa
```

在rac1上将密钥添加到授权文件中,grid和oracle用户分别执行

本机的拷贝

cat id_dsa.pub >> authorized_keys
cat id_rsa.pub >> authorized_keys

拷贝另一台主机的秘钥

ssh rac2 cat ~/.ssh/id_dsa.pub >> authorized_keys
ssh rac2 cat ~/.ssh/id_rsa.pub >> authorized_keys

将本机的authorized_keys的文件拷贝到另一台主机上

scp authorized_keys rac2:/home/grid/.ssh/
scp authorized_keys rac2:/home/oracle/.ssh/

修改两个主机的授权密钥的权限
```
chmod 600 ~/.ssh/authorized_keys
```

测试

获取服务器公钥

ssh-keyscan rac1  >> ~/.ssh/known_hosts
ssh-keyscan rac2  >> ~/.ssh/known_hosts
ssh-keyscan rac1private  >> ~/.ssh/known_hosts
ssh-keyscan rac2private  >> ~/.ssh/known_hosts

测试

ssh rac1 date
ssh rac2 date
ssh rac1private date
ssh rac2private date

安装前检查

$ORACLE_HOME/runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose

错误处理

PRVE-0421
修改/etc/fstab

tmpfs  /dev/shm tmpfs  size=5000m   0 0

安装GI--grid用户下

解压安装压缩包

cd /u01/app/12.2.0/grid/grid
unzip /oraclesoftware/linuxx64_12201_grid_home.zip

安装cvudisk RPM包
root 用户安装
cvuqdisk存于oracle GI安装介质的cv/rpm目录下

cd /u01/app/18.3.0/grid
export CVUQDISK_GRP=asmadmin
rpm -ivh cv/rpm/cvuqdisk-1.0.10-1.rpm

scp cv/rpm/cvuqdisk-1.0.10-1.rpm 10.0.0.20:/root
export CVUQDISK_GRP=asmadmin
rpm -ivh /root/cvuqdisk-1.0.10-1.rpm

grid用户执行$HOME_ORACLE/gridSetup.sh

安装DB--oracle用户下

解压安装压缩包

cd /u01/app/oracle/product/18.3.0/db_1
unzip /oraclesoftware/LINUX.X64_180000_db_home.zip

oracle用户执行$HOME_ORACLE/runInstaller.sh

查看crs服务状态

crsctl status res -t

华为开发者空间

华为开发者空间，是为全球开发者打造的专属开发空间，汇聚了华为优质开发资源及工具，致力于让每一位开发者拥有一台云主机，基于华为根生态开发、创新。

更多推荐

GaussDB技术解读——GaussDB架构介绍（四）

华为开发者空间

一文解读GaussDB(DWS)监控运维诊断优化能力

华为开发者空间

华为云数据仓库专业级开发者认证重磅发布

华为开发者空间

所有评论(0)

查看更多评论

来留去送

@jiashiwen

已为社区贡献1条内容