dolphinscheduler 3.2.1 集群安装部署笔记
dolphinescheduler 3.2.1 集群部署 +virtualbox 虚拟机
·
dolphinscheduler 3.2.1 集群安装部署笔记
1. 集群角色分配
| 服务器 | ip | 角色 | 配置 |
|---|---|---|---|
| centos-1 | 192.168.56.101 | zookeeper,master,worker | 4核8G |
| centos-2 | 192.168.56.102 | zookeeper,master,worker,alert-server | 4核8G |
| centos-3 | 192.168.56.103 | zookeeper,worker,mysql,api-server | 4核8G |
2. 前置条件
- 虚拟机：virtualbox，网卡:NAT(提供外网访问功能), Host-Only(虚拟机间相互访问)
- 安装 centos7最小化版本，root/1q@W3e$R
virtualbox安装最小化CentOS7系统
手把手使用VirtualBox创建centOS7虚拟机并完成网络配置… - yum修改源
Centos7配置yum源 - 安装 ifconfig
Centos 7 安装 ifconfig 管理命令 - 安装 jdk8
CentOS7 安装jdk8教程 - 在centos-3上安装mysql5.7。用户名密码：root/abcdef
CentOS 7安装MySQL 5.7 - 关闭防火墙
centos7 关闭防火墙和selinux
Linux永久关闭iptables防火墙的命令 - 复制虚拟机，配置各自ip地址及hosts文件
- 配置相互免密登录
CentOS 配置SSH免密码登录 - 安装rsync
centos7安装rsync… - 配置集群时间同步
Centos7集群时间同步（Chrony） - 编写脚本xcall和xsync以便集群配置
- 将附录1和附录2的内容保存成文件xcall和xsync放置到/bin目标下
- xcall的作用是在集群各服务器上同时执行相同的命令
- xsync的作用是将centos-1上的目录或文件拷贝到centos-2和centos-3上，避免重复操作
[root@centos-1 ~] vim xcall # 将附录1的内容拷入，保存
[root@centos-1 ~] vim xsync # 将附录2的内容拷入，保存
[root@centos-1 ~] cp xcall /bin/
[root@centos-1 ~] cp xsync /bin/
- 其他注意事项
资料很多，遇到问题随时搜索即可。
3. 创建必要的文件夹
[root@centos-1 ~] mkdir -p /opt/software # 用于存放安装包
[root@centos-1 ~] mkdir -p /opt/module # 安装目录
4. 安装 datax
4.1. 下载datax
[root@centos-1 software] wget http://datax-opensource.oss-cn-hangzhou.aliyuncs.com \
/datax.tar.gz
4.2. 解压
[root@centos-1 software] tar -zxvf datax.tar.gz -C /opt/module/
4.3. 配置
[root@centos-1 software] rm -rf /opt/module/datax/plugin/*/._*
4.4. 自检
[root@centos-1 software] cd /opt/module/datax
[root@centos-1 datax] python2 bin/datax.py job/job.json
4.5. 拷贝到其他虚拟机(xsync 脚本附后)
[root@centos-1 datax] cd ..
[root@centos-1 module] xsync datax
5. 安装 dolphinescheduler
5.1. 下载
[root@centos-1 software] wget https://mirrors.aliyun.com/apache/dolphinscheduler /3.2.1/apache-dolphinscheduler-3.2.1-bin.tar.gz
5.2. 解压
[root@centos-1 software] tar -zxvf apache-dolphinscheduler-3.2.1-bin.tar.gz -C /opt/module
[root@centos-1 software] mv /opt/module/apache-dolphinscheduler-3.2.1-bin /opt/module/dolphinscheduler-3.2.1
5.3. 配置元数据库(在centos-3上执行)
# 使用root用户登录mysql
mysql -uroot -p
mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphinscheduler'@'%' IDENTIFIED BY 'abcdef';
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphinscheduler'@'localhost' IDENTIFIED BY 'abcdef';
mysql> flush privileges;
5.4. 配置install_env.sh与dolphinscheduler_env.sh
- install_env.sh
# ---------------------------------------------------------
# INSTALL MACHINE
# ---------------------------------------------------------
# Due to the master, worker, and API server being deployed on a single node,
# the IP of the server is the machine IP or localhost
ips="centos-1,centos-2,centos-3"
sshPort="22"
masters="centos-1,centos-2"
workers="centos-1:default,centos-2:default,centos-3:default"
alertServer="centos-2"
apiServers="centos-3"
# DolphinScheduler installation path, it will auto-create if not exists
installPath=~/dolphinscheduler
# Deploy user, use the user you create in section Configure machine SSH
# password-free login
deployUser="dolphinscheduler"
- dolphinscheduler_env.sh
# JAVA_HOME, will use it to start DolphinScheduler server
export JAVA_HOME=${JAVA_HOME:-/opt/module/jdk1.8.0_391}
# Database related configuration, set database type, username and password
export DATABASE=${DATABASE:-mysql}
export SPRING_PROFILES_ACTIVE=${DATABASE}
export SPRING_DATASOURCE_URL="jdbc:mysql://centos-3:3306/dolphinscheduler? useUnicode=true&characterEncoding=UTF-8&useSSL=false"
export SPRING_DATASOURCE_USERNAME=dolphinscheduler
export SPRING_DATASOURCE_PASSWORD=abcdef
# DolphinScheduler server related configuration
export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}
# Registry center configuration, determines the type and link of the registry center
export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}
export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING: -centos-1:2181,centos-2:2181,centos-3:2181}
# Tasks related configurations, need to change the configuration if you use the related tasks.
export PYTHON_LAUNCHER=/usr/bin/python2.7
export JAVA_HOME=/opt/module/jdk1.8.0_391
export DATAX_LAUNCHER=/opt/module/datax/bin/datax.py
export PATH=$PYTHON_LAUNCHER:$JAVA_HOME/bin:$DATAX_LAUNCHER:$PATH
export PATH=$PYTHON_LAUNCHER:$JAVA_HOME/bin:$DATAX_LAUNCHER:$PATH
5.5. 修改各组件的 common.properties中的网卡地址获取方式
分别编辑以下几个文件
dolphinscheduler-3.2.1/master-server/conf/common.properties
dolphinscheduler-3.2.1/worker-server/conf/common.properties
dolphinscheduler-3.2.1/alert-server/conf/common.properties
dolphinscheduler-3.2.1/api-server/conf/common.properties
dolphinscheduler-3.2.1/tools/conf/common.properties
将其中的dolphin.scheduler.network.interface.preferred 属性打开并修改其值为enp0s8 (即host-only网卡的地址)
5.6. 将配置好的拷贝到其他虚拟机
[root@centos-1 module] xsync dolphinscheduler-3.2.1
5.7. 创建专用登录用户(在三台虚拟机上均执行)
# 创建用户需使用 root 登录
[root@centos-1 module] useradd dolphinscheduler
# 添加密码
[root@centos-1 module] echo "dolphinscheduler" | passwd --stdin dolphinscheduler
# 配置 sudo
[root@centos-1 module] sed -i '$adolphinscheduler ALL=(ALL) NOPASSWD: NOPASSWD: ALL' /etc/sudoers
[root@centos-1 module] sed -i 's/Defaults requirett/#Defaults requirett/g' /etc/sudoers
# 修改目录权限，使得部署用户对二进制包解压后的 dolphinscheduler-3.2.1 目录有操作权限
[root@centos-1 module] chown -R dolphinscheduler:dolphinscheduler /opt/module/dolphinscheduler-3.2.1
[root@centos-1 module] chmod -R 755 /opt/module/dolphinscheduler-3.2.1
# 配置三台服务器dolphinescheduler的免密登录
# 以dolphinescheduler用户登录：
[root@centos-1 module] su dolphinescheduler
[dolphinescheduler@centos-x ~]$ ssh-keygen
[dolphinescheduler@centos-x ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub -p22 dolphinscheduler@centos-1
[dolphinescheduler@centos-x ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub -p22 dolphinscheduler@centos-2
[dolphinescheduler@centos-x ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub -p22 dolphinscheduler@centos-3
5.8. 初始化元数据库 并启动
[dolphinescheduler @centos-1 dolphinscheduler-3.2.1]$ bash ./tools/bin/upgrade-schema.sh
[dolphinescheduler @centos-1 dolphinscheduler-3.2.1]$ bash ./bin/install.sh
5.9. 启动zookeeper
[dolphinescheduler @centos-1 dolphinscheduler-3.2.1]$ sudo /opt/module/zookeeper-3.9.1 /bin/zkServer.sh start
5.10. 启动dolphinescheduler集群
# 一键停止集群所有服务
bash ./bin/stop-all.sh
# 一键开启集群所有服务
bash ./bin/start-all.sh
# 查看集群所有服务状态
bash ./bin/status-all.sh
# 启停 Master
bash ./bin/dolphinscheduler-daemon.sh stop master-server
bash ./bin/dolphinscheduler-daemon.sh start master-server
# 启停 Worker
bash ./bin/dolphinscheduler-daemon.sh start worker-server
bash ./bin/dolphinscheduler-daemon.sh stop worker-server
# 启停 Api
bash ./bin/dolphinscheduler-daemon.sh start api-server
bash ./bin/dolphinscheduler-daemon.sh stop api-server
# 启停 Alert
bash ./bin/dolphinscheduler-daemon.sh start alert-server
bash ./bin/dolphinscheduler-daemon.sh stop alert-server
附录
附录1. xcall脚本
#!/bin/bash
# 获取控制台指令
cmd=$*
# 判断指令是否为空
if (( #$cmd -eq # ))
then
echo "command can not be null !"
exit
fi
# 获取当前登录用户
user=`whoami`
# 在从机执行指令,这里需要根据你具体的集群情况配置，host与具体主机名一致
for (( host=1;host<=3;host++ ))
do
echo "================current host is centos-$host================="
echo "--> excute command \"$cmd\""
ssh $user@centos-$host $cmd
done
echo "excute successfully !"
附录2. xsync脚本
#!/bin/bash
# 获取输出参数，如果没有参数则直接返回
pcount=$#
if [ $pcount -eq 0 ]
then
echo "no parameter find !";
exit;
fi
# 获取传输文件名
p1=$1
filename=`basename $p1`
echo "load file $p1 success !"
# 获取文件的绝对路径
pdir=`cd -P $(dirname $p1); pwd`
echo "file path is $pdir"
# 获取当前用户（如果想使用root用户权限拷贝文件，在命令后加入-root参数即可）
user=$2
case "$user" in
"-root")
user="root";;
"")
user=`whoami`;;
*)
echo "illegal parameter $user"
esac
echo $user
# 拷贝文件到从机(这里注意主机的host需要根据你的实际情况配置，要与你具体的主机名对应)
for (( host=2;host<=3;host++ ))
do
echo "================current host is centos-$host================="
rsync -rvl $pdir/$filename $user@centos-$host:$pdir
done
echo "complate !"
华为开发者空间，是为全球开发者打造的专属开发空间，汇聚了华为优质开发资源及工具，致力于让每一位开发者拥有一台云主机，基于华为根生态开发、创新。
更多推荐
23
0- 0
已为社区贡献1条内容
所有评论(0)