Win主机+Linux虚拟机搭建Kettle集群
环境说明程序 版本Win:OS win 7Linux:OS Redhat 6.2JDK1.8.0_45Kettle 5.1本机:Vmnet1 192.168.52.1集群Master：192.168.52.1集群Slave1： 192.168.52.128集群Slave2： 192.168.52.1需要在Redhat 6.2上安装好jdk，配置好环境变量，并且...
环境说明
程序 版本
Win:OS win 7
Linux:OS Redhat 6.2
JDK 1.8.0_45
Kettle 5.1
本机:Vmnet1 192.168.52.1
集群Master： 192.168.52.1
集群Slave1： 192.168.52.128
集群Slave2： 192.168.52.1
需要在Redhat 6.2上安装好jdk，配置好环境变量，并且把kettle安装好。（kettle不需要启动spoon来启动图形界面，利用kitchen调试好即可，linux上部署kettle，解压即可，因为kettle是跨平台运行的）。
注意：在集群运行环境中，需要端口打开或者直接关闭防火墙，供主服务器及子服务器互相连通通讯，在Redhat 6.2中，防火墙采用iptables的方式开启和关闭，检查端口是否开通，可以通过本机进行telnet测试。
首先，配置kettle主目录下的文件夹pwd中的carte-config-master-8080.xml、carte-config-8081.xml和carte-config-8082.xml文件，配置文件见下图：
注意：配置文件中的username和password并不是指主机的登陆账号和密码，是集群的账号密码，该账号密码是集群连接的依据，账号密码是通过混淆的方式保存在pwd文件，kettle默认的账号密码是cluster/cluster，修改该账号密码需要修改pwd文件，该文件也在pwd目录中，密码做了混淆。
Tips：
Kettle的集群搭建还是比较简单的，但是简单也有坑。
坑1：基于主机+虚拟机搭建集群的时候，一定要注意虚拟机与主机的网络通信方式，同时要考虑win主机的hosts文件的内容修改；
坑2：官网下载的kettle5或者6的主目录下是没有安装集群所必须的必要文件，可以去该地址下载相应文件：
Maven Central Repository Search
http://repo1.maven.org/maven2/org/owasp/esapi/esapi/2.1.0.1/
集群必要文件列表：
ESAPI.properties
validation.properties
antisamy-esapi.xml
集群必要jar文件：
esapi-2.1.0.1.jar
这个jar包记得放在lib文件夹里。
下面把配置步骤做如下描述：
一. MASTER 配置
进到pwd目录，打开文件carte-config-master-8080.xml，内容如下
| <slave_config> <!-- Document description...
- masters: You can list the slave servers to which this slave has to report back to. If this is a master, we will contact the other masters to get a list of all the slaves in the cluster. - report_to_masters : send a message to the defined masters to let them know we exist (Y/N) - slaveserver : specify the slave server details of this carte instance. IMPORTANT : the username and password specified here are used by the master instances to connect to this slave. --> <slaveserver> <name>master1</name> <hostname>192.168.52.1</hostname> <port>8080</port> <master>Y</master> </slaveserver> </slave_config> |
Tips：这里注意<hostname>的value值选择本机的Vmnet1地址，因为我的主机与虚拟机是通过host-only方式通信的，如果你的虚拟机是通过NAT方式通信，那么此时的value要填入Vmnet8地址，原因自己想，理由很简单。
此时MASTER配置完毕。
二. SLAVE2 配置
进到pwd目录，打开文件carte-config-8082.xml，内容如下
| <slave_config> <!-- Document description...
- masters: You can list the slave servers to which this slave has to report back to. If this is a master, we will contact the other masters to get a list of all the slaves in the cluster. - report_to_masters : send a message to the defined masters to let them know we exist (Y/N) - slaveserver : specify the slave server details of this carte instance. IMPORTANT : the username and password specified here are used by the master instances to connect to this slave. --> <masters> <slaveserver> <name>master1</name> <hostname>192.168.52.1</hostname> <port>8080</port> <username>cluster</username> <password>cluster</password> <master>Y</master> </slaveserver> </masters> <report_to_masters>Y</report_to_masters> <slaveserver> <name>slave2-8082</name> <hostname>192.168.52.1</hostname> <port>8082</port> <username>cluster</username> <password>cluster</password> <master>N</master> </slaveserver> </slave_config> |
Tips：
子服务器比主服务器的配置内容多一点，该子服务器其实是与MASTER配置在一台主机上，除了自身的配置信息以外，要把MASTER的配置信息也配置好，因为启动子服务器的时候，子服务器需要根据配置文件中的MASTER地址和端口去MASTER主机注册服务，这里要注意的是配置子服务器的地方有个K-V配置，即<master>N</master>，如果是配置MASTER，这个位置的值是Y，如果配置的是子服务器，那么这个位置的值是N。
三. SLAVE3 配置
进到pwd目录，打开文件carte-config-8081.xml，内容如下
| <slave_config> <!-- Document description... - masters: You can list the slave servers to which this slave has to report back to. If this is a master, we will contact the other masters to get a list of all the slaves in the cluster. - report_to_masters : send a message to the defined masters to let them know we exist (Y/N) - slaveserver : specify the slave server details of this carte instance. IMPORTANT : the username and password specified here are used by the master instances to connect to this slave. --> <masters> <slaveserver> <name>master1</name> <hostname>192.168.52.1</hostname> <port>8080</port> <username>cluster</username> <password>cluster</password> <master>Y</master> </slaveserver> </masters> <report_to_masters>Y</report_to_masters> <slaveserver> <name>slave1-8081</name> <hostname>192.168.52.128</hostname> <port>8081</port> <username>cluster</username> <password>cluster</password> <master>N</master> |
Tips：slave3是一台linux虚拟机，特别要注意在配置MASTER时的hostname的值，同时还要切记将前文提到的三个必要文件放在kettle的主目录中，否则会报错，报错内容如下：
| Attempting to load validation.properties via file I/O. Attempting to load validation.properties as resource file via file I/O. Not found in 'org.owasp.esapi.resources' directory or file not readable: /opt/data-integration-5.1/validation.properties Not found in SystemResource Directory/resourceDirectory: .esapi/validation.properties Not found in 'user.home' (/root) directory: /root/esapi/validation.properties Loading validation.properties via file I/O failed. Attempting to load validation.properties via the classpath. validation.properties could not be loaded by any means. fail. Exception was: java.lang.IllegalArgumentException: Failed to load ESAPI.properties as a classloader resource. |
启动MASTER主机的carte服务：
| D:\software\data-integration-5.1>Carte.bat pwd\carte-config-master-8080.xml DEBUG: Using JAVA_HOME DEBUG: _PENTAHO_JAVA_HOME=D:\jdk1.8.0_74 DEBUG: _PENTAHO_JAVA=D:\jdk1.8.0_74\bin\java.exe 2017/12/27 16:31:33 - Carte - Installing timer to purge stale objects after 1440 minutes. 2017-12-27 16:31:33.331::INFO: Logging to STDERR via org.mortbay.log.StdErrLog 2017/12/27 16:31:33 - Carte - 创建 web 服务监听器 @ 地址: 192.168.52.1:8080 2017-12-27 16:31:33.433::INFO: jetty-6.1.21 2017-12-27 16:31:33.458::INFO: Started SocketConnector@192.168.52.1:8080 Attempting to load ESAPI.properties via file I/O. Attempting to load ESAPI.properties as resource file via file I/O. Not found in 'org.owasp.esapi.resources' directory or file not readable: D:\soft ware\data-integration-5.1\ESAPI.properties Not found in SystemResource Directory/resourceDirectory: .esapi\ESAPI.properties Not found in 'user.home' (C:\Users\Administrator) directory: C:\Users\Administra tor\esapi\ESAPI.properties Loading ESAPI.properties via file I/O failed. Exception was: java.io.FileNotFoun dException Attempting to load ESAPI.properties via the classpath. SUCCESSFULLY LOADED ESAPI.properties via the CLASSPATH from '/ (root)' using cur rent thread context class loader! Attempting to load validation.properties via file I/O. Attempting to load validation.properties as resource file via file I/O. Not found in 'org.owasp.esapi.resources' directory or file not readable: D:\soft ware\data-integration-5.1\validation.properties Not found in SystemResource Directory/resourceDirectory: .esapi\validation.prope rties Not found in 'user.home' (C:\Users\Administrator) directory: C:\Users\Administra tor\esapi\validation.properties Loading validation.properties via file I/O failed. Attempting to load validation.properties via the classpath. SUCCESSFULLY LOADED validation.properties via the CLASSPATH from '/ (root)' usin g current thread context class loader! |
打开浏览器，输入地址：http://192.168.52.1:8080
输入用户名和密码(就是在配置文件中配置的用户名和密码)
Tips:注意右下角，有提示这是MASTER.
启动slave1
| [root@racnode2 data-integration-5.1]# carte.sh pwd/carte-config-8081.xml /opt/data-integration-5.1 2017/12/27 00:34:21 - Carte - Installing timer to purge stale objects after 1440 minutes. System property [org.owasp.esapi.opsteam] is not set System property [org.owasp.esapi.devteam] is not set Attempting to load ESAPI.properties via file I/O. Attempting to load ESAPI.properties as resource file via file I/O. Not found in 'org.owasp.esapi.resources' directory or file not readable: /opt/data-integration-5.1/ESAPI.properties Not found in SystemResource Directory/resourceDirectory: .esapi/ESAPI.properties Not found in 'user.home' (/root) directory: /root/esapi/ESAPI.properties Loading ESAPI.properties via file I/O failed. Exception was: java.io.FileNotFoundException Attempting to load ESAPI.properties via the classpath. SUCCESSFULLY LOADED ESAPI.properties via the CLASSPATH from '/ (root)' using current thread context class loader! SecurityConfiguration for Validator.ConfigurationFile.MultiValued not found in ESAPI.properties. Using default: false Attempting to load validation.properties via file I/O. Attempting to load validation.properties as resource file via file I/O. Not found in 'org.owasp.esapi.resources' directory or file not readable: /opt/data-integration-5.1/validation.properties Not found in SystemResource Directory/resourceDirectory: .esapi/validation.properties Not found in 'user.home' (/root) directory: /root/esapi/validation.properties Loading validation.properties via file I/O failed. Attempting to load validation.properties via the classpath. SUCCESSFULLY LOADED validation.properties via the CLASSPATH from '/ (root)' using current thread context class loader! 2017/12/27 00:34:21 - Carte - Registered this slave server to master slave server [master1] on address [192.168.52.1:8080] 2017/12/27 00:34:21 - Carte - Registered this slave server to master slave server [master1] on address [192.168.52.1:8080] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See SLF4J Error Codes for further details. 2017/12/27 00:34:22 - Carte - Created listener for webserver @ address : 192.168.52.128:8081 |
打开浏览器，输入地址：http://192.168.52.128:8081
启动slave2
| D:\software\data-integration-5.1>Carte.bat pwd\carte-config-8082.xml DEBUG: Using JAVA_HOME DEBUG: _PENTAHO_JAVA_HOME=D:\jdk1.8.0_74 DEBUG: _PENTAHO_JAVA=D:\jdk1.8.0_74\bin\java.exe 2017/12/27 16:38:09 - Carte - Installing timer to purge stale objects after 1440 minutes. Attempting to load ESAPI.properties via file I/O. Attempting to load ESAPI.properties as resource file via file I/O. Not found in 'org.owasp.esapi.resources' directory or file not readable: D:\soft ware\data-integration-5.1\ESAPI.properties Not found in SystemResource Directory/resourceDirectory: .esapi\ESAPI.properties Not found in 'user.home' (C:\Users\Administrator) directory: C:\Users\Administra tor\esapi\ESAPI.properties Loading ESAPI.properties via file I/O failed. Exception was: java.io.FileNotFoun dException Attempting to load ESAPI.properties via the classpath. SUCCESSFULLY LOADED ESAPI.properties via the CLASSPATH from '/ (root)' using cur rent thread context class loader! Attempting to load validation.properties via file I/O. Attempting to load validation.properties as resource file via file I/O. Not found in 'org.owasp.esapi.resources' directory or file not readable: D:\soft ware\data-integration-5.1\validation.properties Not found in SystemResource Directory/resourceDirectory: .esapi\validation.prope rties Not found in 'user.home' (C:\Users\Administrator) directory: C:\Users\Administra tor\esapi\validation.properties Loading validation.properties via file I/O failed. Attempting to load validation.properties via the classpath. SUCCESSFULLY LOADED validation.properties via the CLASSPATH from '/ (root)' usin g current thread context class loader! 2017/12/27 16:38:09 - Carte - Registered this slave server to master slave serve r [master1] on address [192.168.52.1:8080] 2017/12/27 16:38:09 - Carte - Registered this slave server to master slave serve r [master1] on address [192.168.52.1:8080] 2017-12-27 16:38:09.398::INFO: Logging to STDERR via org.mortbay.log.StdErrLog 2017/12/27 16:38:09 - Carte - 创建 web 服务监听器 @ 地址: 192.168.52.1:8082 2017-12-27 16:38:09.456::INFO: jetty-6.1.21 2017-12-27 16:38:09.475::INFO: Started SocketConnector@192.168.52.1:8082 |
打开浏览器，输入地址：http://192.168.52.1:8082
关于在kettle开发过程中转换和工作流里的集群配置，以后再写个文章说明。
华为开发者空间，是为全球开发者打造的专属开发空间，汇聚了华为优质开发资源及工具，致力于让每一位开发者拥有一台云主机，基于华为根生态开发、创新。
更多推荐
0
0- 0
已为社区贡献1条内容
所有评论(0)