Kafka 监控 EFAK(即原 Kafka Eagle) 安装配置

Kafka Eagle 在 v2.0.7 版后更名为 EFAK

修改 Kafka 启动命令

关闭 Kafka 集群

$ kafka.sh stop

(kafka.sh 是 Kafka 集群启停脚本,在 Kafka 集群安装文章中有代码 Kafka 集群安装及常用命令)

修改启动脚本

$ vim /home/hadoop/local/kafka/bin/kafka-server-start.sh

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    exportKAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
fi

改为:

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then                                                                                                                       
#    export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"                                                                                                                  
    export KAFKA_HEAP_OPTS="-server -Xms2G -Xmx2G -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=5 -XX:Ini
tiatingHeapOccupancyPercent=70"                                                                                                                              
    export JMX_PORT="9999"                                                                                                                                                                                                                                                                                                
fi     

分发启动命令:

分发到 ns2、ns3 服务器上;

$ xsync /home/hadoop/local/kafka/bin/kafka-server-start.sh

修改 Zookeeper 启动文件

如果 Zookeeper 版本在 3.5 以上,需要修改 zkServer.sh 启动文件;

停止 Zookeeper 集群

$ zk.sh stop

(zk.sh 是 Zookeeper 集群启停脚本,在 Zookeeper 集群安装文章中有代码 Zookeeper 集群安装)

修改启动脚本

$ vim /home/hadoop/local/zookeeper/bin/zkServer.sh

在第 77 行添加如下一行代码

ZOOMAIN="-Dzookeeper.4lw.commands.whitelist=* ${ZOOMAIN}"

分发启动脚本

$ xsync /home/hadoop/local/zookeeper/bin/zkServer.sh

重新启动 Zookeeper、Kafka 集群

$ zk.sh start

$ kafka.sh start

安装 Kafka Eagle

官网:http://www.kafka-eagle.org/index.html

Github:https://github.com/smartloli/EFAK

下载 3.0.1 版:

$ wget https://github.com/smartloli/kafka-eagle-bin/archive/v3.0.1.tar.gz

$ cd kafka-eagle-bin-3.0.1

$ tar xzvf efak-web-3.0.1-bin.tar.gz -C /home/hadoop/local

$ cd /home/hadoop/local

$ ln -s efak-web-3.0.1 efak

修改配置文件

$ cd /home/hadoop/local/efak

$ vim conf/system-config.properties

修改 zk 集群信息:把 cluster2 都注释掉

######################################                                                                                                                       
# multi zookeeper & kafka cluster list                                                                                                                       
# Settings prefixed with 'kafka.eagle.' will be deprecated, use 'efak.' instead                                                                              
###################################### 
efak.zk.cluster.alias=cluster1                                                                                                                               
cluster1.zk.list=ns1:2181,ns2:2181,ns3:2181/kafka                                                                                                                
#cluster2.zk.list=xdn10:2181,xdn11:2181,xdn12:2181  

修改 kafka offset storage

######################################                                                                                                                       
# kafka offset storage                                                                                                                                       
######################################                                                                                                                       
cluster1.efak.offset.storage=kafka                                                                                                                           
#cluster2.efak.offset.storage=zk      

修改 MySQL 连接信息

######################################                                                                                                                       
# kafka mysql jdbc driver address                                                                                                                            
######################################                                                                                                                       
efak.driver=com.mysql.cj.jdbc.Driver                                                                                                                         
efak.url=jdbc:mysql://10.2.179.225:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull                                        
efak.username=root                                                                                                                                           
efak.password=123456  

其他保持不变,修改完的所有配置项:

efak.zk.cluster.alias=cluster1                                                                                                                               
cluster1.zk.list=ns1:2181,ns2:2181,ns3:2181/kafka                                                                                                              
cluster1.zk.acl.enable=false                                                                                                                                 
cluster1.zk.acl.schema=digest                                                                                                                                
cluster1.zk.acl.username=test                                                                                                                                
cluster1.zk.acl.password=test123                                                                                                                             
cluster1.efak.broker.size=20                                                                                                                                 
kafka.zk.limit.size=16                                                                                                                                       
efak.webui.port=8048                                                                                                                                         
efak.distributed.enable=false                                                                                                                                
efak.cluster.mode.status=master                                                                                                                              
efak.worknode.master.host=localhost                                                                                                                          
efak.worknode.port=8085                                                                                                                                      
cluster1.efak.jmx.acl=false                                                                                                                                  
cluster1.efak.jmx.user=keadmin                                                                                                                               
cluster1.efak.jmx.password=keadmin123                                                                                                                        
cluster1.efak.jmx.ssl=false                                                                                                                                  
cluster1.efak.jmx.truststore.location=/data/ssl/certificates/kafka.truststore                                                                                
cluster1.efak.jmx.truststore.password=ke123456                                                                                                               
cluster1.efak.offset.storage=kafka                                                                                                                           
cluster1.efak.jmx.uri=service:jmx:rmi:///jndi/rmi://%s/jmxrmi                                                                                                
efak.metrics.charts=true                                                                                                                                     
efak.metrics.retain=15                                                                                                                                       
efak.sql.topic.records.max=5000                                                                                                                              
efak.sql.topic.preview.records.max=10                                                                                                                        
efak.topic.token=keadmin                                                                                                                                     
cluster1.efak.sasl.enable=false                                                                                                                              
cluster1.efak.sasl.protocol=SASL_PLAINTEXT                                                                                                                   
cluster1.efak.sasl.mechanism=SCRAM-SHA-256                                                                                                                   
cluster1.efak.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="kafka" password="kafka-eagle";                     
cluster1.efak.sasl.client.id=                                                                                                                                
cluster1.efak.blacklist.topics=                                                                                                                              
cluster1.efak.sasl.cgroup.enable=false                                                                                                                       
cluster1.efak.sasl.cgroup.topics=                                                                                                                            
cluster2.efak.sasl.enable=false                                                                                                                              
cluster2.efak.sasl.protocol=SASL_PLAINTEXT                                                                                                                   
cluster2.efak.sasl.mechanism=PLAIN                                                                                                                           
cluster2.efak.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="kafka" password="kafka-eagle";                     
cluster2.efak.sasl.client.id=                                                                                                                                
cluster2.efak.blacklist.topics=                                                                                                                              
cluster2.efak.sasl.cgroup.enable=false                                                                                                                       
cluster2.efak.sasl.cgroup.topics=                                                                                                                            
cluster3.efak.ssl.enable=false                                                                                                                               
cluster3.efak.ssl.protocol=SSL                                                                                                                               
cluster3.efak.ssl.truststore.location=                                                                                                                       
cluster3.efak.ssl.truststore.password=                                                                                                                       
cluster3.efak.ssl.keystore.location=                                                                                                                         
cluster3.efak.ssl.keystore.password=                                                                                                                         
cluster3.efak.ssl.key.password=                                                                                                                              
cluster3.efak.ssl.endpoint.identification.algorithm=https                                                                                                    
cluster3.efak.blacklist.topics=                                                                                                                              
cluster3.efak.ssl.cgroup.enable=false                                                                                                                        
cluster3.efak.ssl.cgroup.topics=                                                                                                                             
efak.driver=com.mysql.cj.jdbc.Driver                                                                                                                         
efak.url=jdbc:mysql://10.2.179.225:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull                                        
efak.username=root                                                                                                                                           
efak.password=123456

添加环境变量

$ sudo vim /etc/profile.d/my_env.sh

HADOOP_HOME=/home/local/hadoop                                                                                                                               
ZOOKEEPER_HOME=/home/hadoop/local/zookeeper                                                                                                                  
KAFKA_HOME=/home/hadoop/local/kafka                                                                                                                          
KE_HOME=/home/hadoop/local/efak                                                                                                                              
PATH=$PATH:/home/hadoop/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin:$KAFKA_HOME/bin:$KE_HOME/bin                                              
export HADOOP_HOME ZOOKEEPER_HOME KAFKA_HOME KE_HOME PATH

$ source /etc/profile

启动 EFAK

$ ke.sh start

Welcome to                                                                                                                                                   
    ______    ______    ___     __ __                                                                                                                        
   / ____/   / ____/   /   |   / //_/                                                                                                                        
  / __/     / /_      / /| |  / ,<                                                                                                                           
 / /___    / __/     / ___ | / /| |                                                                                                                          
/_____/   /_/       /_/  |_|/_/ |_|                                                                                                                          
( Eagle For Apache Kafka® )                                                                                                                                  
                                                                                                                                                             
Version v3.0.1 -- Copyright 2016-2022                                                                                                                        
*******************************************************************                                                                                          
* EFAK Service has started success.                                                                                                                          
* Welcome, Now you can visit 'http://10.2.179.225:8048'                                                                                                      
* Account:admin ,Password:123456                                                                                                                             
*******************************************************************                                                                                          
*  ke.sh [start|status|stop|restart|stats]                                                                                                    
*  https://www.kafka-eagle.org/                                                                                                               
*******************************************************************   

在输出信息中写明了 web 的地址和用户名、密码

浏览器打开:http://ns1:8048

用户名:admin,密码:123456

登录进入;

Logo

为开发者提供学习成长、分享交流、生态实践、资源工具等服务,帮助开发者快速成长。

更多推荐