Centos7虚拟机配置Hadoop

jdk安装将jdk安装包上传至/opt目录下执行rpm -ivh jdk-8u281-linux-x64.rpm通过xmanager的Xftp上传hadoop-3.1.4.tar.gz文件到/opt目录解压缩hadoop-3.1.4.tar.gz 文件tar -zxf hadoop-3.1.4.tar.gz -C /usr/local解压后即可，看到/usr/local/hadoop-3.1.4文

刘二光

1653人浏览 · 2022-01-20 01:47:35

刘二光 · 2022-01-20 01:47:35 发布

jdk安装
将jdk安装包上传至/opt目录下
执行
rpm -ivh jdk-8u281-linux-x64.rpm
1. 通过xmanager的Xftp上传hadoop-3.1.4.tar.gz文件到/opt目录

2. 解压缩hadoop-3.1.4.tar.gz 文件
tar -zxf hadoop-3.1.4.tar.gz -C /usr/local

解压后即可，看到/usr/local/hadoop-3.1.4文件夹

3. 配置Hadoop

进入目录：
cd /usr/local/hadoop-3.1.4/etc/hadoop/
4.依次修改下面的文件：
4.1 core-site.xml

fs.defaultFS
hdfs://master:8020

hadoop.tmp.dir
/var/log/hadoop/tmp

4.2 hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_281-amd64

4.3 hdfs-site.xml

dfs.namenode.name.dir file:///data/hadoop/hdfs/name dfs.datanode.data.dir file:///data/hadoop/hdfs/data dfs.namenode.secondary.http-address master:50090 dfs.replication 3

4.4 mapred-site.xml
复制cp mapred-site.xml.template mapred-site.xml

mapreduce.framework.name
yarn

mapreduce.jobhistory.address master:10020 mapreduce.jobhistory.webapp.address master:19888

4.5 yarn-site.xml

yarn.resourcemanager.hostname
master

yarn.resourcemanager.address
${yarn.resourcemanager.hostname}:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>$ {yarn.resourcemanager.hostname}:8030

yarn.resourcemanager.webapp.address
${yarn.resourcemanager.hostname}:8088</value> </property> <property> <name>yarn.resourcemanager.webapp.https.address</name> <value>$ {yarn.resourcemanager.hostname}:8090

yarn.resourcemanager.resource-tracker.address
${yarn.resourcemanager.hostname}:8031</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>$ {yarn.resourcemanager.hostname}:8033

yarn.nodemanager.local-dirs
/data/hadoop/yarn/local

yarn.log-aggregation-enable
true

yarn.nodemanager.remote-app-log-dir
/data/tmp/logs

yarn.log.server.url
http://master:19888/jobhistory/logs/
URL for job history server

yarn.nodemanager.vmem-check-enabled
false

yarn.nodemanager.aux-services
mapreduce_shuffle

yarn.nodemanager.aux-services.mapreduce.shuffle.class
org.apache.hadoop.mapred.ShuffleHandler

yarn.nodemanager.resource.memory-mb
2048

yarn.scheduler.minimum-allocation-mb
512

yarn.scheduler.maximum-allocation-mb
4096

mapreduce.map.memory.mb
2048

mapreduce.reduce.memory.mb
2048

yarn.nodemanager.resource.cpu-vcores
1

yarn.application.classpath /usr/local/hadoop-3.1.4/etc/hadoop:/usr/local/hadoop-3.1.4/share/hadoop/common/lib/*:/usr/local/hadoop-3.1.4/share/hadoop/common/*:/usr/local/hadoop-3.1.4/share/hadoop/hdfs:/usr/local/hadoop-3.1.4/share/hadoop/hdfs/lib/*:/usr/local/hadoop-3.1.4/share/hadoop/hdfs/*:/usr/local/hadoop-3.1.4/share/hadoop/mapreduce/lib/*:/usr/local/hadoop-3.1.4/share/hadoop/mapreduce/*:/usr/local/hadoop-3.1.4/share/hadoop/yarn:/usr/local/hadoop-3.1.4/share/hadoop/yarn/lib/*:/usr/local/hadoop-3.1.4/share/hadoop/yarn/*

4.6 yarn-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_281-amd64

4.7 worker
slave1
slave2
slave3

4.8 start-dfs.sh,stop-dfs.sh
HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root

4.9 start-yarn.sh,stop-yarn.sh

YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

4.10 设置IP映射
编辑各节点/etc/hosts
添加：
192.168.128.130 master master.centos.com
192.168.128.131 slave1 slave1.centos.com
192.168.128.132 slave2 slave2.centos.com

拷贝hadoop安装文件到集群slave节点
scp -r /usr/local/hadoop-3.1.4 slave1:/usr/local
scp -r /usr/local/hadoop-3.1.4 slave2:/usr/local

5.配置SSH无密码登录
（1）使用ssh-keygen产生公钥与私钥对。
输入命令“ssh-keygen -t rsa”，接着按三次Enter键

[root@master ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory ‘/root/.ssh’.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
a6:13:5a:7b:54:eb:77:58:bd:56:ef:d0:64:90:66:d4 root@master.centos.com
The key’s randomart image is:
±-[ RSA 2048]----+
| … |
| . .E|
| . = |
| . . o o |
| o S . . =|
| o * . o ++|
| . + . . o ooo|
| o . …o |
| .|
±----------------+

生成私有密钥id_rsa和公有密钥id_rsa.pub两个文件。ssh-keygen用来生成RSA类型的密钥以及管理该密钥，参数“-t”用于指定要创建的SSH密钥的类型为RSA。
（2）用ssh-copy-id将公钥复制到远程机器中
ssh-copy-id -i /root/.ssh/id_rsa.pub master//依次输入yes,123456（root用户的密码）
ssh-copy-id -i /root/.ssh/id_rsa.pub slave1
ssh-copy-id -i /root/.ssh/id_rsa.pub slave2

（3）验证是否设置无密码登录
依次输入
ssh slave1
ssh slave2

6.配置时间同步服务
（1）安装NTP服务。在各节点：
yum -y install ntp
（2）设置假设master节点为NTP服务主节点，那么其配置如下。
使用命令“vim /etc/ntp.conf”打开/etc/ntp.conf文件，注释掉以server开头的行，并添加:
restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap
server 127.127.1.0
fudge 127.127.1.0 stratum 10

（3）在slave中配置NTP，同样修改/etc/ntp.conf文件，注释掉server开头的行，并添加:
server master

（4）执行命令“systemctl stop firewalld.service & systemctl disable firewalld.service”永久性关闭防火墙，主节点和从节点都要关闭。
（5）启动NTP服务。
① 在master节点执行命令“service ntpd start & chkconfig ntpd on”
② 在slave上执行命令“ntpdate master”即可同步时间
③ 在slave上分别执行“service ntpd start & chkconfig ntpd on”即可启动并永久启动NTP服务。

7.在/etc/profile添加JAVA_HOME和Hadoop路径
export HADOOP_HOME=/usr/local/hadoop-3.1.4
export JAVA_HOME=/usr/java/jdk1.8.0_281-amd64
export PATH= $HADOOP_HOME/bin:$ PATH:$JAVA_HOME/bin

source /etc/profile使修改生效

8. 格式化NameNode
进入目录
cd /usr/local/hadoop-3.1.4/bin

执行格式化
./hdfs namenode -format

9.启动集群
进入目录
cd /usr/local/hadoop-3.1.4/sbin
执行启动：
./start-dfs.sh
./start-yarn.sh
./mr-jobhistory-daemon.sh start historyserver

使用jps，查看进程
[root@centos67 sbin]# jps
3672 NodeManager
3301 DataNode
3038 NameNode
4000 JobHistoryServer
4058 Jps
3589 ResourceManager
3408 SecondaryNameNode

10. 关闭防火墙（在所有节点执行）：
service iptables stop
chkconfig iptables off

11. 浏览器查看：

http://master:9870
http://master:8088