linux下对Hadoop3.1.4源码编译（详细步骤，附所有资源下载地址）

Hadoop源码编译环境准备1.系统环境虚拟机安装 centos7已配置好网卡，能联网root用户安装2.软件安装包下载jdk8下载地址https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.htmlhadoop-3.1.4-src.tar.gz 源码下载https://hadoop.apach

薛定谔的猫不吃猫粮

1647人浏览 · 2021-05-06 16:28:59

薛定谔的猫不吃猫粮 · 2021-05-06 16:28:59 发布

Hadoop源码编译

环境准备

1.系统环境

虚拟机安装 centos7
已配置好网卡，能联网
root用户安装

2.软件安装包下载

jdk8下载地址

https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
hadoop-3.1.4-src.tar.gz 源码下载

https://hadoop.apache.org/releases.html
apache-maven-3.6.3-bin.tar.gz下载地址

https://maven.apache.org/download.cgi
apache-ant-1.10.9-bin.tar.gz下载地址

https://www.apache.org/dist/ant/binaries/
protobuf-2.5.0.tar.gz下载地址

https://github.com/protocolbuffers/protobuf/releases/tag/v2.5.0

jar包安装

1.JDK1.8安装，配置环境变量

略，请自行查询安装

2.Maven配置安装

2.1 解压到指定文件夹（目标文件根据自己的情况而定）

[root@hdp16 hadoop-src]# ls
apache-ant-1.10.9-bin.tar.gz  apache-maven-3.6.3-bin.tar.gz  hadoop-3.1.4-src.tar.gz  protobuf-2.5.0.tar.gz
[root@hdp16 hadoop-src]# pwd
/opt/resource/hadoop-src
[root@hdp16 hadoop-src]# tar -zxvf apache-maven-3.6.3-bin.tar.gz -C /opt/bigdata/
apache-maven-3.6.3/README.txt
apache-maven-3.6.3/LICENSE
apache-maven-3.6.3/NOTICE
apache-maven-3.6.3/lib/
apache-maven-3.6.3/lib/cdi-api.license
apache-maven-3.6.3/lib/commons-cli.license
......

2.2 修改maven配置文件settings.xml，添加阿里镜像

[root@hdp16 conf]# pwd
/opt/bigdata/apache-maven-3.6.3/conf
[root@hdp16 conf]# vim settings.xml

找到，在其内添加如下配置：

<mirror>
    <id>nexus-aliyun</id>
    <mirrorOf>central</mirrorOf>
    <name>Nexus aliyun</name>
    <url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>

2.3 配置maven环境变量

[root@hdp16 apache-maven-3.6.3]# vi /etc/profile
#在文件末尾添加如下内容，注意自己maven的安装路径
#MAVEN_HOME
export MAVEN_HOME=/opt/bigdata/apache-maven-3.6.3
export PATH=$PATH:$MAVEN_HOME/bin

[root@hdp16 apache-maven-3.6.3]# source /etc/profile

3.ant 安装配置

3.1 解压到指定目录

[root@hdp16 hadoop-src]# pwd
/opt/resource/hadoop-src
[root@hdp16 hadoop-src]# tar -zxvf apache-ant-1.10.9-bin.tar.gz -C /opt/bigdata/

3.2 配置环境变量

vi /etc/profile
#在文件末尾添加如下内容
#ANT_HOME
export ANT_HOME=/opt/bigdata/apache-ant-1.10.9
export PATH=$PATH:$ANT_HOME/bin

[root@hdp16 apache-maven-3.6.3]# source /etc/profile

4.验证jdk，maven，ant是否配置正确

[root@hdp16 hadoop-src]# java -version
java version "1.8.0_212"
Java(TM) SE Runtime Environment (build 1.8.0_212-b10)
Java HotSpot(TM) 64-Bit Server VM (build 25.212-b10, mixed mode)
[root@hdp16 hadoop-src]# mvn -version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /opt/bigdata/apache-maven-3.6.3
Java version: 1.8.0_181, vendor: Oracle Corporation, runtime: /usr/java/jdk1.8.0_181-cloudera/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1160.el7.x86_64", arch: "amd64", family: "unix"
[root@hdp16 hadoop-src]# ant -version
Apache Ant(TM) version 1.10.9 compiled on September 27 2020

5.yum安装依赖包

yum install gcc gcc-c++
yum install make cmake 
yum install autoconf automake libtool curl
yum install lzo-devel zlib-devel openssl openssl-devel ncurses-devel
yum install snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop libXtst

6.安装Protobuf

6.1解压安装包到指定目录

[root@hdp16 hadoop-src]# pwd
/opt/resource/hadoop-src
[root@hdp16 hadoop-src]# tar -zxvf protobuf-2.5.0.tar.gz -C /opt/bigdata/

6.2 安装配置

[root@hdp16 protobuf-2.5.0]# pwd
/opt/bigdata/protobuf-2.5.0
[root@hdp16 protobuf-2.5.0]# ls
aclocal.m4   config.guess  configure         COPYING.txt  examples                      install-sh   ltmain.sh    Makefile.in          protobuf.pc.in  src
autogen.sh   config.h.in   configure.ac      depcomp      generate_descriptor_proto.sh  INSTALL.txt  m4           missing              python          vsprojects
CHANGES.txt  config.sub    CONTRIBUTORS.txt  editors      gtest                         java         Makefile.am  protobuf-lite.pc.in  README.txt
[root@hdp16 protobuf-2.5.0]# ./configure 
......

[root@hdp16 protobuf-2.5.0]# make check
......
[root@hdp16 protobuf-2.5.0]# make install
......
[root@hdp16 protobuf-2.5.0]# ldconfig
......

#配置环境变量
[root@hdp16 protobuf-2.5.0]# pwd
/opt/bigdata/protobuf-2.5.0
[root@hdp16 protobuf-2.5.0]# vi /etc/profile
#在文件末尾添加如下内容：
#LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/bigdata/protobuf-2.5.0
export PATH=$PATH:$LD_LIBRARY_PATH
[root@hdp16 protobuf-2.5.0]# source /etc/profile

编译源码

1.解压源码

[root@hdp16 hadoop-src]# ls
apache-ant-1.10.9-bin.tar.gz  apache-maven-3.6.3-bin.tar.gz  hadoop-3.1.4-src.tar.gz  protobuf-2.5.0.tar.gz
[root@hdp16 hadoop-src]# pwd
/opt/resource/hadoop-src
[root@hdp16 hadoop-src]# tar -zxvf hadoop-3.1.4-src.tar.gz -C /opt/
......
[root@hdp16 hadoop-src]# cd /opt/
[root@hdp16 opt]# ls
bigdata  cloudera  cloudera-manager  hadoop-3.1.4-src  module  resource  software
[root@hdp16 opt]# cd hadoop-3.1.4-src/
[root@hdp16 hadoop-3.1.4-src]# ls
BUILDING.txt       hadoop-build-tools            hadoop-common-project  hadoop-mapreduce-project  hadoop-project       hadoop-yarn-project  NOTICE.txt    README.txt
dev-support        hadoop-client-modules         hadoop-dist            hadoop-maven-plugins      hadoop-project-dist  Jenkinsfile          patchprocess  start-build-env.sh
hadoop-assemblies  hadoop-cloud-storage-project  hadoop-hdfs-project    hadoop-minicluster        hadoop-tools         LICENSE.txt          pom.xml

2.编译

源码目录执行如下命令，等待

[root@hdp16 hadoop-3.1.4-src]# mvn package -Pdist,native -DskipTests -Dtar

遇到如下错误

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:43 min
[INFO] Finished at: 2021-04-30T09:16:35+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.1.4:cmake-compile (cmake-compile) on project hadoop-common: CMake failed with error code 1 -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hadoop-common

卸载 cmake，重新安装

[root@hdp16 hadoop-3.1.4-src]# yum erase cmake

下载安装包

cmake-3.13.5.tar.gz，手动安装：

https://cmake.org/files/v3.13/

安装教程参考 https://www.cnblogs.com/liudw-0215/p/9877290.html

[root@hdp16 hadoop-src]# ls
apache-ant-1.10.9-bin.tar.gz  apache-maven-3.6.3-bin.tar.gz  cmake-3.13.5.tar.gz  hadoop-3.1.4-src.tar.gz  protobuf-2.5.0.tar.gz
[root@hdp16 hadoop-src]# pwd
/opt/resource/hadoop-src
#解压
[root@hdp16 hadoop-src]# tar -zxvf cmake-3.13.5.tar.gz -C /opt/bigdata/
...
#编译
[root@hdp16 cmake-3.13.5] ./configure  
...
#安装
[root@hdp16 cmake-3.13.5]#  sudo make && make install
#验证
[root@hdp16 cmake-3.13.5]# cmake -version
cmake version 3.13.5

CMake suite maintained and supported by Kitware (kitware.com/cmake).

重新进去源码目录，重新执行编译命令

经过漫长的等待，编译终于成功

 Reactor Summary for Apache Hadoop Main 3.1.4:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  6.136 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  5.946 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  2.541 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  3.955 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.321 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.979 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  7.253 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  4.419 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  8.949 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  4.395 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:50 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 10.181 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [  6.701 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.089 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [01:03 min]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:43 min]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  3.546 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 19.263 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  5.768 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 49.629 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.076 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.092 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 20.100 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [01:03 min]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  7.901 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.050 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 18.634 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 20.205 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  7.330 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [  7.996 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  6.143 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 35.662 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  0.962 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [  6.662 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  4.611 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  3.948 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [  0.056 s]
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [  6.872 s]
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [  6.143 s]
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [  0.043 s]
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2  SUCCESS [  6.142 s]
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [  3.106 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [  5.439 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.040 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  4.889 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  3.747 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.323 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 28.221 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 20.595 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  5.583 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 10.253 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  9.863 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  8.261 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.927 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [  0.036 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [  1.588 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [  1.521 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.055 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.064 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 10.699 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  3.906 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [  9.688 s]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [  2.401 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  6.423 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  5.309 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  9.735 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  6.077 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  2.907 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  2.749 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  9.198 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  7.328 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  4.882 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  3.831 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  0.451 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  5.952 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 22.555 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [ 32.827 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 32.073 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 43.862 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [  2.023 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  7.329 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 21.866 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 32.568 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [ 13.138 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 14.721 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.028 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [03:23 min]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [02:09 min]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [ 11.921 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [04:12 min]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [  0.201 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [  0.236 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 59.964 s]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [  0.064 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  0.760 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.041 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  28:44 min
[INFO] Finished at: 2021-05-06T16:15:17+08:00
[INFO] ------------------------------------------------------------------------

查看一下，编译成果：

[root@hdp16 hadoop-3.1.4-src]# ls
BUILDING.txt           hadoop-cloud-storage-project  hadoop-maven-plugins  hadoop-yarn-project  pom.xml
dev-support            hadoop-common-project         hadoop-minicluster    Jenkinsfile          README.txt
hadoop-assemblies      hadoop-dist                   hadoop-project        LICENSE.txt          start-build-env.sh
hadoop-build-tools     hadoop-hdfs-project           hadoop-project-dist   NOTICE.txt
hadoop-client-modules  hadoop-mapreduce-project      hadoop-tools          patchprocess
[root@hdp16 hadoop-3.1.4-src]# cd hadoop-dist/target/
[root@hdp16 target]# ls
antrun   hadoop-3.1.4         hadoop-tools-deps               test-classes
classes  hadoop-3.1.4.tar.gz  maven-shared-archive-resources  test-dir