2021-04-25 21:51:03,732 ERROR tool.ExportTool: Encountered IOException running export job: java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1619356750085_0009 to YARN : root is not a leaf queue

​ sqoop导出数据报错,执行命令语句如下：

bin/sqoop export \
--D mapred.job.queue.name=root.root \
--connect jdbc:mysql://mysqlhost:3306/weblog \
--username root --password 12345678 \
--table dw_pvs_referer_everyhour \
--fields-terminated-by '\001' \
--columns referer_url,hour,pv_referer_cnt \
--export-dir /weblog/export/dw_pvs_referer_everyhour

​ 报错信息:

[root@node03 sqoop]# bin/sqoop export \
> --D mapred.job.queue.name=root.root \
> --connect jdbc:mysql://mysqlhost:3306/weblog \
> --username root --password 12345678 \
> --table dw_pvs_referer_everyhour \
> --fields-terminated-by '\001' \
> --columns referer_url,hour,pv_referer_cnt \
> --export-dir /weblog/export/dw_pvs_referer_everyhour
Warning: /home/software/sqoop/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/software/sqoop/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/software/sqoop/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/software/sqoop/bin/../../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
2021-04-25 21:50:19,523 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
2021-04-25 21:50:19,618 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
2021-04-25 21:50:19,718 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
2021-04-25 21:50:19,719 INFO tool.CodeGenTool: Beginning code generation
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2021-04-25 21:50:20,527 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `dw_pvs_referer_everyhour` AS t LIMIT 1
2021-04-25 21:50:20,606 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `dw_pvs_referer_everyhour` AS t LIMIT 1
2021-04-25 21:50:20,661 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/software/hadoop-3.3.0
注: /tmp/sqoop-root/compile/97af14ab892008ad2438f0c1e63caa40/dw_pvs_referer_everyhour.java使用或覆盖了已过时的 API。
注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
2021-04-25 21:50:22,524 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/97af14ab892008ad2438f0c1e63caa40/dw_pvs_referer_everyhour.jar
2021-04-25 21:50:22,535 INFO mapreduce.ExportJobBase: Beginning export of dw_pvs_referer_everyhour
2021-04-25 21:50:22,535 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2021-04-25 21:50:22,675 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
2021-04-25 21:50:24,265 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
2021-04-25 21:50:24,270 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
2021-04-25 21:50:24,270 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2021-04-25 21:50:24,417 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at node01/192.168.3.60:8032
2021-04-25 21:50:24,979 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1619356750085_0008
2021-04-25 21:50:26,909 INFO input.FileInputFormat: Total input files to process : 1
2021-04-25 21:50:26,915 INFO input.FileInputFormat: Total input files to process : 1
2021-04-25 21:50:27,005 INFO mapreduce.JobSubmitter: number of splits:4
2021-04-25 21:50:27,052 INFO Configuration.deprecation: mapred.job.queue.name is deprecated. Instead, use mapreduce.job.queuename
2021-04-25 21:50:27,052 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
2021-04-25 21:50:27,211 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1619356750085_0008
2021-04-25 21:50:27,212 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-04-25 21:50:27,480 INFO conf.Configuration: resource-types.xml not found
2021-04-25 21:50:27,481 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2021-04-25 21:50:27,581 INFO impl.YarnClientImpl: Submitted application application_1619356750085_0008
2021-04-25 21:50:27,640 INFO mapreduce.Job: The url to track the job: http://node01:8088/proxy/application_1619356750085_0008/
2021-04-25 21:50:27,641 INFO mapreduce.Job: Running job: job_1619356750085_0008
^C^C[root@node03 sqoop]# 
[root@node03 sqoop]# 
[root@node03 sqoop]# 
[root@node03 sqoop]# 
[root@node03 sqoop]# 
[root@node03 sqoop]# 
[root@node03 sqoop]# 
[root@node03 sqoop]# 
[root@node03 sqoop]# 
[root@node03 sqoop]# 
[root@node03 sqoop]# 
[root@node03 sqoop]# 
[root@node03 sqoop]# 
[root@node03 sqoop]# 
[root@node03 sqoop]# bin/sqoop export \
> --connect jdbc:mysql://mysqlhost:3306/weblog \
> --username root --password 12345678 \
> --table dw_pvs_referer_everyhour \
> --fields-terminated-by '\001' \
> --columns referer_url,hour,pv_referer_cnt \
> --export-dir /weblog/export/dw_pvs_referer_everyhour
Warning: /home/software/sqoop/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/software/sqoop/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/software/sqoop/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/software/sqoop/bin/../../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
2021-04-25 21:50:48,716 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
2021-04-25 21:50:48,925 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
2021-04-25 21:50:49,127 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
2021-04-25 21:50:49,127 INFO tool.CodeGenTool: Beginning code generation
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2021-04-25 21:50:51,018 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `dw_pvs_referer_everyhour` AS t LIMIT 1
2021-04-25 21:50:51,229 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `dw_pvs_referer_everyhour` AS t LIMIT 1
2021-04-25 21:50:51,316 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/software/hadoop-3.3.0
注: /tmp/sqoop-root/compile/fff77741758ebbac7c72bf0a18a1d75b/dw_pvs_referer_everyhour.java使用或覆盖了已过时的 API。
注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
2021-04-25 21:50:55,260 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/fff77741758ebbac7c72bf0a18a1d75b/dw_pvs_referer_everyhour.jar
2021-04-25 21:50:55,272 INFO mapreduce.ExportJobBase: Beginning export of dw_pvs_referer_everyhour
2021-04-25 21:50:55,272 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2021-04-25 21:50:55,521 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
2021-04-25 21:50:58,108 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
2021-04-25 21:50:58,115 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
2021-04-25 21:50:58,116 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2021-04-25 21:50:58,268 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at node01/192.168.3.60:8032
2021-04-25 21:50:59,202 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1619356750085_0009
2021-04-25 21:51:02,623 INFO input.FileInputFormat: Total input files to process : 1
2021-04-25 21:51:02,630 INFO input.FileInputFormat: Total input files to process : 1
2021-04-25 21:51:02,721 INFO mapreduce.JobSubmitter: number of splits:4
2021-04-25 21:51:02,756 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
2021-04-25 21:51:03,117 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1619356750085_0009
2021-04-25 21:51:03,118 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-04-25 21:51:03,615 INFO conf.Configuration: resource-types.xml not found
2021-04-25 21:51:03,616 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2021-04-25 21:51:03,717 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/root/.staging/job_1619356750085_0009
2021-04-25 21:51:03,732 ERROR tool.ExportTool: Encountered IOException running export job: java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1619356750085_0009 to YARN : root is not a leaf queue
[root@node03 sqoop]# 

报错图片

解决方案，添加–D参数，如下:

--D mapred.job.queue.name=root.root 

命令修改前:

bin/sqoop export \
--connect jdbc:mysql://mysqlhost:3306/weblog \ 
--username root --password 12345678 \
--table dw_pvs_referer_everyhour \
--fields-terminated-by '\001' \
--columns referer_url,hour,pv_referer_cnt \
--export-dir /weblog/export/dw_pvs_referer_everyhour

命令修改后:

bin/sqoop export \
--D mapred.job.queue.name=root.root \  ## 重点
--connect jdbc:mysql://mysqlhost:3306/weblog \
--username root --password 12345678 \
--table dw_pvs_referer_everyhour \
--fields-terminated-by '\001' \
--columns referer_url,hour,pv_referer_cnt \
--export-dir /weblog/export/dw_pvs_referer_everyhour

具体报错图片日志:

相关大数据学习demo地址:

https://github.com/carteryh/big-data