需求

统计hbase某个表的所有rowkey数量

方法

使用hbase提供的类，RowCounter。
具体命令：

./hbase org.apache.hadoop.hbase.mapreduce.RowCounter 'table_name'

出现异常

命令执行后控制台出现以下错误：

2021-04-21 15:46:19,787 INFO  [main] Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2021-04-21 15:46:19,978 INFO  [main] mapreduce.JobSubmitter: Submitting tokens for job: job_1618991128254_0001
2021-04-21 15:46:20,181 INFO  [main] conf.Configuration: resource-types.xml not found
2021-04-21 15:46:20,181 INFO  [main] resource.ResourceUtils: Unable to find 'resource-types.xml'.
2021-04-21 15:46:20,188 INFO  [main] resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE
2021-04-21 15:46:20,188 INFO  [main] resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE
2021-04-21 15:46:20,482 INFO  [main] impl.YarnClientImpl: Submitted application application_1618991128254_0001
2021-04-21 15:46:20,514 INFO  [main] mapreduce.Job: The url to track the job: http://datanode1:8088/proxy/application_1618991128254_0001/
2021-04-21 15:46:20,514 INFO  [main] mapreduce.Job: Running job: job_1618991128254_0001
2021-04-21 15:46:37,613 INFO  [main] mapreduce.Job: Job job_1618991128254_0001 running in uber mode : false
2021-04-21 15:46:37,615 INFO  [main] mapreduce.Job:  map 0% reduce 0%
2021-04-21 15:46:37,625 INFO  [main] mapreduce.Job: -04-21 15:46:36.856]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.


[2021-04-21 15:46:36.857]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.


For more detailed output, check the application tracking page: http://datanode1:8088/cluster/app/application_1618991128254_0001 Then click on links to logs of each attempt.
. Failing the application.
2021-04-21 15:46:37,642 INFO  [main] mapreduce.Job: Counters: 0

问题排查

通过hadoop yarn界面上的日志看到以下错误：

2021-04-21 15:46:35,115 ERROR [Listener at 0.0.0.0/35651] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Webapps failed to start. Ignoring for now:
java.lang.NullPointerException
	at org.apache.hadoop.util.StringUtils.join(StringUtils.java:941)
	at org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer.initFilter(AmFilterInitializer.java:74)
	at org.apache.hadoop.http.HttpServer2.initializeWebServer(HttpServer2.java:605)
	at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:558)
	at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:119)
	at org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:433)
	at org.apache.hadoop.yarn.webapp.WebApps$Builder.build(WebApps.java:341)
	at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:432)
	at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:428)
	at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:147)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1290)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1761)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
2021-04-21 15:46:35,119 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1618991128254_0001

解决方法

以上问题是由于hadoop执行mapreduce任务后，会通过yarn-site.xml获取resoucemanager的地址，但是yarn-site.xml里没有对应的配置。（明明hadoop启动yarn的时候默认了rs的地址，这里又通过配置文件获取，不是很太懂。。。）

所有需要在yarn-site.xml中添加rs的配置，并同步到其他节点即可，不需要重启hadoop集群。

具体配置内容如下：

<property>
    <name>yarn.resourcemanager.webapp.address.rm1</name>
    <value>datanode1:8088</value>
</property>
<property>
    <name>yarn.resourcemanager.webapp.address.rm2</name>
    <value>datanode2:8088</value>
</property>