Hadoop 实战之分析专利引用数据集（三）
环境：Vmware 8.0 和Ubuntu11.04Hadoop 实战之分析专利引用数据集（三）---从专利引用数据集中计算专利被引用的次数第一步：首先创建一个工程命名为HadoopTest.目录结构如下图:第二步: 在/home/tanglg1987目录下新建一个start.sh脚本文件，每次启动虚拟机都要删除/tmp目录下的全部文件，重新格式化namenode，代码如下：
环境：Vmware 8.0 和Ubuntu11.04
Hadoop 实战之分析专利引用数据集（三）---从专利引用数据集中计算专利被引用的次数
第一步：首先创建一个工程命名为HadoopTest.目录结构如下图:
sudo rm -rf /tmp/*
rm -rf /home/tanglg1987/hadoop-0.20.2/logs
hadoop namenode -format
hadoop datanode -format
start-all.sh
hadoop fs -mkdir input
hadoop dfsadmin -safemode leave第三步：给start.sh增加执行权限并启动hadoop伪分布式集群，代码如下:
chmod 777 /home/tanglg1987/ start.sh
./start.sh 执行过程如下：
第四步：上传本地文件到hdfs
在专利局http://data.nber.org/patents/网站下载专利数据
http://data.nber.org/patents/apat63_99.zip
hadoop fs -put /home/tanglg1987/cite75_99.txt input五步：新建一个MyJob3.java，代码如下：
package com.baison.action;
import java.io.IOException;
import java.util.Iterator;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.KeyValueTextInputFormat;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapred.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class MyJob3 extends Configured implements Tool {
public static class MapClass extends MapReduceBase
implements Mapper<Text, Text, Text, Text> {
public void map(Text key, Text value,
OutputCollector<Text, Text> output,
Reporter reporter) throws IOException {
output.collect(value, key);
}
}
public static class Reduce extends MapReduceBase
implements Reducer<Text, Text, Text, IntWritable> {
public void reduce(Text key, Iterator<Text> values,
OutputCollector<Text, IntWritable> output,
Reporter reporter) throws IOException {
String csv = "";
int count=0;
while (values.hasNext()) {
if (csv.length() > 0) csv += ",";
csv += values.next().toString();
count++;
}
output.collect(key, new IntWritable(count));
}
}
public int run(String[] args) throws Exception {
for (String string : args) {
System.out.println(string);
}
Configuration conf = getConf();
JobConf job = new JobConf(conf, MyJob3.class);
Path in = new Path(args[0]);
Path out = new Path(args[1]);
FileInputFormat.setInputPaths(job, in);
FileOutputFormat.setOutputPath(job, out);
job.setJobName("MyJob");
job.setMapperClass(MapClass.class);
job.setReducerClass(Reduce.class);
job.setInputFormat(KeyValueTextInputFormat.class);
job.setOutputFormat(TextOutputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
job.set("key.value.separator.in.input.line", ",");
JobClient.runJob(job);
return 0;
}
public static void main(String[] args) throws Exception {
String [] arg={"hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt","hdfs://localhost:9100/user/tanglg1987/output"};
int res = ToolRunner.run(new Configuration(), new MyJob3(), arg);
System.exit(res);
}
}第六步：Run On Hadoop，运行过程如下：
hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt
hdfs://localhost:9100/user/tanglg1987/output
12/10/18 22:24:57 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
12/10/18 22:24:57 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
12/10/18 22:24:58 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/18 22:24:58 INFO mapred.JobClient: Running job: job_local_0001
12/10/18 22:24:58 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/18 22:24:58 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 22:24:58 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 22:24:58 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 22:24:58 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 22:24:59 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:24:59 INFO mapred.MapTask: bufstart = 0; bufend = 4185926; bufvoid = 99614720
12/10/18 22:24:59 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/18 22:24:59 INFO mapred.JobClient: map 0% reduce 0%
12/10/18 22:25:00 INFO mapred.MapTask: Finished spill 0
12/10/18 22:25:01 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:01 INFO mapred.MapTask: bufstart = 4185926; bufend = 8372612; bufvoid = 99614720
12/10/18 22:25:01 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/18 22:25:01 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:25:02 INFO mapred.JobClient: map 14% reduce 0%
12/10/18 22:25:02 INFO mapred.MapTask: Finished spill 1
12/10/18 22:25:03 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:03 INFO mapred.MapTask: bufstart = 8372612; bufend = 12558934; bufvoid = 99614720
12/10/18 22:25:03 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
12/10/18 22:25:04 INFO mapred.MapTask: Finished spill 2
12/10/18 22:25:04 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:25:04 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:04 INFO mapred.MapTask: bufstart = 12558934; bufend = 16745318; bufvoid = 99614720
12/10/18 22:25:04 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
12/10/18 22:25:05 INFO mapred.JobClient: map 21% reduce 0%
12/10/18 22:25:05 INFO mapred.MapTask: Finished spill 3
12/10/18 22:25:05 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:05 INFO mapred.MapTask: bufstart = 16745318; bufend = 20931889; bufvoid = 99614720
12/10/18 22:25:05 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
12/10/18 22:25:07 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:25:08 INFO mapred.MapTask: Finished spill 4
12/10/18 22:25:08 INFO mapred.JobClient: map 32% reduce 0%
12/10/18 22:25:08 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:08 INFO mapred.MapTask: bufstart = 20931889; bufend = 25118734; bufvoid = 99614720
12/10/18 22:25:08 INFO mapred.MapTask: kvstart = 327677; kvend = 262140; length = 327680
12/10/18 22:25:10 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:25:11 INFO mapred.MapTask: Finished spill 5
12/10/18 22:25:11 INFO mapred.JobClient: map 38% reduce 0%
12/10/18 22:25:11 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:11 INFO mapred.MapTask: bufstart = 25118734; bufend = 29305269; bufvoid = 99614720
12/10/18 22:25:11 INFO mapred.MapTask: kvstart = 262140; kvend = 196603; length = 327680
12/10/18 22:25:13 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:25:14 INFO mapred.JobClient: map 45% reduce 0%
12/10/18 22:25:16 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:25:17 INFO mapred.MapTask: Finished spill 6
12/10/18 22:25:17 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:17 INFO mapred.MapTask: bufstart = 29305269; bufend = 33492764; bufvoid = 99614720
12/10/18 22:25:17 INFO mapred.MapTask: kvstart = 196603; kvend = 131066; length = 327680
12/10/18 22:25:19 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:25:19 INFO mapred.MapTask: Finished spill 7
12/10/18 22:25:20 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:20 INFO mapred.MapTask: bufstart = 33492764; bufend = 37680391; bufvoid = 99614720
12/10/18 22:25:20 INFO mapred.MapTask: kvstart = 131066; kvend = 65529; length = 327680
12/10/18 22:25:20 INFO mapred.JobClient: map 51% reduce 0%
12/10/18 22:25:21 INFO mapred.MapTask: Finished spill 8
12/10/18 22:25:22 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:22 INFO mapred.MapTask: bufstart = 37680391; bufend = 41868206; bufvoid = 99614720
12/10/18 22:25:22 INFO mapred.MapTask: kvstart = 65529; kvend = 327673; length = 327680
12/10/18 22:25:22 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:25:23 INFO mapred.JobClient: map 62% reduce 0%
12/10/18 22:25:24 INFO mapred.MapTask: Finished spill 9
12/10/18 22:25:24 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:24 INFO mapred.MapTask: bufstart = 41868206; bufend = 46056257; bufvoid = 99614720
12/10/18 22:25:24 INFO mapred.MapTask: kvstart = 327673; kvend = 262136; length = 327680
12/10/18 22:25:25 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:25:25 INFO mapred.MapTask: Finished spill 10
12/10/18 22:25:26 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:26 INFO mapred.MapTask: bufstart = 46056257; bufend = 50244288; bufvoid = 99614720
12/10/18 22:25:26 INFO mapred.MapTask: kvstart = 262136; kvend = 196599; length = 327680
12/10/18 22:25:26 INFO mapred.JobClient: map 70% reduce 0%
12/10/18 22:25:27 INFO mapred.MapTask: Finished spill 11
12/10/18 22:25:27 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:27 INFO mapred.MapTask: bufstart = 50244288; bufend = 54432271; bufvoid = 99614720
12/10/18 22:25:27 INFO mapred.MapTask: kvstart = 196599; kvend = 131062; length = 327680
12/10/18 22:25:28 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:25:29 INFO mapred.MapTask: Finished spill 12
12/10/18 22:25:29 INFO mapred.JobClient: map 82% reduce 0%
12/10/18 22:25:30 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:30 INFO mapred.MapTask: bufstart = 54432271; bufend = 58620635; bufvoid = 99614720
12/10/18 22:25:30 INFO mapred.MapTask: kvstart = 131062; kvend = 65525; length = 327680
12/10/18 22:25:31 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:25:32 INFO mapred.MapTask: Finished spill 13
12/10/18 22:25:32 INFO mapred.JobClient: map 88% reduce 0%
12/10/18 22:25:32 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:32 INFO mapred.MapTask: bufstart = 58620635; bufend = 62808941; bufvoid = 99614720
12/10/18 22:25:32 INFO mapred.MapTask: kvstart = 65525; kvend = 327669; length = 327680
12/10/18 22:25:34 INFO mapred.MapTask: Finished spill 14
12/10/18 22:25:34 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:34 INFO mapred.MapTask: bufstart = 62808941; bufend = 66997060; bufvoid = 99614720
12/10/18 22:25:34 INFO mapred.MapTask: kvstart = 327669; kvend = 262132; length = 327680
12/10/18 22:25:34 INFO mapred.MapTask: Starting flush of map output
12/10/18 22:25:34 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:25:35 INFO mapred.MapTask: Finished spill 15
12/10/18 22:25:35 INFO mapred.MapTask: Finished spill 16
12/10/18 22:25:35 INFO mapred.JobClient: map 100% reduce 0%
12/10/18 22:25:35 INFO mapred.Merger: Merging 17 sorted segments
12/10/18 22:25:35 INFO mapred.Merger: Merging 8 intermediate segments out of a total of 17
12/10/18 22:25:37 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:25:38 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 75511476 bytes
12/10/18 22:25:40 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:25:43 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:25:45 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
12/10/18 22:25:45 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:25:45 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.
12/10/18 22:25:45 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 22:25:45 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 22:25:45 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 22:25:45 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 22:25:45 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:45 INFO mapred.MapTask: bufstart = 0; bufend = 4188247; bufvoid = 99614720
12/10/18 22:25:45 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/18 22:25:46 INFO mapred.MapTask: Finished spill 0
12/10/18 22:25:46 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:46 INFO mapred.MapTask: bufstart = 4188247; bufend = 8376832; bufvoid = 99614720
12/10/18 22:25:46 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/18 22:25:47 INFO mapred.MapTask: Finished spill 1
12/10/18 22:25:48 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:48 INFO mapred.MapTask: bufstart = 8376832; bufend = 12565282; bufvoid = 99614720
12/10/18 22:25:48 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
12/10/18 22:25:48 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:25:48 INFO mapred.JobClient: map 59% reduce 0%
12/10/18 22:25:49 INFO mapred.MapTask: Finished spill 2
12/10/18 22:25:49 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:49 INFO mapred.MapTask: bufstart = 12565282; bufend = 16754414; bufvoid = 99614720
12/10/18 22:25:49 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
12/10/18 22:25:50 INFO mapred.MapTask: Finished spill 3
12/10/18 22:25:50 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:50 INFO mapred.MapTask: bufstart = 16754414; bufend = 20943273; bufvoid = 99614720
12/10/18 22:25:50 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
12/10/18 22:25:51 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:25:51 INFO mapred.MapTask: Finished spill 4
12/10/18 22:25:51 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:51 INFO mapred.MapTask: bufstart = 20943273; bufend = 25132396; bufvoid = 99614720
12/10/18 22:25:51 INFO mapred.MapTask: kvstart = 327677; kvend = 262140; length = 327680
12/10/18 22:25:51 INFO mapred.JobClient: map 66% reduce 0%
12/10/18 22:25:52 INFO mapred.MapTask: Finished spill 5
12/10/18 22:25:52 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:52 INFO mapred.MapTask: bufstart = 25132396; bufend = 29321431; bufvoid = 99614720
12/10/18 22:25:52 INFO mapred.MapTask: kvstart = 262140; kvend = 196603; length = 327680
12/10/18 22:25:53 INFO mapred.MapTask: Finished spill 6
12/10/18 22:25:53 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:53 INFO mapred.MapTask: bufstart = 29321431; bufend = 33510738; bufvoid = 99614720
12/10/18 22:25:53 INFO mapred.MapTask: kvstart = 196603; kvend = 131066; length = 327680
12/10/18 22:25:54 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:25:54 INFO mapred.MapTask: Finished spill 7
12/10/18 22:25:54 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:54 INFO mapred.MapTask: bufstart = 33510738; bufend = 37700146; bufvoid = 99614720
12/10/18 22:25:54 INFO mapred.MapTask: kvstart = 131066; kvend = 65529; length = 327680
12/10/18 22:25:54 INFO mapred.JobClient: map 75% reduce 0%
12/10/18 22:25:55 INFO mapred.MapTask: Finished spill 8
12/10/18 22:25:55 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:55 INFO mapred.MapTask: bufstart = 37700146; bufend = 41889795; bufvoid = 99614720
12/10/18 22:25:55 INFO mapred.MapTask: kvstart = 65529; kvend = 327673; length = 327680
12/10/18 22:25:56 INFO mapred.MapTask: Finished spill 9
12/10/18 22:25:56 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:56 INFO mapred.MapTask: bufstart = 41889795; bufend = 46079341; bufvoid = 99614720
12/10/18 22:25:56 INFO mapred.MapTask: kvstart = 327673; kvend = 262136; length = 327680
12/10/18 22:25:57 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:25:57 INFO mapred.MapTask: Finished spill 10
12/10/18 22:25:57 INFO mapred.JobClient: map 85% reduce 0%
12/10/18 22:25:57 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:57 INFO mapred.MapTask: bufstart = 46079341; bufend = 50269061; bufvoid = 99614720
12/10/18 22:25:57 INFO mapred.MapTask: kvstart = 262136; kvend = 196599; length = 327680
12/10/18 22:25:58 INFO mapred.MapTask: Finished spill 11
12/10/18 22:25:58 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:58 INFO mapred.MapTask: bufstart = 50269061; bufend = 54458860; bufvoid = 99614720
12/10/18 22:25:58 INFO mapred.MapTask: kvstart = 196599; kvend = 131062; length = 327680
12/10/18 22:25:59 INFO mapred.MapTask: Finished spill 12
12/10/18 22:25:59 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:25:59 INFO mapred.MapTask: bufstart = 54458860; bufend = 58648475; bufvoid = 99614720
12/10/18 22:25:59 INFO mapred.MapTask: kvstart = 131062; kvend = 65525; length = 327680
12/10/18 22:26:00 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:26:00 INFO mapred.JobClient: map 94% reduce 0%
12/10/18 22:26:00 INFO mapred.MapTask: Finished spill 13
12/10/18 22:26:00 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:00 INFO mapred.MapTask: bufstart = 58648475; bufend = 62838648; bufvoid = 99614720
12/10/18 22:26:00 INFO mapred.MapTask: kvstart = 65525; kvend = 327669; length = 327680
12/10/18 22:26:01 INFO mapred.MapTask: Finished spill 14
12/10/18 22:26:01 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:01 INFO mapred.MapTask: bufstart = 62838648; bufend = 67028992; bufvoid = 99614720
12/10/18 22:26:01 INFO mapred.MapTask: kvstart = 327669; kvend = 262132; length = 327680
12/10/18 22:26:02 INFO mapred.MapTask: Starting flush of map output
12/10/18 22:26:02 INFO mapred.MapTask: Finished spill 15
12/10/18 22:26:02 INFO mapred.MapTask: Finished spill 16
12/10/18 22:26:02 INFO mapred.Merger: Merging 17 sorted segments
12/10/18 22:26:02 INFO mapred.Merger: Merging 8 intermediate segments out of a total of 17
12/10/18 22:26:03 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:26:03 INFO mapred.JobClient: map 100% reduce 0%
12/10/18 22:26:05 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 75507463 bytes
12/10/18 22:26:06 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:26:09 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:26:12 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:26:13 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting
12/10/18 22:26:13 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:26:13 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000001_0' done.
12/10/18 22:26:13 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 22:26:13 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 22:26:13 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 22:26:13 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 22:26:13 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:13 INFO mapred.MapTask: bufstart = 0; bufend = 4190341; bufvoid = 99614720
12/10/18 22:26:13 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/18 22:26:15 INFO mapred.MapTask: Finished spill 0
12/10/18 22:26:15 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:15 INFO mapred.MapTask: bufstart = 4190341; bufend = 8380745; bufvoid = 99614720
12/10/18 22:26:15 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/18 22:26:16 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:26:16 INFO mapred.JobClient: map 71% reduce 0%
12/10/18 22:26:17 INFO mapred.MapTask: Finished spill 1
12/10/18 22:26:17 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:17 INFO mapred.MapTask: bufstart = 8380745; bufend = 12571307; bufvoid = 99614720
12/10/18 22:26:17 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
12/10/18 22:26:18 INFO mapred.MapTask: Finished spill 2
12/10/18 22:26:19 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:19 INFO mapred.MapTask: bufstart = 12571307; bufend = 16762033; bufvoid = 99614720
12/10/18 22:26:19 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
12/10/18 22:26:19 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:26:19 INFO mapred.JobClient: map 75% reduce 0%
12/10/18 22:26:20 INFO mapred.MapTask: Finished spill 3
12/10/18 22:26:21 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:21 INFO mapred.MapTask: bufstart = 16762033; bufend = 20952800; bufvoid = 99614720
12/10/18 22:26:21 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
12/10/18 22:26:22 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:26:22 INFO mapred.JobClient: map 77% reduce 0%
12/10/18 22:26:23 INFO mapred.MapTask: Finished spill 4
12/10/18 22:26:23 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:23 INFO mapred.MapTask: bufstart = 20952800; bufend = 25143412; bufvoid = 99614720
12/10/18 22:26:23 INFO mapred.MapTask: kvstart = 327677; kvend = 262140; length = 327680
12/10/18 22:26:25 INFO mapred.MapTask: Finished spill 5
12/10/18 22:26:25 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:26:25 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:25 INFO mapred.MapTask: bufstart = 25143412; bufend = 29334124; bufvoid = 99614720
12/10/18 22:26:25 INFO mapred.MapTask: kvstart = 262140; kvend = 196603; length = 327680
12/10/18 22:26:25 INFO mapred.JobClient: map 80% reduce 0%
12/10/18 22:26:26 INFO mapred.MapTask: Finished spill 6
12/10/18 22:26:26 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:26 INFO mapred.MapTask: bufstart = 29334124; bufend = 33524778; bufvoid = 99614720
12/10/18 22:26:26 INFO mapred.MapTask: kvstart = 196603; kvend = 131066; length = 327680
12/10/18 22:26:27 INFO mapred.MapTask: Finished spill 7
12/10/18 22:26:27 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:27 INFO mapred.MapTask: bufstart = 33524778; bufend = 37715716; bufvoid = 99614720
12/10/18 22:26:27 INFO mapred.MapTask: kvstart = 131066; kvend = 65529; length = 327680
12/10/18 22:26:28 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:26:28 INFO mapred.JobClient: map 85% reduce 0%
12/10/18 22:26:28 INFO mapred.MapTask: Finished spill 8
12/10/18 22:26:28 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:28 INFO mapred.MapTask: bufstart = 37715716; bufend = 41906544; bufvoid = 99614720
12/10/18 22:26:28 INFO mapred.MapTask: kvstart = 65529; kvend = 327673; length = 327680
12/10/18 22:26:29 INFO mapred.MapTask: Finished spill 9
12/10/18 22:26:29 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:29 INFO mapred.MapTask: bufstart = 41906544; bufend = 46097751; bufvoid = 99614720
12/10/18 22:26:29 INFO mapred.MapTask: kvstart = 327673; kvend = 262136; length = 327680
12/10/18 22:26:31 INFO mapred.MapTask: Finished spill 10
12/10/18 22:26:31 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:31 INFO mapred.MapTask: bufstart = 46097751; bufend = 50288897; bufvoid = 99614720
12/10/18 22:26:31 INFO mapred.MapTask: kvstart = 262136; kvend = 196599; length = 327680
12/10/18 22:26:31 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:26:31 INFO mapred.JobClient: map 92% reduce 0%
12/10/18 22:26:32 INFO mapred.MapTask: Finished spill 11
12/10/18 22:26:32 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:32 INFO mapred.MapTask: bufstart = 50288897; bufend = 54480358; bufvoid = 99614720
12/10/18 22:26:32 INFO mapred.MapTask: kvstart = 196599; kvend = 131062; length = 327680
12/10/18 22:26:33 INFO mapred.MapTask: Finished spill 12
12/10/18 22:26:33 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:33 INFO mapred.MapTask: bufstart = 54480358; bufend = 58671642; bufvoid = 99614720
12/10/18 22:26:33 INFO mapred.MapTask: kvstart = 131062; kvend = 65525; length = 327680
12/10/18 22:26:34 INFO mapred.MapTask: Finished spill 13
12/10/18 22:26:34 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:34 INFO mapred.MapTask: bufstart = 58671642; bufend = 62863227; bufvoid = 99614720
12/10/18 22:26:34 INFO mapred.MapTask: kvstart = 65525; kvend = 327669; length = 327680
12/10/18 22:26:34 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:26:34 INFO mapred.JobClient: map 98% reduce 0%
12/10/18 22:26:35 INFO mapred.MapTask: Finished spill 14
12/10/18 22:26:35 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:35 INFO mapred.MapTask: bufstart = 62863227; bufend = 67054803; bufvoid = 99614720
12/10/18 22:26:35 INFO mapred.MapTask: kvstart = 327669; kvend = 262132; length = 327680
12/10/18 22:26:35 INFO mapred.MapTask: Starting flush of map output
12/10/18 22:26:36 INFO mapred.MapTask: Finished spill 15
12/10/18 22:26:36 INFO mapred.MapTask: Finished spill 16
12/10/18 22:26:36 INFO mapred.Merger: Merging 17 sorted segments
12/10/18 22:26:36 INFO mapred.Merger: Merging 8 intermediate segments out of a total of 17
12/10/18 22:26:37 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:26:37 INFO mapred.JobClient: map 100% reduce 0%
12/10/18 22:26:39 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 75504227 bytes
12/10/18 22:26:40 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:26:43 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:26:45 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000002_0 is done. And is in the process of commiting
12/10/18 22:26:45 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:26:45 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000002_0' done.
12/10/18 22:26:45 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 22:26:45 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 22:26:45 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 22:26:45 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 22:26:45 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:45 INFO mapred.MapTask: bufstart = 0; bufend = 4191578; bufvoid = 99614720
12/10/18 22:26:45 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/18 22:26:46 INFO mapred.MapTask: Finished spill 0
12/10/18 22:26:47 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:47 INFO mapred.MapTask: bufstart = 4191578; bufend = 8383078; bufvoid = 99614720
12/10/18 22:26:47 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/18 22:26:47 INFO mapred.MapTask: Finished spill 1
12/10/18 22:26:48 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:48 INFO mapred.MapTask: bufstart = 8383078; bufend = 12574490; bufvoid = 99614720
12/10/18 22:26:48 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
12/10/18 22:26:48 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:26:48 INFO mapred.JobClient: map 80% reduce 0%
12/10/18 22:26:48 INFO mapred.MapTask: Finished spill 2
12/10/18 22:26:49 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:49 INFO mapred.MapTask: bufstart = 12574490; bufend = 16766127; bufvoid = 99614720
12/10/18 22:26:49 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
12/10/18 22:26:49 INFO mapred.MapTask: Finished spill 3
12/10/18 22:26:50 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:50 INFO mapred.MapTask: bufstart = 16766127; bufend = 20958217; bufvoid = 99614720
12/10/18 22:26:50 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
12/10/18 22:26:50 INFO mapred.MapTask: Finished spill 4
12/10/18 22:26:51 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:51 INFO mapred.MapTask: bufstart = 20958217; bufend = 25150065; bufvoid = 99614720
12/10/18 22:26:51 INFO mapred.MapTask: kvstart = 327677; kvend = 262140; length = 327680
12/10/18 22:26:51 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:26:51 INFO mapred.JobClient: map 85% reduce 0%
12/10/18 22:26:51 INFO mapred.MapTask: Finished spill 5
12/10/18 22:26:51 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:51 INFO mapred.MapTask: bufstart = 25150065; bufend = 29341612; bufvoid = 99614720
12/10/18 22:26:51 INFO mapred.MapTask: kvstart = 262140; kvend = 196603; length = 327680
12/10/18 22:26:52 INFO mapred.MapTask: Finished spill 6
12/10/18 22:26:53 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:53 INFO mapred.MapTask: bufstart = 29341612; bufend = 33533322; bufvoid = 99614720
12/10/18 22:26:53 INFO mapred.MapTask: kvstart = 196603; kvend = 131066; length = 327680
12/10/18 22:26:54 INFO mapred.MapTask: Finished spill 7
12/10/18 22:26:54 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:26:54 INFO mapred.JobClient: map 89% reduce 0%
12/10/18 22:26:54 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:54 INFO mapred.MapTask: bufstart = 33533322; bufend = 37725421; bufvoid = 99614720
12/10/18 22:26:54 INFO mapred.MapTask: kvstart = 131066; kvend = 65529; length = 327680
12/10/18 22:26:56 INFO mapred.MapTask: Finished spill 8
12/10/18 22:26:56 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:56 INFO mapred.MapTask: bufstart = 37725421; bufend = 41917251; bufvoid = 99614720
12/10/18 22:26:56 INFO mapred.MapTask: kvstart = 65529; kvend = 327673; length = 327680
12/10/18 22:26:57 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:26:57 INFO mapred.JobClient: map 92% reduce 0%
12/10/18 22:26:57 INFO mapred.MapTask: Finished spill 9
12/10/18 22:26:58 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:26:58 INFO mapred.MapTask: bufstart = 41917251; bufend = 46108919; bufvoid = 99614720
12/10/18 22:26:58 INFO mapred.MapTask: kvstart = 327673; kvend = 262136; length = 327680
12/10/18 22:26:59 INFO mapred.MapTask: Finished spill 10
12/10/18 22:27:00 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:27:00 INFO mapred.MapTask: bufstart = 46108919; bufend = 50300567; bufvoid = 99614720
12/10/18 22:27:00 INFO mapred.MapTask: kvstart = 262136; kvend = 196599; length = 327680
12/10/18 22:27:00 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:27:00 INFO mapred.JobClient: map 95% reduce 0%
12/10/18 22:27:01 INFO mapred.MapTask: Finished spill 11
12/10/18 22:27:01 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:27:01 INFO mapred.MapTask: bufstart = 50300567; bufend = 54492159; bufvoid = 99614720
12/10/18 22:27:01 INFO mapred.MapTask: kvstart = 196599; kvend = 131062; length = 327680
12/10/18 22:27:02 INFO mapred.MapTask: Finished spill 12
12/10/18 22:27:03 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:27:03 INFO mapred.MapTask: bufstart = 54492159; bufend = 58684086; bufvoid = 99614720
12/10/18 22:27:03 INFO mapred.MapTask: kvstart = 131062; kvend = 65525; length = 327680
12/10/18 22:27:03 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:27:03 INFO mapred.JobClient: map 98% reduce 0%
12/10/18 22:27:06 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:27:07 INFO mapred.MapTask: Finished spill 13
12/10/18 22:27:08 INFO mapred.MapTask: Starting flush of map output
12/10/18 22:27:09 INFO mapred.MapTask: Finished spill 14
12/10/18 22:27:09 INFO mapred.Merger: Merging 15 sorted segments
12/10/18 22:27:09 INFO mapred.Merger: Merging 6 intermediate segments out of a total of 15
12/10/18 22:27:09 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:27:09 INFO mapred.JobClient: map 100% reduce 0%
12/10/18 22:27:12 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 70597223 bytes
12/10/18 22:27:12 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:27:15 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:27:17 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000003_0 is done. And is in the process of commiting
12/10/18 22:27:17 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:27:17 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000003_0' done.
12/10/18 22:27:18 INFO mapred.LocalJobRunner:
12/10/18 22:27:18 INFO mapred.Merger: Merging 4 sorted segments
12/10/18 22:27:18 INFO mapred.Merger: Down to the last merge-pass, with 4 segments left of total size: 297120317 bytes
12/10/18 22:27:18 INFO mapred.LocalJobRunner:
12/10/18 22:27:24 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:27:24 INFO mapred.JobClient: map 100% reduce 73%
12/10/18 22:27:27 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:27:27 INFO mapred.JobClient: map 100% reduce 77%
12/10/18 22:27:30 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:27:30 INFO mapred.JobClient: map 100% reduce 81%
12/10/18 22:27:33 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:27:33 INFO mapred.JobClient: map 100% reduce 85%
12/10/18 22:27:36 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:27:36 INFO mapred.JobClient: map 100% reduce 89%
12/10/18 22:27:39 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:27:39 INFO mapred.JobClient: map 100% reduce 93%
12/10/18 22:27:42 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:27:42 INFO mapred.JobClient: map 100% reduce 96%
12/10/18 22:27:44 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
12/10/18 22:27:44 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:27:44 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now
12/10/18 22:27:44 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to hdfs://localhost:9100/user/tanglg1987/output
12/10/18 22:27:44 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:27:44 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done.
12/10/18 22:27:44 INFO mapred.JobClient: map 100% reduce 100%
12/10/18 22:27:44 INFO mapred.JobClient: Job complete: job_local_0001
12/10/18 22:27:44 INFO mapred.JobClient: Counters: 15
12/10/18 22:27:44 INFO mapred.JobClient: FileSystemCounters
12/10/18 22:27:44 INFO mapred.JobClient: FILE_BYTES_READ=1853504380
12/10/18 22:27:44 INFO mapred.JobClient: HDFS_BYTES_READ=930853207
12/10/18 22:27:44 INFO mapred.JobClient: FILE_BYTES_WRITTEN=2603767507
12/10/18 22:27:44 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=32858613
12/10/18 22:27:44 INFO mapred.JobClient: Map-Reduce Framework
12/10/18 22:27:44 INFO mapred.JobClient: Reduce input groups=3258984
12/10/18 22:27:44 INFO mapred.JobClient: Combine output records=0
12/10/18 22:27:44 INFO mapred.JobClient: Map input records=16522439
12/10/18 22:27:44 INFO mapred.JobClient: Reduce shuffle bytes=0
12/10/18 22:27:44 INFO mapred.JobClient: Reduce output records=3258984
12/10/18 22:27:44 INFO mapred.JobClient: Spilled Records=57431615
12/10/18 22:27:44 INFO mapred.JobClient: Map output bytes=264075431
12/10/18 22:27:44 INFO mapred.JobClient: Map input bytes=264075431
12/10/18 22:27:44 INFO mapred.JobClient: Combine input records=0
12/10/18 22:27:44 INFO mapred.JobClient: Map output records=16522439
12/10/18 22:27:44 INFO mapred.JobClient: Reduce input records=16522439
第七步：查看结果集，运行结果如下：
华为开发者空间，是为全球开发者打造的专属开发空间，汇聚了华为优质开发资源及工具，致力于让每一位开发者拥有一台云主机，基于华为根生态开发、创新。
更多推荐
0
0- 0
已为社区贡献5条内容
所有评论(0)