写操作
根据上一篇的介绍,在hadoop2.x之后,hadoop中的SequenceFile.Writer将会逐渐摒弃大量的createWriter()重载方法,而整合为更为简洁的createWriter()
方法,除了配置参数外,其他的参数统统使用SequenceFile.Writer.Option来替代,具体有:
新的API里提供的option参数:
FileOption
FileSystemOption
StreamOption
BufferSizeOption
BlockSizeOption
ReplicationOption
KeyClassOption
ValueClassOption
MetadataOption
ProgressableOption
CompressionOption
这些参数能够满足各种不同的需要,参数之间不存在顺序关系,这样减少了代码编写工作量,更为直观,便于理解,下面先来看看这个方法,后边将给出一个具体实例。
-
createWriter
public staticorg.apache.hadoop.io.SequenceFile.WritercreateWriter(Configurationconf,
org.apache.hadoop.io.SequenceFile.Writer.Option...opts)
throws IOException
Create a new Writer with the given options.
- Parameters:
-
conf
- the configuration to use
-
opts
- the options to create the file with
- Returns:
- a new Writer
- Throws:
IOException
权威指南第四版中提供了一个SequenceFileWriteDemo实例:
// cc SequenceFileWriteDemo Writing a SequenceFile
import java.io.IOException;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.SequenceFile;
import org.apache.hadoop.io.Text;
// vv SequenceFileWriteDemo
public class SequenceFileWriteDemo {
private static final String[] DATA = {
"One, two, buckle my shoe",
"Three, four, shut the door",
"Five, six, pick up sticks",
"Seven, eight, lay them straight",
"Nine, ten, a big fat hen"
};
public static void main(String[] args) throws IOException {
String uri = args[0];
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(uri), conf);
Path path = new Path(uri);
IntWritable key = new IntWritable();
Text value = new Text();
SequenceFile.Writer writer = null;
try {
writer = SequenceFile.createWriter(fs, conf, path,
key.getClass(), value.getClass());
for (int i = 0; i < 100; i++) {
key.set(100 - i);
value.set(DATA[i % DATA.length]);
System.out.printf("[%s]\t%s\t%s\n", writer.getLength(), key, value);
writer.append(key, value);
}
} finally {
IOUtils.closeStream(writer);
}
}
}
// ^^ SequenceFileWriteDemo
对于上面实例中的createWriter()
方法用整合之后的最新的方法来改写一下,代码如下:
package org.apache.hadoop.io;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.SequenceFile;
import org.apache.hadoop.io.SequenceFile.Writer;
import org.apache.hadoop.io.SequenceFile.Writer.FileOption;
import org.apache.hadoop.io.SequenceFile.Writer.KeyClassOption;
import org.apache.hadoop.io.SequenceFile.Writer.ValueClassOption;
import org.apache.hadoop.io.Text;
public class THT_testSequenceFile2 {
private static final String[] DATA = { "One, two, buckle my shoe",
"Three, four, shut the door", "Five, six, pick up sticks",
"Seven, eight, lay them straight", "Nine, ten, a big fat hen" };
public static void main(String[] args) throws IOException {
// String uri = args[0];
String uri = "file:///D://B.txt";
Configuration conf = new Configuration();
Path path = new Path(uri);
IntWritable key = new IntWritable();
Text value = new Text();
SequenceFile.Writer writer = null;
SequenceFile.Writer.FileOption option1 = (FileOption) Writer.file(path);
SequenceFile.Writer.KeyClassOption option2 = (KeyClassOption) Writer.keyClass(key.getClass());
SequenceFile.Writer.ValueClassOption option3 = (ValueClassOption) Writer.valueClass(value.getClass());
try {
writer = SequenceFile.createWriter( conf, option1,option2,option3,Writer.compression(CompressionType.RECORD));
for (int i = 0; i < 10; i++) {
key.set(1 + i);
value.set(DATA[i % DATA.length]);
System.out.printf("[%s]\t%s\t%s\n", writer.getLength(), key,
value);
writer.append(key, value);
}
} finally {
IOUtils.closeStream(writer);
}
}
}
运行结果如下:
2015-11-06 22:15:05,027 INFO compress.CodecPool (CodecPool.java:getCompressor(153)) - Got brand-new compressor [.deflate]
[128] 1 One, two, buckle my shoe
[173] 2 Three, four, shut the door
[220] 3 Five, six, pick up sticks
[264] 4 Seven, eight, lay them straight
[314] 5 Nine, ten, a big fat hen
[359] 6 One, two, buckle my shoe
[404] 7 Three, four, shut the door
[451] 8 Five, six, pick up sticks
[495] 9 Seven, eight, lay them straight
[545] 10 Nine, ten, a big fat hen
生成的文件:
读操作
新的API里提供的option参数:
FileOption -表示读哪个文件
InputStreamOption
StartOption
LengthOption -按照设置的长度变量来决定读取的字节
BufferSizeOption
OnlyHeaderOption
根据最新的API直接上源码:
package org.apache.hadoop.io;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.io.SequenceFile;
import org.apache.hadoop.io.SequenceFile.Reader;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.util.ReflectionUtils;
public class THT_testSequenceFile3 {
public static void main(String[] args) throws IOException {
//String uri = args[0];
String uri = "file:///D://B.txt";
Configuration conf = new Configuration();
Path path = new Path(uri);
SequenceFile.Reader.Option option1 = Reader.file(path);
SequenceFile.Reader.Option option2 = Reader.length(174);//这个参数表示读取的长度
SequenceFile.Reader reader = null;
try {
reader = new SequenceFile.Reader(conf,option1,option2);
Writable key = (Writable) ReflectionUtils.newInstance(
reader.getKeyClass(), conf);
Writable value = (Writable) ReflectionUtils.newInstance(
reader.getValueClass(), conf);
long position = reader.getPosition();
while (reader.next(key, value)) {
String syncSeen = reader.syncSeen() ? "*" : "";
System.out.printf("[%s%s]\t%s\t%s\n", position, syncSeen, key,
value);
position = reader.getPosition(); // beginning of next record
}
} finally {
IOUtils.closeStream(reader);
}
}
}
我这儿设置了一个读取长度的参数,只读到第174个字节那,所以运行结果如下:
2015-11-06 22:53:00,602 INFO compress.CodecPool (CodecPool.java:getDecompressor(181)) - Got brand-new decompressor [.deflate]
[128] 1 One, two, buckle my shoe
[173] 2 Three, four, shut the door
分享到:
相关推荐
Hadoop安装教程_单机/伪分布式配置_Hadoop2.7.1/Ubuntu 16.04
hadoop2.7.1的eclipse插件,编译环境,eclipse 4.4(luna) ,jdk1.7,ant1.9.6,maven3.3,hadoop2.7.1,centos6.7,jdk1.7 要注意的是开发黄金下jdk版本必须是jdk1.7及以上,否则无法使用
eclipse hadoop2.7.1 plugin 配置,包括操作步骤说明,eclipse hadoop2.7.1的插件,还有hadoop.dll和winutils.exe等文件。
Hadoop2.7.1中文文档
hadoop2.7.1的linux下eclipse支持插件。已经经过检查,安装后在eclipse下,可以正常上传,下载,删除hdfs文件,以及跑mapreduce程序。 编译环境:ant1.9.6 jdk1.8 hadoop2.7.1
hadoop2.7.1平台搭建
Spark所需的hadoop2.7.1相关资源 hadoop2.7.1版本的hadoop.dll,winutils.exe 适用Spark2.0.0+版本
hadoop2.7.1版本的hadoop.dll,winutils.exe hadoop2.7.1版本的hadoop.dll,winutils.exe
本人用7个多小时成功编译 hadoop 2.7.1 64位编译包(JDK1.8 64),由于文件太大,分3卷压缩。 hadoop 2.7.1 相对于2.7.0修复了上百个Bug,是可用于生产环境的版本了。
在网上下了好多2.6版本的hadoop.dll,但是都不好使,昨天有个好心网友给我发了一份,实际测试通过。开发环境是64位win7+hadoop2.7.1+redhat版本的linux。
hadoop2.7.1 windows 64位 相关 dll文件
Hadoop-2.7.1-Windows-64-binaries, 预先编译,非官方的Hadoop 2.7.1的Win64二进制文件 用于 Windows 64位 平台的 2.7.1二进制文件这是针对 Windows 64位 平台的非官方预先编译的Apache Hadoop 2.7.1. 这里的tar.gz ...
Windows 7 or 10 eclipse hadoop2.7.1 配置需要文件和工具
Hadoop 2.7.1 中文文档 Hadoop 2.7.1 中文文档 Hadoop 2.7.1 中文文档
hadoop.dll winutils.exe for hadoop2.7.1 下载版本,
hadoop2.7.1运行Wordcount错误 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012) Exception in thread "main" java.lang.UnsatisfiedLinkError
win32平台winutils,hadoop win32 native code
hadoop2.7.1 windows7 32 位 hadoop.dll winutils.exe.在window7 32位下,编译的源码。
hadoop安装包 包括 win访问hadoop 文件系统所需api
hadoop-2.7.1-64位编译包,本人亲测,绝对有用。