Hadoop添加snappy压缩支持

启动hadoop后，使用如下命令查看压缩支持：

hadoop checknative

出现下列输出，snappy为false，则需要手动添加snappy支持

Native library checking:
hadoop:  true /opt/bigdata/hadoop/hadoop-3.1.4/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
snappy:  false 
lz4:     true revision:10301
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   false libhadoop was built without ISA-L support
PMDK:    false The native code was built without PMDK support.

snappy-1.1.3.tar.gz下载地址：
https://src.fedoraproject.org/lookaside/pkgs/snappy/snappy-1.1.3.tar.gz/7358c82f133dc77798e4c2062a749b73/
下载snappy-1.1.3.tar.gz包，然后执行以下命令

tar -xvf snappy-1.1.3.tar.gz
cd snappy-1.1.3
./configure
make
make install

默认目录是 /usr/local/lib，查看snappy是否安装成功

ll /usr/local/lib | grep snappy

出现下面5个文件，表示安装成功

libsnappy.a
libsnappy.la*
libsnappy.so -> libsnappy.so.1.3.0*
libsnappy.so.1 -> libsnappy.so.1.3.0*
libsnappy.so.1.3.0*

将上述5个文件复制到${HADOOP_HOME}/lib/native，${HADOOP_HOME}为你hadoop的安装目录
修改hadoop下的/etc/hadoop/core-site.xml文件，添加以下属性，如果是集群，记得分发

<property><name>io.compression.codecsname><value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodecvalue>
property>

修改hadoop下的/etc/hadoop/mapred-site.xml文件，添加以下属性，如果是集群，记得分发

<property><name>mapred.output.compressname><value>truevalue>
property>
<property><name>mapred.output.compression.codecname><value>org.apache.hadoop.io.compress.SnappyCodecvalue>
property>
<property><name>mapred.compress.map.outputname><value>truevalue>
property>
<property><name>mapred.map.output.compression.codecname><value>org.apache.hadoop.io.compress.SnappyCodecvalue>
property>

重启hadoop集群，再次执行 hadoop checknative，snappy那一栏不是false则表示成功。

例如：

Native library checking:
hadoop:  true /opt/bigdata/hadoop/hadoop-3.1.4/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
snappy:  true /opt/bigdata/hadoop/hadoop-3.1.4/lib/native/libsnappy.so.1
lz4:     true revision:10301
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   false libhadoop was built without ISA-L support
PMDK:    false The native code was built without PMDK support.

本文来自互联网用户投稿，文章观点仅代表作者本人，不代表本站立场，不承担相关法律责任。如若转载，请注明出处。 如若内容造成侵权/违法违规/事实不符，请点击【内容举报】进行投诉反馈！