Return to site

Hive Error: buildSupportsSnappy()Z

与CDH Cluster中的parcels

[fix hive issues]

· Hive

Error: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z


报错的map , reduce 在log中均显示形如

java.lang.Exception: java.lang.UnsatisfiedLinkError:

Error: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z

的error信息, 而且连续三天出问题的机器均在210与195身上, 通过回顾近期的集群Datanode操作, 这两台机器均进行了reload操作。 因此怀疑是在上线过程中有些环境变量被修改导致MapReduce在执行过程中无法access native libraries。

  1. Ensure that LD_LIBRARY_PATH and JAVA_LIBRARY_PATH contains the native directory path having the** files.
  2. Ensure that LD_LIBRARY_PATH and JAVA_LIBRARY path have been exported in the SPARK environment(



mapred-site.xml 查找要对应的 key value配置。





可见这里某值是在map reduce tasks有必要进行获取native libraries 权限, 以进行例如数据压缩。 此值若为空,则用系统环境变量替换。

通过查找, 发现是后者 第二个环境变量决定了

{ mapreduce.admin.user.env}


Expert: Additional execution environment entries for map and reduce task processes. This is not an additive property. You must preserve the original value if you want your map and reduce tasks to have access to native libraries (compression, etc). When this value is empty, the command to set execution envrionment will be OS dependent: For linux, use LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native. For windows, use PATH = %PATH%;%HADOOP_COMMON_HOME%\\bin.​

备注: 我的同事在第一次修改 后, 问题没有解决。


CDH Cluster上的配置如何生效的?


如: /var/run/cloudera-scm-agent/process/193-hdfs-NAMENODE/core-site.xml

这些配置文件是通过Cloudera Manager启动相应服务(如HDFS)时生成的,例如当我们修改了Cloudera Manage配置界面的值后, 重启后就会生成一份新的配置xml 替换原有的。


【core-site.xml 的 HDFS 复制高级配置代码段(安全阀)】

- fs.oss.core.dependency.path




Environment variables for the MR App Master processes for admin purposes. These values are set first and can be overridden by the user env ( Example : 1) A=foo This will set the env variable A to foo 2) B=$B:c This is inherit app master's B env variable.​



default value:

- /opt/cloudera/parcels/CDH/lib/hadoop/lib/native

core files to search:

Java opts for the MR App Master processes. The following symbol, if present, will be interpolated:

@taskid@ is replaced by current TaskID. Any other occurrences of '@' will go unchanged.

For example, to enable verbose gc logging to a file named for the taskid in /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of: -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc Usage of -Djava.library.path can cause programs to no longer function if hadoop native libraries are used.

These values should instead be set as part of LD_LIBRARY_PATH in the map / reduce JVM env using the and mapreduce.reduce.env config settings.

/opt/cloudera/parcels/ 是指Hadoop相关服务安装目录, 除了以上简单提醒的参数涉及外, 还包括环境运行时的Java直接依赖

hive-log4j Logging initialized using configuration in jar:file:   /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/hive-common-1.1.0-cdh5.14.0.jar


Hadoop 官方对于可能的参数key及解释 :

All Posts

Almost done…

We just sent you an email. Please click the link in the email to confirm your subscription!

OKSubscriptions powered by Strikingly