Return to site

Hive Error: buildSupportsSnappy()Z

与CDH Cluster中的parcels

[fix hive issues]

· Hive

Error: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z

问题描述

报错的map , reduce 在log中均显示形如

java.lang.Exception: java.lang.UnsatisfiedLinkError:

Error: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z

的error信息, 而且连续三天出问题的机器均在210与195身上, 通过回顾近期的集群Datanode操作, 这两台机器均进行了reload操作。 因此怀疑是在上线过程中有些环境变量被修改导致MapReduce在执行过程中无法access native libraries。

https://stackoverflow.com/questions/22150417/hadoop-mapreduce-java-lang-unsatisfiedlinkerror-org-apache-hadoop-util-nativec/28187884

  1. Ensure that LD_LIBRARY_PATH and JAVA_LIBRARY_PATH contains the native directory path having the libsnappy.so** files.
  2. Ensure that LD_LIBRARY_PATH and JAVA_LIBRARY path have been exported in the SPARK environment(spark-env.sh).

然后在hive的配置目录中

core-site.xml

mapred-site.xml 查找要对应的 key value配置。

<property>
<name>yarn.app.mapreduce.am.admin.user.env</name>
<value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH:/opt/cloudera/parcels/CDH/lib/hadoop/lib/native:/mnt/disk1/cloudera/parcels/CDH/lib/hadoop/lib/native</value>
</property>​

 

<property>
<name>mapreduce.admin.user.env</name>
<value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH:/opt/cloudera/parcels/CDH/lib/hadoop/lib/native</value>
</property>

 

可见这里某值是在map reduce tasks有必要进行获取native libraries 权限, 以进行例如数据压缩。 此值若为空,则用系统环境变量替换。

通过查找, 发现是后者 第二个环境变量决定了

{ mapreduce.admin.user.env}

==================================================================

Expert: Additional execution environment entries for map and reduce task processes. This is not an additive property. You must preserve the original value if you want your map and reduce tasks to have access to native libraries (compression, etc). When this value is empty, the command to set execution envrionment will be OS dependent: For linux, use LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native. For windows, use PATH = %PATH%;%HADOOP_COMMON_HOME%\\bin.​

备注: 我的同事在第一次修改yarn.app.mapreduce.am.admin.user.env 后, 问题没有解决。

问题拓展

CDH Cluster上的配置如何生效的?

配置文件放置于/var/run/cloudera-scm-agent/process/

如: /var/run/cloudera-scm-agent/process/193-hdfs-NAMENODE/core-site.xml

这些配置文件是通过Cloudera Manager启动相应服务(如HDFS)时生成的,例如当我们修改了Cloudera Manage配置界面的值后, 重启后就会生成一份新的配置xml 替换原有的。

/opt/cloudera/parcels的依赖列表

【core-site.xml 的 HDFS 复制高级配置代码段(安全阀)】

- fs.oss.core.dependency.path

/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/oss-library-2.2.0.jar

【yarn.app】

- yarn.app.mapreduce.am.admin.user.env

Environment variables for the MR App Master processes for admin purposes. These values are set first and can be overridden by the user env (yarn.app.mapreduce.am.env) Example : 1) A=foo This will set the env variable A to foo 2) B=$B:c This is inherit app master's B env variable.​

【mapreduce.admin】

mapreduce.admin.user.env

default value:

- /opt/cloudera/parcels/CDH/lib/hadoop/lib/native

core files to search:

libsnappy.so

Java opts for the MR App Master processes. The following symbol, if present, will be interpolated:

@taskid@ is replaced by current TaskID. Any other occurrences of '@' will go unchanged.

For example, to enable verbose gc logging to a file named for the taskid in /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of: -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc Usage of -Djava.library.path can cause programs to no longer function if hadoop native libraries are used.

These values should instead be set as part of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and mapreduce.reduce.env config settings.

/opt/cloudera/parcels/ 是指Hadoop相关服务安装目录, 除了以上简单提醒的参数涉及外, 还包括环境运行时的Java直接依赖

hive-log4j Logging initialized using configuration in jar:file: hive-log4j.properties   /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/hive-common-1.1.0-cdh5.14.0.jar

参考

Hadoop 官方对于可能的参数key及解释 :

https://hadoop.apache.org/docs/r2.7.2/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

All Posts
×

Almost done…

We just sent you an email. Please click the link in the email to confirm your subscription!

OKSubscriptions powered by Strikingly