hive dynamic partitions insert java.lang.OutOfMemoryError: Java heap space

动态分区问题，如果数据量大或者当动态分区大甚至只有十几个时，会出现如下异常：

2015-10-23 16:43:54,165 INFO [fetcher#10] org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: assigned 20 of 34 to spark-03:13562 to fetcher#10
2015-10-23 16:43:54,166 WARN [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hive (auth:SIMPLE) cause:org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#9
2015-10-23 16:43:54,167 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#9at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:415)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.OutOfMemoryError: Java heap spaceat org.apache.hadoop.io.BoundedByteArrayOutputStream.(BoundedByteArrayOutputStream.java:56)at org.apache.hadoop.io.BoundedByteArrayOutputStream.(BoundedByteArrayOutputStream.java:46)at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.(InMemoryMapOutput.java:63)at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:304)at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:294)at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:511)at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:329)at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)

  参考issue： https://issues.apache.org/jira/browse/MAPREDUCE-6108
https://issues.apache.org/jira/browse/MAPREDUCE-6447 源码： https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java#L254 默认参数： https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml mem参考 http://stackoverflow.com/questions/24070557/what-is-the-relation-between-mapreduce-map-memory-mb-and-mapred-map-child-jav 参数理解：

mapreduce.map.java.opts   -xmx配置的  heap memory  cloudera mapreduce.map.java.opts.max.heap 一般设置java.opts为memory.mb的75%
mapreduce.reduce.java.opts  -xmx配置的  heap memory  cloudera mapreduce.reduce.java.opts.max.heap 一般设置java.opts为memory.mb的75%
mapreduce.map.memory.mb  1G默认
mapreduce.reduce.memory.mb 1G默认
mapreduce.reduce.memory.totalbytes

mapreduce.reduce.shuffle.parallelcopies  shuffle开启的fetcher线程数 apache默认5，choudera默认10 mapreduce.reduce.shuffle.input.buffer.percent 默认0.7 mapreduce.reduce.shuffle.memory.limit.percent默认0.25  如上3个参数相乘得小于1，否则将报如上错。 将mapreduce.reduce.shuffle.parallelcopies调成5，可以解决此问题。 另外cloudera hive hive.stats.autogather默认为true，即插入数据时会优化统计，如此在大的动态分区时load数据后会有一段很长时间的统计，且操作hive元数据表，例如每个分区的文件数，行数等等。耗时比较长时可能会timeout，需要将其设成false。 详细查看  https://cwiki.apache.org/confluence/display/Hive/StatsDev。

本文来自互联网用户投稿，文章观点仅代表作者本人，不代表本站立场，不承担相关法律责任。如若转载，请注明出处。 如若内容造成侵权/违法违规/事实不符，请点击【内容举报】进行投诉反馈！