大量推送本地文件到hdfs如下
hadoop fs -put ${local_path} ${hdfs_path}报错。
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:)
at java.lang.StringBuffer.toString(StringBuffer.java:)
at java.net.URI.toString(URI.java:)
at java.net.URI.<init>(URI.java:)
at org.apache.hadoop.fs.Path.initialize(Path.java:)
at org.apache.hadoop.fs.Path.<init>(Path.java:)
at org.apache.hadoop.fs.Path.getPathWithoutSchemeAndAuthority(Path.java:)
at org.apache.hadoop.fs.shell.CommandWithDestination.checkPathsForReservedRaw(CommandWithDestination.java:)
at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:)
at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:)
at org.apache.hadoop.fs.shell.CommandWithDestination.recursePath(CommandWithDestination.java:)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:)
at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:)
at org.apache.hadoop.fs.shell.CommandWithDestination.recursePath(CommandWithDestination.java:)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:)
at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:)
at org.apache.hadoop.fs.shell.CommandWithDestination.recursePath(CommandWithDestination.java:)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:)
at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:)
at org.apache.hadoop.fs.shell.CommandWithDestination.recursePath(CommandWithDestination.java:)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:)
at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:)
at org.apache.hadoop.fs.shell.CommandWithDestination.recursePath(CommandWithDestination.java:)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:)
at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:)
at org.apache.hadoop.fs.shell.CommandWithDestination.recursePath(CommandWithDestination.java:)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:)
at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:)
at org.apache.hadoop.fs.shell.CommandWithDestination.recursePath(CommandWithDestination.java:)
在hadoop命令下发现：
hadoop-2.7./bin/hadoop: exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@"
查找JAVA_HEAP_MAX
发现
hadoop-2.7./libexec/hadoop-config.sh: JAVA_HEAP_MAX="-Xmx""$HADOOP_HEAPSIZE""m"
继续查找HADOOP_HEAPSIZE
发现
hadoop-2.7./libexec/hadoop-config.sh: JAVA_HEAP_MAX="-Xmx""$HADOOP_HEAPSIZE""m"
以及
hadoop-2.7./share/doc/hadoop/hadoop-project-dist/hadoop-common/ClusterSetup.html:<li><tt>HADOOP_HEAPSIZE</tt> / <tt>YARN_HEAPSIZE</tt> - The maximum amount of heapsize to use, in MB e.g. if the varibale is set to the heap will be set to 1000MB. This is used to configure the heap size for the daemon. By default, the value is . If you want to configure the values separately for each deamon you can use.</li>
<li>
<tt>HADOOP_HEAPSIZE</tt>
/ <tt>YARN_HEAPSIZE</tt> - The maximum amount of heapsize to use, in MB e.g. if the varibale is set to 1000 the heap
will be set to 1000MB. This is used to configure the heap size for the daemon. By default, the value is 1000. If you
want to configure the values separately for each deamon you can use.
</li>
调整JVM堆的最大值：
export HADOOP_HEAPSIZE=
