Using spark-submit I’m launching a java program. However I’m setting java arguments for the JVM that are not taken into account. I’m trying to specify the max/min heap free ratio.
However, even though the arguments are present (based on visualvm), the memory graph shows that they don’t affect the running program.
Here is the command I’m launching:
“/usr/jdk1.8.0_131/bin/java” “-Xmx4096M” “-Dspark.driver.extraJavaOptions=-XX:MinHeapFreeRatio=30 -XX:MaxHeapFreeRatio=30 -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005” “-Dspark.executor.extraJavaOptions=-XX:MinHeapFreeRatio=30 -XX:MaxHeapFreeRatio=30” “-XX:MinHeapFreeRatio=30” “-XX:MaxHeapFreeRatio=30”
As you can see, I’ve tried setting the arguments on the driver, executor and java command.
Here is the arguments from visualvm:
and here is the graph:
To be clear, what I’m trying to improve is the “unused” heap size.
When starting the program with these arguments (outside of spark), it behaves correctly.