关于SparkConf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

/**
 * Configuration for a Spark application. Used to set various Spark parameters as key-value pairs.
 * ---
 * 这是 Spark Application 的配置，可以用来设置 Spark 的 KV 格式的参数
 *
 * Most of the time, you would create a SparkConf object with `new SparkConf()`, which will load
 * values from any `spark.*` Java system properties set in your application as well. In this case,
 * parameters you set directly on the `SparkConf` object take priority over system properties.
 * ---
 * 大多数情况下，可用通过 new 实例化一个 SparkConf 的对象，这可以把所有的以 spark 开头的一些属性配置好
 * 并且直接对 SparkConf 进行参数设置的优先级是高于系统属性文件的
 *
 * For unit tests, you can also call `new SparkConf(false)` to skip loading external settings and
 * get the same configuration no matter what the system properties are.
 * ---
 * 通过 `new SparkConf(false)` 可以在进行单元测试的时候不去读取外部的设置
 *
 * All setter methods in this class support chaining. For example, you can write
 * `new SparkConf().setMaster("local").setAppName("My app")`.
 * ---
 * 所有的 setter 方法都支持链式表达
 *
 * @param loadDefaults whether to also load values from Java system properties
 *                     是否从 Java 系统属性中读取配置参数
 *
 * @note Once a SparkConf object is passed to Spark, it is cloned and can no longer be modified
 * by the user. Spark does not support modifying the configuration at runtime.
 * ---
 * 一旦 SparkConf 对象传给 Spark，他会被其他组件 clone，并且不能再动态的被任何用户修改
 */

1
2
3
4
5
6
7
8

/** Copy this object */
    override def clone: SparkConf = {
    val cloned = new SparkConf(false)
    settings.entrySet().asScala.foreach { e =>
      cloned.set(e.getKey(), e.getValue(), true)
    }
    cloned
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

 /**
  * Creates a modified version of a SparkConf with the parameters that can be passed separately
  * to SparkContext, to make it easier to write SparkContext's constructors. This ignores
  * parameters that are passed as the default value of null, instead of throwing an exception
  * like SparkConf would.
  * 这里提供了一个在将 conf 传入 SparkContext 前 update SparkConf 的一个方法，基本上就是提供了 master，appName 等的 update
  */
 private[spark] def updatedConf(
     conf: SparkConf,
     master: String,
     appName: String,
     sparkHome: String = null,
     jars: Seq[String] = Nil,
     environment: Map[String, String] = Map()): SparkConf =
 {
   val res = conf.clone()
   res.setMaster(master)
   res.setAppName(appName)
   if (sparkHome != null) {
     res.setSparkHome(sparkHome)
   }
   if (jars != null && !jars.isEmpty) {
     res.setJars(jars)
   }
   res.setExecutorEnv(environment.toSeq)
   res
 }

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

// /Users/runzhliu/workspace/Source/spark/core/src/test/scala/org/apache/spark/SparkConfSuite.scala

// 配置什么得到什么
test("loading from system properties") {
    System.setProperty("spark.test.testProperty", "2")
    System.setProperty("nonspark.test.testProperty", "0")
    val conf = new SparkConf()
    assert(conf.get("spark.test.testProperty") === "2")
    assert(!conf.contains("nonspark.test.testProperty"))
}

// new SparkConf(false) 就不会读取配置信息了
test("initializing without loading defaults") {
    System.setProperty("spark.test.testProperty", "2")
    val conf = new SparkConf(false)
    assert(!conf.contains("spark.test.testProperty"))
}