在执行spark程序时,很容易出现以下错误:
Total size of serialized results of 19 tasks (1069.2 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
解决方法是设置spark.driver.maxResultSize。
设置方法如下:
val sparkConf = new SparkConf().setAppName("your appname") sparkConf.set("spark.driver.maxResultSize", "4g") val sparkContext = new SparkContext(sparkConf)