使用spark-submit时,经常需要传递一些参数,这里写了一个简单的参数解析方法,代码如下:
val sparkConf = new SparkConf().setAppName("intention recognition") sparkConf.set("spark.driver.maxResultSize", "4g") val sparkContext = new SparkContext(sparkConf) val argsIterator = args.toIterator val trainLabel = argsIterator.next val trainRatio = argsIterator.next.toDouble val testRatio = argsIterator.next.toDouble val sampleRDD = sparkContext.textFile(argsIterator.next) while (argsIterator.hasNext) { sampleRDD.union(sparkContext.textFile(argsIterator.next)) }