mrbrahman
tzachz
/ CombineMaps.scala
Last active
January 26, 2023 04:31
Apache Spark UserDefinedAggregateFunction combining maps
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import org.apache.spark.SparkContext | |
| import org.apache.spark.sql.expressions.{MutableAggregationBuffer, UserDefinedAggregateFunction} | |
| import org.apache.spark.sql.types._ | |
| import org.apache.spark.sql.{Column, Row, SQLContext} | |
| /*** | |
| * UDAF combining maps, overriding any duplicate key with "latest" value | |
| * @param keyType DataType of Map key | |
| * @param valueType DataType of Value key | |
| * @param merge function to merge values of identical keys |