Skip to content

Instantly share code, notes, and snippets.

@adrianulbona

adrianulbona/parquet.scala

Last active March 14, 2017 17:21
Show Gist options
  • Save adrianulbona/ade2e20f4dc960a72d9c56f931fe8a30 to your computer and use it in GitHub Desktop.
Save adrianulbona/ade2e20f4dc960a72d9c56f931fe8a30 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
parquet.scala
case class Way(id: Long, nodes: Array[Long], tags: Map[String, String])
val w1 = Way(1, Array(100, 200, 13, 20), Map("type"-> "highway"))
val w2 = Way(2, Array(30, 13, 500), Map("type"-> "ulita"))
val wayRDD = sc.parallelize(Array(w1, w2))
val wayDF = wayRDD.toDF
wayDF.write.parquet("path/to/mighty_map")
val persistedWayDF = sqlContext.read.parquet("path/to/mighty_map")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment