Skip to content

Instantly share code, notes, and snippets.

@gbroccolo

gbroccolo/AWS Glue: load a spark dataframe directly from Parquet files stored in S3

Last active June 21, 2021 08:08
Show Gist options
  • Save gbroccolo/19917534b98680ffee816a5aaf738770 to your computer and use it in GitHub Desktop.
Save gbroccolo/19917534b98680ffee816a5aaf738770 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
AWS Glue: load a spark dataframe directly from Parquet files stored in S3
val inDF = glueContext.getSourceWithFormat(
connectionType = "s3",
options = JsonOptions(
Map(
"paths" -> Array(s"s3://path/to/s3/folder")
)
),
format = "parquet"
).getDynamicFrame()
.toDF()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment