ECMonsen ecmonsen
ecmonsen
/ gist:e1e3e6906def081a5fbf43ad3a653171
Last active
December 7, 2023 22:57
Make an Athena-created view available to Glue Jobs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Making an Athena view queryable in a Glue job | |
| # | |
| # When a view is created by running an Athena SQL query, it appears in the Glue catalog but errors are raised when | |
| # querying the view in a Glue job or a Glue Spark context. | |
| # | |
| # Use this gist to programmatically update the view's metadata in the Glue catalog. | |
| # | |
| # After this you should be able to run `spark.sql("select * from mydb.myview")` without errors. | |
| # | |
| # Assumes you have created the view `mydb.myview` in Athena. |
ecmonsen
/ gist:76759c5ab42a1973ef2dac7668bfe883
Created
October 3, 2023 23:39
Pseudo-python in response to a recent interview question.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| """ | |
| Build the start of an e2e pipeline designed to be robust, extensible and scalable. | |
| Approach and structure is open ended. | |
| Use any packages you like but ensure code is as close to executable as possible. | |
| Input | |
| API endpoint = "testurl.com/endpoint" | |
| - json response that contains some IDs, a description of that ID, and the most | |
| recent modification date for that id’s description. | |
| Has three columns: |