A guide to uploading third-party Python packages for use in Snowflake notebooks running on the warehouse

uploading_third_party_packages_to_snowflake.md

Uploading third-party Python packages to Snowflake

Note that this guide is specifically for notebooks running on a warehouse.

See here for when running a notebook on a container runtime.

1. Create a stage

If yo want to use a Python package within a notebook running on a warehouse and it's not available via Snowflake’s Anaconda channel of pre-approved packages you’ll need to use a stage to upload the source code of the Python packages you want to use.

You can create the stage via Snowsight (Snowflake's web interface) or via a script:

CREATE STAGE PYTHON_PACKAGES 
	DIRECTORY = ( ENABLE = true ) 
	COMMENT = 'Third-party Python packages for use by the analytics team'
  ;

2. Upload the package source code to the stage

Next you need to upload to the stage the source code of the 3rd party Python package as a zip folder.

For example, using the package prettytable as an example, you’d want to upload this directory:

Note that you’re not uploading the entire package code, which would include the README.MD and tests directory.

You’re solely uploading the code needed to execute the package. It should be fairly obvious which directory this is, as generally it will be named src or the name of the package itself.

You can use Snowsight to upload the folder or a command:

PUT file:///Users/Ben/Downloads/prettytable.zip @EXAMPLE_DATABASE.EXAMPLE_SCHEMA.PYTHON_PACKAGES AUTO_COMPRESS=TRUE;

3. Using the package in a Snowflake Notebook

You can now install the package via your Snowflake Notebook.

First click the Package button at the top and then select the Stage Packages tab, where you’ll need to enter the path to the package:

The package file path is the name of the database, schema and stage followed by the name of the package. So in our prettytable example it would be:

@EXAMPLE_DATABASE.EXAMPLE_SCHEMA.PYTHON_PACKAGES/prettytable.zip

Once you’ve clicked save you’ll be able to import and use the library:

Useful SQL commands for managing packages

-- Show all stages
SHOW STAGES ;

-- List files in a stage
list @example_database.example_schema.python_packages ;

-- Remove a specific file from a stage:
REMOVE @example_database.example_schema.python_packages/prettytable.zip ;

Notes

• If the package you’re importing/installing requires dependencies not available via Snowflake’s Anaconda packages you’ll need to install those dependencies too.

• Uploading the entire package (including the README.md, LICENCE.txt, etc) to the stage will result in you not being able to import/use the package in the notebook. You need to make sure you’re uploading the actual source code folder as a zip folder. You can instead upload an actual .py file but likely there will be multiple Python files that will need to be bundled together as a zip folder in order for the package to work.