Update: The information in this post is slightly outdated. In the meantime, we have released a public preview version of the LocalStack Snowflake emulator - see blog post here: Introducing LocalStack for Snowflake: The new emulator to build & test data pipelines locally - LocalStack
Hi all,
About a year ago we have released the first version of LocalStack Extensions: Extensions are a powerful mechanism to plug additional functionality into LocalStack - ranging from additional service emulators, to value-add features like Chaos Engineering, request logging, cloud resource replication/proxying, and more. An initial list of extensions can be found in this repository as well as in the LocalStack Web app.
Over the last couple of months we have been experimenting with a LocalStack Snowflake emulator extension, which allows to develop and test your Snowflake data pipelines entirely on your local machine!
Below is a simple sample snippet that illustrates how you can use the Python Snowflake client library to connect to the the Snowflake emulator running on your local machine (note that the hostname snowflake.localhost.localstack.cloud
resolves to a local IP address):
connection = snowflake.connector.connect(
user="test",
password="test",
account="test",
database="test",
host="snowflake.localhost.localstack.cloud",
)
connection.cursor().execute("CREATE TABLE table1(col1 INT)")
This extension is still highly experimental and covers only a small subset of the Snowflake API and functionality - although some of the key features are already available today, including:
- Basic operations on warehouses, databases, schemas, and tables (e.g., https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-example)
- Storing files in user/data/named stages (Choosing an internal stage for local files | Snowflake Documentation)
- Snowpark libraries (e.g., Snowpark Developer Guide for Python | Snowflake Documentation)
- Snowpipe streaming with Kafka connector (Using Snowflake Connector for Kafka with Snowpipe Streaming | Snowflake Documentation)
- JavaScript and Python UDFs (https://docs.snowflake.com/en/developer-guide/udf/javascript/udf-javascript-introduction)
- … and quite a bit more!
Below is a sneak preview of the extension running in LocalStack, and using DBeaver to connect to it and run Snowflake SQL queries against it:
We are currently looking for early adopters who would like to give it a try with this extension, provide feedback on the current functionality, and help us shape the roadmap going forward.
If you’re interested in learning more, please reach out to us directly - either in this thread, or via DM, or via email to support@localstack.cloud .
We’re looking forward to getting your feedback and learning more about your use cases for developing & testing data pipelines locally with Snowflake!