Upcoming changes for LocalStack v3

Announcement
, ,
1

Hello everyone!
We are happy to announce that we released LocalStack 3.0!
This release introduces a lot of great new features, but also cleans up some legacy code and configurations. This page is meant to give you an overview over the changes such that you can migrate your LocalStack setup accordingly.

Tentative Timeline

  • Thursday November 9., 2023: latest ships the v3 release candidate.
    • :warning: We will merge our release branch for v3 into master, which will update the latest Docker image to contain the current release candidate of v3.
    • This will put into effect all breaking changes that we are introducing for v3, and will affect your pipeline if you are using latest.
  • Thursday November 16., 2023: v3 Release
    • The tagged release v3.0.0 will be published.

What’s changed?

The final release notes can be found here: Release v3.0.0 · localstack/localstack · GitHub

Community

  • Completely new Step Functions provider
  • Improved native S3 provider becoming the default
  • Improved performance for DynamoDB
  • Upgraded to Python 3.11 and Debian Bookworm

Pro

How to migrate

Networking

We have published an extensive migration guide for our networking-related changes on this Discuss page

  • The default container name is renamed from localstack_main to localstack-main
    • Our default container name localstack_main is not a valid URL, so when the container name is used as a hostname, some libraries refuse to connect. With this release, we use the default container name of localstack-main. There may be breakages with tooling that expects the new container name, but a container is running with the old container name. Please update your tooling accordingly.
    • Most prominently, this is container name used in docker-compose files.
    • If you are using the LocalStack CLI, please just make sure to update to the latest version after the release. The CLI will handle this migration automatically.
  • HOSTNAME_EXTERNAL and LOCALSTACK_HOSTNAME are removed in favor of LOCALSTACK_HOST
    • We are simplifying how to configure LocalStack by fully switching over to use LOCALSTACK_HOST to configure URLs returned by AWS services. Please update your usages of HOSTNAME_EXTERNAL (default: localhost) and LOCALSTACK_HOSTNAME to use LOCALSTACK_HOST (default: localhost.localstack.cloud:4566). You can find more details in our Networking Migration Guide for LocalStack 3.0.

Strict Service Loading - SERVICES environment variable

With 3.0, we are introducing strict service loading based on the SERVICES environment variable.

In older versions of LocalStack, this variable defined the set of services which should be loaded by default. New versions of LocalStack load the services on-demand dynamically on their first invocation. With 3.0, the SERVICES variable will be used as a strict list of services that should be loaded. The loading of all other services will be prevented.

  • If you are using the SERVICES environment variable, without using EAGER_SERVICE_LOADING, please remove the variable (it does not have any effect in your current config).
  • If you are using the SERVICES environment variable in combination with EAGER_SERVICE_LOADING, please make sure it contains all services you are using. Any additional services (which would have been loaded on demand in pre-v3 versions of LocalStack) will not be loaded anymore.

Cloud Pods

In our upcoming 3.0 release, we will be removing Community Cloud Pods. Moving forward, Cloud Pods will only be accessible to Pro/Team users. We understand this may affect some users and we are open to feedback, through our GitHub Issues or Discuss Forum.

However, please stay tuned for an announcement in this area in the upcoming weeks.

S3

With 2.3.0 we introduced a new, opt-in, LocalStack-native, faster S3 provider with no external dependencies, improved parity, better memory usage, and better support for bucket versioning and multipart upload. With 3.0, we are promoting this provider to become the default!

S3_DIR is not supported anymore with the new provider. However, S3 persistence has been greatly improved, and you should be able to seed your data with Initialization hooks.

Besides that, users should be mostly unaffected, but may experience some breakage depending on previous behavior that was not aligned with AWS. Should you run into problems, you can, for now, activate the old provider with PROVIDER_OVERRIDE_S3=legacy_v2.

Step Functions

Step Functions in LocalStack has been completely re-written. If you run into issues, you can, for now, use the legacy provider using PROVIDER_OVERRIDE_STEPFUNCTIONS=legacy.

ElastiCache

ElastiCache in LocalStack has been completely re-written. The new ElastiCache implementation provides much better parity with AWS and resolves some issues around Redis cluster management in the previous provider. In case you run into issues, you can, for now, use the legacy provider using PROVIDER_OVERRIDE_ELASTICACHE=legacy.

Multi-Region and Multi-Account Support in SQS and IoT endpoints

In order to improve the multi-account and multi-region support for SQS and IoT, the endpoints generated for certain resources of these services (like SQS queues, or IoT endpoints) now contain the account ID and the region.

When using these services, please do not make any assumptions on the URLs generated by LocalStack, but use the URLs returned in the responses.

  • For SQS queues, please use the QueueUrl contained in responses to CreateQueue operation.
  • For IoT endpoints, please use the endpointAddress contained in responses to the DescribeEndpoint operation.

Other notable changes

  • The IoT services (iot, iot-analytics, iot-data, iot-wireless) do not ship with their third-party packages pre-installed in the Docker image anymore.
    • Instead, these packages will be downloaded on-demand when the services need them. This helps reducing the Docker image size for all users.
    • Please configure the volume directory to cache packages that are downloaded on-demand.
  • RDS now actually uses MySQL (instead of MariaDB) if the engine mysql is used.
    • Up until now, the engine mysql would always create a mariadb instance behind the scenes.
    • With 3.0 we change this behavior, and RDS will create a real mysql instance in a new docker container. The image will be taken from the official MySQL image on Dockerhub, and the EngineVersion defines the image-tag that will be used.
    • Should you run into problems, you can, for now, set RDS_MYSQL_DOCKER=0.
  • RDS now correctly returns only the hostname in the Endpoint field of responses to the CreateDBCluster operation.
    • Previously, this field also contained the port, however the port has its own field in the response.
    • Should you run into problems, you can, for now, set RDS_CLUSTER_ENDPOINT_HOST_ONLY=0.
  • After its initial preview phase, AWS RAM is now part of our paid offering and will be further developed and integrated with our IAM enforcement in future releases.

Removals of Deprecations

The following section lists removals of configurations that have already been deprecated in previous versions, where the usage with current images will already log warnings on the usage of outdated configurations. If you have upgraded to the latest changes in previous releases, these changes will not affect you.

Legacy CLI commands

The following CLI command groups are being removed:

  • infra - Please use localstack start instead.
  • daemons - This feature is being removed.

Legacy Internal Endpoints

The following endpoints, which have been deprecated a long time ago, will now be removed:

  • /health → Please use /_localstack/health instead
  • /cloudwatch/metrics/raw → Please use /_aws/cloudwatch/metrics/raw instead
  • /_localstack/ses → Please use /_aws/ses instead
  • /_pods → Please use /_localstack/pods instead
  • /login → Please use /_aws/cognito-idp/login instead
  • /logout → Please use /_aws/cognito-idp/logout instead
  • /signup → Please use /_aws/cognito-idp/signup instead
  • /forgotPassword → Please use /_aws/cognito-idp/forgotPassword instead
  • /oauth2/token → Please use /_aws/cognito-idp/oauth2/token instead
  • /oauth2/authorize → Please use /_aws/cognito-idp/oauth2/authorize instead

Legacy S3 Provider - PROVIDER_OVERRIDE_S3=legacy

The legacy S3 provider is being removed. This only affects you if you are setting PROVIDER_OVERRIDE_S3=legacy. Please migrate to the new S3 provider by removing this setting.

Legacy Lambda Provider - PROVIDER_OVERRIDE_LAMBDA=legacy

The legacy Lambda provider is being removed. This only affects you if you are setting PROVIDER_OVERRIDE_LAMBDA=legacy. Please migrate to the new Lambda provider by removing this setting. You can find more details in our Lambda migration guide.

With the removal of the legacy lambda provider, the following deprecated environment variables will be removed:

  • LAMBDA_EXECUTOR
  • LAMBDA_STAY_OPEN_MODE
  • LAMBDA_REMOTE_DOCKER
  • LAMBDA_CODE_EXTRACT_TIME
  • LAMBDA_CONTAINER_REGISTRY
  • LAMBDA_FALLBACK_URL
  • LAMBDA_FORWARD_URL
  • LAMBDA_XRAY_INIT

Legacy KMS Provider - KMS_PROVIDER=local-kms

The legacy “local-kms” provider is being removed. This only affects you if you are setting KMS_PROVIDER=local-kms. Please migrate to the new KMS provider by removing this setting.

Legacy port and host config - EDGE_PORT, EDGE_PORT_HTTP, EDGE_BIND_HOST

We have removed the ability to configure what address and port LocalStack listens on via the variables EDGE_PORT, EDGE_PORT_HTTP, and EDGE_BIND_HOST. Please migrate to use GATEWAY_LISTEN instead. You can find more details on the migration path in the release notes of v2.

Legacy region config - DEFAULT_REGION, USE_SINGLE_REGION

We have removed DEFAULT_REGION and USE_SINGLE_REGION, which have been deprecated since 0.12.7. LocalStack is now multi-account and multi-region-aware, and these settings are not used anymore.

Legacy BigData Image Support - BIGDATA_MONO_CONTAINER

The support for the usage of the localstack/bigdata image (with BIGDATA_MONO_CONTAINER=0) is being removed. Please just remove this configuration variable to switch to the (default) “mono container mode”.

With the removal of the legacy bigdata image support, the following deprecated environment variables will be removed:

  • AUTOSTART_UTIL_CONTAINERS

Legacy CloudPods client - ACTIVATE_NEW_POD_CLIENT

The support for the legacy CloudPods client (with ACTIVATE_NEW_POD_CLIENT=0) is being removed. Please just remove this configuration variable to switch to the new (default) cloudpods client.

Others

In addition to the changes above, the following configuration variables have been removed (which has been announced in previous releases):

  • ES_ENDPOINT_STRATEGY, ES_MULTI_CLUSTER, ES_CUSTOM_BACKEND
    • These options have been removed. Please OPENSEARCH__ENDPOINT_STRATEGY, OPENSEARCH_MULTI_CLUSTER, or OPENSEARCH_CUSTOM_BACKEND instead.
  • KINESIS_INITIALIZE_STREAMS
    • Please use init hooks to pre-seed your environment instead.
  • MOCK_UNIMPLEMENTED
    • This feature has been discontinued. Please remove this environment variable.
  • SYNCHRONOUS_*_EVENTS
    • This configurations break parity with AWS, which is why they have been removed. Please remove this environment variable.
  • EC2_AUTOSTART_DAEMON
    • The localstack local daemons have been removed. Please remove this environment variable.
  • *_PORT_EXTERNAL
    • These configuration variables have been superseded by the new networking config (see above).
4 Likes