NoSuchBucket error when copying file

Platform
1

After successfully uploading a file, I’m attempting to copy it to another key (s3.CopyObject) within same bucket. Unfortunately, I’m getting NoSuchBucket error.

The same script is working fine with real S3 credentials + endpoint.

Do my logs give any indication to the possible reason for the error?

CleanShot 2023-12-04 at 21.16.00@2x
CleanShot 2023-12-04 at 21.16.00@2x1920×244 120 KB

integration-localstack-1  | 2023-12-04T20:02:37.456 DEBUG --- [   asgi_gw_1] l.aws.serving.wsgi         : GET localhost.localstack.cloud:4566/_localstack/health
integration-localstack-1  | 2023-12-04T20:02:38.397 DEBUG --- [   asgi_gw_0] l.aws.serving.wsgi         : PUT host.docker.internal:4566/order-source-data/cache/9483623b6076232cd76aef52801c443d.json
integration-localstack-1  | 2023-12-04T20:02:38.398 DEBUG --- [   asgi_gw_0] l.aws.protocol.serializer  : Determined accept type (None) is not supported by this serializer. Using default of this serializer: application/xml
integration-localstack-1  | 2023-12-04T20:02:38.398  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.PutObject => 200; 000000000042/us-east-1; PutObjectRequest({'ACL': None, 'Bucket': 'order-source-data', 'CacheControl': None, 'ContentDisposition': 'inline; filename="order-231204150233-190.json"; filename*=UTF-8\'\'order-231204150233-190.json', 'ContentEncoding': None, 'ContentLanguage': None, 'ContentLength': 330, 'ContentMD5': 'oAzC5jzdQcYerDIBHkBBEA==', 'ContentType': None, 'ChecksumAlgorithm': None, 'ChecksumCRC32': None, 'ChecksumCRC32C': None, 'ChecksumSHA1': None, 'ChecksumSHA256': None, 'Expires': None, 'GrantFullControl': None, 'GrantRead': None, 'GrantReadACP': None, 'GrantWriteACP': None, 'Key': 'cache/9483623b6076232cd76aef52801c443d.json', 'Metadata': {}, 'ServerSideEncryption': None, 'StorageClass': None, 'WebsiteRedirectLocation': None, 'SSECustomerAlgorithm': None, 'SSECustomerKey': None, 'SSECustomerKeyMD5': None, 'SSEKMSKeyId': None, 'SSEKMSEncryptionContext': None, 'BucketKeyEnabled': None, 'RequestPayer': None, 'Tagging': None, 'ObjectLockMode': None, 'ObjectLockRetainUntilDate': None, 'ObjectLockLegalHoldStatus': None, 'ExpectedBucketOwner': None, 'Body': <_io.BufferedReader>}, headers={'Accept-Encoding': '', 'Content-Disposition': 'inline; filename="order-231204150233-190.json"; filename*=UTF-8\'\'order-231204150233-190.json', 'Expect': '100-continue', 'Content-Md5': 'oAzC5jzdQcYerDIBHkBBEA==', 'User-Agent': 'aws-sdk-ruby3/3.190.0 ua/2.0 api/s3#1.141.0 os/macos#22 md/arm64 lang/ruby#3.1.2 md/3.1.2 cfg/retry-mode#legacy ft/resource', 'Host': 'host.docker.internal:4566', 'X-Amz-Date': '20231204T200238Z', 'X-Amz-Content-Sha256': '36f88290181ac7af7eed674761e237d45b2473e5c6c99799ef7e14eb975045cd', 'Authorization': 'AWS4-HMAC-SHA256 Credential=LSIAQAAAAAAVNCBMPNSG/20231204/us-east-1/s3/aws4_request, SignedHeaders=content-disposition;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=b13251477c3c60c69431095a8f6fc6f26dc1c434a5f7aed34166dd69199c98d1', 'Content-Length': '330', 'Accept': '*/*', 'x-localstack-tgt-api': 's3', 'x-moto-account-id': '000000000042'}); PutObjectOutput({'ETag': '"a00cc2e63cdd41c61eac32011e404110"', 'ServerSideEncryption': 'AES256'}, headers={'Content-Type': 'application/xml', 'ETag': '"a00cc2e63cdd41c61eac32011e404110"', 'x-amz-server-side-encryption': 'AES256', 'x-amz-request-id': '0d159dba-4c29-4f14-a8b9-6c111b3c4bf0', 'x-amz-id-2': 's9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=', 'Connection': 'close'})
integration-localstack-1  | 2023-12-04T20:02:38.450 DEBUG --- [   asgi_gw_1] l.aws.serving.wsgi         : PUT host.docker.internal:4566/order-source-data/7c48de5958df44c67330d53994da749a.json
integration-localstack-1  | 2023-12-04T20:02:38.451 DEBUG --- [   asgi_gw_1] l.aws.protocol.serializer  : Determined accept type (None) is not supported by this serializer. Using default of this serializer: application/xml
integration-localstack-1  | 2023-12-04T20:02:38.451  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.CopyObject => 404 (NoSuchBucket); 000000000042/us-east-1; CopyObjectRequest({'ACL': None, 'Bucket': 'order-source-data', 'CacheControl': None, 'ChecksumAlgorithm': None, 'ContentDisposition': 'inline; filename="order-231204150233-190.json"; filename*=UTF-8\'\'order-231204150233-190.json', 'ContentEncoding': None, 'ContentLanguage': None, 'ContentType': None, 'CopySource': 'order-source-data/cache/9483623b6076232cd76aef52801c443d.json', 'CopySourceIfMatch': None, 'CopySourceIfModifiedSince': None, 'CopySourceIfNoneMatch': None, 'CopySourceIfUnmodifiedSince': None, 'Expires': None, 'GrantFullControl': None, 'GrantRead': None, 'GrantReadACP': None, 'GrantWriteACP': None, 'Key': '7c48de5958df44c67330d53994da749a.json', 'Metadata': {}, 'MetadataDirective': 'REPLACE', 'TaggingDirective': None, 'ServerSideEncryption': None, 'StorageClass': None, 'WebsiteRedirectLocation': None, 'SSECustomerAlgorithm': None, 'SSECustomerKey': None, 'SSECustomerKeyMD5': None, 'SSEKMSKeyId': None, 'SSEKMSEncryptionContext': None, 'BucketKeyEnabled': None, 'CopySourceSSECustomerAlgorithm': None, 'CopySourceSSECustomerKey': None, 'CopySourceSSECustomerKeyMD5': None, 'RequestPayer': None, 'Tagging': None, 'ObjectLockMode': None, 'ObjectLockRetainUntilDate': None, 'ObjectLockLegalHoldStatus': None, 'ExpectedBucketOwner': None, 'ExpectedSourceBucketOwner': None}, headers={'Accept-Encoding': '', 'Content-Disposition': 'inline; filename="order-231204150233-190.json"; filename*=UTF-8\'\'order-231204150233-190.json', 'X-Amz-Copy-Source': 'order-source-data/cache/9483623b6076232cd76aef52801c443d.json', 'X-Amz-Metadata-Directive': 'REPLACE', 'User-Agent': 'aws-sdk-ruby3/3.190.0 ua/2.0 api/s3#1.141.0 os/macos#22 md/arm64 lang/ruby#3.1.2 md/3.1.2 cfg/retry-mode#legacy ft/resource ft/s3-transfer', 'Host': 'host.docker.internal:4566', 'X-Amz-Date': '20231204T200238Z', 'X-Amz-Content-Sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=LSIAQAAAAAAVNCBMPNSG/20231204/us-east-1/s3/aws4_request, SignedHeaders=content-disposition;host;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-metadata-directive, Signature=6df16ba0fae5c1720039ff0703670fa0066aceb2829b3351d4c395047c3365ec', 'Content-Length': '0', 'Accept': '*/*', 'x-localstack-tgt-api': 's3', 'x-moto-account-id': '000000000042'}); NoSuchBucket(The specified bucket does not exist, headers={'Content-Type': 'application/xml', 'Content-Length': '234', 'x-amz-request-id': '118c3894-f7ff-4e03-9a01-38cb8870487d', 'x-amz-id-2': 's9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=', 'Connection': 'close'})
integration-localstack-1  | 2023-12-04T20:02:47.719 DEBUG --- [   asgi_gw_0] l.aws.serving.wsgi         : GET localhost.localstack.cloud:4566/_localstack/health
2

Hi @wkrsz,

This is unexpected, I’m not finding an obvious reason for the failure, except maybe the use of a non default account id. I’ll try to reproduce and come back to you soon! Thanks for the report.

3

Could you share what LocalStack version you are using? I tried to reproduce with latest or 3.0.2 with the following script and was unable to do so:

import boto3
from mypy_boto3_s3 import S3Client

s3_client: S3Client = boto3.client(
    "s3",
    endpoint_url="http://localhost:4566",
    aws_access_key_id="LSIAQAAAAAAVNCBMPNSG",
    aws_secret_access_key="test"
)

s3_client.create_bucket(Bucket="test")
s3_client.put_object(Bucket="test", Key="cache/test", Body="test")
s3_client.copy_object(Bucket="test", Key="test2", CopySource="test/cache/test", MetadataDirective="REPLACE")

And the logs:

2023-12-04T22:54:16.075  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.CreateBucket => 200; 000000000042/us-east-1; CreateBucketRequest({'ACL': None, 'Bucket': 'test', 'GrantFullControl': None, 'GrantRead': None, 'GrantReadACP': None, 'GrantWrite': None, 'GrantWriteACP': None, 'ObjectLockEnabledForBucket': None, 'ObjectOwnership': None, 'CreateBucketConfiguration': {}}, headers={'Accept-Encoding': 'identity', 'User-Agent': 'Boto3/1.29.5 md/Botocore#1.32.5 md/awscrt#0.19.17 ua/2.0 os/macos#23.1.0 md/arch#arm64 lang/python#3.11.6 md/pyimpl#CPython cfg/retry-mode#standard Botocore/1.32.5', 'host': 'localhost:4566', 'X-Amz-Date': '20231204T205415Z', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=LSIAQAAAAAAVNCBMPNSG/20231204/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=e70b1cd4a67805237ad143c25122a79b0116dd9eb1f6d0eaec22ae5733a54315', 'amz-sdk-invocation-id': '491fe96d-3375-4d81-971a-1ee661d39410', 'amz-sdk-request': 'attempt=1', 'Content-Length': '0', 'x-localstack-tgt-api': 's3', 'x-moto-account-id': '000000000042'}); CreateBucketOutput({'Location': '/test'}, headers={'Content-Type': 'application/xml', 'Location': '/test', 'x-amz-request-id': '47ef59b9-4208-4f05-93e6-0f4143b5327f', 'x-amz-id-2': 's9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=', 'Connection': 'close'})
2023-12-04T22:54:16.078 DEBUG --- [   asgi_gw_0] l.aws.serving.wsgi         : PUT localhost:4566/test/cache/test
2023-12-04T22:54:16.080 DEBUG --- [   asgi_gw_0] l.aws.protocol.serializer  : Determined accept type (None) is not supported by this serializer. Using default of this serializer: application/xml
2023-12-04T22:54:16.080  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.PutObject => 200; 000000000042/us-east-1; PutObjectRequest({'ACL': None, 'Bucket': 'test', 'CacheControl': None, 'ContentDisposition': None, 'ContentEncoding': None, 'ContentLanguage': None, 'ContentLength': 4, 'ContentMD5': 'CY9rzUYh03PK3k6DJie09g==', 'ContentType': None, 'ChecksumAlgorithm': None, 'ChecksumCRC32': None, 'ChecksumCRC32C': None, 'ChecksumSHA1': None, 'ChecksumSHA256': None, 'Expires': None, 'GrantFullControl': None, 'GrantRead': None, 'GrantReadACP': None, 'GrantWriteACP': None, 'Key': 'cache/test', 'Metadata': {}, 'ServerSideEncryption': None, 'StorageClass': None, 'WebsiteRedirectLocation': None, 'SSECustomerAlgorithm': None, 'SSECustomerKey': None, 'SSECustomerKeyMD5': None, 'SSEKMSKeyId': None, 'SSEKMSEncryptionContext': None, 'BucketKeyEnabled': None, 'RequestPayer': None, 'Tagging': None, 'ObjectLockMode': None, 'ObjectLockRetainUntilDate': None, 'ObjectLockLegalHoldStatus': None, 'ExpectedBucketOwner': None, 'Body': <_io.BufferedReader>}, headers={'Accept-Encoding': 'identity', 'User-Agent': 'Boto3/1.29.5 md/Botocore#1.32.5 md/awscrt#0.19.17 ua/2.0 os/macos#23.1.0 md/arch#arm64 lang/python#3.11.6 md/pyimpl#CPython cfg/retry-mode#standard Botocore/1.32.5', 'Content-MD5': 'CY9rzUYh03PK3k6DJie09g==', 'Expect': '100-continue', 'host': 'localhost:4566', 'X-Amz-Date': '20231204T205416Z', 'x-amz-content-sha256': '9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08', 'Authorization': 'AWS4-HMAC-SHA256 Credential=LSIAQAAAAAAVNCBMPNSG/20231204/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=d15355a87ff83d0a684cfe66fefc64a1c267e35924b8f16c1cdcb1583684c673', 'amz-sdk-invocation-id': 'cd283cef-ae22-4096-9432-ad1ef797429f', 'amz-sdk-request': 'attempt=1', 'Content-Length': '4', 'x-localstack-tgt-api': 's3', 'x-moto-account-id': '000000000042'}); PutObjectOutput({'ETag': '"098f6bcd4621d373cade4e832627b4f6"', 'ServerSideEncryption': 'AES256'}, headers={'Content-Type': 'application/xml', 'ETag': '"098f6bcd4621d373cade4e832627b4f6"', 'x-amz-server-side-encryption': 'AES256', 'x-amz-request-id': 'e9c2f0ab-f95b-4697-a870-cba798872fa7', 'x-amz-id-2': 's9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=', 'Connection': 'close'})
2023-12-04T22:54:16.083 DEBUG --- [   asgi_gw_0] l.aws.serving.wsgi         : PUT localhost:4566/test/test2
2023-12-04T22:54:16.084 DEBUG --- [   asgi_gw_0] l.aws.protocol.serializer  : Determined accept type (None) is not supported by this serializer. Using default of this serializer: application/xml
2023-12-04T22:54:16.084  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.CopyObject => 200; 000000000042/us-east-1; CopyObjectRequest({'ACL': None, 'Bucket': 'test', 'CacheControl': None, 'ChecksumAlgorithm': None, 'ContentDisposition': None, 'ContentEncoding': None, 'ContentLanguage': None, 'ContentType': None, 'CopySource': 'test/cache/test', 'CopySourceIfMatch': None, 'CopySourceIfModifiedSince': None, 'CopySourceIfNoneMatch': None, 'CopySourceIfUnmodifiedSince': None, 'Expires': None, 'GrantFullControl': None, 'GrantRead': None, 'GrantReadACP': None, 'GrantWriteACP': None, 'Key': 'test2', 'Metadata': {}, 'MetadataDirective': 'REPLACE', 'TaggingDirective': None, 'ServerSideEncryption': None, 'StorageClass': None, 'WebsiteRedirectLocation': None, 'SSECustomerAlgorithm': None, 'SSECustomerKey': None, 'SSECustomerKeyMD5': None, 'SSEKMSKeyId': None, 'SSEKMSEncryptionContext': None, 'BucketKeyEnabled': None, 'CopySourceSSECustomerAlgorithm': None, 'CopySourceSSECustomerKey': None, 'CopySourceSSECustomerKeyMD5': None, 'RequestPayer': None, 'Tagging': None, 'ObjectLockMode': None, 'ObjectLockRetainUntilDate': None, 'ObjectLockLegalHoldStatus': None, 'ExpectedBucketOwner': None, 'ExpectedSourceBucketOwner': None}, headers={'Accept-Encoding': 'identity', 'x-amz-copy-source': 'test/cache/test', 'x-amz-metadata-directive': 'REPLACE', 'User-Agent': 'Boto3/1.29.5 md/Botocore#1.32.5 md/awscrt#0.19.17 ua/2.0 os/macos#23.1.0 md/arch#arm64 lang/python#3.11.6 md/pyimpl#CPython cfg/retry-mode#standard Botocore/1.32.5', 'host': 'localhost:4566', 'X-Amz-Date': '20231204T205416Z', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=LSIAQAAAAAAVNCBMPNSG/20231204/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-metadata-directive, Signature=b5944650b52e3059cc514360f6c20191d8dbe3fcf4650925e6873e2f8db3a948', 'amz-sdk-invocation-id': 'de58ecb3-c5f8-4f58-ae0a-5b21f8a24d89', 'amz-sdk-request': 'attempt=1', 'Content-Length': '0', 'x-localstack-tgt-api': 's3', 'x-moto-account-id': '000000000042'}); CopyObjectOutput({'CopyObjectResult': {'ETag': '"098f6bcd4621d373cade4e832627b4f6"', 'LastModified': datetime.datetime(2023, 12, 4, 20, 54, 16, 84341, tzinfo=zoneinfo.ZoneInfo(key='GMT'))}, 'ServerSideEncryption': 'AES256'}, headers={'Content-Type': 'application/xml', 'x-amz-server-side-encryption': 'AES256', 'Content-Length': '177', 'x-amz-request-id': 'a6da63f7-195c-4456-a46e-6c39f6710881', 'x-amz-id-2': 's9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=', 'Connection': 'close'})
4

@bentsku thank you so much for looking into this!

I’ve been using 3.0.3.dev from localstack/localstack Docker image

Your script is working fine. But when I change bucket name to one I’ve been using originally, I can reproduce the error. I have to comment out “create_bucket” operation, obviously.

So there seems to be something wrong with my bucket. I’ll try to dig further.

s3_client = boto3.client(
    "s3",
    endpoint_url="http://localhost:4566",
    aws_access_key_id="LSIAQAAAAAAVNCBMPNSG",
    aws_secret_access_key="test"
)

# s3_client.create_bucket(Bucket="order-source-data")
s3_client.put_object(Bucket="order-source-data", Key="cache/foo", Body="bar")
s3_client.copy_object(Bucket="order-source-data", Key="foo2", CopySource="order-source-data/cache/foo", MetadataDirective="REPLACE")

integration-localstack-1  | 2023-12-05T12:56:19.636 DEBUG --- [   asgi_gw_0] l.aws.serving.wsgi         : PUT localhost:4566/order-source-data/cache/foo
integration-localstack-1  | 2023-12-05T12:56:19.640 DEBUG --- [   asgi_gw_0] l.aws.protocol.serializer  : Determined accept type (None) is not supported by this serializer. Using default of this serializer: application/xml
integration-localstack-1  | 2023-12-05T12:56:19.641  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.PutObject => 200; 000000000042/us-east-1; PutObjectRequest({'ACL': None, 'Bucket': 'order-source-data', 'CacheControl': None, 'ContentDisposition': None, 'ContentEncoding': None, 'ContentLanguage': None, 'ContentLength': 3, 'ContentMD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'ContentType': None, 'ChecksumAlgorithm': None, 'ChecksumCRC32': None, 'ChecksumCRC32C': None, 'ChecksumSHA1': None, 'ChecksumSHA256': None, 'Expires': None, 'GrantFullControl': None, 'GrantRead': None, 'GrantReadACP': None, 'GrantWriteACP': None, 'Key': 'cache/foo', 'Metadata': {}, 'ServerSideEncryption': None, 'StorageClass': None, 'WebsiteRedirectLocation': None, 'SSECustomerAlgorithm': None, 'SSECustomerKey': None, 'SSECustomerKeyMD5': None, 'SSEKMSKeyId': None, 'SSEKMSEncryptionContext': None, 'BucketKeyEnabled': None, 'RequestPayer': None, 'Tagging': None, 'ObjectLockMode': None, 'ObjectLockRetainUntilDate': None, 'ObjectLockLegalHoldStatus': None, 'ExpectedBucketOwner': None, 'Body': <_io.BufferedReader>}, headers={'Host': 'localhost:4566', 'Accept-Encoding': 'identity', 'User-Agent': 'Boto3/1.33.7 md/Botocore#1.33.7 ua/2.0 os/macos#22.6.0 md/arch#arm64 lang/python#3.11.6 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.33.7', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Expect': '100-continue', 'X-Amz-Date': '20231205T125619Z', 'X-Amz-Content-SHA256': 'fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9', 'Authorization': 'AWS4-HMAC-SHA256 Credential=LSIAQAAAAAAVNCBMPNSG/20231205/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=0d1b7ff81c3f26d3eff2cd39ec973866082547032807c01cf1cdf4691cad1781', 'amz-sdk-invocation-id': '2f4c379d-e7e0-482b-95a0-a8058f2265e8', 'amz-sdk-request': 'attempt=1', 'Content-Length': '3', 'x-localstack-tgt-api': 's3', 'x-moto-account-id': '000000000042'}); PutObjectOutput({'ETag': '"37b51d194a7513e45b56f6524f2d51f2"', 'ServerSideEncryption': 'AES256'}, headers={'Content-Type': 'application/xml', 'ETag': '"37b51d194a7513e45b56f6524f2d51f2"', 'x-amz-server-side-encryption': 'AES256', 'x-amz-request-id': '0686d6ed-b14c-4d89-9569-e2ae606a2895', 'x-amz-id-2': 's9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=', 'Connection': 'close'})
integration-localstack-1  | 2023-12-05T12:56:19.649 DEBUG --- [   asgi_gw_1] l.aws.serving.wsgi         : PUT localhost:4566/order-source-data/foo2
integration-localstack-1  | 2023-12-05T12:56:19.651 DEBUG --- [   asgi_gw_1] l.aws.protocol.serializer  : Determined accept type (None) is not supported by this serializer. Using default of this serializer: application/xml
integration-localstack-1  | 2023-12-05T12:56:19.652  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.CopyObject => 404 (NoSuchBucket); 000000000042/us-east-1; CopyObjectRequest({'ACL': None, 'Bucket': 'order-source-data', 'CacheControl': None, 'ChecksumAlgorithm': None, 'ContentDisposition': None, 'ContentEncoding': None, 'ContentLanguage': None, 'ContentType': None, 'CopySource': 'order-source-data/cache/foo', 'CopySourceIfMatch': None, 'CopySourceIfModifiedSince': None, 'CopySourceIfNoneMatch': None, 'CopySourceIfUnmodifiedSince': None, 'Expires': None, 'GrantFullControl': None, 'GrantRead': None, 'GrantReadACP': None, 'GrantWriteACP': None, 'Key': 'foo2', 'Metadata': {}, 'MetadataDirective': 'REPLACE', 'TaggingDirective': None, 'ServerSideEncryption': None, 'StorageClass': None, 'WebsiteRedirectLocation': None, 'SSECustomerAlgorithm': None, 'SSECustomerKey': None, 'SSECustomerKeyMD5': None, 'SSEKMSKeyId': None, 'SSEKMSEncryptionContext': None, 'BucketKeyEnabled': None, 'CopySourceSSECustomerAlgorithm': None, 'CopySourceSSECustomerKey': None, 'CopySourceSSECustomerKeyMD5': None, 'RequestPayer': None, 'Tagging': None, 'ObjectLockMode': None, 'ObjectLockRetainUntilDate': None, 'ObjectLockLegalHoldStatus': None, 'ExpectedBucketOwner': None, 'ExpectedSourceBucketOwner': None}, headers={'Host': 'localhost:4566', 'Accept-Encoding': 'identity', 'x-amz-copy-source': 'order-source-data/cache/foo', 'x-amz-metadata-directive': 'REPLACE', 'User-Agent': 'Boto3/1.33.7 md/Botocore#1.33.7 ua/2.0 os/macos#22.6.0 md/arch#arm64 lang/python#3.11.6 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.33.7', 'X-Amz-Date': '20231205T125619Z', 'X-Amz-Content-SHA256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=LSIAQAAAAAAVNCBMPNSG/20231205/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-metadata-directive, Signature=14d18befad3d7e6cf6698edff9ec901b5aaa29be37227ea2f83081ec831ee10f', 'amz-sdk-invocation-id': '765338b2-487d-431b-9f5d-2ee812837c50', 'amz-sdk-request': 'attempt=1', 'Content-Length': '0', 'x-localstack-tgt-api': 's3', 'x-moto-account-id': '000000000042'}); NoSuchBucket(The specified bucket does not exist, headers={'Content-Type': 'application/xml', 'Content-Length': '234', 'x-amz-request-id': '0e98bb3a-53ae-4207-af8d-0a7d145dc26e', 'x-amz-id-2': 's9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=', 'Connection': 'close'})
5

Ah, maybe the bucket is not in the same account in that case, I have a TODO in the code to verify that part. Maybe if the bucket doesn’t live in the same account, we have some issues. I think I could fix this, it’s just a bit harder to test against AWS. I’ll link the PR fixing this here!

6

That was it!

I had created that bucket with:

docker-compose exec localstack awslocal s3 mb s3://order-source-data

The error was fixed when I re-created it with correct credentials:

docker-compose exec --env=AWS_ACCESS_KEY_ID=LSIAQAAAAAAVNCBMPNSG,AWS_SECRET_ACCESS_KEY=test localstack awslocal s3 mb s3://order-source-data

Thank you very much @bentsku

7

I’m glad it solved your issue! I still think CopyObject should maybe work between different buckets, but at least it unblocks you for now. Thanks again for sharing this!

1 Like
8

I’ve created a PR here which would solve the issue in any case: add cross-account for CopyObject by bentsku · Pull Request #9806 · localstack/localstack · GitHub

Still, I believe creating the bucket with the same credentials is the better solution here. Thanks again!