DynamicPartition not working on firehose

Platform
1

Hello there,

I have been using firehose and S3 for straging application log.
I am trying to replicate it in my local environment using localstack.
but, not working DynamicPartition.

Environment
docker image: localstack/localstack:1.2.0

AWSLOCAL command

awslocal s3api create-bucket \
  --bucket logs-bucket-000000000000 \
  --region $REGION \
  --create-bucket-configuration LocationConstraint=$REGION

awslocal firehose create-delivery-stream \
    --region $REGION \
    --delivery-stream-name log-stream-000000000000 \
    --delivery-stream-type DirectPut \
    --extended-s3-destination-configuration file:///docker-entrypoint-initaws.d/firehose_config.json```

**firehose_config.json**

{
“RoleARN”: “arn:aws:iam::000000000000:role/name”,
“BucketARN”: “arn:aws:s3:::logs-bucket-000000000000”,
“Prefix”: “log/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/!{partitionKeyFromQuery:accountId}/!{partitionKeyFromQuery:sampleId}/”,
“ErrorOutputPrefix”: “errors/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/!{firehose:error-output-type}/”,
“BufferingHints”: {
“SizeInMBs”: 64
},
“CompressionFormat”: “UNCOMPRESSED”,
“ProcessingConfiguration”: {
“Enabled”: true,
“Processors”: [
{
“Type”: “RecordDeAggregation”,
“Parameters”: [
{
“ParameterName”: “SubRecordType”,
“ParameterValue”: “JSON”
}
]
},
{
“Type”: “AppendDelimiterToRecord”
},
{
“Type”: “MetadataExtraction”,
“Parameters” : [
{
“ParameterName”: “JsonParsingEngine”,
“ParameterValue”: “JQ-1.6”
},
{
“ParameterName”: “MetadataExtractionQuery”,
“ParameterValue”: “{sampleId:.tags.sampleId, accountId:.tags.accountId}”
}
]
}
]
},
“DynamicPartitioningConfiguration”: {
“Enabled”: true
}
}


**Current Behavior**
awslocal s3 ls s3://logs-bucket-000000000000 --recursive

log/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/!{partitionKeyFromQuery:accountId}/!{partitionKeyFromQuery:sampleId}/2022/11/24/12/log-stream-000000000000-2022-11-24-12-07-05-07d85038-10ee-48f4-860c-52ce16d73a2b```

Expected Behavior
awslocal s3 ls s3://logs-bucket-000000000000 --recursive

log/year=2022/month=11/day=24/hour=12/1234567/891011213/log-stream-000000000000-2022-11-24-12-07-05-07d85038-10ee-48f4-860c-52ce16d73a2b```

I may have the wrong setup, but could you please give me a solution?

Thanks in advance.

Best,
ryoqn
1 Like
2

I’m having the same problem, did you ever solve this?