On LocalStack, can AWS Data Firehose convert a input JSON stream to parquet, and sink the files to S3?

Platform
1

I’m using terraform to create a Firehose Delivery Stream to S3. I can successfully use boto3’s firehose client and call the put_batch method to push records to S3 files.

Apparently, Firehose can turn these JSON to Parquet, so long as the schema is defined in Glue:

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-dataformatconversionconfiguration.html

I can’t get any Parquet files though. My terraform is successfully planned and applied (using tflocal), and there are no apparent errors in the LocalStack logs.

Does LocalStack’s implementation of Firehose allow for Data Format Conversion, specifically from JSON to Parquet?

2

This option is not currently available, but we would be happy to include it on our roadmap. Please submit a feature request on our GitHub page with a brief description of your use case. That would be greatly appreciated. Thanks.

3

alright, i’ll do that, thanks!