(step_functions): S3 Json Lines Item Reader

[MFC-MiguelFerreira]

Describe the feature

The AWS Step Functions team recently introduced support for JSON Lines (JSONL) in Distributed Map, allowing efficient processing of large datasets stored in this format:
🔗 AWS Blog Post – JSONL Support in Step Functions Distributed Map

Currently, the AWS CDK provides S3JsonItemReader (docs), which supports reading JSON objects from an S3 file. However, this construct does not support JSONL files. Given that JSONL is now natively supported by Step Functions Distributed Map, it would be highly beneficial to have native support for JSONL in the CDK as well.

Use Case

Developers using AWS Step Functions with CDK would be able to seamlessly leverage JSONL for large-scale data processing, without resorting to custom implementations or workarounds.

Proposed Solution

Introduce a new construct (or extend the existing S3JsonItemReader) to support JSONL files, aligning with the latest Step Functions capabilities.

Example:

s3_jsonl_reader = stepfunctions.S3JsonItemReader(
    bucket=s3_bucket,
    key="data.jsonl",
    format=stepfunctions.JsonFormat.JSONL  # Example of possible new parameter
)

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.280.0

Environment details (OS name and version, etc.)

Windows 11, python

[pahud]

Hi @MFC-MiguelFerreira,

Thank you for submitting this feature request! We appreciate you bringing this new Step Functions capability to our attention.

This is a valid feature request that aligns well with the recent AWS service updates. The CDK should indeed provide a way to leverage JSONL support in Distributed Map for large-scale data processing.

Your proposed solution of extending the S3JsonItemReader with a format parameter is a good approach.

I'm making this a p2 feature request. If you or anyone from the community is interested in contributing this feature, we would welcome a pull request.

Thanks again for helping improve the AWS CDK!

[bhowmick-sumit]

I would love to work on this feature.
I have done my initial analysis on this issue and found 3 ways to solve this issue.

1. creating a new construct S3JsonlItemReader which will support JSONL file.
2. modify the existing S3JsonItemReader which can support JSONL/JSON file by adding flexibility to set inputType which is present in S3JsonItemReader
3. Creating a new construct that will support all three files support - JSON, JSONL and MANIFEST.

@pahud can you please assign this task to me?

@MFC-MiguelFerreira for a temporary solution you can use the following code.

class S3JsonlItemReader extends S3JsonItemReader {
  inputType: string = 'JSONL';
}

you can use the S3JsonlItemReader as the itemReader for DistributedMap.

[vishalsatam]

Extending S3JsonItemReader is not a good idea. It violates the Single responsibility principle.

https://en.wikipedia.org/wiki/SOLID

Let's extend the S3FileItemReader class instead similar to the existing S3JsonItemReader class. It's straightforward to do it that way. I might be able to add it to my pull request.

export class S3JsonlItemReader extends S3FileItemReader {
  protected readonly inputType: string = 'JSONL';
}

Example: https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/aws-stepfunctions/lib/states/distributed-map/item-reader.ts#L323-L328

[bhowmick-sumit]

Hi @vishalsatam ,
I was thinking of Creating a new construct that will support all three files - JSON, JSONL and MANIFEST.
what's your opinion on this approach?

[vishalsatam]

If we create a new construct to support all 3 types, then it will become backwards incompatible and existing users will need to update their code. Additionally, if StepFunctions updates one of the types to include some additional configuration in future, like they have for the CSV type, then we will need to do some more refactoring.

I created a pull request to include the new class S3JsonlItemReader I mentioned in my previous comment.
#33831