You work as a machine learning specialist for a mobile network operator who builds an analytics platform to analyze and optimize its operations by leveraging machine learning.
You receive your data from source systems that send data in CSV format in real-time. You require to transform the data to the parquet format before storing it on S3. From there, you plan to use the data in SageMaker AutoPilot to help you find the best machine learning pipeline for your analytics problem.
Which option solves your data analysis machine learning problem in the most efficient manner?

Answer options:

A.Ingest the CSV data using MSK running on EC2 instances and use Kafka Connect S3 to convert the data to the parquet format.
B.Ingest the CSV data using Kinesis Data Streams. Convert the data to the parquet format using Glue.
C.Ingest the CSV data using Spark Structured Streaming in an EMR cluster. Convert data to the parquet format using Spark.
D.Ingest the CSV data using Kinesis Data Streams and use Kinesis Data Firehose, leveraging a Lambda function to transform the data from CSV to JSON, then convert the data to the parquet format.

Answer: D
Option A is incorrect. Implementing the solution using MSK on EC2 instances requires more work than the other options.
Option B is incorrect. Unless you are using Glue streaming ETL, which is not explicitly stated in the question, you should not use Glue on streaming data.
Option C is incorrect. This option also requires more work (spinning up an EMR cluster) than simply using Kinesis Data Streams, Kinesis Data Firehose, and Lambda (all managed services).
Option D is CORRECT. You can ingest the streaming CSV data using Kinesis Data Streams, a managed service. Then use Kinesis Data Firehose and Lambda (also both managed services) to first convert the data from CSV to JSON (Kinesis Data Firehose transformation requires that the data be in the JSON format) then use the Kinesis Data Firehose parquet transformation to convert the data to parquet.
Reference:
Please see the AWS blog titled Stream Real-Time Data in Apache Parquet or ORC Format Using Amazon Kinesis Data Firehose.
Please refer to the Kinesis Data Firehose developer guide titled Converting Your Input Record Format in Kinesis Data Firehose.