A big company has a service to process gigantic clickstream data sets which are often the result of holiday shopping traffic on a retail website, or sudden dramatic growth on the data network of a media or social networking site. It is becoming more and more complex to analyze these clickstream datasets for its on-premise infrastructure. As the sample data set keeps growing, fewer applications are available to provide a timely response. The service is using a Hadoop cluster with Cascading. How can they migrate the applications to AWS in the best way? 

Answer options:

A.Put the source data to S3 and migrate the processing service to an AWS EMR Hadoop cluster with Cascading. Enable EMR to read and query data from S3 buckets directly. Write the output to the RDS database.
B.Put the source data to a Kinesis stream and migrate the processing service to AWS lambda to utilize its scaling feature. Enable lambda to read and query data from the Kinesis stream directly. Write the output to the RDS database.
C.Put the source data to an S3 bucket and migrate the processing service to AWS EC2 with auto-scaling. Ensure that the auto-scaling configuration has a proper maximum and minimum number of instances. Monitor the performance in the Cloudwatch dashboard. Write the output to the DynamoDB table for downstream to process.
D.Put the source data to a Kinesis stream and migrate the processing service to an AWS EMR cluster with Cascading. Enable EMR to read and query data from Kinesis streams directly. Write the output to Redshift.

Correct Answer – D
 
The application needs to process data timely; therefore Kinesis stream should be considered first. After a click event happens, a message should be put into the Kinesis stream in real-time. Moreover, Cascading is a proven, highly extensible application development framework for building massively parallelized data applications on EMR. By using EMR, the application does not need to change a lot for the migration. Refer to https://aws.amazon.com/blogs/big-data/integrating-amazon-kinesis-amazon-s3-and-amazon-redshift-with-cascading-on-amazon-emr/.
Option A is incorrect: Because the RDS database`s output is improper as RDS does not scale well when the traffic is high. Redshift is much more appropriate.
Option B is incorrect: AWS Lambda can potentially work for Hadoop. However, EMR provides native support for Hadoop. Also, RDS is incorrect.
Option C is incorrect: Because EC2 is not suitable for Hadoop processing if compared with EMR. This question asks for the best option. So EMR should be chosen.
Option D is CORRECT: Because the EMR cluster with Cascading can process the data from Kinesis stream in real-time, and Redshift is also a proper place to store the output data.