You work as a machine learning specialist for an audio processing and distribution company. You are currently working on a custom audio recommendation model for a criminal investigation application that recommends which audio file to use based on investigation details. The dataset you are attempting to use to train the model is extremely large, containing millions of data points. You are storing the dataset in an S3 bucket. You need to find an alternative to loading all of the data into a SageMaker notebook instance because it would take too long to load and exceed the 50 GB EBS volume attached to the notebook instance. Which approach should you select so that you can load all the data to train the model?

Answer options:

A.Split the training dataset using scikit-learn or pandas to create a subset of your training data. Load the subset of the training data into the SageMaker notebook and train the model in your notebook instance. Verify that the model trained accurately and that the model parameters produce reasonable results. Use a Deep Learning AMI to start an EC2 instance and attach the S3 bucket to train the full dataset.
B.Split the training dataset using scikit-learn or pandas to create a subset of your training data. Use Glue to load your data into your SageMaker notebook, using your subset of the training data to verify that the model trained accurately and that the model parameters produce reasonable results. Next, run a training job using the entire dataset from the S3 bucket using Pipe input mode.
C.Use a Deep Learning AMI to start an EC2 instance and attach the S3 bucket. Split the training dataset using scikit-learn or pandas to create a subset of your training data. Train using the subset of the training data to verify the training code and hyperparameters. Use SageMaker to train using the full dataset.
D.Split the training dataset using scikit-learn or pandas to create a subset of your training data. Load the subset of the training data into the SageMaker notebook and train in your notebook. Verify that the model trained accurately and that the model parameters produce reasonable results. Run a SageMaker training job loading the complete dataset from the S3 bucket using Pipe input mode.

Correct Answer: D
Option A is incorrect. Using a Deep Learning AMI for your EC2 instance will not help with loading the extremely large training dataset from S3.
Option B is incorrect. Glue cannot be used to load data into your SageMaker notebook to train a machine learning model. Glue could be used to place your data onto an S3 bucket that you could then use to load the data into your SageMaker notebook.
Option C is incorrect. This approach does not address the loading of the extremely large dataset onto the local storage of the EC2 instance.
Option D is correct. With this option, we are using the pipe input mode to stream our training data directly to our training instance instead of downloading it to our EBS storage first. This solves the problem of loading extremely large data into our notebook instance.
References:
Please see the AWS Machine Learning blog titled Using Pipe input mode for Amazon SageMaker algorithms (https://aws.amazon.com/blogs/machine-learning/using-pipe-input-mode-for-amazon-sagemaker-algorithms/)
The AWS Machine Learning blog titled Accelerate model training using faster Pipe mode on Amazon SageMaker (https://aws.amazon.com/blogs/machine-learning/accelerate-model-training-using-faster-pipe-mode-on-amazon-sagemaker/),
The AWS announcement titled Amazon SageMaker Now Supports an Improved Pipe Mode Implementation (https://aws.amazon.com/about-aws/whats-new/2018/10/amazon-sagemaker-now-supports-an-improved-pipe-mode-implementati/),
The Amazon Amazon SageMaker developer guide titled Use Scikit-learn with Amazon SageMaker (https://docs.aws.amazon.com/sagemaker/latest/dg/sklearn.html)