You work for a startup shirt manufacturer that has come up with a new manufacturing process for shirts that is very stylish and has become very popular since your company ran an online Kickstarter fundraiser and shipped its first line of shirts. You now want to use machine learning to classify your shirt styles as either conservative or not based on customer feedback on your website. This classification information will help your designers target new designs based on the customer perception of your current offerings.
You have gathered your data from your website comments and ratings. You have also performed feature engineering of your data. You are now ready to run several model tuning jobs, as many as needed, even if you have to run hundreds of them, to find the best version of your XGBoost model. You plan to do this by running many hyperparameter tuning jobs that test the range of hyperparameters you have available to you. Since you have decided on using a binary classifier algorithm and based on the business problem you are trying to solve, you have decided you need to measure the success of a hyperparameter tuning job based on precision and recall.
Which XGBoost metric is the best objective on which to evaluate your model?

Answer options:

A.accuracy
B.error
C.F1
D.MAE (Mean Absolute Error)
E.MAP (Mean Average Precision)
F.merror

Answer: C
Option A is incorrect. The accuracy metric only measures (right cases)/(all cases), which doesn’t give you precision or recall, which are the two metrics you wish to use to evaluate your model.
Option B is incorrect. The error metric only measures (wrong cases)/(all cases), which doesn’t give you precision or recall, which are the two metrics you wish to use to evaluate your model.
Option C is correct. The f1 metric combines precision and recall into one metric. It represents the harmonic mean of precision and recall. Its formula: 2*precision*recall/(precision+recall).
Option D is incorrect. The MAE metric finds the absolute value of the error between the predicted and target values. This doesn’t give you precision or recall, which are the two metrics you wish to use to evaluate your model.
Option E is incorrect. The map metric finds the mean average precision. This doesn’t give you recall, which is one of the two metrics you wish to use to evaluate your model.
Option F is incorrect. The error metric is a multiclass classification error rate, which is represented as (wrong cases)/(all cases). This doesn’t give you recall, which is one of the two metrics you wish to use to evaluate your model. Also, this metric is used for multiclass classification problems. You are trying to solve a binary classification problem.
Reference:
Please see the Amazon SageMaker developer guide titled Tune an XGBoost Model, the XGBoost docs page titled XGBoost Parameters, and the article titled 20 Popular Machine Learning Metrics. Part 1: Classification & Regression Evaluation Metrics.