Path4.Mod3.d - Perform Hyperparameter Tuning - Sweep Job Implementation

Question 1

cmd mlf

Two things required by your training code/script before using it in a Sweep Job

Answer 1

1. Include a command line argument for each hyperparameter you want to try
2. Log the metric with MLFlow; the Sweep Job needs it to evaluate the performance of each trial and identify the best performing model

The code below includes a hyperparameter reg_rate and logs a metric “Accuracy”:

// train.py code
// Commandline arg "--regularization" gets value
// put into 'reg_rate' input
parser = argparse.ArgumentParser()
parser.add_argument('--regularization', type=float, dest='reg_rate', default=0.01)
args = parser.parse_args()
reg = args.reg_rate

// Training code that uses your 
// Hyperparameter value 'reg_rate'

// Log your target metric
acc = np.average(y_hat == y_test)
mlflow.log_metric("Accuracy", acc)

Question 2

To create a Sweep Job, instantiate a instance of this class

Answer 2

This is not an instance of CommandJob

An instance of command

The code below shows how to create one:
- The command line to execute your script “train.py” script along with its commandline Hyperparameters + input values taken from the input json.
- Note that the inputs field will create those input parameters dynamically, making them dynamically available to use as input to the command parameter:

from azure.ai.ml import command

configure command job as base
cmd_instance = command(
    code="./src",
    command="python train.py --regularization ${{inputs.reg_rate}}",
    inputs={
        "reg_rate": 0.01,
    },
    environment="AzureML-sklearn-0.24-ubuntu18.04-py37-cpu@latest",
    compute="aml-cluster"
)

Question 3

Override a command instance’s input to use a Search Space

Answer 3

To override the input with your Search Space, set the hyperparameter input directly through the command instance. Remember that input creates those key:value pairs dynamically, ergo they are dynamically accessible. You can access them through the command instance by its reference as a function that will return an instance of a CommandJob

Below, the command’s reg_rate input is set to a discrete search space via the Choice function:

from azure.ai.ml.sweep import Choice

// access hyperparameter directly through 
// the command instance
command_job_for_sweep = cmd_instance(
    reg_rate=Choice(values=[0.01, 0.1, 1]),
)

See The Command Entity

Question 4

Python code to create a Sweep Job from your Job
- Initialize the Sweep Job
- Set the Experiment Name
- Set limits for your Sweep Job
- Submit the job

Answer 4

Use sweep method to get an instance

from azure.ai.ml import MLClient

// Get an instance of the actual sweep_job
sweep_job = cmd_job.sweep(
    compute="aml-cluster",
    sampling_algorithm="grid",
    primary_metric="Accuracy",
    goal="Maximize",
)

// The sweep function doesn't set this
// So set the name of the sweep job experiment
sweep_job.experiment_name="sweep-example"

// Define the limits for the sweep
sweep_job.set_limits(
    max_total_trials=4, 
		max_concurrent_trials=2, 
		timeout=7200)

 # Submit the sweep
returned_sweep_job = ml_client.create_or_update(sweep_job)

Question 5

The chart that shows all metrics logged for each hyperdrive child job over the duration of hyperparam tuning, with each line being a child job and each point measuring the primary metric value.

Answer 5

Metrics Chart

Question 6

The chart that shows the correlation between primary metric performance and individual Hyperparameters.

Answer 6

Parallel Coordinates Chart. The chart itself is Interactive, including an axis that plots the best metric value corresponding to hyperparams set for that job instance.

Question 7

The chart that shows the correlation between primary metric performance and any TWO Hyperparameters

Answer 7

2-Dimensional Scatter Chart

Question 8

The chart that shows the correlation between primary metric performance and any THREE Hyperparameters.

Answer 8

3-Dimensional Scatter Chart. You can also reorient along a specified axis.

Question 9

• Python code to download the best trial output
• CLI code to get ALL outputs
• CLI code to get the best trial output

Answer 9

ml_client.jobs.download(returned_sweep_job.name, output_name="model_name")

az ml job download --name <sweep-job-name> --all

az ml job download --name <sweep-job-name> --output-name model_name