Path4.Mod3.c - Perform Hyperparameter Tuning - Sweep Job Early Termination

Question 1

ei de

Two common parameters for configuring an Early Termination Policy

Answer 1

• evaluation_interval - How many trials per evaluation ie 1 means eval on every trial
• delay_evaluation - How many completed trials to wait before we start evaluating for ET policy (allows training a few Models in order to get some performance numbers)

Question 2

BP MSP TSP

Three policies for configuring an Early Termination Policy

Answer 2

• Bandit Policy
• Median Stopping Policy
• Truncation Selection Policy

Question 3

Bandit Policy
- Explain slack_factor vs slack_amount
- Explain lower vs higher factors

Answer 3

Stop a trial if the target performance metric underperforms the best trial so far by a specified margin, defined by either a slack_factor or a slack_amount

• slack_amount: the next score underperforms by this specific amount
• slack_factor: the next score underperforms by (highest / (1+slack_factor)). ex. if slack_factor = 0.1 and highest = 1.0 then ET happens when the next trial score is less than (1/1.1) = 0.91
• higher factors == more lenient, vs lower factors == steeper thresholds. Ex. 0.12 comes out to 0.89, so there’s some wiggle room for a trial.

Question 4

Explain Bandit Policy code:

from azure.ai.ml.sweep import BanditPolicy
sweepjob.early_termination = BanditPolicy(
    slack_amount = 0.2,
    delay_evaluation = 5, 
    evaluation_interval = 1
)

Answer 4

• We delay evaluation for 5 trials, so we let 5 go and we start evaluating at the 6th trial
• We evaluate at each (1) interval, so after the 5th internval we evaluate each one going forward
• We stop the Sweep Job when a new trial’s target metric is 0.2 (or 20%) less than the best performing model thus far. W.r.t. the chart below, since the best was 0.9, the first chart is still good at 0.85, but the second chart shows a value below the slack threshold at 0.65, so we stop the sweep job.

Question 5

Median Stopping Policy

Answer 5

Stop a trial if the target performance metric is worse than the median or average of all trials

Question 6

Explain Median Stopping Policy code:

from azure.ai.ml.sweep import MedianStoppingPolicy
sweepjob.early_termination = MedianStoppingPolicy(
    delay_evaluation = 5, 
    evaluation_interval = 1
)

Answer 6

• We delay evaluation for 5 trials, so we let 5 go and we start evaluating at the 6th trial
• We evaluate at each (1) interval, so after the 5th internval we evaluating each one going forward
• We stop the Sweep Job when a new trial’s target metric comes in below the average metric value across all previous trials, denoted by the dashed line.

Question 7

Truncation Selection Policy

Answer 7

Cancel the lowest performing % of trials at each evaluation interval based on the truncation_percentage parameter

Question 8

Explain Truncation Selection Policy code:

from azure.ai.ml.sweep import TruncationSelectionPolicy
sweepjob.early_termination = TruncationSelectionPolicy(
    delay_evaluation = 5,
    truncation_percentage = 20,
    evaluation_interval = 1
)

Answer 8

• We delay evaluation for 5 trials, so we let 5 go and we start evaluating at the 6th trial
• We evaluate at each (1) interval, so after the 5th internval we evaluating each one going forward
• We stop the Sweep Job when a new trial’s target metric comes in at the lowest 20% of all trials so far. In other words, if the trial isn’t in the lowest percentile of trials, the job continues.