Understand penalties and tradeoffs

A penalty encodes a preference

A regularizer adds a cost for a property such as large parameters. In the teaching objective, a squared parameter penalty competes with fitting the target three. Increasing its nonnegative weight pulls the preferred solution toward zero.

The resulting objective is a different optimization problem from pure target fitting. A higher fitting error may be accepted because the combined cost is lower. Record the component costs so that a better total cannot hide an important tradeoff.

Signs depend on minimization or maximization

The project’s autonomous objective adds exploration and entropy terms while subtracting conflict, resource, and repetition penalties. Under maximization, a positive penalty is subtracted to discourage the penalized behavior.

For a minimized loss, an unwanted positive cost is usually added. A term called entropy regularization can encourage or discourage entropy depending on its sign and whether the objective is minimized or maximized. The label alone does not decide the effect.

The gradient-descent lab elsewhere in the learning studio uses the unregularized reference objective. Its loss and update rule do not include the regularization term in this lesson. Use the worked calculation here when checking the regularized optimum.