First simulate data where \(x_1\) to \(x_3\) are identical and \(x_4\) is uncorrelated with them. Each \(x\) is a z-score.
Set \(y = 0.2 (x_1 + x_2 + x_3) + 0.3 x_4 + \epsilon\).
Now try elastic nets, varying \(\alpha\) from 0 (ridge regression) to 1 (lasso). I used 10‑fold cross‑validation to select \(\lambda\), keeping the coefficients for the smallest \(\lambda\), and used the same fold assignments across all values of \(\alpha\).
Here’s a picture of the coefficients. Note how the correlated predictor slopes separate as \(\alpha\) increases until only one survives with a slope of about three times 0.2 (to compensate for the other missing identical predictors). The uncorrelated predictor slope stays the same.
Here’s a table of coefficients for a selection of \(\alpha\), including the results from an unpenalised regression with only \(x_1\) and \(x_4\) as predictors:
| Variable | α = 0 | α = 0.9 | α = 1 | Unpenalised regression |
|---|---|---|---|---|
| (Intercept) | 0.002 | 0.002 | 0.002 | 0.002 |
| x1 | 0.198 | 0.243 | 0.600 | 0.603 |
| x2 | 0.196 | 0.188 | ||
| x3 | 0.195 | 0.169 | ||
| x4 | 0.271 | 0.287 | 0.287 | 0.290 |