The performance package computes indices of model quality and goodness of fit, including \(R^2\) (including for LMMs), RMSE, ICCs, and tests for overdispersion, zero-inflation, convergence, and singularity. It also simplifies posterior predictive checks for every model with a simulate function.
Author: Andi
Evaluating agentic AI
An extreme case of agentic AI gone wrong, but the recent AISI incident report is worth a read before you let complex agents loose on your personal or organisation’s files, allow them to reply to emails autonomously or go wild on the internet.
Note: it happened during an evaluation of frontier AI models. Internet access was deliberately enabled and filters that block harmful behaviour were deliberately switched off for the evaluation. The affected models are not publicly available.
In 10 runs out of 122, an agent carried out “potentially harmful” actions. There were 19 such actions, 17 from Anthropic’s Mythos 5 and two from OpenAI’s GPT-5.6-Sol. The most serious was when an agent tried to insert malicious code into an open-source project hosted on GitHub. It created fake GitHub accounts (via Tor) to try to persuade the project’s maintainer to approve the change; however, they spotted that the code was dodgy and blocked it.
The aftermath:
“we notified GitHub […] of the agents’ malicious activity, which included actions that GitHub has confirmed violated their terms of service. We worked together with GitHub to remove artefacts left behind by the agent, and to notify the GitHub users the model interacted with. We have also contacted other affected parties.”
Ciaran Martin has written a considered post on the incident, but the clearest summary of important mitigations to prevent this happening again comes from his quotation of Daniel Card:
“if your AI starts hacking s**t… if you monitor what it’s doing you can, you know, turn the f***ing power off”
AISI are now implementing this approach. However, he notes, “I suspect the main regret at AISI is that they didn’t make this change immediately after the OpenAI/Hugging Face disclosure – this test was carried out the week after. Had they done so, the incident disclosed today would not have happened.” Indeed.
I think Martin is right to point out that AI‑related incidents are receiving a disproportionate amount of media attention compared with, for example, Iran hacking into US water systems in at least one and maybe seven states – intrusions that caused problems ranging from low pressure to some areas advising people to boil tap water before use.
The reason for the focus on AI may be that organisations are deploying it at pace, and calamities like AISI’s are vivid (though extreme) illustrations of the kinds of harm AI can cause. It’s interesting to compare and contrast how careful you would be, e.g., running a mail merge or code that writes or deletes files (using an innocent for-loop, perhaps) with how cautious you feel about offering your precious data and your email’s send button to a stochastic AI agent.
Airspace violations by Russia
There’s a Wikipedia page recording violations of non-combatant airspace during the Russo-Ukrainian war. Usual caveats apply with Wikipedia – check the citations.
Two recent notable examples:
| Date | Event |
|---|---|
| 29 May 2026 | A Russian Geran 2 drone entered Romanian airspace and hit the 10th floor of block of flats in Galați, exploding on impact. Two people were injured and about 70 evacuated as the resulting fire was put out. Romania’s president said, “There was a group of 43 drones coming from the east. Some were shot down over Ukraine, and one was hit above the [Ukrainian] city of Reni, which altered its trajectory.” (BBC News) |
| 30 July 2026 | A Russian missile (likely a Kh-101 cruise missile) crossed the border into Poland and crashed into a field close to the village of Tarnawa-Kolonia, 57 miles from the border with Ukraine. The missile left a 10 m-wide crater in the field. (BBC News) |
The International Institute of Strategic Studies (IISS) has argued (July 2026) that it is “highly likely” Russia conducted a UAV campaign over Europe between August 2024 and February 2026, and assessed it as “likely” that Russian-linked commercial vessels, including shadow-fleet tankers, coastal freighters, and smaller craft, were used as launch and recovery platforms. (See also the July 2026 Guardian article.)
Update: Leipzig drone bomb (5 Aug 2026).
Trying elastic net regularisation with correlated predictors
First simulate data where \(x_1\) to \(x_3\) are identical and \(x_4\) is uncorrelated with them. Each \(x\) is a z-score.
Set \(y = 0.2 (x_1 + x_2 + x_3) + 0.3 x_4 + \epsilon\).
Now try elastic nets, varying \(\alpha\) from 0 (ridge regression) to 1 (lasso). I used 10‑fold cross‑validation to select \(\lambda\), keeping the coefficients for the smallest \(\lambda\), and used the same fold assignments across all values of \(\alpha\).
Here’s a picture of the coefficients. Note how the correlated predictor slopes separate as \(\alpha\) increases until only one survives with a slope of about three times 0.2 (to compensate for the other missing identical predictors). The uncorrelated predictor slope stays the same.
Here’s a table of coefficients for a selection of \(\alpha\), including the results from an unpenalised regression with only \(x_1\) and \(x_4\) as predictors:
| Variable | α = 0 | α = 0.9 | α = 1 | Unpenalised regression |
|---|---|---|---|---|
| (Intercept) | 0.002 | 0.002 | 0.002 | 0.002 |
| x1 | 0.198 | 0.243 | 0.600 | 0.603 |
| x2 | 0.196 | 0.188 | ||
| x3 | 0.195 | 0.169 | ||
| x4 | 0.271 | 0.287 | 0.287 | 0.290 |
Counterfactual thinking – common and useful
Counterfactual isn’t a synonym for control group. Here’s a brief Open Encyclopedia of Cognitive Science entry on counterfactual thinking, which argues that it is “extremely common and profoundly useful in ordinary life.” Whether or not you believe that your impact evaluation is counterfactual, there’s a good chance that your participants are using counterfactual thinking to answer your questions.
Brigard, F. D. (2025). Counterfactual Thinking. In Open Encyclopedia of Cognitive Science. MIT Press.
Testing 1 2 3 (4 5 6 7 8 …)
When should you adjust p‑values or confidence intervals for multiple testing? It’s a complicated (and often painful) question, with the wide continuum of possible answers depicted below by Susan Ahmed (1991):
I haven’t met anyone who’s an ultra conservative (on this dimension), but it may be an interesting exercise to estimate how many statistical tests you’re likely to use in a lifetime and what, e.g., a lifetime Bonferroni penality would do to your career.
Sabine Hoffmann and colleagues (2026) offer a helpful guide on what to do. They summarise their principle as follows: “multiple testing should be adjusted for if and only if authors, when reporting and interpreting their findings, put more emphasis on the results of one or several of the tests because of their small p‑value(s)” (p. 3).
This criterion of emphasis applies throughout your reporting, from the title and abstract through to the depths of the results section and discussion. They illustrate this using the example of a study that tested the impact of seven vitamins on mortality (p. 8).
Suppose you find two statistically significant findings out of seven.
If your title were “Vitamin D and vitamin B12 are independent risk factors for all‑cause mortality”, then you need to adjust.
If your title were “Two out of seven vitamins are independent risk factors for all‑cause mortality”, then you don’t.
References
Ahmed, S. W. (1991). Issues arising in the application of Bonferroni procedures in federal surveys. ASA Proceedings of the Survey Research Methods Section, 344–349.
Hoffmann, S., Lemster, S., Collins, G., Hapfelmeier, A., Heinze, G., Mayr, A., Schmid, M., Wilcke, J. C., & Boulesteix, A. (2026). When to Adjust for Multiple Testing: A Unifying Guiding Principle. Biometrical Journal, 68(4), e70148.
RCT Bench
“A curated public collection of randomised controlled trials for evaluating covariate-adjustment methods” – over here. Includes trials of mental health interventions.
Futures
I recently attended Henrik Bengtsson’s excellent UseR! 2026 workshop on the {futureverse}. Here are a few examples to show what it does.
Fun with WebSDR
Two very different transmissions you can hear on SW:
- 14.759 MHz USB: the E07 numbers station.
- 6.130 MHz AM: Radio Europe.
Recorded using the WebSDR shortwave receiver hosted by the ETGD amateur radio club at the University of Twente.
QSL: Radio Europa (aka Radio Europe)
Starting 2350 UTC on 15 June 2026, 6130 kHz, using an XHDATA D-808 with XHDATA AN-80 antenna. Tunes included included “Good Life” by Inner City.
“Bedankt voor je verslag wij vinden dit leuk
Wij doen ons best om zoveel verschillende muziek in een korte tijd uit te zenden
Tevens John is de eigenaar van Radioeuropa
Wij zijn wat broers die deze hobby al vanaf de jaren 1970 doen
Hier word gewerkt met een inverterd V antenne zelfbouw en vorig jaar een fabriekszender
Daarvoor werkte wij nog met zelfbouw zenders en modulators met tubes
Zal u wat foto’s sturen en mocht je meer willen weten horen wij dit graag
73 radio europa team”
CoPilot translate:
“Thank you for your report, we appreciate it.
We do our best to broadcast as many different kinds of music as possible in a short time.
Also, John is the owner of Radio Europa.
We are a group of brothers who have been doing this hobby since the 1970s.
We work here with a home‑built inverted‑V antenna and, since last year, a factory‑made transmitter.
Before that we were still using home‑built transmitters and tube modulators.
We will send you some photos, and if you would like to know more, we would be happy to hear it.
73, Radio Europa team”