Personal website – any views expressed herein are my own unless otherwise obvious
“The ESDist package is designed to calculate and visualise field-specific effect size distributions, based on data that can easily be obtained from meta-analyses.”
Looks like a relatively painless way to develop alternatives to Cohen’s 0.2, 0.5, 0.8 SD landmarks, including using limit meta analysis (Rücker et al.) to adjust individual effect sizes for publication bias.
In multilevel analyses, it is often necessary to estimate a standardised mean difference (SMD) where the numerator is a covariate-adjusted mean difference and the denominator is the SD from a model without covariate adjustment.
I’m aware of two (frequentist, non-bootstrap) packages calculating an SMD and confidence interval in R. Here’s a comparison of them for simulated data.
The problem is, you want to estimate g with a covariate-adjusted treatment effect in the numerator but unadjusted SD in the denominator and an uncertainty interval of some description.
Work in progress R code here – scroll to the bottom for a table comparing results. Use at own risk.
Here is also a comparison of {eefAnalytics} and {lmeInfo}.
This package looks useful (for {nlme} not {lme4}).
“Provides analytic derivatives and information matrices for fitted linear mixed effects (lme) models and generalized least squares (gls) models estimated using lme() (from package ‘nlme’) and gls() (from package ‘nlme’), respectively. The package includes functions for estimating the sampling variance-covariance of variance component parameters using the inverse Fisher information. The variance components include the parameters of the random effects structure (for lme models), the variance structure, and the correlation structure. The expected and average forms of the Fisher information matrix are used in the calculations, and models estimated by full maximum likelihood or restricted maximum likelihood are supported. The package also includes a function for estimating standardized mean difference effect sizes (Pustejovsky, Hedges, and Shadish (2014) <doi:10.3102/1076998614547577>) based on fitted lme or gls models.”
Furukawa and Leucht (2011) give a formula for calculating the number needed to treat (NNT), i.e., (p. 1)
“the number of patients one would need to treat with the intervention in question in order to have one more success (or one less failure) than if treated in the control intervention”
based on the control group event rate (CER; for instance proportion of cases showing recovery) and Cohen’s d – an effect size in standard deviation units.
R code below:
NNT = function(d, CER) {
1 / ( pnorm( d - qnorm(1-CER) ) - CER )
}
Reference
I’ve moaned a bit about (what felt at the time to be a religion of) “effect size”. Recently Thom Baguley has published a paper on the topic, comparing standardised effects measures, which involve scaling with respect to the sample variance, with simple effects measures, which are expressed in the original units of measurement.
Baguley reviews some of the problems with standardised measures, all related to factors affecting sample variance. In general he advises reporting simple effect sizes, and preferably with confidence intervals. If you really want to use standardised measures, for instance to compare conceptually similar measures on different scales, then he advises against reporting absolute and “canned” judgements like “small”, “medium”, and “large”, arguing instead in favour of descriptions about the relative size of effects.
I like his Tukey quote:
“… being so disinterested in our variables that we do not care about their units can hardly be desirable.”
It does seem odd to focus on, e.g., how much variance is explained rather than actually characterising the nature of relationships between variables.
Baguley, T. (2009). Standardized or simple effect size: What should be reported? British Journal of Psychology, 100, 603–617.
Tired of people equating “effect size” with “standardised measure of effect size”? Here’s an antidote, thanks to Shinichi Nakagawa and Innes C. Cuthill (2007). [Effect size, confidence interval and statistical significance: a practical guide for biologists. Biol. Rev. (2007), 82, pp. 591–605.]
They review the different meanings of “effect size”:
They argue in favour of confidence intervals, as these “are not simply a tool for NHST [signifcance testing], but show a range of probable effect size estimates with a given confidence.”
They also cite Wilkinson, L & The Task Force on Statistical Inference (1999) [Statistical methods in psychology journals. American Psychologist 54, 594–604]:
“our focus on these two standardised effect statistics does not mean priority of standardised effect statistics (r or d) over unstandardised effect statistics (regression coefficient or mean difference) and other effect statistics (e.g. odds ratio, relative risk and risk difference). If the original units of measurement are meaningful, the presentation of unstandardised effect statistics is preferable over that of standardised effect statistics (Wilkinson & the Task Force on Statistical Inference, 1999).”
Good stuff, this.