*! version 2.1 February 19, 2022
help mces
help svysd

Title

mces -- Standardized effect sizes for comparisons between
predicted values of continuous outcome variables after margins or
mimrgns

svysd -- Pooled standard deviations for continuous outcome
variables when data are svyset or mi svyset

Syntax

Syntax after margins, pwcompare post or mimrgns, pwcompare post:

mces [options]

To calculate the standard deviation only:

svysd outcomevar, sdbyvar(varname) [options]

options

hedgesg estimate Hedges's g instead of the default RMSE-based Delta [mces only]

cohensd estimate Cohen's d instead of the default RMSE-based Delta [mces only]

sdbyvar(varname) dichotomous indicator variable defining
comparison groups [used with hedgesg, cohensd, and svysd]

unweighted calculate the unweighted standard deviation used for Cohen's d [svysd only]

sdupdate force a re-calculation of the standard deviation [mces only]

nowarning suppress warning messages

force do not check for continuous outcome variable

Description

mces calculates one of three standardized effect size statistics for between-group
contrasts of marginal effects obtained either from margins or
mimrgns (Klein, 2016--type ssc install mimrgns to install). The available effect size measures are Delta figured using the RMSE, Hedges's g (Hedges, 1981), or Cohen's d (Cohen, 1988). Hedges's g is similar to Cohen's d but uses a pooled standard deviation (sd*) that is weighted by the sample sizes in each group, which is preferable in instances where the group sizes are unequal (see Ellis, 2010). Hedges's g reduces to be equivalent to Cohen's
d when the group sizes are the same. The RMSE-based Delta that is estimated by default uses the error from the regression, rather than the raw standard deviation of the outcome variable, in the effect size calculation.

mces can process estimates when data are svyset or mi svyset.
mces calls svysd with complex survey data when Hedges's g or Cohen's d are requested, but remembers the standard deviation if the outcome variable remains the same to reduce computation time. Note that the command uses the sampling weights to estimate population sample sizes, rather than the unweighted number of cases, when weighting the standard deviation. The sdupdate option forces svysd to update the standard deviation if necessary.
If only the survey-adjusted pooled weighted or unweighted standard deviation is desired,
but not the effect size, then svysd can be used as a standalone command.

The mces command should work with most regression-type models followed by
margins, pwcompare post or mimrgns, pwcompare post that store
their coefficients in e(b_vs), such as regress, truncreg, sem and gsem, and tobit. mces is not appropriate for multilevel/hierarchical linear/mixed-effects models (see Lorah, 2018), nor for generalized linear models with categorical outcomes that do not have standard deviations or RMSEs. svysd is not a postestimation command and functions independently.

Options

Options for Hedges's g and Cohen's d

sdbyvar(varname) specifies the variable name that indicates assignment to
one of the two groups whose marginal effects are contrasted (e.g., the treatment
group and the control group). The sdbyvar must be dichotomous.
The program will return an error message if the variable has more than two
values, even if options such as at or subpop mean
that only two of the levels are used by margins. The
recode, generate() command is useful for creating dichotomous grouping
variables to help ensure a valid estimate of the standard deviation. This option is required when Hedges's g and Cohen's d are requested, but not for the default RMSE-based Delta statistic.

cohensd (for mces) requests estimates of Hedges's g.

cohensd (for mces) requests estimates of Cohen's d.

unweighted (for svysd) requests the unweighted pooled standard
deviation used for Cohen's d instead of the weighted pooled standard deviation SD* used to calculate Hedges's g.

Other options

sdupdate requests the re-calculation of the pooled standard
deviation. mces stores the standard deviation from the last estimation
in a scalar, and typically does not re-estimate it if the outcome variable is
the same. Use the sdupdate option to update the standard deviation if
necessary (e.g., if the dataset has changed).

If there are comparisons reported by margins, pwcompare (or contrast)
for which the reported effect size might not be applicable, the program will return a
warning message. The nowarning option suppresses these messages.

force bypasses the program's attempt to ensure that the outcome variable
is continuous. While these effect sizes measures are designed for
continuous outcome variables, this option allows you to use the program (at your own
risk!) for categorical outcomes if you are sure that the estimated standard
deviation is correct and applicable, and that the coefficients from
margins are comparable in this way.

Examples

Simple example

sysuse nlsw88

reg wage age hours i.union i.married i.union#i.married

margins union, pwcompare post

mces

Two comparison variables

reg wage age hours i.union i.married i.union#i.married

margins union, over(married) pwcompare post

mces

Note the warning message: mces effect size apply to only "all else equal"
comparisons between union=0 and union=1. Accordingly, g and d are only valid for rows 1 and 6 in these results. Because Delta is figured using the RMSE and is not variable-specific, other "all else equal" comparisons may also be in the results; in this case, rows 2 and 5 are "all else equal" comparisons, but between married=0 and married=1. The program can't ensure that the results make sense--that's up to you!

Hedges's g

reg wage age hours i.union i.married i.union#c.hours

margins union, at(hours=(20 40) married=1) pwcompare post

mces, sdby(union) hedgesg

Simple svyset example

webuse nmihs

svyset [pweight=finwgt], strata(stratan)

svy: regress birthwgt age i.race i.multiple i.race#i.multiple

margins multiple, pwcompare(effects) post

mces

Using mimrgns

webuse nhanes2

mi set mlong

mi register imputed diabetes

mi impute chained (logit) diabetes = bpsystol female race age bmi, rseed(1111) add(5)

mi svyset [pw=finalwgt], psu(psu) strata(strata) singleunit(centered)

mi estimate: svy: regress bpsystol i.female race age i.diabetes i.diabetes#i.female

mimrgns female, at(diabetes=(0 1) (median) age) pwcompare post

mces, cohensd sdbyvar(female)

svysd as a standalone command

webuse nmihs

svyset [pweight=finwgt], strata(stratan)

svysd birthwgt, sdby(multiple)

Stored results

mces and svysd save the following in r():

Scalars:

r(RMSE) the root mean square error of the regression (estimated by mces even if the regression does not report one)

or
r(sdstar) sd*, the pooled weighted standard deviation for Hedges's g

or
r(pooledsd) the unweighted pooled weighted standard deviation for Cohen's d

r(n__at_#) the sample size in the group sdbyvar=#

r(n__at_#) the sample size in the group sdbyvar=#

r(sd__at_#) the standard deviation for the group sdbyvar=#

r(sd__at_#) the standard deviation for the group sdbyvar=#

Matrices (mces only):

r(Delta) the estimated RMSE-based Delta values

or
r(g) the estimated Hedges's g values

or
r(d) the estimated Cohen's d values

Macros:

r(depvar) the outcome variable

r(sdbyvar) the margins variable

References

Cohen, J. (1988). Statistical power analysis for the behavioral sciences.
Lawrence Erlbaum Associates.

Ellis, P. D. (2010). The essential guide to effect sizes: Statistical
power, meta-analysis, and the interpretation of research results. Cambridge
University Press.

Hedges, L. V. (1981). Distribution theory for Glass’s estimator of effect
size and related estimators. Journal of Educational Statistics, 6(2), 107–128.
https://doi.org/10.3102/10769986006002107

Klein, D. (2016). Marginal effects in multiply imputed datasets.
14th German Stata Users Group Meeting, Cologne, Germany.
https://www.stata.com/meeting/germany16/slides/de16_klein.pdf

Lorah, J. (2018). Effect size measures for multilevel models: definition,
interpretation, and TIMSS example. Large-scale Assessments in Educucation, 6, 8.
https://doi.org/10.1186/s40536-018-0061-2

Acknowledgements

Miguel Dorta, Daniel Klein, and Chris Cheng contributed helpful advice during the development
process.

Author

Brian Shaw, Indiana University, USA

bpshaw@indiana.edu