*! version 1.0.1 10Nov2023

help pmvalsampsize

Title

pmvalsampsize -- Calculates the minimum sample size required for external validation of a multivariable prediction model

Syntax

pmvalsampsize , type(string)
[rsquared(real 0) rsqciwidth(real 0.1)
cslope(real 1) csciwidth(real 0.2)
citl(real 0) citlciwidth(real 0) varobs(real 0)
prevalence(real 0) simobs(int 1000000)
cstatistic(real 0) cstatciwidth(real 0.1)
oe(real 1) oeciwidth(real 0.2) oeseincrement(real 0.0001)
lpnormal(numlist max=2) lpskewednormal(numlist max=4)
lpbeta(numlist max=2) lpcstat(numlist max=1)
tolerance(real 0.0005) increment(real 0.1)
sensitivity(real 0) specificity(real 0) threshold(real 0)
nbciwidth(real 0.2) nbseincrement(real 0.0001)
seed(int 123456) mmoe(real 1.1) trace graph noprint]

pmvalsampsize_general_options

type(string) specifies the type of analysis for which sample size is being calculated

cslope(real 1) anticipated C-slope performance in validation sample

csciwidth(real 0.2) target CI width for calibration slope performance

noprint suppress criteria descriptions in output

Binary outcome models

binary_outcome_options

prevalence(real 0) anticipated prevalence of outcome in validation sample

simobs(int 1000000) sets the number of observations to use for simulated LP calculations

cstatistic(real 0) anticipated c-statistic at validation

cstatciwidth(real 0.1) target CI width for c-statistic performance

oe(real 1) anticipated observed/expected ratio at validation

oeciwidth(real 0.2) target CI width for OE performance

lpnormal(numlist max=2) defines parameters to simulate LP from normal distribution

lpskewednormal(numlist max=4) defines parameters to simulate LP from a skewed normal distribution

lpbeta(numlist max=2) defines parameters to simulate P from beta distribution

lpcstat(numlist max=1) defines starting value for non-events mean

tolerance(real 0.0005) sets tolerance for observed event prop. during iterative procedure for non-events mean

increment(real 0.1) sets increment by which to iterate when identifying mean for non-events for lpcstat()

trace output a trace of the iteration process when using lpcstat()

oeseincrement(real 0.0001) sets increment by which to iterate when identifying the SE(ln(OE)) to meet the target CI width for OE

seed(int 123456) set seed for simulation based calclulations

graph produces histogram of the LP distribution for checking

sensitivity(real 0) anticipated sensitivity at validation

specificity(real 0) anticipated specificity at validation

threshold(real 0) specifies risk threshold used for net benefit calculation

nbciwidth(real 0.2) target CI width for standardised net benefit performance

nbseincrement(real 0.0001) sets increment by which to iterate when identifying the SE(standardised net benefit) to meet the target CI width for standardised net benefit

Continuous outcomes models

continuous_outcome_options

rsquared(real 0) anticipated value of the R-squared in the validation sample

rsqciwidth(real 0.1) target precision in terms of CI width for R-sq

citl(real 0) assumed CITL performance in validation sample

citlciwidth(real 0) target precision in terms of CI width for CITL

varobs(real 0) anticipated variance of observed values in the validation sample

mmoe(real 1.1) set MMOE threshold for acceptable precision of residual variance of CITL & C-slope

Description

pmvalsampsize computes the minimum sample size required for the external validation of an existing multivariable prediction model using the criteria proposed by Archer et al. and Riley et al.
pmvalsampsize can currently be used to calculate the minimum sample size for the external validation of models with continuous or binary outcomes.
Survival (time-to-event) outcome model calculations are a work in progress.

A series of criteria define the sample size needed to ensure precise estimation of key measures of prediction model performance,
allowing conclusions to be drawn about whether the model is potentially accurate and useful in a given population of interest.

For continuous outcome models, Archer et al. specify four criteria to calculate the sample size (N) needed for:

i) precise estimation of the R-squared performance,

ii) precise estimation of the calibration-in-the-large (CITL),

iii) precise estimation of the calibration slope (c-slope), and

iv) precise estimation of the residual variance for CITL & c-slope.

The sample size calculation requires the user to pre-specify
(e.g. based on previous evidence from the development study or information known from the validation sample if available) the following;

- the anticipated R-squared performance

- the target precision (CI width) for the R-squared

- the anticipated CITL performance

- the target precision (CI width) for the CITL

- the anticipated c-slope

- the target precision (CI width) for the c-slope

- the anticipated variance of observed values in the validation sample

For binary outcomes, there are three criteria to calculate the sample size (N) needed for:

i) precise estimation of the Observed/Expected (O/E) statistic,

ii) precise estimation of the calibration slope (c-slope), and

iii) precise estimation of the c-statistic.

And a fourth optional criteria to calculate the sample size (N) needed for:

iv) precise estimation of the standardised net-benefit (sNB).

The sample size calculation requires the user to pre-specify the following;

- the outcome event proportion

- the anticipated O/E performance

- the target precision (CI width) for the O/E

- the anticipated c-slope

- the target precision (CI width) for the c-slope

- the anticipated c-statisitic performance

- the target precision (CI width) for the c-statisitic

- the distribution of estimated probabilities from the model, ideally specified on the log-odds scale - AKA the Linear Predictor (LP)

- the anticipated sensitivity, specificity and relevant risk threshold for sNB calculation

- the target precision (CI width) for the standardised net benefit

General Options

type(string) specifies the type of analysis for which sample size is being calculated:

c specifies sample size calculation for a prediction model with a continuous outcome

b specifies sample size calculation for a prediction model with a binary outcome

cslope(real 1) specifies the anticipated c-slope performance in the validation sample.
Default conservatively assumes perfect c-slope=1.
The value could alternatively be based on a previous validation study for example.
For binary outcomes the c-slope calculation requires the user to specify a distribution for the assumed LP in the validation sample
(or alternatively the distibution of predicted probabilities in the validation sample). See lp*() options below.

csciwidth(real 0.2) specifies the target CI width (acceptable precision) for the c-slope performance.
Default assumes CI width=0.2.

noprint suppresses output showing the detailed descriptions of each of the criteria shown in the results table.

Binary outcome options

prevalence(real) specifies the overall outcome proportion (for a prognostic model) or overall prevalence (for a diagnostic model) expected within the model validation sample.
This is a required input.
This should be derived based on previous studies in the same population or
directly from the validation sample if to hand.

simobs(int) specifies the number of observations to use when simulating the LP distribution for c-slope calculation in criteria 2.
Default observations=1,000,000.
Higher simobs() values will reduce random variation further.

cstatistic(real) specifies the anticipated c-statistic performance in the validation sample.
This is a required input.
May be based on the optimism-adjusted c-statistic reported in the development study for the existing prediction model.
Ideally, this should be an optimism-adjusted c-statistic.
NB: This input is also used when using the lpcstat() option.

cstatciwidth(real 0.1) specifies the target CI width (acceptable precision) for the c-statistic performance.
Default assumes CI width=0.1.

oe(real 1) specifies the anticipated O/E performance in the validation sample.
Default conservatively assumes perfect O/E=1.

oeciwidth(real 0.2) specifies the target CI width (acceptable precision) for the E/O performance.
Default assumes CI width=0.2.
The choice of CI width is context specific, and depends on the event probability in the population.
See Riley et al. for further details.

oeseincrement(real 0.0001) sets the increment by which to iterate when identifying the SE(ln(OE)) to meet the target CI width specified for OE.
The default iteration increment=0.0001.
In the majority of cases this will be suitably small to ensure a precise SE is identified.
The user should check the output table to ensure that the target CI width has been attained and adjust the increment if necessary.

lpnormal(numlist) defines parameters to simulate the LP distribution for criteria 2 from a normal distribution.
The user must specify the mean and standard deviation (in this order) of the LP distribution.

lpskewednormal(numlist) defines parameters to simulate the LP distribution for criteria 2 from a skewed normal distribution.
The user must specify the mean, variance, skew and kurtosis parameters (in this order) of the LP distribution.
NB: lpskewednormal() option uses sknor and can take a little longer than other distributional assumptions.

lpbeta(numlist) defines parameters to simulate the distribution of predicted probabilities for criteria 2 from a beta distribution.
The user must specify the alpha and beta parameters (in this order) of the probability distribution.
The LP distribution is then generated internally using this probability distribution.

lpcstat(numlist) defines parameters to simulate the LP distribution for criteria 2 assuming that the distribution of events and non-events are normal with a common variance.
The user specifies a single input value - the expected mean for the non-events distribution. This could be informed by clinical guidance.
However, this input is taken as a starting value and an iterative process is used to identify the most appropriate values for the event and non-event distributions
so as to closely match the anticipated prevalence in the validation sample.
NB: this approach makes strong assumptions of normality and equal variances in each outcome group, which may be unrealistic in most situations.

tolerance(real 0.0005) for use with lpcstat() option.
Sets the tolerance for agreement between the simulated and expected event
proportion during the iterative procedure for calculating the mean for the non-events distribution.

increment(real 0.1) for use with lpcstat() option.
Sets increment by which to iterate the value of the mean for the non-events distribution.
Trial and error may be necessary as it is dependent on how close the inital input for the non-event mean in lpcstat() is to the required value.
If the process takes a particularly long time then the user could try an alternative increment value, or an alternative non-event mean value in lpcstat().
The trace option may be useful in such circumstances.

trace for use with lpcstat() option.
Specifies that a trace of the values obtained in each iteration when identifying the non-event mean is output.
Useful when finding the approriate values for lpcstat() & increment() is proving difficult!

seed(int 123456) specifies the initial value of the random-number seed used by the random-number functions when simulating data
to approximate the LP distribution for criteria 2.

graph specifies that a histogram of the simulated LP distribution for criteria 2 is produced.
The graph also details summary statistics for the simulated distribution.
Useful option for checking the simulated LP distribution against the source of input parameters.
Also useful for reporting at publication.

sensitivity(real 0) specifies the anticipated sensitivity performance in the validation sample at the chosen risk threshold (specified using threshold()).
If sensitivity and specificity are not provided then pmvalsampsize uses the simulated LP distribution from criteria 2
and the user-specified risk threshold to estimate the anticipated sensitivity and specificity
to be used in calculation of net benefit.
NB: net benefit criteria is only calculated if either i) sensitivity(), specificity() and threshold() or ii) threshold() option are provided.

specificity(real 0) specifies the anticipated specificity performance in the validation sample at the chosen risk threshold (specified using threshold()).
If sensitivity and specificity are not provided then pmvalsampsize uses the simulated LP distribution from criteria 2
and the user-specified risk threshold to estimate the anticipated sensitivity and specificity
to be used in calculation of net benefit.
NB: net benefit criteria is only calculated if either i) sensitivity(), specificity() and threshold() or ii) threshold() option are provided.

threshold(real 0) specifies the risk threshold to be used for calculation of net benefit performance of the model in the validation sample.
If sensitivity and specificity are not provided then threshold() must be given in order for pmvalsampsize to assess sample size requirements for net benefit.
NB: net benefit criteria is only calculated if either i) sensitivity(), specificity() and threshold() or ii) threshold() option are provided.

nbciwidth(real 0.2) specifies the target CI width (acceptable precision) for the standardised net benefit performance.
Default assumes CI width=0.2.
The choice of CI width is context specific.
See Riley et al. for further details.

nbseincrement(real 0.0001) sets the increment by which to iterate when identifying the SE(standardised net benefit) to meet the target CI width specified for standardised net benefit.
The default iteration increment=0.0001.
In the majority of cases this will be suitably small to ensure a precise SE is identified.
The user should check the output table to ensure that the target CI width has been attained and adjust the increment if necessary.

Continuous outcome options

rsquared(real) specifies the anticipated value of R-squared in the validation sample.
This is a required input.
May be based on the optimism-adjusted R-squared reported in the development study for the existing prediction model.
Ideally, this should be an optimism-adjusted R-squared value, not the apparent R-squared value, as the latter is optimistic (biased).

rsqciwidth(real 0.1) specifies the target CI width (acceptable precision) for the R-squared performance.
Default assumes CI width=0.1.

citl(real 0) specifies the anticipated value of CITL in the validation sample.
Default conservatively assumes perfect CITL=0.

citlciwidth(real 0) specifies the target CI width (acceptable precision) for the CITL performance.
The choice of CI width is context specific, and depends on the scale of the outcome.
See Archer et al. for further details.

varobs(real 0) the anticipated variance of observed values in the validation sample.
This is a required input.
This could again be based on the previous development study, or on clinical knowledge.

mmoe(real 1.1) multiplicative margin of error (MMOE) acceptable for calculation of the variance for c-slope and CITL in the calibration model.
The default is a MMOE of 10%.
See Archer et al. reference below for further information.

Examples

Examples based on those included in the papers referenced below by Riley et al. & Archer et al. published in Statistics in Medicine.

Binary outcome models

Use pmvalsampsize to calculate the minimum sample size required to externally validate an existing multivariable prediction model for a binary outcome (e.g. mechanical heart valve failure).
Based on previous evidence, the outcome prevalence is anticipated to be 0.018 (1.8%) and the reported c-statistic was 0.8.
The LP distribution was published and appeared normally distributed with mean(SD) of -5(2.5).
We target default CI widths for all but O/E CI width=1 (see Riley et al. for details).
We can use the graph option to check the simulated distribution is appropriate

. pmvalsampsize, type(b) prev(.018) cstat(.8) lpnorm(-5 2.5) oeciwidth(1) graph

On further inspection a skewed normal distribtion may be more appropriate as below:

. pmvalsampsize, type(b) prev(.018) cstat(.8) lpskew(-5.8 5 -0.5 4) oeciwidth(1) graph

Alternatively, lets assume that the authors provided a distribution of predicted probabilities (e.g. as part of a calibration plot).
We can use this to specify parameters for a beta distribution to simulate the LP distribution as below:

. pmvalsampsize, type(b) prev(.018) cstat(.8) lpbeta(.5 .5) oeciwidth(1) graph

Finally, we can use the anticipated c-statistic to simulate the event and non-event distributions assuming normality and common variances.
We input a starting value for the mean of the non-events distribution as below:

. pmvalsampsize, type(b) prev(.018) cstat(.8) lpcstat(-4) oeciwidth(1) seed(1234) graph

Continuous outcome models

Use pmvalsampsize to calculate the minimum sample size required for validation of a multivariable prediction model for a continuous outcome (here, fat-free mass in children).
We know the existing prediction model has been previously validated in a small sample with an R-squared of 0.9.
Riley et al. compute the anticipated variance of outcome values in the validation sample based on reported upper and lower quartile values deriving var(Y)=0.089 (see Riley et al. for details).
We conservatively assume perfect calibration performance at validation.
If we target a R-sq CI width=0.1, CITL CI width=0.08 and c-slope CI width=0.2, then we can run the following command:

. pmvalsampsize, type(c) rsq(0.9) varobs(0.089) citlciwidth(0.08)

Stored results

pmvalsampsize stores the following in r() as relevant for the specific outcome type chosen:

Scalars

r(sample_size) Minimum sample size required for model validation

r(events) Minimum number of events required assuming input outcome proportion or outcome prevalence (binary models)

r(prevalence) Overall prevalence (binary models)

r(cstatistic) User-specified c-statistic (binary models)

r(se_cstat) Standard error (SE) of c-stat based on target CI width (binary models)

r(cstatciwidth) User-specified target CI width for c-statistic (binary models)

r(oe) User-specified O/E (binary models)

r(se_oe) SE of O/E based on target CI width (binary models)

r(lb_oe) Lower CI of O/E based on target CI width (binary models)

r(ub_oe) Upper CI of O/E based on target CI width (binary models)

r(width_oe) User-specified target CI width for O/E (binary models)

r(citl) User-specified CITL (linear models)

r(se_citl) SE of CITL based on target CI width (linear models)

r(lb_citl) Lower CI of CITL based on target CI width (linear models)

r(ub_citl) Upper CI of CITL based on target CI width (linear models)

r(citlciwidth) User-specified target CI width for CITL (linear models)

r(cslope) User-specified c-slope

r(se_cslope) SE of c-slope based on target CI width

r(lb_cslope) Lower CI of c-slope based on target CI width

r(ub_cslope) Upper CI of c-slope based on target CI width

r(csciwidth) User-specified target CI width for c-slope

r(rsquared) User-specified R-squared adjusted (linear models)

r(se_rsq) SE of R-sq based on target CI width (linear models)

r(lb_rsq) Lower CI of R-sq based on target CI width (linear models)

r(ub_rsq) Upper CI of R-sq based on target CI width (linear models)

r(rsqciwidth) User-specified target CI width for R-sq (linear models)

r(non_event_mean) Derived mean for non-events distribution when using lpcstat() (binary models)

r(event_mean) Mean for events distribution when using lpcstat() (binary models)

r(common_variance) Common variance parameter when using lpcstat() (binary models)

r(tolerance) Tolerance for agreement between anticipated event prop. and simulated event prop. when using lpcstat() (binary models)

r(increment) Increment by which to iterate non-event mean when identifying suitable mean to meet tolerance() when using lpcstat() (binary models)

r(simulated_data_cstat) C-statistic calculated through simulation based process. Can be used to check against target c-statistic (binary models)

r(sensitivity) Sensitivity either user specified or calculated through simulation based process (binary models)

r(specificity) Specificity either user specified or calculated through simulation based process (binary models)

r(nb) Net benefit calculated using sens, spec and risk threshold (binary models)

r(standardised_nb) Standardised net benefit calculated using prevalence, sens, spec and risk threshold (binary models)

r(threshold) User specified risk threshold for net benefit (binary models)

Matrices

r(results) A matrix of all tabular output

Authors

Joie Ensor, University of Birmingham

j.ensor@bham.ac.uk

Acknowledgements

With thanks to Richard Riley for helpful feedback

References

Archer L, Snell K, Ensor J, Hudda MT, Collins GS, Riley RD.
Minimum sample size for external validation of a clinical prediction model with a continuous outcome. Statistics in Medicine. 2020.

Riley RD, Debray TPA, Collins G, Archer L, Ensor J, van Smeden M, Snell KIE.
Minimum sample size for external validation of a clinical prediction model with a binary outcome. Statistics in Medicine. 2021.

Also see

Online: pmsampsize, pmcalplot, matlist, sknor