*! version 1.12 04may2023

Title

[R] repest Estimation with weighted replicate samples and plausible values

Syntax

repest
svyname
[if] [in]
, estimate(cmd [,cmd_options])
[options]

Main
estimate(cmd [,cmd_options]) runs estimation command cmd with options cmd_options.

Optional
by(varname [,by_options]) produces separate estimates by levels of varname. Averaged results can optionally be requested.

over(varlist [,test]) jointly estimates across levels of varlist.

outfile(filename [,of_options])
saves results to disk.

results(results_options) keep, add, and combine estimation results.

display displays results in output window.

flag flags elements of results which should not be reported.

coverage Reports coverage of estimation sample relative to target sample.

svyparms(svy_options) overrides default parameters set by svyname.

fast when a pvvarlist is specified, computes sampling variance only for the first plausible value.

store(string) saves the estimation results stored in e().

PIAAC, PISA, SVY, TALISTCH, and many other survey names are valid - see svynames. Option svyparms() is required with SVY.

You have to specify one estimate() option.

Description

repest estimates statistics using replicate weights (BRR weights, Jackknife replicate
weights,...), thus accounting for complex survey designs in the estimation of sampling
variances. It was originally designed to be used with the PIAAC, PISA and TALIS datasets produced by
the OECD, but works for many other educational large-scale surveys as well. It also allows for analyses with multiply imputed variables (plausible values); where
plausible values are included in a pvvarlist, the average estimator across plausible values is
reported and the imputation error is added to the variance estimator.

svyname is a shortcut for declaring survey settings. Use svyname to indicate the data
source. svyname must be equal to one of the supported survey datasets: ALL, IALS, ICCS, ICILS, IELS, PIAAC, PIRLS, PISA, PISAOOS,
SSES, SSES2023, TALISSCH, TALISTCH, TALISEC_STAFF, TALISEC_LEADER, TIMSS
(see remarks, below, for the corresponding definitions and parameters).
In addition, svyname can be equal to SVY: in this case, you must specify survey settings using option svyparms().

Options

Main

estimate(cmd
[,cmd_options]) runs estimation command cmd
with options cmd_options.

cmd can be any estimation command that posts results in e(b)
and accepts pweights or aweights, including
user-defined e-class programs (see remarks below). In addition, cmd can be
equal to means, freq, corr, summarize or quantiletable (see below).
Options for cmd must be declared after
a comma, as in the original syntax of cmd. Weights are used but should not be declared, as they are passed to the estimation command from the survey
settings. Optional [if] [in] statements are not to be included in cmd and
should be declared after the command repest.

A version ##: prefix can be included in the cmd statement
(as in repest PIAAC, estimate(version 13: ivregress...).
Most other prefix commands cannot be included.

varlists within cmd or cmd_options that include plausible values
should be specified as pvvarlists. varlists
and pvvarlists can contain factor variables.

means, freq, corr, summarize or quantiletable are special
commands for standard descriptive analyses:

means pvvarlist [, pct]
computes means of all variables in pvvarlists. pvvarlists may contain a mix of pvvarnames
and varnames. Option pct multiplies means by 100 - e.g. to report percentages on binary (0/1) indicator variables.

freq pvvarname [, count levels(levelslist)] computes frequency counts of categorical variable pvvarname.
pvvarname may be a varname or a set of plausible values, such as plausible proficiency levels. By default, freq
reports frequencies as percentages. If sub-option count is specified, raw counts are reported.
If sub-option levels() is specified, only levels of pvvarname in levelslist are reported.

corr pvvarlist [, pairwise] computes pearson correlation coefficients among variables of pvvarlists.
pvvarlists may contain a mix of pvvarnames and varnames. Option pairwise requests pairwise
correlations (see pwcorr). The default is to delete observations listwise.

summarize pvvarlist , stats(statlist) computes summary statistics of all variables in pvvarlists.
pvvarlists may contain a mix of pvvarnames and varnames. You have to specify sub-option stats().

stats(statname [statname ...])
specifies the statistics to be estimated; ou have to specify at least one statname. Multiple statistics may be specified
and are separated by white space, such as stats(mean sd).
Available statistics are

statname

mean mean

sd standard deviation (the sample standard deviation is computed; for an unbiased estimator of the population standard deviation, use sd_ub)

kurtosis kurtosis

skewness skewness

min minimum

max maximum

p1 1st percentile

p5 5th percentile

p10 10th percentile

p25 25th percentile

p50 50th percentile

p75 75th percentile

p90 90th percentile

p95 95th percentile

p99 99th percentile

iqr inter-quartile range (p75 - p25)

idr inter-decile range (p90 - p10)

Var variance (the sample variance is computed; for an unbiased estimator of the population variance, use Var_ub)

sum sum of variable

N number of observations

sum_w sum of the weights

quantiletable index outcome [, nquantiles(n)
noindexquantiles nooutcomequantiles relrisk oddsratio summarize(var1) regress(var2 var3) test]
computes a quantile table, similar to tables introducing indices in PISA international reports. quantiletable index outcome generates quantile categories of index and
reports the average of index and outcome within index quantile categories.
nquantiles(n) specifies how many quantile categories are generated - the default is 4 (quarters).
noindexquantiles drops averages of index within index quantile categories from the results.
nooutcomequantiles drops averages of outcome within index quantile categories from the results.
relrisk adds to the results a measure of relative risk for the likelihood of outcome being in the top quantile category of outcome if index
is in the bottom quantile category of index.
oddsratio adds to the results the odds ratios for the likelihood of outcome being in the top quantile category of outcome if index
is in the bottom quantile category of index.
summarize(var1) adds to the results the mean and standard deviation of var1.
regress(var2 var3) adds to the results the r-squared from the regression of var2 on var3.
test adds to the results the difference between the average of outcome in the top index quantile category and the average of outcome in the bottom index quantile category.

Note: while corr and summarize look similar to regular stata r-class commands, it is not possible, in general, to use r-class command within repest. These commands have therefore been adapted and their syntax may differ from the usual syntax. For example, it is not possible to abbreviate summarize or to type correlate instead of corr.

Other

by(varname [, levels(string) average(string) ]) produces separate estimates by levels of varname. The following options can be specified:

levels(string) requests estimates only for the listed levels of varname. The default is to produce estimates for all levels of varname.

average(string) requests averaged estimates for the listed levels of varname.

over(varlist [,test]) requests estimates to be obtained
separately for each level of categorical variables varlist. The variables must be
numerical. If more than one variable is specified, long_over is assumed.

test computes the difference between estimates obtained for the
highest and the lowest values of varname. It is useful to test for differences. Also see results(combine()).

Note: Use by() if sources of sampling uncertainty are independent across groups, e.g. by(cnt). Use over() if each subset does not corresponds to an independent sample, particularly if you want to test for differences across subsets, e.g. over(gender, test).

results([keep(string) add(string) combine(string)]) can be
used to keep only estimation results of interest, to add scalars from ereturn to
the results, and to combine elements of e(b) to form new scalars (e.g. to test
for differences).

keep(keeplist) requests that only the listed elements of e(b) be
kept. keeplist must specify names exactly as in the output, except if the name
contains a pvvarname, in which case the "@" character must be used.

add(addlist) requests that the listed elements be added to the results of repest. add() can only be used after estimate(stata: e_cmd).
addlist must contain names of scalar returned by e_cmd, such as r2 (rsquared), ll (log-likelihood), N (number of observations).
To see the list of available scalars, type help e_cmd in the command prompt.

combine(name: myexp [, name: myexp] [, ...]) computes myexp and adds the result to the output of repest, with name name.
myexp must contain names of estimated results, together with
operators and/or mathematical functions.
Estimated results must be enclosed by _b[...] .

outfile(filename [, long_over pvalue]) saves the results of repest to a .dta file (Stata dataset). This makes further manipulation of results easy. .dta files can be exported to excel using export excel.

long_over Requests the outfile to be organised in long form (see reshape long/wide). The default is to store results in wide form. (Only to be used in combination with the over() option.)

pvalue Requests the outfile to include p-values for tests of statistical significance. The reported pvalues are based on a z-test assuming an asymptotically normal distribution for the estimated parameter.

display forces the results of repest to be displayed in Stata's result window. The default is to display results only if no outfile() option is specified.

flag replaces estimation results which are based on fewer observations than required for reporting with a specific missing code (.f) in the outfile, and by (omitted) in display.
repest PISA, flag checks that each estimation result is based on at least 30 observations and 5 schools. repest PIAAC, flag
checks that each estimation result is based on at least 30 observations. repest TALISSCH, flag checks that each estimation result is based on at least 10 schools.
repest TALISTCH, flag checks that each estimation result is based on at least 30 observations and 10 schools.

coverage computes the (weighted) proportion of observations in the target sample (i.e. accounting for any [if] [in] statements) that is included in the estimation sample.
Coverage proportions are stored as additional coefficients in e(b).
Note: if option over is specified, the target population is restricted to observations where variables specified within this option are not missing; if, in addition,
suboption test is specified within over, the coverage reported for the difference corresponds to the lowest coverage of the two.

fast When a pvvarlist is specified, the share of variance accounted for by the sampling error is computed only for the first plausible value. This is an unbiased shortcut that makes computation of the variance-covariance matrix faster.

store(string) saves the results stored in e(), renaming the output with string as prefix (see store).

svyparms([NBpv(#) final_weight_name(string) rep_weight_name(string) variancefactor(#) NREP(#)])
sets or overrides survey parameters, which are usually defined by svyname. Option svyparms() is required
if svyname equals SVY.

Remarks

Plausible values

A pvvarname is a set of plausible values. A pvvarname looks like a valid stata name
that contains the character "@" (e.g. "pv@math"). When it encounters a pvvarname, repest checks that variables
"*1*" to "*M*" exist (e.g. pv1math-pv5math), where M is the number of plausible values in survey
svyname, and uses them as appropriate in the estimation.

pvvarlists are varlists that may, or not, include a pvvarname .

Plausible values are multiply imputed measures of unobserved characteristics, such as students'
skills, that allow estimates to be unbiased by measurement error.

Survey options

svyname sets the following parameters (bold indicates variable names that must
exist in the dataset in use). These parameters can be overridden by values set in option svyparms(svy_options).

ALL: Adult Literacy and Lifeskills Survey

IALS: International Adult Literacy Survey

Final weight popwt

Replicate weights REPLIC1-REPLIC30

Variance factor Jackknife 2

Number of replications 30

Number of plausible values 10

ICCS: International Civics and Citizenship Study

ICILS: International Computer and Information Literacy Study

Final weight* TOTWGTS

Replicate weights* SRWGT1-SRWGT75

Variance method Jackknife 2

Number of replications 75

Number of plausible values 5

*To use teacher weights, use ICCS_T or ICILS_T. For school weights, use ICCS_C or ICILS_C.

IELS: International Early Learning Study

Final weight CHILDWGT

Replicate weights SRWGT1-SRWGT92

Variance method balanced repeated replication with Fay's adjustment

Fay's parameter 0.5

Number of replications 92

Number of plausible values 5

Primary sampling unit (for flags) IDCNTRY IDCENTRE

PIRLS: Progress in International Literacy Study

TIMSS: Trends in International Mathematics and Science Study

Final weight* WGT

Replicate weights* JR1-JR150

Variance method JRR

Number of replications 150

Number of plausible values 5

*final and replicate weights must be generated prior to running repest, using the following code :

gen WGT = TOTWGT // note: use TOTWGT for student weights, SCHWGT for school weights, MATWGT for math teacher weights, etc.
forval i = 1/75 {
local j = 75 + `i'
gen JR`i' = WGT
replace JR`i' = 2*WGT*JKREP if JKZONE == `i'
gen JR`j' = WGT
replace JR`j' = 2*WGT*(1-JKREP) if JKZONE == `i'
}

PISA2015: Programme for International Student Assessment

Final weight w_fstuwt

Replicate weights w_fsturwt1-w_fsturwt80

Variance method balanced repeated replication with Fay's adjustment

Fay's parameter 0.5

Number of replications 80

Number of plausible values 10

Primary sampling unit (for flags) cnt cntschid

PISA: Programme for International Student Assessment

Final weight w_fstuwt

Replicate weights w_fstr1-w_fstr80

Variance method balanced repeated replication with Fay's adjustment

Fay's parameter 0.5

Number of replications 80

Number of plausible values 5

Primary sampling unit (for flags) cnt schoolid

Note: with PISA 2015 data, repest automatically changes to PISA2015 options.

PISAOOS: PISA for Development - assessment of out-of-school youth

Final weight spfwt0

Replicate weights spfwt1-spfwt30

Variance factor Jackknife 1

Number of replications 30

Number of plausible values 10

SSES: Survey of Social and Emotional Skills (Round 1)

Final weight WT2019

Replicate weights rwgt1-rwgt76

Variance method Jackknife 1

Number of replications 76

SSES2023: Survey of Social and Emotional Skills (Round 2)

Final weight WT2023

Replicate weights rwgt1-rwgt80

Variance method balanced repeated replication with Fay's adjustment

Fay's parameter 0.5

Number of replications 80

TALISTCH: Teaching and Learning International Survey (teacher weights)

Final weight tchwgt

Replicate weights trwgt1-trwgt100

Variance method balanced repeated replication with Fay's adjustment

Fay's parameter 0.5

Number of replications 100

Primary sampling unit (for flags) cntry idschool

TALISSCH: Teaching and Learning International Survey (school weights)

Final weight schwgt

Replicate weights srwgt1-srwgt100

Variance method balanced repeated replication with Fay's adjustment

Fay's parameter 0.5

Number of replications 100

TALISEC_STAFF: Teaching and Learning International Survey - Starting Strong (staff weights)

Final weight staffwgt

Replicate weights srwgt1-srwgt92

Variance method balanced repeated replication with Fay's adjustment

Fay's parameter 0.5

Number of replications 92

Primary sampling unit (for flags) idcntpop idcentre

TALISEC_LEADER: Teaching and Learning International Survey - Starting Strong (leader weights)

Final weight cntrwgt

Replicate weights crwgt1-crwgt92

Variance method balanced repeated replication with Fay's adjustment

Fay's parameter 0.5

Number of replications 92

PIAAC: Programme for the International Assessment of Adult Competencies

Final weight spfwt0

Replicate weights spfwt1-spfwt80

Variance method* Jackknife 1 or 2, depending on vemethodn

Number of replications 80

Number of plausible values 10

*The correct estimation of standard errors with PIAAC data is done with Jackknife 1
(Australia, Austria, Canada, Denmark, Germany) or Jackknife 2 (all other countries).
In Jackknife 1, the covariance matrix of replicate estimates is adjusted with a factor equal to 79/80.
When Jacknife 1 and 2 country samples are pooled, the adjustment factor is computed as a convex mixture
of 79/80 and 1, with mixture weights proportional to the share of the sample from Jackknife 1 countries;
this share is computed using spfwt0 weights.

SVY: User-defined survey settings (requires option svyparms())

Final weight set by suboption final_weight_name(string)

Replicate weights set by suboption rep_weight_name(string)

Variance factor set by suboption variancefactor(#)

Number of replications set by suboption NREP(#)

Number of plausible values set by suboption NBpv(#)

User-defined estimation commands

The syntax of user-defined estimation commands must be such that
all variables required for their execution, except weights, are
declared on the command line, as in the last example below.
User-defined commands must accept aweights or pweights. A typical syntax statement
for user-defined programmes is

. syntax varlist [if] [in] [pweight] [, options]

Examples

Built-in estimation commands: means, summarize, corr, etc.

. repest PIAAC, estimate(means age_r) by(cnt)

. repest TALISTCH, estimate(freq tchagegr) by(cnt)

Combining results

. repest PISA, estimate(summarize escs, stats(p5 p95)) by(cnt) results(combine(escs_length: _b[escs_p95] - _b[escs_p5]))

Using plausible values

. repest PISA, estimate(corr pv@math pv@read pv@scie) by(cnt)

Stata estimation commands: regress, logistic, sureg, etc.

. repest PIAAC, estimate(stata: reg lnwage pvlit@ yrsqual) by(cnt)

. repest PIAAC, estimate(stata: sureg (lnwage pvlit@ yrsqual) (lnwage pvnum@ yrsqual)) by(cnt)

Estimation on pooled country samples

. repest PIAAC, estimate(stata: reg lnwage pvlit@ yrsqual)

Adding estimated scalars to results (e.g. r2)

. repest PIAAC, estimate(stata: reg lnwage pvlit@ yrsqual) by(cnt) results(add(r2))

Requesting averaged results

. repest PIAAC, estimate(stata: reg lnwage pvlit@ yrsqual) by(cnt, levels(ITA FRA DEU) average(FRA DEU))

Saving results to excel

. repest PIAAC, estimate(stata: reg lnwage pvlit@ yrsqual) by(cnt) outfile(wagereturns)

. use wagereturns, clear

. export excel using wagereturns.xls, first(var)

User-defined estimation command: 1. simultaneous weighted quantile regressions

cap program drop mysqreg

program define mysqreg, eclass

syntax varlist [if] [in] [pweight] [, flag Quantiles(numlist)]
version 12.1
// compute quantile regressions, store results in vectors
foreach q in `quantiles' {
tempname q`q'
qreg `varlist' [pw `exp'] `if' `in', quantile(`q')
matrix `q`q'' = e(b)
matrix coleq `q`q'' = q`q'
local results = "`results'" + "`q`q''" + " , "
}
// post results
tempname b
local results = regexr("`results'", "\,.$" ,"")
matrix `b' = [`results']
ereturn post `b'

end

. repest PISA, estimate(stata: mysqreg pv@math escs girl, q(25 75)) by(cnt) results(keep(q25_girl q75_girl) combine(qdiffgirl: _b[q75_girl] - _b[q25_girl]))

User-defined estimation command: 2. logit postestimation

cap program drop mylogitmargins

program define mylogitmargins, eclass

syntax [if] [in] [pweight], logit(string) [margins(string) loptions(string) moptions(string)]
tempname b m
// compute logit regressions, store results in vectors
logit `logit' [`weight' `exp'] `if' `in', `loptions'
matrix `b'= e(b)
// compute logit postestimation, store results in vectors
if "`margins'" != "" | "`moptions'" != ""{
margins `margins', post `moptions'
matrix `m' = e(b)
matrix colnames `m' = margins:
matrix `b'= [`b', `m']
}
// post results
ereturn post `b'

end

. repest PISA, estimate(stata: mylogitmargins, logit(repeat pv@math escs ib1.st04q01) margins(st04q01) moptions(atmeans))

Stored results

repest stores the following in e():

Macros

e(properties) b V

Matrices

e(b) estimator vector

e(V) variance-covariance matrix of the estimators.
Note: If option over() is used, only diagonal (variance)
elements are included in stored results.

Authors

Francesco Avvisati
Directorate for Education and Skills
Organisation for Economic Co-operation and Development (OECD)
francesco.avvisati@oecd.org

Francois Keslair
Directorate for Education and Skills
Organisation for Economic Co-operation and Development (OECD)
francois.keslair@oecd.org

Also see

svy, pisastats, pisareg, piaacreg, piaacdes, piaactab