*! version 1.5.0 25Mar2013

*! version 1.2.1 02Aug2012

*! version 1.1.0 19Jun2012

*! version 1.0.0 18Jun2012
help simpplot

Title

simpplot -- Plot describing p-values from a simulation by
comparing nominal significance levels with the coverages

Syntax

simpplot
varlist
[if] [in]
[,
options
]

options description

main#opt(graph_opts ) options governing the look the #th variable
ra(off | graph_opts) options governing the look of the Monte
Carlo region of acceptance
level(#) set confidence level for the Monte Carlo
region of acceptance; default is level(95)
overall displays a region of acceptance with an
overall error rate of level. The default is
to display a region of acceptance with a
point-wise error rate of level
reps(#) number of replications used for computing
the overall error rate. the default is 1000.
ref0(off | graph_opts) options governing the look of a reference
line
nodeviations displays the observed coverage against
nominal significance levels; default is to
display the deviations from the nominal
significance level against the nominal
significance level
by(varlist [, byopts]) option for repeating simpplot command
generate(newvars) specifies that the deviations or observed
coverages and optionally the x-coordinate,
upper and lower bound of the Monte Carlo
region are to be stored
addplot(plot) add other plots to the graph
other graph options

Description

simpplot describes the results of a simulation that inspects the coverage of
a statistical test. If a null-hypothesis is true than a statistical test at a 5%
siginificance level should reject the true null hypothesis in 5% of the
replications. In a simulation we can make sure that the null hypothesis is true
in the "population", draw many times from that "population", perform the test in
each of these "samples", and count how many times the null hypothesis is rejected.
The proportion of replications in which the hypothesis was rejected is our
estimate of the coverage of the test, and this coverage should be close to the
nominal siginficance level. There is nothing special about the 5% significance
level other than that it is the conventional level in many disciplines. We could
just as easily look at the coverage of tests at a 1%, 2%, 10%, or any other
significance level. simpplot displays by default the deviations from the
nominal significance level against the entire range of possible nominal
significance levels. It also displays the range (Monte Carlo region of acceptance)
within which we can reasonably expect these deviations to remain if the test is
well behaved.

If a test is well behaved than the p-values of that test of a true null-hypothesis
should follow a standard uniform distribution. That way 1% of the replication will
have a p-value less than 0.01, 5% of the replications will have a p-value less than
0.05, 10% will have a p-value less than 0.10, etc. So whichever significance level
we choose, we will on average reject the null hypothesis the right number of times.
One could see the graph produced by simpplot as a graphical test whether the
distribution of p-values follows a standard uniform distribution.

Even if the test we are checking with our simulaiton works perfectly, we would
still expect slight deviations in the distribution of the p-values from a standard
uniform distribution because of the randomness in a Monte Carlo experiment. To
quantify the amount of this uncertainty simpplot by default also displays a
point-wise Monte Carlo region of acceptance. This gives for each nominal significance
level the range in which we would expect most simulations (by default 95%) to be if
we were repeating our simulation experiment many times and we were simulating a well
behaved test. If we did a simulation with n replication, than for a given
nominal significance level a we would expect the number of rejections to follow
a binomial(n,a) distribution. This is what is being used to compute the
end points of the point-wise region of accpetance.

A 95% point-wise region of acceptance means that 5% of the replications of our Monte
Carlo experiment is expected to lie outside the point-wise region of acceptance. So,
the chance that the entire set of replications lies within the point-wise region of
acceptence is a lot less than the nominal 95%. simpplot can also display an
approximate overall region of acceptance, that is the region in which we expect 95%
of the curves to remain if we were repeating the Monte Carlo experiment many times
using an algorithm discussed by Davison and Hinkley (1997, Chapter 4).

1. Define a grid of nominal significance level consisting of 300 equally spaced values
between .001 and .999, i.e. .001, .0043, .0077, ... .992, .996, .999. Take reps
samples of the same size from a standard uniform distribution. For each sample from the
uniform distribution and each value on the grid, compute the proportion of that sample
that is less than or equal to that grid value.

2. Order each sample from smallest to largest.

3. Set L to ceil((100 - level)/200 * reps).

4. For each sample, create an envelope using the remaining samples by
storing for each grid value the Lth and (reps - L)th smallest value, and determine
whether the entire sample falls within this envelope. The proportion of
samples for which this is not true is an estimate of the overall error rate
of that envelope.

5. Decrease L until the overall error rate is less than 100 - level. If
the nominal level has not been reached when L = 1, then the entire range is
returned and a warning is displayed reporting the approximate overall error
rate for that envelope.

6. For each value on the grid, calculate reference intervals using the Lth and
(1-L)th sample value.

options

main#opt(graph_opts) options governing the look of the #th variable specified
in varlist. the relevant options are listed in twoway_scatter.

ra(off | graph_opts) options governing the look of the Monte Carlo region of
acceptance. One can suppres the display (and computation) of the Monte Carlo region
of acceptance by specifying ra(off). Alternatively, one can specify options
that change the look of the region of acceptance. The relevant options are listed
in twoway_rarea.

level(#) set confidence level for the Monte Carlo region of
acceptance; default is level(95)

overall specifies that the approximate overall Monte Carlo region of acceptance
is displayed instead of the default point-wise Monte Carlo region of acceptance.

reps(#) specifies the number of samples used to compute the overall Monte Carlo
region of acceptance. The default is reps(1000), which is often not enough. The
overall option needs to be specified when specifying the reps(#) option.

ref0(off | graph_opts) options governing the look of a reference line. By
default this is a horizontal line at 0. If the nodeviations option is specified
than the reference line is a diagonal line from 0,0 to 1,1. One can suppres the
reference line by specifying ref0(off). Alternatively, one can specify
options than change the look of the reference line. The relevant options are listed
in twoway_line.

nodeviations displays the observed coverage against nominal significance levels;
default is to display the deviations from the nominal significance level against
the nominal significance level.

by(varlist [, byopts]) Option by() draws separate plots
within one graph. The byopts are documented in by_option.

generate(newvars) specifies that the deviations (the default) or coverages
(when the nodeviations option has been specified) for each variable in varlist
is to be stored. If the number of newvars is the number of variables in
varlist + 3, than the x-coordinate, the upper and lower bound of the Monte Carlo
region of acceptance are also stored.

So if one uses simpplot to display k p-values, than one can specify either
k or k + 3 newvars. The first k newvars will contain the
deviations or coverages of the corresponding p-value in varlist. The k+1th
newvar will contain the x-coordinate for the Monte Carlo region of acceptance, the
k+2th and k+3th will contain respectively the lower and upper bound of that
area.

addplot(plot) allows adding more graph twoway plots to the graph

other graph options

Examples

In this example we test how well a t-test performs when the data is from a
non-Gaussian (some prefer to say non-normal) distribution, in this case a
Chi-square distribution with 2 degrees of freedom. We know that the mean of that
distribution is 2, so a t-test whether the mean equals 2 is a test of a true
null-hypothesis. We also look at how well this test performs in different sample
sizes. In a sample size of 50 the test does not perform too well, the true
null-hypothesis is too often rejected, but a sample size of a 500 seems already
big enough for this test to work.

program drop _all
program define sim, rclas
drop _all
set obs 500
gen x = rchi2(2)

ttest x=2 in 1/50
return scalar p50 = r(p)

ttest x=2
return scalar p500 = r(p)
end

set seed 12345
simulate p50=r(p50) p500=r(p500), ///
reps(5000) : sim

label var p50 "N=50"
label var p500 "N=500"

simpplot p50 p500, main1opt(mcolor(red)) ///
main2opt(mcolor(blue))

Author

Maarten L. Buis

Wissenschaftszentrum Berlin für Sozialforschung, WZB

Research unit Skill Formation and Labor Markets

maarten.buis@wzb.eu

Reference

Davison, A.C. and Hinkley, D.V. 1997.
Bootstrap methods and their application.
Cambridge: Cambridge University Press.

Also see:

if installed: simsum, qenvnormal, qenvchi2, qenvF