*! version 1.2.1 21 May 2022

Title

sgpv -- Calculate the Second-Generation P-Value(s)(SGPV) and their associated diagnosis statistics after common estimation commands.

Syntax

sgpv
[subcmd]
[,
options] [: estimation_command]

Main
* estimate(name) takes the name of a previously stored estimation.

* matrix(name) takes the name of matrix as input for the calculation.

coefficient(sgpv##coeflist:coeflist) the coefficients for which the SGPVs and other statistics are calculated.

noconstant do not calculate SGPVs for the constant term.

Null hypothesis
nulllo(boundlist) change the lower bound(s) of the null-hypothesis interval(s).

nullhi(boundlist) change the upper bound(s) of the null-hypothesis interval(s).

Reporting
level(#) set confidence level; default is level(95)

quietly suppress the output of the estimation command.

matlistopt(matlist:options) change the options of the displayed matrix.

deltagap calculate and display the delta gap.

fdrisk calculate and display the false discovery risk.

all calculate and display both, the delta gap and the false discovery risk.

format(%fmt) display format of the results.

nonullwarnings disable warning messages when the default point 0 null-hypothesis is used for calculating the SGPVs.

Fdrisk
truncnormal use truncated normal distribution as probibility distribution for null parameter space.

likelihood(#) use the likehood support interval with level 1/k

pi0(#) prior probability of the null hypothesis.

Menu
permdialog install permanently the dialog boxes into the User menubar.

remove remove the entries created by the option permdialog.

estimation_command can be any of Stata's or user-provided estimation commands which return their results in a matrix named r(table).
This should be true so long the estimation command runs ereturn display at some point.
If you want to calculate SGPVs for an estimation command or any other command which does not follow this convention, then you can use the matrix(matrixname) option. The matrix must be pre-processed to meet the expectations of sgpv.
See the description here for more information.

* ONLY one thing can be used to calculate the SGPVs: an estimation command, a stored estimation result, a matrix with the necessary information or the previous estimation results.

Description

sgpv allows the calculation of the Second-Generation P-Values (SGPV) developed by Blume et.al.(2018,2019) for and after commonly used estimation commands.
The false discovery risk (fdr) can be also reported.
The SGPVs are reported alongside the usually reported p-values.

An ordinary user should use this command and not other commands of this package on which sgpv is based upon.
sgpv uses sensible default values for calculating the SGPVs, the delta-gaps and the accompaning false discovery risks (fdr), which can be changed.
This package comes also with the example leukemia dataset from https://github.com/ramhiser/datamicroarray/wiki/Golub-(1999).

Dialog boxes for each command (including this one) are also provided to make the usage of the commands easier.
The dialog boxes can be installed into the User menubar.
See this example for how to do it.

sgpv typed without any options calculates the SGPVs for the last estimation command with the default settings. All options except for estimate() and matrix() can be used.

For more information about how to interpret the SGPVs and other common questions,
see Frequently Asked Questions by Blume et al. (2018).
An example of how to interpret the result from sgpv can be found in the examples section of this help file.

The formulas for the Second-Generation P-Values can be found here.

The sgpv-package consists of: sgpv - the main command to calculate SGPVs, delta-gaps and false discovery risks after an estimation.

sgpvalue - calculate the SGPVs.

sgpower - power functions for the SGPVs.

fdrisk - false confirmation/discovery risks for the SGPVs.

plotsgpv - plot interval estimates according to SGPV rankings.

Subcommands

It is possible to call the other commands of the sgpv-package with the sgpv-command.
This is mostly a convenience feature so that the help-files for the individual commands should be consulted for the options of these commands.
Supported subcommands are: value, power, risk, plot and menu.
Two examples how to use the subcommands are given here.
Using the individual commands directly is comparable to the immediate form of other Stata commands, like ttesti.

Options

Main

replay is the default behaviour if no estimation command, matrix or stored estimate is set.
The replay-option is only available in the dialog box.
sgpv behaves like any other estimation command (e.g. regress) which replays the previous results when run without a varlist.
Currently, the results from previous runs of sgpv are not used to display the results.
Instead, the results are calculated fresh on every run of sgpv.

To see the results from a previous run of sgpv without recalculation, use the command matlist r(comparison)
if no other commands were run after sgpv.

Only one thing can be used to calculate the SGPVs: an estimation command, the results from the previous estimation command, a stored estimation result or a matrix with the necessary information.

estimate(name) takes the name of a previously stored estimation.

matrix(name) takes the name of matrix as input for the calculation.
The matrix must follow the structure of the r(table) matrix returned after commonly used estimation commands due to the hardcoded row numbers used for identifiying the necessary numbers.
Meaning that the parameter estimate has to be in the 1st row, the standard errors need to be in the 2nd row, the p-values in 4th row, the lower bound in the 5th and the upper bound in the 6th row.
As additional check, the row names of the supplied matrix need to match the rownames of the r(table) matrix.
The rownames are: b se t pvalue ll ul

To the set rownames run: mat rownames your_matrix = b se t pvalue ll ul

Example code is located in the file sgpv-leukemia-example.do which can be viewed here, if installed.
To run the example code, go to the respective example section.

coefficient(sgpv##coeflist:coeflist) allows the selection of the coefficients for which the SGPVs and other statistics are calculated.
The selected coefficients need to have the same names as displayed in the estimation output. If you did not use factor-variable notation, then the names are identical to the variable names.
Otherwise, you have to use the factor-variable notation e.g. 1.foreign if you estimated reg price mpg i.foreign.
Multiple coefficients must be separated with a space.
You can also select only an equation by using "eq:" or select a specific equation and variable "eq:var". See the multiple equations example for an example and the definition of coeflist below.

A coeflist is:
coef [coef ...]
eq:coef [eq:coef ...]
eq: [eq: ...]

noconstant do not calculate SGPVs, delta-gaps and Fdrs for the constant term. The constant term is also removed from the list of coefficients if the coefficient-option is used and only equations are specified.

Null-Hypothesis

nulllo(boundlist) change the lower bound of the null-hypothesis interval.
The default is 0 (the same bound as for the usually reported p-values).
Missing values, strings and variable names are not allowed.
Expressions/formulas are also allowed as input.
More than one null-hypothesis is also supported. Each lower bound must be separated with a space.
The number of lower bounds must match the number of arguments set in the coefficient-option.
The number of lower and upper bounds must also match.
See these examples for a demonstration.

nullhi(boundlist) change the upper bound of the null-hypothesis interval.
The default is 0 (the same bound as for the usually reported p-values). Missing values, strings and variable names are not allowed.
Expressions/formulas are also allowed as input.
More than one null-hypothesis is also supported. Each upper bound must be separated with a space.
The number of upper bounds must match the number of arguments set in the coefficient-option.
The number of lower and upper bounds must also match.
See these examples for a demonstration.

A boundlist is:

# [# ...]

The default value 0 is just meant to be used for an easier beginning when starting to use SGPVs.
Please change this value to something more reasonable. Reasonable lower or upper bounds depend on your dataset and your research question.
Using this default value will always result in having SGPVs of value 0 or 0.5!

Reporting

level(#) set the confidence level, as a percentage, for the confidence interval(s).
The default is level(95) or as set by set level. See also [R] estimation options.
This option overwrites the same named option of an estimation command.
A warning is displayed in the beginning if this happens.

quietly suppress the output of the estimation command.

nonullwarnings disable warning messages when the default point 0 null-hypothesis is used for calculating the SGPVs.
You should disable these warning messages only if using the default point 0 null-hypothesis is
what you want to do and you understand the consequences of doing so.

matlistopt(matlist:options) change the format of the displayed matrix. The same options as for matlist can be used.

deltagap calculate and display the delta-gap if the SGPV is 0.

fdrisk calculate and display the false discovery risk if the SGPV is 0.

all calculate and display both, the delta-gap and the false discovery if the SGPV is 0.
This options takes precedence over the options deltagap and fdrisk.
Using both options together has the same effect as using this option alone.

format(%fmt) specifies the format for displaying the individual elements of the result matrix. The default is format(%5.4f).
This option is NOT identical to the same named option of matlist, but works independently of it.
Setting the format option via matlistopt() overrides the setting here and also changes the format of the column names.

The following options are only needed for the calculations of the False Discovery Risk (fdr).
More information about each option can be found in the help for fdrisk.

truncnormal uses the truncated normal distribution as the probability distribution for the null and alternative parameter space.
The default is to use the uniform distribution as the probability distribution for the null and alternative parameter space.
The mean and standard deviation of the distribution are automatically set based on the estimated coefficient.

likelihood(#) use a 1/k likelihood support interval (LSI) instead of a (1-α)100% confidence interval to calculate the Fdr.
The level is 1/k (not k).
This option works only in combination with the option matrix() for a user supplied matrix.
The level should be set equal to the level of the LSI which was used to calculate the lower and upper bound of the estimated coefficients.
For technical reasons, a fraction like 1/8 as the level must be converted to a real number like 0.125 first, when used for this option.
No official Stata command reports likelihood support intervals yet.

pi0(#) prior probability of the null hypothesis. Default is 0.5. This value can be only between 0 and 1 (exclusive).
A prior probability outside of this interval is not sensible.
The default value assumes that both hypotheses are equally likely.

Menu

permdialog install permanently the dialog boxes into the User menubar.
The necessary commands are added to the user's profile.do.
If no profile.do exists or can be found then a new profile.do is created in the current directory.
Currently, a profile.do will be only be found in the current directory and in the Stata installation base folder.
Other possible places like the user's home folder are not accessed yet.
See profile to get more information about how to setup the profile.do.
Without this option, the dialog boxes will only available from the menubar until the next restart of Stata.
The dialog boxes can be accessed as usual by for example db sgpv.

remove remove the entries created by the option permdialog from the profile.do file.
A backup of the original file is kept with the name profile.do.bak.
ONLY tested under Windows, other operation systems should work but could not be tested.
The profile.do file should not contain a line like "global F4 `" or like "global F5 '".
Otherwise, the option returns an error and will not delete the menu entries.
Also make sure that profile.do is not open in another program nor
that the files profile.do.bak or profile.do.new already exist.
If they exist, you can delete them.

As an alternative to this subcommand, open the profile.do file by running the following two commands and delete manually the lines added by the permdialog-option:

. findfile profile.do, path(STATA;.)

. doedit "`r(fn)'"

Examples

Contents

Usage of sgpv as a prefix-command

How to interpret results

Exporting results

Using stored estimations

Calculating for SGPVs with different levels of confidence

Setting a different null-hypothesis

Setting an individual null hypotheses for each coefficient

Selecting coefficients

Calculating SGPVs for a large dataset of estimation or t-test results

Using subcommands

Usage of sgpv as a prefix-command:

. sysuse auto, clear

. sgpv, all: regress price mpg weight foreign

Example Output:

Source | SS df MS Number of obs = 74
-------------+---------------------------------- F(3, 70) = 23.29
Model | 317252881 3 105750960 Prob > F = 0.0000
Residual | 317812515 70 4540178.78 R-squared = 0.4996
-------------+---------------------------------- Adj R-squared = 0.4781
Total | 635065396 73 8699525.97 Root MSE = 2130.8

------------------------------------------------------------------------------
price | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-----------+------------------------------------------------------------------
mpg | 21.8536 74.22114 0.29 0.769 -126.1758 169.883
weight | 3.464706 .630749 5.49 0.000 2.206717 4.722695
foreign | 3673.06 683.9783 5.37 0.000 2308.909 5037.212
_cons | -5853.696 3376.987 -1.73 0.087 -12588.88 881.4934
------------------------------------------------------------------------------

Comparison of ordinary P-Values and Second Generation P-Values for a point Null-Hypothesis of 0

Variables | P-Value SGPV Delta-Gap Fdr
-------------+--------------------------------------------
mpg | .7693 .5 . .
weight | 0 0 2.2067 .0479
foreign | 0 0 2300 .048
_cons | .0874 .5 . .

Interpretation example:

There is inconclusive evidence for an effect of mpg on price, while there is no evidence for the null-hypothesis of no effect for weight and foreign.
Remember that the null-hypothesis is an interval of length 0 with both lower and upper bounds being also 0.
This is the standard null-hypothesis of no effect.
You will usually have a more relastic interval which is larger than 0 due to measurement errors, scientific relevance, etc.
The delta-gap can be used to compare two different studies for the same model/estimation when both report second-generation p-values of zero (p_δ = 0).
So for this example, the delta-gap is not needed, but it still provides information about the distance between either the upper or the lower bound of the confidence interval and 0.
Finally, the False Discovery Risk (Fdr) tells you that there is a roughly a 5% chance that the null-hypothesis is true although you calculated a second-generation p-value of zero (p_δ = 0).
Whether 5% is much, is for you to decide.

NB: Note that the given interpretations are based on my understanding of Blume et.al (2018,2019).
I cannot guarantee that my understanding is correct.
These interpretations are just meant as examples how to make sense out of the calculated numbers, but not meant as a definitive answer.

Exporting results

You can export the results with the help of Ben Jann's estout-command. If you have not installed it yet, then you can do so by clicking here.

. postrtoe // Transfer the matrix r(comparison) to e(comparison) to make repeat usage of estout easier.

. estout e(comparison) // Display the results.

. estout e(comparison) using sgpv-results.tex, style(tex) replace // Export results for later use in a LaTeX-document.

Using stored estimations

. qui sgpv, all: regress price mpg weight foreign // Re-run the previous estimation

Save estimation for later usage

. estimate store reg

The same result but this time after the last estimation.

. sgpv

Now run a quantile regression instead

. qreg price mpg weight foreign

. estimates store quantile

Calculate sgpvs for the stored estimation and only the foreign coefficient

. sgpv, estimate(reg) coefficient("foreign")

. sgpv, estimate(quantile) coefficient("foreign")

Calculating for SGPVs with different levels of confidence

Usually the SGPVs are calculated for the standard 95% confidence interval, but other confidence levels are also possible.

At first the SGPVs are calculated for a 99% confidence interval:

. sgpv, all level(99): regress price mpg weight foreign

Then for the stored quantile regression results also for 99% confidence level

. sgpv, estimate(quantile) level(99) all

Setting a different null-hypothesis

Set an alternative null-hypothesis -> 1% of the mean value of the price variable (-62, 62) and remove the constant from the calculations

. sgpv, all nulllo(-62) nullhi(62) quietly noconstant: regress price mpg weight foreign

Comparison of ordinary P-Values and Second Generation P-Values for an interval Null-Hypothesis of [-62,62] based on a 95% confidence interval

Variables | P-Value SGPV Delta-Gap Fdr
-------------+--------------------------------------------
mpg | .7693 .5 . .
weight | 0 1 . .
foreign | 0 0 36.2405 .0394

The SGPV for the weight-coefficient has changed from 0 to 1 while the P-Value remained the same compared to the default point 0 null-hypothesis.
The example illustrates the need to set a scientifically reasonable null-hypothesis.
For the weight-coefficient, the null-hypothesis of [-62,62] is probably too wide.

Setting an individual null hypotheses for each coefficient

To set a separate/different null-hypothesis for each coefficient, you need to separate the individual lower or upper bounds of the null hypotheses with a space.
The number of coefficients set in the coefficient-option needs to match the number of lower and upper bounds set in the nulllo and nullhi-options.

. sgpv, coefficient(mpg weight foreign) nulllo(20 2 3000) nullhi(40 4 6000) quietly: regress price mpg weight foreign

The same null hypotheses but this time one null-hypothesis for each selected equation or quantile:

. sgpv, coefficient(q10: q50: q90:) nulllo(20 2 3000) nullhi(40 4 6000) quietly: sqreg price mpg rep78 foreign weight, q(10 25 50 75 90)

Selecting coefficients

Calculate sgpvs for a multiple equation estimation command and select coefficients

. sqreg price mpg rep78 foreign weight, q(10 25 50 75 90)

Select only the foreign coefficient for sgpv calculation

. sgpv, coefficient(foreign)

Select only the 10%, 50% and 90% quantile equation for sgpv calculation

. sgpv, coefficient(q10: q50: q90:)

Select only the 50% quantile equation and foreign coefficient for sgpv calculation

. sgpv, coefficient(q50:foreign)

Calculating SGPVs for a large dataset of estimation or t-test results

For this example you need to install the ancillary files of this package by by clicking here which will install the example leukemia dataset together with other examples.
The example leukemia dataset can be used to show how the SGPVs can be calculated for a large dataset which contains the information usually returned in the r(table) matrix.
The leukemia dataset contains 7218 gene specific t-tests for a difference in mean expression.
More information about the dataset are in the dataset itself: Use sysuse leukstats,clear and notes to access this information.
The example file below will calculate the SGPVs and bonus statistics for the leukemia dataset.
You can view the code.

. do sgpv-leukemia-example.do

This example code is rather slow on my machine and demonstrates some ways around the current limitations of the program code.
Should your maximum matrix size be higher than the number of observations in the dataset (7128), then the example code should run faster.
You can run display c(matsize) to see your current setting.

Subcommands examples

The subcommands can be used in case you want to use only one command instead remembering the names of the other commands of this package:

. sgpv value, estlo(log(1.3)) esthi(.) nulllo(.) nullhi(log(1.1))

. sgpv power, true(2) nulllo(-1) nullhi(1) stderr(1)

Install the dialog boxes permanently in the User menubar: User -> Statistics

. sgpv menu, permdialog

Remove the installed dialog boxes (permanently) from the User menubar:

. sgpv menu, remove

Stored results

sgpv stores the following in r():

Scalars

r(level) confidence level

Matrices

r(comparison) a matrix containing the displayed results

r(table) coefficient statistics

Formulas & Interpretation

The formulas below are taking from the help-files of sgpvalue and fdrisk.
An example about how to interpret the results is given in the example section.

The SGPV is defined as : p_δ = |I ∩ H_0|/|I|*max{|I|/(2|H_0|), 1}
= |I ∩ H_0|/|I| when |I|<=2|H_0|
= 1/2*|I ∩ H_0|/|H_0| when |I|> 2|H_0|
with I = {θ_l,θ_u} and |I|= θ_u - θ_l.

θ_u and θ_l are typically the upper and lower bound of a (1-α)100% confidence interval but any other interval estimate is also possible.

H_0 is the null hypothesis and |H_0| its length.

The null hypothesis should be an interval which contains all effects that are not scientifically relevant.

The p-values reported by most of Stata's estimation commands are based on the null hypothesis of a parameter being exactly 0.

Point null-hypothesis are supported by SGPVs, but they are discouraged.
See answer 11 in the Frequently asked questions to Blume et al. (2018).
You could set a small null-hypothesis interval which includes effects of less than 1% or 0.1%. The exact numbers depend on what you deem a priori as not scientifically relevant.

p_δ lies between 0 and 1.

A p_δ of 0 indicates that 0% of the null hypotheses are compatible with the data.

A p_δ of 1 indicates that 100% of the null hypotheses are compatible with the data.

A p_δ between 0 and 1 indicates inconclusive evidence.

A p_δ of 1/2 indicates strictly inconclusive evidence.

For more information about how to interpret the SGPVs and other common questions,
see Frequently Asked Questions by Blume et al. (2018).

The delta-gap is have a way of ranking two studies that both have second-generation p-values of zero (p_δ = 0). It is defined as the distance between the intervals in δ units with δ being the half-width of the interval null hypothesis.

The delta-gap is calculated as: gap = max(θ_l, H_0l) - min(H_0u, θ_u)
delta = |H_0|/2
delta.gap = gap/delta

For the standard case of a point 0 null hypothesis and a 95% confidence interval, delta is set to be equal to 1.
Then the delta-gap is just the distance between either the upper or the lower bound of the confidence interval and 0.
If both θ_u and θ_l are negative then, the delta-gap is just θ_u, the upper bound of the confidence interval.
If both bounds of the confidence interval are positive, then the delta-gap is equal to the lower bound of the confidence interval.

The false discovery risk is defined as: P(H_0|p_δ=0) = (1 + P(p_δ = 0| H_1)/P(p_δ=0|H_0) * r)^(-1)

The false confirmation risk is defined as: P(H_1|p_δ=1) = (1 + P(p_δ = 1| H_0)/P(p_δ=1|H_1) * 1/r )^(-1)

with r = P(H_1)/P(H_0) being the prior probability.

See equation(4) in Blume et.al.(2018)

References

Blume JD, D’Agostino McGowan L, Dupont WD, Greevy RA Jr. (2018). Second-generation p-values: Improved rigor, reproducibility, & transparency in statistical analyses. PLoS ONE 13(3): e0188299.
https://doi.org/10.1371/journal.pone.0188299

Blume JD, Greevy RA Jr., Welty VF, Smith JR, Dupont WD (2019). An Introduction to Second-generation p-values. The American Statistician. In press. https://doi.org/10.1080/00031305.2018.1537893

Author

Sven-Kristjan Bormann, School of Economics and Business Administration, University of Tartu.

Bug Reporting

Please submit bugs, comments and suggestions via email to:	sven-kristjan@gmx.de

Further Stata programs and development versions can be found under https://github.com/skbormann/stata-tools

See Also

Related commands:
plotsgpv, sgpvalue, sgpower, fdrisk