*! version 1.0.0 03mar2026

Title

xtmispanel -- Comprehensive Missing Data Detection, Imputation and Diagnostics for Panel Data

Syntax

1. Detection and testing (multiple variables allowed)

xtmispanel [varlist] [if] [in]
[, detect test]

2. Imputation (exactly one variable required)

xtmispanel varname [if] [in]
, impute(method)
[generate(newvar) replace
knn(#) mice(#)]

3. Sensitivity analysis (exactly one variable required)

xtmispanel varname [if] [in]
, sensitivity
[methods(method_list)
knn(#) mice(#)]

4. Visualization (multiple variables allowed)

xtmispanel [varlist] [if] [in]
, graph
[impvar(varname)]

5. Combined usage

xtmispanel [varlist] [if] [in]
, detect test

Prerequisites: Data must be declared as panel data using xtset
before using xtmispanel. The panel variable and time variable are
automatically detected from xtset.

Default behavior: If no option is specified, detect is assumed.

Variable list: If no varlist is specified, all numeric variables
(excluding the panel and time identifiers) are analyzed.

Description

xtmispanel is a comprehensive, all-in-one Stata command for handling
missing values in panel (time-series cross-sectional) data. It implements
the complete missing data workflow:

Step 1. Detect and summarize missingness patterns

Step 2. Test the missing data mechanism (MCAR, MAR, MNAR)

Step 3. Choose and apply an imputation method

Step 4. Validate via sensitivity analysis

Step 5. Visualize results with publication-quality graphs

The command is designed specifically for panel data and respects the panel
structure in all computations (panel-specific means, within-panel
interpolation, panel-aware regression, etc.).

Key features:

o 4 formatted detection tables (by variable, panel, time, pattern)

o Little's MCAR test + logistic MAR tests + pattern classification

o 13 imputation methods from simple to ML-inspired

o Multi-method sensitivity analysis with automatic recommendation

o 8 publication-quality diagnostic graphs

o All imputation is panel-aware (respects panel structure)

o Imputed variable is added to the dataset; original is never modified

Options

Module 1 — Detection

detect displays four formatted summary tables:

Table 1: Per-variable summary — N total, N missing, % missing,
mean, SD, and a severity status (Complete / Low / Moderate / High / Severe).

Table 2: Per-panel summary — N observations, N missing, % missing,
number of gaps, maximum gap length, and status.

Table 3: Per-time-period summary — N missing, % missing, and an
ASCII visual bar showing severity for each period.

Table 4: Missing data pattern co-occurrence matrix showing which
combinations of variables have missing values simultaneously. Shows up to
10 variables. Pattern '.' = observed, 'X' = missing.

This is the default action if no option is specified.

Module 2 — Testing

test runs three diagnostic tests for the missing data mechanism:

Test 1: Little's MCAR Test (Approximate). Compares means of each
variable across groups defined by missingness patterns of other variables.
Reports chi-square statistic, degrees of freedom, and p-value.
H0: Data are MCAR (p >= 0.05 = fail to reject).

Test 2: MAR Logistic Regression Test. For each variable with
missing values, regresses its missingness indicator (0/1) on all other
variables. Reports chi2, p-value, Pseudo-R2, and conclusion (MCAR or MAR)
per variable.

Test 3: Pattern Classification. Determines whether the
missingness pattern is monotone (once missing, stays missing) or
arbitrary (irregular gaps). Monotone patterns allow sequential
imputation; arbitrary patterns favor MICE.

Overall Recommendation. Combines all test results to classify the
mechanism as MCAR, MAR, or possibly MNAR, with method recommendations.

Module 3 — Imputation

impute(method) imputes missing values in the specified variable using
the given method. Exactly one variable must be specified.

Output after imputation:

o Fill report: missing before, values imputed, still missing, fill rate

o Before vs After comparison table: N, Mean, SD, Min, Max with Delta and Delta%

o Pearson correlation between original and imputed (on observed pairs)

o Auto-generated density overlay graph (xtmis_impute_density) comparing distributions

generate(newvar) specifies the name for the imputed variable.
If not specified, the default name is varname_imp.

replace allows overwriting an existing variable with the same name
as the generated imputed variable. Without this option, the command will
exit with an error if varname_imp already exists.

knn(#) specifies the number of nearest neighbors for the knn
method. Default is 5. Higher values produce smoother imputations
but may over-smooth. Recommended range: 3–10.

mice(#) specifies the number of multiple imputations for the mice
method. Default is 5. More imputations improve precision but
increase computation time. Recommended range: 5–20.

Important: The imputed variable is added to your dataset in memory.
The original variable is never modified. To save permanently,
use save after imputation. See Important Notes.

Module 4 — Sensitivity

sensitivity runs all (or selected) imputation methods on the specified
variable and produces a comparison table showing:

— Mean, SD, min, max of the imputed series

— Number of observations filled

— Percentage change in mean from the original (observed) distribution

The method with the smallest absolute mean change is automatically recommended.
Color coding: green (< 1%), yellow (1–5%), red (> 5%).

methods(method_list) specifies a subset of methods for sensitivity
analysis. Separate methods with spaces. Default uses all 13 methods:
mean median locf nocb linear spline regress pmm hotdeck knn rf em mice.

Module 5 — Visualization

graph generates 8 publication-quality diagnostic graphs. All graphs
are stored in Stata memory and can be displayed or exported after the command.
See Graphs for the full list.

impvar(varname) specifies a pre-existing imputed variable for the
density overlay graph (Graph 6). If omitted, the command automatically
imputes the first variable using linear interpolation to generate the
density comparison.

Imputation Methods

xtmispanel supports the following 13 panel-aware imputation methods:

Simple Methods

mean Panel-specific mean imputation. Replaces missing values with
the mean of observed values in the same panel. Falls back to the global mean
if the entire panel is missing. Fast but ignores trends.

median Panel-specific median imputation. Same as mean but uses the
median, which is more robust to outliers.

locf Last Observation Carried Forward. Fills each missing value
with the most recent non-missing value in the same panel. Runs up to 10
passes for consecutive gaps. Cannot fill leading missings (first
observations); those are left as missing.

nocb Next Observation Carried Backward. Like LOCF but fills
backward from the next available observation. Falls back to LOCF + panel
mean for any remaining values. Ensures 100% fill rate.

Interpolation Methods

linear Linear interpolation within each panel using Stata's
ipolate command. For leading/trailing missings outside the observed
range, falls back to LOCF and NOCB. Best for smooth trends.

spline Cubic spline interpolation with extrapolation using
ipolate, epolate. Produces smoother curves than linear for
nonlinear trends. Falls back to linear for any remaining values.

Model-Based Methods

regress Panel regression imputation. Fits a fixed-effects
regression (xtreg, fe) of the variable on time, predicts missing
values, and adds a stochastic component from the residual variance.
Falls back to OLS with panel dummies if FE fails.

pmm Predictive Mean Matching. Fits a regression, then for each
missing value finds the observed value whose predicted value is closest
(the "donor"). Preserves the observed distribution better than regression.
Falls back to panel mean if regression fails.

hotdeck Temporal hot-deck imputation. Uses the nearest observed
value in time within the same panel (like LOCF/NOCB) and adds a small
random perturbation (5% of panel SD) for stochastic variation.

ML-Inspired Methods

knn K-Nearest Neighbor imputation. For each missing observation,
finds the k nearest observations in time within the same panel and
fills with their mean. Controlled by the knn(#) option. Default k=5.

rf Random forest-style iterative regression. Initializes with
panel mean, then iterates 5 rounds of regression using lag, lead, time,
and panel dummies as predictors. Updates only missing values each round.
Captures nonlinear temporal patterns.

em Expectation-Maximization algorithm. Iterates 20 rounds between
M-step (fit regression on current complete data) and E-step (update missing
values using conditional expectation). Uses damped updates (70/30 blend)
for convergence stability.

mice Multiple Imputation by Chained Equations. Creates multiple
(default 5) imputed datasets using regression with random draws, each
initialized with mean + noise. Chains 3 rounds of equations per imputation.
Final values are Rubin's pooled point estimates (average across imputations).
Controlled by mice(#).

Which method to use?

Mechanism Suggested Methods

MCAR Any method is acceptable. Simplest: mean, linear, locf

MAR Model-based: mice, pmm, regress, knn, em

MNAR Run sensitivity across multiple methods. Consider selection models.

Smooth trends Interpolation: linear, spline

Block missing Temporal: locf, nocb, hotdeck

Best overall Run xtmispanel var, sensitivity and follow the recommendation

Modules

xtmispanel consists of 5 integrated modules:

# Module

1 Detection — 4 formatted summary tables (by variable, panel, time, pattern)

2 Testing — MCAR test, MAR tests, pattern classification, recommendation

3 Imputation — 13 methods; creates new variable in dataset

4 Sensitivity — compares all methods; recommends lowest distortion

5 Visualization — 8 publication-quality Stata graphs

Modules can be combined in a single call (e.g., detect test) or run
separately. Module 3 (imputation) and Module 4 (sensitivity) require exactly
one variable; all other modules accept multiple variables.

Graphs

The graph option generates 8 diagnostic graphs:

Graph Name Description

xtmis_heatmap Missing data heatmap (panel x time). Green squares =
observed, red squares = missing. Shows the spatial-temporal pattern of
missingness at a glance.

xtmis_barvar Horizontal bar chart showing % missing per variable.
X-axis always starts at 0. Color indicates severity.

xtmis_barpanel Bar chart showing % missing per panel unit across
all variables. Identifies which panels have the most missing data.

xtmis_bartime Area chart showing % missing over time across all
panels and variables. Reveals whether missingness is concentrated in specific
periods (e.g., economic crises, data collection gaps).

xtmis_pattern Pattern frequency bar chart showing how many
observations share each unique missingness pattern. Pattern rank 1 is the
most common pattern (usually "all observed"). Limited to 20 patterns and
variables <= 15.

xtmis_density Kernel density overlay comparing the distribution of
the original (observed-only) data versus the complete (imputed) data.
If no impvar() is specified, auto-imputes using linear interpolation.
Useful for checking whether imputation distorts the distribution.

xtmis_timeline Scatter plot with circles (observed) and X marks
(missing) for each panel-time cell. Alternative to the heatmap.

xtmis_combined 4-panel combined diagnostic dashboard containing
the heatmap, barvar, bartime, and pattern graphs.

xtmis_impute_density Auto-generated density overlay after imputation.
Compares original observed distribution with completed (imputed) distribution.
Created automatically when using impute().

Viewing and exporting graphs:

. graph display xtmis_heatmap

. graph display xtmis_combined

. graph export xtmis_heatmap.png, name(xtmis_heatmap) replace width(1200)

. graph export xtmis_combined.pdf, name(xtmis_combined) replace

All graphs use white backgrounds, clean typography, and professional color
palettes suitable for journal submissions.

Stored results

xtmispanel, detect stores the following in r():

Scalars

r(total_missing) total missing values across all variables

r(total_obs) total observation-variable pairs

r(overall_pct) overall percentage missing

r(n_panels) number of panels

r(n_vars) number of variables analyzed

xtmispanel, test stores the following in r():

r(mcar_chi2) MCAR test chi-square statistic

r(mcar_df) MCAR test degrees of freedom

r(mcar_pval) MCAR test p-value

r(mcar_result) REJECT or FAIL_TO_REJECT

Macros

r(mechanism) classified mechanism: MCAR, MAR, or MNAR

xtmispanel, impute() stores the following in r():

r(n_missing) number of missing values before imputation

r(n_imputed) number of values successfully imputed

r(n_remain) number still missing after imputation

r(orig_mean) mean of original observed values

r(orig_sd) SD of original observed values

r(imp_mean) mean of imputed (complete) variable

r(imp_sd) SD of imputed (complete) variable

r(correlation) Pearson correlation between original and imputed

Macros

r(method) imputation method used

r(imputed_var) name of generated variable

xtmispanel, sensitivity stores the following in r():

r(best_method) recommended method name

r(best_dmean_pct) smallest mean change percentage

Matrices

r(sensitivity) matrix of results (methods x 5 statistics)

Using stored results:

. xtmispanel GDP, impute(linear)

. display "Imputed " r(n_imputed) " values using " r(method)

. display "Fill rate: " (r(n_imputed)/r(n_missing))*100 "%"

. xtmispanel GDP, sensitivity

. display "Best method: " r(best_method)

. matrix list r(sensitivity)

Important Notes

How imputed variables work

1. The imputed variable is added to your current dataset in memory as a
new variable. The original variable is never modified.

2. By default, the imputed variable is named varname_imp. You can
customize this with generate(newvar).

3. To save the imputed variable permanently, you must save the dataset
after imputation:

. xtmispanel GDP, impute(linear)

. save mydata_complete.dta, replace

4. Use replace to overwrite an existing imputed variable (e.g., to try
a different method):

. xtmispanel GDP, impute(linear)

. xtmispanel GDP, impute(mice) replace

(overwrites GDP_imp with MICE results)

Sensitivity analysis

5. The sensitivity module creates temporary imputed variables for each
method. These are automatically dropped. Only the comparison table and
recommendation are retained.

6. The "best" method is chosen based on the smallest absolute mean change
from the original distribution. This is a simple criterion; users should
also consider SD, range, and theoretical appropriateness.

Data requirements

7. Data must be xtset before using xtmispanel. The command
automatically detects the panel and time variables.

8. The command works with both balanced and unbalanced panels.

9. At least 2 non-missing observations per panel are needed for most
imputation methods to produce meaningful results.

10. For mice and rf, computation time increases with panel count.
For very large datasets (>10,000 obs), consider using a subset of methods
in the sensitivity analysis.

Mechanism tests

11. The MCAR test is an approximation of Little's (1988) test using
pairwise mean comparisons. It is not identical to the full multivariate
version but provides a practical diagnostic.

12. The MAR test uses logistic regression. If the regression does not converge
(e.g., perfect prediction), that variable is reported as "Failed".

13. These tests cannot distinguish between MAR and MNAR. If the mechanism
is classified as "possibly MNAR", consider domain-specific knowledge and
running sensitivity analysis across multiple methods.

Graphs

14. All graphs are stored in memory with nodraw. Use
graph display to view and graph export to save.

15. The density overlay (Graph 6) automatically generates an imputed version
using linear interpolation if no impvar() is specified. To compare
with a specific imputed variable, use impvar(varname).

16. The pattern plot (Graph 5) is limited to datasets with <= 15 variables to
avoid excessive pattern counts.

17. After modifying the .ado files, run program drop _all and
discard to force Stata to reload the updated programs.

Recommended Workflow

The recommended workflow for handling missing data in panel data:

Step 1. Diagnose

. xtmispanel varlist, detect test

Examine the four tables and mechanism test results.

Step 2. Visualize

. xtmispanel varlist, graph

. graph display xtmis_combined

Check the heatmap and time trend for patterns.

Step 3. Compare methods

. xtmispanel varname, sensitivity

See which method minimizes distributional distortion.

Step 4. Impute

. xtmispanel varname, impute(recommended_method)

The imputed variable is added to the dataset.

Step 5. Validate

. xtmispanel varlist, graph impvar(varname_imp)

. graph display xtmis_density

Compare original vs imputed distributions.

Step 6. Save

. save mydata_complete.dta, replace

Save the dataset with imputed variables for analysis.

Examples

Setup

. webuse nlswork, clear

. xtset idcode year

Example 1: Basic detection (default)

. xtmispanel ln_wage age hours tenure

When no option is given, detect is assumed. Displays four tables
summarizing missing data by variable, panel, time period, and pattern.

Example 2: Detection with explicit detect option

. xtmispanel ln_wage age hours tenure, detect

Example 3: Mechanism testing only

. xtmispanel ln_wage age hours tenure, test

Runs Little's MCAR test, logistic MAR tests, and pattern classification.
Results stored in r(mechanism), r(mcar_chi2), r(mcar_pval).

Example 4: Detect and test together

. xtmispanel ln_wage age hours, detect test

Produces all four tables plus all three tests in one call.

Example 5: Impute using linear interpolation (default name)

. xtmispanel tenure, impute(linear)

Creates tenure_imp in the dataset. Original tenure is unchanged.

Example 6: Impute with a custom variable name

. xtmispanel tenure, impute(mice) generate(tenure_mice) mice(10)

Creates tenure_mice using MICE with 10 imputations.

Example 7: Impute using KNN with 7 neighbors

. xtmispanel hours, impute(knn) knn(7)

Creates hours_imp using 7 nearest time-neighbors within each panel.

Example 8: Overwrite a previous imputation

. xtmispanel tenure, impute(linear)

. xtmispanel tenure, impute(pmm) replace

First creates tenure_imp (linear), then replaces it with PMM results.

Example 9: Sensitivity analysis (all 13 methods)

. xtmispanel tenure, sensitivity

Runs all 13 methods and displays a comparison table. The recommended method
(lowest mean distortion) is shown at the bottom and stored in r(best_method).

Example 10: Sensitivity with specific methods

. xtmispanel tenure, sensitivity methods(mean linear pmm mice knn em)

Only compares the 6 specified methods.

Example 11: Generate all diagnostic graphs

. xtmispanel ln_wage age hours tenure, graph

. graph display xtmis_combined

Creates 8 graphs. The density overlay auto-imputes the first variable.

Example 12: Graphs with explicit imputed variable for density

. xtmispanel tenure, impute(linear)

. xtmispanel ln_wage age hours tenure, graph impvar(tenure_imp)

. graph display xtmis_density

The density overlay compares observed ln_wage vs the specified tenure_imp.

Example 13: Export graphs as PNG

. xtmispanel ln_wage hours, graph

. graph export dashboard.png, name(xtmis_combined) replace width(1200)

Example 14: Full best-practice workflow

. * Step 1: Diagnose

. xtmispanel tenure, detect test

. * Step 2: Compare methods

. xtmispanel tenure, sensitivity

. * Step 3: Impute with recommended method

. xtmispanel tenure, impute(linear)

. * Step 4: Validate

. xtmispanel ln_wage tenure, graph impvar(tenure_imp)

. graph display xtmis_density

. * Step 5: Save

. save mydata_complete.dta, replace

Example 15: Using stored results programmatically

. xtmispanel tenure, impute(linear)

. local nfill = r(n_imputed)

. local method = r(method)

. display "Filled `nfill' values using `method'"

. xtmispanel tenure, test

. if "`r(mechanism)'" == "MAR" {

. xtmispanel tenure, impute(mice) replace

. }

Example 16: Using if/in restrictions

. xtmispanel tenure if year >= 2000, detect

. xtmispanel tenure in 1/100, impute(mean)

Example 17: All numeric variables (no varlist)

. xtmispanel, detect

Analyzes all numeric variables excluding the panel and time identifiers.

Frequently Asked Questions

Q1: Does imputation modify my original variable?

No. The original variable is never modified. A new variable
(default: varname_imp) is added to your dataset.

Q2: Is the imputed variable saved automatically?

It is added to the dataset in memory but NOT saved to disk until
you explicitly run save.

Q3: Which method should I use?

Run xtmispanel varname, sensitivity to compare all methods.
The command will recommend the one with least distributional distortion.
Also consider the mechanism (use test first).

Q4: Can I impute multiple variables at once?

No. Imputation works on one variable at a time. Run the command
separately for each variable:

. xtmispanel var1, impute(linear)

. xtmispanel var2, impute(mice) generate(var2_mice)

Q5: Why does sensitivity show "FAILED" for a method?

Some methods (e.g., regression-based) may fail if there are too few
observations or insufficient variation. The method is skipped and marked
as FAILED.

Q6: What does the MCAR test p-value mean?

p >= 0.05: fail to reject MCAR (missing is likely random).

p < 0.05: reject MCAR (missing depends on observed data = MAR/MNAR).

Q7: Can I use this with time series only (not panel)?

No. This command is designed for panel data. For pure time series,
use Stata's ipolate or mi impute.

Q8: Why does LOCF leave some values missing?

LOCF cannot fill the first observation if it starts as missing
(there is no previous value to carry forward). Use linear or
nocb instead for complete imputation.

Q9: How do I reload updated .ado files?

program drop _all

discard

Q10: Does this package require external dependencies?

No. All methods are implemented using base Stata commands. No
additional packages (ssc install) are needed.

References

Little, R.J.A. 1988. A test of missing completely at random for multivariate
data with missing values. Journal of the American Statistical Association
83(404): 1198-1202.

Rubin, D.B. 1987. Multiple Imputation for Nonresponse in Surveys.
New York: Wiley.

van Buuren, S. and K. Groothuis-Oudshoorn. 2011. mice: Multivariate
Imputation by Chained Equations in R. Journal of Statistical Software
45(3): 1-67.

Honaker, J. and G. King. 2010. What to do about missing values in
time-series cross-section data. American Journal of Political Science
54(2): 561-581.

Schafer, J.L. 1997. Analysis of Incomplete Multivariate Data.
London: Chapman & Hall.

Azur, M.J., E.A. Stuart, C. Frangakis, and P.J. Leaf. 2011. Multiple
imputation by chained equations: What is it and how does it work?
International Journal of Methods in Psychiatric Research 20(1): 40-49.

Stekhoven, D.J. and P. Buhlmann. 2012. MissForest: non-parametric missing
value imputation for mixed-type data. Bioinformatics 28(1): 112-118.

Troyanskaya, O. et al. 2001. Missing value estimation methods for DNA
microarrays. Bioinformatics 17(6): 520-525.

Dempster, A.P., N.M. Laird, and D.B. Rubin. 1977. Maximum likelihood from
incomplete data via the EM algorithm. Journal of the Royal Statistical
Society, Series B 39(1): 1-38.

Allison, P.D. 2001. Missing Data. Sage University Papers Series on
Quantitative Applications in the Social Sciences, 07-136. Thousand Oaks, CA: Sage.

Author

Dr. Merwan Roudane

Email: merwanroudane920@gmail.com

Please cite as:

Roudane, M. (2026). xtmispanel: Comprehensive Missing Data Detection,
Imputation and Diagnostics for Panel Data. Statistical Software Components,
Boston College Department of Economics.