*! version 1.0 30sep2015

Import Panel data from Eurostat - (c) Duarte Gonçalves 2016

v1.2 2016/05/20

xteurostat -- Imports data from Eurostat in panel data structure.

WARNING: This version of xteurostat may be out of date. To ensure that you have the latest version, type or click on ssc install xteurostat, replace, and then restart STATA.

Citation: Gonçalves, D. 2016. XTEUROSTAT: Stata module to import data from Eurostat in panel data structure, Statistical Software Components s458089, Boston College Department of Economics.
URL: http://EconPapers.repec.org/RePEc:boc:bocode:s458089.

Syntax

xteurostat tablecode [filters] [, geo(geovarname) time(timevarname) isocountrynames start(starttime) end(endtime) vname(newvarname) date(frequency) datemask(mask) clear path7zg(path)]

Main
filters retain only observations with dimensions equal to such strings.

Other
time existing/to be created time variable name.

geo existing/to be created geo variable name.

isocountrynames replace Eurostat's country codes with ISO 3166-1 Alpha-3 country codes
and generates a variable with ISO 3166-1 Numeric country codes, already labelled.

start specify the start time for the extraction, e.g. start(2000); the default is extract from the first year available.

end specify the end time for the extraction, e.g. end(2010); the default is extract until the last year available.

vname specify the name for the variable to be generated from the extraction, if only one is to be extracted.

date specify the date format, e.g. y for yearly data, q for quarterly, m for monthly, d for daily and c for clock - see date format.

clear clears existing data; the default is retain existing data and just merge the new variables.

path7zg determines an ad-hoc path to 7zG.exe that not C:\Program Files\7-Zip\7zG.exe (the default one)

Advanced
datemask specify the date mask of the time variable.

by is not allowed. if is not allowed.

Note: xteurostat is especially useful for very large extractions from Eurostat

getdata is better for more flexible extractions from several sources

Description

xteurostat imports data from Eurostat producing ready-to-use panel data structure. tablecode corresponds to Eurostat target dataset code, e.g. gov_10a_main.
It requires having 7zip installed - see install below.

tablecode should include one Eurostat data file to be downloaded, unzipped and processed.

You should just specify the code, not the .tsv or .gz extension.
The table codes can be found in the navigation tree. The codes are indicated in brackets after the titles.
The navigation tree is located here http://ec.europa.eu/eurostat/data/database.

See also http://ec.europa.eu/eurostat/data/bulkdownload and http://ec.europa.eu/eurostat/estat-navtree-portlet-prod/BulkDownloadListing.

Options

Main

filters retain only observations with dimensions equal to such strings

e.g. if MIO_NAC S13 is specified, STATA will loop over variables values and retain only those observations where the unit of measurement is millions of national currency (MIO_NAC)
and refer to General Government (S13). Only one filter per dimension is permitted and each filter should appear only in one of the dimensions, otherwise the code will yield an error so as to avoid dropping data unintentionally.

Important If there are dimensions with "-", the dashes will be removed, given that variables' names in STATA cannot have dashes.

Other

time name of the time variable. It should match the time variable name - e.g. "year", if it already exists - otherwise, it will be the name of the time variable generated.

geo name of the geo variable. It should match the geo variable name - e.g. "country", if it already exists - otherwise, it will be the name of the geo variable generated.

isocountrynames replace Eurostat's country codes with ISO 3166-1 Alpha-3 country codes
and generates a variable with ISO 3166-1 Numeric country codes, already labelled, named geovarname_num.
Although some options were added to contemplate some regional groupings in the numeric 900 section (e.g. EA19, EU28), some other of Eurostat's add-hoc regional groupings may not have an attributed numeric code.

start specify the start year for the extraction, e.g. start(2000); the default is extract all the available data.

end specify the end year for the extraction, e.g. end(2010); the default is extract all the available data.

vname specify the name for the variable to be generated from the extraction, if only one is to be extracted; the default is a combination of the parameters.

date specify the date format, e.g. y for yearly data, q for quarterly, m for monthly and d for daily - see date format.

path7zg determines an ad-hoc path to 7zG.exe that not C:\Program Files\7-Zip\7zG.exe (the default one).

clear clears existing data; the default is retain existing data and just merge the new variables

Advanced

datemask specify the date mask of the time variable - see datetime translation, mask. Requires date option and a preexisting daily-structured time variable if clear option is not selected.

Common situations: d(y) datem(Y) for yearly-frequenced data; or d(q) datem(YQ) for quarterly-frequenced data; or d(m) datem(YM) for monthly-frequenced data; or d(d) datem(YMD) for daily-frequenced data

Citation

xteurostat is not an official STATA command. It is a free contribution to the research community to improve data accessibility. Whenever used, please cite it as follows:

Gonçalves, D. 2016. XTEUROSTAT: Stata module to import data from Eurostat in panel data structure, Statistical Software Components s458089, Boston College Department of Economics.
URL: http://EconPapers.repec.org/RePEc:boc:bocode:s458089.

Install

If you use Windows you also need to install 7-zip into the program files directory (C:\Program Files\7-Zip\7zG.exe). If you install it elsewhere, just specify the path7zg(path_to_7zg) option.
OSX users don't need to do anything. You can download 7-zip from http://www.7-zip.org/download.html.

Remarks

The Eurostat repository at http://ec.europa.eu/eurostat/data/database
contains more a large number of EU economic time series. Each time series is
stored in a separate file that also contains a string-date variable and header
with information about the series.

xteurostat imports a panel data structured data into STATA directly from Eurostat. It can only refer to a single table at a time, but can download the whole data, one variable only, or several variables.

Flags are erased from the data.

Execution can sometimes be time-consuming mainly due to the necessary reshaping of the data: the more variables are to be imported, the greater the running time.
For a large dataset (gov_10a_exp), to download the table, process it and extract only one series from it takes about 1 minute.
To extract from it only data for General Government sector in millions of national currency (approx. 1500 variables) it takes less than 20 min.
To import the whole of GDP and main components (nama_10_gdp) - 570 variables and 1,360 obs, the runtime is under 3 minutes.

The Eurostat's table codes are case sensitive.

Examples

A simple extraction

xteurostat une_ltu_a, clear

Fully working example

xteurostat gov_10a_main S13 TE MIO_NAC, isocountrynames g(country) t(year) v(govsp) date(y) clear

xteurostat nama_10_gdp B1GQ CP_MNAC, isocountrynames g(country) t(year) v(gdp) date(y)

drop if substr(country,1,2) == "EA" | substr(country,1,2) == "EU"

xtset country_num year

gen govsp_sh = govsp/gdp*100

xtline govsp_sh if year > 2000

Using datemask option

datemask option specifies a particular pattern in the date string variable. As this can take many forms, the program tries to be as broad as possible in recognizing them,

but it may not cover all the cases - especially if there is any innovation from the data providers. It converts the date automatically to yyyymmdd format

- if neither date(c) nor date(C) are chosen as an option - and provides the date in the frequency determined by date option as well as daily frequency.

It requires a daily-structured time variable to merge new data with the already existing one. Exemplifying:

xteurostat nama_10_gdp B1GQ CP_MNAC, isocountrynames g(country) t(date) v(gdp) date(y) datem(Y) clear

xteurostat irt_st_m MAT_M01, isocountrynames g(country) t(date) v(intrates) d(m) datem(YM)

xteurostat ei_bsin_q BSICUPC SA, isocountrynames g(country) t(date) v(caput) date(q) datem(YQ)

Daily data can take a while to process due to reshaping and the number of observations. Run this example only with fast computers or if willing to wait

xteurostat ert_h_eur_d, geo(curr) t(date) d(d) datem(YMD) clear

Please, after a few weeks of using the program, send an email with your remarks in order to improve the code.

Authors

Duarte Gonçalves duarte.goncalves.dg@outlook.com

Draws on code by Sem Vandekerckhove (sem.vandekerckhove@kuleuven.be).