*! version 1.22 24mar2016

Import SMDX Data - (c) Duarte Gonçalves 2016

v1.22 2016/03/24

getdata -- Imports data from several SDMX Data providers (OECD, EUROSTAT, ECB, IMF, ILO, ...) in raw, cross-sectional, time series or panel data structure using SDMX rest codes.

WARNING: This version of getdata may be out of date. To ensure that you have the latest version, type or click on ssc install getdata, replace, and then restart STATA.

Citation: Gonçalves, D. 2016. GETDATA: Stata module to import SDMX data from several providers, Statistical Software Components S458093, Boston College Department of Economics.
URL: http://EconPapers.repec.org/RePEc:boc:bocode:s458093.

Syntax

getdata structure provider, rest(restcode) [geo(s1 [s2]) time(s1 [s2]) isocountrycodes(codetype) start(starttime) end(endtime) merge(1:1|1:m|m:1|m:m) replace update
frequency(frequency) datemask(mask) varname(varname) set clear]

Main
structure raw, cs (cross-sectional), ts (time series) or xt (panel).

provider SDMX data provider.

rest rest code, e.g. EO/USA.YRGT.A (from OECD) or irt_h_euryld_q/Q.YIELD.Y1. (from EUROSTAT); check getdatacodes.

CS, TS and XT
geo s1: geo variable name as is in the "raw" dataset; s2: existing/to be created geo variable name.

time s1: time variable name as is in the "raw" dataset - default is DATE; s2: existing/to be created time variable name.

isocountrycodes replace country codes with ISO 3166-1 country codes: alpha3 for ISO 3166-1 Alpha-3; alpha2 for ISO 3166-1 Alpha-2;
num for ISO 3166-1 Numeric country codes (labelled);
more than one is permitted, e.g. iso(alpha3 num).

merge specify the type of merge with existing data, 1:1, 1:m, m:1 or m:m; the default is 1:1 - see merge.

replace specify replace in merge options - see merge.

update specify update in merge options (no need to use both replace and update) - see merge.

start specify the start time for the extraction, e.g. start(2000); the default is extract from the first year available.

end specify the end time for the extraction, e.g. end(2010); the default is extract until the last year available.

varname specify the name for the variable to be generated from the extraction, if only one is to be extracted.

frequency specify the date format, e.g. y for yearly data, q for quarterly, m for monthly, d for daily and c for clock - see date format.

set sets the data as time series or panel data, depending on the structure chosen.

clear clears existing data; the default is retain existing data and just merge the new variables.

Advanced
datemask specify the date mask of the time variable; must be used with frequency.

by is not allowed. if is not allowed.

Note: getdata is better for more flexible extractions from several sources

xteurostat is especially useful for very large extractions from Eurostat

Description

getdata imports data from several SDMX Data providers (OECD, EUROSTAT, ECB, IMF, ILO, ...) in raw, cross-sectional, time series or panel data structure using SDMX rest codes.
It is able to produce ready-to-use cross-sectional, time series panel data structure. rest corresponds to the rest query code, EO/USA.YRGT.A (from OECD) or irt_h_euryld_q/Q.YIELD.Y1. (from EUROSTAT).

This package installs automatically Bank of Italy's SDMX Connector for STATA.

rest codes can be obtained by calling getdatacodes, selecting a Provider, a Flow ID and the Dimensions - it appears in the white box above.
To copy it select everything in the white box on the top and click Ctrl+C.

Some providers' websites, e.g. http://stats.oecd.org, also enable users to get the rest code from the URL.

Options

Main

structure raw, cs (cross-sectional), ts (time series) or xt (panel).

provider SDMX data provider, e.g. OECD, EUROSTAT, ECB, IMF, ILO, or other.

rest rest code, e.g. EO/USA.YRGT.A (from OECD) or irt_h_euryld_q/Q.YIELD.Y1. (from EUROSTAT). Check getdatacodes as mentioned above.

Important If there are dimensions with "-" or other characters aside alphanumeric ones and the underscore (_), these characters will be removed, given that variables' names in STATA cannot contain special characters.
The variables' names are also shortened to comply with STATA's limit on varname characters (32).

CS, TS and XT

geo s1 geo variable name as is in the "raw" dataset - e.g. LOCATION (common in OECD tables) or GEO (the defaul in EUROSTAT tables); s2: existing/to be created geo variable name.
If the existing data already contains a geo variable name, it should be specified (s2) - e.g. "country", if it already exists. Otherwise, s2 will be the name of the geo variable generated.
The default assumes the s2=s1; there is no default for s1 in itself.

time s1 time variable name as is in the "raw" dataset - e.g. DATE (the default in raw tables); s2: existing/to be created time variable name.
If the existing data already contains a time variable name, it should be specified (s2) - e.g. "year", if it already exists.
Otherwise, s2 will be the name of the time variable generated. The default assumes s2=s1 and s1="DATE".

isocountrycodes replace country codes with ISO 3166-1 country codes: alpha3 for ISO 3166-1 Alpha-3; alpha2 for ISO 3166-1 Alpha-2; num for ISO 3166-1 Numeric country codes (labelled), named geovarname_s2_num.
Although some options were added to contemplate some regional groupings in the numeric sections greater than 999 (e.g. EA19, EU28), some other of the providers' add-hoc regional groupings will not have an attributed numeric code
and thus frustrate the use of the panel setting. The first option in isocountrycodes is considered the default; any other option creates an additional variable. E.g. isocountrycodes(alpha3 num) assumes that the geo variable
follows ISO 3166-1 Alpha-3 and creates an additional geo variable, geo_num, following the numeric ISO coding.

merge specify the type of merge with existing data, 1:1, 1:m, m:1 or m:m; the default is 1:1 - see merge.

replace specify replace in merge options - see merge.

update specify update in merge options (no need to use both replace and merge) - see merge.

start specify the start year for the extraction, e.g. start(2000); the default is extract all the available data.

end specify the end year for the extraction, e.g. end(2010); the default is extract all the available data.

varname specify the name for the variable to be generated from the extraction, if only one is to be extracted; the default is a combination of the parameters.

frequency specify the date format, e.g. y for yearly data, q for quarterly, m for monthly, d for daily and c for clock - see date format.

set sets the data as time series or panel data, depending on the structure chosen.

clear clears existing data; the default is retain existing data and just merge the new variables

Advanced

datemask specify the date mask of the time variable - see datetime translation, mask.
Requires date option and a preexisting daily-structured time variable if clear option is not selected.
Must be used with frequency

Common situations: d(y) datem(Y) for yearly-frequenced data; or d(q) datem(YQ) for quarterly-frequenced data; or d(m) datem(YM) for monthly-frequenced data; or d(d) datem(YMD) for daily-frequenced data.

Citation

getdata is not an official STATA command. It is a free contribution to the research community to improve data accessibility. Whenever used, please cite it as follows:

Gonçalves, D. 2016. GETDATA: Stata module to import SDMX data from several providers, Statistical Software Components S458093, Boston College Department of Economics.
URL: http://EconPapers.repec.org/RePEc:boc:bocode:s458093.

Install

Requires STATA version 13 or above.

Remarks

Imports data from several SDMX Data providers (OECD, EUROSTAT, ECB, IMF, ILO, ...) in raw, cross-sectional, time series or panel data structure using SDMX rest codes.

getdata imports cross-sectional, time series or panel data structured data into STATA directly from data providers. It can only refer to a single table at a time, but can download the whole data, one variable only, or several variables.

Flags are erased from the data.

Execution can sometimes be time-consuming mainly due to the necessary reshaping of the data: the more variables are to be imported, the greater the running time. For a big dataset (gov_10a_exp), to download the table, process it
and extract only one series from it takes about 1 second. To extract from it only data for General Government sector in millions of national currency (over 1500 variables) it takes less than 5 min.

The rest codes are case sensitive.

EUROSTAT has currently a cap on the quantity of data one can extract directly at once. Try xteurostat as an alternative for very large extractions.

This program uses the SDMX Connector for STATA, licensed to Banca d'Italia (Bank of Italy) under a European Union Public Licence
(world-wide, royalty-free, non-exclusive, sub-licensable licence).
See https://github.com/amattioc/SDMX/wiki/SDMX-Connector-for-STATA and
https://joinup.ec.europa.eu/sites/default/files/eupl1.1.-licence-en_0.pdf.

I dearly thank Attilio Mattiocco (Attilio.Mattiocco@bancaditalia.it) for all the help regarding the SDMX Connector for STATA
and Bo Werth (Bo.WERTH@oecd.org) for additional clarifications.

For a statistics provider to be able to be included, please contact Attilio Mattiocco (Attilio.Mattiocco@bancaditalia.it). See also Adding a Provider.

Examples

Calling the getdatacodes

getdatacodes

Simple extractions

getdata raw IMF, r(DM/111.PPPGDP.WEO.H.A) clear

getdata ts IMF, r(DM/111.PPPGDP.WEO.H.A) g(REF_AREA) t(DATE) clear

getdata ts IMF, r(DM/111.PPPGDP.WEO.H.A) g(REF_AREA country) t(DATE year) f(y) v(us_gdp_ppp) set clear

getdata cs EUROSTAT, r(inn_cis5_gen/.MAR_LREG.TOT_INN.TOTAL.NR.INNOACT.) g(GEO country) v(nr_innov_firms_2006) clear

getdata xt OECD, r(EO/USA.YRGT.A) geo(LOCATION country) iso(num) time(DATE year) f(y) set clear

Different ISO Country codes

getdata xt EUROSTAT, r(gov_10a_exp/A.MIO_NAC.S13.TOTAL.TE.) geo(GEO country) iso(alpha3) time(DATE year) f(y) v(govsp) clear

getdata xt OECD, r(SNA_TABLE11/.TLYCG.T.GS13.C) geo(LOCATION country) iso(alpha2) time(DATE year) f(y) v(govsp) clear

getdata xt OECD, r(SNA_TABLE11/.TLYCG.T.GS13.C) geo(LOCATION country) iso(num) time(DATE year) f(y) v(govsp) set clear

getdata xt OECD, r(SNA_TABLE11/.TLYCG.T.GS13.C) geo(LOCATION country) iso(num alpha3) time(DATE year) f(y) v(govsp) set clear

Fully working example

getdata xt OECD, r(EO/AUT+BEL+DEU+ESP+EST+FIN+FRA+IRL+ITA+LUX+LVA+NLD+PRT+SVK+SVN.GDP.A) geo(LOCATION country) iso(num) time(DATE year) f(y) v(gdp) clear

replace gdp = gdp/(10^6)

getdata xt OECD, r(SNA_TABLE11/AUT+BEL+DEU+ESP+EST+FIN+FRA+IRL+ITA+LUX+LVA+NLD+PRT+SVK+SVN.TLYCG.090.GS13.C) geo(LOCATION country) iso(num) time(DATE year) f(y) v(govsp_educ)

getdata xt OECD, r(SNA_TABLE11/AUT+BEL+DEU+ESP+EST+FIN+FRA+IRL+ITA+LUX+LVA+NLD+PRT+SVK+SVN.TLYCG.T.GS13.C) geo(LOCATION country) iso(num) time(DATE year) f(y) v(govsp) set

gen govsp_on_gdp = govsp/gdp

gen govsp_educ_on_gdp = govsp_educ/gdp

label var govsp_on_gdp "Total Gvt Spending"

label var govsp_educ_on_gdp "Gvt Spending on Education"

xtline govsp_on_gdp govsp_educ_on_gdp, byopts(title("Government Spending (%GDP)") note("OECD, SNA Table 11")) name("govsp_on_gdp", replace) xtitle("") ytitle("")

Update data using different sources and the same variables

getdata xt EUROSTAT, r(gov_10a_exp/A.MIO_NAC.S13.TOTAL.TE.) g(GEO country) t(DATE year) set f(y) iso(num) v(govsp) clear

getdata xt OECD, r(SNA_TABLE11/.TLYCG.T.GS13.C) g(LOCATION country) t(DATE year) set f(y) iso(num) v(govsp) update

Merge regional and national data

getdata xt OECD, r(REGION_ECONOM/2.ES11+ES12+ES13+ES21+ES22+ES23+ES24+ES30+ES41+ES42+ES43+ES51+ES52+ES53+ES61+ES62+ES63+ES64+ES70+FR10+FR21+FR22+FR23+FR24+FR25+FR26+FR30+FR41+FR42+FR43+FR51+FR52+FR53+FR61+FR62+FR63+FR71+FR72+FR81+FR82+FR83.SNA_2008.GDP.CURR_PR.ALL.) g(REG_ID) t(TIME year) f(y) clear v(y_reg)

getdata xt OECD, r(REGION_ECONOM/2.ES11+ES12+ES13+ES21+ES22+ES23+ES24+ES30+ES41+ES42+ES43+ES51+ES52+ES53+ES61+ES62+ES63+ES64+ES70+FR10+FR21+FR22+FR23+FR24+FR25+FR26+FR30+FR41+FR42+FR43+FR51+FR52+FR53+FR61+FR62+FR63+FR71+FR72+FR81+FR82+FR83.SNA_2008.INCOME_DISP.CURR_PR.ALL.) g(REG_ID) t(TIME year) f(y) v(yd_reg)

gen country = substr(REG_ID,1,2)

getdata xt OECD, r(EO/FRA+ESP.GDP.A) g(LOCATION country) t(DATE year) f(y) merge(m:1) iso(alpha2 num) v(y)

Merge data of different frequencies

getdata ts OECD, r(EO/USA.GDP.Q) g(LOCATION country) t(DATE year) f(q) datem(YQ) v(gdp) clear

replace year = yofd(year)

format year %ty

getdata ts OECD, r(EO/USA.YPGT.A) g(LOCATION country) t(DATE year) f(y) v(govsp) m(m:1)

Using datemask option

datemask option specifies a particular pattern in the date string variable. As this can take many forms, the program tries to be as broad as possible in recognizing them,

but it may not cover all the cases - especially if there is any innovation from the data providers. It converts the date automatically to yyyymmdd format

- if neither frequency(c) nor frequency(C) are chosen as an option - and provides the date in the frequency determined by frequency option as well as daily frequency.

It requires a daily-structured time variable to merge new data with the already existing one. E.g.

getdata ts EUROSTAT, r(irt_h_euryld_d/D.YIELD.Y1.EA) geo(GEO) time(DATE year) f(d) datem(YMD) clear

getdata ts OECD, r(EO/USA.GDP.Q) geo(LOCATION) time(DATE year) f(q) datem(YQ) clear

getdata ts ECB, r(EON/D.EONIA_TO.RATE) g(GEO) t(DATE day) v(eonia) f(d) datem(YMD) set clear

Please, after a few weeks of using the program, send an email with your remarks in order to improve the code.

Providers

Adding a Provider

Is is possible to add custom data providers using the configuration.properties file given that the providers follow SDMX 2.1 standards.

1. Do cd "`c(sysdir_plus)'jar"

2. If you have downloaded the configuration.properties file already, go to step 3. Otherwise, do net get getdata.pkg

3. Go to the ado\plus\jar folder (do di "`c(sysdir_plus)'jar" to see which one it is)

4. Open the configuration.properties file on Notepad (or a similar text editor) and under "Providers settings" find "Add New Providers" and follow the instructions

International Sources

Eurostat http://ec.europa.eu/eurostat/web/sdmx-web-services/rest-sdmx-2.1
European Central Bank (ECB) https://sdw-wsrest.ecb.europa.eu/
International Monetary Fund (IMF) http://sdmxws.imf.org/
Organisation for Economic Co-operation and Development (OECD) http://stats.oecd.org (export to SDMX tool)
United Nations (UN) http://data.un.org/Host.aspx?Content=API
UN Food & Agriculture Organization (FAO) http://api.data.fao.org/1.0/esb-rest/sdmx/introduction.html
UN International Labour Organization (ILO) http://www.ilo.org/ilostat/faces/home/statisticaldata/technical_page?_adf.ctrl-state=25zdozvi8_9&_afrLoop=1131342564621899
World Bank (WB) http://databank.worldbank.org/data/ (export to SDMX tool)

National Sources

AUS Australian Bureau of Statistics (ABS) http://stat.abs.gov.au/ (export to SDMX tool)
BEL National Bank of Belgium http://stat.nbb.be/ (export to SDMX tool)
CAN Statistics Canada http://www.statcan.gc.ca/
GER Deutsche Bundesbank https://www.bundesbank.de/Navigation/EN/Statistics/Time_series_databases/time_series_databases.html
FRA Institut National de la Statistique et des Etudes Economiques (INSEE) http://www.bdm.insee.fr/bdm2/statique?page=sdmx (export to SDMX tool)
MEX Sistema Nacional de Información Estadística y Geográfica de México (SNIEG) http://www.snieg.mx/opendata/
ESP Instituto Nacional de Estadística http://www.ine.es/fmi/nsdp.htm (export to SDMX tool)
GBR Office of National Statistics (ONS) https://www.ons.gov.uk/ons/apiservice/web/apiservice/how-to-guides#sdmx
USA US Federal Reserve http://www.federalreserve.gov/DataDownload/help/default.htm (export to SDMX tool)

Configuring a Network Proxy

If there is any need to configure a network proxy in order to the program to work, after installing getdata follow the steps below

1. Do cd "`c(sysdir_plus)'jar"

2. If you have downloaded the configuration.properties file already, go to step 3. Otherwise, do net get getdata.pkg

3. Go to the ado\plus\jar folder (do di "`c(sysdir_plus)'jar" to see which one it is)

4. Open the configuration.properties file on Notepad (or a similar text editor) and under "Network Configuration" add the following:

http.proxy.default = URL_FOR_INTERNET_PROXY:NUMBER_OF_PROXY_PORT

For example,

http.proxy.default = proxy.institution.org:80

5. After editting the configuration.properties file, restart STATA and do getdataconfig.

It is advisable to follow the instructions on the configuration.properties file, that you can edit with any text processor, e.g. Notepad.

Author

Duarte Gonçalves duarte.goncalves.dg@outlook.com