*! version 2.0.1 09 May 2018

Title

splitit -- splits overlapping spells in spell data

Syntax

splitit
varlist,
[options]

where varlist denominates 5 existing variables in the working file, standing for case_ID, start_date (num int), end_date (num int), spell_type, and spell_ID. The named sequence of variables is obligatory.

Main
srt(varlist) variables serving as additional sort criteria of the split data

portions(#) specifies the number of portions the source file is divided into for processing. Default is 1 portion for ca. 25,000 obervatons

cvars orders four specific count variables for the split spells

Description

splitit splits overlapping spells within a case. The result is a file where each split spell of a case is either chronologically unique or
chronologically completely parallel to other split spells of the same case (having exactly the
same start and end_dates, in the following called "isochronic spells"). The graph and table below illustrate the principle of this operation.

Spell splitting is an essential step for most of the methods to analyze episode data, and is a
preliminary step in transforming spell data into sequence data.
In contrast to other tools for spell data, (i.e. newspell), splitit exclusively splits the data and leaves it untouched in every other aspect.
Therefore no prior knowledge regarding the precedence of spell_types is needed. Together with combival.ado (also written by the authors),
splitit can be used to explore the data in order to investigate the existing parallelities and, subsequently, determine a rank order or
define new spell_types.

splitit works by an elaborate comparison of every begin and end_date within each case. The resulting file has the fewest possible number of split spells, satisfying the condition not to overlap, for a given data set. Because of this
property, splitit can be used to split data that has - compared to the whole represented time span - very finely grained time units.

A special feature of portioned processing optimizes the runtime and makes splitit applicable to very large data sets. It has been tested with a dataset of about 30 Mio observations. A feature that measures the runtime of the program is
included.

The versions from 2.0.0 are adapted to work with Stata 14, and are tested to work with previous versions from Stata 11.2

Graphical example:

Original spells of a case in the source file
A
|----------------------|
B
|------------------|

Split spells of this case
A1 A2
|-------------||-------|
B1 B2
|-------||---------|

Tabular example:

Source file table
case	spellid start end spelltype
10001 1 jan2010 dec2012 A
10001 2 apr2012 nov2013 B

Table after splitting
case	spellid start end spelltype start_split	end_split sid_split
10001 1 jan2010 dec2012 A jan2010	mar2012 1
10001 1 jan2010 dec2012 A apr2012	dec2012 2
10001 2 apr2012 nov2013 B apr2012	dec2012 3
10001 2 apr2012 nov2013 B jan2013	nov2013 4

Options

Main

srt(varlist) The sort order of the split data after the execution of splitit is case_ID, start_date of the split spells, spell_type, and, if specified, the variables named in the srt() option. The sort order is
reflected in the newly generated variable sid_split, which is the ID of the split spells within a case.

portions(#) By default, splitit divides the data into portions of approximately 25,000 observations, providing for placing spells of the same case into the same portion. This feature is implemented because Stata tends to become
disproportionally slow with increasing computer capacity utilization. Depending on the users' computer environment, the default might result in portions that are too large or too small. splitit produces a message with a time stamp for
each portion being processed. As a rule of thumb, the default number of portions ought to be increased if more than 5 minutes elapse before the next portion is completed.

cvars generates 4 variables relating to the number of split spells:

- levela counts spell splits per case and spell_type, beginning with 0, for each value of split_start.

- levelb counts spell splits per case, beginning with 0, for each value of split_start.

- nlevela represents the sum of spell splits per case and spell_type for each value of split_start.

- nlevelb represents the sum of spell splits per case for each value of split_start.

Please note: with split spell data all spells having a certain split_start always have the same split_end.

Remarks

The varlist assigned to splitit is required to contain 5 variables, standing for case_ID, start_date, end_date,
spell_type, and spell_ID in the active working file.

The following requirements are mandatory:

- The above named sequence of the variables in varlist is compulsory, because splitit
derives the meaning of the variables from their position within varlist.

- None of the 5 variables of varlist must have missing values, (or blanks, if string type). If there are any, splitit issues an error message and stops processing.

- The start and end_date variables in varlist must be numeric and hold integer values, as is the case with Stata date variables. splitit copies the format of the date variables in varlist to the newly generated variables
start_date_split and end_date_split.

- The duration of spells, calculated as end_date minus start_date, must not be negative. If there are any spells with a negative duration, splitit issues an error message and stops processing.

Further information:

All variables of varlist except the start and end_date of the spells may be numeric or string variables.

If a spell has the same start_date as the end_date of the previous spell, the two spells will be treated as overlapping for one time unit. If this is not desired, one of the two spells must be shortened by one time unit before applying
splitit.

Gaps in the sequence of spells do not cause problems.

After splitting, the original spell_ID is no longer unique. Therefore a new unique split spell_ID is generated, named "sid_split".

Undoing and repeating the splitting:

You can undo the splitting very easily with:

. keep if start_date_split == start_date

Alternatively, and provided that case_ID and spell_ID identify the observations in the original unsplit file uniquely,
you can use:

. duplicates drop case_ID spell_ID , force

The variables generated by splitit are senseless after that and should be dropped.

If you want to repeat the splitting because the data has changed (i.e. spells have been dropped or merged), the best way is to undo the splitting as described above and then apply splitit again.

If for any reason you want to apply splitit on previously split data, you will need to pay attention to the names of the variables in varlist:

start_date, end_date and spell_ID of the first run have become start_date_split, end_date_split and sid_split in the second run.

If you apply splitit inadvertently on your working file with the same parameters more than once, splitit will produce duplicates and additional sets of generated variables. Provided that case_ID and spell_ID identify the observations
in the original unsplit file uniquely, you can get rid of those with:

. duplicates drop case_ID spell_ID start_date_split, force

(subsequently drop the additionally generated variable sets).

Examples

. use testdata_splitit, clear
(testdata_splitit.dta)
. splitit persno begin end sptype spellno

. use testdata_splitit, clear
(testdata_splitit.dta)
. splitit persno begin end sptype spellno, srt(netincd)

. use testdata_splitit, clear
(testdata_splitit.dta)
. splitit persno begin end sptype spellno, cv

. use testdata_splitit, clear
(testdata_splitit.dta)
. splitit persno begin end sptype spellno, por(4) cv

Authors

Klaudia Erhardt, SOEP, DIW Berlin
Email kerhardt@diw.de

Ralf Kuenster, NEPS, WZB Berlin
Email ralf.kuenster@wzb.eu

Edition: V.2.0.1, May 2018