help for sctostreamsum

Title

sctostreamsum -- This command calculates statistics from sensor stream files outputted during data collection using SurveyCTO and summarizes them at the submission level in a .dta file so that they can be merged with the main data.

Syntax

sctostreamsum ,
mediafolder(folder_path) outputfolder(folder_path)
[
sensors(string) replace quiet still moving
llbetween(string) slbetween(string)
spbetween(string) mvbetween(string)
]

options

Required options:

mediafolder(folder_path) Folder where the .csv stream files used as an input to this command are saved.

outputfolder(folder_path) Folder where the .dta file generated by this command will be saved.

Output options:

sensors(string) Indicates the sensor streams (LL, SL, SP or MV) for which basic statistics should be calculated.

replace Replace the .dta file in the output folder even if it already exists. Default is that only .csv files with key-IDs not already in the .dta file are processed and appended to the already existing file.

Standardized Statistics options:

quiet Add a statistic indicating percentage of time periods when the sound level around device was quiet. Requires sound level sensor files.

still Add a statistic indicating percentage of time periods when the device was completely still. Requires movement sensor files.

moving Add a statistic indicating percentage of time periods when the device was being moved around. Requires movement sensor files.

Customizable Statistics options:

llbetween(range_string) Manually specify statistics for the light level stream files.

slbetween(range_string) Manually specify statistics for the sound level stream files.

spbetween(range_string) Manually specify statistics for the sound pitch stream files.

mvbetween(range_string) Manually specify statistics for the movement stream files.

Description

sctostreamsum is a command that calculates statistics from the sensor stream .csv files outputted by the sensor_stream field in SurveyCTO's data collection tool. This command reads .csv files with sensor data (light level, sound level, sound pitch and movement) recorded during the interview by the device used in the data collection. The unit of observation (what each row represents) in the .csv files this command uses as input is the time period the sensor data was reported on (the default is one second), but this command outputs a .dta file where the statistics have been collapsed down to one row for each submission so that the .dta file can be merged to the main data.

Before reading a .csv file in the media folder, the command first checks if the .dta file already exists, and if a file exists then it checks if that file already has an observation with the same key (ID generated for each submission on SurveyCTO's server). If an observation with the same key already exist, then the default behavior is to skip that .csv file. If the option replace is used, then no .csv files are skipped and the existing .dta file is overwritten with a new .dta file that is generated from all .csv files in the media folder.

This command cannot make updates to observations already in the .dta file, so the only way to change the statistics for an observation already in the .dta file is to use the replace option and overwrite the .dta file with a new file were all observations have been re-calculated. Note that any manual edits to the .dta file done to observations in that file after they were outputted will be lost when the replace option is used.

The command calculates two types of statistics. Basic statistics (the mean of the sensor, the standard deviation of the sensor, etc., see full list below) that are calculated the same way for all sensors, and sensor specific statistics that are all reported as a percentage of time periods which had a mean in the sensor within a specific range. If a sensor specific statistic has been specified for a sensor, then the basic statistics are also calculated. If no sensor specific statistics has been specified but basic statistics are still wanted, then the sensor should be listed in the sensor() option. If a sensor is not listed in sensor() and has not sensor specific stats, then all the .csv files for that sensor will be skipped.

Basic statistics

The basic statistics are calculated the same way for all sensor streams and the statistics are described in the table below. They are calculated for all sensors listed in sensor() or all sensors for which a sensor specific statistics was specified. In the .dta file the variables woth these statistics will have the same name as in the table below but with the sensor stream abbreviation (LL, SL, SP and MV) as a prefix. For example, LL_mean, SL_mean etc.

Name Description

mean The mean of all time period means. Each row in the .csv file is based on more than one raw sensor recording. The value in the variable mean in the .csv file is the mean of those raw recordings. The basic statistics mean calculated by this command is the mean of all those means.

period The period length (in seconds) used for each row in the .csv file. The default if not explicitly specified in the questionnaire form definition is period=1.

period_obs Number of time periods (i.e. rows) in the .csv file. If the period is 1, then this is the duration of the interview in seconds.

raw_obs Number of raw recordings of the stream. Unless the time period for the sensor stream is set to 0 in the questionnaire form definition, each time period is made up of many raw sensor recordings. This is the total number of raw sensor recordings.

min The minimum of all time period means. Note that this is not the lowest raw recording, it is the lowest time period mean.

max The maximum of all time period means. Note that this is not the highest raw recording, it is the highest time period mean.

sd The standard deviation of all time period means.

median The median of all time period means.

Sensor specific statistics

Sensor specific statistics consist of Standardized Statistics where the range is predefined and Customizable Statistics where the range is specified by the user. Each of these statistics are calculated as booleans, i.e. either true or false. For example, was it quiet or not, was the sensor within a certain range or not. This boolean, represented as 1 or 0, where 1 is true, is calculated for each time period for that sensor. This command then calculates the percentage of time periods for which this variable was 1 and reports that value in the .dta file. The variable for each statistic in the .dta file therefore represents the percentage of the interview that was quiet, where the sensor was within a certain range, etc., for each submission.

Options

Required options:

mediafolder(string) indicates where the .csv files exported from the SurveyCTO server are saved. This is called the media folder because that is the name of the folder where SurveyCTO Sync saves these files. Other files not relevant to this command may also be stored in this folder as this command can tell which files are sensor stream files based on the file name.

outputfolder(string) indicates where the .dta file will be saved. This folder may not be the same folder as the folder in mediafolder(). If the .dta file already exists there, then only files in the media folder with a key not already in the .dta file will be processed and appended to the already existing file (unless the replace option is used).

Output options:

sensors(string) lists all the sensor streams for which to calculate basic statistics. If a sensor specific statistic is already specified then there is no need to also specify it here as basic statistics for that sensor will also be calculated, but doing so will not cause an error. For each sensor listed here there must be at least one sensor stream .csv file in the media folder. Valid values in this option are LL for light level, SL for sound level, SP for sound pitch and MV for movement.

replace makes the command overwrite any file generated by this command already in the outputfolder(). If this option is used, then all .csv files in the mediafolder() will be used, whether they are in the file that already exists or not. Since this will overwrite the file already in the output folder, all manual edits after the file was originally generated will be lost. Using this option is the only way to update existing observations in a .dta file already generated using this command.

Standardized statistics options:

quiet adds a variable called quiet to the .dta file. This variable will be the percentage of time periods (expressed in decimals) for which the sound level around the device used during the interview was less than 25dB. An error will be generated if this option is used and no sound level sensor stream files exist in the mediafolder() folder.

still adds a variable called still to the .dta file. This variable will be the percentage of time periods (expressed in decimals) for which the movement of the device used during the interview is less than 0.25 m/s^2. An error will be generated if this option is used and no movement sensor stream files exist in the mediafolder() folder.

moving adds a variable called moving to the .dta file. This variable will be the percentage of time periods (expressed in decimals) for which the movement of the device used during the interview is greater than 2 m/s^2. An error will be generated if this option is used and no movement sensor stream files exist in the mediafolder() folder.

Customizable Statistics options:

All of the following options take a range_string as the value. The range_string is used to indicate the name of the new variable this command should create and the range that this command will use to calculate the percentage (in decimal points) of time periods that the sensor was within that range. Each new variable in the range_string must be specified as: varname(min max), where varname is the name of the new variable to be created, and min and max are the lower and upper boundaries for the range. Round brackets indicate that the boundary is exclusive, and straight brackets indicate it is inclusive. One of min and max can be replaced with a question mark to have a greater-than or less-than expression instead of a range. Multiple new variables can be specified in the same range_string. See examples below.

llbetween(range_string) allows the user to manually specify statistics for the light level stream. See documentation on range_string above and examples below. An error will be generated if this option is used and no light level sensor stream files exist in the mediafolder() folder.

slbetween(range_string) allows the user to manually specify statistics for the sound level stream. See documentation on range_string above and examples below. An error will be generated if this option is used and no sound level sensor stream files exist in the mediafolder() folder.

spbetween(range_string) allows the user to manually specify statistics for the sound pitch stream. See documentation on range_string above and examples below. An error will be generated if this option is used and no sound pitch sensor stream files exist in the mediafolder() folder.

mvbetween(range_string) allows the user to manually specify statistics for the movement stream. See documentation on range_string above and examples below. An error will be generated if this option is used and no movement sensor stream files exist in the mediafolder() folder.

range_string examples:

llbetween(indoors_lit(100 750)) will create a variable from the light level stream named indoors_lit where the values are the percentage (in decimal points) of time periods where the mean light level was between 100 lux (exclusive) and 750 lux (exclusive).

slbetween(quiet(? 25)) will create a variable from the sound level stream named quiet where the values are the percentage (in decimal points) of time periods where the mean sound level was below 25 dB (exclusive). This is identical to the variable created when using the option quiet.

mvbetween(mv1[.2 .25) mv2[1 ?]) will create two variables from the movement stream named mv1 and mv2. mv1 is the percentage (in decimal points) of time periods where the mean movement was between .2 m/s^2 (inclusive) and .25 m/s^2 (exclusive). mv2 is the percentage (in decimal points) of time periods where the mean movement was greater than 1 m/s^2 (inclusive).

Examples

All examples will use the following globals as folder paths:

global project "C:\Users\username\Documents\ProjectA"

global media "$project\raw_data\media"

global output "$project\outputs"

Example 1.

sctostreamsum, mediafolder("$media") outputfolder("$output") quiet still

This is a very simple way to run the command. The command will read sensor stream .csv files in the media folder with the prefix SL (because quiet was used) and the prefix MV (because still was used) and create a .dta file with the key in variable key, the basic statistics for the sound level sensor in addition to the quiet variable, and the basic statistics for the movement sensor in addition to the still variable and save it in the output folder. If the command has already been run and the .dta already exists in the folder, then only submissions with a key not already in the .dta file will be processed and appended to the already existing file. Any sensor stream files with the prefix SP or LL in the media folder will be ignored as no statistic applicable to either of those streams was specified.

Example 2.

sctostreamsum, mediafolder("$media") outputfolder("$output") quiet slbetween(loud(60 ?)) replace

In this example, the command will only read SL sensor stream .csv files from the media folder as sound level is the only sensor required by the statistics that are specified. Basic statistics will be calculated for the sound level stream, in addition to the quiet and loud variables. All SL files in the media folder will be included and the command will start over with a new .dta file, replacing the .dta file in the output folder if it already exists.

Author

This command was developed by SurveyCTO.

See this command's repository for more information where you can also submit feedback and feature requests.