help for sctostreamcsv

Title

sctostreamcsv -- This command adds variables to sensor streams files outputted alongside data exported from SurveyCTO. The added variables are calculations based on the sensor recordings in those files.

Syntax

sctostreamcsv ,
mediafolder(folder_path) outputfolder(folder_path)
[
keyvar replace quiet still moving
llbetween(string) slbetween(string)
spbetween(string) mvbetween(string)
]

options

Required options:

mediafolder(folder_path) Folder where the .csv stream files used as an input to this command are saved.

outputfolder(folder_path) Folder where the new .csv files generated by this command will be saved.

Output options:

keyvar Saves the UUID key of the submission the sensor stream belongs to in a variable inside the file. By default the key is only in the file name.

replace Replace files already in the output folder. Default is to skip files already in the output folder and not update them.

Standardized Statistics options:

quiet Add a statistic indicating which time period the sound level around the device was quiet. Requires sound level sensor files.

still Add a statistic indicating which time period the device was completely still. Requires movement sensor files.

moving Add a statistic indicating which time period the device was moving. Requires movement sensor files.

Customizable Statistics options:

llbetween(range_string) Manually specified statistics for the light level stream.

slbetween(range_string) Manually specified statistics for the sound level stream.

spbetween(range_string) Manually specified statistics for the sound pitch stream.

mvbetween(range_string) Manually specified statistics for the movement stream.

Description

sctostreamcsv is a command that adds statistics to the sensor stream .csv files outputted by the sensor_stream field in SurveyCTO's data collection tool. The .csv files include sensor data (light level, sound level, sound pitch and movement) recorded during the interview by the device used in the data collection. The unit of observation (what each row represents) in the .csv files this command uses as input is the time period the sensor data was reported on. The default is one second, but this can be changed in the survey form definition.

This command adds one column for each Standardized Statistics option and/or each Customizable Statistics option specified by the user to the individual sensor stream .csv files. Each of these statistics are calculated as booleans, i.e. either true or false. For example, was it quiet or not, was the sensor within a certain range or not. This boolean, represented as 1 or 0 where 1 is true, is calculated for each time period for that sensor and added in the row in the .csv file corresponding to that time period.

After calculating the statistics the command generates a new .csv file and saves it in the outputfolder(folder_path), giving it the same name the file had in the mediafolder(). If the file already exists in the output folder, then the file is skipped. So the only way to add new statistics to already generated files is to use the option replace. Note that this overwrites any manual edits made to the files after they were generated by this command.

Options

Required options:

mediafolder(string) indicates where the .csv files exported from the SurveyCTO server are saved. This is called the media folder because that is the name of the folder where SurveyCTO Sync, SurveyCTO's desktop export tool, saves these files. Other files not relevant to this command may also be stored in this folder as this command can tell which files are sensor stream files based on the file name.

outputfolder(string) indicates where the newly generated csv files will be saved. This folder should not be the same folder as the folder in mediafolder(). The output folder is used to list which files have already been processed and therefore should be skipped by this command (unless replace is used).

Output options:

keyvar tells the command to save the uniquely identifying ID key of the submission the sensor stream belongs to in a variable called key in the new .csv file. The key is always stored in the file name even if this option is not used. This is not the same ID as any ID entered manually by the enumerators in the field. This ID is generated automatically by SurveyCTO for each submission. Multiple submissions might have the same human-entered ID (e.g. if the same respondent is surveyed multiple times for a panel survey) but each such submission will have a unique ID key generated by SurveyCTO. This column is not added by default as that saves disk space and the information is stored in the file name regardless.

replace makes the command overwrite any files already in the outputfolder() instead of skipping those files. Since this will overwrite the files in the output folder, all manual edits made after the files were originally generated will be lost. Using this option is the only way to update a .csv file already generated by this command.

Standardized Statistics options:

quiet adds a variable called quiet to each SL (sound level) sensor stream .csv file. This variable will be 1 for all time periods for which the mean sound level is less than 25dB. For all other non-missing values this variable will be 0. An error will be generated if this option is used and no SL sensor files exist in the mediafolder() folder.

still adds a variable called still to each MV (movement) sensor stream .csv file. This variable will be 1 for all time periods for which the mean movement is less than 0.25 m/s^2. For all other non-missing values this variable will be 0. An error will be generated if this option is used and no MV sensor files exist in the mediafolder() folder.

moving adds a variable called moving to each MV (movement) sensor stream .csv file. This variable will be 1 for all time periods for which the mean movement is greater than 2 m/s^2. For all other non-missing values this variable will be 0. An error will be generated if this option is used and no MV sensor files exist in the mediafolder() folder.

Customizable Statistics options:

All of the following options take a range_string as a value. The range_string is used to indicate the name of the new variable this command should create and for which range this variable should be 1 for each time period. The new variable is 0 for all other non-missing values. Each new variable in the range_string must be specified as: varname(min max), where varname is the name of the new variable to be created and min and max are the lower and upper boundaries for the range. Round brackets indicate that the boundary is exclusive, and straight brackets indicate it is inclusive. One of min or max can be replaced with a question mark to have a greater-than or less-than expression instead of a range. Multiple new variables can be specified in the same range_string. See examples below.

llbetween(range_string) allows the user to manually specify statistics for the light level stream. See documentation on range_string above and examples below. An error will be generated if this option is used and no light level sensor files exist in the mediafolder() folder.

slbetween(range_string) allows the user to manually specify statistics for the sound level stream. See documentation on range_string above and examples below. An error will be generated if this option is used and no sound level sensor files exist in the mediafolder() folder.

spbetween(range_string) allows the user to manually specify statistics for the sound pitch stream. See documentation on range_string above and examples below. An error will be generated if this option is used and no sound pitch sensor files exist in the mediafolder() folder.

mvbetween(range_string) allows the user to manually specify statistics for the movement stream. See documentation on range_string above and examples below. An error will be generated if this option is used and no movement sensor files exist in the mediafolder() folder.

range_string examples:

llbetween(indoors_lit(100 750)) will create a variable in light level stream files named indoors_lit that is 1 for each time period where the mean light level was between 100 lux (exclusive) and 750 lux (exclusive).

slbetween(quiet(? 25)) will create a variable in sound level stream files named quiet that is 1 for each time period where the mean sound level was below 25 dB (exclusive). This is identical to the variable created when using option quiet.

mvbetween(mv1[.2 .25) mv2[1 ?]) will create two variables in movement stream files named mv1 and mv2. mv1 will be 1 for each time period where the mean movement is between .2 m/s^2 (inclusive) and .25 m/s^2 (exclusive). mv2 will be 1 for each time period where the mean movement is greater than 1 m/s^2 (inclusive).

Examples

All examples will use the following globals as folder paths:

global project "C:\Users\username\Documents\ProjectA"

global media "$project\raw_data\media"

global output "$project\outputs"

Example 1.

sctostreamcsv, mediafolder("$media") outputfolder("$output") quiet still

This is a very simple way to run the command. The command will read sensor stream .csv files in the media folder with the prefix SL (because quiet was used) and the prefix MV (because still was used) and create the quiet variable in the SL files and the still variable in the MV files and save the new files in the output folder. If a file already exists in the output folder then it is skipped. Any sensor stream files with prefix SP or LL in the media folder will be ignored as no statistic applicable to either of those streams was specified.

Example 2.

sctostreamcsv, mediafolder("$media") outputfolder("$output") quiet slbetween(loud(60 ?)) replace

In this example, the command will only read SL sensor stream .csv files from the media folder as sound level is the only sensor for which statistics are specified. The quiet and loud variables will be added to all SL files in the media folder. Any files in the output folder already in the output folder will be overwritten if the corresponding original file is still in the media folder.

Author

This command was developed by SurveyCTO.

See this command's repository for more information where you can also submit feedback and feature requests.