ensemble_export_respnstim

 tabulates subject-level and stimulus-based response-level data, exports one row per stim

 function outData = ensemble_export_respnstim(inData,params)

 This function takes given response table data, extracts data that are
 subject-level data (such as scores on a personality scale), extracts
 stimulus-level data (such as the responses made to each stimulus) and
 returns data in a form where there is one row per stimulus presented per
 subject, and any subject-level scores are applied to each row for that
 subject. Data can be saved to a demilited file (comma default, other
 delimiters can be specified), and an optional SAS script can be generated
 to import the data into a SAS dataset.
 
 INPUT/PARAMS
 
  inData should be a cell array of structs, containing the following
    possible data:
    - the 'response_data' struct from ensemble_load_expinfo - REQUIRED
    - the 'stimulus_metadata' struct from ensemble_load_expinfo - OPTIONAL
    - the 'subject_info' struct from ensemble_load_expinfo - OPTIONAL
    - the results of any function that has calculated subject-level
    constants and contains a subject_id column. currently, all vars other
    than subject_id that are passed on will be included in the output data
    as constants for that subject.
 
  params.filt - any filtering criteria found here are applied at once to
    the response_data struct. This can be utilized to include or exclude
    forms, questions, subquestions, sessions, subjects, etc (AND compqids)
    NOTE: make sure not to limit your filtering parameters so that only
    questions for a given stimulus or form are included, IF you are
    passing that data onto a function that is called within this script
    (through the params.export.by_subject.(fieldname) or
    .by_stimulus.(fieldname) mechanisms) for subject or stimulus-level
    constants
 
  params.export.by_subject.(fieldname) - fieldname can either be the
    analysis struct name of a dataset within inData, or the name of a
    function to be executed to return a dataset, containing a subject_id
    variable and data that you would like to use as subject-level constant
    data. - OPTIONAL
    NOTE: if any given dataset contains more than one row per
    subject_id, a warning will be issued, and this script will output only
    the first observation for that subject_id
  params.export.by_subject.(fieldname).vars - a cell array of strings of
    vars to export, from the dataset provided by (fieldname) - OPTIONAL
    NOTE: if vars in this cell array do not exist within the output of
    (fieldname), then all vars in (fieldname) will be exported as
    subject-level constants
 
  params.export.by_stimulus.(fieldname) - similar to
    params.export.by_subject.(fieldname), except for stimulus-level
    metadata ... the return of this function handle or inData struct will
    be treated as stimulus-level constants - OPTIONAL
  params.export.by_stimulus.(fieldname).vars - similar to
    params.export.by_subject.(fieldname).vars, except for the output of
    params.export.by_stimulus.(fieldname) - OPTIONAL
 
  params.export.non_num_delim - if present, the value of this param is
    placed before and after non-numeric cell values when exporting (i.e.
    single or double quotes around non-numeric cells)
  params.export.delimiter - character to delimit cell values. if none is
    provided, a tab is used
  params.export.var_name_map - a struct whose fieldnames are the
    calculated fieldnames from question/subquestion ids. These fieldnames
    will be located within the outData.vars array and converted to their
    values within params.export.var_name_map. For instance, if
    params.export.var_name_map.s123_01 = 'renamedq', and if question 123,
    subquestion 1 exists within the dataset, the varname for this question
    in the output will be renamed to 'renamedq'. Original var names are
    returned in the outData.origvars field.
 
  params.export.R - currently a boolean that indicates whether the output
    file should be formatted in a manner that facilitates loading into R.

  params for ensemble_init_fid
    params.write2file, params.print, params.fname, params.filemode
  params.sas.fname - REQUIRED if you want to automatically generate a SAS
    import file for your data.
  params.sas.options - an optional 'options' string for the beginning of
    your sas file
  params.sas.libsave - if you set this = 1, and you have defined
    params.sas.libpath and params.sas.libname, it will include the command
    to save your imported data to a SAS data library.
  params.sas.libpath - the path to your SAS data library directory
  params.sas.libname - the filename that will be given to your new SAS
    data library, if you have chosen to save this dataset as a library

 OUTPUT
  outData - contains one row per stimulus per subject, one column for
    subject_id, stimulus_id, and trial_id, one column per subject-level
    score calculated, one column per stimulus_metadata var that was
    included in the inData struct, and one column per stimulus-level qnum.
  outData.origvars - the original variable names, most of which will be
    constructed from question and subquestion numbers. This is provided as
    a reference for re-named variables (see params.
 
 NOTE: the function gets it's list of valid subject_ids from
 response_data, so if this is not required, the function will return
 empty. But ... if you provide response_data, and then do not specify any
 params.export.by_stimulus fields, then it will only return subject-level
 data. this is a bit messy, a better way to do it would be to glean
 subject ids from the 'subject_info' struct, and make it a hard
 requirement, but that is not implemented yet ... it was coded this way at
 first since this is supposed to be mainly a function with a
 stimulus-response level focus, but there is room for improvement here so
 I will say that this warrants a FIXME
 
 NOTE: if a given subject-level score variable contains more than one of
 any given score for a subject, the first of each will be used. also, if
 more than one iteration of a qnum exists for any given question/stimulus
 pair, the first of the list will be reported for that stimulus
 
 NOTE: this script assumes that a given question will only be asked once
 for a given stimulus_id within a given subject. If the same compqid is
 found more than once for a given stimulus within a given subject's data,
 the first response will be returned and the subsequent responses will be
 ignored.
 
 FIXME - need to extract/integrate data from the misc_info column of a
 given response table, include in the row for each stimulus ... up to this
 point, I have dealt with data stored in misc info by writing a separate
 function to extract that data and add it as another variable in the
 response_data struct. This might be a good way to deal with misc_info dat
 moving forward.

 SEE ALSO: ensemble_print_datast()

0001 function outData = ensemble_export_respnstim(inData,params)
0002 
0003 % tabulates subject-level and stimulus-based response-level data, exports one row per stim
0004 %
0005 % function outData = ensemble_export_respnstim(inData,params)
0006 %
0007 % This function takes given response table data, extracts data that are
0008 % subject-level data (such as scores on a personality scale), extracts
0009 % stimulus-level data (such as the responses made to each stimulus) and
0010 % returns data in a form where there is one row per stimulus presented per
0011 % subject, and any subject-level scores are applied to each row for that
0012 % subject. Data can be saved to a demilited file (comma default, other
0013 % delimiters can be specified), and an optional SAS script can be generated
0014 % to import the data into a SAS dataset.
0015 %
0016 % INPUT/PARAMS
0017 %
0018 %  inData should be a cell array of structs, containing the following
0019 %    possible data:
0020 %    - the 'response_data' struct from ensemble_load_expinfo - REQUIRED
0021 %    - the 'stimulus_metadata' struct from ensemble_load_expinfo - OPTIONAL
0022 %    - the 'subject_info' struct from ensemble_load_expinfo - OPTIONAL
0023 %    - the results of any function that has calculated subject-level
0024 %    constants and contains a subject_id column. currently, all vars other
0025 %    than subject_id that are passed on will be included in the output data
0026 %    as constants for that subject.
0027 %
0028 %  params.filt - any filtering criteria found here are applied at once to
0029 %    the response_data struct. This can be utilized to include or exclude
0030 %    forms, questions, subquestions, sessions, subjects, etc (AND compqids)
0031 %    NOTE: make sure not to limit your filtering parameters so that only
0032 %    questions for a given stimulus or form are included, IF you are
0033 %    passing that data onto a function that is called within this script
0034 %    (through the params.export.by_subject.(fieldname) or
0035 %    .by_stimulus.(fieldname) mechanisms) for subject or stimulus-level
0036 %    constants
0037 %
0038 %  params.export.by_subject.(fieldname) - fieldname can either be the
0039 %    analysis struct name of a dataset within inData, or the name of a
0040 %    function to be executed to return a dataset, containing a subject_id
0041 %    variable and data that you would like to use as subject-level constant
0042 %    data. - OPTIONAL
0043 %    NOTE: if any given dataset contains more than one row per
0044 %    subject_id, a warning will be issued, and this script will output only
0045 %    the first observation for that subject_id
0046 %  params.export.by_subject.(fieldname).vars - a cell array of strings of
0047 %    vars to export, from the dataset provided by (fieldname) - OPTIONAL
0048 %    NOTE: if vars in this cell array do not exist within the output of
0049 %    (fieldname), then all vars in (fieldname) will be exported as
0050 %    subject-level constants
0051 %
0052 %  params.export.by_stimulus.(fieldname) - similar to
0053 %    params.export.by_subject.(fieldname), except for stimulus-level
0054 %    metadata ... the return of this function handle or inData struct will
0055 %    be treated as stimulus-level constants - OPTIONAL
0056 %  params.export.by_stimulus.(fieldname).vars - similar to
0057 %    params.export.by_subject.(fieldname).vars, except for the output of
0058 %    params.export.by_stimulus.(fieldname) - OPTIONAL
0059 %
0060 %  params.export.non_num_delim - if present, the value of this param is
0061 %    placed before and after non-numeric cell values when exporting (i.e.
0062 %    single or double quotes around non-numeric cells)
0063 %  params.export.delimiter - character to delimit cell values. if none is
0064 %    provided, a tab is used
0065 %  params.export.var_name_map - a struct whose fieldnames are the
0066 %    calculated fieldnames from question/subquestion ids. These fieldnames
0067 %    will be located within the outData.vars array and converted to their
0068 %    values within params.export.var_name_map. For instance, if
0069 %    params.export.var_name_map.s123_01 = 'renamedq', and if question 123,
0070 %    subquestion 1 exists within the dataset, the varname for this question
0071 %    in the output will be renamed to 'renamedq'. Original var names are
0072 %    returned in the outData.origvars field.
0073 %
0074 %  params.export.R - currently a boolean that indicates whether the output
0075 %    file should be formatted in a manner that facilitates loading into R.
0076 %
0077 %  params for ensemble_init_fid
0078 %    params.write2file, params.print, params.fname, params.filemode
0079 %  params.sas.fname - REQUIRED if you want to automatically generate a SAS
0080 %    import file for your data.
0081 %  params.sas.options - an optional 'options' string for the beginning of
0082 %    your sas file
0083 %  params.sas.libsave - if you set this = 1, and you have defined
0084 %    params.sas.libpath and params.sas.libname, it will include the command
0085 %    to save your imported data to a SAS data library.
0086 %  params.sas.libpath - the path to your SAS data library directory
0087 %  params.sas.libname - the filename that will be given to your new SAS
0088 %    data library, if you have chosen to save this dataset as a library
0089 %
0090 % OUTPUT
0091 %  outData - contains one row per stimulus per subject, one column for
0092 %    subject_id, stimulus_id, and trial_id, one column per subject-level
0093 %    score calculated, one column per stimulus_metadata var that was
0094 %    included in the inData struct, and one column per stimulus-level qnum.
0095 %  outData.origvars - the original variable names, most of which will be
0096 %    constructed from question and subquestion numbers. This is provided as
0097 %    a reference for re-named variables (see params.
0098 %
0099 % NOTE: the function gets it's list of valid subject_ids from
0100 % response_data, so if this is not required, the function will return
0101 % empty. But ... if you provide response_data, and then do not specify any
0102 % params.export.by_stimulus fields, then it will only return subject-level
0103 % data. this is a bit messy, a better way to do it would be to glean
0104 % subject ids from the 'subject_info' struct, and make it a hard
0105 % requirement, but that is not implemented yet ... it was coded this way at
0106 % first since this is supposed to be mainly a function with a
0107 % stimulus-response level focus, but there is room for improvement here so
0108 % I will say that this warrants a FIXME
0109 %
0110 % NOTE: if a given subject-level score variable contains more than one of
0111 % any given score for a subject, the first of each will be used. also, if
0112 % more than one iteration of a qnum exists for any given question/stimulus
0113 % pair, the first of the list will be reported for that stimulus
0114 %
0115 % NOTE: this script assumes that a given question will only be asked once
0116 % for a given stimulus_id within a given subject. If the same compqid is
0117 % found more than once for a given stimulus within a given subject's data,
0118 % the first response will be returned and the subsequent responses will be
0119 % ignored.
0120 %
0121 % FIXME - need to extract/integrate data from the misc_info column of a
0122 % given response table, include in the row for each stimulus ... up to this
0123 % point, I have dealt with data stored in misc info by writing a separate
0124 % function to extract that data and add it as another variable in the
0125 % response_data struct. This might be a good way to deal with misc_info dat
0126 % moving forward.
0127 %
0128 % SEE ALSO: ensemble_print_datast()
0129 
0130 
0131 % FB 12/17/07 - started scripting
0132 % FB 10/25/09 - added support for multiple trials with the same stimulus.
0133 % If a stimulus is presented multiple times, the trial number should be
0134 % incremented, and if this is done, export_respnstim will use the trial
0135 % number to dissociate responses to the same stimulus at different
0136 % presentations.
0137 % FB 01/22/10 - added 'response_order' to the output for 'by_stimulus'
0138 % output ... now outputs the response order of the first response for a
0139 % given session/subject/stimulus/trial. In this way, if you sort your data
0140 % by stimulus, or any other variable, you can later retrieve the order of
0141 % presentation of stimuli by sorting by response_order.
0142 % PJ 24/09/10 - fixed passing of qnums to make_valid_struct_key
0143 % PJ 04May2012 - improved handling of question type to differentiate enums
0144 % from text and numeric data, e.g. reaction times, written to response_text
0145 % PJ 29May2012 - fixed handling of multi-part questions; fixed indexing
0146 % when overall list of questions involves checkbox enums; fixed handling
0147 % of output datatypes so that they are dynamically determined based on
0148 % qinfo
0149 % PJ 13Jan2013 - fixed handling of subquestions
0150 % PJ 02May2013 - added handling of case when a stimulus_id is not found in
0151 %                a particular list
0152 % PJ 04May2013 - if a file intended as an input file for R is being written,
0153 %                added 2nd row of output specifying variable datatypes
0154 % PJ 10May2013 - optimized code by replacing costly and unnecessarily
0155 %                repeated calls to ismember(), among other things
0156 % PJ 28Dec2013 - added default handling when question datatype is 'enum',
0157 %                but html_field_type is ''.
0158 % PJ 26Feb2014 - fixed strange handling of trialMask (see additional
0159 %                comment below with this date stamp)
0160 % PJ 11Mar2014 - tried further to improve use of either trialMask or
0161 %                trialMaskMtx
0162 
0163 % % initialize output data struct
0164 outData = ensemble_init_data_struct;
0165 outData.type = 'resp_x_stim'; % unique analysis type
0166 outData.vars = {}; % names of columns that will end up inas outData.data
0167 outData.datatype = {}; % tracks non-numeric data, for SAS output
0168 if isfield(params,'outDataName') && ~isempty(params.outDataName)
0169     outData.name = params.outDataName;
0170 else
0171     outData.name = 'resp_x_stim';
0172 end;
0173 
0174 % % check for export criteria
0175 if ~isfield(params,'export')
0176     % if no export criteria are provided, we can not process export data
0177     warning('no export params specified');
0178     return
0179 end
0180 
0181 % % check inData, convert to cell if needed
0182 if ~iscell(inData)
0183     inData = {inData};
0184 end
0185 
0186 % % get response data, filter
0187 rdSearchCrit.name = 'response_data';
0188 respIdx = ensemble_find_analysis_struct(inData,rdSearchCrit);
0189 if respIdx
0190     rsData = inData{respIdx};
0191     % check for, generate compqids
0192     rsData = ensemble_check_compqid(rsData);
0193 else
0194     % if no response data are present, you can't export them
0195     warning('no response_data struct was found');
0196     return
0197 end
0198 
0199 % apply filtering criteria, if present
0200 if isfield(params,'filt')
0201   rsData = ensemble_filter(rsData,params.filt);
0202 end
0203 rsCols = set_var_col_const(rsData.vars);
0204 
0205 % See if we are outputing for R
0206 usingR = isfield(params.export,'R') && (isstruct(params.export.R) || params.export.R);
0207 
0208 % See whether we are converting checkbox NaNs to false or leaving them as
0209 % NaNs
0210 if isfield(params.export, 'convertCheckboxNaNToFalse')
0211   convertCheckboxNaNToFalse = params.export.convertCheckboxNaNToFalse;
0212 else
0213   convertCheckboxNaNToFalse = 0;
0214 end
0215 
0216 % % initialize subject-level data structure
0217 % sNum is an auto-incrementing subject number within this dataset
0218 sub_st.vars={'subject_id','sNum'};
0219 % sub_st.datatype tracks data type
0220 %   (s)tring
0221 %   (n)umeric
0222 %   (l)ogical
0223 sub_st.datatype = {'s','n'};
0224 sub_st.data{1} = unique(rsData.data{rsCols.subject_id});
0225 subs = sub_st.data{1};
0226 nsub = length(sub_st.data{1});
0227 sub_st.data{2} = 1:nsub;
0228 
0229 
0230 % % extract/calculate subject-level scores
0231 if isfield(params.export,'by_subject')
0232 
0233     % % cycle through params.export.by_subject fieldnames
0234     % % if struct is found in inData, load that data (expects fldname)
0235     % % if struct is NOT found in inData, try to excute it as a function
0236     % % if no subject_id var in output or struct, warn & continue
0237     % % if include.(fldname).vars, include only those vars
0238     subinc = params.export.by_subject;
0239     subfldnames = fieldnames(subinc);
0240     for ifld=1:length(subfldnames)
0241         lsData = {};
0242         lfld = subfldnames(ifld);
0243         if iscell(lfld), lfld = lfld{1}; end
0244         lSchCrit.name = lfld;
0245         lIdx = ensemble_find_analysis_struct(inData,lSchCrit);
0246         if lIdx
0247             lsData = inData{lIdx};
0248         else
0249             lfh = str2func(lfld);
0250             try lsData = lfh(rsData,params);
0251             catch
0252                 lsData = {};
0253             end;
0254         end 
0255         if ~isempty(lsData) && isstruct(lsData)
0256             lsCols = set_var_col_const(lsData.vars);
0257 
0258             % sanity check, make sure one row per subject
0259             if isfield(lsCols,'subject_id')
0260                 usubs = length(unique(lsData.data{lsCols.subject_id}));
0261                 if usubs
0262                     nrows = length(lsData.data{lsCols.subject_id});
0263                     if nrows ~= usubs
0264                         warning(['subject-level data (%s) did not have '...
0265                             'one row per subject'],lfld)
0266                     end
0267                 end
0268             else
0269                 warning(['subject-level data (%s) did not have '...
0270                     'a val named ''subject_id'''],lfld)
0271                 continue
0272             end
0273             % if 'vars' field found under params.export.by_subject.(lfld)
0274             % then mask out all other vars & data
0275             if isfield(subinc.(lfld),'vars') && iscell(subinc.(lfld).vars)
0276                 xvars = subinc.(lfld).vars;
0277                 if ~ismember(lsData.vars,xvars)
0278                     warning(['vars defined for %s can not be found, '...
0279                         'exporting all vars for this subject-level '...
0280                         'data source',lfld]);
0281                 else
0282                     if ~ismember(xvars,'subject_id')
0283                         xvars = [xvars 'subject_id'];
0284                     end
0285                     xmask = ismember(lsData.vars,xvars);
0286                     lsData.vars = lsData.vars(xmask);
0287                     lsData.data = lsData.data(xmask);
0288                 end
0289             end
0290             lsMask = ~ismember(lsData.vars,{'subject_id','session_id'});
0291             sub_st.vars = [sub_st.vars lsData.vars{lsMask}];
0292             lsIdxs = find(lsMask);
0293             % check, track, data type (s)tring or (n)umeric
0294             for iidx = 1:length(lsIdxs)
0295                  if isnumeric(lsData.data{lsIdxs(iidx)})
0296                      dtype = 'n';
0297                  elseif islogical(lsData.data{lsIdxs(iidx)})
0298                      dtype = 'l';
0299                  else
0300                      dtype = 's';
0301                  end
0302                  sub_st.datatype = [sub_st.datatype dtype];
0303             end
0304             vidxs = find(ismember(sub_st.vars,{lsData.vars{lsMask}}));
0305             nvars = length(vidxs);
0306             for ivar = 1:nvars
0307                 vi = vidxs(ivar);
0308                 if sub_st.datatype{vi} == 's'
0309                     sub_st.data{vi} = cell(nsub,1);
0310                 else
0311                     sub_st.data{vi} = zeros(nsub,1);
0312                 end
0313             end
0314 
0315             % parse rows
0316             for isub = 1:nsub
0317                 sidx = find(ismember(lsData.data{lsCols.subject_id},subs{isub}));
0318                 if length(sidx) > 1
0319                   sidx = sidx(1);
0320                 elseif isempty(sidx)
0321                   continue
0322                 end
0323                 for ivar = 1:nvars
0324                     vi = vidxs(ivar);
0325                     li = find(ismember(lsData.vars,sub_st.vars{vi}));
0326                     if sub_st.datatype{vi} == 's'
0327                         svdata = lsData.data{li}{sidx};
0328                     else
0329                         svdata = lsData.data{li}(sidx);
0330                     end
0331                     svdata = sanitize_cell_value(svdata,sub_st.datatype{vi});
0332                     if sub_st.datatype{vi} == 's'
0333                         sub_st.data{vi}{isub} = svdata;
0334                     else
0335                         sub_st.data{vi}(isub) = svdata;
0336                     end
0337                 end
0338             end
0339         else
0340             warning('couldn''nt find inData struct or function handle for %s',...
0341                 lfld);
0342         end % for ~isempty(lsData
0343     end % for ifld=1:length(subfldnames
0344 end % if isfield(params.export,'by_subject'
0345 
0346 % initalize vars
0347 outData.vars = sub_st.vars;
0348 outData.datatype = sub_st.datatype;
0349 
0350 % % extract/calculate stim-level scores
0351 % filter by each stim-level qnum, loop over each subject, data2enum and
0352 % store scores
0353 if isfield(params.export,'by_stimulus')
0354 
0355     % add var for trial_id, stim repetition number, and response order
0356     outData.vars = [outData.vars 'trial_id' 'stim_rep' 'response_order'];
0357     outData.datatype = [outData.datatype 'n' 'n' 'n'];
0358     
0359     % % initialize stimulus-level data structure
0360     smData.vars={'stimulus_id'};
0361     smData.datatype = {'n'}; % tracks data type, (s)tr, (n)um, (l)ogical
0362     smData.data = {[]};
0363     smCols = set_var_col_const(smData.vars);
0364     smIdxs = [];
0365     stims = [];
0366 
0367     if isstruct(params.export.by_stimulus)
0368         stiminc = params.export.by_stimulus;
0369         stimfldnames = fieldnames(stiminc);
0370         for ifld=1:length(stimfldnames)
0371             sfld = stimfldnames(ifld);
0372             if iscell(sfld)
0373                 sfld = sfld{1};
0374             end
0375             sSchCrit.name = sfld;
0376             sIdx = ensemble_find_analysis_struct(inData,sSchCrit);
0377             if sIdx
0378                 lsData = inData{sIdx};
0379             else
0380                 sfh = str2func(sfld);
0381                 try lsData = sfh(rsData,params);
0382                 catch
0383                     lsData = {};
0384                 end;
0385             end
0386 
0387             if ~isempty(lsData) && isstruct(lsData)
0388                 lsCols = set_var_col_const(lsData.vars);
0389 
0390                 % sanity check, make sure one row per stimulus
0391                 if isfield(lsCols,'stimulus_id')
0392                     ustim = unique(lsData.data{lsCols.stimulus_id});
0393                     if ~isnumeric(ustim)
0394                         warning(['stimulus_id column for %s is non-numeric, '...
0395                             'please use only numeric stimulus_ids. %s will '...
0396                             'not be included as stim-level constant data'],...
0397                             sfld,sfld);
0398                         continue;
0399                     end
0400                     if ustim
0401                         nrows = length(lsData.data{lsCols.stimulus_id});
0402                         if nrows ~= length(ustim)
0403                             warning(['stimulus-level data (%s) did not have '...
0404                                 'one row per stimulus'],sfld)
0405                         end
0406                     end
0407                     % add stimulus ids that aren't already in smData
0408                     newstim = setdiff(ustim,stims);
0409                     if newstim
0410                         smData.data{smCols.stimulus_id} = [stims newstim];
0411                         stims = smData.data{smCols.stimulus_id};
0412                         nstim = length(stims);
0413                         % initialize smData.data cols for each new row
0414                         for ivar = 2:length(smData.vars)
0415                             if smData.datatype{ivar} == 's'
0416                                 smData.vars{ivar}{nstim} = [];
0417                             else
0418                                 smData.data{ivar}(nstim) = 0;
0419                             end
0420                         end
0421                     end
0422                 else
0423                     warning(['stimulus-level data (%s) did not have '...
0424                         'a val named ''stimulus_id'''],sfld)
0425                     continue
0426                 end
0427 
0428                 % if 'vars' field found under params.export.by_stimulus.(sfld)
0429                 % then mask out all other vars & data
0430                 if isfield(stiminc.(sfld),'vars') && iscell(stiminc.(sfld).vars)
0431                     xvars = stiminc.(sfld).vars;
0432                     if ~any(ismember(lsData.vars,xvars))
0433                         warning(['vars defined for %s can not be found, '...
0434                             'exporting all vars for this stimulus-level '...
0435                             'data source',sfld]);
0436                     else
0437                         if ~ismember(xvars,'stimulus_id')
0438                             xvars = [xvars 'stimulus_id'];
0439                         end
0440                         xmask = ismember(lsData.vars,xvars);
0441                         lsData.vars = lsData.vars(xmask);
0442                         lsData.data = lsData.data(xmask);
0443                     end
0444                 end
0445                 lsMask = ~ismember(lsData.vars,{'stimulus_id'});
0446                 lsIdxs = find(lsMask);
0447                 smData.vars = [smData.vars lsData.vars{lsMask}];
0448                 smCols = set_var_col_const(smData.vars);
0449                 smIdxs = find(~ismember(smData.vars,'stimulus_id'));
0450                 % check, track, data type (s)tring or (n)umeric
0451                 for iidx = 1:length(lsIdxs)
0452                      if isnumeric(lsData.data{lsIdxs(iidx)})
0453                          dtype = 'n';
0454                      elseif islogical(lsData.data{lsIdxs(iidx)})
0455                          dtype = 'l';
0456                      else
0457                          dtype = 's';
0458                      end
0459                      smData.datatype = [smData.datatype dtype];
0460                 end
0461                 vidxs = find(ismember(smData.vars,{lsData.vars{lsMask}}));
0462                 nvars = length(vidxs);
0463                 for ivar = 1:nvars
0464                     vi = vidxs(ivar);
0465                     if smData.datatype{vi} == 's'
0466                         smData.data{vi} = cell(nstim,1);
0467                     else
0468                         smData.data{vi} = zeros(nstim,1);
0469                     end
0470                 end
0471 
0472                 % parse rows
0473                 for istim = 1:nstim
0474                     sidx = find(ismember(lsData.data{lsCols.stimulus_id},stims(istim)));
0475                     if isempty(sidx)
0476                       continue
0477                     end
0478                     if length(sidx) > 1
0479                         sidx = sidx(1);
0480                     end
0481                     for ivar = 1:nvars
0482                         vi = vidxs(ivar);
0483                         li = find(ismember(lsData.vars,smData.vars{vi}));
0484                         if ~isempty(li)
0485                             if smData.datatype{vi} == 's'
0486                                 svdata = lsData.data{li}{sidx};
0487                             else
0488                                 svdata = lsData.data{li}(sidx);
0489                             end
0490                             svdata = sanitize_cell_value(svdata,smData.datatype{vi});
0491                             if smData.datatype{vi} == 's'
0492                                 smData.data{vi}{istim} = svdata;
0493                             else
0494                                 smData.data{vi}(istim) = svdata;
0495                             end % if smData.datatype{vi
0496                         end % if ~isempty(li
0497                     end % for ivar=1:
0498                 end % for istim = 1:
0499             else 
0500                 warning('couldn''nt find inData struct or function handle for %s',...
0501                     sfld);
0502             end % for ~isempty(lsData
0503         end % for ifld=1:length(stimfldnames
0504     end % isstruct(params.export.by_stimulus
0505 
0506     outData.vars = [outData.vars smData.vars];
0507     outData.datatype = [outData.datatype smData.datatype];
0508 
0509     cqids = {};
0510     ustims = {};
0511     % filter out non-stimulus response data
0512     ustims = unique(rsData.data{rsCols.stimulus_id});
0513     stimidxs = ~isnan(ustims);
0514     ustims = ustims(stimidxs);
0515     filt = {};
0516     filt.include.all.stimulus_id = ustims;
0517     aqData = ensemble_filter(rsData,filt);
0518     % get unique qnums - now compqids
0519     cqids.vars = {'compqid','question_id','subquestion',...
0520         'qinfo','scqid','bitmask'};
0521     cqCols = set_var_col_const(cqids.vars);
0522     cqScq = cqCols.scqid;
0523     cqCqi = cqCols.compqid;
0524     cqQid = cqCols.question_id;
0525     cqSub = cqCols.subquestion;
0526     cqQin = cqCols.qinfo;
0527     cqBit = cqCols.bitmask;
0528     ucqids = unique(aqData.data{rsCols.compqid});
0529     ucqididxs = find(~isnan(ucqids));
0530     nucqid = length(ucqididxs);
0531     for icqc = 1:length(cqids.vars)
0532         cqids.data{icqc} = cell(nucqid,1);
0533     end
0534     cqids.data{cqScq} = ucqids(ucqididxs);
0535 
0536     qnums = {};
0537     % parse out checkbox enums
0538     for icq = 1:length(cqids.data{cqScq})
0539         % check each cqid
0540         % could be converted to 'ensemble_compqid2dfid'
0541         [qidx] = find(ismember(rsData.data{rsCols.compqid},cqids.data{cqScq}(icq)));
0542         if length(qidx) > 1
0543             qidx = qidx(1);
0544         end
0545         cqids.data{cqQid}(icq) = num2cell(rsData.data{rsCols.question_id}(qidx));
0546         cqids.data{cqSub}(icq) = num2cell(rsData.data{rsCols.subquestion}(qidx));
0547         lcqid = cqids.data{cqScq}(icq);
0548         if iscell(lcqid)
0549             lcqid = lcqid{1};
0550         end
0551         cqids.data{cqCqi}(icq) = {sprintf('%1.2f',lcqid)};
0552         lqid = cqids.data{cqQid}(icq);
0553         if iscell(lqid)
0554             lqid = lqid{1};
0555         end
0556         lsqid = cqids.data{cqSub}(icq);
0557         if iscell(lsqid)
0558             lsqid = lsqid{1};
0559         end
0560         lqinfo = mysql_extract_metadata('table','question',...
0561             'question_id',lqid,'conn_id',params.mysql.conn_id);
0562         lqicq = [lqinfo.subquestion] == lsqid;
0563         cqids.data{cqQin}(icq) = {lqinfo(lqicq)}; % cqids.data{cqSub}{icq}
0564         uhft = {lqinfo.html_field_type};
0565         if ~isempty(uhft{lqicq}) && ~isempty(strmatch('checkbox',uhft{lqicq}))
0566             % get dfid for qid, expand qnums for this compqid
0567             ueval = {lqinfo.enum_values};
0568             for istr = 1:length(ueval{lsqid})
0569                 lqnum = sprintf('%1.2f_c%02d',lcqid,istr);
0570                 qnums = [qnums lqnum];
0571             end
0572         else
0573             qnums = [qnums cqids.data{cqCqi}(icq)];
0574         end
0575     end
0576 
0577     % add vars/init data cells
0578     if iscell(qnums)
0579       tmp_qnums = qnums;
0580     else
0581       tmp_qnums = num2cell(qnums);
0582     end
0583     qStructs = make_valid_struct_key(tmp_qnums);
0584     numconst = length(outData.vars);
0585     for iadvar = numconst+1:numconst+length(qnums)
0586       outData.data{iadvar} = [];
0587       
0588       % Figure out which compqid row we are dealing with
0589       stridx = regexp(qnums{iadvar-numconst},'_c\d{2}');
0590       if isempty(stridx)
0591         cqid = qnums{iadvar-numconst};
0592       else
0593         cqid = qnums{iadvar-numconst}(1:stridx-1);
0594       end
0595       cqidrow = find(ismember(cqids.data{cqCqi}, cqid));
0596       switch cqids.data{cqQin}{cqidrow}.type
0597         case {'text','varchar'}
0598           cdt = 's';
0599         case {'enum'} % distinguish between checkboxes and scales
0600           switch cqids.data{cqQin}{cqidrow}.html_field_type
0601             case 'radiogroup'
0602               cdt = 'n';
0603             case 'checkbox'
0604               cdt = 'l';
0605             otherwise
0606               cdt = 'n';
0607           end
0608         otherwise
0609           cdt = 'n';
0610       end
0611       outData.datatype = [outData.datatype cdt];
0612     end
0613     outData.vars = [outData.vars qStructs];
0614     outCols = set_var_col_const(outData.vars);
0615     
0616     % 06Jul2013 PJ - this is weird. Why is this here? outData.data needs to
0617     % be filled down rows, but initializing like this alone does not force
0618     % filling down rows.
0619     nrows = 0;  
0620     
0621     for ivar = 1:length(outData.vars)
0622         if outData.datatype{ivar} == 's'
0623             outData.data{ivar} = cell(nrows,1);
0624         else
0625             outData.data{ivar} = zeros(nrows,1);
0626         end
0627     end
0628     
0629     % Create mask matrices that we can use to access the data
0630     [subMask, subIDs] = make_mask_mtx(rsData.data{rsCols.subject_id});
0631     [stimMask, stimIDs] = make_mask_mtx(rsData.data{rsCols.stimulus_id});
0632     nstim = length(stimIDs);
0633     if any(~isnan(rsData.data{rsCols.trial_id}))
0634       [trialMaskMtx, trialIDs] = make_mask_mtx(rsData.data{rsCols.trial_id});
0635       usingTrialMaskMtx = 1;
0636       trialMask = [];
0637     else
0638       usingTrialMaskMtx = 0;
0639       trialMask = ones(size(rsData.data{rsCols.trial_id}));
0640     end
0641     
0642     % row counter
0643     outRow = 0;
0644     
0645     % % collate stim-level data
0646     for isub = 1:nsub
0647         subid = subIDs(isub);
0648 
0649         for istim = 1:nstim
0650           currStimID = stimIDs(istim);
0651           
0652           % Make sure this subject encountered this stimulus
0653           if ~any(subMask(:,isub) & stimMask(:,istim))
0654             continue
0655           end
0656 
0657           if exist('trialMaskMtx','var')
0658             uTrials = unique(rsData.data{rsCols.trial_id}(subMask(:,isub) & stimMask(:,istim)));
0659             nTrials = ~isnan(uTrials);
0660             nUt = sum(nTrials);
0661           else
0662             uTrials = NaN;
0663             nUt = 1;
0664           end
0665 
0666           for in=1:nUt
0667             % 26Feb2014 PJ - the current conditional is really odd. It
0668             % assumes that a situation with the unique number of trials of
0669             % 1 cannot be valid. It seems the test here should be whether
0670             % uTrials is NaN, in which case it should throw an error
0671             % because it cannot construct the trial mask. Currently it
0672             % throws an error because trialMask is undefined.
0673             %if nUt > 1 % original code
0674             % 11Mar2014 PJ - added usingTrialMaskMtx check to disambiguate
0675             %    the two possible conditions
0676             if nUt && usingTrialMaskMtx
0677               trialMask = trialMaskMtx(:,trialIDs == uTrials(in));
0678             elseif isempty(trialMask)
0679               error('Failed to construct trialMask');
0680             end
0681            
0682             rsMask = subMask(:,isub) & stimMask(:,istim) & trialMask;
0683             currCompIdxs = find(rsMask);
0684             
0685             outRow = outRow + 1;
0686             
0687             % set id vars
0688             outData.data{outCols.subject_id}(outRow,1) = subid;
0689             outData.data{outCols.sNum}(outRow,1) = isub;
0690             outData.data{outCols.stimulus_id}(outRow,1) = currStimID;
0691             outData.data{outCols.trial_id}(outRow,1) = uTrials(in);
0692             outData.data{outCols.stim_rep}(outRow,1) = in;
0693             
0694             % set subject-level scores
0695             % start at 3, since subject_id and sNum are always 1 and 2
0696             for icon = 3:length(sub_st.vars)
0697                 outData.data{icon}(outRow,1) = sub_st.data{icon}(isub);
0698             end
0699             % set stimulus metadata
0700             if smIdxs
0701                 stimidx = ismember(smData.data{smCols.stimulus_id},currStimID);
0702                 for ismd = 1:length(smIdxs)
0703                     smdidx = smIdxs(ismd);
0704                     smvar = smData.vars{smdidx};
0705                     lsmpt = smData.data{smdidx}(stimidx);
0706                     ltype = outData.datatype{outCols.(smvar)};
0707                     lsmpt = sanitize_cell_value(lsmpt,ltype);
0708                     if ltype == 's'
0709                         outData.data{outCols.(smvar)}{outRow,1} = lsmpt;
0710                     else
0711                         outData.data{outCols.(smvar)}(outRow,1) = lsmpt;
0712                     end
0713                 end
0714             end
0715 
0716             % set response_order
0717             outData.data{outCols.response_order}(outRow,1) = rsData.data{rsCols.response_order}(find(rsMask,1,'first'));
0718             for iq = 1:length(qnums)
0719                 qi = qnums(iq);
0720                 if iscell(qi)
0721                     qi = qi{1};
0722                 end
0723                 % if not checkbox, go for it, if check box, parse out vals
0724                 qcidx = regexp(qi,'_c\d{2}');
0725                 if iscell(qcidx)
0726                     qcidx = qcidx{1};
0727                 end
0728                 qdata = {};
0729                 if ~isempty(qcidx)
0730                     % data2bitmask, get items
0731                     lcqid = qi(1:qcidx-1);
0732                     lcqidx = find(strcmp(lcqid,cqids.data{cqCqi}));
0733                     lqinfo = cqids.data{cqQin}(lcqidx);
0734                     if (length(lqinfo)>1)
0735                         lqi = strmatch('checkbox',lqinfo.html_field_type);
0736                     else
0737                         lqi = 1;
0738                     end
0739                     evals = {lqinfo{lqi}.enum_values};
0740                     nenum = length(evals{lqi});
0741                     lqid  = {lqinfo{lqi}.question_id};
0742                     if iscell(lqid)
0743                         lqid = lqid{1};
0744                     end
0745                     bidx  = str2double(qi(qcidx+2:end));
0746                     
0747                     lsqidx = find(rsData.data{rsCols.question_id}(rsMask) == lqid);
0748                    
0749                     if length(lsqidx) > 1
0750                         lsubq = cqids.data{cqSub}(lcqidx);
0751                         lsubq = lsubq{1};
0752                         if length(lsqidx) < lsubq
0753                             lsqidx = lsqidx(1);
0754                         else
0755                             lsqidx = lsqidx(lsubq);
0756                         end
0757                     end
0758                     if lsqidx
0759                         qenum = rsData.data{rsCols.response_enum}(currCompIdxs(lsqidx));
0760                         if bidx <= nenum && ~isnan(qenum)
0761                             if isempty(cqids.data{cqBit}{lcqidx})
0762                                 qdata = data2bitmask(qenum,nenum);
0763                                 cqids.data{cqBit}{lcqidx} = qdata;
0764                             else
0765                                 qdata = cqids.data{cqBit}{lcqidx};
0766                             end
0767                             qdata = qdata(bidx);
0768                         elseif isnan(qenum)
0769                           if (~isfield(rsCols,'decline') || strcmp(rsData.data{rsCols.decline}(currCompIdxs(lsqidx)),'T')) && ...
0770                              ~convertCheckboxNaNToFalse
0771                            if usingR
0772                               qdata = 'NA';
0773                             else
0774                               qdata = '.';
0775                             end
0776                           else
0777                             qdata = 0;
0778                           end
0779                         else
0780                             qdata = '.';
0781                         end
0782                     else
0783                         qdata = '.';
0784                     end % if lsqid
0785                 else
0786                     lqidx = find(strcmp(qi,cqids.data{cqCqi}));
0787                     lqid  = cqids.data{cqQid}(lqidx);
0788                     sqid = cqids.data{cqCols.subquestion}(lqidx);
0789                     if iscell(lqid)
0790                         lqid = lqid{1};
0791                     end
0792                     if iscell(sqid)
0793                       sqid = sqid{1};
0794                     end
0795                     
0796                     qmask = rsData.data{rsCols.question_id}(rsMask) == lqid;
0797                                         
0798                     % Make sure this question existed for this particular
0799                     % stimulus_id
0800                     if ~any(qmask)
0801                         ocol = outCols.(qStructs{iq});
0802                         qtype = outData.datatype{ocol};
0803                         if qtype == 's'
0804                             qdata = '';
0805                         else
0806                             qdata = NaN;
0807                         end
0808                     else
0809                         if sum(qmask) > 1
0810                             lsubq = cqids.data{cqSub}(lqidx);
0811                             lsubq = lsubq{1};
0812                             if sum(qmask) < lsubq
0813                                 qidx = find(qmask,1,'first');
0814                             else
0815                                 qidx = find(qmask,1,'last');
0816                             end
0817                         else
0818                           qidx = find(qmask);
0819                         end % if isempty(qidx
0820                     
0821                         % Determine whether the response should be drawn from
0822                         % response_enum or response_text
0823                         srcType = cqids.data{cqQin}{lqidx}.type;
0824                         if strcmp(srcType,'enum')
0825                             srcCol = rsCols.response_enum;
0826                         else
0827                             srcCol = rsCols.response_text;
0828                         end
0829                                             
0830                         qenum = rsData.data{srcCol}(currCompIdxs(qidx));
0831                         if isnumeric(qenum)
0832                             switch srcType
0833                               case 'enum'
0834                                 qdata = enum2data(qenum);
0835                               otherwise
0836                                 qdata = qenum;
0837                             end
0838                         else
0839                             switch srcType
0840                                 case {'varchar','text'}
0841                                     qdata = qenum{1};
0842                                 otherwise
0843                                     qdata = str2num(qenum{1});
0844                             end
0845                         end
0846                     end
0847                 end % if length(qcidx)
0848                 ocol = outCols.(qStructs{iq});
0849                 qtype = outData.datatype{ocol};
0850                 qdata = sanitize_cell_value(qdata,qtype);
0851                 if qtype == 's'
0852                     outData.data{ocol}{outRow,1} = qdata;
0853                 else
0854                   if isstr(qdata) && strcmp(qdata,'NA')
0855                     outData.data{ocol}(outRow,1) = NaN;
0856                   else
0857                     outData.data{ocol}(outRow,1) = qdata;
0858                   end
0859                 end
0860             end % for iq=1:length(qnums)
0861             cqids.data{cqBit} = cell(length(cqids.data{cqScq}),1);
0862           end % for in=1:nUt
0863         end % for istim = 1:nstim
0864     end % for isub = 1:nsub
0865 else
0866     outData.data = sub_st.data;
0867     outRow = length(outData.data{1});
0868 end
0869 
0870 % % % % CONVERT outData.vars to meaningful codes
0871 %
0872 if isfield(params.export,'var_name_map')
0873     vnmap = params.export.var_name_map;
0874     if isstruct(vnmap)
0875         % save original var names in outData.origvars
0876         outData.origvars = outData.vars;
0877         % convert var names
0878         for iv = 1:length(outData.vars)
0879             if isfield(vnmap,outData.vars{iv})
0880                 outData.vars{iv} = vnmap.(outData.vars{iv});
0881             end
0882         end
0883     else
0884         warning(['vnmap is not a valid struct, therefore variables '...
0885             'will not be re-named']);
0886     end
0887 end
0888 
0889 %% Write data to file
0890 % init file
0891 fid = ensemble_init_fid(params);
0892 fprintf(fid,'%s\n',cell2str(outData.vars,','));
0893 
0894 % if we are generating an R file, insert a row of datatype information
0895 if usingR
0896   datatype = strrep(outData.datatype,'n','numeric');
0897   datatype = strrep(datatype,'l','logical');
0898   datatype = strrep(datatype,'s','character');
0899   
0900   fprintf(fid,'%s\n',cell2str(datatype,','));
0901 end
0902 
0903 if fid ~= 1
0904     % non-numeric cell delimiter
0905     if isfield(params.export,'non_num_delim')
0906         nndlm = params.export.non_num_delim;
0907     else
0908         nndlm = '';
0909     end
0910     % delimiter
0911     if isfield(params.export,'delimiter')
0912         dlm = params.export.delimiter;
0913     else
0914         % \t is the default delimiter
0915         dlm = '\t';
0916         params.export.delimiter = '\t';
0917     end
0918     for irow = 1:outRow
0919         row = '';
0920         for ivar = 1:length(outData.vars)
0921             item = outData.data{ivar}(irow);
0922             datatype = outData.datatype{ivar};
0923             if iscell(item)
0924                 if isempty(item)
0925                   if usingR
0926                     item = 'NA';
0927                   else
0928                     item = '.';
0929                   end
0930                 else
0931                     item = item{1};
0932                     if iscell(item)
0933                         item = cell2str(item);
0934                         % does this break anything??? FIXME
0935                         if isempty(item)
0936                           if usingR
0937                             item = 'NA';
0938                           else
0939                             item = '.';
0940                           end
0941                         end
0942                     end
0943                 end
0944             end
0945             if isnan(item)
0946               if usingR
0947                 item = 'NA';
0948               else
0949                 item = '.';
0950               end               
0951             elseif isnumeric(item) && ~strcmp(datatype,'l')
0952                 item = num2str(item);
0953             elseif islogical(item) || strcmp(datatype,'l')
0954                 if item
0955                   if usingR
0956                     item = 'TRUE';
0957                   else
0958                     item = '1';
0959                   end
0960                 else
0961                   if usingR
0962                     item = 'FALSE';
0963                   else
0964                     item = '0';
0965                   end
0966                 end
0967             end
0968             try item = regexprep(item,'\n','; ');
0969             catch
0970                 error('error trying to remove white space from a cell')
0971             end
0972             item = regexprep(item,'[^a-zA-Z0-9.]','_');
0973             if ~isempty(regexp(item,'[^0-9.]','once')) && ~isempty(nndlm)
0974                 item = [nndlm item nndlm];
0975             end
0976             if isempty(row)
0977                 row = item;
0978                         else
0979                             format_str = sprintf('%%s%s%%s',dlm);
0980                 row = sprintf(format_str,row,item);
0981             end
0982         end
0983         fprintf(fid,'%s\n',row);
0984     end
0985 
0986     fclose(fid);
0987 
0988     % % % % GENERATE SAS IMPORT CODE
0989     %
0990     if isfield(params,'sas') && isfield(params.sas,'fname')
0991         % expects params.sas.fname & params.fname
0992         fname = params.fname;
0993         sfname = params.sas.fname;
0994         sid = fopen(sfname,'wt');
0995         if sid == -1, error('Could not open SAS file: %s\n', sfname), end
0996 
0997         % Generate some header information
0998         fprintf(sid,'/* File generated by ensemble_export_respnstim.m, %s  */\n',datestr(now));
0999         if isfield(params.sas,'libpath')
1000             fprintf(sid,'LIBNAME xportlib ''%s'';\n',params.sas.libpath);
1001         end
1002         if isfield(params.sas,'options')
1003             fprintf(sid,'OPTIONS %s;\n',params.sas.options);
1004         end
1005         fprintf(sid,'\nDATA exported;\n')
1006         fprintf(sid,'  INFILE ''%s'' DLM=''%s'' FIRSTOBS=2;\n',fname,params.export.delimiter);
1007         %% FIXME: need some kind of logic to identify non-numeric columns and
1008         %% include a $ in the SAS INPUT command for that variable
1009         fprintf(sid,'  INPUT %s;\n\n',generate_sas_input_str(outData));
1010         fprintf(sid,'PROC CONTENTS;\nPROC MEANS mean stddev min max n;\nrun;\n');
1011         if isfield(params.sas,'libpath') && isfield(params.sas,'libsave') ...
1012                 && isfield(params.sas,'libname')
1013             fprintf(sid,'DATA xportlib.%s;\nSET exported;\n',params.sas.libname);
1014         end
1015         fclose(sid);
1016     end
1017 end % if fid ~=1
1018 
1019 % % % % functions
1020 % % sanitize_cell_value
1021 
1022 function cval = sanitize_cell_value(cval,ctype)
1023 
1024 if isempty(cval)
1025     if ctype ~= 's'
1026         cval = NaN;
1027     else
1028         cval = '.';
1029     end
1030 end
1031 if iscell(cval) && length(cval) == 1
1032     cval = cval{1};
1033 end
1034 if ctype ~= 's' && strcmp(cval,'.')
1035     cval = NaN;
1036 end
1037 
1038 % % generate_sas_input_str
1039 
1040 function input_str = generate_sas_input_str(struct)
1041 
1042 input_str = '';
1043 tempstr = '';
1044 
1045 for iv=1:length(struct.vars)
1046     tempstr = sprintf('%s %s',tempstr,struct.vars{iv});
1047     if struct.datatype{iv} ~= 'n' && struct.datatype{iv} ~= 'l'
1048         tempstr = sprintf('%s $',tempstr);
1049     end
1050     if length(tempstr) > 100
1051         input_str = sprintf('%s%s\n',input_str,tempstr);
1052         tempstr='';
1053     end
1054 end
1055 
1056 if ~isempty(tempstr)
1057     input_str = sprintf('%s%s\n',input_str,tempstr);
1058 end
1059 
1060 input_str = sprintf('%s',input_str);