Detrending FreeSurfer Data - Revision history

Chris: /* Scripting */

2022-09-27T20:20:12Z

Scripting

← Older revision		Revision as of 16:20, 27 September 2022
Line 114:		Line 114:
	===Scripting===		===Scripting===
	These steps have been combined with the detrend.sh script above to remove linear trends, motion parameters and wm+csf signal (as a single component). You will find detrend_wmcsf.sh in the ubfs/Scripts/Shell folder. However the script is reproduced below:		These steps have been combined with the detrend.sh script above to remove linear trends, motion parameters and wm+csf signal (as a single component). You will find detrend_wmcsf.sh in the ubfs/Scripts/Shell folder. However the script is reproduced below:

			'''Note: I think there's a bug that prevents the script from working correctly when passed more than 1 subject at a time.'''

	====detrend_wmcsf.sh====		====detrend_wmcsf.sh====
	#!/bin/bash		#!/bin/bash

Chris: /* detrend_wmcsf.sh */

2022-09-27T20:19:14Z

detrend_wmcsf.sh

← Older revision		Revision as of 16:19, 27 September 2022
Line 116:		Line 116:
	====detrend_wmcsf.sh====		====detrend_wmcsf.sh====
	#!/bin/bash		#!/bin/bash
	USAGE="Usage: detrend_wmcsf.sh filepattern surf ~~sub1 ... subN~~"		USAGE="Usage: detrend_wmcsf.sh filepattern surf SUBJECT"

	if [ "$#" == "0" ]; then		if [ "$#" == "0" ]; then

Chris: /* Running the Script */

2022-09-27T19:59:47Z

Running the Script

← Older revision		Revision as of 15:59, 27 September 2022
Line 70:		Line 70:

	== Running the Script ==		== Running the Script ==
	Before running this ~~script~~, you will need to create a text file called 'runs' in the bold/ directory for each subject's dataset, e.g.,		===Before you run these scripts===
			Before running either of the scripts on this page, you will need to create a text file called 'runs' in the bold/ directory for each subject's dataset, e.g.,
	*FS_T1_501/		*FS_T1_501/
	**bold/		**bold/
Line 86:		Line 87:
	#if there are fewer than 10 runs of BOLD data, then the run directories will probably have 2 leading zeros		#if there are fewer than 10 runs of BOLD data, then the run directories will probably have 2 leading zeros
	ls -1 \| grep "^00*" > runs		ls -1 \| grep "^00*" > runs
			===Example script===
	Assuming all your subject folders have the same run folders to detrend, you would detrend multiple subjects using detrend.sh, specifying a file pattern for the source data (i.e., the name of the preprocessed files generated by FS-FAST, omitting anything after the '?h' hemisphere identifier), followed by a list of subject IDs:		Assuming all your subject folders have the same run folders to detrend, you would detrend multiple subjects using detrend.sh, specifying a file pattern for the source data (i.e., the name of the preprocessed files generated by FS-FAST, omitting anything after the '?h' hemisphere identifier), followed by a list of subject IDs:
	#A SPECIFIC EXAMPLE: (note these parameters may differ '''substantially''' from what you would be typing in)		#A SPECIFIC EXAMPLE: (note these parameters may differ '''substantially''' from what you would be typing in)

Chris: /* detrend_wmcsf.sh */

2020-04-24T00:15:26Z

detrend_wmcsf.sh

← Older revision		Revision as of 20:15, 23 April 2020
Line 151:		Line 151:
	echo "${source_dir} does not exist!"		echo "${source_dir} does not exist!"
	else		else
	#The subject bold directory exists. Proceed.		#The subject bold directory exists. Proceed.
	fcseed-sess -s ${sub} -cfg ${SUBJECTS_DIR}/wmcsf.cfg		fcseed-sess -s ${sub} -cfg ${SUBJECTS_DIR}/wmcsf.cfg
	echo "Completed wm & csf nuisance regressor calculation for ${sub}"		echo "Completed wm & csf nuisance regressor calculation for ${sub}"
Line 162:		Line 162:
	runlist+=($r)		runlist+=($r)
	done < runs		done < runs
	for r in "${runlist[@]}"; do		for r in "${runlist[@]}"; do
	if [ -n "${r}" ]; then		if [ -n "${r}" ]; then
	#the -n test makes sure that the run number is not an empty string		#the -n test makes sure that the run number is not an empty string
	#caused by a trailing newline in the runs file		#caused by a trailing newline in the runs file
	cd ${source_dir}/${r}		cd ${source_dir}/${r}
	pwd		pwd
	#subject_id does exist. Detrend		#subject_id does exist. Detrend
	if [ "${surf}" = "self" ]; then		if [ "${surf}" = "self" ]; then
	SURFTOUSE=${sub}		SURFTOUSE=${sub}
	else		else
	SURFTOUSE=fsaverage		SURFTOUSE=fsaverage
	fi		fi

			#this will generate the QA matrix in lh.detrend if needed:
			if [ ! -f ${source_dir}/${r}/lh.detrend/Xg.dat ]; then
			mri_glmfit --y ${source_dir}/${r}/${filepat}.${surf}.lh.nii.gz \
			--glmdir ${source_dir}/${r}/lh.detrend \
			--qa \
			--surf ${SURFTOUSE} lh
			fi

	#~~this will generate~~ the QA matrix ~~in lh.detrend if needed:~~		#copy and merge the QA, WMCSF and MC matrices
	~~if [ ! -f~~ ${~~source_dir~~}/${r}~~/lh.detrend/Xg.dat ]; then~~		echo "Generating nuisance regressor matrix for ${hemi} ${r}"
	~~mri_glmfit --y~~ ${source_dir}/${r}/~~${filepat}.${surf}.~~lh.~~nii.gz \~~		paste ${source_dir}/${r}/lh.detrend/Xg.dat wmcsf mcprextreg > nuisance
	~~--glmdir ${source_dir}~~/~~${r}/lh~~.~~detrend \~~
	~~--qa \~~
	~~--surf ${SURFTOUSE} lh~~
	fi

	~~#copy and merge the QA, WMCSF and MC matrices~~		for hemi in "${hemis[@]}"; do
	~~echo "Generating nuisance regressor matrix for ${hemi} ${r}"~~		#regress out nuisance
	~~paste ${source_dir}/${r}/lh.detrend/Xg.dat wmcsf mcprextreg > nuisance~~		mri_glmfit --y ${source_dir}/${r}/${filepat}.${surf}.${hemi}.nii.gz \


	for hemi in "${hemis[@]}"; do
	#regress out nuisance
	mri_glmfit --y ${source_dir}/${r}/${filepat}.${surf}.${hemi}.nii.gz \
	--glmdir ${source_dir}/${r}/${hemi}.detrend \		--glmdir ${source_dir}/${r}/${hemi}.detrend \
	--X nuisance \		--X nuisance \
	--no-contrasts-ok --save-yhat --eres-save \		--no-contrasts-ok --save-yhat --eres-save \
	--surf ${SURFTOUSE} ${hemi}		--surf ${SURFTOUSE} ${hemi}
	#move and rename detrended files		#move and rename detrended files
	mv ${source_dir}/${r}/${hemi}.detrend/eres.mgh ${source_dir}/${r}/${filepat}.${surf}.${hemi}.mgh		mv ${source_dir}/${r}/${hemi}.detrend/eres.mgh ${source_dir}/${r}/${filepat}.${surf}.${hemi}.mgh
	done		done

Line 204:		Line 204:
	--X nuisance --no-contrasts-ok --save-yhat --eres-save		--X nuisance --no-contrasts-ok --save-yhat --eres-save
	mv ${source_dir}/${r}/mni305.detrend/eres.mgh ${source_dir}/${r}/${filepat}.${surf}.mni305.2mm.mgh		mv ${source_dir}/${r}/mni305.detrend/eres.mgh ${source_dir}/${r}/${filepat}.${surf}.mni305.2mm.mgh
	fi		fi
	done		done
	fi		fi

Chris: /* detrend_wmcsf.sh */

2020-04-24T00:14:00Z

detrend_wmcsf.sh

← Older revision		Revision as of 20:14, 23 April 2020
Line 118:		Line 118:

	if [ "$#" == "0" ]; then		if [ "$#" == "0" ]; then
	echo "$USAGE"		echo "$USAGE"
	exit 1		exit 1
	fi		fi
	#first parameter is the filepattern for the .nii.gz time series to be detrended, up to the surface indicator		#first parameter is the filepattern for the .nii.gz time series to be detrended, up to the surface indicator
Line 128:		Line 128:
	surf="$1"		surf="$1"
	shift		shift

	#after the shift command, all the arguments are shifted down one place and the first argument (the filepattern)		#after the shift command, all the arguments are shifted down one place and the first argument (the filepattern)
	#falls off the list. The remaining arguments should be subject_ids		#falls off the list. The remaining arguments should be subject_ids
Line 153:		Line 153:
	#The subject bold directory exists. Proceed.		#The subject bold directory exists. Proceed.
	fcseed-sess -s ${sub} -cfg ${SUBJECTS_DIR}/wmcsf.cfg		fcseed-sess -s ${sub} -cfg ${SUBJECTS_DIR}/wmcsf.cfg
	echo "Completed wm & csf nuisance regressor calculation for ${sub}"		echo "Completed wm & csf nuisance regressor calculation for ${sub}"
	cd ${source_dir}		cd ${source_dir}

Chris: /* White Matter and CSF Signal */

2020-04-24T00:13:18Z

White Matter and CSF Signal

@@ Line 94: / Line 94: @@
 ==White Matter and CSF Signal==
-Work in progress: I'm figuring out how to also get the mean white matter (wm) and ventricle (csf) signal to also include among nuisance regressors in the detrend steps. So far we have:
+It is among best practices in functional connectivity studies to regress out WM and CSF signals from your time series before computing time series correlations, at least for resting state studies (Muschelli et al (2014)). We can add WM and CSF signal to the set of nuisance regressors when detrending our data (note that the procedure below removes the mean WM + CSF as a single component; it can be modified to remove WM and CSF independently):
 ===Create a configuration file===
 An answer to a question I posted to the FreeSurfer mailing list pointed me to [http://surfer.nmr.mgh.harvard.edu/fswiki/FsFastFunctionalConnectivityWalkthrough a walkthrough] for functional connectivity analyses in FreeSurfer is the lead that I'm following. We will be using fcseed-sess to pull out our mean signal from seed regions. In our case, our seed regions will be from the white matter and csf masks. It's a 2-step process. The first step is to create a configuration file:
@@ Line 107: / Line 108: @@
 This will use the config file just created to extract a seed regressor for ${SUBID} (or for all subjects listed in your ''subjects'' file if you use the -sf switch to run in batch mode). You'll find a file called '''wmcsf''' in each of your BOLD/run directories. This file is a plaintext file with one column of values that you can use as a nuisance regressor. The detrend script could be modified to first run fcseed-sess to generate this value to include in the nuisance regressor matrix.
 == Using variables ==

Chris: /* Computing WM and CSF Regressor Values */

2020-03-27T20:46:06Z

Computing WM and CSF Regressor Values

← Older revision		Revision as of 16:46, 27 March 2020
Line 104:		Line 104:
	fcseed-sess -s ${SUBID} -cfg wmcsf.cfg		fcseed-sess -s ${SUBID} -cfg wmcsf.cfg
	or		or
	fcseed-sess -sf subjects -cfg wmcsf.cfg		fcseed-sess -sf ''subjects'' -cfg wmcsf.cfg

	This will use the config file just created to extract a seed regressor for ${SUBID} (or for all subjects if you use the -sf switch). You'll find a file called '''wmcsf''' in each of your BOLD/run directories. This file is a plaintext file with one column of values that you can use as a nuisance regressor. The detrend script could be modified to first run fcseed-sess to generate this value to include in the nuisance regressor matrix.		This will use the config file just created to extract a seed regressor for ${SUBID} (or for all subjects listed in your ''subjects'' file if you use the -sf switch to run in batch mode). You'll find a file called '''wmcsf''' in each of your BOLD/run directories. This file is a plaintext file with one column of values that you can use as a nuisance regressor. The detrend script could be modified to first run fcseed-sess to generate this value to include in the nuisance regressor matrix.

	== Using variables ==		== Using variables ==

Chris at 20:45, 27 March 2020

2020-03-27T20:45:19Z

← Older revision		Revision as of 16:45, 27 March 2020
Line 95:		Line 95:
	==White Matter and CSF Signal==		==White Matter and CSF Signal==
	Work in progress: I'm figuring out how to also get the mean white matter (wm) and ventricle (csf) signal to also include among nuisance regressors in the detrend steps. So far we have:		Work in progress: I'm figuring out how to also get the mean white matter (wm) and ventricle (csf) signal to also include among nuisance regressors in the detrend steps. So far we have:
	===Create a configuration file==		===Create a configuration file===
	An answer to a question I posted to the FreeSurfer mailing list pointed me to [http://surfer.nmr.mgh.harvard.edu/fswiki/FsFastFunctionalConnectivityWalkthrough a walkthrough] for functional connectivity analyses in FreeSurfer is the lead that I'm following. We will be using fcseed-sess to pull out our mean signal from seed regions. In our case, our seed regions will be from the white matter and csf masks. It's a 2-step process. The first step is to create a configuration file:		An answer to a question I posted to the FreeSurfer mailing list pointed me to [http://surfer.nmr.mgh.harvard.edu/fswiki/FsFastFunctionalConnectivityWalkthrough a walkthrough] for functional connectivity analyses in FreeSurfer is the lead that I'm following. We will be using fcseed-sess to pull out our mean signal from seed regions. In our case, our seed regions will be from the white matter and csf masks. It's a 2-step process. The first step is to create a configuration file:
	fcseed-config -cfg wmcsf.cfg -wm -vcsf -fsd bold -mean		fcseed-config -cfg wmcsf.cfg -wm -vcsf -fsd bold -mean
	This assumes that the BOLD data are in ${SUBJECTS_DIR}/${SUBID}/'''bold''' (-fsd), which is likely going to be the case if you've been following the FreeSurfer pipeline on this wiki. I ran the above step in ${SUBJECTS_DIR}.		This assumes that the BOLD data are in ${SUBJECTS_DIR}/${SUBID}/'''bold''' (-fsd), which is likely going to be the case if you've been following the FreeSurfer pipeline on this wiki. I ran the above step in ${SUBJECTS_DIR}.

			===Computing WM and CSF Regressor Values===
	Now you will have a configuration file, '''wmcsf.cfg''' that you will pass to the next step that performs the math on the fmri data:		Now you will have a configuration file, '''wmcsf.cfg''' that you will pass to the next step that performs the math on the fmri data:
	fcseed-sess -s ${SUBID} -cfg wmcsf.cfg		fcseed-sess -s ${SUBID} -cfg wmcsf.cfg
	This will use the config file just created to extract a seed regressor for ${SUBID}. You'll find a file called '''wmcsf''' in each of your BOLD/run directories. This file is a plaintext file with one column of values that you can use as a nuisance regressor. The detrend script could be modified to first run fcseed-sess to generate this value to include in the nuisance regressor matrix.		or
			fcseed-sess -sf subjects -cfg wmcsf.cfg

			This will use the config file just created to extract a seed regressor for ${SUBID} (or for all subjects if you use the -sf switch). You'll find a file called '''wmcsf''' in each of your BOLD/run directories. This file is a plaintext file with one column of values that you can use as a nuisance regressor. The detrend script could be modified to first run fcseed-sess to generate this value to include in the nuisance regressor matrix.

	== Using variables ==		== Using variables ==

Chris at 20:43, 27 March 2020

2020-03-27T20:43:00Z

← Older revision		Revision as of 16:43, 27 March 2020
Line 4:		Line 4:

	<span style="color:red;">'''Update:''' The script has been modified to also regress out motion parameters from the mcprextreg files. This modification is not (yet) reflected in the code below.</span>		<span style="color:red;">'''Update:''' The script has been modified to also regress out motion parameters from the mcprextreg files. This modification is not (yet) reflected in the code below.</span>
	==== detrend.sh ====		==The Script ==
			The most recent version of the BASH script (detrend.sh) is below:
			=== detrend.sh ===
	#!/bin/bash		#!/bin/bash
	USAGE="Usage: detrend.sh filepattern surf sub1 ... subN"		USAGE="Usage: detrend.sh filepattern surf sub1 ... subN"
Line 90:		Line 92:
	#For a more generalizable example of how you should call this function, see the section below using variables		#For a more generalizable example of how you should call this function, see the section below using variables
	The gist is that it calls the mri_glmfit function and saves the residuals after the linear trend has been removed from the data. Multiple files are generated in ?h.detrend/ directories in each run directory. The detrended data is subsequently copied back to the run directory as a new file called ${filepat}.?h.mgh, where ${filepat} is whatever file pattern you provided to the script (note that the source data are .nii.gz files, whereas the detrended data are .mgh files).		The gist is that it calls the mri_glmfit function and saves the residuals after the linear trend has been removed from the data. Multiple files are generated in ?h.detrend/ directories in each run directory. The detrended data is subsequently copied back to the run directory as a new file called ${filepat}.?h.mgh, where ${filepat} is whatever file pattern you provided to the script (note that the source data are .nii.gz files, whereas the detrended data are .mgh files).

			==White Matter and CSF Signal==
			Work in progress: I'm figuring out how to also get the mean white matter (wm) and ventricle (csf) signal to also include among nuisance regressors in the detrend steps. So far we have:
			===Create a configuration file==
			An answer to a question I posted to the FreeSurfer mailing list pointed me to [http://surfer.nmr.mgh.harvard.edu/fswiki/FsFastFunctionalConnectivityWalkthrough a walkthrough] for functional connectivity analyses in FreeSurfer is the lead that I'm following. We will be using fcseed-sess to pull out our mean signal from seed regions. In our case, our seed regions will be from the white matter and csf masks. It's a 2-step process. The first step is to create a configuration file:
			fcseed-config -cfg wmcsf.cfg -wm -vcsf -fsd bold -mean
			This assumes that the BOLD data are in ${SUBJECTS_DIR}/${SUBID}/'''bold''' (-fsd), which is likely going to be the case if you've been following the FreeSurfer pipeline on this wiki. I ran the above step in ${SUBJECTS_DIR}.

			Now you will have a configuration file, '''wmcsf.cfg''' that you will pass to the next step that performs the math on the fmri data:
			fcseed-sess -s ${SUBID} -cfg wmcsf.cfg
			This will use the config file just created to extract a seed regressor for ${SUBID}. You'll find a file called '''wmcsf''' in each of your BOLD/run directories. This file is a plaintext file with one column of values that you can use as a nuisance regressor. The detrend script could be modified to first run fcseed-sess to generate this value to include in the nuisance regressor matrix.

	== Using variables ==		== Using variables ==

Chris at 15:47, 16 July 2019

2019-07-16T15:47:23Z

← Older revision		Revision as of 11:47, 16 July 2019
Line 2:		Line 2:

	A script has been written called detrend.sh that removes the linear trend in your BOLD data. The latest version of this script can be found in /usr/local/sbin (which should be in your $PATH):		A script has been written called detrend.sh that removes the linear trend in your BOLD data. The latest version of this script can be found in /usr/local/sbin (which should be in your $PATH):

			<span style="color:red;">'''Update:''' The script has been modified to also regress out motion parameters from the mcprextreg files. This modification is not (yet) reflected in the code below.</span>
	==== detrend.sh ====		==== detrend.sh ====
	#!/bin/bash		#!/bin/bash