Collection

Data Collection using the Ultrasound with the Kinect

This manual will show how to set up and collect data from a subject. The end result will be a folder for the individual subject that holds the ultrasound video, Kinect data and coordinates, stimulus list, as well as individual frames of the tongue images.

Setting Up the Hardware Before the Subject Arrives: NEED PICTURE

• Make sure the ultrasound is connected to the lockbox

• The svideo cable connects to the back of the ultrasound

  It is the cable that connects the ultrasound and the lockbox

  The lockbox has a label for this svideo cable

  Important: if it is not connected, make sure the ultrasound is off before connecting, otherwise it may malfunction

• Microphone amplifier should be connected to computer

  It connects in the spot with the pink ring around it

• Make sure the lockbox is on

  There should be a blue light on the lockbox that shows that it is on

• If it does need to be turned on, press the power button on the lockbox

  When you do this a box will pop up on the monitor which you should exit out of

Setting up the Ultrasound Software with the Kinect

• Turn on the ultrasound

• There is a power button in the top left corner of the ultrasound

INCLUDE PICTURE

• Test the video feed of the ultrasound

• There should be a black screen on the Titan with white grain that shows the ultrasound is on and functioning

• Make sure the Kinect is connected to the computer

  The USB cord connects to the front of the computer

• On the main monitor, locate the ultrasound icon and double click on it

  Two new windows should open, a Kinect window and an ultrasound window

• Move the Kinect window to the right of the ultrasound window

• On the monitor, locate the stimulus display icon

• Click stimuli.py and a new window should open

• Close the explorer window

  The stimuli will be a group of words in any language that the subjects will be prompted with in a random order

• To determine how many times these words are repeated, you will be prompted with a screen asking to specify the number of repetitions

  These repetitions will repeat the same set of words in a different order each time

• Click enter

CREATING A STIM LIST-NEED DIRECTIONS

• A new window will open that says click to begin

• Drag the click to begin window onto the second monitor and resize to fit screen

• Minimize this window so you can still see the other windows

• In the ultrasound window, press preview to ensure the image feed from the ultrasound is correct

Setting up Subject

• Sit subject comfortably in chair

• Mount microphone over subject’s ear

• Apply a small strip of ultrasound gel to the top of the probe and place directly under subject’s chin to get an image of the tongue and palate

• Adjust fit of probe to subject

• Adjust the depth on the ultrasound to make sure the tongue is displaying correctly

NEED TIPS FOR THIS

Note: if the video appears extremely zoomed in the best solution is to turn the ultrasound on and off again

Naming the folder

• In the ultrasound window, there is a space entitled recording ID

• In the blank provided, include the study subject number, year, month, day, hour, and minute of the experiment in that order

  Use leading zeros in the case of single digit months

  Note: try not to use window’s special characters

  This creates a separate file that includes the subject number and demographic data that will contain all of the data collected

• Check the Kinect display

• Make sure there are no obstructions in the way of the Kinect

• Make sure the image feed recognizes the head of the subject

  There should be a grid around the face of the subject as shown below

Collecting Data

• Press start on the ultrasound window to begin recording

• The record button should turn red

  Note: once the recording is done it will turn grey

• Click on click to begin in the stimuli window

• Instruct the subject to click through the stimuli as the subject reads the words out loud

• Once the subject has read through all of the stimuli, the stimulus display window should close

• Hit stop on the ultrasound window to end the recording

  Note: do not close this window it is very important that it stays open for post processing

• The Kinect window should close when you hit stop

Post Processing and Saving Data

• Post processing must be done to ensure that all of the proper data is recorded and gathered in the subject’s folder

• After closing the Kinect, an icon entitled coords should appear on the desktop

• Drag the coords icon into the folder you created previously

• In the top right of the ultrasound window click on the post processing tab

• Click Run Post Processing

  Note: this may take a while, so be patient

• A window will pop up that shows it is complete

• Close out of this window by clicking ok

• Open Stimulus Display icon again

• Drag the newly created stimulusresponse.csv file into the subject’s folder that was created when you filled in the Subject ID

  This file contains the stimuli, subject information, repition number, and timestamp of when the subject clicked to start each stimuli

• Open Kinect Matching icon on desktop by double-clicking

• Select the appropriate files for the post processing

• The Kinect Matching program opens with a window that allows you to select the folders that contain the frames, video, and Kinect video and then combines them into one document labeled output

  Note: Double check with each file you open that they are coming from the correct folder

Can it default to the correct folder?

• Frames: locate in the subject’s folder

• Video: locate the vidTimes section of the subject’s folder

• Kinect: locate the coords section of the subject’s folder

• Output: in the output file box, the output file will be automatically named output and placed in the same folder.

  If you want it to have another name, you can fill that name into this box

• Once you have located the four different parts of the folder, hit process file

• A window should pop up that says file ready

• Close the Kinect Matching window

• Open the folder you created for the subject

• A new file, entitled output should now be in the folder as well as the vidTimes, coords, and frames folders

• Play the vidTimes video in order to make sure that it recorded properly

  Do this by clicking the on the vidTimes file within the subject’s folder

• Open the output file to make sure that the post-processing worked correctly

  This should have timestamps, frame numbers, and coordinates of the locations of the subject’s head throughout the experiment as shown below