Methodology

The iCAN Project is a two-year exploratory project consisting of five phases. Each is designed to allow     us to answer two fundamental research questions:

  1. Which attributes of the Landmark College model are most salient at a large urban institution?
  2. What technological and community supports add value to the model?

The following objectives are associated with the iCAN Project:

  1. Identify essential aspects of the Landmark College Model
  2. Collaboratively design a hybrid model of supports for undergraduates in STEM majors that can be used at UCF and other large institutions
  3. Iteratively implement, test, measure, and revise the model
  4. Conduct a pilot study to inform a future efficacy study
  5. Analyze the data
  6. Create a plan to digitize as many aspects of the model as possible

 

Objective 1: Identify essential aspects of the Landmark College Model

The PIs at UCF and Landmark will conduct semi-structured interviews with a cross-section of 20 undergraduate students with executive function disabilities in STEM majors. They will also interview five academic coaches and five advisors at Landmark College. Participants will be purposefully selected to represent a range of ages, gender, and ethnic backgrounds. The purpose of these interviews will be to establish common themes each group views as essential components of a successful STEM learning and persistence model. Specific questions related to potential types of physical and technology-based supports will be included.

Interviews will be video recorded, transcribed, coded, and analyzed using a basic interpretive qualitative methodology 71. Categories and themes will be determined through a constant comparative method using data from transcribed interviews across the groups of participants 34. The units of data will be sorted and grouped first by regularities and then by irregularities into tentative categories and subcategories. To increase reliability, transcriptions will include pauses, facial expressions, and tone of voice. Following the initial qualitative findings, a secondary analysis will occur. During this member checking process, the PIs will present initial results and ask participants for help and clarification in interpreting the results.

 

Objective 2: Collaboratively design a hybrid model of supports

Once critical aspects of the Landmark Model have been identified, we will work with our advisory board to develop a preliminary graphical representation of the hybrid model at UCF. We will share the model with the 18,236 students in STEM majors at UCF and solicit specific feedback from students in the Calculus with Analytic Geometry, Probability and Statistics for Engineers, and Physics for Engineers & Scientists courses. We will use Qualtrics, an electronic survey platform, to solicit input from the 2,005 students with disabilities in STEM majors at UCF. The survey will prompt students to rank order the features of the model they anticipate would be most beneficial. Based on the survey results and feedback from students in the STEM courses, the PIs will work with members of the advisory board to develop a working prototype of the model.

 

Objective 3: Iteratively implement, test, measure, and revise the model

During the spring of 2015, we will iteratively implement, test, measure, and revise the model with six groups of students. There will be two groups from each of the three target courses. Students will be selected from a convenience sample recruited through the office of disability services at UCF. Students will be selected to represent a range of ages, gender, and ethnic backgrounds.

Each model test will last approximately four weeks. During the first week, students will be introduced to the model. They will participate in the model for the next two weeks. During week four students will evaluate the model, recommend changes, and assist the project team with revisions. The model will be revised during week five and then a new iterative cycle will begin. We plan to include the same students in each group over the course of the spring semester. Compensation for the students is included in the budget. We will recruit a new group of students for iterative testing during the summer session if we do not achieve model continuity by the end of the spring semester. Once the hybrid model has been established, the team will develop a survey to assess the model during the pilot study. Specifically, the survey will ask students to rank the value of each of the model components including technologies and community supports. Our intention is to have the hybrid model and survey finalized for pilot testing during the fall of 2016. We will then develop a model implementation protocol manual for the pilot study.

 

Objective 4: Conduct a pilot study

The pilot study will utilize a quasi-experimental mixed method research design to answer the primary research questions.

Pilot Study Logic Model
Figure 3. Logic Model for the pilot study.

Power Analysis: We conducted a power analysis to ensure this pilot study would have a large enough sample to allow for the detection of differences in the performance of students with disabilities across the two conditions. Using the “ANOVA: Repeated measures, between factors” algorithm of the G*Power 3.1 software 31 for this analysis and assuming a medium effect size of 0.25, and a of 0.05 and a power of 0.8 19, we would need a sample size of 120 total participants. We anticipate a rate of attrition at 20% based on our previous experiences with this population and therefore will include 144 students in the pilot study.

An initial convenience sample will be identified from the disabilities resource center at UCF. The director of the disability resource center at UCF will serve on the advisory board for this project. He has identified approximately 350 undergraduate students in STEM majors. Students will be randomly selected from those who consent to participate in the study. When target numbers are reached, we will administer the Tasks of Executive Control 44 assessment. This standardized assessment will allow us to compare participants’ working memory and inhibitory control capabilities. In addition, we will administer the Success Navigator 67 to assess participants’ social skills. Psychometric properties for these instruments are available in the instruments section of the proposal.

We will assign students to equivalent groups in the treatment (i.e., hybrid support model) or control (no intervention) condition based on the composite scores from these assessments. See Figure 4 for a graphical representation of this process. This approach, using multiple construct-specific standardized assessments, will allow us to create more valid and reliable equivalency groups than by relying on traditional disability classifications. Madaus et al., 52 noted that questions surrounding the reliability of traditional disability eligibility determination at the postsecondary level have been present for more than a decade.

During iCAN we will compile an extensive database of participant information.

Four Primary Data Sources
Table 2. Four primary data sources.

Instruments

Reseach Proceedure
Figure 4. Participant selection and assignment criteria.

Instruments in the study will be used as repeated measures to examine critical aspects of persistence (e.g., self- determination, self-efficacy, etc.) and academic achievement. Students across conditions will participate in equivalent assessment measures.

 

 
Tasks of Executive Control The Task of Executive Control (TEC) is a computer administered standardized test consisting of two executive functioning tasks measuring working memory and inhibitory control. These include the n-back paradigm (i.e., working memory) and the go/no go task (i.e., executive control). The test is designed to take 20 – 30 minutes to complete. Reported reliability (Cronbach’s alpha) ranged from .75 to .98 for factor scores and from .68 to .99 for summary scores 44. This test will be administered once at the beginning of the study to students in both groups.

Success Navigator – The Success Navigator 67 is a standardized online assessment of social factors critical for STEM persistence including: personality, motivation, study skills, intrapersonal and interpersonal, skills. Reported reliability (Cronbach’s alpha) ranged from .78 to .90 across the skill scales. This test will be administered at the beginning, middle, and end of the semester.

Content Specific Pre/Post Tests – Items from the UCF Standardized Calculus, Engineering, and Physics Assessments will be administered to all students in both conditions (treatment and control) using a pre/posttest design at the beginning and end of each semester. The final list of question items will be identified using a Delphi process with content experts from the respective disciplines and advisory board.

Motivated Strategies for Learning Questionnaire A computer-based version of The Motivated Strategies for Learning Questionnaire 82 will be administered to all students at the beginning, middle, and end of the semester. The survey contains 44 items rated using a 7-point Likert-type scale. Reported reliability (Cronbach’s alpha) for the self-efficacy subscale is .93.

Semi-Structured Interviews – Twenty-four students will participate in semi-structured interviews at the conclusion of the pilot study. Participants will include equal numbers of students from the treatment and control condition across courses. Interviews will include those who intend to persist and those who intend to change majors or withdraw from the STEM program. The questions will focus on academic and social aspects of college life in a STEM major. Broad questions will be designed to illuminate the institutional, situational, and individual barriers or scaffolds that influenced the students’ decisions.
Specific questions will ask students to identify how they engaged in the STEM community and why they are choosing to persist or withdraw. Students will also be asked to identify the types of technologies and community supports they find most beneficial when participating in STEM courses. We are particularly interested in understanding how these technologies and community supports enhance metacognition.

Qualtrics Technology, Community & Model Survey – The research team will develop a model-specific survey in order to determine undergraduate students’ perceptions of the most salient factors within the model. The survey will also include questions about the types of software and community supports the students find most beneficial and why.

Procedure

Graduate student coaches will receive formal training during their courses with Drs. Marino and Vasquez. The courses will provide students with multiple opportunities to learn and practice facilitating the coaching techniques with a high degree of fidelity. Doctoral students from STEM disciplines will be employed by the project and also participate in the courses. The graduate students will follow the model implementation protocol developed during objective three. While the content in each group will be course specific, the social skills, as measured using the Success Navigator, will be consistent across all three groups and align with the findings of Gregg et al. 38.

Threats to validity

Threats to validity include contamination, history effects, and the use of an initial convenience sample. It is possible students from the treatment and control condition will be in the same courses. To reduce cross contamination, students will commit to not discussing the intervention with their peers at the outset of each semester and will be asked if they complied at the end of the semester. Randomization within the convenience sample and the use of covariates will reduce error. Concurrent measurement of achievement and persistence measures across the treatment and control conditions will minimize potential confounds and increase the ability to detect an intervention effect.

 

Objective 5: Analyze the data

To examine differences across groups in STEM persistence, we will compare students’ aggregate number of course completion rates across all STEM and non-STEM courses. In addition, we will examine changes in students’ plans of study to a major outside STEM and enrollment in STEM courses each semester. Finally, we will examine trend data from the Success Navigator and Motivated Strategies for Learning Questionnaire. Semi-structured interviews with students who do and do not persist will be transcribed, coded, and analyzed using a basic interpretive qualitative methodology 71. Categories and themes will be determined through a constant comparative method 34 using triangulated data from transcribed interviews, Qualtrics 84 survey responses, and data for students in the treatment condition. The qualitative analysis will follow the same procedure described in objective one.

Content area gain scores will serve as the dependent variable to measure student learning. Differences from pre to post tests across each of the three courses will be converted to z-scores. To examine differences in academic achievement across groups, a repeated-measures between-factors analyses of variance (ANOVA) will be performed. Condition (2 levels, treatment and control) will be the between-subjects factor and time will be the within-subject factor. Our initial analysis will compare differences in the groups’ aggregated performance (i.e., µ percent gain). Subsequent analyses will examine content specific differences.

We will triangulate data from the interviews, group session recordings, and Qualtrics surveys in order to determine which aspects of the model are most salient. Once common themes have been identified, we will share those with students who participated in the project and other students without executive function difficulties. Once consensus regarding the most salient aspects of the model has been reached by the participants, we will prepare a synthesis of our findings and disseminate it to all STEM majors at UCF. We will follow the synthesis with a brief survey to determine whether results from this exploratory project are generalizable. We will also revise the implementation protocol if necessary. All of the materials will be posted to the project website and disseminated through our existing partnerships.

Objective 6: Create a plan to digitize the model

The finalized model will by analyzed by the project team. All components that can be digitized will be identified and we will work with our partners in computer science to create a digital design document. This document will include all of the features, functionality, software descriptions, and user interface elements necessary to build an electronically enhanced version of the model. This document will also serve as a template for a future phase II proposal.

Project Management

          Table 3 presents an overview of major events across the two years of the project. We have created a detailed study implementation checklist based on our extensive previous research projects. The document is available on request.

Project Table
Table 3. Project objectives and dates.

PI Dr. Marino will assume primary responsibility for the project and will oversee all project activities. Co-PI Dr. Banerjee will coordinate and lead efforts at Landmark College. Co-PI Dr. Vasquez will collaborate with the project team and will be the lead on data collection and analysis during the pilot study. Co-PI Dr. Moore will be the content specialist and aid in the selection and oversight of the STEM doctoral students. A visual representation of the project team is presented in Figure 5.

Project Team Members
Figure 5. Project Team Members.