Case Study

CAEP Component 4: A Pilot Case Study

 

Component 4 highlights how the program prepared the completers to be successful in their employment. Specifically, it focuses on the impact that the completers have on P-12 student learning and development. This component also evaluates the satisfaction of employers as well as completer satisfaction with the program. To demonstrate our program’s effectiveness in these areas, we have designed a case study that uses multiple measures to assess completer’s success.

 

Design steps 

  1. Purpose of the study
  2. Preparation of data collecting instruments
  3. Identification of participants
  4. Collection of data
  5. Triangulation of data
  6. Analysis of the results

 

Purpose 

Case studies are tools that institutions use to evaluate the effectiveness of their program. According to Balbach (1999), case studies are “particularly useful for evaluating unique programs, programs with unique outcomes, and programs carried out in turbulent or unpredictable environments” (4). PPM UPRAg is a unique program because it is the only one in Puerto Rico that specifically trains completers in educational technology integration. In Addition, the program has unique outcomes because it serves an almost exclusive Latinx population. These factors, coupled with the unique environment that Puerto Rico faced over the past few years (hurricanes, earthquakes, and the pandemic), make a case study the perfect tool for our EPP to use in this endeavor.

Like the one presented here, case studies lend themselves well to program evaluation because they provide a comprehensive and holistic snapshot of how the program influences the completers, the stakeholders, and the students. Ultimately, “case studies enable a researcher to gain a more detailed, un-biased understanding of a complex situation, through the use of a range of research tools” (Salmon 2017). Our goal is to use the results of this case study to improve and assure the quality of the program.

 

The pilot case study will address four principal research questions:

  1. What is the impact of PPM UPRAg preparation on completers’ P-12 students learning and development?
  2. How satisfied are P-12 students with the completer’s teaching?
  3. How satisfied are PPM UPRAg completers with the relevance and effectiveness of their preparation?
  4. How satisfied are the employers with the relevance and effectiveness of PPM UPRAg completers preparation?

 

Data Collection 

Since 2017, the university has faced several unique challenges due to power outages, school closings due to the weather and environment, and the transition to distance learning at the beginning of 2020. Due to these circumstances, data collection has been our biggest obstacle.

The program has several schools in the Western area that frequently employ completers. PPM UPRAg reached out to these and one school entered into a partnership with our institution. The partnership school provided data on their teachers, who are our completers, and their students. PPM UPRAg will continue to work harder on forging partnerships with other schools that recruit our completers as the pandemic restrictions are lifted.

 

Four instruments were used for data collection in relation to the research questions:

  1. To examine the impact that PPM UPRAg completers have on their student’s learning and development, a teaching observation instrument will be used. The instrument used for this measure is called the “Online Education Teacher Evaluation.” During the pilot study phase the data set came from seven completers. Additionally, the partnership school provided these completers’ students’ grades.
  2. To determine how satisfied P-12 students are with the completer’s teaching, a Student Satisfaction Survey was created and will continue to be used in future case studies.
  3. To determine how satisfied completers are with the relevance and effectiveness of their preparation, we employed the use of a Completer Survey. A sample of twenty-six completers who graduated from 2015-2019 completed the survey.
  4. An employer’s satisfaction survey was used to determine how satisfied employers are with the relevance and effectiveness of the completers and their preparation.

 

 

Participants 

The Department of Education of Puerto Rico does not publish standardized test scores or statistics of any kind that relate to student learning. Because of this, we attempted to get participants from different private schools and grade levels. However, the situations that Puerto Rico faced from 2017 to the present made it challenging to collect a large amount of data. Ultimately, the data gathered comes from the partnership school. These completers graduated between 2015-2018.

 

Method 

The data was collected during the 2020-2021 academic year due to the many hardships that completers have faced consecutively since 2017. The partnership school was contacted digitally because Puerto Rico was in lockdown since March 2020 due to the pandemic.

The data-gathering phase of the case study was conducted by the faculty and stakeholders of the PPM UPRAg program. In the initial phases, each CAEP standard was assigned to faculty driven committees that shifted and regrouped as time went on to attend specific tasks related to the specific strands within each standard.

 

Results 

Overall, the results of this pilot case study show favorable results. For three out of four research questions, the data points to the fact that the program was instrumental for completer success. Not only were the completers satisfied with the program, but the administrators and students were as well.

 

What is the impact of PPM UPRAg preparation on completers’ P-12 student learning and development? 

A total of seven completers from 2015-2018 cohorts were evaluated and considered. Most of the completers found that the EPP prepared them for the teaching environment with the necessary knowledge, skills, and dispositions. In general, all completers evaluated were measured as excellent and/or good with their preparation in the different areas. Furthermore, supervisor evaluations of the completers indicated the success achieved with the preparation received. These findings can be triangulated with the data that we collected on student grades. Of the seven completers who submitted their teacher evaluations, six also included their student grades. This data highlights how the PPM UPRAg prepared the completers to be successful in their field. In addition, the data sets used here are valid and reliable because they accurately portray how completers work in a classroom setting. Moreover, the Online Education Teacher Evaluation is an instrument that has used by the partner school for several academic cycles. Also, the student grades are valid and reliable because they represent a snapshot of how the student is performing in the class.

 

How satisfied are P-12 students with the completer’s teaching? 

For this area of inquiry, we developed a survey set (P-3 and 4-12) that seeks to provide data on the impact of completers on P-12 students. This instrument is a digital version and translation of the one used by UPR Mayaguez for the same purpose. We are using it with permission from our sister institution. The instrument is valid and reliable because it has been tested by completer students in the Mayaguez area since 2016. We distributed these instruments to completers and their administrators in two partner schools in early Spring 2021. However, we did not receive any results within the time frame allotted for this endeavor. This was unexpected because in Mayaguez, the program received results. We think that the COVID pandemic had a role to play in the lack of responses. Student and teacher burnout due to the strain of distance learning is a real phenomenon (Pressley 2021). Furthermore, students and their families might not have the time or resources available to easily respond to an electronic questionnaire. Nevertheless, we are confident that we can get data from these questionnaires once the COVID restrictions are lifted, and school is back in session in a face-to-face format. Add validity and reliability

 

How satisfied are the PPM UPRAg completers with the relevance and effectiveness of their preparation?

The UPR Aguadilla EPP initiated a phase-in plan in September 2015 which includes evaluating completers’ satisfaction with their preparation to assume their assigned responsibilities teaching P-12 students. In order to collect data PPM UPRAg gathered alumni surveys for completers within the range of the past five years. This Completer Survey is sent to every completer as soon as they earn their degree. We are using the data of 26 completers that answered the survey and fall within the range studied. Most of the completers found that the TPP prepared them for the teaching environment with the necessary knowledge, skills, and dispositions. In general, all completers are either satisfied or very satisfied with their preparation. In summary, the completers indicated their level of satisfaction with the preparation received in the PPM UPRAg. Add validity and reliability

 

How satisfied are the employers with the relevance and effectiveness of the UPRAg PPM completers preparation?

The data set needed to answer this data set is provided in the CAEP Addendum to 4.1. The EPP developed an employer satisfaction survey that was submitted as evidence that employers perceive that our completers’ preparation was sufficient for the job responsibilities. Sixty-seven surveys were completed and analyzed. In general terms, the data showed that the employers were very satisfied with the knowledge that the completers obtained from the program. The instrument was validated by the Office of Planning and Institutional Research at UPR Aguadilla and through one cycle of administration. Add validity and reliability

 

Suggestions for Case Study Follow Up 

After the careful consideration of the pilot case study, the instruments, participants, and the results, it is our conclusion that the main study is feasible with changes to the protocol. The area where we must focus on is data collection. This case study yielded several instruments that had a poor or nonexistent completion rate. We believe that this is due to factors outside of our control, such as natural phenomena and the pandemic that have plagued us since 2017. Over the last few years, Puerto Rico, the completers and their students have been in survival mode. We hope that the effects of these obstacles will normalize in future semesters. When pandemic restrictions are lifted, we will have more access to school administrators, completers, and community members. All of these instruments presented here were distributed in digital format as electronic forms, but some instruments, like the student perception surveys, could yield more data if the students had access to these surveys in paper form. We feel confident that we are heading in the right direction.

 

 

References

 Balbach, E. D. (1999) California Department of Health Services, (1999). Using case studies to do program evaluation. Retrieved from website: http://www.case.edu/affil/healthpromotion/ProgramEvaluation.pdf

Pressley, T. (2021). Factors Contributing to Teacher Burnout During COVID-19. Educational Researcher. https://doi.org/10.3102/0013189X211004138

Salmon, L. (2018, August 24). The importance of case studies in social research. Changeworks. https://www.changeworks.org.uk/news-and-events/blog/the-importance-of-case-studies-in-social-research#:~:text=The most significant benefit of,of tools on one subject.