Observation Instrument
The EPP at UPR Aguadilla is initiating a pilot study to demonstrate completer’s teaching effectiveness. So far, a partnership was established with one school where several completers are employed at both the elementary and secondary levels. Although our EPP has an assessment instrument for teacher performance evaluation (Assessment 4.1: (Candidate’s) Student Teacher evaluation, used to evaluate candidates during clinical practice), the data used as evidence for this component was gathered using the partnership school’s Teaching Teacher Evaluation. (Note: Due to the pandemic, this instrument is now being used to evaluate teachers’ distance teaching.)
The instructions for the instrument state that “The administrative personnel will carefully evaluate the performance of the employee in relation to the areas to be examined and their respective criteria. The principal will evaluate each area and assign a score that best represents your performance. The subtotals will be added to obtain the total result of the performance of each evaluated employee.” The observation instrument is used by the school’s principal to evaluate the established criteria.
The observation instrument is composed of 12 areas to be observed and the measurement scale used has four levels. These are:
- Personal Qualities: It refers to the dominant skills exhibited by an individual in the way they behave in certain situations.
- Classroom Management: It refers to the wide variety of skills and techniques the teacher uses to keep students organized, orderly, focused, attentive, on task, and academically productive during a class.
- Professional Skills: Ability to perform work correctly with ease, skills, and specialized knowledge.
- Planning: It refers to the teacher’s written and detailed execution plan of daily activities to achieve objectives for each area of instruction in a correct and timely fashion.
- Instructional Techniques: It refers to the various educational strategies and demonstrates effective rapport towards the students as part of the class.
- Evaluation and Assessment: It refers to the objective way the teacher executes different assessments and demonstrates effective techniques as part of the class.
- Online Grading System Usage: It refers to the accountability under which the work is performed in a correct and timely fashion.
- Environment and Work Habits: It refers to the conditions under which the work is performed and demonstrated practices that seek to maintain a pleasant working environment.
- Course Information: This section refers to the class syllabus and class information including objectives, student learning outcomes, and class requirements.
- Learning Environment: Elements to set the learning environment in an online course, such as opportunities and settings for interaction, welcoming environment, dynamic communication, and collaboration.
- Interaction and Collaboration: Student-instructor, student-student, and student-content interaction. Courses should integrate many different ways to interact and collaborate online.
- Assessment and Evaluation: The assessment category focuses on the ways in which the student is evaluated toward achieving the student learning outcomes and the quality, type, structure, and security of the assessments used.
There is also an additional section where the Technology Tools used in the course are observed.
The partnership school’s instrument areas were correlated to UPRAg’s (Candidate’s) Student Teaching Evaluation (Assessment 4.1). This assessment is composed of seven competences: (I) Knowledge and mastery of the subject matter and the effective teaching planning process; (II) Knowledge of the students, their learning process, and comprehension of student diversity; (III) Management of Educational Setting; (IV) Communication Skills; (V) Learning Evaluation and Assessment; (VI) Professional development, ethics, collaboration, relationship with parents and the community, and (VII) Management of Professional Documents and Reflection on the Process. The components in the UPRAg’s (Candidate’s) Student Teaching Evaluation (Assessment 4.1) are theoretically grounded on UPRAg’s Conceputal Framework of the Teachers’ Preparation Program. This framework highlights the philosophical principals UPRAg’s EPP embraces and emphasizes the candidates’ development of the professional knowledge, skills and disposition that will ensure their effectivity in the classroom. (See Conceptual Framework Diagram add)
Describe Instrument Validity and Reliability
The partnership school is an accredited institution that regularly uses the Teacher Teaching Evaluation Observation instrument to assess teacher performance. The instrument assesses teachers’ effectiveness in applying professional knowledge, skills, and dispositions when teaching as well as their effect on student’s learning and development. It has been in use for the past 10 years’ instrument, with appropriate modification applied as needed. It was originally created by two educational specialists with doctoral degrees in education. It was last revised three years ago. The criteria on the instrument are aligned to the specifications of teacher’s duties as specified by Puerto Rico’s Department of Education. To assure the instruments’ data validity, during this pilot phase, the partnership school’s Teacher Teaching Evaluation Instrument criteria were correlated to the (Candidate’s) Student Teaching Evaluation used in UPRAg’s EPP. The construct validity for the Teacher Teaching Evaluation Instrument has been established since the instrument criteria is focused on assessing teachers’ effective use of professional knowledge, skills, and dispositions. The data obtained from the instrument has been used by the partnership school to implement changes for improvement and for accreditation purposes. Additionally, the data is used to make decisions such as contract renewal, assigning permanent positions and professional development needs for teachers. When the components in the UPRAg’s (Candidate’s) Student Teaching Evaluation were correlated to the criteria in the partnership school Teacher Teaching Evaluation the criteria aligned (see correlation chart add #) thus demonstrating that the same areas were being evaluated in both instruments. The data obtained would be valid for UPRAg’s EPP assessment purposes in demonstrating completer’s effectiveness in applying the professional knowledge, skills, and dispositions, acquired through the PPM UPRAg, which result in effective student learning and development.
The data gathered from the Teacher Teaching Evaluation Instrument has also proven to be reliable since the instrument has been used regularly and the data gathered has allowed the partnership school to consistently implement changes for improvement based on the information.
Data Analysis
The Teacher Teaching Evaluation Instrument’s criteria data was analyzed using three focus areas: professional knowledge, skills, and dispositions. PPM UPRAg teacher candidates’ professional knowledge is developed and demonstrated throughout the design of preparation experiences during the course of their professional development. Throughout this process, teacher candidates’ professional knowledge is corroborated by means of structured and validated assessment instruments. Subsequently, during their Clinical Practice, candidates demonstrate professional knowledge through the implementation of competencies stipulated in the (Candidate) Student Teaching Evaluation assessment. Specifically, competences 1, 2, and 5 demonstrate candidates’ effective professional knowledge development and thus, completers’ teaching effectiveness.
Competence # 1: Knowledge and mastery of the subject-matter and the effective teaching planning process with emphasis on candidates’ professional knowledge regarding lesson planning. That is, knowledge in developing different types of observable and measurable learning objectives (conceptual, procedural, attitudinal) for all learning levels (DOK) and aligned to grade level expectations. Candidates also demonstrate knowledge in creating relevant and pertinent teaching and learning activities that link students’ prior knowledge to learning objectives and develop their critical thinking skills. In addition, candidates demonstrate knowledge in the selection and adaptation of materials and resources, as well as technological support to enhance the teaching and learning processes.
Competence # 2: Knowledge of the students, their learning process, and comprehension of student diversity, emphasizes candidates’ professional knowledge regarding the design of lesson planning to address student diversity and accessibility, among others.
Competence # 5: Learning evaluation and assessment, emphasizes candidates’ professional knowledge regarding the design of formative, summative, and performance assessments to promote students’ knowledge.
Criteria stipulated in the (Candidate) Student Teaching Evaluation assessment instrument (4.1) demonstrate correlation with those stipulated in the partnership school’s Online Teacher Teaching Evaluation assessment instrument. Criteria in both instruments is aligned demonstrating similarities among areas evaluated regarding professional knowledge. Consequently, the data gathered demonstrated the following completers’ success rates in applying professional knowledge throughout their teaching process: competence # 1 and # 2 (area IV), completers demonstrated an average of 96% effectivity. Completers demonstrated an average of 95% effectivity for competence # 5 (area VI). Regarding skill development, a cross-cutting theme considered a core area of the program is technology (Conceptual Framework, 2019). Technology usage and integration are incorporated in every core course of the degree and candidates are required to take a minimum of 12 credits in Educational Technology with the purpose of preparing teachers who can integrate technology in the classroom purposefully and effectively. These skills are exhibited in Areas VII: Online Grading System Usage, IX: Course Information, and XII: Technology Tools, obtaining a 98% average.
Results show that completers model clarity of expression and communication in the L1 & L2 in language arts measuring a 96% as a general average in the following observed areas: I: Personal Qualities, II: Classroom Management, III: Professional Skill, V: Instructional Techniques, VI: Evaluation and Assessment, XI: Course Information. PPM UPRAg completers measured 87% in area III: Planning. Planning instruction, preparation of activities, selection of appropriate level resources is established as an area for improvement. PPM UPRAg Assessment 3 is the instrument used for evaluating Student Teaching in Clinical Practice. The guidelines and rubric are currently under revision by a Committee composed of Clinical Practice Supervisors and Methodology and Seminar professors, Clinical Educators and stakeholders. In Area VI: Evaluation and Assessment completers obtained a 95%. PPM UPRAg establishes clear standards for developing skills to evaluate and assess and the creation of techniques and their application. PPM UPRAg completers should demonstrate the following dispositions when teaching: (1) respects for student diversity; (2) modeling and promoting justice among the students; (3) demonstrating a reflective approach to improve planning and practice; (4) taking responsibility for student learning and recognizing their potentiality to learn; and (5) understanding and application of ethics throughout the clinical practice process. These are considered essential for completers to be effective in their future practice and are surveyed at four transition points during the program. The data gathered using the partnership school’s Teaching Teacher Evaluation Instrument shows the following completers success rates in applying the dispositions in their teaching: For candidate respects student’s diversity: areas V.2, Makes the subject or discipline accessible and meaningful for students, and V.8, Works individually with those students that need additional attention, both demonstrated an average of 97% effectivity. For candidate models and promotes justice among the students: areas V.4, Provides positive and descriptive feedback to guide student’s progress, V.5, Check frequently for understanding and adjust instruction when needed, and XI.1, Students participation requirements/expectations are clearly stated, demonstrated an average of 97%, 97% and 89% respectively. For candidate promotes a reflective approach to improve planning and practice: areas XII.1, Assessments are used throughout the course, demonstrated an average of 89% effectivity. For candidate takes responsibility for student learning and recognizes their potentiality to learn: areas V.2, Makes the subject or discipline accessible and meaningful for students. and V.3, Uses creativity in the classroom and motivates students to learn, demonstrated 97% and 97% respectively. For candidate understands and applies ethics throughout the clinical practice process: areas XI.6, Uses an appropriate tone of voice during lessons and student interactions, and XI.8, Clarifies questions and doubts in a respectful manner, demonstrated 89% and 89% effectivity. All data for completers effective use of dispositions with students when teaching had an average of 89% or higher.
Interpretation of Results
Discussions of results should include any comparisons that are supported by the quantity of data; these could include comparisons of results across licensure areas at the EPP, between completers’ results and external benchmarks (e.g., district, state, national, or other relevant benchmarks), or over time.
The measures provided should be reflectively analyzed and interpreted to reveal specific implications for the program.
Program goals for dispositions were met as evidenced by the data. Completers demonstrated to effectively apply professional knowledge, skills and dispositions to support student learning and development in the classroom. The information highlighted the programs need to focus more on.
The completers work at different grade levels, elementary and secondary, and hold teaching licenses at these levels. The licensure test (PCMAs) scores support completers preparation for effective implementation of professional knowledge, skills and disposition.
Over time, the data collected will be compared to document completers effectiveness in the classroom when applying professional knowledge, skills and dispositions for the benefit of student learning and development.
Additionally, the results of this pilot study have made evident the need for PPM UPRAg to work closely with partnership school to improve the teacher teaching observation instruments so it will provide a more detailed picture of how completers are applying their professional knowledge, skills and dispositions in the classroom to support student achievement. The changes will be aligned the instruments used by PPM UPRAg for candidates’ clinical practice. Therefore, future data gathered will allow the program to continue improving in its preparation of future educators and supporting completers as well as the schools they work in.