MCaP EVALUATION PLAN
The MCaP Evaluation is being conducted by Wexford Inc., under the direction and coordination of Sheila Cassidy and Rachel Saldivar.
For each MCaP Stage (1-4), common data are collected across all goals and specific to each goal, using:
- Stages 1 and 2 (S1&2) Formative data to monitor and understand participation, quality, usability and feasibility of services and resources for systemic development and statewide use
- S1-3 Implementation and outcome analyses to determine implementation fidelity and factors affecting implementation
- S3&4 Sustainability and impact analyses to identify changes within and across local, county, IHE and state networks/systems
For each MCaP Stage (1-4), common data are collected across all goals and specific to each goal, using:
- Stages 1 and 2 (S1&2) Formative data to monitor and understand participation, quality, usability and feasibility of services and resources for systemic development and statewide use
- S1-3 Implementation and outcome analyses to determine implementation fidelity and factors affecting implementation
- S3&4 Sustainability and impact analyses to identify changes within and across local, county, IHE and state networks/systems
Liberatory Design processes, Concerns Based Adoption Model (CBAM) tools to track adoption of innovations, NIRN Capacity Assessments (NIRN), and these tools are the basis for our data collection: ELR Self-Reflection Rubric (ELRSRR); Professional Learning System Review (PLSR); QPLS with ELR descriptors; ELR Teacher Toolkit Self-Assessments (ELRTTSA); and, the MEGA Dashboard. Quantitative and qualitative data are triangulated for rigorous analysis and findings useful within and across networks. Data collection includes participation data, milestone monitoring, surveys, interviews, focus groups, observations, and artifacts. Project staff, partners, and evaluators work together to collect, summarize, analyze and synthesize the data for continuous feedback, action steps and impact. Progress on each metric is reported annually, and cumulatively for Year 3.
The following section refers to 5 Goals (1-4 from RFA)
1. School Leader Capacity
2. Instructional Practices
3. PL Models
4. Systemic ELR Policy
5. Statewide Collaborative Network
In person and online participation is disaggregated and reported annually by goal, activity type and the following categories, with disaggregation by classrooms, schools, LEAs, counties and regions. Annual targets: 12,500 including 8,375 teachers; 625 paraeducators; 1,000 LEA and school leaders; 625 school counselors; 1,875 families.
- 90% of respondents indicate resources and PL will help them implement ELR, and identify factors inhibiting or supporting their planned action steps - 85% of Stage 2 and 3 survey respondents indicate the resources or PL helped them take action steps to implement the ELR; and factors affecting those actions - # of respondents annually indicating a mindset change toward implementation of the ELRPs (adaptation of CBAM Survey)
Goals 1-3
- Teachers completing online ELR courses, earning badges and micro-credentials targets: Y1 100, Y2 250, Y3 300
- Increase/strengthen # of bilingual authorization programs and participants in IHEs in project regions, with role groups providing feedback related to Quality Standards
- By Y3, using process and fidelity data, and artifact evidence generated through Liberatory Design to change mindset and practice
- # of teachers and administrators in 60 LEAs who have changed practices to implement the ELR (including evidence of student and family engagement, and student outcomes) based on ELRSRR, ELRTTSA, CBAM tools, and MEGA Dashboard
- At least 60% of all schools in Stage 3 LEAs implement their ELR innovation at CBAM Level 4 or higher
- Stage 3 LEAs increase implementation of each ELR Principle by at least one level on the ELRSRR, and show improvement on the PLSR of at least 2 levels for at least 5 Elements
Goal 4– By Year 3
- Expand Seal of Biliteracy support in participating LEAs PreK-16, and award credits for heritage languages, through interviews and artifacts
- Increase # of participating LEAs creating and implementing their LEA Roadmap
- Increase # and quality of biliteracy programs, based on documentation, artifacts observations
Goal 5– By Year 3
- Increase levels of collaboration (based on artifacts and the Tamarack Institute Collaboration Spectrum) with partner network, advisory council, IHE networks, other CA networks, CA-SS (e.g., EL Directors)
- Each PIT increases at least one level on the NIRN tool to assess their capacities to facilitate their work
- Inventory of grant-produced resources from Language Centers and partner agencies
- # LEAs committed to Multilingual CA as evidenced by Pledge artifacts
- 2 members of District Roadmap Team (20 total) are trainers for their LEAs/regions
The following section refers to 5 Goals (1-4 from RFA)
1. School Leader Capacity
2. Instructional Practices
3. PL Models
4. Systemic ELR Policy
5. Statewide Collaborative Network
In person and online participation is disaggregated and reported annually by goal, activity type and the following categories, with disaggregation by classrooms, schools, LEAs, counties and regions. Annual targets: 12,500 including 8,375 teachers; 625 paraeducators; 1,000 LEA and school leaders; 625 school counselors; 1,875 families.
- 90% of respondents indicate resources and PL will help them implement ELR, and identify factors inhibiting or supporting their planned action steps - 85% of Stage 2 and 3 survey respondents indicate the resources or PL helped them take action steps to implement the ELR; and factors affecting those actions - # of respondents annually indicating a mindset change toward implementation of the ELRPs (adaptation of CBAM Survey)
Goals 1-3
- Teachers completing online ELR courses, earning badges and micro-credentials targets: Y1 100, Y2 250, Y3 300
- Increase/strengthen # of bilingual authorization programs and participants in IHEs in project regions, with role groups providing feedback related to Quality Standards
- By Y3, using process and fidelity data, and artifact evidence generated through Liberatory Design to change mindset and practice
- # of teachers and administrators in 60 LEAs who have changed practices to implement the ELR (including evidence of student and family engagement, and student outcomes) based on ELRSRR, ELRTTSA, CBAM tools, and MEGA Dashboard
- At least 60% of all schools in Stage 3 LEAs implement their ELR innovation at CBAM Level 4 or higher
- Stage 3 LEAs increase implementation of each ELR Principle by at least one level on the ELRSRR, and show improvement on the PLSR of at least 2 levels for at least 5 Elements
Goal 4– By Year 3
- Expand Seal of Biliteracy support in participating LEAs PreK-16, and award credits for heritage languages, through interviews and artifacts
- Increase # of participating LEAs creating and implementing their LEA Roadmap
- Increase # and quality of biliteracy programs, based on documentation, artifacts observations
Goal 5– By Year 3
- Increase levels of collaboration (based on artifacts and the Tamarack Institute Collaboration Spectrum) with partner network, advisory council, IHE networks, other CA networks, CA-SS (e.g., EL Directors)
- Each PIT increases at least one level on the NIRN tool to assess their capacities to facilitate their work
- Inventory of grant-produced resources from Language Centers and partner agencies
- # LEAs committed to Multilingual CA as evidenced by Pledge artifacts
- 2 members of District Roadmap Team (20 total) are trainers for their LEAs/regions