The purpose and effect of the proposed rule is to set the formulas used for measuring student learning growth on the statewide, standardized assessments in English Language Arts and Mathematics, and the statewide, standardized end-of-course ...

 

DEPARTMENT OF EDUCATION

State Board of Education

RULE NO.:RULE TITLE:

6A-5.0411Calculations of Student Learning Growth Using Statewide Assessment Data for Use in School Personnel Evaluations

PURPOSE AND EFFECT: The purpose and effect of the proposed rule is to set the formulas used for measuring student learning growth on the statewide, standardized assessments in English Language Arts and Mathematics, and the statewide, standardized end-of-course assessment in Algebra I. Additionally, the proposed rule sets the standards for determining the performance levels of highly effective, effective, needs improvement or developing, and unsatisfactory in school district evaluation systems for the student performance component in teacher evaluations of courses associated with a statewide, standardized assessment in English Language Arts or Mathematics, or the statewide, standardized end-of-course assessment in Algebra I (or any combination of such courses). Also, the proposed rule establishes the procedures associated with implementing the formulas and standards.

SUMMARY: The proposed rule adopts and describes the covariate adjustment value-added model with the statewide, standardized assessments in English Language Arts and Mathematics, as well as the statewide, standardized end-of-course assessment in Algebra I. It describes how scores and confidence intervals determine the performance levels of highly effective, effective, needs improvement or developing, and unsatisfactory for the student performance components of annual evaluations based on these models. The rule establishes the procedures for data collection and reporting necessary for the value-added model calculations and the requirements for the Department to report results of the calculations to school districts for use in evaluations of instructional personnel pursuant to s. 1012.34, F.S. The proposed rule incorporates two documents; one describes the courses that are associated with statewide, standardized assessments in English Language Arts and Mathematics, and the statewide, standardized end-of course assessment in Algebra I. The second is a Value-Added Model (“VAM”) Methodology document that describes the data used, the sources of the data, and the statistical model for the value-added formulas.

SUMMARY OF STATEMENT OF ESTIMATED REGULATORY COSTS AND LEGISLATIVE RATIFICATION: The Agency has determined that this will not have an adverse impact on small business or likely increase directly or indirectly regulatory costs in excess of $200,000 in the aggregate within one year after the implementation of the rule. A SERC has been prepared by the Agency.

In summary, the proposed rule is expected to result in a cost savings to school districts because the Department will not only continue providing districts with data as it has in the past, but the proposed rule also establishes performance-level standards from that data for use by the school districts. The Agency has determined that the proposed rule is not expected to require legislative ratification based on the statement of estimated regulatory costs.

Any person who wishes to provide information regarding a statement of estimated regulatory costs, or provide a proposal for a lower cost regulatory alternative must do so in writing within 21 days of this notice.

RULEMAKING AUTHORITY: 1012.34(7), (8) FS.

LAW IMPLEMENTED: 1012.34 FS.

A HEARING WILL BE HELD AT THE DATE, TIME AND PLACE SHOWN BELOW:

DATE AND TIME: July 23, 2015, 8:30 a.m.

PLACE: Tampa Airport Marriott, 4200 George J. Bean Parkway, Duval Conference Room, Tampa, FL 33607

THE PERSON TO BE CONTACTED REGARDING THE PROPOSED RULE IS: Jason Gaitanis, Policy Research and Accountability Coordinator, Florida Department of Education, Florida Department of Education, 325 West Gaines Street, Suite 544, Tallahassee, Florida, 32399-0400, [phone]

 

THE FULL TEXT OF THE PROPOSED RULE IS:

 

6A-5.0411 Calculations of Student Learning Growth Using Statewide Assessment Data for Use in School Personnel Evaluations.

(1) Purpose. The purpose of this rule is to provide the formulas for measuring student learning growth, the statewide standards for determining each performance level for use in school district instructional personnel evaluation systems developed under s. 1012.34, Florida Statutes, and procedures associated with implementing the formulas and standards.

(2) Definitions. For the purposes of this rule, the following definitions apply.

(a) “Confidence interval.” A confidence interval is derived from the standard error.  It expresses the precision of a statistic as a range of values.  An individual teacher’s VAM score is an estimate of that teacher’s contributions to student learning growth. The 95% confidence interval used in classification represents a range of possible values that would include the teacher’s VAM score 95% of the time if VAM scores were repeatedly re-estimated with different students for each teacher.

(b) “Courses associated with statewide, standardized assessments” or “courses associated with statewide, standardized assessments under s. 1008.22, F.S.,” are those courses listed in the publication, “Florida VAM Course List,” (effective _____ 2015), which is incorporated herein by reference (insert link). A copy of the Florida VAM Course List may be obtained from the Florida Department of Education, 325 West Gaines Street, Room 544, Tallahassee, FL 32399-0400.

(c) “Covariate.” A covariate is a variable or set of variables used in computing a statistical model.

(d) “Covariate adjustment model.” A covariate adjustment model is a statistical model that controls for the influence of one or more of the covariates.

(e) “Expected score.” An expected score generated by a value-added model for a statewide, standardized assessment is based on the student’s prior statewide, standardized assessment score history and measured characteristics, as well as how other students in the state actually performed on the assessment. For each individual student, the expected score is the sum across all covariates of the value of the covariate multiplied by that covariate’s contribution to student learning as estimated by the covariate adjustment model.

(f) “Observed score.” An observed score is the actual score a student received on an assessment.

(g) “Staff information system.” The staff information system is the comprehensive management information system maintained by the Department containing staff data reported by school districts in accordance with Rule 6A-1.0014, F.A.C., (insert link) which is incorporated by reference herein. A copy of Rule 6A-1.0014, F.A.C., may be obtained from the Florida Department of Education, 325 West Gaines Street, Room 544, Tallahassee, FL 32399-0400.

(h) “Standard error.”  A standard error is a measure of the precision of a statistic.  It is determined by both sample size and sample variability.

(i) “Student information system.”  The student information system is the comprehensive management information system maintained by the Department containing student data reported by school districts in accordance with Rule 6A-1.0014, F.A.C., as incorporated in paragraph (2)(g) of this rule.

(j) “Value-added model” or “VAM.”  A value-added model is a statistical model used for the purpose of determining an individual teacher’s contribution to student learning.

(3) Formulas for measuring student learning growth.

(a) The English Language Arts and Mathematics value-added models.

1. The formula for measuring student learning growth using student English Language Arts and Mathematics results shall be a covariate adjustment value-added model.

The value-added model statistically establishes the expected learning growth for each student, called an expected score. When a student’s actual performance differs from these expectations, a portion of that difference is attributed to the teacher’s and a portion is attributed to the school’s influence. Together, this information is used to compute a teacher’s value-added score.

2. The formula for the model. A full technical description of the data sources, formula, covariates, and methodology for calculating VAM scores is provided in the publication, “Florida VAM Methodology” (Effective _____) (insert link), which is incorporated herein by reference. A copy of “Florida VAM Methodology” may be obtained from the Florida Department of Education, 325 West Gaines Street, Room 544, Tallahassee, FL 32399-0400.

3. The covariates included in the value-added model shall be:

a. The number of subject-relevant courses in which a student is enrolled.

b. At least one (1) and up to two (2) prior years of achievement scores on the applicable statewide, standardized assessment for each student.

c. A student’s disabilities. The disabilities used within the model are limited to language impaired; deaf or hard of hearing; visually impaired; emotional/behavioral disabilities; specific learning disability; dual sensory impaired; autism spectrum disorder; traumatic brain injured; other health impaired; and other intellectual disability.

d. A student’s English Language Learner (ELL) status. This covariate is used to control for effects related to whether a student is an English language learner and has been receiving English for Speakers of Other Languages (ESOL) services for less than two (2) years; at least two (2) years but less than four (4) years; at least four (4) years but less than six (6) years; or six (6) years or longer.  

e. A student’s gifted status.

f. Student attendance.

g. Student mobility. This covariate is used to control for effects related to changing schools during the school year.

h. Difference from modal age in grade. This covariate is used to control for effects related to differences in a student’s age from the most common age for students enrolled in the same grade across the state.

i. Class size. This covariate is used to control for effects related to the number of students in a class.

j. Homogeneity of students’ entering test scores in the class. This covariate is used to control for the variation in student proficiency within a classroom at the beginning of the year.

4. The formula produces a value-added score for a teacher. For English Language Arts and Mathematics, this value-added score consists of two (2) parts:

a. The teacher effect. The teacher effect is an estimate of a teacher’s contributions to student achievement as measured by scores on statewide, standardized assessments. It is based on the difference between expected scores and actual scores for a teacher’s students relative to other teachers in the school, among students assessed in the same subject at the same grade level during the same year.

b. The school component. The school component is an estimate of the part of a student’s performance that is common to students within a school.  It is based on the difference between expected scores and actual scores for the school’s students relative to other schools in the state, among students assessed in the same subject at the same grade level during the same year.  It represents school-level factors influencing performance of all students in a school among students assessed in the same subject at the same grade level during the same year. Fifty (50) percent of the school component shall be added to the teacher effect to create the teacher’s value-added score.

(b) The Algebra I value-added models.

1. The formula for measuring student learning growth using student results from the statewide, standardized end-of-course assessment in Algebra I pursuant to s. 1008.22, F.S., shall be a covariate adjustment value-added model.

The value-added model statistically establishes the expected learning growth for each student, called an expected score. When a student’s actual performance differs from these expectations, a portion of that difference is attributed to the teacher’s and a portion is attributed to the school’s influence. Together, this information is used to compute a teacher’s value-added score.

2. The formula for the model.  A full technical description of the data sources, formula, covariates, and methodology for calculating VAM scores is provided in the publication, Florida VAM Methodology.

3. The covariates included in the Algebra I value-added model shall include those listed under sub-subparagraph (3)(a)3.a.-j., as well as the following:

a. Mean prior test score. Mean prior test score is the average of the most recent prior score on the statewide, standardized assessment in Mathematics for all students within the class.

b. Percent of students in the class who are reported in the student information system as Gifted.

c. Percent at modal age in grade. Percent at modal age in grade is the percentage of students in the class whose age on September 1 of the school year is the same as the modal age of all students in the same grade.

4. The formula produces a value-added score for a teacher. For Algebra I, the score is the teacher effect. The teacher effect is an estimate of a teacher’s contributions to student achievement as measured by scores on statewide, standardized assessments. It is based on the difference between expected scores and actual scores for a teacher’s students relative to other teachers in the school, among students assessed in the same subject at the same grade level during the same year.

(4) Data Collected and Reported for VAM.

(a) Data collection and reporting procedures for purposes of VAM calculations shall be as provided in the publication, Florida VAM Methodology. As set forth in this publication in more detail, data from the Student Information System and Staff Information System obtained from Surveys 2 and 3 are used in VAM calculations. School districts submit Survey 2 and 3 data to the Department’s Student Information System and Staff Information System pursuant to Rule 6A-1.0014, F.A.C. (Comprehensive Management Information System) and Rule 6A-1.0451 (Florida Education Finance Program Student Membership Surveys) (insert link) . These rules are incorporated by reference herein, and a copy of the rules may be obtained from the Florida Department of Education, 325 West Gaines Street, Room 544, Tallahassee, FL 32399-0400.

(b) Results provided to districts shall include the following information for each statewide, standardized assessment for which a formula has been adopted:

1. A value-added score for each teacher based on the statewide, standardized assessment associated with the course(s) that the teacher taught during the current year. This score shall be reported for each grade level and subject area covered by the statewide assessment.

2. Three-year aggregate value-added scores for each teacher, which includes data for the teacher from the current school year and each of the two (2) prior years for which data are available, for a total of at least one (1) and up to three (3) years of data for the teacher, as follows:

a. Three-year aggregate English Language Arts score. A combination of all value-added results for the teacher from all grades and courses associated with the statewide, standardized assessments in English Language Arts.

b. Three-year aggregate Mathematics score. A combination of all value-added results for the teacher from all grades and courses associated with the statewide, standardized assessments in Mathematics.

c. Three-year aggregate combined score. A combination of all value-added results associated with the statewide, standardized assessments in English Language Arts and Mathematics.

d. The aggregate score shall be calculated by standardizing the value-added scores by converting them to a proportion of a year’s average growth within the grade and subject for the year, and combining them across all grades and subjects for as many of the last three (3) years as data are available.

3. The standard error for each value-added score.

(5) Performance-Level Standards for Courses Associated with Statewide, Standardized Assessments.

(a) Data elements used to set performance-level standards are as follows:

1. The statewide average year’s growth for students in each grade and subject. For each student learning growth formula, an average year’s growth for students across the state on the statewide assessment is calculated, and once standardized, uses a threshold of zero (0) to establish performance expectations. A score of zero (0) indicates that a teacher’s students scored no higher or lower, on average, than expected.

2. The educator’s value-added score.

3. The confidence interval. A confidence interval is computed using the standard error associated with the educator’s value-added score.    

(b) Performance-level standards for the Performance of Students Criterion. The performance standards for the performance of students criterion in performance evaluations under s. 1012.34, F.S., for classroom teachers of courses associated with statewide, standardized assessments shall be as follows.

1. Performance-level standards for Florida’s value-added models. The performance-level standards for the English Language Arts and Mathematics value-added models shall be established using the 3-year aggregate combined VAM score for English Language Arts and Mathematics and the grade-level VAM score for Algebra I provided by the Department, except as follows:

a. When a teacher has any student in their VAM score used to determine the Performance of Students component of a teacher’s annual evaluation who had an expected score on any assessment that was higher than the score it was possible to achieve on that assessment;

b. The number of assessments used to calculate the VAM score used to determine the Performance of Students component of a teacher’s annual evaluation is fewer than ten (10);

c. The teacher is not present for more than 50% of the school days associated with the course; or

d. The teacher’s VAM score used to determine the Performance of Students component of a teacher’s annual evaluation includes Advanced Academics courses (7755040 or 7855040).

In the circumstances described in (5)(b)1.a. through d. above, the district shall not be required to use the results of the VAM formula but instead may select an alternate measure of student performance to be used in the educator’s evaluation.

2. The performance-level standards for the English Language Arts and Mathematics value-added models are as follows:

a. Highly Effective. A highly effective rating on Performance of Students criteria is demonstrated by a value-added score of greater than zero (0), where all of the scores contained within the associated 95-percent confidence interval also lie above zero (0).

b. Effective. An effective rating on Performance of Students criteria is demonstrated by the following:

I. A value-added score of zero (0);

II. A value-added score of greater than zero (0), where some portion of the range of scores associated with a 95-percent confidence interval lies at or below zero (0); or

III. A value-added score of less than zero (0), where some portion of the range of scores associated with  both the 68-percent and the 95-percent confidence interval lies at or above zero (0).    

c. Needs Improvement, or Developing if the teacher has been teaching for fewer than three (3) years. A needs improvement or developing rating on Performance of Students criteria is demonstrated by a value-added score that is less than zero (0), where the entire 68-percent confidence interval falls below zero (0), but where a portion of the 95-percent confidence interval lies above zero (0).

d. Unsatisfactory. An unsatisfactory rating on Performance of Students criteria is demonstrated by a value-added score of less than zero (0), where all of the scores contained within the 95-percent confidence interval also lie below zero (0).

(c) Implementing the performance-level standards.

Beginning with the evaluations for performance during the 2015-16 school year, each district school board will implement the performance-level standards for Florida’s English Language Arts, Mathematics and Algebra I value-added models, as described in this rule.

Rulemaking Authority 1012.34, FS. Law Implemented 1012.34, FS. History–New______.

 

NAME OF PERSON ORIGINATING PROPOSED RULE: Juan Copa, Deputy Commissioner, Division of Accountability, Research and Measurement

NAME OF AGENCY HEAD WHO APPROVED THE PROPOSED RULE: Pam Stewart, Commissioner, Department of Education

DATE PROPOSED RULE APPROVED BY AGENCY HEAD: June 24, 2015

DATE NOTICE OF PROPOSED RULE DEVELOPMENT PUBLISHED IN FAR: June 3, 2015