90-002164 Don S. Bates vs. Betty Castor, As Commissioner Of Education Status: Closed Recommended Order on Wednesday, July 31, 1991.

Summary: Essay written for Florida educational leadership examination fairly graded using holistic method pursuant to rule.

STATE OF FLORIDA

DIVISION OF ADMINISTRATIVE HEARINGS

DON BATES, )

)

Petitioner, )

)

vs. ) CASE NO. 90-2164

)

BETTY CASTOR as COMMISSIONER OF )

EDUCATION, )

)

Respondent. )

___________________________________)

RECOMMENDED ORDER

Pursuant to notice, the Division of Administrative Hearings, by its duly

designated Hearing Officer, Claude B. Arrington, held a formal hearing in the

above-styled case on April 22 and 23, 1991, in Tallahassee, Florida.

APPEARANCES

For Petitioner: Thomas W. Young, III, Esquire

FEA/United

118 North Monroe Street

Tallahassee, Florida 32399-1700

For Respondent: Charles S. Ruberg, Esquire

Assistant General Counsel

Florida Board of Education

The Capitol, Suite 1701

Tallahassee, Florida 32399-0400

STATEMENT OF THE ISSUE

Whether the scores given to Petitioner's essay on the communication subtest

of the May 1989 Florida Education Leadership Examination were inconsistent with

and the result of an improper application of the holistic scoring methodology,

and whether the scores given to his essay on the communication subtest of the

November 1989 Florida Education Leadership Examination were inconsistent with

and the result of an improper application of the holistic scoring methodology.

PRELIMINARY STATEMENT

Petitioner, a professional educator, sat for the Florida Education

Leadership Examination (FELE) in May 1989. After Petitioner was advised that he

had not passed the communications subtest of the May 1989 FELE, he attempted to

determine the reasons he did not pass. While he was awaiting a response from

Respondent as to the May 1989 scores, he retook the communications subtest in

November 1989. After he was advised that he had not passed the communication

subtest in either May or November 1989, he filed these challenges to the manner

in which the essay portion on both the May 1989 FELE and the November 1989 FELE

were scored. Petitioner contends that a proper application of the holistic

scoring method, the method set forth by rule, would produce a higher score.

Petitioner does not challenge the underlying rule that requires application of

the holistic scoring method.

At the formal hearing, Petitioner testified on his own behalf and presented

the additional testimony of Betty Owen, Charles R. Blackmon, and David Kirby.

Ms. Owen is an experienced reader for the FELE and was one of the readers who

scored Petitioner's essay for the November 1989 FELE. Dr. Blackmon is a retired

educator and educational administrator who was accepted as an expert witness in

the field of educational administration and in the evaluating written

communication using the holistic scoring method. Dr. Kirby is a McKenzie

professor of English at Florida State University and was accepted as an expert

witness in the field of evaluating written communication using the holistic

scoring method. The parties submitted the following joint exhibits which were

accepted into evidence: A1-6, B1-6, C1-4, D, E, F, and G. Petitioner submitted

one additional exhibit, which was accepted into evidence. Respondent submitted

three additional exhibits, which were accepted into evidence.

A transcript of the proceedings has been filed. At the request of the

parties, the time for filing post-hearing submissions was set for more than ten

days following the filing of the transcript. Consequently, the parties waived

the requirement that a recommended order be rendered within thirty days after

the transcript is filed. Rule 22I-6.031, Florida Administrative Code. Rulings

on the parties' proposed findings of fact may be found in the Appendix to this

Recommended Order.

FINDINGS OF FACT

1. Petitioner is a professional educator who moved to Florida in 1988 to

teach in the Broward County public school system. He completed Florida's

beginning teacher program and passed the Florida Teacher Certification

Examination. At the time of the formal hearing, Petitioner was employed as a

social studies teacher in one of the public schools of Broward County.

2. Petitioner received his Bachelor of Science degree from East Tennessee

State University (ETSU) in 1967, majoring in Political Science and Speech. In

1971, he received a Masters of Art degree in Educational Administration and

Supervision with a minor in Instructional Communication from ETSU. He earned

his Doctorate of Education degree from the University of Alabama in 1974, with a

major in Educational Administration and Supervision and a minor in Curriculum

Development and Instructional Communication.

3. Petitioner has been teaching continuously since 1967, either at the

high school or college level. He has considerable experience in reading and

evaluating written communication of high school and graduate students. He has

served as an assistant principal in the State of Alabama and holds

superintendent's credentials as well as principal's credentials and supervisory

and teaching credentials for the states of Alabama and Louisiana. Petitioners

desires to work as an assistant principal in the public schools of Florida.

4. A person who desires to hold the position as a principal or as an

assistant principal in a Florida public school district must earn an Educational

Leadership Certificate from Respondent. Among the requirements for achieving

this Certificate is the successful completion of the Florida Education

Leadership Examination (FELE).

5. FELE is administered twice each year to those seeking certification in

educational leadership. The examination covers the eight areas of the Florida

Educational Leadership Core Curriculum listed in Rule 6A-4.0082, Florida

Administrative Code. The candidates for certification are notified by the FELE

registration bulletin that the examination consists of three subtests with

Subtest One covering "Management, Leadership, Personnel"; Subtest Two covering

"Communications (includes essay)"; and Subtest Three covering "Curriculum,

Finance, Law, Technology". The registration bulletin also notifies candidates

that:

With the exception of the Communications essay

question, all questions will be multiple choice.

There will be one essay question designed to

demonstrate proficiency in written communication.

The essay question will be scored holistically

by trained experts.

6. As indicated by the registration bulletin, the Communications Subtest

consists of multiple choice questions and of an essay. The scores received on

the two parts of the Communications Subtest are combined and statistically

adjusted to determine whether the candidate passed the subtest. (There are

several different versions of FELE that have been developed over the years. The

statistical adjustment is made to maintain consistency in the examination

process by making adjustments in the scoring process to account for differences

in the comparative difficulty between the different versions of FELE. No

challenge is raised to the fact that statistical adjustments are made or as to

how those adjustments were made for the examinations in question.)

7. A candidate must pass all three subtests of the FELE in order to

receive certification, and he can retake any subtest that he failed to pass.

8. A candidate chooses the topic for his essay from the two topics that

are offered by the essay portion of the Communications Subtest. While the

topics relate to assignments that a school administrator might receive, the

essay portion of the examination is designed to test the candidate's

communication skills; it is not designed to test the candidate's administrative

skills or substantive knowledge of the topic of the essay.

9. Rule 6A-4.00821, Florida Administrative Code, pertains to FELE and

provides, in pertinent part, as follows:

(7) Scoring.

(b) The essay portion to the school communications

subtest shall be scored by two (2) trained judges

using a scale of one (1) which is an unsatisfactory

score to four (4) which is an outstanding score.

In the event the two (2) ratings are two (2) or more

points different, or in the event the summed ratings

equal three (3), the writing sample will be rated by

a referee and the referee's score will replace the

most discrepant of the original ratings.

(d) Beginning July 1, 1988, a passing score for each

subtest of the Florida Education Leadership Examination

shall be:

1. School Communications. Examinee scores for the

school communications subtest shall be reported as an

average scaled score from the essay test and the scaled

score from the multiple choice questions. The passing

score shall be the scaled score equivalent to the

combination of the essay total raw score of four (4)

and a multiple choice total raw score of fifteen (15)

on the November, 1987 administration of the subtest.

(8) Essay performance standards.

(c) Rating scale. The four-level scale for judging

the written essays is defined as follows:

1. A rating of one (1) indicates the essay lacks

unity and focus. It is distorted or ambiguous, and

it fails to treat the topic in sufficient depth and

breadth. There is little or no discernible

organization and only scant development of ideas,

if any at all. The essay betrays only sporadically

a sense of paragraph and sentence structure, and it

is syntactically slipshod. Usage is irregular and

often questionable or wrong. There are serious errors

in spelling, capitalization, and punctuation.

2. A rating of two (2) indicates the essay has some

degree of unity and focus, but each could be improved.

It is reasonably clear, though not invariably so, and

it treats the topic with a marginal degree of

sufficiency. The essay reflects some concern for

organization and for some development of ideas, but

neither is necessarily consistent nor fully realized.

The essay reveals some sense, if not full command of

paragraph and sentence structure. It is syntactically

bland and, at times, awkward. Usage is generally

accurate, if not consistently so. There are some

errors in spelling, capitalization, and punctuation

that detract from the essay's effect if not from its

sense.

3. A rating of three (3) indicates the essay is

focused and unified, and it is clearly if not

distinctly written. It gives the topic an adequate

though not always thorough treatment. The essay is

well organized, and much of the time it develops

ideas appropriately and sufficiently. It shows a

good grasp of paragraph and sentence structure, and

its usage is generally accurate and sensible.

Syntactically, it is clear and reliable. There may

be a few errors in spelling, capitalization, and

punctuation, but they are not serious.

4. A rating of four (4) indicates the essay is

unified, sharply focused, and distinctively effective.

It treats the topic clearly, completely, and in

suitable depth and breadth. It is clearly and fully

organized, and it develops ideas with consistent

appropriateness and thoroughness. The essay reveals

an unquestionably firm command of paragraph and

sentence structure. Syntactically, it is smooth and

often elegant. Usage is uniformly sensible, accurate,

and sure. There are very few, if any errors in

spelling, capitalization, and punctuation.

10. In addition to the foregoing, Rule 6A-4.00821, Florida Administrative

Code, contains minimal qualifications for those persons who will score the

essays. The chief reader and the scorers involved in the scoring of

Petitioner's essays for both the May 1989 and the November 1989 examinations are

well qualified, appropriately trained, and experienced in the application of the

holistic scoring method. 1/

11. The holistic scoring method is widely used to evaluate essays such as

the ones on the FELEs involved in this case. 2/ Unlike the analytical method

of evaluating essays, the holistic method is designed only for scoring purposes

and is not designed to give the writer feedback as to how he can improve his

writing or why the essay received a particular score. For that reason,

Petitioner received no feedback following the examinations, and he was not

informed what he needed to do to improve his essay.

12. The following instructions were among the general instructions given

to the candidates for the May 1989 FELE and the November 1989 FELE:

Your essay will be judged on your ability to

write in a logical, easily understood style

and on the quality of information provided,

NOT on the opinion expressed.

13. The following essay rating criteria were provided the candidates for

the May 1989 FELE and the November 1989 FELE:

Unity, organization, and focus (described as

"consistency, coherence, order and interdependence

of parts, creating a single, integrated effect")

Sufficiency and development (described as

"appropriateness of expression to meet writer's

and subject's needs; breadth and depth; use of

detail, examples, illustrations")

Usage and syntax (described as "appropriate

ordering of words to convey intended meaning;

appropriate use of language, inflection, tense,

agreement, vocabulary")

Clarity and paragraph and sentence structure

(described as "lucidity of expression; paragraph

structure; variety, logic, and relatedness of

sentences within paragraphs")

Spelling, capitalization, punctuation (described as

"currently accepted standards")

14. To produce reliable and consistent scoring of the essays, a well

developed process is closely followed. The first stage of the holistic scoring

process is the selection of a chief reader, an assistant chief reader, and

readers. 3/

15. The chief reader meets with the assistant chief reader and with other

readers, normally selected because of their experience, to select range finders

and sample papers. This group, referred to as selectors, first discusses the

scoring definitions and builds a consensus as to the criteria by which the

papers will be judged. The members of the group review range finders from a

previous administration of the examination so that the criteria used will be

consistent with past scoring. They then read, rank, and discuss 40-50 sample

papers, selected at random from the current examination. Thereafter, in a

process that builds towards a consensus as to what scores the different sample

papers should receive, they discuss what ratings these papers received and how

the rating criteria applies. They then select at least six range finders as

being typical of the one, two, three, and four categories plus two scores that

are borderline. These range finders are backed up by 15-20 sample papers which

also exemplify scale level and borderline cases. It is the responsibility of

the chief reader to determine that the selectors have reached an appropriate

understanding as to the criteria to be applied to scoring these essays.

16. After the range finders and the samples have been selected, the chief

reader and the assistant chief readers meet with those readers who will have the

responsibility of scoring the essays. The process used to build consensus among

the selectors as to the appropriate scoring criteria is followed to build

consensus among the readers. It is the responsibility of the chief reader to

determine that all readers have a clear understanding as to the criteria to be

used in scoring the essays. The actual scoring does not begin until after the

chief reader has made that determination. Throughout the scoring process, the

chief reader (or the table leader, if an examination is taken by enough people

to justify a table reader) monitors the reading to ensure that the readers are

applying the appropriate criteria in scoring the essays.

17. Each essay is independently scored by two readers, each of whom is

unaware of the score given by the other reader. No participant in the scoring

process knows the identity of the essay writer.

18. Essays which receive the same or contiguous scores from two readers

are completed and those scores are recorded. For example, if the two readers

each scored an essay as a two, the total score the candidate would receive on

his essay would be a four. If one reader scored the essay as a two while the

other scored it as a three, the total score the candidate would receive on his

essay would be a five. That score would be added to the score he received on

the multiple choice portion of the communications subtest and statistically

adjusted to determine his final score on the communications subtest. The

passing score for the communications subtest is determined as provided by Rule

6A-4.00821(7)(d)1, Florida Administrative Code.

19. Petitioner took the FELE in May 1989. He failed the communications

subtest, but he passed the remaining portions of the examination. On the May

1989 examination, Petitioner received on his essay a score of two from each of

the two readers who scored his essay. The four points he received on the essay

portion of the subtest, when added to the score he received on the multiple

choice portion of the subtest and statistically adjusted were insufficient to

attain a passing grade on the communication subtest. Petitioner needed a score

of three from each of the two readers to pass the communications subtest.

20. Petitioner retook the communications subtest in November 1989 and

again failed to achieve a passing score. On the November 1989 examination,

Petitioner again received on his essay a score of two from each of the two

readers. Petitioner needed a score of three from one of the two readers to pass

the communications subtest.

21. There was nothing about the testing conditions on either occasion that

treated the Petitioner differently from any other candidate or that interfered

with his opportunity to do his best writing. The holistic scoring process for

each examination was carried out according to expectations without any problems

or unusual circumstances. The procedural safeguards, designed to ensure

uniformly fair scoring, were followed for both examinations. In neither case

did the scoring of Petitioner's essay trigger the referring part of the process.

22. In the May 1989 examination, eight range finders were used, but only

two dealt with the topic Petitioner selected. Those range finders were assigned

a score of 2 and a 4, respectively.

23. Dr. Robert Blackmon and Dr. David Kirby, both of whom have impressive

academic credentials and both of whom were accepted as experts in the field of

holistic scoring of essays, were of the opinion that the Petitioner's essay on

the May 1989 examination merited a score of three. Prior to forming that

opinion, each had reviewed the two range finders pertinent to the topic selected

by Petitioner and the scoring criteria used by the readers. Each then evaluated

Petitioner's essay based on their review of the range finders and the criteria.

24. In the November 1989 examination, six range finders were used, with

two dealing with the topic selected by Petitioner. Following the procedure they

used in evaluating the May 1989 essay, Dr. Blackmon and Dr. Kirby reviewed

Petitioner's essay for the November 1989 exam. Again, both of these witnesses

were of the opinion that the Petitioner's essay merited a score of three.

25. The scoring of an essay using only two range finders is not as

accurate as the scoring would be if there were additional range finders to be

used for comparison. Neither Dr. Blackmon or Dr. Kirby reviewed range finders

from other administrations of the FELE, neither reviewed the samples that had

been selected, and neither had participated in any process designed to build

consensus as to the appropriate scoring technique. Both reviewed Petitioner's

two essays in greater detail than that employed by the holistic scoring

contemplated for the FELE examination.

26. The use of only two range finders on the topic selected by Petitioner

for the May 1989 exam and for the November 1989 exam did not invalidate the

scoring process. The readers involved in the scoring of Petitioner's essays for

the May and November 1989 examinations had sufficient range finders and

sufficient samples to enable them to fairly score Petitioner's essays. While

Petitioner's experts were of the opinion that a higher score should be awarded,

these opinions were formed using a pre-scoring process different than that used

by Respondent in evaluating the FELE and are not afforded greater weight than

the scores given by Respondent's readers.

27. It is found, based on the greater weight of the evidence, that

Petitioner's essays were fairly scored by Respondent's readers, that the scoring

process used by Respondent complied in all material ways with the holistic

scoring method, and that the scores awarded Petitioner by Respondent's readers

should not be changed.

CONCLUSIONS OF LAW

28. The Division of Administrative Hearings has jurisdiction over this

matter. Section 120.57(1), Florida Statutes.

29. Petitioner has the burden of proving by a preponderance of the

evidence that he is entitled to the relief he seeks. Rule 28-6.08(3), Florida

Administrative Code. See also, Florida Department of Transportation v. J.W.C.,

Co., 396 So.2d 778 (Fla. 1st DCA 1981). Petitioner has failed to meet that

burden.

RECOMMENDATION

Based upon the foregoing Findings of Fact and Conclusions of Law, it is

RECOMMENDED that a Final Order be entered which denies Petitioner's

challenge to the scoring of his essay on the May 1989 Florida Educational

Leadership Examination and which denies his challenge to the scoring of his

essay on the November 1989 Florida Educational Leadership Examination.

DONE AND ORDERED in Tallahassee, Leon County, Florida, this 31st day of

July, 1991.

___________________________________

CLAUDE B. ARRINGTON

Hearing Officer

Division of Administrative Hearings

The DeSoto Building

1230 Apalachee Parkway

Tallahassee, Florida 32399-1550

[phone]

Filed with the Clerk of the

Division of Administrative Hearings

this 31st day of July, 1991.

ENDNOTES

1/ Dr. Charles R. Blackmon, one of Petitioner's expert witnesses, expressed the

opinion that the readers were not qualified because of their lack of experience

in school administration. Dr. Blackmon's opinion in this regard is rejected as

being contrary to the greater weight of the evidence. Likewise rejected is

Petitioner's challenge to the qualifications of the readers because no

statistics were used to evaluate their performances. The qualifications of the

readers were established by the testimony presented at the hearing and by the

exhibits which were accepted into evidence.

2/ Petitioner does not challenge Respondent's use of the holistic method of

evaluating his essays. His challenge is limited to the scores that resulted by

the application of the scoring method.

3/ The readings for the May 1989 and November 1989 FELEs were not large enough

to justify the use of table leaders. If table leaders are necessary, those

individuals would be chosen and would typically participate in the selection of

range finders and samples.

APPENDIX TO THE RECOMMENDED ORDER

IN CASE NO. 90-2164

The following rulings are made on the proposed findings of fact submitted

on behalf of the Petitioner.

1. The proposed findings of fact in paragraphs 1, 2, 3, 4, 5, 6, 11, 12,

13, 16, and 20 are adopted to the extent the proposed findings are deemed

necessary to the conclusions reached.

2. The proposed findings of fact in paragraphs 7, 8, 10, 14, 15, 17, 19,

and 22 are adopted in material part by the Recommended Order.

3. The proposed findings of fact in paragraph 9 are rejected because the

proposed finding is not necessary to the conclusions reached and to protect the

confidentiality of the examination question.

4. The proposed findings of fact in paragraphs 18 and 21 are rejected as

being subordinate to the findings made.

5. The proposed findings of fact in the first sentence of paragraph 23 are

rejected as being the recitation of testimony that is without probative value.

The proposed findings of fact in the second sentence of paragraph 25 are

rejected as being subordinate to the findings made.

The following rulings are made on the proposed findings of fact submitted

on behalf of the Respondent.

1. The proposed findings of fact in paragraphs 1, 2, 3, 4, 5, 6, 8, 9, 11,

12, 13, 14, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 30, and 34 are adopted

in material part by the Recommended Order.

2. The proposed findings of fact in paragraphs 7, 10, and 15 are adopted

in part by the Recommended Order and are rejected in part as being subordinate

to the findings made.

3. The proposed findings of fact in paragraphs 28, 29, 31, and 33 are

rejected as being subordinate to the findings made.

COPIES FURNISHED:

Thomas W. Young, III, Esquire

FEA/United

118 North Monroe Street

Tallahassee, Florida 32399-1700

Charles S. Ruberg, Esquire

Assistant General Counsel

Florida Board of Education

The Capitol - Suite 1701

Tallahassee, Florida 32399-0400

George A. Bowen

Acting Executive Director

301 Florida Education Center

325 W. Gaines Street

Tallahassee, Florida 32399-0400

Jerry Moore, Administrator

Professional Practices Service

352 Florida Education Center

325 W. Gaines Street

Tallahassee, Florida 32399-0400

NOTICE OF RIGHT TO SUBMIT EXCEPTIONS

All parties have the right to submit written exceptions to this Recommended

Order. All agencies allow each party at least 10 days in which to submit

written exceptions. Some agencies allow a larger period within which to submit

written exceptions. You should contact the agency that will issue the final

order in this case concerning agency rules on the deadline for filing exceptions

to this Recommended Order. Any exceptions to this Recommended Order should be

filed with the agency that will issue the final order in this case.