17-000424 Daryl Bryant vs. Pam Stewart, As Commissioner Of Education Status: Closed Recommended Order on Friday, October 13, 2017.

Summary: Petitioner did not prove any grounds for invalidating his failing score on the GK essay test.

STATE OF FLORIDA

DIVISION OF ADMINISTRATIVE HEARINGS

DARYL BRYANT,

Petitioner,

vs. Case No. 17 - 0424

PAM STEWART, AS COMMISSIONER OF

EDUCATION,

Respondent.

_______________________________/

RECOMMENDED ORDER

On June 14, 2017, Elizab eth W. McArthur, Administrative Law

Judge, Division of Administrative Hearings (DOAH), conducted the

hearing in this cause in Orlando, Florida.

APPEARANCES

For Petitioner: Jennifer Diane Rose, Esquire

Post Office 924

Melbourne, Florida 32902

For Respondent: Bonnie Ann Wilmot, Esquire

Darby G. Shaw, Esquire

Department of Education

Suite 1244

325 West Gaines Street

Tallahassee, Florida 32399

STATEMENT OF THE ISSUE

The issue for determination is whether PetitionerÓs

challenge to the failing score he received on the essay portion

of the Florida Teacher Certification ExaminationÓs (FTCE) General

Knowledge (GK) test should be sustained.

PRELIMINARY STATEMENT

Petitioner, Daryl Bryant (Petitioner), took the GK essay

test on June 25, 2016. The score report he subsequently received

showed that he did not earn a passing score, having received a

score of seven ( on a scale of two to 12), when a score of eight

was required to pass. Petitioner utilized the score verification

procedures in statute and rule, and by letter dated September 26,

2016, the Department of Education (DOE) informed Petitioner of

its determina tion that his essay had been scored correctly.

Petitioner was informed of his right to an administrative hearing

pursuant to sections 120.569 and 120.57, Florida Statutes

(2017), 1/ to dispute the decision.

Petitioner initially requested an informal hearin g not

involving disputed facts, but after discussions with counsel for

DOE, Petitioner notified counsel by email on December 15, 2016,

that he wanted a formal (i.e., disputed - fact) hearing instead of

an informal hearingeating the email as a petition, DOE

issued an Order dismissing the petition without prejudice, and

allowed Petitioner to amend his petition to conform to the

requirements for petitions in Florida Administrative Code Rule

28 - 106.201(2). Petitioner timely filed an amended request

(petiti on) for a disputed - fact hearing on January 4, 2017, and

the matter was transmitted to DOAH for assignment of an

administrative law judge to conduct the requested hearing.

After being unable to reach Petitioner, DOE filed a response

to the Initial Order tha t sought information related to

scheduling the hearing. DOE identified another DOAH case

characterized as similar to this case, with overlapping witnesses

for DOE. Consolidation was not requested, but DOE requested that

if possible, the two hearings be c oordinated and scheduled on

back - to - back days. DOE suggested some dates, and requested a

live hearing in Orlando. The hearing was scheduled for March 14,

2017, in Orlando, by Administrative Law Judge J.D. Parrish, and

the similar case was scheduled for M arch 13, 2017. Shortly

thereafter, DOE filed a Motion to Limit the Scope of Review in

T his Matter, followed by an amended motion. DOE learned that

counsel would be representing Petitioner, although no notice of

appearance was filed, and DOE served its mo tion and amended

motion on PetitionerÓs counsel.

After a motion for continuance was filed on February 28,

2017, in the related case by the petitioner, DOE filed a motion

for continuance in this case, so as to be able to agree to the

requested continuance i n the related case, while trying to keep

the hearing schedules coordinated. Petitioner did not file a

response opposing the continuance. Judge Parrish granted DOEÓs

motion for continuance and rescheduled the hearing for May 2,

2017, in Orlando. A contin uance was also granted in what had

become the companion case for scheduling purposes (DOAH Case

No. 17 - 0423), and that hearing was reset for May 1, 2017.

On March 20, 2017, Petitioner filed a response to DOEÓs

pending motion to limit the scope of the hear ing. On April 3,

2017, this case and the companion case were transferred to the

undersigned. On April 5, 2017, an Order was issued denying DOEÓs

motion to limit the scope of the hearing; that Order was amended

(corrected) by Order issued on April 7, 2017 . A similar DOE

motion to limit the hearing scope in the companion case was also

denied by Order issued on April 5, 2017.

Meanwhile, a second motion for continuance was filed in the

companion case by the petitioner, based on the uncertainty caused

by DOEÓ s motion while it had been pending, affecting such matters

as the scope of permissible discovery and potential evidence to

prepare for hearing. Since the same issues would apply to this

case, a joint telephonic status conference was scheduled, with

counse l for parties in both cases participating. It was agreed

that both hearings would be continued and rescheduled as soon as

feasible, while allowing the parties sufficient time to complete

discovery and hearing preparation. Based on the agreement of all

pa rties regarding how much time was needed to prepare, the

hearing in this case was reset for June 14, 2017, in Orlando (and

the companion case was reset for June 13, 2017).

To resolve confidentiality issues raised by DOE in its

motion to limit the hearing, the parties entered into a

Confidentiality Agreement. Pursuant to that agreement, DOE filed

an agreed Motion for Protective Order on June 6, 2017, to address

the handling of confidential materials and testimony at the

hearing. A Protective Order was issu ed on June 7, 2017.

On June 7, 2017, a Joint Pre - hearing Stipulation was filed,

in which the parties identified their witnesses and proposed

exhibits, and agreed to a few facts. The agreed facts are

incorporated below.

On June 9, 2017, Petitioner filed a motion to permit a

witness who was in New York to testify by telephone. The motion

was granted by Order issued on June 12, 2017.

At the hearing, the parties offered Joint Exhibits 1

through 8, identified as confidential testing material subject to

the P rotective Order, which were admitted as such and are sealed.

Petitioner testified on his own behalf. Petitioner did not offer

the testimony of the witness in New York for whom leave to

testify by telephone had been granted. Petitioner did not offer

any additional exhibits besides the Joint Exhibits.

Respondent presented the testimony of the following

witnesses: Michael Grogan, Pearson director of performance

assessment scoring services; Phil Canto, DOE bureau chief of

post - secondary assessment; Betsy G riffey, a FTCE GK essay chief

reviewer; and Mary Jane Tappen, DOE vice chancellor for K - 12

student achievement and student services. In addition, the

parties stipulated to adopt by reference as testimony in this

case the non - confidential testimony given b y Dr. Grogan,

Mr. Canto, and Ms. Tappen the previous day in DOAH Case

No. 17 - 0423, since both Petitioner and counsel for Petitioner

were in attendance for that testimony, and counsel for Petitioner

was permitted to cross - examine the witnesses as to their previous

dayÓs testimony. 2/ RespondentÓs Exhibits 1 through 7 (which are

not confidential) were admitted.

As stated on the record, the undersigned took official

recognition of the statutes and rules, including the publications

incorporated by reference, r elated to the FTCE.

In addition to the confidential exhibits under seal,

portions of the hearing were deemed confidential and the hearing

room was cleared of persons not bound by the Protective Order.

Those designated portions of the transcript are also u nder seal.

At the conclusion of the hearing, it was agreed that the

deadline to submit proposed recommended orders (PROs) would be an

extended deadline of 30 days from the filing of the transcript.

The one - volume Transcript of the hearing was filed on July 10,

2017. By subsequent motions to extend the PRO deadline by the

petitioner in the companion case and by Petitioner in this case,

the PRO filing deadline was extended until August 30, 2017.

Both parties timely filed their PROs by the extended deadline ,

and the PROs have been carefully considered in the preparation of

this Recommended Order.

FINDING S OF FACT

1. Petitioner has been employed as a teacher for the past

three years. He had a temporary Florida teacher certificate, but

at the time of the hea ring, he said that he believes it was

expired. Petitioner is seeking to qualify for a (non - temporary)

Florida teacher certificate. Petitioner first must pass the

essay part of the GK test to complete the GK requirements. He

would then be qualified to pr oceed to address the remaining

certification requirements. See § 1012.56(2)(g), (h), (i), and

(7), Fla. Stat.

2. Respondent, Pam Stewart, as Commissioner of Education,

is the stateÓs chief educational officer and executive director

of DOE. §§ 20.15(2) a nd 1001.10(1), Fla. Stat.

3. One of DOEÓs responsibilities is to review applications

for educator certification and determine the qualifications of

applicants according to eligibility standards and prerequisites

for the specific type of certification sou ght. See § 1012.56,

Fla. Stat. One common prerequisite is taking and passing

examinations relevant to the particular certification.

4. Respondent is authorized to contract for development,

administration, and scoring of educator certification exams.

§ 1012.56(9)(a), Fla. Stat. Pursuant to this authority,

following a competitive procurement in 2011, Pearson was awarded

a contract to administer and score FloridaÓs educator

certification exams, including the FTCE.

5. The State Board of Education (SBE) is the collegial

agency head of DOE. § 20.15(1), Fla. Stat. As agency head, the

SBE was required to approve the contract with Pearson. The SBE

is also charged with promulgating certain rules that set forth

policies related to educator certification, su ch as requirements

to achieve a passing score on certification exams. DOE develops

recommendations for the SBE regarding promulgating and amending

these rules. In developing its recommendations, DOE obtains

input and information from a diverse group of F lorida experts and

stakeholders, including active teachers, school district

personnel, and academicians from colleges and universities.

FTCE Development, Administration, and Scoring

6. DOE develops the FTCE, as well as the other educator

certification ex ams, in - house. The FTCE is developed and

periodically revised to align with SBE - promulgated standards for

teachers. In addition, as required by statute, certification

exams, including the FTCE, must be aligned to SBE - approved

student standards.

7. Deta ils about the FTCE, such as the competencies and

skills to be tested, the exam organization, and passing score

requirements, are set forth in Florida Administrative Code Rule

6A - 4.0021 (the FTCE rule). The FTCE rule has been amended

periodically, but the current version includes a running history,

setting forth FTCE details that applied during past time periods,

as well as those currently in effect.

8. The FTCE is not actually a single examination. It

consists of multiple separate examinations to meet th e different

requirements for teacher certification and the different options

for specific subject areas. Descriptions of the areas to be

tested by each FTCE component are set forth in a publication

incorporated by reference in the FTCE rule. The version of this

publication that was in effect when Petitioner took the exam at

issue in this proceeding is identified in the FTCE rule as:

ÐCompetencies and Skills Required for Teacher Certification in

Florida, Twenty - Second Edition.Ñ

9. As set forth in the FT CE rule, the GK exam consists of

four subtests. Subtest one is the essay test; subtest two,

three, and four are multiple choice tests covering English

language skills, reading, and math, respectively.

10. Petitioner met the requirements for GK subtests two,

three, and four, by virtue of having taken and passed the College

Level Academic Skills Test (CLAST) in those areas prior to

July 1, 2002. 3/ Therefore, Petitioner only had to take and pass

subtest one, the essay exam, to satisfy all GK requirements.

11. The competency and skills to be tested by the GK essay

test, as promulgated by the SBE and codified by reference in the

FTCE rule, are as follows:

Knowledge of formal college - level writing

• Determine the purpose of writing to task

and audience.

• Pro vide a section that effectively

introduces the topic.

• Formulate a relevant thesis or claim.

• Organize ideas and details effectively.

• Provide adequate, relevant support by

citing ample textual evidence; response

may also include anecdotal experience for

a dded support.

• Use a variety of transitional devices

effectively throughout and within a

written text.

• Demonstrate proficient use of college -

level, standard written English (e.g.,

varied word choice, syntax, language

conventions, semantics).

• Provide a co ncluding statement or section

that follows from, or supports, the

argument or information presented.

• Use a variety of sentence patterns

effectively.

• Maintain a consistent point of view.

• Apply the conventions of standard English

(e.g., avoid inappropriat e use of slang,

jargon, clichés). (Competencies and

Skills Required for Teacher Certification

in Florida, Twenty - Second Edition, page 2

of 247, incorporated by reference in the

FTCE rule).

12. Prior to January 1, 2015, a score of at least six

(using a s coring range from two points to 12 points) was required

to pass the GK essay test.

13. Based on input from educators, academicians, and other

subject matter experts, DOE recommended that the passing score

for the GK essay test be raised from a score of six to a score of

eight (using the same range of two points to 12 points). The SBE

adopted the recommendation, which is codified in the FTCE rule:

eight is the required passing score for GK essays as of

January 1, 2015.

14. Without question, the highe r passing score requirement

makes it more difficult to pass the GK essay. The policy

underlying this scoring change is to make the GK essay test more

rigorous, in recognition of the critical importance of writing

skills. By raising the standards for demo nstrating mastery of

the writing skills tested by the GK essay test, the GK essay test

better aligns with increasingly rigorous SBE - approved student

standards for written performance. This policy change is

reasonable and within the purview of the SBE; in any event, it is

not subject to debate in this case, because Petitioner did not

challenge the FTCE rule.

15. Not surprisingly, since the passing score was raised

for the GK essay, the overall passage rates have dropped. The

passage rates were 96 percent in 2013 and 93 percent in 2014,

when the passing score was lower. After the passing score was

raised, the passage rates were 63 percent in 2015 and 69 percent

in 2016. While Petitioner characterizes the 69 percent passage

rate as ÐlowÑ (Pet. PRO at 4, ¥ 13), that is an opinion that is

unsupported by any testimony offered at hearing. Petitioner did

not offer any expert witness to testify on his behalf. Instead,

based on the testimony offered on this subject at the final

hearing, the more reasonable infe rence to draw from the overall

GK essay passage rates is that the passage rates were too high

prior to 2015. The overall GK essay passage rate, standing

alone, is not evidence that the GK essay is arbitrary,

capricious, unfair, or invalid.

16. Pursuant t o its contract with DOE as the test

administration and test scoring agency, Pearson administers and

scores GK essay exams. Pearson employs holistic scoring as the

exclusive method for scoring essays, including GK essays (as

specified in PearsonÓs contract with DOE). The holistic scoring

method is used to score essay examinations by professionals

across the testing service industry. Pearson has extensive

experience in the testing service industry, currently providing

test scoring services to more than 20 states.

17. Dr. Michael Grogan, PearsonÓs director of performance

assessment scoring services and a former chief rater, has been

leading sessions in holistic scoring or training others since

2003. He described the holistic scoring method as a process o f

evaluating the overall effect of a response, weighing its

strengths and weaknesses, and assigning the response one score.

Through training and use of tools, such as rubrics and exemplars,

the evaluation process becomes less subjective and more

standardi zed, with professional bias of individual raters

minimized, and leading to consistent scoring among trained

raters. Training is therefore an integral part of PearsonÓs

testing services for which DOE contracted. In an intensive two -

day training program, p rospective raters are trained in the

holistic scoring method used to score GK essays.

18. PearsonÓs rater training program begins with a review

of background about the holistic scoring method generally,

including discussions about rater bias. From there, trainees are

oriented to GK essay - specific training material. They thoroughly

review and discuss the rubric, the score scale (which is one

point to six points), the operational prompt raters will be

scoring, and exemplars (other responses to the prompt t hat have

been pre - scored). The rater candidates then employ these tools

to begin independently scoring exemplars. Raters - in - training

conduct many rounds of independent scoring sessions, interspersed

with group discussions regarding how the essays should have been

scored. The trainees then move into the calibration test phase,

in which they independently score essay exemplars, paired with an

experienced rater who independently scores the same exemplars.

The trainees score essay after essay, then compare scores with

the experienced rater, with the goal to achieve consistency in

scores, by equaling or coming within one point of the other

raterÓs score. Ultimately, the raters must pass the calibration

test by achieving scoring consistency to qualify for app ointment

as raters to score actual GK essays.

19. Raters who conduct scoring of the GK essay must meet

qualifications specified by DOE (including teacher certification

and experience). Pearson proposes qualified individuals to DOE,

and then DOE must a pprove proposed raters. Then the approved

raters must undergo and successfully complete PearsonÓs training.

20. Each GK essay is scored independently by two qualified

raters. Pairs of raters receive scoring assignments, one prompt

at a time. The assig nments are received anonymously; one rater

does not know who the other assigned rater is. And neither rater

knows anything about the examinee, as the essay is identified

solely by a blind number. GK essay raters work in one room, at

individual computer t erminals, in Hadley. Security of all

testing information is vigilantly maintained, through

confidentiality agreements and secure, limited, and protected

computer access.

21. For each scoring assignment, raters adhere to a step -

by - step process that reinfo rces their initial training. Raters

must first score sample responses to a historic prompt that is

different from the assigned prompt, as a training refresher to

invoke the holistic scoring mindset. From there, raters review

the assigned prompt and the s coring rubric. Raters then must

score an anchor set of six sample responses, one exemplifying

each score category; the historic scores are not revealed until

the raters complete their scoring. Raters compare their scores

with the anchor scores and work t hrough any discrepancies.

Raters then go through a calibration process of scoring 10 more

sample responses to the same prompt. After scoring all 10

essays, the raters learn the scores deemed appropriate for those

responses, and must work through any disc repancies until

consistency is achieved. Only after scoring many sample essays

and achieving scoring consistency are the raters permitted to

turn to the assigned GK essay for review and scoring.

22. Pearson also employs chief raters to supervise and

mon itor the raters while they are engaged in their scoring work.

Chief raters must meet specified qualifications and be approved

by DOE. Chief raters must be certified and experienced in the

field of teaching, plus they must have prior experience working

as raters. Chief raters conduct the training sessions to train

raters in the holistic scoring method in Hadley.

23. A chief rater supervises and monitors raters by being

physically present in the same room with the raters while they

are engaged in their s coring work. The chief rater monitors

rater work online in real time. As raters enter scores, those

scores are immediately known by the chief rater, so that any Ðred

flagÑ issues in scoring results and trends can be addressed

immediately.

24. The scores of the two raters assigned to score a GK

essay are added together for the total holistic score. Thus, the

total score range for a GK essay is between two points and 12

points: the lowest possible score of two points would be achieved

if each rater assigns a score of one point; and the highest score

of 12 points would be achieved if each rater assigns six points.

25. The sum of the two ratersÓ scores will be the score that

the GK essay receives unless the ratersÓ scores disagree by more

than one p oint. If the two ratersÓ scores differ by more than one

point, then the chief rater steps in to resolve the discrepancy.

26. After GK essays are scored, the examinee is informed of

the final score of between two and 12 points, and the examinee is

told w hether the score is a passing or failing score. Eight

points is a passing score, according to the FTCE rule.

27. Raters do not develop written comments as part of their

evaluation of GK essays. Their holistic evaluation is expressed

by the point value they assign to the essay.

28. Through the intensive training and the subsequent

calibration and recalibration before each GK essay scoring

assignment, Pearson has achieved consistency in rater scoring of

GK essays that meets industry standards for holist ic scoring.

Consistency in this context means that the scores assigned to a GK

essay by a pair of raters are either identical or adjacent (within

one point), and when adjacent, are balanced (i.e., each rater is

as often the higher scorer as he or she is t he lower scorer). DOE

makes sure that Pearson maintains rater scoring consistency in

accordance with industry standards, by monitoring monthly

performance reports provided by Pearson.

Examinee Perspective: Preparation for the GK Essay

29. DOE provides detailed information and aids on its

website regarding all four subtests of the GK exam, including the

GK essay, for potential examinees. This includes a 39 - page test

information guide for the FTCE GK test.

30. The test information guide sets forth the complete SBE -

adopted competencies and skills to be tested by each of the four

GK subtests, including those specific to the essay test quoted in

Finding of Fact 11.

31. The test information guide explains the GK essay and

scoring process, as follows:

For your essay, you will choose between two

topics. The 50 minutes allotted for this

section of the exam includes time to prepare,

write, and edit your essay.

Your work will be scored holistically by two

raters. The personal views you express will

not be a n issue; however, the skill with

which you express those views, the logic of

your arguments, and the degree to which you

support your position will be very important

in the scoring.

Your essay will be scored on both the

substance and the composition skil ls

demonstrated, including the following

elements: ideas, organization, style

(diction and sentence structure), and

mechanics (capitalization, punctuation,

spelling, and usage).

The raters will use the categories on

page 14 when evaluating your essay. T he

score you receive for your essay will be the

combined total of the two ratersÓ scores.

(R. Exh. 2 at 12 of 39).

32. At the referenced page 14, the test information guide

sets forth in full the scoring rubric used by raters to evaluate

GK essays. Th e rubric is simply a comparative description of the

extent to which an essay demonstrates the competency and skills

to be tested, on a scoring scale of one to six points. The

rubric descriptions differentiate between the various skills to

be tested in a w ay that identifies, as to each skill or group of

skills, which essay is best, better, good, not - so - good, worse,

and worst. But the evaluation of each skill is not separately

scored; instead, the essay response is evaluated as a whole, with

the various str engths and weaknesses weighed and balanced.

33. Finally, the test information guide provides a sample

essay test, with representative essay prompts in the same format

that the examinee will see on the exam: two topics are set

forth, with instructions that the examinee is to select one of

the two topics.

34. The information DOE makes publicly available is

appropriate and sufficient to explain the GK essay exam and

scoring process, and to allow an examinee to know what to expect

in a prompt and what is expected of the examinee in a response.

Score Verification

35. An examinee who fails the GK essay test (or any other

FTCE test or subtest) may request score verification to verify

that the failed exam was scored correctly. The examinee has the

righ t, by statute and rule, to review the test question(s) and

response(s) that received a failing score. The score verification

procedures, providing this review opportunity, are set forth in

the FTCE rule.

36. The score verification rule provides that DOE makes the

determination as to whether an examineeÓs test was scored

correctly. DOE is authorized to consult with field - specific

subject matter experts in making this determination. In practice,

though not required by the FTCE rule, when a score verifica tion

request is directed to the scoring of a GK essay, DOE always

consults with a field - specific subject matter expert known as a

Ðchief reviewer.Ñ

37. Chief reviewers are another category of experts (in

addition to raters and chief raters) retained by Pearson, pursuant

to qualifications identified by DOE, and subject to DOE approval.

Once approved by DOE, prospective chief reviewers undergo the same

rater training in the holistic scoring process as do all other

raters, to gain experience in scoring es says and undergo

calibration to achieve scoring consistency. In addition, chief

reviewers are given training for the chief reviewer role of

conducting review and scoring of essays when scores have been

contested. Unlike raters and chief raters, chief rev iewers do not

work at Pearson in Hadley; they are Florida experts in the field,

with certification and experience teaching in Florida schools.

38. Chief reviewers only become involved with GK essays when

an examinee who failed the GK essay invokes the sc ore verification

process. A chief reviewer is assigned to evaluate whether that

essay was scored correctly. As with the initial scoring, a chief

reviewer is not given any information about the raters or about

the examinee; the essay is assigned a blind, anonymous number.

The chief reviewer conducts the evaluation by first going through

the same step - by - step process as raters, following the same

retraining and calibration steps that involve scoring many sample

essays. Upon achieving success in the cal ibration test, the chief

reviewer moves on to evaluate the assigned essay response

independently, before reviewing the scores the raters gave to that

essay. After reviewing the ratersÓ scores, the chief reviewer

offers his or her view as to whether the es say score should stand

or be changed, and provides a summary rationale for that opinion.

This information is conveyed to DOE, which determines the action

to take -- verify or change the score -- and notifies the examinee of

the action taken.

39. In the 14 - m onth period from January 2016 through

February 2017, two failing GK essay scores were changed by DOE to

passing scores as a result of the score verification process. As

with the overall passage rates, Petitioner characterizes this

reversal rate as low, bu t no evidence is offered to prove that

characterization. It is as reasonable or more reasonable to infer

from the fact that GK essay scores are only rarely reversed

through score verification that the scoring process works well.

PetitionerÓs GK Essay Atte mpts

40. Petitioner took the GK essay test for the first time in

July 2015. He received a failing score of four, with two points

assigned by each of the two raters. Petitioner admits that he

did little to nothing to prepare for the GK essay the first ti me.

When taking the essay test, he ran out of time and recalls that

he left the essay incomplete. The time pressure Ðhad a huge deal

with me not being able to provide enough specifics for it to make

any sense at all where I was going with the essay.Ñ (T r. 75).

Petitioner thought the passing score was six at the time, but his

recollection is incorrect. The higher passing score of eight has

been in place since January 2015, and has been the passing score

for each of PetitionerÓs GK essay attempts.

41. F TCE examinees can retake failed subtests/sections, and

need only retake the parts failed. There are no limits on the

number of retakes. The requirements for retakes are that at

least 30 days must have elapsed since the last exam attempt, and

that examinee s pay the registration fees specified in the FTCE

rule for each retake of a failed subtest and/or section.

42. Petitioner retook the GK essay test in February 2016.

In preparation for this second attempt, Petitioner did not seek

tutoring or spend much ti me training. As he explained, ÐIÓm

under the impression that I can write an essay.Ñ (Tr. 21).

Instead, he focused mostly on preparing for the timed aspect of

the exam, making sure that he started when the clock started.

Although his score improved from four to six, it was still a

failing score.

43. Petitioner did not invoke the score verification

process to question the failing scores he received on his first

two GK essays. Those two failing scores stand as final, as he

did not challenge them.

44 . Petitioner took the GK essay test for the third time on

June 25, 2016. This time, he prepared to some extent. In the

month before the exam, Petitioner sought help from someone he

described as a writing coach. The writing coach did not evaluate

Petiti onerÓs writing so as to identify weaknesses; instead, she

asked him what he thought his weaknesses were, and he responded

that he did not know what his weakness is besides not being able

to formulate his plan and map out his essay faster. As a result,

she coached him on some mapping techniques, and on how to

structurally organize an essay -- with an introduction, followed by

three points in paragraphs begun with transitional phrases, and a

conclusion. Petitioner practiced a little with his writing

coach, by email: she would send a prompt and he would write an

essay, which he timed, and then send it back to her. They did

this Ða few times.Ñ (Tr. 24). There is no evidence of record

regarding the writing coach, other than that her name is

Ms. Martin. She may have been PetitionerÓs proposed witness who

was allowed to appear from New York by telephone, but who was not

called to testify.

45. One of the things Petitioner learned from Ms. Martin

was that in his introduction, he should Ðspeak vaguely aboutÑ

wh at will be covered. When asked if Ms. Martin actually said to

be ÐvagueÑ in the beginning, Petitioner said, ÐShe may not have

used the word vague, but that is the meaning that I got from what

she said.Ñ (Tr. 70).

46. In preparation for his third attem pt at the GK essay

test, Petitioner also sought help from Jordan Gibbs, who was

described as an educator who taught language arts for over 20

years. Petitioner testified that Mr. Gibbs is Ðour academy

leader there[.]Ñ (Tr. 24). However, Petitioner did n ot

elaborate; it is unknown which academy is led by Mr. Gibbs , or

where ÐthereÑ is . Like Ms. Martin, Mr. Gibbs also addressed

mapping techniques with Petitioner. Petitioner never sent any

essay drafts to Mr. Gibbs for his review.

47. Petitioner also re viewed GK essay preparation material

on the DOE website. He reviewed sample prompts, but did not

practice writing complete essays. He just looked at the sample

prompts for purposes of mapping and planning an essay.

Petitioner said that he found the prep aration material useful to

an extent, but did not think the sample prompts reflected the

type of GK essay prompts in use when he took the test. A

comparison of the sample GK essay prompts in the test information

guide (R. Exh. 2 at 17 of 39) with the actu al GK essay prompt

Petitioner chose for his essay topic (Jt. Exh. 1 at 3 of 4)

suggests otherwise. Although DOE obviously does not make

available as samples the actual essay prompts actively being used

in GK examinations, the sample prompts appear to be s imilar to

PetitionerÓs actual prompt in style, substance, and tone. It

would be unreasonable for examinees to expect more from a testing

agency than what DOE makes available.

48. PetitionerÓs score improved slightly in his third

attempt at the GK essay test, but it was still a failing score of

seven. One rater assigned the essay a score of three, while

another rater scored the essay a four.

49. Each of the three times Petitioner took the GK essay

test, the two raters assigned scores that were consi stent, in

that they were either identical or adjacent (within one point of

each other). Accordingly, a chief rater was never assigned for

discrepancy resolution, as there were no discrepancies.

50. After receiving notification of his third failing

score , this time Petitioner invoked the score verification

process. Petitioner completed a statement explaining why he

believes his score was erroneous, which is in evidence as part of

the confidential testing material. (Jt. Exh. 1 at 2 of 4). The

statement set forth why he believes the essay demonstrated good

organization, used transitional phrases, and addressed the topic .

He acknowledged one misspelling, and acknowledged that his

conclusion ended in mid - sentence, as he ran out of time. He

added three wor ds to complete the last sentence, and suggested

that the ending should have been inferred from what he did say.

51. DOE conducted its review, and the score was verified

through a process consistent with DOEÓs practice of consulting a

chief reviewer who wa s qualified as a subject matter expert in

the field of teaching in Florida and approved by DOE.

52. The chief reviewer who undertook to verify PetitionerÓs

essay score conducted an independent evaluat ion of PetitionerÓs

essay following the same holistic m ethod. Then the chief

reviewer considered the scores separately assigned by the two

raters who scored PetitionerÓs essay. She concluded that the

assigned scores of three/four should stand. The chief reviewer

provided a summary rationale for her determin ation, offering her

view that the essay borders on a three/three due to weak

development. 4/

53. The chief reviewerÓs summary was provided to DOE for

consideration. By letter dated September 27, 2016, Petitioner

was notified by DOE that the Ðessay score t hat you questioned has

been reviewed by a Chief Reviewer. As a result of this review,

the Department has determined that the written performance

section that you questioned is indeed scored correctly.Ñ

Petitioner was notified of his right to an administr ative hearing

pursuant to sections 120.569 and 120.57 to dispute the decision.

Petitioner availed himself of that opportunity, and was given the

chance in a de novo evidentiary hearing to present evidence to

support his challenge to his exam score.

54. A t the hearing, Petitioner offered only his own

testimony as support for his challenge to the scoring of his

essay. Petitioner was not shown to be, tendered as, or qualified

as an expert in either formal college - level English writing or

scoring of essays. His attempt to compar e isolated parts of the

rubric to isolated parts of his essay is contrary to the holistic

scoring approach used to score the GK essay. Petitioner offered

no comprehensive, holistic evaluation of his essay as a whole,

nor was he shown to be qualified to do so.

55. Besides being contrary to the holistic scoring method,

PetitionerÓs critique of the scoring of his essay was wholly

unpersuasive. Without undermining the confidentiality of the

ingredients of PetitionerÓs testimony (the e ssay prompt, his

essay, and the historic anchors), overall, the undersigned did

not find PetitionerÓs critique credible or accurate. Although

awkward to try to explain in code, some examples follow to

illustrate the basis for this overall finding.

56. Petitioner began his critique by reading the first

three sentences -- the introductory paragraph -- of his essay. He

said that each sentence had one topic, and that each of the

subsequent three paragraphs in the body addresses one of those

three topics. The problem with PetitionerÓs explanation for the

substantive organization of his essay is that the essay prompt

identifies a single topic, not three topics. Petitioner failed

to respond to the promptÓs single topic by introducing that topic

as the essayÓs th eme, and developing that single theme in the

body of that essay. Similarly, the concluding paragraph offers

scatter ed thoughts, somewhat related to the three topics

discussed in the essay. The essayÓs weakness in development was

a prominent point in the scoring rationale summaries written by

the raters and chief reviewers.

57. Petitioner specifically addressed only one aspect of

the rubric considerations, addressing the extent to which an

essay has errors in sentence structure, usage, and mechanics. As

to this consideration, Petitioner stated that there were three

spelling errors in his essay (up from the one error he identified

in his score verification statement). He was critical of one

raterÓs comments for referring to grammatical errors, because

Petitioner does not believe there were any grammatical errors in

his essay. PetitionerÓs assessment of his essay reflects his

bias, because it fails under any objective analysis.

58. In fact, PetitionerÓs essay (Jt. Exh. 3) has both

spelling errors and grammatical errors. In addition, the essay

uses poor sentence structure in several instances, as well as

poor word choices that interfere with an understanding of what

Petitioner means. A n example of a sentence with a grammatical

error is the fifth sent ence in paragraph 4. At the very least,

the word ÐhavingÑ is required after the comma. With that

addition, the sentence would only be awkward, instead of

grammatically incorrect.

59. An example of a poorly written sentence is the second

sentence of the second paragraph. This sentence combines a

misspelling, a misused word, and syntax that is awkward, at best.

60. Petitioner must also acknowledge that the last sentence

of his essay is another example of poor sentence structure, since

it is an incompl ete sentence without punctuation. It would be

inappropriate for raters reviewing essays to fill in the gaps

left by writers, whether those gaps were because of running out

of time or otherwise. What Petitioner meant to write to complete

the sentence is n ot something that can be added after - the - fact to

cure the defect on the face of the essay.

61. By the undersignedÓs count, there are five misspellings

in the essay, unless one counts Ðin to,Ñ which should be Ðinto,Ñ

as an error of grammar or syntax. The other misspellings were:

easire (easier); savy (savvy); yeild (yield); and evironment

(environment). In addition, Petitioner made several punctuation

errors, failing to hyphenate two compound adjectives preceding

nouns and presenting a single idea: cut ting - edge technology;

tech - savvy students. Petitioner also improperly omitted a hyphen

in Ðself discipline.Ñ

62. Petitioner acknowledged some repetitive use of a

particular word, but thought he only used that word twice. In

fact, he used the word in bo th sentences one and two of the

second paragraph, and then again in paragraph four. Only the

first usage is arguably correct (but in an awkwardly written

sentence). While used once, the word is an interesting one,

PetitionerÓs overuse and misuse of this word suggests a

mechanical, as opposed to thoughtful, approach of injecting

interesting words into the essay.

63. PetitionerÓs essay demonstrated good superficial

structure, with an introductory paragraph, three paragraphs in

the body that begin with good transitional words, and a

concluding paragraph. The organizational structure may have

earned Petitioner a score of four, as stated in that raterÓs

comments, but that same rater also repeated the comments of

others that where the essay is weakest is i n development.

64. Petitioner offered his view that the only reason his

essay received a failing score was because the raters considered

it to be too short in length. While Petitioner is correct in

noting that length is not a criterion, he mischaracteri zed the

comments on this subject, by ignoring the criticisms of his essay

that were made when the length of the essay was noted. The

comments only mention the length of PetitionerÓs essay as it

correlates to other considerations, such as the weakness in

d evelopment, the lack of specifics or examples, or the impact of

a Ðnumber of misspellings, . . . usage issues, . . . and

punctuation errors,Ñ which accumulated to a notable level Ðgiven

the shortness of the response.Ñ (Jt. Exh. 5 - A). Petitioner

failed to prove his contention that an unauthorized criterion --

essay length alone -- was applied in scoring PetitionerÓs essay.

65. Petitioner failed to prove that the holistic scoring of

his essay was incorrect, arbitrary, capricious, or devoid of

logic and reason. He offered no evidence that a proper holistic

evaluation of h is essay would result in a higher total score than

seven; indeed, he offered no holistic evaluation of h is essay at

all. PetitionerÓs critique of various parts in isolation did not

credibly or effectively prove that his score of seven was too

low; if anything, a non - expertÓs review of various parts in

isolation could suggest that a score of seven would be generous.

But that is not the scoring approach called for here.

66. Petitioner present ed no evidence that any aspect of the

GK essay process overall, including development, administration,

evaluation, and score review, was arbitrary, capricious, unfair,

discriminatory, or contrary to requirements imposed by law.

CONCLUSIONS OF LAW

67 . Th e Division of Administrative Hearings has

jurisdiction over the parties and subject matter, pursuant to

sections 120.569 and 120.57(1), Florida Statutes.

68. Petitioner has the burden of proving by a preponderance

of the evidence that he is entitled to th e relief he seeks. See

DepÓt of Transp. v. J.W.C. Co. , 396 So. 2d 778 (Fla. 1st DCA

1981); § 120.57(1)(j), Fla. Stat.

69. As the one who has failed the essay component of a

certification exam, Petitioner shoulders a heavy burden to prove

that the subject ive evaluation of his exam by Pearson raters, who

are experts in the field, is arbitrary and capricious. Harac v.

DepÓt of ProfÓl Reg. , 484 So. 2d 1333, 1338 (Fla. 3d DCA 1986);

State ex rel. Glaser v. Pepper , 155 So. 2d 383, 384 (Fla. 1st DCA

1963); Stat e ex rel. Topp v. Bd. of Elec. Examiners , 101 So. 2d

5832, 586 (Fla. 1st DCA 1958).

70. In Harac , an applicant seeking licensure as an

architect successfully challenged the failing grade received on

the design portion of the exam, because of unique circu mstances

established in the administrative hearing. In particular, it was

shown in the hearing that one of the three graders did not follow

the holistic scoring method described in the design test

handbook, and instead, gave a score of one, which all part ies

agreed was invalid. As the court noted, a score of one would

only have been proper if the design solution was incomplete,

which everyone agreed was not the case. Therefore, the invalid

grade had to be thrown out. In the administrative hearing, two

e xpert witnesses provided testimony as to their evaluations of

the design and the grades they would assign. One of the experts

used the holistic method and followed the original grading

procedures as closely as possible without reconvening the

original gra ders; this expert assigned a passing grade. The

other expert did not evaluate the examineeÓs design in accordance

with the holistic method or approved procedures, but offered his

opinion that the design should earn a failing grade. The grade

assigned by the expert who used the holistic method and followed

the approved procedures was accepted as substituting for the

admittedly invalid grade, and licensure was approved.

71. In marked contrast to Harac , there was no proof in this

case that either of the tw o ratersÓ scores was invalid, contrary

to PearsonÓs scoring procedures, or improper in any way. Without

such a showing, arguably it would be inappropriate to reach the

second level of Harac where, under the unique circumstance of an

admittedly invalid gra de, expert testimony was accepted to

regrade the design test by following the holistic grading method

and approved procedures, to substitute for the invalid grade.

See, e.g. , The Florida Bar Re Williams , 718 So. 2d 773, 778 - 779

(Fla. 1998) (in a certifica tion examineeÓs challenge to the

credit given on two essay answers, the Court refused the

invitation to regrade the essays and award a higher score,

Ðabsent clear and convincing allegations establishing fraud,

imposition, discrimination, manifest unfairnes s, or arbitrary or

capricious conduct.Ñ) . In this de novo hearing, Petitioner had

his opportunity to prove fraud, imposition, discrimination,

manifest unfairness, or arbitrary or capricious conduct.

Petitioner failed to meet his burden of proof in this r egard.

72. If it were appropriate to reach the second level of

Harac , PetitionerÓs proof would fall well short of the necessary

showing to sustain his score challenge. Unlike the examinee in

Harac , Petitioner failed to offer expert testimony by an expert

in holistic scoring and/or an expert teacher to attempt to

replicate as closely as possible the holistic scoring method used

by Pearson to score PetitionerÓs exam. PetitionerÓs non - expert

testimony was far off the mark; as found above, he did not

underta ke an overall evaluation in accordance with the holistic

scoring method. H is attempted non - holistic , self - serving

critique of the score given to his essay was wholly unpersuasive.

73. To the extent Petitioner contends that his challenge

should succee d because scoring essay examinations are, by their

nature, subjective, that contention is rejected. The fact that

subjectivity plays some role in the scoring process is not,

standing alone, a basis upon which to overturn the results. That

is particularly true where, as here, the unrebutted evidence

showed that the scoring process in place is not only designed to

minimize subjectivity, but that it actually functions that way.

Instead, as shown by Harac and cases cited therein, to prevail,

Petitioner was r equired to also prove that those who subjectively

evaluated his examination acted arbitrarily or without reason or

logic in giving him a failing score. Petitioner failed to meet

his burden of proof in this regard.

74. Petitioner hints at criticism direc ted to the SBE for

making the policy decision to toughen the certification standards

by raising the score required to pass the GK essay. As noted,

that choice, codified in the FTCE rule, was the SBEÓs prerogative

and is not a matter subject to debate in t h is proceeding.

Moreover, raising the standards for writing skills required to

pass the GK essay was appropriate to align the FTCE with SBE -

adopted student standards, which have increased the focus on, and

raised the expectations for, student achievement in writing. See

§ 1012.56(9)(f), Fla. Stat.

75. Petitioner failed to prove his contention that the

passage rates on the GK essay have been ÐlowÑ since the passing

score was raised. The fact that fewer GK essay examinees are

passing is the expected cons equence of the SBEÓs policy choice,

codified in the FTCE rule, to increase the passing score

requirement. Thus, while the passage rate was shown to be lower,

there was no proof that the recent scores are Ðlow,Ñ as opposed

to the prior scores having been t oo high. The passage rates,

standing alone, do not provide grounds for invalidating

PetitionerÓs essay score.

76. Similarly, PetitionerÓs challenge to his essay score is

not aided by the fact that the score verification process rarely

results in a change from a failing score to a passing score. The

score change rate, standing alone, does not establish that the

score verification process is arbitrary, capricious, unfair,

discriminatory, or otherwise improper.

RECOMMENDATION

Based on the foregoing Findings of Fact and Conclusions of

Law, it is RECOMMENDED that a final order be entered rejecting

PetitionerÓs challenge to the failing score he received on the

General Knowledge essay test taken in June 2016, and dismissing

the petition in this proceeding.

DONE AND ENTERED this 13th day of October , 2017 , in

Tallahassee, Leon County, Florida.

S

ELIZABETH W. MCARTHUR

Administrative Law Judge

Division of Administrative Hearings

The DeSoto Building

1230 Apalachee Parkway

Tallahassee , Florida 32399 - 3060

(850) 488 - 9675

Fax Filing (850) 921 - 6847

www.doah.state.fl.us

Filed with the Clerk of the

Division of Administrative Hearings

this 13th day of October , 2017 .

ENDNOTE S

1/ References herein to Florida Statutes are to the 2017

codifi cation unless otherwise provided. Any amendments to the

applicable substantive and procedural statutes in effect at the

time Petitioner took his exam and at the time the hearing was

held appear inconsequential; the relevant law addressed in this

proceedin g was not changed.

2/ The transcript portions from the hearing in DOAH Case

No. 17 - 0423 that were adopted by reference have not been

duplicated, but the following portions should be considered part

of the record of this case: Volume I, pages 167 throug h 204

(Michael Grogan); Volume II, pages 219 through 257 (Phil Canto);

and Volume II, pages 296 through 316 (Mary Jane Tappen).

3/ CLAST scores used to be accepted to demonstrate mastery of

general knowledge for purposes of teacher certification .

Use o f CLAST scores was eliminated as of July 1, 2002, and

instead, the Legislature directed the SBE to develop a Ðbasic

skills examination,Ñ which was the precursor to the GK four - part

exam , to substitute for CLAST scores as the means to demonstrate

mastery of general knowledge . See § 1012.56(3)(a) and (b), Fla.

Stat. (2002). CLAST scores earned prior to July 1, 2002, are

still accepted in lieu of passing scores in the corresponding

subtests of the GK exam.

4/ In this case, after Petitioner contested DOEÓs determination

that h is essay was scored correctly, DOE asked the original

raters to prepare written justifications for their scores. In

addition, although DOEÓs practice in the score verification

process is to have one chief reviewer prepare written comme nts to

explain why the original score should stand or why it should be

changed, in this case, after Petitioner had contacted DOE

regarding a possible request for an administrative hearing, DOE

had a second chief reviewer conduct an additional review and

pr epare written comments. Only the second chief reviewer

testified at hearing, but her testimony was more about the

process followed by chief reviewers in conducting score

verification reviews, and she did not specifically address her

written comments. The original raters did not testify. All of

the written comments are in evidence under seal. (Jt. Exhs. 5

and 6). The written comments were utilized at hearing only by

Petitioner in his critique of his essay score, comparing parts of

the comments with part s of his essay, but not doing so

effectively or persuasively.

COPIES FURNISHED:

Daryl Bryant

3607 Brophy Boulevard

Cocoa, Florida 32926

Bonnie Ann Wilmot, Esquire

Department of Education

Suite 1244

325 West Gaines Street

Tallahassee, Florida 32399

(eS erved)

Jennifer Diane Rose, Esquire

Post Office 924

Melbourne, Florida 32902

(eServed)

Darby G. Shaw, Esquire

Department of Education

Suite 1244

325 West Gaines Street

Tallahassee, Florida 32399

(eServed)

Matthew Mears, General Counsel

Department of Education

Turlington Building, Suite 1244

325 West Gaines Street

Tallahassee, Florida 32399 - 0400

(eServed)

Pam Stewart, Commissioner of Education

Department of Education

Turlington Building, Suite 1514

325 West Gaines Street

Tallahassee, Florida 32399 - 0 400

(eServed)

Chris Emerson, Agency Clerk

Department of Education

Turlington Building, Suite 1520

325 West Gaines Street

Tallahassee, Florida 32399 - 0400

(eServed)

NOTICE OF RIGHT TO SUBMIT EXCEPTIONS

All parties have the right to submit written except ions within

15 days from the date of this Recommended Order. Any exceptions

to this Recommended Order should be filed with the agency that

will issue the Final Order in this case.