98-002429 Clasina Vanthul vs. Department Of Environmental Protection Status: Closed Recommended Order on Monday, November 16, 1998.

Summary: Petitioner and agency both applied "rounding" rule incorrectly. No proof of pilot or trick questions. Petitioner was not short-changed on time. Agency entitled to delete flawed questions without giving additional credit.

STATE OF FLORIDA

DIVISION OF ADMINISTRATIVE HEARINGS

CLASINA VANTHUL, )

)

Petitioner, )

)

vs. ) Case No. 98-2429

)

DEPARTMENT OF ENVIRONMENTAL )

PROTECTION, )

)

Respondent. )

_________________________________)

RECOMMENDED ORDER

Upon due notice, this cause came on for formal hearing on

September 11, 1998, in Tallahassee, Florida, before Ella Jane P.

Davis, a duly assigned Administrative Law Judge of the Division

of Administrative Hearings.

APPEARANCES

For Petitioner: Gerrit Vanthul, as Qualified Representative

5 279 Southeast 39th Street

Trenton, Florida 32693

For Respondent: Cynthia Christen, Esquire

Department of Environmental Protection

3900 Commonwealth Boulevard

Mail Station 35

Tallahassee, Florida 32399-3000

STATEMENT OF THE ISSUES

(1) Is Petitioner entitled to credit for her answers to any

of the questions she specifically challenged or for the four

questions deleted by the Department of Environmental Protection

(Department) on the February 1998 Class "B" Domestic Wastewater

Operator Certification examination (wastewater examination)?

(2) Was the Department's administration or grading of

Petitioner's examination arbitrary, capricious or otherwise

unfair so as to entitle Petitioner to either additional points

for a passing grade or an opportunity to retake the examination

without cost?

PRELIMINARY STATEMENT

By letter of April 6, 1998, the Department notified

Petitioner that she had failed the February 1998 wastewater

examination. Petitioner timely requested an administrative

hearing by a letter dated May 5, 1998. The Department referred

the matter to the Division of Administrative Hearings on or about

June 6, 1998. At the time of its referral, the Department had

already received another letter from Petitioner dated May 26,

1998. The second letter alleged additional disputed issues of

material fact. The Department never forwarded the May 26, 1998,

letter to the Division.

On June 30, 1998, an Order of Prehearing Instructions and

Notice of Hearing for September 11, 1998, was mailed to both

parties.

At the commencement of the disputed fact hearing, the

Department stipulated that Petitioner's second letter could be

treated as an amended petition and that the Department had

adequate notice of the additional disputed issues of material

fact included in Petitioner's second letter. Accordingly, the

case went forward on the merits of all issues raised.

After appropriate examination on the record, and without

objection by the Department, Gerrit Vanthul, Petitioner's

husband, was accepted as Petitioner's Qualified Representative

for purposes of this proceeding.

Respondent Department had timely complied with the Order of

Prehearing Instructions. Petitioner had not complied with the

Order of Prehearing Instructions. Nonetheless, Petitioner was

permitted to present documentary evidence and her sole witness.

Petitioner presented the oral testimony of Gerrit Vanthul

and had eight exhibits admitted in evidence. Part of

Petitioner's Exhibit 1 was not admitted, and Petitioner's Exhibit

9 was not admitted. Petitioner did not testify on her own

behalf.

Respondent Department presented the oral testimony of Greg

Dawkins, Mary Smith, Mary Patt Peterson, Gary Peterson, Ph.D.,

and William Allman. Greg Dawkins was accepted as an expert in

emergency response and community right-to-know. Dr. Gary

Peterson was accepted as an expert in examination development and

psychometrics. William Allman was accepted as an expert in the

operation of wastewater treatment plant operations. Respondent

had nine exhibits admitted in evidence.

A transcript was filed in due course, and all timely filed

proposed recommended orders have been considered in the

preparation of this recommended order.

FINDINGS OF FACT

1. Petitioner took the Class "B" wastewater operator

certification examination given in February 1998 by the

Department of Environmental Protection.

2. Pilot questions are questions submitted by licensees and

educators which do not yet have a "performance record" of testing

validity. Petitioner alleged that pilot questions were used on

her examination contrary to rules of the Department. Petitioner

submitted no competent evidence to establish this allegation and

it was credibly refuted. There are no pilot questions in the

bank of potential questions from which the examination was

composed.

3. Petitioner conjectured that questions on her "B" level

examination may have been drawn from a bank of questions for a

higher level ("A") certification examination. In fact, the

examination questions for the "B" level examination were selected

from a bank of questions developed by the Department of Business

and Professional Regulation. The Department of Business and

Professional Regulation was the agency that had jurisdiction over

the operator certification program before the Department of

Environmental Protection assumed responsibility therefor. The

selection of the examination questions was accomplished by

selecting the percentage of questions from a range in a subject

area already predetermined by rule and a computer program

inserting the number of questions to fill that percentage. There

is no way the computer program can select questions from another

level of examination, for instance "A" level or "C" level.

4. Prior to the examination, candidates for examination

were advised they would have four hours to complete the

examination. Examinees for the February 1998 examination in fact

were provided four full hours after all preliminary matters and

reading of instructions were completed.

5. Prior to the examination, the Department provided

candidates for examination with a list of subject areas that it

intended to place on the examination, so that candidates could

effectively prepare. All subject areas, except one, were in fact

covered on the February 1998 examination. In some instances, a

single question/answer satisfied two or more subject areas

because of content equally applicable to each subject area. In

other instances, the same subject area was covered by several

questions/answers. Only one subject area that was listed in the

pre-examination information did not appear on the February 1998

examination. That subject area was "energy." The reason that

the subject area of "energy" was not included on the February

1998 examination was that there were no energy questions in the

bank of questions which the Department of Environmental

Protection had inherited from the Department of Business and

Professional Regulation.

6. The sole result of the absence of any energy question on

the examination is that Petitioner and all other examinees in her

group did not receive as thorough an examination in a single

subject matter area as the licensure board had aspirationally

intended. However, all examinees were equally treated.

7. Originally, there were 87 questions on the February 1998

examination. After the examination was administered and a

special analysis report on the grades was produced, the

Department's examination review committee met with the

examination consultants. The committee recommended to the

Department, and the Department accepted the recommendation, that

four questions should be deleted because they were misleading or

for some other reason failed to adequately and reliably measure

the examinees' ability to practice at a Class "B" license level.

Removal of the four questions only lowered the mean score by one

point, thereby creating a slightly easier examination while

simultaneously slightly increasing its reliability.

8. Examinees were instructed to select the best multiple

choice answer for each question. Each of the questions was

equally weighted.

9. The Department interpreted Rule 61E12-41.005(5), Florida

Administrative Code , as requiring that examinees achieve at least

a 65% rounded score on the examination in order to pass the

examination.

10. In order to determine an examinee's success on the

examination, the Department multiplied the initial 87 questions

by 65% (.65) and so determined that an examinee would need at

least 54 correct questions/answers to earn a passing grade. In

determining a candidate's grade on an operator licensure

examination, the Department determines the number of correct

answers needed to reach the minimum rounded score of 65%. A

special analysis report also indicates how many correct answers

equal each percentage grade. If this number is not a whole

number, the Department uses the rounding method to reach a whole

number, based on 0.5 percentage.

11. By the foregoing grading interpretation, before

deletion of the four questions, Petitioner's rounded score was

60%, with 52 correct answers. Petitioner's grade improved with

the deletion of the four questions, because she had incorrectly

answered each of the four questions which were later deleted.

After the four questions were deleted, the same grading system

resulted in a rounded score of 63% with 52 correct answers.

12. By letter dated April 6, 1998, the Department notified

Petitioner that she had failed the examination because she did

not get a rounded 65% score based on 52 correct answers.

13. After receiving the letter, Petitioner requested a

review of the examination. Petitioner was allowed to review the

questions and answers she had missed. Petitioner was also

allowed to write comments on the question sheets which she

reviewed.

14. Petitioner's comments were submitted to the examination

review committee of the Department for the committee's review.

Upon review of Petitioner's comments and the examination, the

committee determined that the questions and required answers were

accurate and fair. It recommended no change to Petitioner's

score. Petitioner was notified that no change would be made to

her score.

15. Petitioner then timely requested an administrative

hearing. Although Petitioner's two letters/petitions (see

Preliminary Statement supra .) initially raised issues concerning

a number of examination questions, Petitioner only presented

evidence concerning the contents of question 78 at formal

hearing.

16. Question 78 tested examinees' knowledge of appropriate

emergency response activity and notification concerning the

release of chlorine gas.

17. Petitioner asserted that question 78 was vague,

ambiguous, and misleading because it did not specifically state

that a "reportable quantity" was to be considered in choosing the

best answer from among multiple choice options of reporting a

chlorine spill to one entity, two entities, three entities or no

entities. For this reason, Petitioner alleged that her answer

could have been an answer which was equally correct ("multi-

keyed") with the answer selected as correct by the Department.

18. Mr. Dawkins, who was accepted as an expert in emergency

response and community right-to-know, testified that the question

was not misleading. Mr. Dawkins is not associated with the

Respondent Department, any of its committees, or the examination

preparation process. He oversees actual reportage of dangerous

chemical spills for the Department of Community Affairs.

Although Mr. Dawkins indicated that he, personally, would not

have written question 78 quite the way it was posed on the

examination, he still felt that since it addressed reporting

requirements, examinees should have assumed that a reportable

quantity was involved and answered accordingly.

19. All three of Respondent's experts testified that the

answer chosen as correct by the Department was the most accurate

of the multiple choice answers provided on the examination and

that the subject matter and correct answer should have been

understood by a qualified operator of a wastewater treatment

plant at the "B" licensure level.

20. The Department has under contract an expert in

examination and psychometrics. The Evaluation Services

Instructional Support Center Learning Systems Institute of

Florida State University provides to the Department as part of

the examination grading, a special analysis report for each

examination. This report contains statistics about the scores,

difficulty of each question, and how the spread of answers by the

examinees compared to the four quadrants of grade results.

21. The February 1998 examination was an extremely

difficult examination, as evidenced by the fact that more

examinees failed than passed. However, it was demonstrated that

77% of examinees who took the examination got question 78

correct. Question 78 also discriminated between high and low

scoring examinees.

22. The item analysis performed before the other four

questions were deleted did not show that question 78 was

misleading in any way, but did show that each of the four

questions deleted were misleading or otherwise flawed.

23. One of the proctors for the February 1998 examination

personally observed that at the time the examination ended, only

two examinees remained in the examination room and that neither

of these examinees was Petitioner. It can be inferred therefrom

that Petitioner had finished the examination, had time to spare,

and had left the room.

24. Finally, the inclusion of examination questions which

were later deleted is not a concern as to the time allotted.

This type of examination is a "power exam" and speed is not a

factor.

CONCLUSIONS OF LAW

25. The Division of Administrative Hearin gs has

jurisdiction over the parties and subject matter of this cause,

pursuant to Section 120.57(1), Florida Statutes.

26. The burden of proof and duty to go forward is upon the

Petitioner in this cause. Petitioner must show by a

preponderance of the evidence that the examination was faulty,

arbitrarily or capriciously worded or graded, or that Petitioner

was arbitrarily or capriciously denied credit through a grading

process devoid of logic or reason. Harac v. Department of

Professional Regulation , 484 So. 2d 1333, 1338 (Fla. 3rd DCA

1986); State ex rel Glaser v. J.M. Pepper , 155 So. 2d 383 (Fla.

1st DCA 1963).

27. For the regulation of operators, the Department of

Environmental Protection is entitled to use rules developed by

the Department of Business and Professional Regulation until the

Department of Environmental Protection promulgates its own rules,

as set forth in Section 17 of Chapter 97-236, Laws of Florida .

28. Rule 61-11.010(1)(b), Florida Administrative Code ,

provides as follows:

(1) Pursuant to Section 455.217, Florida

Statutes, grading of all examinations shall

be processed only as follows:

* * *

(b) Departmentally developed objective,

multiple choice examination shall be graded

by the Department or its designee. After an

examination has been administered the Board

shall reject any questions that do not

reliably measure the general areas of

competency specified in the rules of the

Board. The Department shall review the item

analysis and any statistically questionable

items after the examination has been

administered. Based upon this review, the

Department shall adjust the scoring key by

totally disregarding the questionable items

for grading purposes or by multi-keying,

giving credit for more than one correct

answer per question. All questions which do

not adequately and reliably measure the

applicant's ability to practice the

profession shall be rejected. The Department

shall calculate each candidate's grade

utilizing the scoring key or adjusted scoring

key, if applicable, and shall provide each

candidate a grade report.

29. Therefore, Petitioner had no right to have the four

deleted questions scored as part of her examination.

30. Petitioner failed to establish that she suffered any

disadvantage as a result of having too little time to complete

the examination. Her theory that she would have achieved more

correct answers and thus a higher percentage score if she had

spent less time or no time on the four questions which were

ultimately deleted and more time on the remaining 83 questions

which ultimately were counted, is purely speculative and is

belied entirely by the fact that she left the examination room

before the entire time allotted for taking the examination had

elapsed.

31. Petitioner's theory that she would have done better on

other subject matter areas of the examination if she had not

spent time studying the subject matter of "energy" which was not

included in the examination also is pure speculation. Her

assertion that she was somehow prejudiced by not being examined

at all in the subject area of energy also is purely speculative

and does not indicate any arbitrary, capricious, or prejudicial

treatment by the Department under the facts of this case.

32. Petitioner's theory with regard to the correct answer

on question 78 was not proven, and clear, competent, and

substantial evidence was presented by Respondent to demonstrate

that question 78 was not a pilot question or a "trick" question,

or otherwise arbitrary or capricious. Moreover, clear and

convincing evidence (not just preponderant evidence) was produced

by the Department to show that Petitioner did not select the best

multiple choice answer for question 78. Question 78 was

incorrectly answered by Petitioner. Petitioner's answer to

question 78 was not "multi-keyed" because to do so would have

given her credit for a wrong answer The people of Florida would

be best protected by licensing at the Class "B" level only

persons with the required skills, education, training, and

experience necessary to be able to correctly answer question 78.

33. Petitioner's proposed calculation of her score as set

out in various ways in the testimony of Mr. Vanthul, several

exhibits, and Petitioner's proposed recommended order, is not in

accord with the clear requirements of Rule 61E12-41.005(5),

Florida Administrative Code.

34. The Department's methodology of scoring as set forth in

the Findings of Fact, supra , also does not follow the precise

language of the rule. The Department applied the percentage

backwards to determine the number of correct questions/answers

each examinee would need to equal 65%, instead of dividing the

number of possible correct answers into each respective

examinee's number of actual correct answers so as to determine

each examinee's exact correct percentage score and then rounding.

However, this makes no difference because the Department got the

correct percentages.

35. Even when the clear language of the rule is literally

applied, Petitioner has failed to make a minimum passing score of

65%.

36. Rule 61E12-41.005(5) reads:

Examination answer sheets shall be

electronically scored. The minimum passing

score on the examination is 65%. In rounding

percentages, any percentage which is 0.5 or

above shall be rounded up to the next higher

whole number. Percentages less than 0.5

shall be rounded to the next lower whole

number.

37. If the clear language of the rule is literally applied,

the following calculations are made: Originally, there were 87

equally-weighted questions on the examination, and Petitioner got

52 correct answers. 52 correct answers out of 87 possible

correct answers constitutes an exact percentage score of 59.77%.

Using the "rounding" rule as written, this would have rendered

Petitioner's score as 60%. Petitioner had incorrectly answered

the four deleted questions, so their removal from consideration

gave her 52 correct answers out of 83 possible correct questions.

52 out of 83 constitutes an exact percentage score of 62.65%,

which rendered Petitioner's score, according to a literal

application of the "rounding" rule, as 63%, or two percent short

of a minimum passing rounded score of 65% on the examination.

38. The examination was not faulty. All candidates' grades

were calculated in the same manner. No one was given credit for

any of the four deleted questions. Neither the wording nor the

grading of those questions were arbitrary or capricious. The

Department's refusal to credit Petitioner for correct answers on

the four deleted questions or for the one specifically challenged

question 78 was not devoid of logic or reason.

RECOMMENDATION

Upon the foregoing findings of fact and conclusions of law,

it is

RECOMMENDED that the Department of Environmental Protection

enter a Final Order denying the Petitioner's challenge to the

February 1998 Class "B" wastewater operator certification

examination and assigning her a final percentage grade of 63%

thereon.

DONE AND ENTERED this 16th day of November, 1998, in

Tallahassee, Leon County, Florida.

_____________________ ______________

ELLA JANE P. DAVIS

Administrative Law Judge

Division of Administrative Hearings

The DeSoto Building

1230 Apalachee Parkway

Tallahassee, Florida 32399-3060

[phone] SUNCOM 278-9675

Fax Filing [phone]

Filed with the Clerk of the

Division of Administrative Hearings

this 16th day of November, 1998.

COPIES FURNISHED:

Gerrit Vanthul, Qualified Representative

5279 Southeast 39th Street

Trenton, Florida 32693

Cynthia Christen, Esquire

Department of Environmental

Protection

2600 Blairstone Road

Mail Station 35

Tallahassee, Florida 32399-3000

Kathy Carter, Agency Clerk

Department of Environmental

Protection

3900 Commonwealth Boulevard

Mail Station 35

Tallahassee, Florida 32399-3000

F. Perry Odom, General Counsel

Department of Environmental

Protection

3900 Commonwealth Boulevard

Mail Station 35

Tallahassee, Florida 32399-3000

NOTICE OF RIGHT TO SUBMIT EXCEPTIONS

All parties have the right to submit written exceptions within 15

days from the date of this Recommended Order. Any exceptions to

this Recommended Order should be filed with the agency that will

issue the final order in this case.