(SPEECH)
[AUDIO LOGO]

(DESCRIPTION)
A logo depicting a person raising their arm in the air.

(SPEECH)
SPEAKER 1: OTAN, Outreach and Technical Assistance Network.

(DESCRIPTION)
OTAN - To lead California Adult Education in the integration of technology into the educational process, ultimately empowering learners to meet their academic, employment and civic goals. OTAN dot U S.. 9 1 6 2 2 8 25 80.

OTAN Online. Technology and Distance Learning Symposium. Four Easy Steps to Improving Student Writing Skills. Presenters: Janice Fera, CASAS. Luke Philley, CASAS. March 3, 2023.

Janice Fera appears on a video call.

(SPEECH)
JANICE FERA: So thank you to Luke Philley for CASAS tech support. He's our expert on CASAS Essay Test, and I asked him to join us today. Also thank you to Cynthia King who is in the room, room host assistant, and I totally appreciate having both of you along for the ride today.

So for the next hour, we're going to be talking about the two new CASAS assessments. One is called essay, and one is called language. And the title of my presentation-- let me go ahead and share my screen-- is Four Steps To Improving Writing Skills, and its Ideas For Implementing the CASAS Essay and Language Tests.

(DESCRIPTION)
A slideshow. Title slide. Text, Janice Fera. CASAS National Program & Technology Specialist. j-f-e-r-a at casas dot org. Copy of the presentation: tiny u r l dot com slash casas dash essay.

(SPEECH)
And the reason I talk about the ideas for implementing, it's not just your old-- you're multiple choice type of test. What we found with the pilot agency is there's a whole lot of preparation that needs to happen in order to have things go successfully both for the proctors and the students. And I'm going to be talking a little bit about that.

So just to start, we talk about writing, and we talk about how important it is. I like to think of it as something that's not just a call to action, but it's also persuasive. It can allow people to clarify a position or document an event. I think good writing is inspirational. And probably the biggest challenge to agencies in helping students to be better writers is the amount of time that it takes for a teacher to score an essay.

There's also a question about ambiguity or about the fairness of three different teachers all scoring the same essay, do they all come up with the same rubric or a score? And we believe that what we've done with CASAS Essay Test is the best of both worlds. We've used the artificial intelligence and the knowledge of teachers to train this auto scorer to do that work. And the great thing about is it does the scoring in about four or five seconds.

So sit with me for a few minutes. I'll give you a quick tour of what it's all about, and share a little bit about the experiences that our pilot agencies have had.

(DESCRIPTION)
Slide, Questions for the Attendees. Two questions.

(SPEECH)
If I could just for a minute before I dive into that little video, if you could either in the chat, or I think we've got enough people. It might be a little hard to raise hands and go around the room, but what is your role in Adult Ed?

Because what I've found is when I talk about essay, the administrators have a view of, oh, wouldn't this be great for our school. And sometimes the first impression from teachers is, oh, my gosh, not another assessment. And really what the benefit that we found is that the students are the ones that really embrace this once they've gotten over the hump of taking that first dive into the deep water of an essay test online.

Also looking for ideas if you currently offer writing classes, like a dedicated writing class, or maybe it's bundled into part of your other curriculum as part of your ABE reading, GED prep, that kind of thing. And there's lots of great comments coming here. Lots of ESL teachers, beautiful. And I'll say that the essay test is geared for intermediate, high, and above. So if you have ESL students in that ability level currently, these tests are for them.

OK. Now Andrew, I see you're teaching beginning level students. This would be probably too hard. I'll take you through a tour and show you a little bit about what the questions look like on the language test and what the essay prompt looks like. And I think we'll agree that it takes a certain amount of a command of the language.

When I took Spanish language and literature in school years ago, that for me to be able to write an essay in Spanish, took a little bit of extra prep and ability in different kinds of ways.

(DESCRIPTION)
Slide, The New "Essay Test". A video.

(SPEECH)
OK. Let me go ahead and queue up this video, I'm actually really excited about it. It's I can't talk and type at the same time, but here it is. This is an introduction to the CASAS Essay Test.

(DESCRIPTION)
The video plays. An animation of a man working on a laptop. Text, Effective writing is a valuable skill for all adults.

(SPEECH)
[MUSIC PLAYING]

(DESCRIPTION)
The animation continues. The man shakes hands with a woman. Text, For a job application cover letter.

The woman writes on a clipboard. Text, A request for information from a school or business.

A man types on a laptop. Text, Or an email describing a customer issue.

A person raises their arms in the air and shrugs. Text, Does your agency assess writing skills?

A man holds a lock and a woman holds a key. Text, If not, we've got the key!

Hands type on a keyboard. Text, The new CASAS Essay Test is here!

A woman holds a gold star. Text, No staff time is required to grade the essay; it's auto-scored!

A woman raises her arms and cheers. Text, Results are available immediately.

She draws a checkmark in a bubble. Text, The CASAS Essay Test evaluates.

A computer screen. Text, Organization, Word choice, Development of ideas, Response to the writing prompt, Standard English grammar.

A hand holds up a screen. Text, Students create an essay based on one of three prompts.

A web page. Text, Tests are administered through CASAS e-Tests.

Two people high five. Text, Interested in learning more? Email: info at casas dot org.

The slideshow and video call reappear. Slide, Are Writing Skills Important? A text list and a question.

(SPEECH)
To summarize the video, we believe that students will benefit from these different assessments and the preparation that you can deliver to them through your curriculum. You will help your HSE candidates as they prepare for their HISET or GED exams, and it'll support students who are transitioning into different roles and employment, or a secondary education where writing skills are required. The video gave some examples about this.

Now, the pilot sites did just exactly what I'm showing you here, is they group the students by their goals and their abilities, and they created different classes. Some needed a short workshop. Others needed more of a bootcamp for four to six weeks. And then others needed more of a semester-long writing assignment to practice with things.

(DESCRIPTION)
Slide, Four Steps: Improving Writing Skills. A graphic with the steps in the four corners of a square, and arrows pointing from each step to the next.

(SPEECH)
So the next few minutes I'm going to introduce what I call the four steps to improving writing skills. And as you know teachers, the writing skills have four basic components of preparing, assessing, instruction, and then repeating. And I like to think this is similar to what we do when we write. That we're thinking about things and then we compose and then we edit and then we reread it, and it really is that cyclical kind of in nature.

So I'm going to go into these four different areas and talk a little bit in more detail.

(DESCRIPTION)
Slide, Prepare Proctors & Teachers: A list.

(SPEECH)
When I talk about preparation, first off, for the practicing teachers, we have documents, we have a lot of materials, and that reading and preparation helps a lot. It includes screenshots and sample reports, and also includes a rubric, which is very helpful so that you can see what the auto scorer is going to be grading you on.

Also we found that when you take an online test-- I mean, thinking back when I used to use a blue book and I had to write my essays by hand and have them scored, it's a little bit different in the world today because they're in an online word processor. Well, there's a few tricks to that. Do they know how to copy and paste? Do they realize that after a punctuation like a period or a comma that there needs to be a space?

Because the end of a sentence with a dot with the next word starting right next to it, that's going to be scored as a typo. So these kinds of essay test advisory notes are very helpful for teachers and for students to learn some of the insider ways to make things better when you're taking a test.

(DESCRIPTION)
Slide, Prepare Students: CASAS "E-Tests Sampler". A screenshot of a list titled Sample Test Items. A link, casas portal dot org slash e tests.

(SPEECH)
Another thing that we created at CASAS is what we call the essay eTest Sampler. OK, the eTests Sampler already exists. Most of you are already familiar with it, but we've added a component called Essay. And teachers can use this in the classroom because it just runs in a browser. It's not actually connected to eTest. It's not going to do any scoring.

But it gives the students 10 minutes to play inside the word processor, and you can describe what it's going to look like. And that preparatory step, it really reduces a lot of anxiety. And that was the feedback that we got from our pilot agencies that the more preparation and the paint the picture for the students of what that is going to look like, it makes it so much better.

(DESCRIPTION)
Slide, Inside the "Sampler". A screenshot of the word processor, featuring a question prompt at the top, and essay text at the bottom.

(SPEECH)
So if they go into this tool and they click on it, it shows them what it looks like inside the essay test. It starts at the top where there's a timer up here, and this counts down from 10 minutes in the practice. In the real test it would be 60 minutes. It gives them a prompt. Here the prompt for the just a sample is healthy eating habits are important but often not followed. Why is this? What can be done about it?

So that is the style of prompt that the CASAS Essay Test use. And then below it is a box where the students are going to type their essays in. You notice that the last word ingredients there has a little red underline underneath it. That's the spell checker cut that. And we want the students to see that so they can right click on it and correct the word.

Down at the bottom there's a directions button that they can click on to get more information like to remind them you're allowed to use scratch paper, no copying and pasting from other places. And actually, that's prevented because in the CASAS eTest environment as you know we use a secure browser. So they can't exit early or move things back and forth. All right. At the very end, when they're done, they click the blue arrow and that's it.

(DESCRIPTION)
Slide, More Student Preparation Tips: How are your keyboarding skills? A bar graph and a statement summarizing the results.

(SPEECH)
All right. Another topic of preparation, and this one came out in the pilot study as well. We found that of the 120 students initially surveyed, we asked them a question, do you think this would have been easier on you if you had better typing skills? And 63% of them said, oh, yeah, we definitely could use some kind of a tool or a lab.

So the labs and workshops that became popular at South Bay Adult School was for students to use something like typingclub.com. And they would take an initial test with it and get a baseline score, and then the counselors would set a goal of something like maybe an additional 10 or 15 words a minute, or getting up to 25 words a minute to be able to type. And then they would have that typing certificate in their portfolio, which is good if you're trying to get a job.

But I just shared that little piece of information that came up as part of the study. So those are the slides I have for preparation. Does anyone have anything they'd like to share or comment on or shall I keep plowing into the next section, which is the actual writing assessments.

(DESCRIPTION)
Slide, 2, The Writing Assessments. Essay Test and Language Test. A link, casas dot org slash product dash overviews slash research dash and dash evaluation slash field dash testing dash opportunities.

(SPEECH)
Looking for hands or comments. OK. All right. Let's look, the second step assessment. Essay is looking at how writing flows.

CYNTHIA KING: There's a hand in the back here.

JANICE FERA: Oh, I'm sorry. I can't see. Let me go ahead and stop sharing for a moment.

(DESCRIPTION)
The slideshow disappears.

(SPEECH)
OK, yes, please can you share us?

(DESCRIPTION)
A classroom full of people appears on the video call. Text, OTAN Staff 2 0 9.

(SPEECH)
AUDIENCE: OK. Well, no, not to share, just a question. In the preparation do you have material that help the students develop the writing in the way that the CASAS test is going to be graded? Does that make sense? What are the things that whatever software is being used to grade it is looking for? And are there materials to help teachers teach them?

JANICE FERA: All right, so the question was, what curriculum do we recommend, or do we have any materials that we suggest? And yes, that answer is coming in the third section. We've been gathering suggestions from agencies in the pilot program, and we are open if there are other ones you have.

I'll just jump ahead to answer your question because I hate to get left-- when I ask a question, they tell me to wait. I hate to wait. Within CASAS, we have a QuickSearch Online, and that has the ability to look up keywords like conclusion, or inference, which is two important things when you're writing if you're trying to go for a high-level essay created.

We also have one of our agencies like the-- it's called preGED Writing 1 and 2 by New Readers Press. And we have another school that said we're going to write our own. And I have a link coming up to share for you with that one. It was called hook, line, and sinker.

I love that title, which is talking about how to write an introduction that hooks the audience. And I do have a copy of that PowerPoint on CASAS.org. We can get that for you if you're interested. It's posted in the-- if you go to CASAS.org, and you click on the National News and Updates webinar, you'll find that PowerPoint in there. Is that a start?

AUDIENCE: Sounds great.

JANICE FERA: OK. Super. Let's go back.

(DESCRIPTION)
The slideshow reappears.

(SPEECH)
Thank you for your question. So the essay test and the language test I like to think of them as complementary where the essay test is looking at the lyrical flow of the language, the creative part of it, the essay is.

And then the language test is more of the grammar and mechanics down at the lowest level. The language test is 20 multiple choice questions, 20 minutes in a sentence completion format, fill in the blank. That's also scored immediately, and there's a report. We're going to get into the reports in four slides. So I showed you what the essay test looks like and what a sample question looks like. Let's take a look at a couple of the sample questions from the language test.

(DESCRIPTION)
Slide, Sample Language Test Question. A fill-in-the-blank sentence, and three answer choices.

(SPEECH)
The chef cook, cooks meals at the restaurant. OK, cooks. This is an example where a question is aligned to the verb category in the report. The language is looking at nouns, verbs, phrases, and punctuation. OK, let's look at another one.

(DESCRIPTION)
Slide, Sample Language Questions (Continued). A fill-in-the-blank question, and three answer choices.

(SPEECH)
OK, so the class that he signed up for begins in the spring. This is a question that's aligned more to the clauses, the category of the report. Again, going back to which ones of the-- which students these questions would apply for, I think you might agree with me that they need to at least be probably a higher intermediate ESL student to be able to grasp these kinds of constructs.

The pilot sites that use the essay and language tests as far as timing is concerned, that's leading into my next slide.

(DESCRIPTION)
Slide, Administer the Essay & Language Tests. A screenshot of an e-Test page, titled Testing Sessions.

(SPEECH)
Let me hold that thought for a second. This is inside the eTest configuration screen. Any proctors or eTest coordinators out there, this should look familiar to you.

So we added three different essay prompts, and we also added the language test. And that language test is we have great vision, great high hopes for it to be expanded into multiple levels with multiple pre and posttest forms. As of now it's labeled a field test because we don't have the full battery that we normally would launch in a CASAS test suite.

But this is what you do. You basically just like any other eTest you go in. You start the test. Your students log into the testing stations, and they start to write. And as essay test runs under eTest, it creates that secure environment so students can't copy materials from a website or another source, and they can't exit while taking the test.

Then when they're done, the personal score report is displayed. Then you start the language assessment and get that report. I like doing both of them at the same time. Even though we allocate 60 minutes for the essay test and 20 minutes for the language, we found looking at over 200 essays in the pilot and language, they don't take that long.

Most students will finish up a decent two paragraph, three paragraph essay, they can crank that out in a half an hour. And when they go to take the language test, that's usually maybe only 15 minutes just for planning sake. OK.

(DESCRIPTION)
Slide, Essay Personal Score Report (P S R). A document with information about a test, and a progress bar with values from 1 to 4, filled halfway.

(SPEECH)
So here's the essay personal score report.

It's got a little green bar, looks familiar. Got the student's name and information up at the top, and it gives him a score. I remember the frustration of waiting for my essay to be scored by my teachers, and it would take days. And these it pops up immediately as soon as they're done with the test. Huge improvement.

It's not just a pass or fail. It's anywhere between 1 and 4 as the score. Inadequate essay is given an asterisk along the same lines as the other CASAS tests.

(DESCRIPTION)
Slide, Personal Score Report (P S R), Page 2. A document titled Becoming a Better Writer, with a list of four areas of improvement.

(SPEECH)
And at the bottom of each report is a specific areas for improvement, which is on this slide, main idea, organization, word choice, grammar and mechanics. It's divided up into those areas. So anybody who scores a 2, let's say, would be given a set of suggestions on what it would take to get to the next level, a 3.

Let's dive into a little bit more on the bottom item here, the grammar and mechanics because this is the part that the language test really gets into.

(DESCRIPTION)
Slide, Definitions.

(SPEECH)
It breaks this up into a finer granularity. The grammar is the structure of language while mechanics refers to the rules of written language such as capitalization, spelling, that kind of thing.

(DESCRIPTION)
Slide, P S R for Language. A screenshot of a Personal Score Report, with test information, and scores for four items of language content area.

(SPEECH)
So when we run the language test, you'll see that report reflects that. With nouns-- I'm looking at the language content areas, nouns, verbs, phrases, and punctuation. And the way that I recommend teachers look at that is they go over to the far right and they look at the percentage correct. And they look at the areas like the ones I've highlighted here where the student needs the most work.

There's also a CCR language level which is helpful for you to pick out the right curriculum level for the student. So this is the end of my section, second section, which is to summarize that the way you use these two reports. You have a student that's got a comprehensive road map on areas that you need to concentrate on to help them to improve their writing.

(DESCRIPTION)
Slide, 3. Instruct - How to Teach Writing? What's Your Methodology? A bullet point list.

(SPEECH)
All right. So the next section, we're going to talk about how to teach writing. And again keeping in mind the student populations could be ABE, and NRS levels 2 and up and ESL high-intermediate and advanced. There's more than one way to teach writing.

Again, old school I used to use an outline. I'd get this paper, and I'd start drawing things on a piece of paper, and I would come up with an outline. And then I would switch to the keyboard or whatever and start creating it. I find the Generation Z students they go straight for the keyboard. So we have to consider how we're going to teach them. Are we going to teach them to look at the prompt and pick out key words, or are we going to teach them to grab phrases?

I know writer's block is a real thing for a lot of people. So that's something that we'll need to be considered in how they're going to effectively use their time. I mean, the pressure of a time test is created by just saying the fact that it's a timed test, and they don't know what the topic is, and that adds to some of the anxiety. However, I think that you'll find that the benefits outweigh some of the initial discomfort.

(DESCRIPTION)
Slide, 3. Instruction - Writing Skills (Continued). A bullet point list.

(SPEECH)
These were tools that I mentioned a moment ago about inference and conclusion. The introductory hook, line, and sinker. PowerPoint from Andrew and Kay Hartley. I think she's in the audience here. She's going to be teaching at the CASAS Summer Institute in just a couple of months some more ideas for writing skills. So hats off to that group. And again, the New Readers Press preHSE Writing 1 and 2 was the other tool that I mentioned.

Can you share a slideshow [ INAUDIBLE ] Yes, absolutely. Did I already do that once? Let's see. There it is, tinyurl.com/casas-essay.

(DESCRIPTION)
Slide, 4. Repeat the Process. The graphic of the four steps from a previous slide, and a bullet point list.

(SPEECH)
So to bring it all together, we're talking about this cyclical nature of composition which is writing and editing and rewriting. And hopefully, this is giving you a little bit of a sneak preview on how the CASAS assessments could be used in that environment.

We'll be planning a few more in-depth presentations in April. So if you're interested in doing that, please send an email to me jayfera@casas.org, and I'll get you onto the list for that. We do have field testing opportunities for other tools, but these are the two that today I'm super excited to be sharing with you about.

(DESCRIPTION)
Slide, Questions?

(SPEECH)
And now I'll take a little break before I get into some weird techie stuff. If you have any questions so far? Cynthia, do you have any questions there in room-- is that 209?

CYNTHIA KING: No.

AUDIENCE: I have a question.

CYNTHIA KING: Oh, one question.

AUDIENCE: Hi, Janice.

JANICE FERA: Hi. I can't see you, but I can hear you.

AUDIENCE: Carla, from CAERC. My question, Janice, this particular assessment is more for in-class information and not necessarily placement purposes. Or if it is for placement during an intake, how would you use that in combination with life and writing or life and reading like work or goal series?

JANICE FERA: Excellent question. So the question is, would you use this test for intake purposes? And if so, how would you go ahead and-- how would you use that for blending it in with the other levels? Let me just throw that one out to either Barbara or Kay or Portia. If either of you three possibly have a suggestion on that one? I have a couple ideas, but let me just ask the California program specialists if they wanted to tackle that, I don't know. Ladies, no?

Well, I know that one of the adult schools we're considering using that in place of the oral assessment, and they felt that preparation step by taking a student that hasn't been through any of the preparation discussion about, be sure to put a space after the period, or how to manipulate an online word processor, that that was asking a little too much for a lot of students.

That they wanted to spend a little bit of time with them to prepare them before putting them into the essay test. So the agencies that I talked to had already put the students through an appraisal and had already level tested the students before the essay was administered.

KAY HARTLEY: Wouldn't it be challenging to figure out if they're at a high enough level before administering it? I mean, it seems like you might be putting them up for failure if you gave it to them without discerning whether or not it's at an appropriate level.

AUDIENCE: See my thought is it's primarily for instructional information rather than for placement, and that's what I'm hearing from you today?

KAY HARTLEY: Yes. Yes, yes, definitely. I mean, it's real if for instance you had an ABE class or a preGED class where they're preparing for the test, it'd be an excellent tool to determine whether or not they're beginning to approach readiness for essay writing for the test. But for standard ESL, I think it would be a real daunting experience if they haven't had been appropriately leveled before they took it.

BARBARA LEHMAN: I would think that you would want, especially for your ESL learners, you would want to find out how literate they are in their native tongue and go from there because some students have advanced education in their native tongue but yet they're still struggling with English.

And a lot of-- a lot of those students can write much better than they could speak, so it could help you that way as well.

JANICE FERA: Thank you, Barbara. Great.

AUDIENCE: Do you think-- one more question to the group-- do you think the language assessment, the shorter language one, that one would seem to be more helpful for placement purposes, right? The grammar, right?

AUDIENCE: Yeah, [ INAUDIBLE ]

JANICE FERA: The question is, would the language test would it help if I showed that report again so that you know what that looks like in case you're not-- this one.

(DESCRIPTION)
She returns to the slide titled P S R for Language, showing a screenshot of a Personal Score Report.

(SPEECH)
BARBARA LEHMAN: I would think this would be been informative for you as a teacher and for the student, especially if they are educated in their native tongue or your advanced ABE, your advanced ASE student.

JANICE FERA: Also a question in the chat room. Carol Lazarus asking, can it be used as a posttest for moving up a level? Answer to that is no. It's not considered a posttest Remember, a pre and a posttest have to be of the same modality. And the CASAS Essay Test isn't one of the ones that is already approved by the NRS as having a pair. That's something that we're going to talk about going forward once we have more agencies that have administered it, and we have more statistics.

We're hoping that we can use it as a predictor for success for the GED test because of the writing component that's part of the RLA within GED, but we need a lot more data and more time to be able to come up with that kind of a reference.

CYNTHIA KING: Janice, there's a question here.

JANICE FERA: I'm sorry.

CYNTHIA KING: There's a question here in the room.

JANICE FERA: Yes, go ahead.

AUDIENCE: So one of my instructors who teaches HISET-- we just started implementing computer-based testing because of the problem with HISET. But with one of the problems we're running into is that the students are so used to-- and I think it came out one of your example-- seeing the underline of the error. How can we prepare, or what can we do so that the students like not the pencil?

Because that's been one of the challenges. That the student is so used to seeing, oh, there's a grammar error. There's a spelling error. So in their mind, they're like, oh, it's on the computer so it's going to show me the mistake. So how or what advice would anyone in the room or online suggest for us to try to relay the message to the students?

Because we've tried. Apparently we've not been successful but that they're not going to see that, or is there a practice tool kind of what you showed that's not going to show the errors of the student to start training them that it's not going to show you your errors.

AUDIENCE: I don't know if I'm making sense.

CYNTHIA KING: You're talking about underlining mistakes?

AUDIENCE: Yes.

CYNTHIA KING: [ INAUDIBLE ] grammar error.

AUDIENCE: Yes.

JANICE FERA: So as you recall when I showed the eTest sampler that that shows that as they're typing and they get that red underline, that they need to right click to fix the spelling mistakes. If you had a Chromebook and you turned on the spell checker, invariably you also get a grammar checker and that's a blue line underneath.

Agencies have to decide if they want to have that turned on or not. And we think it's an unfair advantage to give the student a grammar checker, and then they find themselves in a HISET testing environment and they don't have that anymore.

So the one agency that has Chromebooks in their eTest lab decided to turn off both the spell checker and the grammar checker because it's like buy one get one free within the Chrome environment. It's not a problem on Windows, but that's how Chrome implemented their spell checker. Any ideas from the field?

I mean, this is a panel discussion, and I'm so glad that all of you are here and so many of you are teachers. Yeah, Anders comment about AutoCorrection to be turned off by the individuals, it's not actually AutoCorrection, it's merely flagging the error. It's not doing any AutoCorrect for them. And thank you Luke for saying it more eloquently than I did about the correlation with the modalities. Thank you.

Kind of an interesting fact I picked up. So the agencies did a lot of their-- in the pilot-- they did a lot of their pretest on their students the reading, and then they gave them the essay tests around the October, November, December time frame. And now I'm circling back with them and saying, no. So they're in their writing classes now, they're in their boot camp, they're in their workshops. What are you going to do for the retest?

And it brought up a really interesting bunch of questions, is it valid and is it actually beneficial to give the student the same essay question after the writing instruction is completed? To see how they do with the same question in terms of their use of words, in terms of the flow, the grammar, the punctuation. Would that be the way that you and your agencies would use it to retest on the same of the three prompts?

So if you started with the first one about-- I think one is about employment, one is about a health thing, and then one is about consumer skills. Would you-- I'll just give you the answer for the essay. The pilot site that I was working with is they said we want to test on the same prompt. We want to see what they'd do with it. It's not like they're memorizing a bunch of answers.

We want to see how they've matured in their writing style. And that way that they can cycle through that and see if they've gone from a 1 to a 2 or 2 to a 3. Maybe they've made a mega jump and gone from a 1 to a 4. We're still waiting for that data to become available to us on the retest.

PORTIA LA FERLA: Janice.

JANICE FERA: Yes.

PORTIA LA FERLA: I would think that the student would really like to that, to see their own progress. I think that would be a really inspirational for the student himself or herself.

JANICE FERA: Good point. I like that. You're right. Because I know when I used to-- I'd get back my essay, and it would be redlined, and the teacher would sit next to me, and she'd go, hey, this is what you need to do differently. And she would send me back, and I would have two more weeks or a week, and I would have to redo it. And then I would come out with something that was so much better than the first one I did. And that's just such the joy of a teacher really to watching the student mature from where they were into their new abilities.

PORTIA LA FERLA: Yeah, that is the writing process.

JANICE FERA: Angela, to answer your question, when the student is in the essay test, yes, they are shown the spelling mistakes. No they are not shown the grammar mistakes. The one caveat is the way that the Chromebooks have implemented it, which is either get both of them or you get neither of them. OK. A question from Danielle, after they take the assay test, can we as teachers view what they wrote? Am I being recorded?

[LAUGHS]

Oh, my gosh, that's a giant-- OK, I'm going to take off my CASAS hat for a second. That's a giant debate within CASAS. How much we're going to open up the entire database to let people pull out chunks of text. So currently yes, you can see what they wrote. I don't know if they're going to lock it down or not in the future, but as of today, you can go into TopSpro Enterprise, you can go into the tests, you can find the student in the test, and you can bring up their text. OK, CASAS hat back on.

[LAUGHS]

That's how we've implemented so far, and that was how we found the mistake with the punctuation with the no spaces after the periods. One of the agencies called me, and she says, this student, Janice, is so capable. We can't believe she got an asterisk. I went, what?

So I have some magic that you guys don't have that I can actually do eTests and copy and paste from other places outside of the secure browser. So I grabbed the student's essay out of TE, plopped it in there, and then scored it and got an asterisk. And then I put spaces after all her commas and periods. And it was a beautiful essay. It was like four paragraphs long, nice choice of words. She got a 3 already just for making that one important correction.

And I mean, if you think about it, she's got maybe 18 sentences total in three paragraphs. And every one of those ones ended up with a weird word with a dot in the middle of it, and that was enough to throw the auto scorer into thinking that wasn't a great essay.

AUDIENCE: Question, please. Does it weight the grammar versus the content?

JANICE FERA: There's lots of magic in them [ INAUDIBLE ]. Yes, there's all kinds of waiting about which things are more important, is the punctuation more important than the creative use of words? And that's all part of the AI behind the scenes. But I do have a few things to share with you if you want to go into the ChatGPT discussion.

[LAUGHS]

ChatGPT. Anybody familiar with-- go ahead.

AUDIENCE: Speaking of writing, there should be a huge emphasis on content, and you can overlook the small little errors because are you understanding the individual writing and speaking? And you spend time, not you but us as a group, going backwards to picking apart all these little details, I think is unfortunate. And your example highlights that. Fantastic. I mean, let's give her 80% on content, and 10%, 15%, 20% on the other in my opinion. But I'm not an expert.

JANICE FERA: I see your point 100%. And all I can say is that's something that I would take back to my development team and take back to a few others to discuss a little bit more. The waiting on content versus little nitpicky things like punctuation.

For now, all I can say is that the amount of hundreds of hours of training-- let me go back to a couple of those slides-- that have gone into training those three prompts that we've prepared so far would suggest that just a little bit of that advisory document that I shared earlier saying that these are a few tricks that you need to teach your students so they score highly on the test. These were just a few things that they'll need to know.

Because if you forgive them, and then they get to HISET test or the GED test and it's not as forgiving, there's going to be a disappointment there. So unfortunately, teaching them the way that an online scoring algorithm works, it's a bit of a dinosaur at the beginning, but it will serve them in the long run. And it's really not too hard to teach those little tricks. But I hear you loud and clear, content is super important.

I mean, look at the stuff that the kids are texting each other, and it's almost unintelligible, but it's not unintelligible. It would fail an essay test, but it conveys meaning. I had a class in college, what is the meaning of meaning? UCLA Linguistics 601. We sat around and talked about what is meaning? I get that.

Any other questions in the chat that I've missed? Let's see. We want to see the last few slides I have for-- because this is the technology and distance learning, and technology being the big word there. I prepared a couple of slides that I didn't share with the national audience just to talk a little bit more about the technology behind the-- let me skip ahead-- the auto scorer.

(DESCRIPTION)
Slide, How Does "Autoscoring" Work? A bullet point list.

(SPEECH)
I talked briefly about the algorithms and the way that we pretrain the prompts. We used CASAS program specialists. We used experts. We've been using these prompts within our NEDP, National External Diploma Program, which is popular in a lot of states besides California.

We share with you the scoring rubric and just give you the advice that these are the things that go into the auto scorer. And we feel that the fact that it doesn't have any teacher bias-- not that you guys are bias-- but the bias of what one teacher might grade is an A versus an A- versus a B+, this thing is a machine. If you give it the same essay 15 times, it will give you the same score. And this consistency that comes with that some find is as beneficial.

(DESCRIPTION)
Slide, How Lenient is the Autoscorer? A bullet point list.

(SPEECH)
Here's a question I got from one of the agencies, how lenient is the auto scorer? How hard is it to get a 4? So, the good student, I sat down, and I wrote a bunch of essays. I wrote a good Janice quality hard-- a good-- I got a 4 on it. And then I backed it off a little bit, and I got a 3. And then I made some mistakes, and I took out a paragraph, and I got it down to a 2, and I got it down to a 1.

And then I messed around with it a little bit to see what kinds of things it flags for an asterisk score like too short, too many certain types of errors, violent speech. So there's a bunch of things that are included in the rubric which describe the things that would give it an asterisk.

But I found that it's not easy to score a 4. And if any of you have heard about ChatGPT, which is a little online tool. It's free currently. Students are using it a lot, especially in the community colleges, I hear about that a lot. So anyway, you feed it a prompt, and it gives you some information that's snooped out of the internet in different places.

So I took the CASAS essay prompts and I fed it to ChatGPT. And sure enough, I gave it that healthy eating habits are important prompt, and it produced this little couple paragraphs essay. And then I took it and I fed it to the auto scorer. What do you think it got on it?

AUDIENCE: 3 or 4.

JANICE FERA: 2.

AUDIENCE: Wow.

JANICE FERA: So it's looking for something good. And I like to think of it as a pretty tough 11th grade English teacher. OK, full caveat, if I had changed the prompt around, and I had told it to do something like create a collegiate level essay on healthy eating habits which include-- it probably would have put out something a little more collegiate level.

But the fact that you can take the prompts and you can feed it to ChatGPT, and it only scores a 2 gives me a feeling that we are going into some level of detail that is exhaustive and analytical.

(DESCRIPTION)
Slide, References. A bullet point list with links.

(SPEECH)
Let's see. Here's a couple of references which are included in the PowerPoint.

There was a great presentation by Kristi Reyes of MiraCosta College part of the Lunch with LAREC talking about ways to teach writing in a collaborative way with cool online tools. I loved that presentation. Also, TESOL they had a note starting on page 60 about advice for teaching writing to students. Those are I thought probably two helpful things.

I'm watching the clock.

(DESCRIPTION)
Slide, How to Get Started? A bullet point list.

(SPEECH)
I want to go ahead and suggest that if any of you want more information, you can always email me jayfera-- F-E-R-A, just one R-- @casas.org. You can talk to your teachers and administrators. And if you think this is something you're really interested in, reach out to tech support. That's Luke and his gang, and they will be able to install the different sessions on your eTest sessions. Remember that was the slide we looked at here.

(DESCRIPTION)
She returns to the slide titled Administer the Essay & Language Tests, with the screenshot of the Testing Sessions.

(SPEECH)
They'll be able to install those for you and get you set up with all the copies of the documentation. There's oodles and oodles. There's the rubric. There's a frequently asked questions, which is a short thing that I would send up to your administrators if you're trying to explain to them the benefits of it and the short of how it works.

Then there's a document that's also posted that's-- we call it the TED internally. I can't even-- Test Administration Directions. And that's got screenshots, sample report. It's got the verbiage that a teacher would say to the students as you're sitting down getting ready to take the test assessment, lots of information in there.

It's been reviewed twice. It's had two different whole revision cycles since we started the pilot. And I'm pretty happy with the state that it's in right now as being good and comprehensive. And those are all available through the tech support or on CASAS.org

So I want to thank you for joining with me today. We're thinking about writing and thinking about ways that your students could possibly benefit from having that skill set in their arsenal as they go out into the world and go into the next things.

If you have any comments or questions, reach out to me please jayfera@casas.org. And we have a couple more minutes before we're going to say goodbye so if anyone wants to stick around, I'd be glad to chat with you and answer any questions. And that's pretty much it. I want to say thank you to Cynthia for room hosting there in room 209 and Luke Philley for being in the chat. Thank you all.

(DESCRIPTION)
Text, OTAN Online. Technology and Distance Learning Symposium. Four Easy Steps to Improving Student Writing Skills. Presenters: Janice Fera, CASAS. Luke Philley, CASAS.

Subscribe. YouTube dot com slash OTAN Serves Adult Education. Like Us. Facebook dot com slash OTAN Serves Adult Education. Follow Us. Twitter dot com slash OTAN.