And here we go. The attendees are starting to come in. We've got one. [laughter] Sometimes, those links are a little hard to find, and we do have to hit the Refresh button every once in a while so we get the Join button. We're up to two, yay! I mean, it was just-- here we go.

All right. And we're starting to fill in. It's looking good. All right, everybody. My name is Melinda Holt. I am the tech host today for this webinar, NRS Performance Goals-- doo, doo, doo, doo, doo, doo, doo.

Everybody come in. Have a seat. There's lots of room up front. Everybody just move towards the front. You'll get much better seating there. They're nice and cushy seats towards the front, hard chairs in the back.

And we're going to go over a few housekeeping items really quick here. So Jay, if you could release Share, thank you. This will be really quick, folks. And you've probably seen these before, but we have to do this every time.

We are recording this webinar. All of the recordings will be available on the VFairs platform as soon as they are downloaded, rendered, and uploaded. So that takes time. Please give us two to three business days, if not more. If you don't see the recordings on the VFairs platform, contact TAP after about three days.

Audio-- you control your own audio. It's one of the few things you do control here in the webinar land. If you cannot hear, or if your sound's really soft or really loud, turn your volume up or down. If the sound appears to be coming from your speakers instead of your headset, then check the little carrot next to the Audio Settings button. And then you can control your output, which is what you're listening to.

The webinar Chat will be used, as well as-- the other button down there is the Q&A. For the webinar Chat, just keep that for general chat. And make sure that All Panelists and Everyone are selected. If you just choose All Panelists, then that's going to be the only people that see your chat. So if you want everyone to view the chat, select All Panelists and Everyone. That's just for general chit-chat.

Or if the presenters ask you to say "yes" or "no," use the Chat. All other questions should be typed in the Q&A. So to get ready, why don't you go ahead and click on the Q&A button. Just pull it off to the side, and wait until you have a question where you can type your questions related to the presentation, please.

You can also Upvote someone else's questions, so you don't have to type it. You let them do the work. You Upvote it. And then the viewers will see that more people really want that question answered.

You have View Options. This is the other thing that you can control during a webinar. You can fit to window, or you can increase the size of the presentation. So if things look a little dim, you might want to increase your setting. You could also Exit Full Screen. That way, you can multitask while you're listening to the presentation.

There will be an evaluation at the end of this webinar. When we end the webinar, you'll see a Continue button. You click on that, and it should open the evaluation. This time or today, it should open up the evaluation so you can actually input.

Please don't edit, all right? And we hope you're having a great summit. I'm going to go ahead and stop sharing and hand off to our lead presenter, Jay Wright from CASAS. Jay?

Hello, everybody. Hopefully, everybody can hear me OK. I'm sharing my quick little PowerPoint here.

We can hear, and we can see.

Going to get into full screen here. Hopefully, everybody can see what we need to see.

Yes.

But I want to start by introducing my all-star panel. I have Steve Curiel from Huntington Beach Adult School. I have Carol Hirota from Delta Siera Consortium. I have Thoibi Rublaitus from Corona-Norco Adult School. And I have John Russell from the Citrus Consortium.

Just in the interest of giving full credit where credit is due, Branka Marceta from the CAERC Capital Consortium was a fifth panelist. She did do a lot of the preparatory work with the other four panelists, but she was triple-booked. So she was unable to actually be a panelist.

Got to say, we have way more than enough already, though, to fill the time with the panelists we have. Because we have a great panel, there is a short little presentation I was going to do just so everybody's 100% clear on what the heck we're talking about when we use the term "NRS Performance Goals." But I really wanted to get the panel out here in front, so I wanted to introduce them.

Don't worry about this crazy bunch of questions here, but this is what the panelists had to face by being on here. So a good appreciation for the panelists, as I gave them, like, 20 pages of this sort of stuff. So just wading through all the questions I overloaded them with was a big heavy lift applied itself.

But back to this slide, which is a lot easier to view, I wanted to have each panelist introduce him or herself and kind of talk about how they've implemented this over the years. It was no accident who I picked. I knew exactly who I wanted to pick for this panel because they've shown superstar qualities for a very long time.

We've had lots of detailed discussions, each one of these panelists, about their data and about how they use their data to excel at their agency and at their consortium. So I'll start with Steve at the top. He appears at the top of my screen, and he's at the top bullet on this slide.

--everyone. Just happy to be here. And of course, any time Jay asked for me to pitch in and talk about my school, I'm happy to do so. And so we've been using these performance reports, the NRS tables, the TE tables ever since I jumped into ESL, what, 12 years ago. I wasn't an ESL person. I was a more CTE administrator.

And so I started learning about the NRS tables, and some of you guys or folks are new to this. I will say, honestly, it takes a little bit of time. You got to be talking with-- definitely networking with your colleagues, finding out about where your data is in terms of how you guys are performing and comparing to other schools.

And-- I'm looking at the-- let's see here-- also, one of the things that I highly recommend is attending the workshops that CASAS puts on and CDE Those have been crucial and beneficial to me and my staff, as we've learned about the performance goals and the changes that happen as they come in. So with that, I'll save more of my comments to my next section, where I've got a whole section just to myself.

OK, thank you. Carol, if you could go next?

Hi, my name is Carol Hirota. And I'm representing Stockton Unified School District, which is Stockton School for Adults. I'm currently the Delta Sierra Adult Education Alliance Exec. Director, but formerly the director of Educational Services, Adult Ed.

I've been in adult education since 1995. I am a credential speech language pathologist, so data is very important. And that's always been a priority. I will be talking about our journey with our colleagues in Stockton.

But I will tell you, I think the change process started when Marian Thacher came from OTAN and created a technology integration program project with our faculty. And I think the other significant professional learning opportunity was the professional learning communities and the PLC Institute, and the process. And I will be talking about that with the data teams, so thank you.

Thank you. Thoibi?

Good morning, everybody. My name is Thoibi Rublaitus. I am honored to be part of this team. And before I go ahead with anything, I'd like to share that I'm a relatively new principal. However, I have been with Corona-Norco Adult School for about 13 years now.

And under the leadership of my director, Jodee Slyter, we've had a very good data team here. And I've been part of the data team. And today, I will be sharing with you a little bit of how we do what we do with a student-centered perspective. Thank you.

Thank you. John?

I'm John Russell. I'm at the Citrus College Adult Ed. Consortium. I'm also assistant principal of Monrovia Community Adult School. I don't know that we're necessarily superstars in our results, but what I will say is that we're superstars in looking at the data. So we do pretty extensive deep dives into our data as a consortium. That fuels a lot of conversations as to how successful we're being as individual member institutions and as a consortium.

And you know, I'm just happy to be here to share that process and answer any questions as to how we look at all these things and say, maybe our results are not spectacular. But we know what's fueling and driving our decisions. And so with that, I'll just pass it back to Jay.

OK. Thank you, everybody. I just wanted to get everybody in before I went in to stuff. So I'm going to be super brief, but just because we have this title, NRS Local Performance Goals. I know some of you have been into this and know this, but it's a little bit of an ambiguous title. So I wanted to spend five minutes or so running through some slides, just so everybody knows what we're all talking about.

So for starters, we have the NRS goals. NRS stands for National Reporting System. Most of you know this. But just to be clear, that refers to the feds. To be even more clear, the NRS isn't really the feds. They're contracted by the feds to provide regular evaluation at the national level, I like to say very similar to what we at CASAS provide at the state level. That is standardized evaluations, standardized accountability services.

So when we talk about NRS goals, we refer to our state goals that we have in California. Making a long story short is California and all other states are required to establish goals with the NRS or the feds every year, so that's a negotiation process. So we have our statewide goals.

Once we know what our statewide goals are, we're looking at our local data to establish local goals so we can be obviously all superstars and be above average. Maybe if we're not above average, we can get closer to average, depending on where our data is, depending on how we look, ABE versus ESL and so on.

So I don't want to get too detailed, but I'll just say can use the NRS tables in TE. And/or you can use the CASAS Data Portal to look at your local data, compare it to statewide performance totals, as well as our statewide performance goals we establish every year with the federal government. That's the name of the game.

When you look at how the feds measure California, how the feds compare us to everybody else, that's the way they do it, is they set goals for us and look and see across these 12 levels on the federal tables do we meet or exceed our goals that as many of these 12 areas as possible?

OK. So here's the screen shot for the CASAS Data Portal and a link. Some of you, I'm sure, have looked at it. But we have data dating all the way back to the '04, '05 year, so it goes way back for everybody. If you've been WIOA II for any period of time over the last 15 years, your data rests here. We've got Table 4, 4B and persistence results dating back some 15 years on the Data Portal. It's really user-friendly to compare and contrast your data with state totals, with your neighbors, with the other large or other small agencies, and so on.

So in the training, we get into that's a lot more. But just to summarize, in general, if you're just trying to see how you compare with the Joneses, so to speak, we say, use the Data Portal because you can stack up your data year to year or against neighbors or whatever and really get a good view of how you compare with everybody else. Once you determine what your strengths and needs are, then we suggest getting into TE using the NRS tables and taking advantage of features like drill down, right clicking, so you can really get to the bottom of areas where you're strong and areas where you're not so strong.

So here is a screenshot of the CASAS Data Portal. This is what it looks like. If you've never been there before, again, we show the goals. We show the averages. And you can see where it says Agency, we whited out the agency. But you can use it to add your agency totals or other statewide totals, county totals, and so on, and see how you compare across the six levels of ABE, well as the six levels of ESL.

I'm not going to get into gory details here, but you can see you can stack it up. You can look at areas where your percentages might be better, areas where your percentages might be worse. So here I'm moving quickly, just to show-- here's an example with the red arrow. That might be a below-average area that we would want to target ASE Low.

Don't worry about all the details for now, but it shows we're at 22.6. The goal is 37 or 39 this year. So you can see that's an area where we're way below average. That might be an area where we need to improve.

Just to keep the concept going, ESL Beginning Literacy is an example where we're strong. We're above average. We're doing really well. It includes a pretty good number of students.

So that would be an area where we might say, yeah, we're on the right track with that. So we're all going to have some areas where we're doing really well and others maybe not so well relative to how we do with others, relative to how we do across our other areas. That's how we can decide our areas to set as goals year to year.

So here's an example with persistence. So we look at performance data. We can also look at persistence data. We'll talk about different ways of measuring persistence with the panel. But in federal reporting land, that's just looking at the percentage of students that each level that stick around long enough or do, quote, unquote, "persist long enough to at least complete a pre-test and a post-test."

We've done lots of workshops where there's alternative ways of looking at persistence, but the NRS data just looks at the number of students who qualify for each of those 12 levels and looks at the percentage of those who persist with the pre- and post-test. So here is another way of looking at that same concept.

We've got one example, ABE Intermediate High, where we're not persisting very well, and another one here at ASE Low where we are. So in this example, this is presuming that we've got two areas where our performance is low. We're looking at targeting it this because we're a little disappointed in our results, so we talk about looking at persistence as a first way to go, where, if we know we're particularly weak in an area, we evaluate whether we've been able to achieve persistence, yes or no.

So an example of where it's "no" is with this red arrow. We're performing weak and intermediate high. We see that we also need to improve persistence. So this is a good indicator that that's really what we need to look at first. We need to get everybody tested and/or improve our demographics or instructional hours data first and then start looking at improvement in the classroom.

The alternate example is ASE Low, where we know we're struggling. But you can see by this graphic, persistence does not explain it. Our performance is weak, but our persistence is actually pretty good. So we know we're not going to get that far by correcting it. We thereby want to look at other reasons. Typically, we want to look at reasons in the classroom.

So what we want to do then is evaluate our data. This is just an example of looking at it longitudinally. Once we identify areas where we know we want to improve, and we start looking at some of the reasons or areas we need to improve in order to get that level up, we can start digging in our data.

We can dig down on the Data Portal by looking at longitudinal results, look to see whether it's always been a problem spot, whether we've improved over the years, whether maybe it was a good spot earlier, and for whatever reason we've gone down, lots of different explanations. But we want to look at the long game here and kind of develop a short-term and long-term plan for how we might improve.

The other big issue with persistence is some of you have probably been to these workshops, where we talk a lot about this basic 2 by 2 diagram, where, once you determine performance high or low, persistence high or low, we've got this quick little graphic that kind of helps us point out what we need to do in order to improve things.

Long story short-- again, I want to be quick and turn it over to the panelists-- is if we know we need to improve and persistence is also low, that suggests we need to improve our data and our testing. We need to fix demographics, hours of instruction, and get everybody pre- and post-tested.

If our performance is low, like in that upper left, but persistence is actually pretty good, then we know we've gotten everybody tested. We know we've done pretty well with our demographics and instructional hours data. But yet, our performance is still low. So that means we need to start looking in the classroom in order to get those levels to improve.

So this graphic is just a real generalized way to show everybody what steps they might need to take to improve that particular level based on what our data is telling us. So that was a very quick-- that summarized about six hours of NRS Local Performance Goals workshops in 5 or 10 minutes. Hopefully, that told you something instead of nothing. I'm not sure, but I wanted to provide a really quick overview for everybody, just so you know what we're talking about overall.

So here's the outline. We talked about those first two. The second bullet is when the panel introduced themselves. That first one is what I just got through doing. So we're all going to have our moment in the Sun with each panelist, starting with Steve, where he's going to talk about using NRS levels and how it can apply across the board in different areas. So I'm going to turn it over to Steve now. I think you might be muted.

Steve, you're muted.

Sorry about that.

There you go.

Sorry about that. Yeah, so over in Huntington Beach Adult School, we've been-- actually, I've been really into the data tables that I look up on Data Portal. It was a kind of a-- I won't say surprising, but it was something that I had wanted to see, some kind of data, statewide data and local data, that was in many ways absent from some of the other programs I was overseeing before that.

And so when I started looking into the data, I saw exactly what Jay was pointing out, is some data that kind of interrelated with persistence and outcomes. And it was a concept that, at that time in our programs, was not really connected at that time. And so it happened to be also right that year, the next year, we were hit with the Great Recession. And our district was in the news, I think one of the first ones to make some major cuts. And I had to face some difficult decisions in terms of layoffs and figuring out how to preserve our program.

And so what I did was-- and me being new to the school, I didn't have a good feel for all the staff. And I based it-- our layoff decisions outside of what our district had negotiated with unions-- there were some flexibilities. There were some things that I had to decide in terms of staffing, and so I based it on persistence. And that was one of the easiest ways to preserve our program and to look at our teaching staff and to ultimately protect the most number of students.

And that was a very difficult time, but that became the year when our data became the driving force in a lot of things that we would do in that department. And then later on, when I became principal, schoolwide now, a lot of people didn't like that approach. But at the time, it was what I felt the best time and really turned our school, our programs into more data-driven entities.

And so we have built in, through those years, processes around reviewing the data every year, both at the administrative level, and then also at the department level, where I think it's more critical to do so. And in the review, we looked at performance levels by class. And that means by teacher.

And it is a scary-- at first, it was kind of a-- there was a pushback. But really, the idea was if we're going to figure out how to do better, we need to see who's doing best. And then much the same way that Jay has picked this panel, and I'm honored to be on this panel, that he's recognized of our data that it's doing well. And we do perform well in our outcomes.

I did the same thing with our department, and not for pointing out that someone's not doing their job, but more importantly for pointing out those who are doing well. And how do we learn from them? What are they doing?

And most of our meetings look at, who are our top performers? And we have them lead discussion groups around persistence, how do they get the persistence up, and then what other strategies they use in terms of curriculum and targeting specific questions. I think somebody else will get more specific about all the different reports you can get at CASAS. But really, looking at the data and then deciding, making curriculum decisions from there.

One of the things that I've always been a fan of is looking at low-hanging fruit. And what are the easiest things we could do and address and focus on as a team, as a school that do not cost us a lot of heartache and headaches? And so one of the easiest ones right at the beginning was persistence. How do we improve our persistence? We invest in a lot of persistence, so that was one right off the bat.

Next thing we did was we looked at, how many paired scores are we getting? And so looking at the number of students that we have in our program, and tracking who's missing a paired score, and looking at strategies for those teachers that had the best performance in terms of paired scores, that was the next step. And so we continue to look for those things that were easy to identify and track, and then actually implement some strategies to address.

And so like I said, we review-- ESL department is probably our high performing-- I would say our high performing-- that's where we tend to produce the best outcomes. That's where they have regular meetings, and the teachers are discussing the outcomes and performance levels, sharing ideas, breaking into groups as far as-- by level, and then having the best performers lead those discussions. I got some questions here. I want to check to make sure--

There is one question, Steve, from-- let me go back out.

Which great teacher--

Isn't there a risk at misalignment when you are setting goals based on two-year-old data?

So yeah, you could look at it that way. But the-- I won't say it's accurate. One of the things that I've always been-- I've gotten used to, but it took some getting used to was data where the outcomes or the results of the data in terms of revenues and grand amounts was based on two-year-old data.

So there is the issue. We're looking at data on an annual basis. So at the end of the year, when you submit your data, whoever your CASAS person is can give you your TOPS, your Table 4 report, Table 4B, Persistence Reports. You don't have to wait for the data posted on the CASAS website Data Portal. You have that data live.

In fact, we run those reports quarterly to look at it. Sometimes, they don't get it into the department because it's a little fuzzy data in terms of stuff's not cleaned up. You haven't made sure you got all your data integrity information cleaned up. But we can look at the data right after we end the fiscal year. But more importantly, we do look at persistence, which is live data, or attendance persistence, the pairs scores report.

One of the other things that we did early on is as a leadership team, we decided that no student would enter a program, would start a class, and get any instructional hours until they had a pre-test. So that was a mandate in both the diploma program, or ASE program and ESL. That we took care of right away, so that's not an issue for us pretty much anymore.

And so with that, there's another question about, are the majority of teachers part-time, full-time? Like most adult schools, majority of our teachers are part-time. We have invested in full-time teachers.

I would say, we have about the same always. And that's probably-- I would say, a fifth of our teaching staff is a full-time, 20%. So I try to keep it right around there in all our programs if I can, just to build some consistency, some long-term commitment, and some folks at that I can go to consistently and talk about data from the current data and then years past data.

And so let's see here-- anything else . We do implement a testing calendar that everyone, and I mean everyone, from teacher to classified staff-- our receptionist knows about it. Things change during test week when we do test students for our post-tests and for EL Civics.

And so that's been something that's been very helpful because support staff-- and I know somebody's going to talk about them later-- they're critical to this whole piece, and they need to be in the know, what's going on. And that's where-- as far as communication is people calling in. They want to be aware if it's testing week. They want to know what kind of questions they're going to be getting so they can prepare for that.

And Data Integrity Report-- that is definitely a support staff report that we review on a quarterly basis, maybe-- yeah, on a quarterly basis, maybe. Maybe after the-- not until December. Or do it December, and then March, and then June, near the end of June to make sure all our data is clean. And then we're getting all the date of births and all the requirements that-- or all the data that's reported in the NRS table.

And we're always looking to beat the previous year numbers. So if we're missing a certain percentage of date of births, we make sure the next year it's less than the year before. But most of our NRS data is pretty clean, pretty good.

And I think with that, I'm going to conclude and hand it over to Carol for the next--

Hi. So in Stockton, we've been collecting and really looking at our data since around 2003. And every year, what we would do is we would-- in the summer, we would put our percentages for the 12 program areas or the 12 educational functional levels. And then we would put the performance goal that the state had established.

The only data that we would have six months later would be comparing ourselves with all of California. But we have our data, so we can-- before the school year starts, based on our previous school year, we are able to create our goals by looking at our previous data. We could look at also how many students are in each of the different educational functional levels, and that actually triggers conversation with our faculty and our staff, our staff because we want to make sure that our data integrity is accurate.

So I'll go into a little bit more about how working on this kind of data and the performance-based payment grant for WIOA has influenced the way the school, the faculty work with their professional learning communities.

So I could go ahead and talk about this. So when I share data results with staff, they are actually super critical in this whole process. They are really our rock stars, our office staff. We have a school secretary, and we have six student data technicians. So they are the ones entering the data.

And they are more than just the student data technician. They are also the registrar, the attendance clerk. And they are our number-one customer service right off the bat, the counter, phones, email. And so they are constantly looking at how registration, TOPS Forms, test data is entered into our MIS.

They look at that Data Integrity to make sure that it is accurate, and they support teachers and the students so they can ensure that there's a pre- and post-test, and how we schedule make-up tests. They also support students in making sure that they know how to register for the GED or HiSET exams. They're looking at accurate attendance, TOPS Forms, and data entry, so critical.

We share quarterly and annual data results, and we exchange feedback with each other. I think the other critical thing that's really important is to support your office staff and send them to professional learning opportunities, like CASAS Institute, or professional organizations like CCAE. They also have regular staff meetings. And sometimes, they'll attend a faculty meeting with teachers and counselors and the administrators. And of course, reinforcing daily communication is essential.

So those are some of the things that we think are really important things. Oh, OK. And then sharing data results with teachers and counselors, we can start with any weekly-- looking at in the TOPS Enterprise system for monitoring reports that the teachers can have so that they know which students have paired scores, which students have completed EL Civics, which ones have had learning gains, those kinds of things, and what areas and gaps that they might have within an ESL class.

It is much more difficult in our adult basic Ed classes and our adult secondary education classes with a high school diploma and GED because of so many gaps. And also, we have found-- and this is more of a subjective observation-- that our students in our ESL class are very diligent when they take the classes test.

Some of our students-- not all, but some of our students in our ABE and ASE programs-- they kind of just go through the test, and they're done in 15 minutes. And so we try to encourage them and remind them that the reason why we're able to offer these classes for free is based on our learning gains and student performance, so we do talk about that. Looking at quarterly reports and the annual reports, and of course the professional learning community data team process-- and I can talk about that some more later on too.

Oh, OK. And then, on this one-- the reason why I shared this page is here's a typical day when they come back to school. We look at the data from the previous year. We not only look at the WIOA data. We look at course completions, high school diplomas, and number of GED certificates. And then we interpret and compare the data. We celebrate the data.

But here's the kaizen part of continuous improvement. We ask our teachers to have this discussion. What are factual statements that can be communicated based on this data? What are some of the questions that you have about the data? What do we celebrate? And what do we need to improve for the next year?

So we're doing this day one, when all of the faculty come back together. And then, of course, they set up their data teams, and they set up their format in Google Drive and their team goals in Google Drive. Every other week, they meet in a faculty meeting. And then they meet an PLC for 90 minutes the alternating weeks. So basically, our teachers are meeting together every week.

So that is actually a really nice feature in continuing professional learning communities. It is actually part of the teacher contract. Our teachers are part of the teacher union, and they're full-time teachers. We also have part-time teachers or actually hourly teachers that work for us during the day or in the evening.

So those are some of the things that we talk about. And looking at communities of practice, this one year, we looked at creating learning spaces. How can we improve our learning experiences for adult learners by changing our classroom environment? So it's just some things to start that discussion for our faculty to engage. So that's just one example.

Oh. And the reason why I showed this on the Table 4 is that when we look at this data, because we chart it, and you'll see it in the next slide, we're really low in the ABE/ASE area. We really struggle. We don't struggle in the ESL area.

But let's take, for instance, Beginning Literacy. It's at 25%. The state goal is 55, and the state average is 54. And we're at 25%, and we have 4 students that were in that educational level.

So we can look at how many hours those students were in, but it isn't a priority as much as when we look at the highest students, like the area that-- ASE High. If you go to ASE High-- or excuse me, ABE Intermediate High, we're at 40%. And-- oh, wait a minute. I meant to go to ASE High. OK, so the state average is 43, and we have 132 students that were in that area.

But the other area that I think is significant is if you go to ABE Intermediate High, we have 387 students. We were at 40%, and the state average was basically 42%. So those are some things. They're close, but still we want to pay attention to make sure that we're maximizing our opportunities for students. So we also look at our numbers, not just the percentages.

But we do compare our percentages from year to year to make sure that our teachers have an understanding that we're not just looking at goals, but we also look at the California state average. We think that we want to be above the California state average. So we pretty much just state that right out.

Hey, Carol, there's a question from Serina, asking whether all teachers, full-time and part-time, are included in your weekly PLCs.

Yes.

If you do, how do you arrange their schedule?

Right. So it is scheduled every Tuesday for our full-time teachers. Our hourly teachers are all invited to attend, and we pay them. Some of them are able to attend.

But if they work in the evening, and they also are at another district site, they are not able to participate, because they work in the K-12 system. During the day, they have to participate in their PLC or their faculty meetings at their sites. So we don't get 100% hourly and full-time, but we get 100% of our full-time faculty at these meetings.

Thank you.

You're welcome.

Jay, can I make a comment about ABE and ASE? Those--

Yes.

--also lower-- those are areas that we've been, I've been looking at for a few years now, trying to get those numbers, because our performance is typically low in those areas too. And I think-- and I'm not sure. Because obviously, if you have a state average, you have some folks that are above the average, doing well. I'm always looking for folks that do have those higher averages, see what they're doing differently.

I don't have solid answers right now. But I know one reason we always get pushback from our ASE teachers is that the students-- and, I think, Carol said, they're not as interested in taking these tests. They don't really have value to them when they're working on a credit economy. They just want to get their classes completed, their credits, and they want to graduate.

So that's always a challenging conversation with our staff, and trying to integrate the testing into the program is not as seamless or as easy as it is in ESL. And then the other compounding factor is that our ABE students and ASE-- they tend to be more the independent study participants. They're coming and going. They're not staying for a direct instruction, and so it makes it harder to start tackling some of those standards in the instruction.

They're basically taking courses that are designed to get them through and get them through history, and math, and science as fast as possible. And so just throwing that out there. These other folks are struggling with those areas because they are-- ASE program does present some different challenges. And I think it might be tied to the format of the program in terms of how we deliver instruction to students.

OK, thanks. For Steve, for Carol, or any of the panelists, Linda asks, what specific data are you analyzing with your data teams?

Linda, that's a great question. So it is kind of evolved. So I was just looking at the agenda for each one of the different years on how we got started with the PLCs. Formative assessments are critical for teachers to do some comparison and getting feedback on the teaching and the learning that is going on in the classroom.

So yes, we look at both. This is the annual data that we look at and quarterly data, and then we look at the monitoring reports that are set up in TOPS Enterprise. But as far as day-to-day assessments, formative assessments are really critical. And those are teacher-generated that they would have to do. Some of them use a lot of electronics now. They will use an app to get some feedback before the students walk out the door.

In our--

Thank you.

In our program, we started off-- that whole idea, the low-hanging fruit-- we started off first with the persistence, improving persistence rates in our programs, with the idea of being-- like Jay had explained that that's a good indicator or it could be an indicator to better performance or lower performance.

So really making sure we had good attendance practices on the teacher's part-- what were they doing? How are they interacting with students, welcoming students at the beginning of the day, making sure they knew their names? Things of that nature can really encourage others to keep participating, keep attending classes.

Then we moved on to paired scores. Where are we getting the paired scores on? And many of our ESL classes-- follow that responsibility, in many ways, falls on the teacher who is administering the test, who is encouraging students to show up on the day of testing, and then following up on those students who passed.

And so we started focusing on paired scores. That definitely relied on more data because they were tracking the students and reports that we've given. These students are still missing a post-test, for example.

Then we moved on to the individualized reports, where teachers would be given-- here's this list of the number of students who are missing-- or the highest missed question-- forget it, whatever the name of that report is.

But TE will give you reports for your teachers. Which test question are the students missing the most? And teachers can go back and just try to hit on those questions, knowing that it'll benefit the most number of students and hopefully get a few more points to get in that game.

All right, thanks. Carol, are you still on this one? Or are you on the next--

No. No, please go to the next slide--

OK.

--quickly. And I'll go through these slides quickly. So here's an example of-- since 2003, this is how we set up our data, using the EFLs on the left-hand side. And then we have Stockton School for Adults, the Performance Goal of the state, which is on the Data Portal. And then, usually in January, we can see how our performance stacks up against all enrollees in California. And we put that data in.

So this is a good comparison for us to see where our areas are. I have to say, though, we do struggle between ABE and ASE. ESL has never been an issue. So you can go to the next slide.

And so this is really important to look at your performance-- Payment Points Summary. And there are lots of areas. These are questions that we ask our teachers to take a look at to see how they can make some improvements. But yeah, it's-- so like for EL Civics, how many have one assessment, versus how many have two, and how many have three?

So the following year, we would say, OK, we want a little bit more of students that have one assessment completed and passed. We would like to increase the 400, and then see how many can do three per year. So it's those kinds of things that teachers have the conversation and how they schedule it, and how they schedule the EL Civics participation in the lessons. OK.

Whoops, sorry.

Oh. And the reason why-- OK, so we really like this Persister Report because we believe that learner persistence really does help your learning gains. So you can go on the Data Portal and look at your Persistence Report. And this is just a basic one for WIOA. And it has it by the years, and it goes by percentages.

And then you have the final one in the final category. And of course, the goal here is 67. We always like to be above 70% because we think that that's really important. We don't want to go below 70. So CALPRO has some amazing learner persistence, communities of practice, and resources there. I highly recommend you do that.

And then here's a California Adult Education Program worksheet that correlates to your Summary Report in CAEP. So all you have to do is put the data in based on the columns, and then you just do the computation. And you will come up with your persistence rate. And you want 70% or higher, measuring persistence and just literacy programs, program persistence. And it goes in these columns, and then outcomes performance.

So this is a really valuable worksheet. We did this one worksheet in our consortium. We haven't really done data teams with our consortium as a group. Thoibi may be able to explain a little bit more about that.

But what I did like is some of the principals went back to their school sites, and they started looking at this, because it's the discovery and the discussions with your faculty and your office staff on, how can we make this better? So this worksheet I believe came from Jay and CASAS, so we just put it all on one page.

And one of the things I'd add also is that when you look at the data, sometimes you scratch your head. Why did it go up? Why did it go down? And keep in mind that a lot of the data or all the data is based on student behavior and how they're attending classes.

And so in years, when the economy is doing really well and everyone's working, that's when we start seeing our students kind of more-- just, it's harder to get them that post-test because they're in and out of classes. And they're just busy. And then in times when they're not as busy, maybe they're not employed, then we see a little bit-- a difference.

So there's a lot of factors that go into this. And so I learned over the years not to be too critical with our teachers and push them too hard, especially when it's the students and what's going on in the environment that's impacting them. And they're making the right choices from their perspective as far as income, and so forth. So just keep that in mind. There's a lot of factors that impact our data.

Great.

Yeah. And so what I wanted to share in something like this is something that was set up for our data team process. This is one teacher, one section of ESL. And so what we do is we want to analyze the data, and she describes what she's analyzing, create a SMART goal. What are her instructional and management strategies? What are the results? And then the review date, and then a reflection.

So you will notice that the time period went from August 2019 to November 2019. We try to get in at least three or four data team cycles, where they specifically look at data. And each teacher will identify which area that they really want to see learning gains. But we feel that when teachers focus on this, it's really helpful. And this is some of the discussion that they have at their professional learning communities.

So this is an example of one. If you go to the next one Jay, this one is for ESL. This other one is for-- oh, this was for mathematics. This is our ASE adult secondary ed. teacher, same kind of data, wanting to make sure that there are some level gains, and then doing a SMART goal analysis, looking at the outcomes, like low attendance, transportation, housing. So documenting all of those kinds of things about their students is really important.