Jay Wright: OK. Thank you very much, Veronica. I did screen share. I guess, for sanity's sake, I'll make sure that everybody sees the screen in the manner in which they would expect to see it. That is, we've got the slide with the list of four presenters. Is that what everybody sees at the moment? Yes or no?

Thoibi Rublaitus: Yes.

Jay Wright: OK, great. Just making sure. I've had a history of messing up screen sharing before, so I'm making sure it's not messed up here.

So we have our all-star panel. We've been joking we've done this quite a bit now, at least five times. I think I've done it seven times, although maybe with a slightly different lineup each time. But Thoibi, Carol, and John have been the most frequent fliers, I think, on these NRS performance goal panels. They've both appeared multiple times.

So what we'll do is I'm going to start off with about five minutes of overview at this point, where another part of our joke is we're thinking everybody's probably had a chance to do this one before, because we did it at last year's CAEP Summit. We've done it at a couple of COABE's We've done it just for kicks on the CASAS website. We've done it in several different ways with the same bill. But here we go.

Just out of curiosity, have you-- any of you here willing to admit that you're here for a second showing? You've seen Carol, and you've seen Thoibi. You've seen this presentation from start to finish, but you just couldn't resist coming on for a second time? Just wondering, because we have done this a lot.

In any case, this will be the order. But I guess, I'll start-- Oh, yes. Frequent flyer. OK. Somebody's sticking up-- Thank you, Dana. So I'll keep going here. And I'm babbling about nonsense and realizing I better move on.

So anyway, here's what we'll talk about. I'll do an overview. We'll talk about some of the NRS issues. I think probably by panelist is better than the agenda though.

I'll just say I will do an overview of that whole NRS performance goal concept for those of you that have never heard this. There is a minimum amount of knowledge that I really do think does help in understanding what the panelists are talking through. So I will do five or 10 minutes of goal setting 101 to get started.

And then Carol will start off. Her specialty will be engaging staff in this process. And then Thoibi will pick up and talk about different ways of engaging students. And then John Russell will bat cleanup. And he'll basically work more from an external point of view, that is, how to look at this in terms of collaborating with your CAEP consortium, collaborating with your WIOA partners, and so on.

So I'll stop talking, and I'll just say-- Yeah, it was off-Broadway right? Yeah. Now we're on Broadway. We've got the gigantic marquee that you all were able to see via vFairs. So, much bigger marquee this time than what we get for other conferences for sure. OK. And I'm trying to keep myself from talking, so I'm just going to click and move to my first slide, here, to plug myself in so I, mercifully, get moving.

So the whole idea here is called NRS local performance goals. This is a topic that yahoos like myself have been talking about for 15 years now. The feds have been imploring all states to do this for about that same amount of time.

That is, we're looking at NRS Table 4. We're not going to get into the gory details of that right now. If you don't know all the gory details, you should be fine. But Table 4 is the federal table that looks at the 12 federal levels-- six for ABE, six for ESL.

And it basically looks in terms of measurable skill gains. With WIOA there's multiple ways of dealing with measurable skill gains now, but historically, that relates to making gains between pre and post. What we've been talking about for years is looking at these levels, identifying your strengths, identifying your weak spots, and identifying those areas of need, and looking at a game plan to try to improve in the areas here that represent areas in need of improvement.

So I'll just say, is it just Chris that's having the screen problem or is it everybody?

Veronica: I believe it's just Chris.

Jay Wright: OK. Just making sure. All right. So I'll move on then, Chris. Sorry. It's just you. I'm not sure. I think maybe you need to log back in. Yeah. It is a black hole anyway, though, whether it's our screen or something else.

So anyway, you can use the NRS tables and TE to identify strengths and needs. You can also use the CASAS data portal. When we look at these examples, we'll use the data portal. We do feel like it's a little more user friendly. You can look at multiple years, multiple agencies, on one screen rather than spreading it all out on the kitchen table, so to speak.

So this slide-- again, we're not going to spend a lot of time on data portal 101 here either. But here's the link to the CASAS data portal with an arrow showing here's the areas that we're looking at. These are the links that you would click to get into the areas we're showcasing here this afternoon. And so what we say overall is we suggest using the data--

[garbled voice]

Veronica: Jay? Jay?

Jay Wright: Yeah.

Veronica: OK. Go ahead. Sorry.

Jay Wright: I unwittingly got muted, I think. Anyway, so what we say is use the data portal when you're troubleshooting. Somehow-- I don't know if I can-- OK. I can't go back. There we go. Use the data portal when you're troubleshooting, and then use the NRS tables and TE so you can drill down once it comes to figuring out ways to make improvements and so on.

So the process-- here we go-- is, again, we're looking at the data portal. We're reviewing our NRS performance across all 12 levels. So here's what the data portal looks like from a recent year. You can see it's got the goal for the current year. It's got the goal and average from the previous year.

So what we're doing is we're comparing our own agency's performance to the statewide average and the statewide goals from the current and previous year. We can go blow by blow and we can see, in some cases, the percentage is better than the average. In other cases, it's worse than the average. So for starters, we're really just looking and seeing which areas are we looking good, which areas are we looking not so good.

The one quick data portal 101 I will bring up is the number in parentheses at the bottom are the number of students represented. So we're, obviously, looking at a large agency here. So, for example, we have 147 ABE beginning lit students. We have 265 ASE high students and so on-- or 71 ASE high students and so on. Again, the numbers in parentheses are just showing the number of students at each level.

So, again, we're looking at our strengths and needs here using the CASAS data portal. So just getting into a brass tacks sort of example, we're blowing it up so you can see it a little bit better. And we're skipping a few steps. But in any case, we're showing you one good area and one area in need of improvement.

The green arrow to the right represents an area in which we're doing well. You can see we're at 63%. The average is at 60%. So that's an area where we're looking pretty good.

On the other hand, ASE low with the red area arrow-- we're well below the statewide average and well below the goal, so that suggests an area where we might need to improve. So, again, look at how far below the average we are. Look at the total number of students in that particular area. You might need to use some local factors, as well. But look at the degree of need and how far below the average we are when we're looking at targets.

So here is-- once we've identified our areas of strength and area of weakness, another good thing that we like to look at is persistence. So when we do the longer presentations here, we'll look and compare performance to persistence. Persistence, in terms of NRS performance, means, of those who qualify for federal reporting, what percentage of students stick around long enough or "persist" long enough to at least complete a pre-test and a post-test. There's different ways of measuring this based on testing, based on attendance hours, based on lots of other things. But just know when we're looking at federal reporting, where WIOA II land anyway, persisters relate to those that complete a pre-test and a post-test.

So what we can also do with the data portal is look at our persistence rate-- so that's what this slide is looking at. So we've already noted these two areas-- ABE intermediate high and ASE low-- are both areas in which we need to improve in terms of performance. They're both bad areas for us. So we're looking at these two bad areas and comparing it to the persistence rate.

So over here to the left, you might say we're sort of seeing what we're expecting to see-- that is, a lousy persistence rate. I'm saying that's what we're expecting to see, because we've already determined that our performance rate is not that good in ABE intermediate high. So this is confirming our suspicion where low persistence seems to be the main thing that's contributing to low performance here.

On the other hand, we're also underperforming in ASE low, but you can see our persistence rate there is actually pretty good. It's not great, but it's definitely not bad at all. It's well above the statewide average.

So the point here is, sometimes we're performing below average, but persistence isn't necessarily the reason. So on the left hand, red arrow, that's an area in which persistence probably is the big explanation for our low performance. Where, over here to the right with the green arrow, there's some other factor, potentially, that seems to be contributing to that low performance.

And then, again, what we've talked about is applying strategies at the agency and the student level to improve our pre/post persistence rate and strategies at the student and agency level to improve our overall areas of performance. Kind of oversimplifying it here is what we've looked at in previous sessions-- again, we're rolling an hour and a half into five minutes here-- is, at the end of the day, in all these different areas, persistence is either good or bad. Similarly, in all these different areas, performance can be good or bad.

So, again, at the end of the day, that allows us to sweep it nice, neat, and over-tidily into four areas on this quadrant-- that is, persistence high or low, performance high or low. Generally speaking, we're looking at the left-hand side, where we're looking at those areas when performance and persistence are both low. And then other times, maybe where our performance low, but our persistence is actually pretty good. And then we get into lots of suggestions, based on these areas, with strategies to try to move our way forward based on our persistence rate in tandem with our performance rate.

OK. So that's my quick little overview just to look at it from an NRS point of view. Before I move on to Carol, I'll give a little plug that we'll do this review very quickly again and apply it to CAEP data collection in a session we're doing Thursday, I think at 8:30. I believe Carol and Thoibi are listed. Carol and Thoibi are welcome to join me on Thursday morning. But by definition, they're not required to attend.

The one I'm doing Thursday will be the deeper dive where we'll look into all these and spread it out. And the emphasis will be using those CAEP reports, not necessarily NRS, given that it's the CAEP Summit. So I'll push myself off the cliff so I stop talking and yield the floor to Carol here. You better jump in, Carol, or I'll think of something else to talk about, and we'll never get out of this mess.

Carol Hirota: Thank you, Jay, very much. Thanks.

Jay Wright: I will slide click.

Carol Hirota: OK. Thank you. So on this slide and during this portion that I'm talking about, it's really a focus on data and also the accuracy. And it's also the who. It's the process and the procedures that are part of this system, but it's about the people who make this system work. So it's a lot of kudos to my colleagues out in Stockton, who have been working on this year after year and continue to do this.

The first thing, obviously, is your customer service, to ensure that your counter, your phone, and your email services, that you are responding to your students continuously. And how you register them-- you want to make it really friendly, using-- whether it's online or they're walking in and doing their registration.

And how you're entering everything in on your TOPS forms or in your system, into your management information system-- that goes over to TOPS Enterprise. Your data integrity-- that report is critical. It's having everybody's eyes on data integrity, on the accuracy of what you're collecting.

And if you're not collecting something really well, go figure out why and how can I do it better. And every time-- it's a Kaizen, continuous improvement. What can I do better?

So you want to also support your teachers and, mainly, your students to do pre and post testing of CASAS. And so having makeup tests when they're not there is really important. And I know our office staff coordinate that with the teachers and the students.

Supporting your students on registering for their exams online, whether it's a GED or HiSET and also-- I learned this from another consortium-- I believe Iris is on this call-- she shares-- she observes students that are testing, and they're going to be taking the GED test. And if they do not have the skills, she lets that teacher know, hey, you're going to really have to work on these computer skills for them to take the essay part.

So that kind of collaboration is really critical and, of course, your accuracy on your attendance, your TOPS forms, and your data entry, sharing your quarterly and annual data results, and exchanging feedback back and forth, really looking at that data together. You can't be afraid to look at the data and say, oh, I don't want all the teachers looking at each other's data. Everybody looks at the data.

You want to also support your professional learning. There's CASAS Institute. There's CCAE. There are CALPRO events and OTAN. Attending all of those is really, really good.

The critical thing is communication-- your staff meetings with your office staff and your faculty meetings, which include your teachers and counselors. And, in Stockton, their teachers have a professional learning community. They have that built into their collaboration time, so it works really well. So they are meeting weekly and, of course, daily communication.

So, on the next slide, it's really important for the data team people, the office people, your assessment people, your data people to work very closely with your teachers and counselors. You need to share those data results in TOPS Enterprise. There are weekly, monthly, quarterly, annual monitoring reports that you can share with your teachers-- quarterly reports, annual reports, and, of course, using the professional learning community.

If your teachers are using a PLC process, you can embed a data team process within that system so they are constantly looking at their data and what they're doing to help students learn. The whole idea is student learning-- not just what teachers are teaching, but how are students learning. And they rely on all of this data that they receive from the data technicians.

Next slide. So part of this system is setting this up at the very, very beginning. And I use this. This is from 2017, but this has been going on for several years.

This is the first day you meet with your teachers. You start looking at your summative data, and you ask things like, what statements can you make about this data. What does-- what is communicated about this? What are some questions you have about the data? What can we celebrate, and what do we need to improve for the following year?

Let them ask these questions, and ask more questions. And exchange information, so that they're looking at the real data together. Then, afterwards, make sure you give them time to set up their data teams within-- and in Stockton, they set up their data teams by ESL, CTE, Adult Basic Education, Adult Secondary Ed, and high school equivalency-- all meet together as one. And sometimes they could meet together all together, but meeting separately helps them look at their data and look at their data goals. And I'm going to share some of that with you in a few minutes.

So the next slide, we're going to be looking at some of the WIOA slides that we look at. Data Table 4, you can easily just put all of the dates that you want so you can compare from year to year. So, on this one, you can see that Stockton-- it's up to 2019-20, but we've done it for like the 10 years.

So you know exactly which areas are doing really well, and you compare it with your state goal and the state average. Because we want to know, not only did we reach the state goal, but how do we compare with other schools in California. So that's really critical. And I'll show you a local report that we use later on.

The next slide is Persister Report, critical in making sure that your students are there long enough so that they are able to demonstrate some learning. So what you want to do is take a look-- and we always use 70% as the minimum for our students-- and, as you can see, our beginning literacy areas and some of our-- and that's for ABE-- are some of our lowest. So what are we doing with those 7, 4, 17-- those first students? Critical.

And then there were 45 students that were at 60-- what-- 69% on beginning basic and ABE. What did we do with those 45 students? That's a lot of students to lose in an Adult Basic Ed program.

So taking a look at all of that-- and, Thoibi, I have to thank you for your session that you had, because I think you've got some-- had some really good ideas on how we can take a look at some of that data and use it efficiently and more effectively so we can make improvements.

So the next report is a local report. And so this is-- and it goes up to 2018-19-- but what this report does is be WIOA. It's a WIOA chart.

So for Stockton School for Adults, it's their percentage in each of the program areas or the entering educational functioning levels, what the performance goal was for the state, and what was the performance against all enrollees in California. So what we want Stockton to do is make sure that they've hit all the performance goals and that they are above the California performance against all enrollees.

That is really important, because we want to make sure that we are doing our very best at this school to provide services for students. And it's critical. So using this is really important for our teachers to take a look at.

OK, next slide. That's a local one. The other one-- and it's actually a longer page-- but for your CAEP we use this as a program worksheet with the-- it's your manager summary, your annual manager summary, or you can use it for your quarterly data. You can take a look at it.

But you can see that there are Column C, Column B, your totals. Anyway, it's a template, and you put your numbers in there. And it will give you your persistence rate. It will also give you your program persistence. And if you look at the entire worksheet, you can also find out evidence of student learning, how many student learning gains there are.

But I believe this worksheet got cut off. It was something that was created from one of Jay's workshop that he got. So we put it in this template, and I know Delta C-- our Adult Education Alliance-- that that consortium has been using this. We've also been using this worksheet in our targeted technical assistance with our coaches when we're evaluating some of our consortium that we're working with.

So, anyway, it's good. So you look at persistence and also learning gains.

The next slide goes into a little bit about what the teachers do. I talked about professional learning communities and the teachers setting up their data systems. You'll see some of the data systems in the next few slides, but quarterly there's a data team system that's going on. And they put down their section number, and then they put their level of their-- what data they're going to be reviewing.

They analyze this data quarterly. They put their data in their SMART goal and their management strategies. And the fourth week, they do a reflection and comments so that they can do better the next round of-- the next quarter that they're going to be doing their data team process. But the whole idea is that they're checking on student learning, and they want to make sure that they are paying attention to what students are doing. So this is a formalized data team process embedded in the professional learning community.

I asked if Stockton was able to follow up on this during the pandemic, and it's been a little harder to do this exact same process. But, however, the teachers have all set up their data systems, which you will be able to see in the next few slides.

The next one-- here's another one where a teacher takes a look at-- flip back, please. They take a look at paired scores, learning gains, and their annual goals, how many students, their SMART goals. They analyze it. They have negative variables affecting the outcomes and the positive intervention affecting the outcomes.

Next slide. These are all the discussions that they have at their professional learning communities. And future interventions-- you'll see like, conferences, flexibility, emailing, communication, modifying expectations, following up. So those are really important kinds of things that we can do to have a positive impact on student learning.

Next slide is another one where teacher-- this was for high school equivalency. I wanted to share with you-- one of the things that I like that this teacher did is not how many students passed the GED, but how many subtests were passed. And so this was for the first quarter, wanting x amount of subtests pass-- oh, this would be for, by February of 2020.

So she would evaluate all of the subtests and where they were having areas of need and more support. So she wasn't just looking on whether they passed or not passed the GED, but each, every single test. So that was really critical. Her data system is very, very involved. And I don't believe I was able to make a picture of that one, because it's so involved.

But the next slide gives you an example of some of the things that the teachers do. They all have their data system. The one where you see the green shaded areas, those are the names of the students and the ESL-- additional assessments for the EL civics project that they're doing-- how many students passed some of these modules-- we call them modules, although they are additional assessments. But this is another way to demonstrate learning gains in your ESL program besides just your CASAS level gains when they take the test.

The other one is data on-- you can see what level, how many gains they made in CASAS, and if they completed anything, and if they had a drop code. So teachers are expected to have a data system so they are tracking their students and tracking their learning. And this is what they use when they have those discussions in those professional learning community sessions.

The whole idea is that this is a system where teachers work very closely with the data technicians. And they, obviously, have support from their administrator, who-- he makes sure-- Jeff Dundas and Bryan Wright-- they make sure that they have all of the technology and have access to all of the reports that they need so that they can take a look at their student learning. So it's really critical.

And they meet very closely with counselors. And-- just a little plug-- we have three counselors in Stockton that are working in the areas of high school diploma, high school equivalency-- one in ESL and one with the transition-- making sure that students get over to the community college or employment. We have found that having credentialed counselors have really complimented the team in Stockton. So, thank you.

Jay Wright: Hey, Carol? A question came through chat wondering how you calculate graduation rate when you're doing this.

Carol Hirota: How we calculate graduation rates?

Jay Wright: Yes.

Carol Hirota: So, there are several ways. We have one chart that just lists numbers-- how many students graduate or complete their high school diploma-- high school equivalency. So that number is tracked every year. The other way we do it is, if you look on Learning Gains, it will tell you how many students passed and how many did not. You can look that up in one of your reports. We-- you can use your manager summary for that, which would be your CAEP manager summary.

Jay Wright: Thanks.

Carol Hirota: Thank you.

Jay Wright: OK. So, anything else, Carol? The next slide is Thoibi's. Anything else you wanted to present, or should we go ahead and move along to Thoibi?

Carol Hirota: Yes, please, Thoibi. Thank you.

Jay Wright: Thoibi, you are on.

Thoibi Rublaitus: Thank you. Thank you. Thank you, Jay. And thank you, Carol. And, just a little introduction before we get into deeper-- Carol has shared a lot of great work that they have done. And every time I listen to her, I learn something new.

And what she shared was mostly from a bigger perspective. I am a principal at a school that serves about 2,500, 3,000 in good times. And we are trying our best to get back to our previous enrollment, as well. What I'm going to share, mostly, is the student-centered approach.

So, as the slide here says, we all, as teachers and administrators, do a lot to serve our students and to provide the best possible education that they need to meet their goals. But we are serving adults. And we have all learned this in our training courses and credential courses about adult learning as opposed to K-12 learning-- adults have different way of learning. And andragogy is what teaches us that.

Adults have a different way of learning. They have to be self-motivated. They need to know what they are doing. They need to build on their previous knowledge. They are responsible for their own growth.

They have immediate needs. And we have to serve their immediate needs, or they'll walk away. And we all know this, and we've heard this often-- that they vote with their feet. They also have problems in their life, so we have to understand their problems and focus on building on what they already have and developing the motivation that they have by coming and enrolling themselves in our courses.

So as we consider all that and work with our students, it is important-- we have seen in our practices-- that constant communication with them is important from the get go to get them to understand why they are doing the test they do. Because CASAS testing-- I'm sure you will all agree with me-- a lot of students, the moment you say test, they are already put off.

But if they know why they do the test with us, then they are more inclined to do it willingly. And so helping them to be the owners of their own learning by providing them the communication from the very beginning and also giving them the empowerment to track their own learning--

So, on this slide, you see those two sheets. The yellow sheet that you see on the top is a report card that our team created a few years ago-- it's about 5, 7, 10 years ago, now-- that we provide for our students. And that, in that report-- when we just hand out the report to somebody, some will look deeply. But not everybody pays attention to look at everything in detail. But they want to know what is in it for me, and when they write it down themselves, they are more involved.

So that report there gives students a worksheet that says, this is your goal-- CASAS test number one, CASAS test number two, number three, number four-- because we give test every quarter. So if the students get to write down there which test they took and what was their score; and then, when they do a post-test and they get to see, OK, I take the next test and then my score went up by three points, or four points, or 15 points, whatever; they get to write it down-- they set their goals for themselves.

Then the onus is not only on the administrators and the teachers, but it is a shared ownership. And so that has helped our students understand why we take the test and how they're tracking their own test. It also helps them to come back and then say to us, oh, I took the test, but I didn't take the other test. I want to take the other test.

Because, as a teacher, when we-- each student has this worksheet. And getting the students to write it down there, they're comparing with each other-- or they're at least paying attention-- and saying, oh, she's got three tests, and I got only one test. Why is that, right? Or, she's got a score, and I haven't got--

But that's not where we stop. I think we also have to get our students to understand and that we try to do this on a regular basis-- with our EL students, particularly, and ABSE teachers are also trying to do it in different ways-- is to get the students to also understand, hey, we are adults. We don't compare ourselves with each other.

We compare with our own work. What I did in the first test, I compare it to the second test, not to my neighbor. Because adults all have different entry level they come to, so they all don't have the same test. They don't all have the same beginning level. So that's another constant communication, as teachers and as administrators, we try to do with our students, which will help them to understand the why, what, and how.

The other thing we also try to do is with the teachers, getting the teachers to look at the data-- like Carol mentioned, the data team that meet, or the PLCs that meet with the data, on a regular basis. In the beginning, it was a little difficult, but over time, the teachers get to share data with each other.

And at the PLCs, during the data discussion, they have a discussion about-- or look at their own class report. So our CASAS data technician provides the data report for students, performance report for students, as well as teachers. The teachers get their own packets.

So the second sheet on the slide that you see, which is the white one in the background, is the one where the teachers do the analysis of their class report-- so where they look at the kind of test the students take, how many students took what test, and what was the areas of growth that most students have.

So by looking at-- so when I was a teacher-- this was a while ago-- when I was a teacher, I got all our students to complete their yellow sheet. And then I collect all the yellow sheets, and then I use them to then go through and look at how many students took test number 81R, how many took 81RX, and how many took 83R, et cetera, et cetera. And then I tally them and see how many students have the goals to meet for their next test.

So that helps me, to give me a data-driven instruction design. So I look at that and say, OK, 30 students took 81RX, and they all did not do well on-- or 50% of them did not do well on-- question number 18, for instance. And then we get to see, then, that, OK, question number 18-- the competency in 18 was to look at maps. So then I teach more on learning to look at maps for students.

Similarly, if all the teachers then go and all intermediate level teachers get together and they say, OK, these are the questions that my students constantly have a problem with-- and then they get together to then create a lesson that meets those students' needs, majority of the students' needs. So we're doing some kind of analysis using the worksheet and the white one.

Jay, the next slide, please. Thank you.

So all this is born out of the mission of what our school was, right-- to provide the learners more educational opportunities and to help them transition. So for all that, we prepare our students. And in this whole big arch of preparing our students to be productive members of the community, there is something that we have built in our school over time. It's a culture of change and the ethical rationale.

Let me preface this by saying that this came because I understand most of our EL students, especially, are immigrants from other countries. So a lot of them come to this new country, and they get free education, right? And the free education-- why do we get free education? You get nothing for free. And how can we get this free education, what does it mean for us, and what do we do with it, is a very big philosophical question, right?

And so, when I was a new teacher at Corona-Norco, I had just come from India. And I was shocked that our students are all getting free education, myself included, right? And then, the more we think about it, the more we realize that it's a matter of paying it forward.

So when we have to test students with CASAS, a lot of our teachers and ourselves included-- CASAS is tough. Why should we do it? Well, we have to do it. Why? Because we're getting free education.

How can we afford a free education? It's the funds that we get from the federal, and the state government, and the region, that allows us to provide this free education. And why would they give us this free education? It's because-- and students laugh at this when I share this with them.

Just two weeks ago, I was walking classrooms and talking to them the importance of taking their post-test, because we were having post-test last week. And so when I tell them, hey, why do you get to do the test, and why do you do the test, and how is it important-- so and then I share with them, if you do well, and you get a good job, you pay more taxes to the state. And that's good for the state. But then, that's not only good for the state-- that's where they laugh.

You pay more taxes. That's why they giving you for free classes. But that's not where it stops. It's beyond that.

People, I share with them, students two years ago did very well, and today you get free education because those students bring the funds to be able to teach you today. Similarly, you do well this year, and the students two years later will benefit from it. So we are paying it forward.

We're getting it for free, but for a good reason-- to improve the state, and the government, and our country together. Now this is our country, because we live here. And then, we are also helping more people to serve more people in the future. So this idea of civic responsibility is something that is important for our students to understand.

And this is also, this also plays out in ABSE classes, as well. If you tell the students, do the test, do the test, they're like, OK, why do I have to keep doing this test when my goal is to get a GED or a HiSET. But if I say that, when you do the test, you get to see your own improvement-- that helps you to grow. Even if you didn't get your GED, you already made some gains, which itself is a motivation, a pat on your back.

And then, similarly, from the teacher's perspective, our teachers can find out which areas of growth they need to focus on you, right? Because when you do the test, each question is related to a competency. And that competency is where, if you did not do the test, if you did not know the answer to that, then the teacher needs to help you on that competency. And that's why we need to do this test.

So this whole idea of changing the mindset, getting our adult students to understand why they are doing what they do and how it helps them, not just the school, brings us more performance and persistence-- or, at least, students doing the test. So that's the idea of this slide.

Also, from a teacher's perspective, the performance-based funding-- For some time, as administrators, we are afraid to let our teachers know about all the detail that goes into it. But, as adults, if they know the why we do what we do, why we have to work on the data, why we have to get outcomes so that we can get our performance funding, it helps them, as well as us.

Of course, we all know it's to help students to grow and students to get their goals. But, above all of this, to help them to grow better, we need the funds. And to be able to provide the support they need, we need more payment points. And to get the payment points, we have to get the CASAS testing done.

And those are some cultural discussions. It's difficult, sometimes, but sharing that helps everybody understand the why. Like, Carol always shares a list of professional development books that she uses with her team. Today, I didn't see it, Carol, somehow. That sharing, and learning together, and growing together-- one of the books she has on her list is Simon Sinek's, The Why, right?

So knowing the why helps everybody else to be more participative and having a vision together as a school. I think, with that, my part is up. Thank you.

Jay Wright: OK. John still isn't here, so I think I'll just move along and have to wear my John hat, here. That's just the way that one is, but when he gets in, I guess I'll let him take over. But I'll just keep talking.

This is the part from the Citrus Consortium, and it focuses on looking on the outside, working with other stakeholders, and also looking at the geographic region that you're in. So this is from his consortium, like official, annual publication here, OK? So this is what they're doing, by community.

A big part of this presentation relates to presenting information by zip code, where he rolls up the data, and gets these consortium and agency level averages, and compares the agencies by their NRS levels. So this first one is just looking at the NRS tables-- the six levels of ESL, six levels of ABE, ASE-- looking at the average, looking at each individual agency, and showing the agencies within the consortium, looking at the pluses and minuses there.

I'll also point out with this, as, hopefully, he hops in here soon-- but anyway, looking at this, he also is tracking which agencies in the consortium are going down, which are going up, and you know when, in particular, which ones are moving up and moving down in relationship to each other. This is looking at overall enrollment, looking at the different agencies in the consortium, and looking at, by percentage, when each individual agency, each individual community, which program comprises the bulk at each agency.

So you can see ESL is almost all of the population at the college. It's most of Claremont's. In other agencies, you can see it's not quite as dominant a program. So it's looking at, by program, which ones are most important by agency and by community.

OK. Not sure how I got there. OK. I guess there was more for John to present. And, obviously, he's not here, so I guess I'll really just have to open it up for questions. He has a lot more to talk about on this. I just have the slides.

I guess I'll just open it up for questions and see if he shows up here in the next couple of minutes. If he doesn't, we just might be minus a panelist here. Sorry about that. He was going to show up about 1:50 and come in for the cleanup role.

I'll just say, any questions here for anybody on any of this? I'll try to guide through the external part any questions for Thoibi, any questions for Carol, or anything for myself here. I was expecting a final panelist and didn't get it, so I'll just make it Q&A, anything about this process that you would like to dig deeper in-- lots of ways we could go.

OK. They're saying they're having trouble with that access. I'll just say, I've put my email in, and Carol sent it. There does seem to be an issue. Maybe there's something with this Zoom. But the area where I normally upload documents when I'm in my own Zoom definitely is not working in this Zoom.

I'm guessing TAP must have disabled that feature so people can't post during the summit. I'm not quite sure what that might be, but I'm not able to post the way I normally do that when I'm doing my own Zoom. I'm assuming that's a difference between the CAEP Zoom and the Zoom that I use from CASAS.

OK? OK, great. OK, thank you. Veronica's looking into that one. All right.

So, Carol or Thoibi, Neil's question-- have you been looking at any of this? I guess I'll say, Thoibi, from a student point of view, or Carol, from a staff point of view-- whether you've looked at any of these things in terms of the new CAEP three-year planning goals and all that?

Thoibi Rublaitus: So we haven't deeply looked into CAEP planning goals yet. But when we did our planning goals for our NRS-- the SIP plan, we'll be using a similar process. But, again, it's the same thing about looking at data as a team, as well as picking the teachers and staff, the data team, our data technician, et cetera, to work together on looking at what we have done, where we have met our state level goals, and where we want to go next, and whether it's practical to take 5% point above more as a goal or--

For our NRS, SIP goal, we were thinking that when we had 35 students passed the, achieved the diploma in the year of the pandemic, the following year we could make at least 70 to 80. And that was the goal we set. It looks like it's going to be still challenging, because the pandemic is still not over. But we have been able to get at least 15% so far. And it is quarter one, so there is still ways to feel quite optimistic that we may be able to meet our goals. So, similarly, for CAEP, as well, we'll be doing similar to what we did for our SIP goals. I don't know if that answers the question.

Carol Hirota: So, in regards to mandatory metrics for three-year planning for the CAEP, I think what we're going to be doing-- because we looked at this when we were doing our coaching-- is we looked at the CAEP manager summary. And on there, there are some really important things in there that document learning gains and different ways to document learning gains, as well as employment.

And then on the right-hand side, there's also services. So as we're trying to improve learning gains, we also need to make sure that we are always improving our services to support services to students.

So that one document, that one data piece, can really give a lot of feedback to your consortia, because you can look at it as a consortium or you can look at it as individual members of the consortium. And so I think that that's really critical, because in NRS and WIOA we have different ways of looking at student learning, which is-- there's, like Jay just shared, a bunch of information that's online. But with CAEP, I think we're just working through some of the new systems on how we are demonstrating learning gains, as well as learner persistence and number of students that we're serving.

But also, how are we serving our students with student support? So I think that's critical. Should this-- should we be holding each other accountable? As far as member effectiveness, I believe that the consortium members should all be holding each other accountable to make sure that we are serving as many students as we can, looking at the gaps that you're not serving in your consortium area, as well as-- are the students learning anything, and am I documenting any of that, and what kinds of support services do they need?