Jay Wright: OK. Hello, everybody. Hopefully, everybody can hear me OK. I'll blow up my screen now. OK, hopefully, everybody can see the blown up screen, and everybody can hear me just fine. And you can see, it's a power-packed agenda.

Everybody is probably wincing as they're looking at this now because it's, how can you fit this much goofy text on one slide? Cut it in half. And it's probably still way, way, way, way, way too much.

What we're going to try to do is talk a little bit about both quantitative and qualitative. You see the words there. So we'll spend the first half talking quantitative, looking at some TE reports. Again, the purpose here-- we just have an hour and a half-- is not to get into the gory of TE details. That's like six or eight hours we don't have, but there's about 10 or 12 suggestions all of which relayed around TE reports.

So the idea will be not to necessarily give you step by step how to follow every step in TE or how to follow every step in your agency's data system, but to just give suggestions on reports you can use, goals you can set, approaches you might want to look to isolate, key areas of your data. And then, we'll use the same general approach for qualitative. We'll review some information about performance and persistence that my hunch is most of you have seen many, many times, looking at agency and student level strategies that again, looking at the participant list, I know many of you have heard about many, many times. But we'll start looking at some things you can do at your agency and your consortium.

Also, again, in the name of ideas, some of them will seem like wonderful ideas, some of them probably not so much. But if you give 10 or 15 quantitative related ideas related to TE reports and so on and roughly the same number of strategies you can implement with your students and staff, there you go. You have something. Hopefully, you'll find a few good nuggets in both sections as we go.

The other thing I'll bring up is there's actually three handouts. This PowerPoint you see is where we'll spend 99% of our time. If we have time at the end, which I have to admit I'm pessimistic about, there is some information that relates a little bit more to what I'm talking about Thursday actually than today. However, my understanding is I have an hour and a half today and just an hour Thursday. So there is one that I probably will have a tough time getting into Thursday, so I'm throwing it out there for right now.

And then there's another one that really goes specific on barriers to employment and ad hoc reporting. We'll talk a little bit about those concepts here. But there's one that gets a little deeper, you might say, in the TE weeds than what we have time for now.

The last thing I'll bring about before I dive in is there will be more detailed sessions at the CAEP Summit. There's a panel discussion. Again, from looking at you, a couple of you might even be panelists before, but a lot of you have been to the panel discussions we've hosted.

The panelists do an overwhelmingly outstanding job in talking about this goal setting concept using TE incorporating students, incorporating staff. So there's a really good session that we'll be doing at the summit with the panel on some superstar ways of looking at this. There's a session I am doing where I'll dig a little bit deeper than I do today into TE and crisis-- and so on. And there's a couple others where we're looking at rallying TE data managers and administrators. So there's a bunch of sessions there.

The final thing I'll say is Veronica and I just had a discussion yesterday on what strategies we need to look at at CAEP TAP for all the various ways of looking at goal setting. So if any of this looks especially interesting to you, or I guess if anything looks especially God-awful, keep that in mind. Let me know. Let Veronica know because we are looking at what do we need to really map out as the specific training ideas probably for January, February. There's a lot of little nuggets we're talking about here that probably fit into that.

So without further ado, let's start. So starting agonizingly enough with NRS table 4, this is just the table set I realize it's a CAEP training. But we're talking a lot about the concepts of NRS performance and persistence. So we need to do a quick overview of table 4 and 4B, just a table set, and give the basic definition.

So here's a screenshot of NRS table 4. A lot of you have seen this already. Hopefully, my understanding is, this is an advanced session, so this one should be familiar to almost all of you. This says what California and all states use for federal reporting.

This is the one a lot of people would say is the biggest plan because it's the one that relates to measurable skill gain. From our point of view, specifically, it captures all the students that make gains from pre-test to post-test. It also has information about high school equivalency and high school diploma.

I'm not going to get into too much gory detail here, but the way it's set up is we've got the six levels for ABE, six levels for ESL. The pre-test is what puts the student in one of these levels. The gain between pre and post is what determines whether they make a gain. You can see columns E and F basically relate to measurable skill gain and the high school diploma and equivalency, respectively.

So columns E and F is where we're looking for our payout. That's where we're looking to show student success. We've got the percentage shown in columns I and M. That's basically looking at each level of all the students that qualified for that level.

What percentage of students at each level made that measurable skill gain? We're looking at that percentage. This table is for WIOA II funded agencies. However, there is a table that is there for CAEP. It's in the CAEP reports as well. I think there's an example or so.

So if you don't mind them, hold that question in about 15 or 20 minutes. It is one of the examples. This specifically is for NRS, but there is a carbon copy we use for CAEP that basically mimics the table exactly. The difference, though, is the one for CAEP will incorporate-- workforce, prep, and CTE, and so on where obviously the one for NRS does not.

OK, these look like-- yeah, well, for this one, yeah, that will actually-- the agent-- well, this agency level, probably not for this NRS report. You would want to have agency level again at a specific agency for the NRS. If you're just looking at the CAEP side, then you would be able to access this. Again, this is more for concept just to give everybody the basic concept of what we're talking about with the NRS performance and persistence.

So, Mitch, you'd have access to CAEP. NRS, though, you wouldn't need to have agency level access for one of your particular agencies. The pre-test required is a longer answer than you think, Kelly.

In general, the short answer is yes, a free test is required to get on NRS tables, period. The caveat that makes it a little longer than you're expecting is because of COVID. There's that federal memo from May of 2020 where they changed that. To get gains, you must have the pre-test, still. But just to get on the tables, period, since May of 2020, you can self-report learners into these levels by using that self-reported level in TE.

And again, that was a change the feds came up with just over a year ago, given all the obvious issues with being able to test everybody during COVID. I have not seen any updates to that since May 2020. So as far as I know, that caveat still is the same as what it was from a year ago.

So hopefully-- that was a lot of questions at once. Hopefully, that got everybody's question. But again, just to keep it so I can move a long. Table 4 is what we use to measure performance.

Then we have table 4B, which is-- looking more at persistence, we have the same 12 federal levels, but this is just looking at only those that complete a pre/post payer. So we're looking at just those that have a pre/post payer to more measure classroom performance.

When we're looking at federal performance overall, we must use table 4, that is, we're incorporating everybody that qualifies for federal reporting. From an instructional point of view, a lot of people would say the only fair way to do it is look at only those that complete a pre/post payer so we can more fairly evaluate instruction. So table 4B gives you the same data, but only looking at those that can complete the payer.

And then, the other one I wanted to just point out is the NRS persister. Again, this is another one used for federal reporting, but it's comparing table 4 to 4B. That is, it's looking at the percentage of students on table 4 that also make it on 4B. Looking at it more simply, of everybody who qualifies for federal reporting at each level, what percentage of students stuck around long enough or persisted long enough in order to at least complete a pre-test and a post-test? That gives us that percentage of persister.

On column D of the report, that is, table 4, we can look at all 12 levels and look at performance level by level. Persister, we can look at those same 12 levels and look at pre/post test persistence level by level. So I bring this up just to get the baseline.

When we're looking overall at NRS performance and NRS persistence, this is what we mean. We're looking at those that make gains on NRS table 4. We're looking at those that qualify-- of those that qualify for that, how many at least complete a pre-test to the post-test.

This is just one of many ways to look at it. But when we're looking at the one way for federal reporting, this is what it is. So I'm just going to give-- I'm going to stop there.

Quick sanity check. That was a lot. There were a lot of questions. I think I got them all, but let's quickly do a sanity check here.

Everybody hanging in and for the three or four of you that had questions, especially, does that wrap it up? Does everybody feel like they've got their foot on the rail at least to get started?

[sighs]

OK, not seeing too much. I'll have to assume no news is good news. OK, foot is on the rail. Thank you, Maryanne. You're giving me the lifeline. I'll move forward.

So we're in a CAEP training, so let's transition now from the basic definitions for NRS. Now, we're going to go ahead and move on and look at it in terms of CAEP reporting. So here's that CAEP summary.

We've talked a lot, and again, most of you have been to a lot of these sessions, I know. So you've heard us talk about the CAEP summary in terms of those three sections-- left-hand section, middle section, right-hand section. For the record, this is the left-hand section. We're looking at literacy gains, pre/post.

Again, this is getting the information directly from NRS table 4. That's another reason why I started there. But again, it's looking at the enrollees in column B. That's basically the same concept as what we looked at in column B for table 4.

Column C, that's basically giving us the number of enrollees from table 4B. That is, of those who qualify in each program, how many of those stuck around long enough to complete a pre-test and a post-test? And then, column D, that's looking at that performance metric of everybody in each program that has the pre and post. How many of those actually made enough of a gain to achieve an EFL gain on table 4?

The red flags. OK, that's a little bit the whole presentation. I guess, that's about an eight-hour answer, so I'm not going to do that off the cuff. But I'll just say we're going to have 8 or 10 or 12 quantitative suggestions that all get your question in different ways that by no means exhaust your question. But I feel when we get to quantitative and we rattle that off, that's going to basically give you a few ideas related to that question.

So, again, literacy gains for pre/post. This is summarizing our pre/post information and maybe more specifically, summarizing this as it relates to that and NRS reporting. The consortium-- yeah, you can look at the CAEP summary with that consortium point of view. Again, you can't drill down, but yes, you do have access to the CAEP summary.

OK, looking at next, we've got the middle section with outcomes. So this is where CAEP starts veering a little bit from federal reporting, where one big outcome for CAEP reporting is pre-post, but we've got several other categories. We've talked about immigrant integration, and then, again, those six areas of AB 104. We have those other literacy gains, like occupational skills gain and workforce prep milestone. We've got secondary, post-secondary employment, wages, and transition.

So this is looking at those other areas in which students can achieve outcomes. So that requires a lot of the same information that it required in the left-hand section. The big difference here, like we've said many times, is the left-hand section requires that qualifying three test. The middle section does not relate to testing, so it doesn't require anything as it relates to testing.

And then, over here on the right-hand section, we have the CAEP services. The bar is a lot lower here. You don't need hours. You don't need demographics to qualify here. You don't even need class or program enrollment.

So this is a way in which we can capture all of those other students that maybe don't stick around long enough to enroll in a class, that really don't enroll in one of your official CAEP programs. So number one, this captures all of this and all of these students that might be receiving short-term services. At the same time, it's a way to capture some of those students that maybe didn't have anything to do with services but were looking at enrolling in a class or program but didn't really stick around long enough to do so. So it looks at people who enroll in services and/or those that were fly-by-nighters that you did do something instead of nothing but maybe not enough to get 12 hours or complete an entry record or whatever.

OK, then the final thing that I wanted to just over view on is the NRS, or is the CAEP Data Integrity. We've got the-- I'm going to get more into this and a couple of the quantitative, and we'll get into the basic definition Thursday. But you can look at the CAEP DIR.

There's also these new CAEP Hours Reports. One is called Cape Enrollees by Hours. The other is called Cape Service Enrollees by Hours. I'll just apologize in advance that those titles are pretty obnoxiously similar, but they're both basically ways to inventory your learners by what we're calling the three buckets. We've started to make a lot bigger deal about the three buckets because NOVA seems to-- looks like it's going to be forcing this issue a lot when you're doing the CAEP data or a goal setting in NOVA.

I won't be getting into that. Yeah, I don't think we'll be exhausting the subject because it's morphing. But yes, we'll be looking at some of those examples here very shortly.

So one example your question is at a good time. This is what these new reports are for, is one big state priority is being able to look at those three buckets. That's probably my word, not NOVA's. But in NOVA, it's looking at the students that have 12 or more hours, that is, those that are participants versus those that are in 1 to 11 hours versus those that are services only students with zero hours. So you've got the participants versus adults served versus services only.

You can see here, it gives those buckets 12 or more hours, 1 to 11, and 0. It seems innocuous, but it is a way in which CAEP is looking at goal setting. That is, ideally, you're going to be moving as many students as possible from the zero hours bucket to the 1 to 11 hours bucket.

Similarly, you're going to want to be moving people from 1 to 11 hours to 12 hours, that is, those students that might show up for short-term services. Maybe, it's just students that are kicking the tires on some of your adult debt classes, a lot of different reasons in which they might have for doing so. But you're looking to capture more students and get them enrolled in instruction and get them to do the 12 or more hours. So they're enrolling in your class so they can start moving toward career pathways and so on. So this is breaking it down into those buckets in a little bit more detail.

The enrollees by hours focusing in on those again that are in that services section. So we look at some of those with services, a lot of those are presumed to be those that don't have any hours. But a lot of times, it might be students that have some hours, or maybe students that even have 12 or more hours but just didn't enroll in an official CAEP class. So what this is doing is getting into a little bit more detail so you can see why students might be in that right-hand section instead of those middle sections, where you can get more information to figure out exactly how many hours they have, exactly what level of effort it's going to take with each student to move them into more detailed instruction or more detailed enrollment.

The service enrollees by hours has the same basic purpose. But instead of looking at those three buckets of hours, it's looking at the specific services received. So it's turning that other report inside out, looking at that same right-hand services section that we're looking to improve upon. But instead of detailing by the three buckets, it's looking at those that actually receive supportive training and transition services, respectively. And then, once it determines which specific services are involved, then detailing it.

So one is assuming probably, hey, we're not really doing services. This is detailing it so we can look and see, what is it we did that brought those students in so we can move forward just like in that other example.

OK, I'm going to do another quick sanity check here. Hopefully, those new hours reports at least made sense, a little bit quick transition. So they can be either one. They are-- by self-reported, I'll just assume you mean manually and putting it into TE. So yes, you can manually input it into TE.

A lot of you import services from a third-party system. It's not technically on the update form, but it is on the supplemental update form. So I got to say not many people do bubbling and scanning overall. The list that does bubbling and scanning for this is much smaller because it's a special form.

So I'll just say, if you really want to technically, there is a form that has this, but again, it's a separate form, so you don't do it that often. I'll just say, for services, probably the majority, bring it in from another system. But there is a way to type it in manually. There's also a way to bubble and scan if you want. Hopefully, that's your question, Crystal.

So getting into the quantitative and qualitative. Again, the idea here is to just give you an overview at high level to equip you with some ideas. There's lots of detailed trainings here that we could do to hone in on just one or two ideas and really get into the weeds. But for now, we're looking at several ideas that you might consider for goal setting using a quantitative approach and then roughly the same number of ideas using a qualitative approach with the theme across everything being goal setting and different ways of attacking goal setting and program evaluation.

So we'll look at quantitative. The process is the same. It's just what you're dealing with.

So on the quantitative side, we have some of those targeted data reports. So a couple of you already brought up some of them, so we're going to use a few of them randomly. Just to suggestions, there's lots more, of course, you can use. But the ones you're suggesting are some of the examples that we'll go through.

So we're looking at using some reports and looking at the neediest areas and then looking at hot spots. That is, we look at a report. We know maybe there's some programs where we're doing well, other programs, we're not doing well.

If it's looking at classes, there always are going to be some classes that are doing well, some other classes that are not doing well. So it's a generic concept, but I really do think it's worth reinforcing that those hot spots are always what you're looking at when you're looking at any data report, whether it's NRS, CAEP, payment points, assessment to instruction, and so on. As you're looking for those hot spots, that is, you're looking for where you find those really strong areas and those really weak areas and figuring out which classes are strong and which ones are weak, which programs, which students, which federal levels, and so on, and then rinse and repeat. Go to those targeted areas, look at those areas of strength and areas of need, start looking for areas of commonality.

When you find those particular strong and weak areas, we'll try to reinforce that concept with the examples. And then, toward the end, we'll look at qualitative examples, where we'll talk more about feedback from staff, feedback from students. What are some things we can put in place to establish clear channels of communication?

And then a lot of that rinse and repeat, where once we figure that out, how are we going to go ahead and identify our strengths and needs from a qualitative point of view? What can we really build on at our agency or our consortium to adopt and really have everybody follow an area that we already know is strong or the other way? What's an area where we know where we're weak? And where do we need to build in order to get a little bit better? And then, overall, we're developing that culture of data, whether we're looking at it from a quantitative or a qualitative point of view.

So here, we're looking at how to align with the continuous improvement plan. That's the CTE plan for WIOA II to be clear. But we've talked a lot more in CAEP land about how we're looking as much as possible to align that step with the goals-- with that CAEP goal setting.

A lot of the concepts that we brought forward with that CTE continuous improvement plan are basically the same concepts we're looking at with CAEP goal setting. So for one, a good way to kill two birds with one stone is go back-- if we're WIOA II, go back and look at some of the things you looked at with your continuous improvement plan. Maybe not all of them will be one to one, apples to apples that you can also do with CAEP, but it's been pretty above board that if you can keep some commonality or consistency with your CIP, that's something that's officially something you're encouraged to do. At the same time, that would be a way to make it easier.

But just some ideas there, we're look at in NRS performance goals, ways you can align to your WIOA and CAEP partners, ways you might want to align with your last visits or your CTE federal program monitoring. There's ways to look at special programs like IET and lots of leadership project resources, CALPRO, OTAN, WestEd, and so on.

So back to quantitative, this is what we just talked about. It's a placeholder to try to get into some specifics. So I just talked about it. So I'm going to move on.

We've got some different ways within CAEP reporting to look at this concept of goal setting. So these are just ideas. There's many, many other ways you could do, but these are the specific ideas we'll touch on, is look at enrollment and participant criteria.

We put that one in bold because it's very clear that's one that everybody is going to have to address whether you like it or not. Some of that relates to the fact that that's where TE and COMS are basically the same. We're finding more and more than it tends to be different more often than it's the same quite frankly. But one way that we know, it's basically the same as when we're just looking at enrollment and that concept of participant versus adults served. So we'll look at it that way.

I'll just say, for now, I don't think there's any state level benchmarks that I know of. I don't think we have Carolyn or Gary or anybody like that here. If anybody like that is here, you could come out. But sorry, I got to say, that's a better question probably for Gary or Carolyn.

In any case, we'll look at enrollment and participant criteria with that being involved because we know that something everybody is going to address. There's ways where we can evaluate our pre and post-test results from a CAEP lens. We'll look at targeting CAEP-related outcomes. We'll look at using the CAEP Data Integrity Report.

Another suggestion is we'll look at it from a straight NRS point of view. For some of you, that'll be a great approach, others, not so much. But certainly, we're going to put NRS federal reporting related activities on the table.

Another thing you might want to consider is looking at special populations, that is, looking at individuals with specific barriers to employment. Maybe looking at that concept of equity that lots of people have been talking about, or maybe just in your region, there's a specific population maybe at a specific ethnicity. Maybe it's individuals with disabilities. Maybe it's a special workforce population, but all kinds of ways of looking at special populations. Sometimes looking at it from a consortium point of view to make sure you're going hand in hand by consortia source instead of by agency.

And then, this last one is generic, but another one is defining goals with external stakeholders. Sometimes, that might be just looking at it as a consortium between college and K12s. Sometimes, that might be measuring yourself with WIOA partners. Sometimes, that might be looking at it with programs like IET, where you're relating your agency information to employers, things like that.

So the first one we're looking at is enrollment criteria. So this is going back to that new report we just mentioned. So we've table-set it because we are looking at this as an example.

So in this case, we're saying, of everybody that shows up on the CAEP summary, how many students actually make it into a CAEP instructional program? So we're looking at these arrows and saying, OK, we've got this number of enrollees total. We've got this number and column G that have the 12 or more hours.

So we can combine those other two and say, OK, we've got 609 with 12 or more hours. We've got about 246 that don't. So you've got these numbers and each bucket with the 12 hours, the 1 to 11 to 0. So there's different ways you could use this simple report and say, OK, we want a certain percentage of that 148 to get at least some hours. Instead of being services only, we want to move people from columns H and I into column G and get them enrolled in official CAEP program.

These are just dummy numbers here. But you can look at your agency numbers in this report or the consortium numbers and figure out maybe some specific programs where you're looking strong at their programs where you may need to improve, and just look at moving more students to have 12 or more hours. Maybe ESL, like in this example, is looking good. Some other programs are not looking as good, so we want to move some students into that column G. Lots of different ways to look at it, but this is just using enrollment criteria to get at least 12 hours to move them from that adults served to participants bucket like we've been talking about.

This is another way of the same question of everybody that we have reported for CAEP reporting overall. How many make it into a CAEP program? Again, the same basic concept. We're looking at having official enrollees rather than service only.

OK, more participant criteria. This is just looking at of those who enroll in a CAEP program, how many stay long enough to accrue at least 12 hours? So just so it makes a little bit better sense, go into the previous slide. I'm sorry.

So go into this one. You could do it either way. You could say, of services only, how many have at least one hour of instruction? Or you can say, of those that have at least some hours of instruction, how many persist at least long enough to have 12 hours and qualify for federal reporting?

I'll bring those up. Those are both very innocuous goals. But I'll just say that basically outlines what you're all going to be required to do, is you're going to be able to show that you're moving people from zero hours to at least some hours, and you'll also going to be, one, able to show you're moving them from less than 12 hours to at least 12 or more hours. That's the universe of everybody there, where you can look at enrollment criteria to get to monitor and get more hours of instruction globally.

OK, I'll just say any questions? I'm trying to just look at it overview. I know I'm getting into weeds no matter what I do, so I'm just going to stop. Does that at least make sense to everybody?

I'm not sure if this is going anywhere or whether Jay is just talking to a brick wall. So just for sanity, let me stop. All right, thank you, a fraction is at least saying yes.

All right, sorry, I'm being a little bit of a pill here. But I can't quite tell whether any of this is going or not. OK, thank you. All right, you're given me the warm, cozy, if nothing else.

So next topic relates to pre and post-test. So again, we're just looking at eight different ideas. Some of these ideas will be good for you, not all. So some of you might really want to focus on testing, depending on what consortium you are and what programs to feature.

This might be an obvious no-brainer for some of you. It may not be your bread and butter for others. But one thing you might want to look at is setting goals for pre/post test results. That's what we're doing for federal reporting.

So this is looking at the pre/post federal reporting way. But instead of looking at NRS, it's adapting it to the CAEP summary. The bottom line difference is this incorporates any testing you might be doing with non-WIOA programs like Workforce Prep and CTE and so on.

So this all hinges on that left-hand literacy gains section, obviously, of the CAEP summary, so we look at those three columns. We've looked at this before-- number of enrollees that qualify for federal reporting in column D. Of those that qualify, how many complete a pre/post payer? And C, of those that complete a payer, how many actually made enough of a gain to complete a level for federal reporting in column D?

So you can see those numbers in those three rows in this sample. So we're comparing-- so one thing we can do is, we're looking at the number who have a pre/post payer. That would be persistence. That would be one way of doing goal setting, or we can look at the number of who make an actual gain. That would be a way to look at performance.

So again, we're looking at that literacy gain so to get the persistence rate, or simply doing that column C divided by column B. So we can simply do that to get persistence. I'll just note 70% is my informal rule of thumb for good persistence. Obviously, last year, that's a little different. Then I'll stick with 70% from a historical point of view.

The other thing you can do is monitor performance. That would be column D divided by column B. So again, if we want persistence, or just saying in column C, what percentage in each program completed a pre/post payer?

For performance, we're looking at column D. What percentage of students in each program actually made that gain between pre and post-test? So again, this is the basic way of doing it for pre and post testing.

I'll say, this is the way we've looked at it from a CAEP point of view because number one, it uses the CAEP summary instead of NRS report. So it's easier for you to access. Number two, it's incorporating all 7-K programs. So if you're really looking at it fairly from your CAEP programs, you'd want to look at everybody in CAEP reporting, not just your WIOA II students.

So I'll say again, that's my second example. Everybody still clear on this one? I'm going to go example by example here. But still it only-- so sorry, what do you mean by that route?

Speaker 2: I just mean that effort gains can only get us so far with measuring performance and persistence. That's not-- that's just a side note. The tables tell me that--

Jay Wright: OK, meaning that they might need to possess more than just making a gain or getting a correct posting.

Speaker 2: Correct. Correct. Yeah. Yeah.

Jay Wright: OK. Got you. OK, I'll just say, yeah. That's just one way of measuring it. So we've had two ways of measuring enrollment, I guess, from a CAEP point of view-- one based on enrollment, one based on free post.

So the third thing we're bringing up is just targeting outcomes that's probably more for performance than persistence. But this relates to that middle section of the CAEP summary. Again, we've got that one new one we've talked about for a year or so. The I-3, that's using EL-7 co-apps for immigrant integration. And then those same tried but true represented columns G through L, 1, 2, 3, 4, 5, 6 columns, representing each of those six areas of AB 104 that we've talked about for quite a while.

I'm going to stop right there, do a bird walk for a minute. That's, one, an area where probably it would have been better if I did my Thursday presentation first and this one second. But just the way it is, my Thursday presentation that gets it. Work it 101 issues is the other way around, but we'll talk more about AB 104 on Thursday, just FYI.

But again, we've got those six areas that the California legislature has required us to follow for adult education. Those six areas are columns G through L. That is the areas that we're monitoring for student outcomes.

So in terms of using those outcomes-- sorry, my slide stick in-- we want to go ahead and just take a look and see if we can tally those numbers with each one of those columns. So we pared it down just to look at a couple to keep it a little easier, so we can look at specific outcomes.

So in this example, just for the sake of argument, we're looking at that column math, the ESL learners who passed I-3. We're keeping it simple and just assuming that for immigrant integration, we're focusing on ESL. That's not going to be true all the time, but for this example, we'll say it is.

So we'll just say, yeah, we've got the 2,200 ESL learners. About 1,000 of them have made some kind of I-3 outcome. That is, about 1,000 of our 2,200 ESL learners have passed at least one co-app. So we can just go ahead and take that.

And you can see my math is wrong, but the everything else is right. I'm not sure where I got the 67. New math, so everything's right here, except for the math. But you can just see you can look at-- you know what, I don't know why that math is wrong.

But anyway, I can visualize it, but I can't calculate it, I guess. But in any case, well, that's a good reason why we'll update the PowerPoint after the presentation. But we can just look. We might want to look at it as ESL only. We might want to look at it for all students, but we can simply keep it super simple here.

The percentage of students by program, how many made and I-3 outcome? We're just saying, OK, right now, we're right about 50%, give or take a little. So, hey, in a year, we want to have it be-- we're at 65%. I'm just making up the numbers.

To get here on the CAEP outcome section, it's only looking at those with 12 or more hours. So the CAEP outcomes is using that same criteria-- is that left-hand pre/post test section, that is. It's using the same criteria as NRS reporting, except it's completely ignoring everything related to pre and post testing. Hopefully, that makes sense, Crystal.

So, yeah. In order for it to be an official CAEP outcome, they need to have at least 12 hours of instruction. I'll just say that's one that was a big issue originally. But I think that one was resolved four or five years ago, where here, everybody agrees that it needs to be a participant with 12 or more hours of instruction.

I'm being a little loose here, but in general, I think the consensus was that in order for you to really report it as an outcome achieved by adult education, were basically saying, well, yeah, 12 hours isn't going to be perfect in every situation. But it's a good loosey goosey cut off to say, yeah. For us to say, yeah, the reason for that wonderful outcome is because of adult education, that 12 hours is the cutoff to where we can safely say, yeah, we're claiming that outcome.

A big reason why the student did that is because they enrolled in adult debt enrollees. Right, it's only those with-- I'm not sure. There's a little double negative in your question, Eric, but back to Crystal's question. Everybody who shows up in the south-hand section must have 12 or more hours of instruction. I think that's what you're saying, Eric, but you can confirm or deny but to you, Eric, it requires 12 hours. OK, so, Eric, Crystal, hopefully, that answers your questions.

But again, we can just look at an individual outcome, is another way to check it. What I'll just say is the thing that I think what somehow got lost in translation here is another thing you could do that we've brought up in the past, is you could look at it for an individual outcome, or you could target the number that made all outcome. So another way of looking at it is you can look at that total number of outcomes. And you could just aggregate the number here, where you're looking at column F through L and aggregating that total number of outcomes and comparing that to the number of enrollees and say OK your average enrollee made more than one outcome or less. We can look at it that way as well.

OK, so let me see. Moving away from outcomes and looking at the DIR, there is an update that I know some of you are looking at. We haven't quite gotten there yet by doing the updated targets with the end of year numbers. So I'll just say, moving here, there is a link that we did in April that has this updated up through the third quarter of the '20/'21 year. Sorry. For right now, that's still the most recent version we have.

In mid-October, after we finish the matter of the deadline to the feds and all that and getting the data to CDE for WIOA II under the CAEP office brigade, we'll have to wait and get all that done before we can follow up and finish things like the DIR numbers. But we should have it by mid-October. But you can look, and there's a document that gives you the statewide averages for all 27 items on the CAEP DIR, so you can compare your agency or consortium percentages to those averages. I'll just say, if you want something recent, you could compare it to third quarter of '20/'21, or you might want to compare it to end of year for '18/'19 or '19/'20.

I'll just say, as an aside-- throw that link-- this link. OK, hang on just a minute then. OK, hang on. OK, you have a meet-- OK, hang on. It's not wanting to go away from full screen. OK, hold on. OK, there we go.

All right, anyway-- so again, this has the 10 quarters. What I'll bring up is I'm making a lot of excuses here for why we don't have that updated document, obviously. But one question that is relevant is, when you get those end of year numbers from '20/'21, the quite frankly, there is all of you probably are going to be looking to improve mightily from most of the numbers we have on '20/'21.

I'll just stop here and just sanity check everybody. Do we all understand? OK, you did it again. OK, hang on just a minute. Why don't we-- well, we'll do this at the end of the presentation. Sorry.

I'm just going to wait. That was a little more time than I have. Anyway, maybe if one of you would be volunteering to do it. That might be a lot better than me pasting it into the chat.

In any case, where was I? So does everybody know-- sorry, I'm talking about way too many things at once. In any case, does everybody understand off the cuff with what-- without lots of detailed explanation why the end of year numbers from '20/'21 may be a lot different from the numbers we're hoping to get here in '21/'22? Let me start with that, and that's going to get me back on track.

Does everybody know what I'm referring to there? Or do I need to dive into that for a minute? You can chat or talk. Does everybody understand why we probably don't want to take the '20/'21 end of year numbers too seriously as it pertains to goals we're setting for the '21/'22 year? I'll just say, does everybody know what I'm talking about right now?

OK, there we go. That's our Corina. Thank you. All right, somebody does.

Right, so '20/'21, we know is probably the worst year we've had on the one time. If we have performance in '21/'22 like we did in '20/'21, we know things are turning really, really, really, really sour. So in a lot of ways, I've got to say, I'm not sure if it's fair to say, yeah, let's just use '18/'19 because there's a lot of reasons why maybe we're not going to be quite as good as we were in '18/'19. That might be a little bit of a reach.

But we're certainly hoping we're a lot closer to '18/'19 than we are to '20/'21. That's another really good discussion topic that we probably don't have time for now. But I'll just say, take it with a grain of salt. We've got all of these quarters of statewide DIR performance '18/'19, '19/'20, and '20/'21.

But take it with a grain of salt. We're not quite sure what this one's going to be. We definitely want to avoid just saying, oh, we'll improve on '20/'21. There's a term I use called setting ourselves up for failure. You probably heard me say it.

So this is the exact opposite of that, where we might be a little too close to setting ourselves up for guaranteed success. If all we're doing is improving on last year, we probably want to do a little bit better than just improving on last year. Sorry to be a downer there in a way, but Ryan, Todd, Karina, you're all bringing up good points to that.

OK, so here's just an example where we're just using the '18/'19. I might be setting the bar too high here, but hey, we have-- in this example, we'll just say, we're at 34.3 students with less than 12 hours of instruction in the '18/'19 year. So we're saying, yeah, we're going to improve from 34% to 24%.

So in this case, we're saying, yeah, we're going to use that '18/'19 number because that's really the metric that we want to improve upon as the last year that wasn't affected by COVID. Again, just an example. Easy way to do an example.

OK, so there's four examples we've given you. The fifth one is saying, hey, let's just use straight NRS address. For those of you that have mostly WIOA II agencies, you might want to just take advantage of NRS reporting, things like the CASAS Data Portal that give you a lot of additional resources.

So hey, if you have a lot of NRS agencies, you might want to take advantage that you can use the Data Portal, a lot of more user friendly ways to look at data, a lot of ways to drill down into these different programs. So in this example, we're just using the Data Portal to generate our agency level information up through '19/'20. So we're just cutting to the chase and saying, yeah, we're looking for federal levels.