Jay Wright: Hopefully everybody can hear me OK. I put it up here on the cover slide, I'll just go straight to the agenda here. So as I believe Veronica noted it's part 3. I believe they go together somewhat, but it is three separate sessions. If you didn't do parts 1 and 2, probably not the end of the world. It's not mandatory or anything like that.

All three in some way or another relate to the CAEP reports in TE. We started on April 28 where we were less than TE and more just talking about outcomes. And most of that discussion was defining all the different CAEP outcomes and looking at it less on how you would market and more on what sort of student scenarios might transpire that would result in you marking specific outcomes and clarifying scenario x versus scenario y and what exact outcome. You know what looks like, quote unquote.

Then we did one just a week ago where we started getting more into TE reports. That one looked at targeting special populations. We focused on that a lot because of the goal setting activities. Everybody is going to have to do a NOVA here starting this summer, lots of focus on barriers. There's nothing mandatory on immigrant integration but it is one of the options.

So we started looking at some different ways of looking at, quote unquote, "special populations", looking at ways in TE you can isolate special populations. Identified some reports you can use that will give you that sort of information.

Now, fast forward to May 28, part 3 of 3 and we're going to talk about goal setting using TE for a goal setting. We'll start with the concept of NRS performance goals. My guess is that lots of you have probably attended those. Got to say we've done tons of sessions over the last year on NRS performance goals. Again, more on the WIOA federal reporting side than the CAEP side but we did a lot of them a year ago.

Quite frankly, we did it a lot of year ago because we were webinar crazy there for a couple of months when we were first into COVID. Then we did a bunch more of them here early this year as we were ramping up to that set that a lot of you completed a few weeks ago for CDE we went to recording. CAEP you might say is on the exact same bandwagon as we are with to moving more toward performance and goal setting.

So we'll be looking at a lot of those same concepts look at our performance goals. Only instead of the focus being on NRS federal reports, we'll look at the same concepts, but in the lens of CAEP state level reports. So we'll review NRS performance goals and then we'll start looking at how you might adapt those concepts to CAEP reports and CAEP reporting metrics. Then we'll get more specifically into different ways to measure CAEP performance and persistence.

That turns back the clock a little bit to the last in person data that we had in spring 2019. I know a lot of you remember that. That one we already, we did a bunch of trainings there spring of 2019 where we started looking at those alternative metrics, looking at the different sections of the CAEP summary, looking at enrollment to see who is an enrollee and who's not, looking at attendance to see who has 12 hours and who doesn't. That is a variety of ways of looking at performance and persistence from a CAEP point of view with students that may or may not have all the same data metrics as rest students would.

And then most of the time will be that second to last bullet, that is, we'll look at CAEP performance goals examples. My feeling is that by far the bullet everybody will have the most interest in or at least that's my prognosis. So hopefully, that's where we'll be spending the bulk of time.

And then for reference, we'll look at the 19-20 Statewide CAEP summary. It's not going to really be a data dive on that, but it is there and has been shared. So I think it's OK and basically we'll use that as kind of reference. We'll point out how some of these metrics look at the state level. There's all kinds of comparisons you could do, but we'll at least give you a couple for starters here by looking at that summary.

And so I'll stop talking. I'll look for a thumbs up or thumbs down again. Do you hear me? Do you hear me clearly? Does this sound more or less like you were expecting to get yourselves into this afternoon? Et cetera, et cetera, et cetera.

OK, there's a few thumbs up. That sounds like we'll say that's good enough here. So we're going to start.

That's a very predictable question. As predictable or more, we're going to start this presentation by talking about the CAEP summary fresh with a brand spanking new screenshot, not unveiled at other presentations, I don't think. Anyway, we're doing it into the three sections.

So again, we've talked about those three sections for four years now. So we're skipping the overview and digging right in, you might say. So we're starting with that left hand literacy gains section. Again, that's where we look at our three post outcomes.

The pre/post outcomes section for the most part comes straight from NRS table 4 or most of it does with the exception that it allows non-NRS programs. Again, column B, enrollees from the NRS federal tables, column C shows you everybody that completed a pre/post, and then column D of those who achieved a pre/post level gain. So again, the left hand section, just giving you a review here. Left-hand section is all about what's happening in terms of pre and post testing.

That the big meaty middle section relates to CAEP outcomes. You can see the number of enrollees in column E. I'll stop here, that should be reviewed for a good 80% or 90% of you. But let's just get everybody steet well, let's warm everybody up, insert your favorite cliche here. That's what we're doing here.

So again, middle section CAEP outcomes, you can see enrollees in column E, the enrollees in the CAEP outcomes section. Very, very similar to, but not exactly the same as that left hand pre/post test section. Both the left hand and middle sections require the demographics.

Both sections require the 12 or more hours of instruction. But the difference is the left hand section relates to testing, so it requires testing. The middle section doesn't, so it doesn't. That's the big difference.

I'll also point out this year we added that new column F for past I-3. We talked a good bit about that in the presentation last week. That's the new column. You can see all the other columns on this section slide backwards by one letter, but it's in the same order and basically meets the same purpose that's reporting all the outcomes and organizing it according to those areas of AB 104.

OK, so moving right along here to the right hand section, that's your services only section. Again, that shows the folks who only received services or those who might have done other things in the CAEP programs that have less than 12 hours of instruction. So everything falls here if it doesn't have the demographics or hours.

We have the total number of enrollees in this section in column M to clarify that who really has services versus who is just top tier because they didn't meet some of the other requirements as clarified in that column N by stipulating those in column like that actually received services. Then Oscar, Papa, Quebec, and Romeo, all basically look at the different types of short term services.

OK, and I think that was enough review for just the summary. Nobody will be surprised. We'll be talking a lot about it over the next hour and change. But that's just the review that we've talked about many times.

Here, we've got the CAEP Data Integrity. I spared you the slide with the 20 text boxes talking about all that action talk. But just to summarize, I will say like I always say that top summary information, very critical to understanding the CAEP DIR, kind of that way, for all the DIRs. But this CAEP version, I'd say especially, it uses that big map, looks at everybody that has any conceivable association whatsoever with CAEP reporting and then goes backwards, it reads out those that are not enrolled in one of the seven CAEP programs.

As it reads some of the students out, it identifies some of those that it's reading out that might have some valuable fruit inside hey, we need to toss it out. It doesn't meet specs, but hey, it sure is a nice looking fish. Maybe we need to figure out a way that we can put it on our menu, so to speak. But anyway, those that might have outcomes that we necessarily cannot retain on the report.

So it basically discards that and comes up with that number in bold at the bottom that serves as the denominator for all 27 items. And then the CAEP DIR digs into 27 items that are important for state and federal reporting. Some of them, the top 10 are the showstoppers.

There's a bunch roughly item 10 through 22 that are not really showstoppers for CAEP but get into things like periods of participation and integrated education and training that is important topics. The topics that quite frankly relate more to federal WIOA reporting than state level CAEP reporting. And then at the end we have that item 23 to 27 that get into CAEP specific outcomes.

OK, and then here is a sneak preview. This is coming, I believe Friday, if it's not Friday, it'll be sometime next week after the holiday weekend. I'm not sure of the exact release date, but it's not there yet, but coming soon. A new CAEP hours report in TE.

So this is what it looks like. I was going to use this for some of the calculations, but there's still some issues being worked out with it, so I'll do it. So making a long story short, I am doing some of those calculations some of you remember from a couple years ago. I'm going to continue to do the calculations the old fashioned way using the CAEP summary.

Some of you remember there are some tricky-- you can do all the calculations, but there are a couple that are a little tricky. Once we have this CAEP hours report, my feeling is we won't have anything all that tricky anymore. But again, it's not quite ready for this presentation.

So I'm going to show you those CAEP summary ways of doing it because I've got to admit, I'm not sure exactly how this is going to look exactly in terms of numbers. If I'm just showing you what it looks like, we'll fine, you can see it. But I have to admit I get a little less confident if I dig in to show you exactly how you're going to calculate the numbers to get inside baseball type figures like I'm going to go into with this example.

For now, no, we just have the four or some level reports. If you want to email me, I can certainly send this up as a request, though is something that could maybe be added to that list. For now, though, it will just be an age of C-level report for now, but it's a good suggestion.

OK, but just sorry, I spent way too much time making excuses for why I'm not using it right now. But just so you know, it's going to be hopefully a simpler report that itemizes out what was called the participants versus the adults served. Well, a lot of people would say that but I got to say I've heard those terms from Neal more than I've heard from anybody else. So at least for now we're going to attribute them to Neal. Anyways it will give you those adult served versus participants.

Look at those same three sections of the CAEP summary that we've just reviewed. That's right, somebody's ears are burning red after that. Anyways, we'll look at the different-- that report will look at it in terms of who has 12 or more hours? Who has 1 to 11 hours versus 0 hours?

Obviously, that's really, really important that can serve as the section to help you figure out who's there just because they received a short term service and really didn't have any CAEP enrollment versus those that did have CAEP enrollment but just had something, whether it was hours or missing demographics or whatever that prohibited the person from getting into one of those other sections. So for troubleshooting and you might say categorizing, students in different buckets this report will hopefully be really, really helpful.

OK, and then here's a view of the consortium manager reports. Again, we have four now. The three we've had forever, demographics, CAEP summary barriers to employment. Relatively recently, I forget exactly which one, but in the last six months anyway, we've added the CAEP DIR to the consortium level reports.

And then here is what I just said for proof in the pudding. It doesn't aggregate everything, but because of that whole common denominator, an issue really is a nightmare with this level. But it will allow you to compare and contrast all the different agencies in your consortium, putting it on spreadsheets, whatever makes it easier for you to look at that and review everybody together.

OK, so I think that was the table setting section of this presentation. I'll just stop. Any questions? Sorry, our sanity check everybody out again because I'm coming up for air here for a few seconds.

Anybody want to indulge me? You still got it. All right, we all got it, we all got it. If you didn't have any after 20 minutes, we'd be in trouble sitting there, wouldn't we?

Anyway, so all good, we're all good. Yes, we are good, aren't we? Anyway, so let's start here with NRS performance goals.

So sorry, another shameless annoying self-promotional comment. How many of you have been to those sessions? Maybe it was a year ago, maybe it was more recent than that, maybe it was way back when we were doing them in person. But how many of you have been to one of those NRS local performance goals presentations?

Same type of presentation but it focuses on the same concepts from an NRS federal reporting lens rather than a CAEP state level reporting lens. So if you have, you've seen these nifty screenshots, that is, we tell you to look at the NRS. It is a crazy series, that's right, it's up and down, it always goes seven gains, doesn't it? They're always going to triple altitude, don't they? You never think those games are going to end.

Anyway, we have the screen shot, we tell you look at the NRS tables, in particular NRS table 4. And we also refer you to the closest Data Portal which for the NRS, our agency summarizes your table 4 performance, your 4B performance, your NRS persister. And now recently, it also includes information about your students' responses to the employment and earnings survey.

So lots of stuff there, but the big question, of course, is the Data Portal is there, just for we owe it to NRS reporting agencies. Not necessarily everybody, but most of you are NRS we owe it to. So for most people in this line, again, this is at least somewhat help.

So anyway, from an NRS lens, look at the Data Portal and/or look at the NRS tables to identify agency strengths and needs. We're not going to spend a lot of time here, but table 4 is the NRS reporting table that's used to report on measurable skill gains that report using the 12 federal educational functioning levels, 6 for ABE, 6 for ESL, the student places with a pre-test. The gain is based on their pre/post test progress.

We've aligned our class system to those 12 levels. So based on how your students score on their pre and post tests, they'll placed into the federal levels and hopefully a lot of them will make progress on their pre and post tests to achieve a measurable skill gain. And then the report gives you percentages for each of those 12 levels, that is, at the end of the day at each of those 12 federal reporting levels, what percentage of students at each level made enough gain between pre-test and post-test to move to the next federal level?

There's a lot of blanks I didn't fill in. This is a whole hour presentation onto itself. But hopefully everybody gets the basic just to how table 4 works. This is not how we do everything CAEP wise.

But if you look at that left cam part for pre and post testing, you might say this represents what's, quote unquote, "under the hood" in that pre/post section of the CAEP summary. It basically is using the results from the NRS tables, more specifically, results from NRS table 4 to report out on those columns B, C, and D that you all know and love on the CAEP summary.

In federal reporting layer, there's also table 4B. It basically is set up a lot like table 4, but it only includes those students that completed a pre-test and completed a post-test. For federal reporting, you always use table 4, number 4B, but there are some that might cite, quote unquote, "fairness" issues from a classroom point of view that says if you're measuring an agency or state, maybe you use table 4. But if you're measuring student performance or classroom performance, you really ought to use table 4B. That is, you've really got to only evaluate those students that completed a pre/post pair when looking at whether the agency or the class or the student is performing well or not so much.

And then here is a screenshot of the class's Data Portal. I split it up maybe that was log, I made the log choice already. I moved the persister back in slider 2 so I could show you the class's Data Portal screenshot from table 4. This is just an example.

Again, another 45 minutes, I'm consolidating into two or three minutes where this is a screenshot from the class's Data Portal focusing on NRS table 4 results that's just part of one agency's performance where we're looking at a few levels and just showing you how the concept works in general. Where when you generate this on the Data Portal, it'll show you Statewide goals and Statewide averages. It has some 15 years worth of California data performance from which to compare it. And so you can run your own agency, compare to the state, compare to the neighbors, et cetera.

So in this example, the agency is comparing 2018, '19 local performance to Statewide goals and averages. Again, at the end of the day, we found one area, ASE low represented by the red arrow where we really need to improve. Another area, ESL, beginning lit, represented by the green arrow where we're doing pretty well.

Again, we're keeping it super simple for now and just saying we know ASE low is an area of weakness because our percentage is way below the state goal and way below the state average. We know ESL beginning lit is looking good because we're above the state goal and above the state average. Again, there's 20 more minutes of this we could get into when we want, but now we're just showing it's simple looking at straight and needs. How can we use this to reach forward conclusions and start looking at where we need to improve versus where we might be able to pat ourselves on the back.

And here is the NRS persister. This is what I almost automatically got into saying I might rearrange this just because I can't really seem to get out of my own way. But the persister basically compares NRS table 4 with table 4B. It looks at it from the lens of the same 12 federal level, 6 for ABE, 6 for ESL. Column B shows how many of each level placed into table 4, column C is showing of those, how many also placed on the table 4B.

And then that percentage of persister there in column D is simply column C divided by column D. That is of everyone at each level that qualified for NRS federal reporting, what percentage of those students at each level stuck around long enough for a, quote unquote, "persisted" long enough to at least complete a pre-test and post-test.

Here is another section of the class's Data Portal. Again, it's a CAEP training. So I'm showing you this but not spending a lot of time using it or whatever that will be and WIOA II trainings. But in any case, here is the persister section where we compare our persistence to the Statewide average rather than our performance.

You can see we're looking at two areas that we already know and identified as two areas of weakness for this agency. One is ABE Intermediate High, the other is ASE Low. So in this example, we're assuming we already did the homework. We already know 125% these are lit areas that we need to improve.

So one thing we always talk about is a good first step is look at persistence. If you know what's an area of weakness, the one thing you want to do right off the bat is look at persistence and see that is have we completed pre/post pairs for the bulk of these students? Yes or no. If so, there's one set series of steps we might want to follow. If not, there's another series of steps we might want to follow.

Here in this example 2, areas of weakness. The one to the left, intermediate high, you might say confirms our suspension. Our persistence in that weak area is very low. So what that means is knowing that our persistence is exceedingly low, our first step is going to be to get more pre/post pairs and/or engage in basic data clean up.

We need to get our demographics hours of instruction pre/post pairs in place. We need to worry about all those things first and then once we get all those ducks in a row, then we can move forward and start looking at things like classroom performance.

The green arrow to the right, that gives us one that shows our persistence, surprisingly enough, already pretty gosh darn good. So yes, it's an area of weakness, but not necessarily an area of weakness explained away by a lack of pre/post pairs. Because as you can see, persistence for ASE low in this example is actually well above the state average. So in this example, we might want to look at things like classroom instruction and look at ways to directly improve our student scores on the actual pre and post tests.

OK, so there is about 120 minutes worth of training frantically wrapped up into a neat little bow in five minutes. So that obnoxiousness notwithstanding, does everybody more or less follow me here on these concepts related to NRS performance goals?

I know a lot of you have been in these sessions a lot, so I do think most of you understand this from previous sessions. But I'll just say, hey, if you don't understand this in basic terms or wonder how retrofitting it for CAEP. Is going to make getting certain concepts I got to say?

OK, thank you for the positive remarks. So this is that goofy graphic. I know a lot of you have seen this, we've been using this for a while, to say. Yeah, at the end of the day, you look at those levels, you identify areas of strengths and need.

At the end of the day, every level is performance, it's going to be good or bad. At every level, persistence is going to be good or bad. That allows us this nice little oversimplified grid, that is, everything falls into one of four quadrants. And we can look at each quadrant and basically show where are our priorities? And what do we need to do?

The short answer is that to the left of the priorities, if we're low on both, we focus on data testing, and then we move on to instruction after we fix the data issues and the testing issues. If we are low performance but high persistence, that suggests that our data and testing is already in good order. So we don't pass go, we don't collect $200, we just go straight to the classroom and start working on fixing instructional performance.

OK, hopefully, that makes sense. Yes, so that's true. This graphic applies to CAEP just like it does to NRS. As long as you're willing to look 40,000 feet above, then yes, the way this graphic applies to CAEP, no different at all to how it applies to NRS. Maybe if you dig into the weeds, then it's not so much the same, but certainly at 40,000 feet, this graphic is the same for CAEP as it is for federal report.

OK, so I already asked whether everybody is doing all right, so I won't burden you with that question again. So here we're going to get into CAEP. Here is a little bit more probably what everybody signed up for. So what we just spent the last 5 or 10 minutes doing is table setting with that NRS approach to setting goals. We've talked many times about how you can use the same approach for CAEP.

So this is a lot like those data days from a couple of years ago, but I got to say a more exhaustive list and it will focus in on details or other types of details you might say, probably a little more goal setting oriented than we did in the past as well. I think right now is before I dig in, I'll also reference that supplemental handout. I believe Veronica mentioned to hand out at the beginning of the session, not just one.

One is the presentation, of course, that we're reviewing right now. That's the one that we're going to be spending 99.9% of our time covering here. The second one we're really going to be covering like not at all today, it's called the supplemental handout.

The screenshots are all old, everything about it is old other than it was cleaned up and that were some garbage slides that eliminated just so it's cohesive and at least makes sense. But other than that, everything's really old. So updated screenshots but you delete the handout. The supplemental handout is 0% mandatory and 125% gravy.

It gets into a lot of stuff we got into a couple of years ago, like Excel spreadsheets and getting into really detailed calculations, slides that do nothing but calculate and provide calculations so that calculations are the same. And a lot of that stuff is the same. So it's just given to you as is from a couple years ago. Everything about it is factually accurate other than a lot of screenshots are old.

So again, if that just bugs you at a high heaven looking at old screenshots, then my advice is just delete it and don't worry about it because it's not mandatory. It was purposely just passed on as something is better than nothing to give you something supplemental. Again, if that bothers you, delete it and just focus on this handout that has all updated stuff rather than stuff that's two years old. But there's a lot of stuff like Excel that I really wasn't interested in updating but it's still good information, so I just passed that along again as is for everybody's consumption or not.

So moving back to the feature presentation here. Here are a bunch of different things we're going to cover here looking at goal setting from a CAEP lens. The top three are the ones that we covered two years ago and that day to dive in person. Whistle stop tour, we did in 2019.

We'll be looking at enrollment criteria in the 2019 series, that's what we called the kick the tires metric. That is looking at all the students that have any connection whatsoever to CAEP reporting. What percentage of those students enrolled in a CAEP program? That is they showed up for orientation or to receive a short term service.

They kicked the proverbial adult dead tires. Ideally, they hung in there and they went into a class and actually accrued an hour or more. If so the enrollment criteria, that's one way of looking at CAEP persistence.

The second way is looking at it from a little longer game perspective. That is looking at attendance related criteria. That's the participants versus adults served. That is, of those that accrued at least an hour of instruction, what percentage of those hung around or persisted long enough to at least accrue 12 or more hours of instruction and thereby qualify for CAEP outcomes and qualify for NRS some pre and post testing? So in any case, that's the attendance.

And then the third one that we covered a lot a couple of years ago was looking at pre and post testing. Basically looking at using the exact same concepts of persistence that we just covered from the NRS plans over the last 5 or 10 minutes, but getting your calculations from the CAEP summary rather than the NRS persist rate.

And then we'll get in do a few others. Targeting CAEP outcomes is another one that we've looked at in previous years where you just summarize the total number of outcomes at your agency or your consortium and divide it by the number of enrollees to get an outcomes percentage. We'll look at that one and review that one and different ways of applying that for goal setting.

We'll look at the CAEP DIR. And again, that will be a goal from a goal setting point of view. More so than drilling down, look at some specific goal setting examples using the NRS model. We'll double back to some goals for special populations, like we discussed one week ago. And then we'll briefly cover consortium level reporting and how to set this up with external stakeholders.

I'll just say these bottom four bullets, you might say, are all presentations unto themself in a lot of ways. There are all multiple presentations unto itself. NRS performance goals, of course, has been done many times. Defining goals for special populations was done at least once because we just did it last week. The other two, we haven't really fleshed out and done sessions that focus just on this.

But I will tell you right now those bottom two bullets, like the two above it, could easily fill an hour, hour and a half presentation by itself. We might really want to look at doing exactly that here in the summer early fall. So a lot of this discussion is a big bad excuse to say we're looking at these. But we're not going to go too far because it's 30 or 45 minutes each that we really don't have now, but we will start looking at how you can set goals at the consortium level as well as ways to set goals with that external stakeholders.

One more birdwatch that's not on the slide as we have heard some panel presentations with some folks in the field that have done an exceptionally good job with the goal setting. Some panelists like to talk about this from a WIOA II federal reporting point of view and use things like table 4. Others are CAEP directors that really use CAEP reports to talk about goal setting.

We've done it a lot here this spring. There will be another session on June 23 of the class's institute and we'll probably be doing this again here in late summer, early fall. All kinds of stuff we'll be doing later for more information just at WIOA.

OK, so starting with CAEP goal setting. And again, we'll start with enrollment criteria. So for starters, you can use the CAEP summary or that new TE CAEP hours report. I mentioned I was really going to dig into this one a little bit here, but it's not quite ready yet. So we'll have to do this one at the next CAEP day to dive.

So I am showing you what this new report looks like here on the screen. I will use this for table setting. But then from this point forward when I actually show how you might calculate this kind of stuff, I'm going to stay conservative and stick with the CAEP summary so as to ensure the way I show you is something that's accurate, rather than something that's inaccurate and just sets you on nothing other than a wild goose chase or whatever.

So here is the look and feel of that new report. And the question for the new report is the exact same question as the one for the old report, that is the kick the tires question. When we look at the overall number of students available for CAEP reporting, how many actually make it into a CAEP instructional program? That is, how many students accrue at least one hour of instruction?

And so now, there is the CAEP summary just to go back into that way. Just kind of a translation, same question, different graphic. Using the CAEP summary to measure enrollment. And now, here is a computation.

So again, when you're using the CAEP summary, I'll admit that it's a little bit complicated to do this calculation. Again, with the new report, I think it will be a straight looking at columns rather than having to dig out particular cells. But what we're doing here is we're starting with that N/A. So does everybody see-- we'll go through this some painstaking just to make sure everybody gets it.

So first off, does everybody see that number 79 listed under column M in that row labeled N/A in the services section? So thank you, Chris, you jumped in. OK, people are jumping in. Yes, thank you, everybody.

All right, so everybody sees that. So does anybody know what does that 79 represent that I just asked you all to at least let me know whether you can read the 79 or not? What did those 79, what 79, students, what is it about those 79 students in that cell? Anybody have a quick answer for me? Anybody know?

Outside CAEP, we'll say, yeah, no program area. Good. Right, they are not part of any program, right, right, right, right, good. OK, different ways of explaining it, but right.

They make the meal, they make the map, but they do not have any program. And generally it's going to be a student that received a short term service, but has no enrollment, not even the evidence of anything that relates to CAEP enrollment.

So what we're doing is we're taking those total unduplicated students in this section and then we're going to be basically subtracting those 79. So those that are enrolled in this section and have a connection to a CAEP program divided by everybody in that section gives us our program enrollment rate. By apology's that's a little bit of irrational. Exuberance agency, they're doing really, really well, 98% of their learners are enrolled in a CAEP program.

I will say in 1920, these percentages were higher. Hard to say whether it's for positive just getting more students in class sort of reasons or it might have been kind of an inside out negative, where it just means you have a lot fewer stray bullets because of COVID. You say typically, I say tomato there, glass half full, glass half empty. How many cliches can shape fit in a bottle here? I'll stop, but in any case, there were a lot of agencies with high percentages in this area in '19/'20.

I'll just say if you want a more realistic number, you might look this at '18/'19. Hard to say there, but that is the program enrollment rate. Here's the math on the slide right here. But if you want to see at the end of the day of everybody who makes the report what percentage have some enrollment connection to one of the seven CAEP programs, here's where you can figure out that information using the CAEP summary.

And I'll go out of my way again to say this is the same metric that we've reviewed in 2019. This is totally pete and repeat admittedly with new screenshots in slightly different terms. I think we've gotten more consistent with our terms. So I do feel a lot better about the terms right now probably than I did before. Yes, unmute, ask away.

Peggy: Hi, Jay, it's Peggy.

Jay Wright: Yes.

Peggy: So I'm struggling a little bit about 98.8% being a good number. And the reason why I'm struggling is I'm thinking about the fact that we are trying to pull in as many students as we can into the adult schools and then refer them to our colleges if they don't fit the adult school. And we're coming up with a whole way to transition them and we have outreach all by ENEN.

And so I could see in our consortium that, that number may be down to 96 or 95. But that's because we'd be serving community members by helping them set goals and go to another agency that is not an adult school. So my question is, am I looking at this wrong here? Is this going to be kind of a new measures of the state--

Jay Wright: No, I think you're looking at it correctly. Sorry, so I'm going to give three goofy answers. The first one is that at the end of the presentation, we have some Statewide slides, I think Statewide percentage was 96. So we were a lot higher than we used to be.

We don't have any set levels, Neal clarified that. So what we're showing is not what you're officially responsible for, it's just kind of rule of thumb for reference only just to see how you compare with the overall, quote unquote, "average". I'll go back to the how many cliches can jade jam into a bottle to wear?

Short answer is in '19/'20, this metric was a lot higher than it was in '18/'19 or '17/'18. I do think that COVID had a little bit to do with it. It is a glass half full, glass half empty, maybe it's just because we're a lot cleaner. We have a lot fewer stray bullets, we're getting everybody in class, we're doing a better job of managing our data.

Maybe it's looking at it more half empty where we just don't have as many students who are coming in to kick the tires. We batten down the hatches. Once the cliche is in this explanation, we batten down the hatches. We're only serving those that are all the way in already, so that we don't have those coming in off the street kicking the tires much in the last year or so.

So I'll just say Statewide at a lot of agencies, the last year or two these percentages are a lot higher than when we did this presentation two years ago spring of 2019. I forget what the percentage was that we presented a couple of years ago, but I remember it being more in the low 80s, I think 82% or something like that. I don't know if that's right, but it was a much lower percentage a couple of years ago than it is now. We increased this metric by quite a bit in '19/'20.

So there's a lot of different answers, none of which is the silver bullet you're looking for but are all possible. I guess the other thing I'll just say related to Neal's response is if you know your agency or your consortium has something specific that might make numbers appear a little higher, appear a little low, that's what I've always called a do your homework issue.

If a number looks weird but you've done your homework and you know the number looks weird because that's just how you roll at your agency and/or your consortium, then fine. You've done your homework, you've done all that, you've taken the steps you need to explain that number. Maybe you need to take steps to change it, maybe you're 100% happy with the way you're doing things. And you may just need to know that yeah, that number is always going to look a little bit different because you've decided to say tomato instead of tomato or some to speak.

Does that at least make sense, Peg? Yeah. All right, thank you.

All right, so I'll move on here. Good question. Thank you, Peggy. That was great. That really helped me a lot to break and break it up a little bit. I think everybody should be really thanking you to break that up because it does feel a lot better having broken up that diet right.

Anyway, the next one we're calling CAEP participants. This is another one that we talked about extensively on the data dive two years ago. So again, there's that nifty screenshot of the new report. Here's the question we asked two years ago, every bit as relevant now.

Of those that enrolled in a CAEP program, that is, those that have at least an hour and have official special enrollment, how many stick around long enough for, quote unquote, "persist" long enough to accrue at least 12 hours of instruction? And you might say it's not there, but in parentheses and also qualify for CAEP outcomes than NRS, quote unquote.

So here, we're going to go back to the CAEP summary. We're going to make sure that our calculations are correct. Here's that outcome section. So again, we're going to look at those unduplicated enrollees in column E, that is, those that have the demographics of 12 hours. You can see that's how many show up in column E.

And then we're going to divide it. It's a little tricky here. Again, we're not dividing it by the unduplicated in the right hand section represented by column M. We're putting a little bit of a twist in and making it a little trickier than anybody wants.

And it's what we divide it by is that number we used in the previous metric, that is, the number of unduplicated in column M but after we subtract those in N/A. So I think, Maureen, you dutifully and graciously typed it in, it was that 6771 minus 79. That's correct, thank you, Maureen.

Is that again, we're using that as the denominator here, not the total in M. Because again, we're at those that are services only, we already kicked out. We've already eliminated them or voted them off the island, so we're now looking at those that were still left standing to measure against. But we're looking at the 12 or more hours.

And Neil says yes, different regions. Not all are the same, your mileage will vary, yes, indeed. But in any case, just to go back to the metric, you can see here 4372 divided by 6692, that's not right. I think I used the same one, sorry, that one I got to change.

I had the right percentage, I overdid it. But anyway that's not 69, that's like 67, I think. Sorry, I used the same thing and typed it twice. I'll have to go back and set a new PowerPoint out.

I think I tested it twice because I have another version that had the right number in it. Yeah, thank you. Sorry, that was a typo. I used the exact same percentage and pasted it twice by mistake. Yeah, everybody's jumping in.

But you can see the right numerator and the right denominator. But yeah, I failed math, I used the same result as the other one, sorry about that. But in any case, hopefully, you can see how you do it.

Again, you're looking at those that get in the middle section, comparing it against those that are in the right hand section. But again, that little caveat or twist here is to make sure you're calculating it correctly. You need to eliminate those who are in the end A cell, that is, the 79 that we address there at the beginning of this discussion before you come and calculate that right percentage.

OK, I'll stop here and just make sure everybody hanging in all right. Sorry, I put that stupid number in. I'm going to leave it and just send it back to Veronica after. But everybody understand these metrics OK, yes or no? Good quiz, that's right. Giving you wrong answers and everything to check on right.

OK, so in any case, the third and final one from that data dive two years ago was looking at it the old fashioned NRS way. That is if we're doing it the federal reporting way, we're looking at pre and post testing. I'll send that to Veronica. Yeah, it was just a stupid mistake. I hit Control-V too many times, giving you two TMI, I think.

Anyway, CAEP pre and post testing. Here is where we look at the same way of measuring persistence exactly as the feds or NRS else would do it, but we're using it giving our calculations with the CAEP summary rather than NRS. Bottom line, if we are WIOA II agency, of course, we can use NRS table 4 or the constance Data Portal and that will work just fine. Thank you very much.

But again, we're looking at CAEP that we should use the CAEP summary. The other probably more important reason is obviously CAEP includes CTE workforce crap and those other programs that are not included in NRS federal reporting. So to have it be apples to apples, we might well want to consider using this method instead of the NRS way. So as to include all of those CAEP students that are not in ABE/ASE or ESL.

Again, apples to apples suggest that we will use this left hand literacy gains section. So again, we're looking at column B to show how many qualify, that column C is basically looking at the, quote unquote, "table 4B" point of view, and that column D is looking at the table 4 so to speak, where it's EFL gains. So again, it's a lot of the same type-- it's the same approach that we would use in a report such as the NRS persister, but setting it up in columns by program, rather than by NRS level. And of course, including all those non-WIOA II students.