Jay Wright: OK. So now I'm sharing my screen. Hopefully everybody can see my cover slide, and not my Outlook calendar.

Anthony: Yes we do see that--

[interposing voices]

Jay Wright: --that is in fact what everybody sees, I guess. OK. Good to have everybody here. Not sure if accountability is a better category than goal setting, or if it's just more a matter of 10:30 is a better time slot than 8:30. But I guess I was kind of thinking I'd have about the same 35 or 40 I had at 8:30 in this one, but I doubled in size somehow. I'm not sure if it's topic or just time slot. But good to have you here.

So this one is kind of the basic accountability 101. You might say-- I'll liken this one to the old Neil And Jay show which has obviously been referenced in a lot of different sessions. There is no Neil And Jay show anymore. So this is what we strung together in lieu of that. So we had something instead of nothing for this.

But the topics are similar. A little more CASAS-centric because it's just me, myself, and I hear. But we'll talk a little bit about programs, update. Some of these are updates from last year, not necessarily this year. But we'll look at some of the big program updates the last couple of years.

We'll bring back those goal-setting targets. That was a big discussion and the presentation earlier this morning. The one this morning is where we really dig into goal setting, and give those examples. We're not really doing that here, but we will list off what the official goal-setting targets are in NOVA here. And a little bit about how you might do that.

Most of this session is going to be looking at the different outcomes. We'll look at-- with a focus by key training issues, that big list of outcomes as represented by the bubble boy slides that I know a lot of you have been reviewing for years and years here now. So we have all of those outcomes that are there for CAEP.

A lot of you have been to these sessions before, where there's those four or five outcomes that seem to be the most confusing, and generate the most questions above all of them. So we'll try to itemize out some of those outcomes that tend to confuse people the worst, and generate the most questions. That's where we'll focus. And try to really itemize out a lot about the different outcomes, added a few new slides with some newer questions we've been getting.

And then in the spirit of just basic accountability, the last few minutes will cover the deliverables, timeline, and resources. We'll do a quick overview of the CAEP reports in TE. I got to say, if you want that you missed it. The deep dive for CAEP report's really was the session I did at 8:30 when we did goal setting, but for this one we will at least identify what some of those basic reports are, and what the purpose of each one is, and so forth.

So I guess I'll just say, does that sound more or less to everybody what you were expecting to get yourselves into here this morning, yes or no? I'll just say, we gain participants since that spiel, miraculously enough. So I guess nobody's voting with their feet. To say no, I guess, right? All right. Thank you for that.

So let's go ahead and jump in. So this is really nothing new at all, but just a review of the CAEP program structure. Every time I do these presentations people usually say, hey, well, can you do something a little bit more basic so I can kind of get my feet firmly planted in the ground before you barrage me with all these crazy details.

So that's what this is really trying to do, is this is the basic CAEP program structure. Historically we've always said CAEP is comprised of seven programs. Looking at it from a global state level perspective, we actually have five. We have three, what we call the basic programs. You can see that up top.

We have ABE, ESL, and CTE. There are those two programs, Adults With Disabilities, Parents Supporting k-12 Success, that really don't fit neatly into one of those three primary programs. So we say we've got the three primary programs, and then the two other programs to comprise five total programs.

Getting into CAEP plan structure a little bit, where a CAEP is a little bit different from the real world, you might say is in CTE and ABE. So breaking down the 3 basic programs. Start in the middle with ESL. ESL is ESL, is ESL, no differences there.

But over to the left with ABE, technically in CAEP land, ABE, high school diploma, and HSE are all one uniform program. In the real world they're usually separate programs, but for CAEP reporting, that's all one program. Similarly over there to the right, with career tech-ed you've got those for somewhat different programs all locked into career tech-ed, that is CTE, short-term CTE, pre-apprenticeship and workforce preparation

This is just how the whole structure is organized, just FYI. Looking into CTE a little bit, because I think this is the one that probably confuses people the most, is we have all these different programs under CTE. We obviously have career tech-ed. In CAEP land we have short term CTE that applies to the college side.

I don't think it applies anywhere else, but CAEP is run by the Chancellor's Office, so it's part of the program. Then we have pre-apprenticeship and workforce prep. What we're doing here is just trying to distinguish because a lot of times people think of CTE as kind of all the same. But just to get in the weeds a little bit and distinguish it, CTE is occupation-specific and usually long-term.

Short-term CTE is obviously occupation-specific and short term. Pre-apprenticeship is just a different delivery method. It's got the same goals as CTE, and the same purpose of CTE. But it's a different delivery method. But it's also occupation-specific and long-term.

And all that is opposed to workforce preparation, which gets locked into the same category. But that's short-term and not occupation specific. Just say over the years there's been confusion about that. So that's getting in the weeds to distinguish those different workforce related programs.

Also reviewing some of the things. This is kind of recapping the last year or so with the Neil and Jay show. We had those trainings back in 2019. Then COVID hit, we didn't really do this, or really haven't done much of it since. So there were a lot of things we introduced as new in 2019.

I got to say, because of COVID and other things, kind have long since been forgotten. But just to perk everybody's memory just moving forward. Some things about the program structure that were big before COVID. We used to have that program, workforce reentry, that we now call workforce preparation. So again, that's now for all learners.

In the old days if you've been at this we used to have a lot of derived fields that I got to say, we're very, very unwitting ideas that we fortunately got rid of. So we're not deriving anything now for workforce reentry. We're now just making it Workforce preparation, and it serves all ages, all types of learners.

We have that adult served with those three categories of learners. We covered that in, I got to say, ridiculously gory detail in the early morning session when we were looking at goal setting. But again, by three categories I mean those three buckets that people like Neil have been talking about for a long time.

That is, you've got the learners with the 12 or more hours that are participants. You have those adults served that have 1 to 11 hours. And you have those reportable individuals that may have zero hours, but have some kind of information related to a program.

So again, we've looked at those three buckets a lot. That is, the 12 or more hours, the 1 to 11 and 0. And a lot of that goal setting now is oriented toward maximizing the number that have 12 or more hours, and trying to do that by moving people out of the 1 to 11 hours bucket. And moving students out of that 0 hours bucket, and moving them to where you have more participants and fewer adults served.

We've also been looking at hours by program and multiple programs. I think there's a slide on that, so I'll talk about that here shortly. We're not recording hours of instruction for short-term services. Hours of instruction only apply to CAEP classes. This seems kind of obvious, but we're recording barriers, do employment that's required. And then we have that passage of exam, MSG. More on that, we'll recap that one a little bit when we get into outcomes.

But that's been a big question asked. When we started this we have that occupational skills gain, which is aligned to a federal outcome called skills progression. At the federal level, a couple of years ago, they changed the name of the federal outcome from skills progression to passage of an exam. It's the same outcome, but you've got to have some kind of exam involved in order to record the outcome. So I'll get into that here in a few minutes.

OK. So sanity check here. I went through a big list. I'm worried I'm losing people already. So just because I have that little bug in the back of my head saying, gee, you're losing people. You're just talking about nonsense already. Everybody hanging in there. Just sanity check me just because my voice is telling me I better. OK, thank you. Keep going!

All right. So I'll try to squelch that little voice next time. But yeah, my little voice was really sabotaging me. And saying, yeah, nobody knows what you're talking about. Everybody's already looking at their exit strategy for this one. That's what my voice was telling me. So let me just double-check.

All right. So getting into that laundry list, and giving a little detail here. Hours by program. So that was a big issue a couple years ago. I'm not quite sure where this is going from a statewide point of view, but I know locally a lot of you at different consortia are worried about how you're recording hours-- you're looking to maximize hours. And a lot of you are looking at that from an instructional program point of view.

So that CAEP hours by program report records that for you. The big issue has come up as if you have learners that are in more than one program. The CAEP summary is duplicated across program unduplicated within program. So we already are going out of our way to record all program enrollments. We're not unduplicated it like we do for WIOA.

So what we do is we go out of our way to show enrollment in both programs, and then just divide the hours equally. So if you have a student-- we'll use an obvious example, a student who's an ESL and also in CTE. If you do nothing, that's OK. But just know if you do nothing, the default behavior is the TE report will assume it's 50-50, and just divide those hours equally.

If you happen to know that it really should be 70% ESL or 30% career tech-ed or whatever, that's fine. But just know then you need to go out of your way to create two separate classes. Make sure all the ESL hours go into the ESL class. And all the CTE hours go into the CTE class. In my example, if you want to bunch it all into one class that's fine. But the default behavior will always be 50-50.

OK. More on hours. This was the big issue in summer of 2017, or spring of 2019. There were a lot of questions going on for about a year about whether we should count service hours as part of instructional hours for about a year when you went to the Neil and Jay show we would waffle on that question, because we really didn't have an official answer.

So there was about a year of waffling. And after about a year of waffling, the final answer is CAEP will not track service hours. It's still requiring you to track those provision of short term services. So yes, you should include those students, but no you should not record any hours just to services, the hours are just for those classes in the CAEP instructional programs.

So again, yeah, when you're doing NOVA reporting, when you're looking at that CAEP hours by program reported TE. It should only be including those hours from CAEP classes, not hours from short-term services.

OK. Here's that big thing that's new this year with the goals is again, we've got adult served. So to be clear, when we use adult served, that actually includes everybody. But then we've got those individuals-- these are the three buckets. We have the service only students. That is students that make the map, and show up as reportable individuals on the CAEP summary, but who have zero hours of instruction. Usually that's service only student. Sometimes that can be data cleanup that needs to happen, but we'll call them service only students.

Then the second bucket are students that have received instruction, but they have not received the 12 or more hours. And then the third bucket-- that's what we're looking for. That is, we're looking to maximize the number of participants. That's the federal term for the students that have the 12 or more hours who also have all those required demographics. That is, they've met all of those requirements to be an official CAEP enrollee, an official CAEP participant.

OK. We talked about this report at length in the first hour. So for a couple of you, this might be review. But these are the newest reports in TE. Is the CAEP enrollees by hours. The other is the CAEP service enrollees by hours. The purpose of these two reports is really exactly the same. They're just displaying the data in different ways.

The one you probably want to review first is the one on the screen now, the enrollees by hours. What this is doing is basically showing those three areas on the CAEP summary. Those three areas in which students can qualify on the CAEP summary. And then for the outcomes section and the services section, breaking it down into those three buckets we just described. That is of those that qualify for outcomes, how many have 12 or more? How many have 1 to 11? How many have 0?

Of those that get dumped into the services section, how many have 12 or more hours? How many have 1 to 11? How to zero? Usually you're going to be looking at the services section for this usually, but you can look at these different sections. So in this example, we have 855 ESL learners that show up in that services section. You can see 609 of them actually have 12 hours. There's some other issue that's keeping them from getting into the outcome section.

So can look at those 609. We can compare that to the 98 and 148. Again, to the maximum extent possible. We're trying to move people out of that column H and I, and getting them into F and G. That is, getting more with 12 or more hours to become a participant.

As we look at that I'll just say the service enrollees breaks it down to the service categories. So if we see we have a lot of services only that might have 1 to 11, or 0. We can go ahead and look at those service categories, and drill down and try to figure out where those students are as again, a way to troubleshoot and try to bring more into the 12 hours bucket.

OK. Moving on to the next big issue. This is that passage of an exam issue. So because I just got through playing musical screens, I'm going to be a little bit of an annoying stickler on sanity checks. Because I'm talking about way too many things at once, everybody following me as I just casually move out of that really big topic, and casually move into another confusing topic called passage of an exam. Even though I'm just cavalierly moving from one complicated topic to the next, that's your story and you're sticking to it. You're following me, no problem.

All right. So passage of an exam we brought this up. Again, this is another one from a couple of years ago. This used to be called skills progression at the federal level. So when way, at the federal level we have those five measurable skill gains, or five MSGs. One of them under WIOA is been called skills progression.

A couple of years ago the feds changed the name of skills progression to passage of an exam. Stepping out for a minute, the skills progression was always intended to be one more for Title I, not us in Title II. That is some measurable skill gain for workforce programs. When we started CAEP five or six years ago, we looked at those WIOA I outcomes as a good way to align and use it for our CTE outcomes in CAEP.

So those CTE related outcomes-- one is called occupational skills gain, the other is called Workforce prep milestone, where based on that federal skills progression MSG for WIOA I programs. We thought it was the closest approximation to WIOA that we could use to report those CTE gains.

So anyway, a couple of years ago the feds changed it to passage of an exam. The purpose didn't change at all. The way that you measure this outcome really didn't change at all, except for from this point forward in order for those workforce clients to get the gain, they need to pass some kind of an exam.

That specific exam is pretty loose, but what we decided to do in CAEP land is pass on the love for our outcomes. So now whenever you record that occupational skills gain there ought to be some kind of exam that they pass. You can see the examples below. You can decide what that exam is.

Sometimes you already have an exam that they pass when they move on from one semester to the next. Some of you might say, hey, it's CTE, so doing skills checks makes a lot more sense than written exams. If so, that's fine. If you want to do CASAS testing, again, that's not required for this. But if you want to use CASAS testing as the passage of an exam, that's also fine.

Just earning a diploma. So do you mean earning a high school diploma, or something else? I assume you mean high school diploma. If so, no. That's not an occupational skills gain. But as we move along, you'll see high school diploma is its own outcome.

So no, high school diploma does not count as an occupational skills gain because we already count high school diploma as an official CAEP outcome. So we kind of go out of our way to count it as a high school diploma, and not an occupational skills gain. Because hey, we're tracking high school diplomas.

GED is an outcome. A section of a GED is not. Just say that's never really been aligned. You're in a big club of people that have brought that up in the past. But at least as of now, when they pass the entire GED, yes, that absolutely is an official outcome. But the official outcome is only for passing the entire GED, not one of those subsections.

Good questions, anything else like that? OK. So we'll move along. These are good questions. These make me feel better that you're asking smart questions. So thank you for making me feel better.

So moving along here, these are our official CAEP goal setting and targets. We're just pretty much dumping all this information in one fell swoop one yet. So here's our goal setting and target. So it's broken down into three basic categories.

We've got our mandatory consortium level metrics. Our mandatory member metrics. And then the larger list of optional member level metrics. I'll get more into that later. But the short answer, Karen, is no. Not the CPR exam. There are things that they could do as a CNA that would be occupational skills gain. But I would say CPR exam, that's way too short.

Potentially you could count CPR as a workforce prep milestone. I mean, if you're marking it as that-- I think that's a small problem, not a big one. But if you're asking me I would say no. CPR is way too little for occupational skills gain. Occupational skills gain it's more like completing a semester worth of work, not just a two or three hour course like CPR.

OK. I see what you're saying Kim. But for now, sorry, it's not. So in any case, let's move on. So here are our mandatory consortium level metrics. So the mandatory at the consortium level, we've got those four barriers. And I should have looked it up knowing that I was going to present it again.

But I think it's low literacy. It's low literacy, long-term unemployed. Low income, and I think English literacy. I think those are the four that we're looking for. Maybe somebody's going to jump in and correct me on those. I always forget that fourth one. But anyway, there's four barriers that are presented to you in NOVA. You don't really need to get that complicated for it with the mandatory metric.

All the mandatory metric is doing is giving you an item count, and you're basically monitoring your item count of those four barriers, and showing that you're doing due diligence to actively serve those barriers a stated priority. Statewide for CAEP is that we're actively going out of our way to serve students who have barriers to employment. A big topic of this summit has been equity.

One real good way we would say we show that in our data is by reporting on barriers to employment, and dutifully showing that we're serving students that have all those barriers. So the mandatory metric is to show that we're doing exactly that. And then we've just got that bottom line head count of reportable individuals, that's all three buckets that we were talking about. We're just looking to have the biggest number possible in terms of overall number of reportable individuals by consortium.

OK. Then member level there's that little metric that Neil's talked about for a year or longer now, where he's looked at the amount that each agency is funded. Looked at the total number of adult served at each agency, and kind of given a dollar amount per learner. I believe that was programmed in to NOVA. So everybody is going to look at how much money by student they're putting in. That's a mandatory metric.

And then here's that moving from one bucket to another that we talked about. The number of enrolled adults that become participants. So that is how many 1 to 11, and how many zero hours are moving into that 12 or more hours bucket. That's the mandatory metric. That's also why I'm making such a big stinking deal about those two new TE reports.

And then here's optional metrics covering all sorts of areas. The more I'm hearing about this, the more and more flexible it is. I think at the beginning, there was an interest to pin your nose to using that number that's in LaunchBoard. And using only that one specific metric.

As it's morphed along I think the flexibility, much to everybody's relief, is increased a lot that you can use those numbers in LaunchBoard. You can use numbers from TE reports. You can use numbers from COMIS reports. You can use numbers from other systems. You might be using it, your agency.

But bottom line is you're documenting the number of diplomas, HSC. Maybe you're looking at post-secondary. Looking at pre-post gains overall, or by specific program. Looking at the transition to CTE. Transition into four credit.

You can look at your follow up data two quarters after exit for employment. Look at earnings. Or you can look at that immigrant integration milestone, that is that I3 outcome you get by students passing EL Civics COAAPs.

So again, a lot of flexibility in the way you look at this. But you're going to be required to select optional metrics that address these specific areas. So I think the long story short is when we're looking at categories for reporting, you really need to use these areas that relate to our official outcomes. Secondary, post-secondary employment, literacy gains, and so on.

But when you get to specific data systems, or whether you want to measure it by agency or by program, or when you're looking at specific reports that you might use to measure it, you've got some flexibility as far as that's concerned. OK. So we're in a transition here. We're kind of getting into more rudimentary Neil and Jays show stuff from this point forward.

So let's do another sanity check. Does that goal setting information make sense? Does all of this stuff that we've been talking about makes sense so far? Sorry, a convenient time to do a sanity check yet again. OK. That's your story and you're sticking to it. Why E3.

OK. So anyway. This slide, if you've been a career follower of this stuff, this slide looks familiar. This is the slide that we put up when we first started doing AB 86 accountability, not even ABG, I don't think. I think we had this slide up when it was still AB 86. But this is the slide we were showing to give an overview of those outcomes.

And specifically, we use this slide to show that we didn't just make it up. That first off, we were basing the outcomes on the AB 104 legislation specifically passed by the California State legislature. Number two, we also wanted to show that the state legislature wasn't making it up either. The state legislature approved these as the official outcomes based on their understanding that these outcomes ensured that we were aligning our outcomes to the federal WIOA system.

Part of the whole ABG, prior to that AB 86 framework is we would have a statewide accountability system aligned to the federal WIOA system. In that federal WIOA system we have what it called performance indicators, and measurable skill gains. Performance indicators, I like to say are outcomes that you can hang your hat on. You got a job. You got a pay raise. You got your GED, whatever. That is you have an outcome that you can really measure by completing something, or getting a piece of paper.

But we also have those outcomes called measurable skill gains that suggest improvement without necessarily finishing anything. So the issue there is everybody agrees things like diploma, and job, and wage increase are all really, really good outcomes to achieve. But if we rely solely on performance indicators, a lot of times we're setting our students up for failure because, yeah, some students get jobs, and some students complete their diploma.

But a lot of students out there maybe haven't finished it to where they've gotten the job, or finished their diploma yet. But we know that those students nonetheless they're doing a really, really, really, really, really good job. For that, the feds have MSGs or are measurable skill gains. That is a way to measure progress even though the student hasn't gotten the job, or finished the diploma or GED yet.

The obvious example of that is CASAS' pre-post testing, where we might be able to show a lot of students that we have not yet gotten a job. We know they haven't yet passed the GED, but we know they've done a great job on their CASAS' pre-post, so we report those measurable skill gains so we can show that even though our students have it finished their final outcome yet, that they've still done a great job. Because we can show these gains on CASAS testing as a means to prove it.

Like we talked about for things like occupational skills gain, we've retrofitted some of these other federal measurable skills gains so we can apply that same concept to ASE programs, and to CTE programs, and so on. When you combine those performance indicators, and measurable skill gains in unison, that basically almost word-for-word, equates to the four areas of AB 104 that the California legislature passed six or seven years ago that comprise our mandatory areas for CAEP report.

OK. So springing on to that. What we did then is looked at these key areas. We align some outcomes to the 6 areas of AB 104. I know a lot of you have probably seen this slide way too many times. But again, it's the four areas of CAEP reporting. Literacy gains, secondary, post-secondary, employment, wages, and transition. So those are the six areas. And you can see each of those areas has a handful of outcomes that fall into one of those six categories.

So anyway, what I'm going to do is I'm going to bird walk through some of these areas, focusing mostly on the literacy gains, and secondary. But some of these like get a job, get a high school diploma, don't need a lot of explanation. But others, like especially some of these non-testing literacy gains, and some in post-secondary do.

So again, we're going to pick out some that are the biggest problems. So I'll start with this one. This is on the ASE side is-- and this one directly relates to federal reporting, that generally when we report measurable skill gains for our NRS tables, we're looking at those CASAS' pre-post test gains.

But when WIOA started, there's an alternative you can use for federal reporting, specifically for students in high school diploma, where they can report gains through high school credits instead of through pre and post tests. So the long story short is for high school diploma students that start at the 9th or 10th grade level, and finish at the 11th or 12th grade level through high school credits, that's a way in which they can make a gain.

This screenshot shows the work around we have we're in TE. Where in TE you record this by using the TE self-reported level as demonstrated on the screenshot. And the basic TE directions are shown in that little text box where we have a place in TE to record credits. But every district has a different number.

So in order to record this we use that self-reported level shown here. But one, just to give you the basic info if you've got those high school diploma students that are making progress, that is an alternative way to show an MSG.

OK. Looking at some other literacy gains for CTE. Here are the ones that probably generate questions the most over the last few years. That is, those CTE related literacy gains. Again, we have one called occupational skills gain. Another called workforce prep milestone.

Before I jump in I'll say if you've got this confused. If you've been reporting this backwards, or maybe you'll use slightly different criteria-- I'll go out of my way to say that's a very small problem because at the end of the day, it all gets kind of locked up the same way in the CAEP summary. But for most of you, that doesn't sit very well.

You want to make sure you have a very concrete distinction. So this is the concrete distinction I like to provide here is occupational skills gain suggests accomplishment of a portion of a longer term program, whereas workforce milestone suggests full completion of something shorter term.

So our example for occupational skills gain, it says welding but I'll use CNA because somebody brought up CNA in a question earlier. So if you just subtract welding and put CNA, I would say the same example applies. So let's just say you've got a CNA program that's five semesters, or five modules long.

We'll just say the CNA student enrolls in the first semester, does a great job. She finishes that first semester. And you did a little skills check. You know what? Hey, she's able to do the-- it's the first module so it's all the basic stuff. She shows that she can check blood pressure and temperature really well. So she does that little skills check.

That's the passage of an exam. So she passes that with flying colors, and finishes the first module, and moves in to module number 2, that would be an example of occupational skills gain.

Well, CTE is what we're calling it just because usually these outcomes on the slide apply to your CTE programs. It's not required that it'd be just for students and CTE. It can be earned by students and other programs, but we call it CTE related outcomes because the assumption is these are usually going to apply to students and CTE.

I think that's your question. If it's some other question than sorry, I don't understand your question. OK. I'll assume that that's the question you're asking. So in any case, on the other hand. I think to the question-- Karen, your question was what if they pass a CPR course? I would say that's a good example for workforce prep milestone, but not really enough to be a good example for occupational skills gain.

But if you fully complete something shorter term-- so in your example, I'm assuming you're saying yeah, you need to enroll in CNA. You need to complete these five modules. There is a CPR course that we're requiring CNAs to do, but we're having you do that separate from the regular instruction, or whatever.

So yeah, you could use that as a potential workforce milestone. If they complete it that would be an example of full completion of something shorter term. You can see the example on the slide is not for CNAs, but it's kind of similar.

So hopefully that answers the question, and clarifies these two outcomes. Well, no no, no. Occupation-- hold that thought. Another no. Occupational skills gain would be more like the example I'm putting on the slide, where they did-- I mean, if they're only doing CPR, I would say that's way less than a full module. So it would be more like, hey, they passed the full module. They completed the class and they completed a written test that shows they understand everything in that first module, not just CPR.

The CNA exam would actually be a post-secondary outcome. That's the next slide. So I'm showing an old slide just to give you what I'm comparing it to. But again, we've got post-secondary outcomes. And we've got literacy gains. The post-secondary, I'll relate here to what we might call performance indicators.

That is, they get the final outcome. The occupational skills gain workforce prep is more than measurable skill gains. That is, they're making progress but they haven't really finished yet. That post-secondary outcome would suggest they finish something.

So again, we'll use that same CNA example, where they do CPR. Where they complete something shorter term. Where they finish module one of five, that is partial completion of something longer term. Keeping with that same example is once that CNA student passes module five out of five. And as you say, actually gets that CNA certification, that's when it rises out of this literacy gain. And rises to the level of post-secondary.

So when they actually get their CNA certificate, you should be marking one of these post-secondary outcomes, not just one of the literacy gains. When they get the CNA certificate that's the post-secondary final outcome we're looking for. Where congratulations, that are now certified to be a CNA officially in the state of California.

Good questions. You kind of guided me along. I'll just say hopefully those questions Karen asked were helpful to everybody. I'll say it was helpful to me, because you gave me something concrete to talk about instead of all this esoteric stuff on the slide. So I'll just stop for a moment.

Does everybody feel like they've got a handle now on workforce prep milestone versus occupational skills gain? And in turn, these two CTE literacy gains, versus these higher level secondary outcomes. Right. Hold that thought, Iris, that's a good question. The next few slides address that question, I believe.

So starting here. And this is actually straight from the OCTAE website, I've got to say. So I feel pretty good about this chart. The titles up at the top don't really do much for me. But I really did like this chart, because it put things into four neat and tidy categories.

What I did is I drew that little line here. You see that dotted line? That was my work to show-- what the feds are calling a certificate to the left doesn't really quite cut the mustard. What they're calling certification degree or license over to the right, I would say, does.

But we've been talking a lot about how when we're looking at things like IET, ILCE, and WIOA Title II reporting. As well as looking at these different CAEP outcomes for post-secondary and for literacy gains. Everybody has had a devil of a time distinguishing what are higher level outcomes, versus whether or not higher level outcomes.

So for my two sets, what the feds are calling certificates are good outcomes, but they're definitely examples that do not qualify. These are definitely not post-secondary outcomes. They're not occupational skills gains. And they're not certificated programs that qualify for workforce training for integrated education and training for WIOA II reporting either.

They're just short term things like food Handler, like CPR. Like OSHA ServSafe and all that kind of stuff. That would be what's demonstrated in that left-hand column. On the other hand, in the right-hand column are the outcomes that actually qualify as quote, unquote, post-secondary outcomes.

One big source of confusion, I'll point out, by that column-- that middle column called degree. Where when we use the word post-secondary one confusing thing about it at the federal level is post-secondary includes those quote, unquote, academic related outcomes for post-secondary. It also includes those CTE occupational outcomes for post-secondary. The two rather dissimilar types of outcomes are grouped together in the same category.

So in certification, that means they get their certified welder. Or technician sort of stuff. They're under license. There's the RN, CNA, cosmetology, all that. So sometimes it's a certificate, sometimes that's a license. But all of those CTE related occupational certificates all count at the same time the associate, the bachelor's, advanced degrees also all count.

That's why I say it's kind of two dissimilar types of outcomes all bunched together. But back to this chart, what I think is convenient here is these right-hand columns. I'll give you some tangible examples of things that rise to the level of secondary, versus these things over to the left that do not rise to that level.

OK. And so here we're digging deep. Here's some peat and repeats of slides I've used before that give you the same information. But it's not that nifty little neat and tidy chart like I just showed you. And it's state level information, not from the almighty feds like that chart.

But anyway, here's examples again a lot like the chart I just showed you, but just in bullet format. Again, all of those different higher institution degrees associate's, bachelor's, higher ed. Again, occupational licensure. CNA is an example there. Occupational certification like automotive. Registered apprenticeships.

If you're in the CTE system in California, obviously, that works. And then the big one that we've talked about a lot in IET. And we brought it up at CAEP a little bit as well, is any recognized program in your local employment, trainer provider list. We've brought that up a lot with IET.

Where maybe you've got something, I'll bring this back. Maybe it really is a program that really looks and acts a lot more like something in that left-hand column here. That is something that really, if you use all the benchmarks it really looks like it fails to rise to this level, rather than it passes.

But you know it's a really good program. And you know students are getting jobs at it. Probably way more jobs at it than any of these over to the right that are officially sanctioned. So what we've said, if you know things like that, what you really need to do is work with your local WIOA Title I, and get it on the official WIOA-- what we all want employment training provider list.

Sorry, I'm getting some excessive noise here. So I'm closing my door and trying to make that go away. So if my voice is echoey it's because I'm walking around trying to do some things here. Anyway, work with your Title I. Get on your local ETPL. Bottom line is if the programs on the ETPL, then that means by definition, it qualifies an official certification program.

Because if it's on the ETPL, that means locally, in your local region, everybody has gotten together and agreed that your program addresses an important need in your local area. And thereby, locally, you're recognizing it as something that contributes to job attainment in your local region.

So that was a big mouthful. And I said some of it walking around and standing up. So for all of those poor presentation reasons, is everybody hanging in there? Well, it depends. Usually security guard license, I've got to say is something shorter term. So there's a little bit-- how long is that security guard program.

If you're saying, hey, it's something that's-- OK, well then it's probably going to be fine. What I would say if you have a security guard program, you'll want to look through and make sure do they actually get some official license or certification when they finish that program. Yes they do.

So as long as you have that documented, then the short answer is yes. And again, what you're bringing up, Iris, I guess is why some of this a lot of times seems kind of loosey goosey, is there are big card carrying programs like automotive, and CNA that everybody does that's been pretty standardized.

Things like you're saying with security guard, probably a lot less standardized. So a lot harder to determine. Just say, what you need to do first off is look, has there already been something institutional put in place where there's an official certified licensed for that, yes or no? If yes, just plug it in and insist that they get that certified license.

If so, then go ahead and record them as occupational certificate, or occupational license. I'll just say, I've worked with a lot of you over the last four years or so though, where the answer is no. If that answer is no, then you really kind of need to do a little more legwork. The legwork I would recommend is working with your WIOA I, and getting that program on your local ETPL.

And usually, if you can prove that your local program contributes to students getting jobs, my understanding is usually your Title I really wanted to work with you because they're eager to have more programs that contribute to everybody getting jobs obviously. So I'll stop right here, everybody. But especially Iris, and especially Karen, is everybody hanging in there with all this information?

OK. There we go. Late returns, but everybody is looking positive, good. OK. So here are some positive examples. These are all great things, but just to show these are the sort of things-- kind of that left-hand column of the chart from a couple of slides ago. Things that are wonderful. Things for students to do.

But it's not long enough. It just doesn't quite cut the mustard for post-secondary things like health and safety courses, Microsoft Office training. There's ServSafe and CPR showing up again. Those real short term things are good to do, but they're not necessarily official post-secondary credentials. OK?

And then back to your question, Iris. Yeah, it could be a workforce prep milestone, yes. The WSCS certification I would say would be a good one to use for that passage of an exam. So if you're doing CASAS' WSCS, then yeah, potentially that would be more of like a workforce prep milestone, I think.

But what the WSCS would do is that would be a good way to cover that passage of an exam, and be able to show, yeah, we're running people through CNA courses, and through welding courses, and automotive courses, and so on. And we're using WSCS as a more foolproof way to officially certify that the students really are ready for the world of work. And so that I would say would be kind of the ideal model agency way to cover that passage of an exam. Does that make sense, Iris?

To your question, Chris. Yeah, it could be workforce prep. I'll just say the MS office usually is a stand alone. Yeah, that's an example of something that doesn't cut the mustard. I'll just say that's an example of something where I know some agencies have tried to work, and put it on the ETPL. When we first started IET I remember lots of contentious meetings where I had to point out that Microsoft Office just doesn't do it.

Some agencies really didn't like that feedback. The feedback I would give is, hey, if you know it's working, if you know that really leads to business skills, or tech skills, whatever. And you can prove that students are getting jobs and so forth, then work with your Title I, put it on the ETPL.

That's always my way of saying if you feel like it's a lot better than Yahoo's like me, say it is. Get it on the ETPL. And then you can by definition, count it as a secondary credential. Right. You could also use MS offices some of those other outcomes on the update record. Good point, Tonuche.