(SPEECH) HOLLY CLARK: Jay? (DESCRIPTION) California Adult Education Program. (SPEECH) JAY WRIGHT: OK, thank you. Hey, I'm going to share-- I'm just going to go ahead and share my screen, I think, here for the moment. So according to my screen-- or according to my Zoom, I should say, everybody should be able to see my screen. I guess, if some of you can just sort of say, yeah, you can see my screen, or yeah, you can see that goofy cover slide, or whatever. That's at least what my screen is telling me. OK, thank you. All right, so I'll point out, hey, this is going to look suspiciously familiar, maybe even disappointingly so, because, hey, I'm going to start with about 10 or 12 slides, the usual suspects. I think it read reports, but it was supposed to be another accountability. So no, it's not going to be the same accountability we did five or six weeks ago in the same thing, but I am going to, for 5 or 10 minutes, do some of that just to make sure. I see it's a mixed group. For some of you, it will be kind of silly, but for some of you I think it'll be worthwhile. Then we'll rearrange the furniture and dig into reports. This is not a dramatically different set of slides or different set of training topics than usual, but I did rearrange it because it has been noted-- I mean, a little bit. It's been noted a little bit. I've noted it myself that, hey, we always run low on time at the end. Hey, 5-minute warning. I've got 15 slides to go. And what do you know? Those are always the ones that have to do with the reports in TE. So what we did is rearrange the furniture a little bit and move that up front. So hey, what do you know? There's a good chance that will happen again because it seems like it always happens. But hey, this time, it'll happen with some of the stuff related to outcomes and services, or perhaps resources, rather than giving short shrift to the section on reports. I'll just say, hey, back to the mixed group. A few of you, I kind of think, might benefit from a little bit of basic data collection requirements. Obviously, there's an equal number of you that will benefit from that-- not at all. But we'll at least-- so there are some new slides for you in the report section that we haven't really spent a lot of time talking about. So without further ado, let's go, or we'll never make it. So again, just a few minutes, for starters, a basic data collection requirements. Again, for some of you, this will be silly, but for about half of you, hopefully, it will help. Hey, a review never hurts. So we'll start with the really, really basic slide. This is the one that I like to bill as kind of the most basic one that I try to use to synopsize our requirements as a CAEP-funded program. It's, if you're WIOA II, a very similar slide here, but just trying to capture the absolute basics of what everybody's got to do. So anyway, for everybody in the seven CAEP programs, you record enrollment at intake. We've always been very prescriptive about that. We've been that way for a long time. That is, you record enrollment as soon as that student sets foot inside your doorway. Day one, square one of your interaction with the student-- that's what should trigger you to record enrollment or complete an Entry Record. And then record outcomes after a substantial block of instruction. That's represented by the CASAS Update Record-- that we've always been a lot looser in terms of when you need to do it. For enrollment, we're very prescriptive. Do it day one, square one. For update, we say, you can do that according to your own agency's way of organizing your schedule. That is, based on your academic calendar, your course schedule, whatever makes the most sense for your agency, is how often you do updates. You're required to do it for everybody with 12 or more hours of instruction. You pre- and posttest everybody in the federal WIOA II programs-- that is, ESL, ABE, and ASE. By ASE, we mean Adult Secondary. That means high school equivalency and high school diploma. And then follow up for exited learners. That is, for some things, we care about data on who's exited more so perhaps than those that are with you at this time. We follow up for exited learners via that Employment and Earnings Survey and through data match-- we'll explain a little bit, of course, coming up here. But starting with those seven CAEP programs, this just clarifies what those seven programs are. For most of you, this probably covers everybody. But I know for some of you at larger agencies, you might have some other programs not represented in the slide. So just for absolute clarity, for CAEP reporting, you only need to include the students that are in one of the seven programs on this slide. In CAEP land, we tend to set it up as the big three. That's why it's set up on top with ABE, ESL, and Career Tech Ed. I'll point out there's two that are under Career Tech Ed, but technically, we consider them separate programs. They are a little bit different. Preapprenticeship is one of them. Workforce Preparation is the other. I don't think we have this clarification, so I'll point it out now. Career Tech Ed obviously involves workforce-related training. It's something longer term and focuses on a specific occupation. Preapprenticeship also does that exact same thing. It's also long term and also focuses on a specific occupation. But the standards are a little bit different. The day-to-day activities are probably different. It usually involves the person working at a job site or with a union. That is, it's more hands on in most instances than Career Tech Ed. Workforce Prep is also, well, from the title, workforce-related. It's situated a lot more closely to Career Tech Ed, but Workforce Prep is generally for courses that are short term and not occupation-related, things like resume writing and job interviews, Microsoft Office, workplace safety skills-- that is, activities that apply to all occupations, not one specific occupation. And then we've got these two over here to the left, one called Adults with Disabilities, the other Parents/K12 Success. Again, that's because the Legislature says so. The funding covers these seven specific programs. So when we talk about data deliverables, here's the basic-level synopsis of that. We have quarterly and end-of-year data submission. For quarterly data submission, that means doing it through the TE Quarterly Data Submission Wizard. That is generally a pretty popular feature. It takes care of your CAEP data, your CAEP Data Integrity, your WIOA data, and your WIOA DIR all in one fell swoop. At the end-of-year, you're using that same TE Quarterly Data Submission Wizard. If you're WIOA II, you also have a little bit of a certification process related to payment points, where you need to submit a copy of that TE Payment Point Summary. The law, it's on the WIOA II side, so we'll leave alone the details. But long story short, if you're in WIOA II, you earn payment points. Very important that the agency and CASAS agree no less than 100% of the time so we have a certification process that certifies mutual understanding that we all agree on the same number. End-of-year also has a couple of those personnel charts. In your TE agency record, you don't need to do that every quarter, but you do that once a year prior to end-of-year data submission. Also, quarterly, you're doing Employment and Earnings. Again, that's quarterly, just like the data submission is. The due dates are exactly the same. But it's a different TE Wizard in this case. It's the TE Employment and Earnings Follow-Up Wizard. I gotta say, it's a bit less popular probably than the Quarterly Data Submission Wizard, but more popular than it's used to be. I'd say, five, six years ago, it was a major source of confusion statewide. Most of you by now are getting the hang of it and doing a lot better job with it. So that's why I say its popularity is improving perhaps. (DESCRIPTION) Data Submission Calendar. (SPEECH) Here are our due dates. Our due dates have been the same for a long time. That is, it's usually a month after the last day of the quarter. We just got through October 31. Next due date for the second quarter is January 31. (DESCRIPTION) Data Collection Requirements. (SPEECH) A little bit we talked about entry on enrollment update. After 12 hours of instruction, little more on each one for Entry Records. Again, it's recording that student enrollment in a WIOA II or CAEP program. As you do so, please do collect gender, race and ethnicity, date of birth. All those demographics are required. Please do select education level. Please do select labor force status. Please do select barriers to employment. All of those things are required. I'll point out, there's screenshots from the CASAS Entry Record here so everybody knows exactly what I'm talking about. Also, for Entry Records, not really new anymore, but please collect barriers to employment. That field gets a little bit more important each year. It was new under WIOA. Gotta say, everybody does a good job collecting it. I'll just continue to use the same encouragement here because it does feel like that's one that's going to get bigger, not smaller. If you have students that are cross-enrolled in another WIOA program, that is, they're in Title I or Title III workforce programs or Title IV, rehabilitation, please mark which title or title numbers that might apply to the student. That's the TE Personal Status field. Historically, we've not done very well on that one. I'll say, last year, we spiked up a little bit. It is getting a little bit better. But it's still a priority. If you have students that are co-enrolled with your partners, please mark it. I'll sidestep a little bit. For those of you that are WIOA II, you've heard a lot about federal-level chaos this year. Hard to say what's going to happen, but one thing everybody pretty much universally agrees upon is, no matter how good or how bad, the net result of all this chaos ends up being-- everybody agrees we'll probably be tied more to labor, we'll probably have more requirements related to labor. So the more effort you put into getting to know your WIOA partners, the better off you'll be because it definitely looks like that. Co-enrollment and cooperation with those labor agencies is going to get more and more and more and more important. On the update side-- again, Entry Records, very prescriptive. Update's, we always say, a lot looser. We say a substantial block of instruction. That might mean a semester, a trimester. It might mean monthly. It might mean quarterly-- whatever makes the most sense for your agency. That applies to all of the Update Record, except for that one piece related to attendance hours. For attendance hours, you can see here. (DESCRIPTION) Three bullet points. (SPEECH) You should do attendance at least once a month. Those three bullets are not different caveats to the once-a-month but are just representing the three ways in which you might get those hours into TE. Whether you're doing it the old-fashioned way, using the Update Record, or whether you're hand-entering it directly into TE, or whether you're relying on a third-party import, either way, get those attendance hours into TE at least one time a month. (DESCRIPTION) Update Records. A table contains learner results and milestones. (SPEECH) OK, here's one more screenshot of the Update Record. This is just to clarify-- not really super recent confusion, but confusion over the last year or two. There's certain fields like Update Status and Update Progress that used to be super duper important and are now really best left blank. When people like me say, hey, leave something blank, you don't need to do it, no matter how I say it. That tends to yield a lot of confusion, no matter what. So just pointing out, yes, there are a couple that you don't really want to fill out anymore. But that's not to say the Update Record isn't important. It's as important as ever-- represented by the screenshot. The screenshot is Update field 9. That's the field that has all the various outcomes. The fact that we're talking CAEP training right now is one really, really big reason why we're now doing CAEP reporting, not just federal reporting. So in CAEP land, as a lot of you know, we have a lot more outcomes on the table, with the short explanation being we have seven CAEP programs instead of the three WIOA II. It's a larger number of instructional programs. So what do you know? A lot more outcomes are relevant. So really looking at those outcomes and marking as many as apply-- always been really important. But I dare say now that's more important than ever. OK, so that concludes the, quote unquote, "basics." I gave it really short shrift because, as promised by the description, we'd spend most of our time talking about reports. So I'm jumping to that right now. I'm just going to jump to it, but I will point out, hey, any questions about basic requirements? I am going to drop that one like a ton of bricks. For most of the rest of the time, we'll talk CAEP reports. If we have any time left over, I'll cover a few of those things related to outcomes. But for the most part, I'll focus on these slides related to reports for the rest of the session. Don't see anything, so I'll just jump right away. We'll start with the CAEP Reports Setup Window, or CAEP Tables Setup Window. We'll get everybody's foot on the rail with the basic TE directions. That is, you want to go to Reports, State Reports, CAEP Tables. That's the simple direction on the TE main menu for how to get to that setup window that you can see on this screen here. Not all, but most of the reports we have for CAEP are represented by that menu option called CAEP Tables. When you do that, because it includes most of them, that explains number two-- is as soon as you select that option on the menu, you'll get to the setup window. The suggestion is go to Report Selection. That's what you can see on the screenshot right here, the bottom option. But you click Report Selection, this does not necessarily apply to all TE reports. But it definitely applies to this option where we have not all, but most of our CAEP-related reports included within the same setup window. So mechanically, you want to go to Report Selection. I believe what you see on the screen is the default option. The CAEP Tables at the bottom are left unchecked. All of the other ones are checked. In most situations, you probably won't want to select all of these, so you'll want to click Deselect All and then just check the boxes for the specific reports you wish to generate. You can select all of them if you want. If you want to run all the CAEP Tables and all these other reports at the same time, the answer is go ahead, knock yourself out. But usually, you just want to run one or two. It'll be less time-consuming if you deselect and select one or two. And then back to the first point, this does include most of the CAEP-related reports in TE, but not 100% of them. There are a few of them. The big one is CAEP Data Integrity, which that one in a couple others are separate menu options on the CAEP main menu rather than ones you necessarily select using Report Selection. (DESCRIPTION) Slide title CAEP Summary. (SPEECH) OK, I'm still not seeing any questions, so we'll start with the CAEP Summary. We'll give a basic overview, and then we'll double back to it a couple more times. But this is the one that I call the hub of the CAEP universe. Hey, if I was on a deserted island and I could only select one CAEP report, this would definitely be the one I selected. You can see it here. I've always liked to present this as three reports in one. As you can see from the slide, simply left-hand section, middle section, right-hand section. Well, obviously it's organized a little that way, as you can see, but I also like to present it this way because we set the bar on this report at three different levels-- high bar to the left, middle bar in the middle, low bar to the right. We'll start with the low bar to the right. That's number three on your slide. That's the Services section. By the book, definition is that simply reporting out those students that receive short-term services-- that is, things that they might have done with the counselor, field trips they did, things like that; that is, things they did at your agency, but not necessarily activities completed in the classroom. I say low bar to that because we don't really require anything for the Services section other than an ID. So that's very low bar, just about all your CAEP-related records you would expect to show up in that right-hand Services section. The bar is a lot higher there in the middle section, labeled CAEP Outcomes. Those are all of our outcomes related to CAPE except for pre- and posttesting. You can see all the columns on your screen, but that bar is set higher. For Services, we don't require any of this stuff, but for Outcomes, we require most of the same things that you know and love from federal reporting-- that is, all of those demographics, like gender and date of birth. It requires 12 or more hours of instruction, and it also requires some kind of enrollment in one of those seven CAEP programs. Moving over to the left, Literacy Gains-- that bar is set a little higher. It's almost the exact same level as CAEP Outcomes, but a little bit higher. Literacy Gains includes all of the requirements as Outcomes, and in addition, has that same requirement that some of you know from the NRS Federal Tables, that is, and also requires that qualifying pretest. We think it's set up right. That is, Literacy Gains includes all of your information related to pre- and posttesting. CAEP Outcomes includes all of those outcomes you do for CAEP except for pre- and posttesting. So the part that reports your test results requires testing. All those other outcomes for everything else do not. So Monica asks if the Services will record some of those that are not in classes. The short answer is yes. That's a good point. The Services section, a lot of times, will include some of your, quote unquote, "clutter." That is, you're going to have stuff that maybe you didn't mark hours or demographics or program enrollment that most of the time is going to show up in your Services section. It's a little bit cut off here, but there's that row called N/A, which is grayed out across the board except for that column M, where usually it will give you a number. That Not Applicable will be that number of students that you might call clutter-- that is, ones that got out of the Services section but don't have any documented CAEP program enrollment. How do we know what each table holds? I'm not sure what you mean. I'm sorry. What do you-- I'm not quite sure what that means. Yeah, I'm not sure what you mean by that question. I guess you can rephrase it. Yeah, I really don't know what that one means on that one. What do you mean each table? You mean each report? Well, that's what we're doing now by the CAEP Summary. I'm still not sure what you're asking about. I think you just mean all these reports, and I'll just say, yeah, that's going to take another half hour to explain. That's what the training's for. That's a snarky answer, but that's the only answer I can come up with to your question. I assume that's what you mean. If it's not, I'm really not sure what you mean here. OK, yeah, that's what the whole training is for. So yeah, this is the first one. So hey, I'll just say, yeah, Sylvia, one down, another 10 or 12 to go. It's going to take some time, though, to get through all the different things we want to talk about, I guess. Sorry for that wiseacre answer, but that's the best I can do, I think, right now. OK, anyway, so that's the CAEP Summary. We'll move on to the Cape Data Integrity. This, again, is one that's very important, at the very least, because this is the one that you're submitting with each quarterly data submission. So the short definition is up there at the top. We've got 33 items on the CAEP DIR. We put them on the DIR, so obviously, we think they're at least somewhat important to CAEP programs and CAEP outcomes. It gives an item count of the number of records that relate to each of those 33 elements, with the percentage for each of those items. What I'd like to say about the CAEP DIR and all the other DIRs is the way to understand the DIR is to be able to understand all of this information that's at the top part of the report. If you can figure out what's going on at the top, you usually can do a pretty good job figuring out the DIR in its entirety. If that's a big ball of confusion, then the DIR will probably be confusing. But the analogy I like to use is a fishing analogy. That is, for starters, CAEP-- or the DIR has this big net, basically collecting all of the records in TE that might conceivably have something to do with CAEP. Once it collects all of that, it then sorts through it, throws out all the clutter. As it tosses out the clutter, it might find a few that have some valuable information, as represented by these different types of outcomes. As it is, I mean, again, this example, I admit, a little too clean for its own good, but it will throw out some records related to clutter. It might find some good outcomes as it throws out clutter. So TE is just going to point out if and when there's situations that have good outcomes but are going to be thrown out because it's basically clutter without program enrollment and so on. It'll weed through all of those different student records. It will discard the ones that don't really fit. Bottom line is it will come up with this number in bold. In this example, that 737 at the bottom, the significance here-- this sounds a little bit wonky, but that we have that 737 to serve as that uniform denominator for all 33 items on the Data Integrity Report. We see that as really important for consistency. That is, for all 33 items, we're going to find the number of issues. For each of the 33 items, that's what you see under that Item Count column. And then each one of those item counts will subsequently be divided, in this case, by 737. So that is, we'll divide it by the same number each time, so all of those percentages will be a, quote unquote, "apples-to-apples" comparison. We use that uniform denominator to calculate all the percentages on this report, so it will give you a meaningful number related to your own agency's data. So you look at the DIR. Some items are more important than others. This has come up a little bit. So I'm going to actually stop sharing here for a little bit, for a minute. I guess I shouldn't have done that because I was sharing my screen. But anyway, I'll reshare my screen, and I'm going to go to-- OK, that's not what I wanted, but hopefully, you can all see that CASAS California web page. Is that what you're seeing right now? Yes or no? Just wondering. Oh, OK, thank you, Holly. Thanks for the quick response. So I think most of you are familiar with this page. I guess I'll go ahead and ask, are you all familiar with this illustrious page, that is, the CASAS California Accountability web page? Most prominently, it includes the CAEP Data Dictionary, as well as the WIOA II Data Dictionary. That's what you see on your screen right now. There's many different resources on this page. I'll add, this year, we actually added a lot of new resources to this page that we didn't have posted before. So if you haven't looked at it in a while, good idea to do so. In late July, we did post a lot of new documents that weren't there before. But for this purpose, I'm going to this item under Administrative Resources called Data Integrity Report. We never really have time to get into all the items, and this training is no exception. But I will point you out to that DIR Help document that gives lots of detailed information for all three DIRs, including, of course, the CAEP DIR. It includes what we call Report Analysis by Quarter. That is, it looks at it first through fourth quarter. Kind of points out, when does a big number on the DIR potentially indicate a problem? Sometimes, you've got a number there, and it probably represents a problem. That is, items like missing pretest or missing some of those demographics-- there really isn't any time that you should have a high number there. Your pretest level should always be pretty good. Those basic demographics should also be pretty good. So you can see we mark all four quarters for those items because we feel like those should be in good shape all year long. On the other hand, there's things like pretest but no posttest, or less than 12 hours of instruction. That really depends on what time of year it is. So it shows things like first and second quarter are left blank because it goes without saying it's early in the year. It may just be kind of early in the year for everybody to have posttested or got enough hours. But hey, we get to third and fourth quarter. Now, we're looking at a situation where we should be in better shape. So that's what quarterly analysis does, is, again, it may not always be in complete agreement with what you think, I suppose, but for the most part, it should be-- and it just gives our opinion, basically, on the periodicity of when things should be copacetic and when maybe it's not such a big deal if it isn't. And then we also have the section called Notes, where we provide some basic definitions. In some cases, it's less of a definition and more of an explanation for why on Earth we saw it as a good idea to add it to the DIR in the first place. So we're looking OK on time, but to stay on track, I'm going to go back to the PowerPoint. I'm just going to sanity check. Does everybody see the PowerPoint again? According to my Zoom, you should. OK, that's what you're verifying. Thank you for that. So I'm just going to blow this up and move on-- move on to the next slide here on this presentation. So I'll just point out, just to get everybody's foot back on the rail, I'll go back a few slides. Here is that setup window. So we looked at that top option, CAEP Summary. We also looked at the DIR. Now we're getting into what I call supplemental reports-- that is, ones that we talk a lot less about. There's the screenshot in the upper left. So over here to the right, a few of these we're going to detail more, a few of these we're just going to point out that they exist. But here's some other ones that we want to at least point out, starting with CAPE Program Hours. More on that in a minute, but that's the one that you can use to report hours in NOVA. Hey, 'tis the season for that, so added a few bonus slides on that one. I believe that everybody right now is amidst adding that information to know of as we speak, so a little bit more on that a couple slides from now. But again, that's CAEP Program Hours, where you can just run it, see the aggregate number of hours for each of those CAEP programs, and basically just copy that numbers into NOVA to complete that NOVA requirement. We also have instructional hours reports called Enrollees by Hours and Service Enrollees by Hours. Those are the ones that relate to those three buckets, where you can track the number of learners with zero hours, versus those with 1 to 11 hours, versus those that qualify and have 12 or more hours of instruction. We also have a couple ZIP code reports. Anybody use those? Those are ones, I admit, we haven't-- that I don't think that many people really use. But we have one that's called Enrollment-- we have one that's called Enrollment by ZIP Code. OK, hey, I've got all this clutter on my screen. I'm going to get rid of it here so I can read my slide a little bit better. Too much Zoom stuff on here. Anyways, so we've got Enrollment by ZIP and City. We also have that Race, Gender, and ZIP. I'm not sure I found many that use that Race, Gender, and ZIP. I haven't figured out a good practical use for that one. But that occupies many, many, many, many pages. If you've got some good use for it, I'd love to hear it. It's a great idea. Gotta say, we've had trouble getting our arms around that one, though. That ZIP and City is the more practical of the two. That one, I think, maybe has a little bit of guilt by association because that other ZIP Code one is a bit unwieldy. ZIP Code and City, though, just gives you a breakdown of student enrollment by zip code. Some of you dutifully track that. Others not so much. But if you do, it can really help for your enrollment and for your marketing. That is, you can see areas where you have really strong enrollment versus areas where you might be weak. Most of you would say you've got areas where you're really solid, students really seem to show up, and you seem to do great with it. But other geographic areas that seem like they ought to be dead ringers for adult ed participation, but for whatever reason, they aren't-- that's a real good way to break it down and try to figure that out a little bit. The other one here is Outcomes by Hours. That one will point a little bit more about-- I say that one's a good one, for that can serve as a summary audit for CAEP enrollees. If you're really into this, you know, for NRS and for payment points, we have that report called the Summary Audit that I've always named the follow-the-money reports. If you really like to chase NRS or payment points outcomes, that's probably the best one to use. Some of you point out we don't really have a good one like that for CAEP. That CAEP Outcomes Enrollees by Hours, I think, is probably the best one where you can look and see how, of all those different enrollees, who might be missing 12 hours of instruction. OK, here's Program Hours. This is just the basic screenshot. If you open up your setup window and you just select CAEP Program Hours and generate it, this is what you'll show on your screen. Again, it simply lists those CAEP programs and gives you an item count of hours-- that is, the aggregate number of hours across all students at your agency for each of those CAEP programs. This is the easy way where you just get that NOVA number. You can just copycat these numbers directly into NOVA. No muss, no fuss, I'll point out-- and I should have done in this order-- here's another way to look at it. This is after going into the setup window and tinkering with it a little bit. But if you want to get a look at maybe some that might be elusive, or you want to see a little bit better idea of how TE accounted for the hours, you can see the total hours. That is, you get the higher number in column C for students that are enrolled in classes, designated for more than one class. If so, you're going to get more hours than you really have. So you've got the total hours that provides that duplicated number, and then the proportional hours is what's done after TE on duplicates. Sorry, I'm going backwards on you here. Sorry for that. But I'll just point out that before and after relates to that left-hand checkbox. You can see I have those two goofy arrows. Lots of things on the setup window. I don't want to go over all of it, but two things that relate here. One is that Use NOVA Format for Hours. By default, that box will be checked. If you just leave it alone, it'll give you that simple format. If you want that more involved format so you can better account for duplicated hours in multiple programs and things like that, uncheck that box. It'll give you the same bottom line, item counts, but it'll give you that expanded format so you can see where those hours are duplicated. I'll point out there's also those radio buttons up top. For CAEP program areas, that relates to Program Hours. It also relates to your CAEP Summary. But in CAEP land, a lot of times, they'll roll up things like ABE, ASE. They'll roll up all those CTE programs into one. So you can decide whether you want it to roll up or whether you want it to expand out and list ABE and ASE separately or list Prep separate from CTE and so on. So again, this is what I just showed you. Sorry, probably a little out of order based on the explanation. But back to Program Hours, here's a slide. We've had more on the accountability side of it, but this relates to hours by program. That is, if you identify it as integrated, meaning if you assigned more than one instructional program to one TE class instance, by default, those hours will be divided equally. Like, if you have an IET class, and you designated it for ESL, and you also designated it for CTE, if you do nothing, the default answer is TE will divide it straight down the middle. 50% of the hours go to CTE. 50% of the hours go to ESL. If you know that it's not really 50/50 and should be more like 75% ESL or something like that, then you want to create two separate class instances so you can allocate those hours accordingly. OK, here's another Hours by Program. Gotta get a little bit more into this one because this one's kind of my favorite FAQ here from the last couple months related to this topic. That's what I like to call unduplicated by program and duplicated across program. That's how we do hours and enrollment in CAEP land. By unduplicated by program, that means, if you have multiple enrollments in the same program, it's only going to be counted once. So goofy example, but if you have a student with 18 different enrollments and ESL, multiple classes, multiple enrollment dates, whatever, lots of enrollment, that's a little bit of a crazy example, but no matter how many enrollments you have in the same program, that's going to show up on the same row. It'll be counted once, and all of those hours will go in to the same program. On the other, duplicated across program. that is, if you have a student enrolled in more than one program, that will be duplicated. A lot of times, you'll see that, and you'll think that's a mistake. And I'll just point out, no, that's not a mistake. It's on purpose. So we'll stick with that same example from a few minutes ago. If they have CTE enrollment and they also have ESL enrollment, that CAEP Summary or Hours by Program will have that student included in different rows. That is, in my example, they'll show up for ESL, and they'll also show up for CTE. That's by design. That will credit both of those programs. There is no real way to say one program is more important than the other, so we purposefully show the person is enrolled in both. So it will show up in both item counts. I'll point out there is that unduplicated count. Here, we'll move in and show exactly what we mean by this. Looking at the CAEP Summary, this is where it usually comes up. So we're looking at that column B there to the left. I just blew it up a little bit. But you can see we've got the number of enrollees by program, we've got the total number of enrollees, and then it will show, hey, we've got a certain number that are in two or more programs. That is, on that left-hand example, that 472, is not necessarily the number of students, but it's the number of program enrollments. In this example, we have 76 students that are in two or more programs, so that weeds down to 367. If you're wondering why that math didn't add up, that, too, should be expected because, again, it's not necessarily two programs. It's two or more. In most of these duplicated examples, it's going to be two instead of one. But hey, in this example, you can see there's at least a few examples where they might have been enrolled in three or more. So again, don't expect this math to always add up because, sometimes, it might be three or more, or four or more. It's not going to be two or more 100% of the time. Hopefully, that makes sense. Then that arrow over there to the right-- if you want the number of unduplicated students, just look at that very bottom row. That will give you that item count by student. And then you can see that asterisk example. That gives you that reminder, giving in shorthand what I just tried to spend the last 5 minutes explaining. (DESCRIPTION) A table. (SPEECH) OK, here, we're getting into the weeds a little bit much. I'm not going to read this whole thing. But I'll just point out this is what I'm using to explain why I think this tends to cause a lot of confusion and why we have been getting questions on this one-- is when you're looking at federal reporting, it's the exact opposite. That is, in federal reporting land, it requires TE and requires all agencies in all states to unduplicate everything. So TE has a built-in hierarchy for those NRS reports and payment points. For what it's worth, that's always ESL first, ABE second, and ASE third. That one is-- why? Because the feds say so. But we always go out of our way to have that hierarchy because we're always required to go out of our way to unduplicate that item count. I'll just point out, over there in CAEP land, we've got the exact opposite, where it's for state reporting. There's a lot of, quote unquote, credit that's done by program rather than overall item count. So we go way out of our way to duplicate that item count for enrollment, and we also duplicate that item count for CAEP outcome. So you need to look at those bottom rows if you're wanting to do an item count by student. And then this one's a little bit out of place. I didn't know where to put it. But it also has a little bit to do with Program Hours, so I just left it in. But this is the big issue of the year back in 2019. Some of you, I'm sure, remember it fondly, but there was a big issue with how to count hours, specifically hours related to services. For about a year, people ask this all the time, and we would always give a nonanswer because we really weren't sure. After a year of goofy nonanswers, we did the research. It was super clear that service hours do not count. So ever since, we've been using this slide to answer that question. That is, it's got to be associated with an instructional program. You'll point out I don't necessarily say it has to be in the classroom because, obviously, you all know now you're going to have some distance learning classes that won't be in the classroom but definitely should count as part of instructional hours. But the point here is those hours spent with short-term services are not part of instructional hours. So those should not be counted as part of CAEP Program Hours, and those don't count when you're looking at that issue related to 12 or more hours of instruction. (DESCRIPTION) CAEP Hours Reports. (SPEECH) OK, here's a couple of those other hours reports. We've got Enrollees by Hours and Service Enrollees by Hours. These are the ones that govern it according to the three buckets. Some of you remember, a few years ago, we had these uniform CAEP goals. They went away three or four years ago. So they're no longer relevant, but that's just my explanation of why we added these reports and formatted the way we did. We still think it's a pretty helpful format because there still is a big imperative to get everybody with 12 or more hours of instruction. So it's a good couple of reports to use to try to manage the hours of instruction. Back to your question from earlier-- I think it was Monica-- I'll also point out this allows you to manage-- look at those people in the Services section. It's not the screenshot, but the twin brother or sister of this one, called Service Enrollees by Hours, has the same format. But it also aligns it to those different categories of services, with the short answer being the Service Enrollees by Hours is a good one to look at if you're trying to figure out whether that Services section of the CAEP Summary is valuable information related to students receiving services or is more related to just that clutter that doesn't have hours, or missing demographics, or missing enrollment, or things like that, so it just got dumped into that Services section. Again, Service Enrollees by Hours is good for that. The last thing on those supplemental reports is you have that Drill Down to NRS Monitor. I'll just point out, when you're using these hours reports, when you're using things like the CAEP Summary-- I'm just going a little crazy here. It's true for the DIR as well. I'm going to go back to the slide. But I'll just point out, whenever you right-click, there's always that option called Drill Down to NRS Monitor. This gives you a chance to govern how the monitor displays. The one that I find the most useful is this one on the bottom. It Includes dropped or qualified students. By default, it will show only qualified. A lot of times, when you're looking at the NRS Monitor, it means, Houston, we do have a problem. So this is one that a lot of times you might want to change before you generate the report and before you drill down to NRS Monitor because, a lot of times, I find it helps to be able to review that clutter and have all of that information with missing enrollment and less than 12 hours and so on to display in this drill-down. If you have Only Qualified selected, then the NRS Monitor will skim from the top when it makes the display for that drill-down. OK, the last part of it here are the TE CAEP Tables. So again, that's setup window. We looked at those at the top half of the setup window. Now we've got those CAEP Tables at the bottom half, which, as I pointed out earlier, are, by default, always left unchecked a little bit because, hey, people don't really use the CAEP Tables much. I'll just point out, 10 years ago, we probably thought CAEP would align a lot more to NRS and WIOA than it has. The mandate was to align, but obviously, we haven't really done all that great a job aligning. So we really didn't need these CAEP Tables that much. I'll Just point out the CAEP Tables are modeled very closely to those NRS Tables. The content and the format of these CAEP Tables-- I'd say, 95% to 98% the same-- is what you would see when you generate NRS Tables. So this over to the right isn't really how to use them but more why you might want to consider using them. That number one is the biggie. If you're doing pre- and posttesting in non-WIOA programs-- I think most of you are dutifully testing, but probably just restricting that to ABE, ASE, and ESL. But I know some of you are testing programs like Workforce Prep and CTE. If so, those CAEP Tables are useful because it has the information that directly relates to that left-hand Pre/Posttest section of the CAEP Summary. If you're selective about who those are, you might want to use the CAEP Tables as a way to follow up and figure out exactly which students might be wise to do that bonus testing. A lot of you like to do that extra testing, but you're kind of selective about it. So the CAEP Tables would be one way to get that crowbar out and figure out which ones might be wise to test and which ones may not. If you've got students in programs that involve co-enrollment-- like EL Co-enrollment, obviously, IET, or Healthcare Pathways-- the CAEP Tables, a lot of times, might show which of those are co-enrolled, why you did or did not do the testing. The CAEP Tables collectively, and especially Table 2A, can be that summary on it for that Pre/Post section. I pointed out those Outcomes Enrollees by Hours is a really good way to figure out who may be missing from that CAEP Outcome section because of less than 12 hours. Using these CAEP Tables will allow you to find out. That same information, if you're concerned, is with the Literacy Gains section of the CAEP Summary rather than the Outcome section. And then this number five is kind of coming along the bend as the front runner for why you might. That is, we've got Table 4B. We've talked a little bit more about 4B on the NRS side lately. But 4B only includes those students that completed a pretest and a posttest, so of course, you can use that NRS Table. But a lot of you are pointing out, hey, you might have some tests and programs outside of those NRS programs. If so, you might want to use that CAEP Table 4B as a good way to measure your persistence from a CAEP point of view. (DESCRIPTION) Two tables. (SPEECH) And then here, this is just summarizing that drill-down piece, that follow-up piece, where, again, if your concern is that left-hand Literacy Gains part of CAEP Summary, you've got numbers that you don't understand or don't agree with or whatever, use those CAEP Tables to follow up and try to figure out what might be going wrong. If your issue is with that middle CAEP Outcome section, then you can use that CAEP Outcomes Enrollees by Hours. At a bare minimum, it will allow you to troubleshoot who does and who doesn't have 12 or more hours of instruction, which anecdotally is the number one reason why you might get people excluded from certain sections of this report. OK, we're looking OK on time. We've got 5 minutes. We just have to wrap up a few of these others. We also have I-3 Reports. That is, I-3 stands for Immigrant Integration Indicators. Those relate to EL Civics. So most of you, I'll say, have really good numbers on these reports, but probably, a lot of you don't realize it or understand why your numbers are so good. The reason why is you're probably doing a great job on the WIOA EL Civics side. With EL Civics COAAPs, that's basically what that I-3 is looking for. That is, those EL Civics COAAPs relate very closely to those Immigrant Integration Indicators. So if you have students doing a great job on EL Civics COAAPs, by definition, that suggests that you're also doing a good job with these Immigrant Immigration Indicators. So there's some reports that show you data on how well your students are doing with this. I'll just point out the I-3 Reports, quite frankly, give you a lot better detail than anything we have for EL Civics. So if you're looking at more content-related information, you're wanting to know what your student performance is by specific COAAP, there are some I-3 Reports that probably give you a lot more useful information than what we have for EL Civics. And then finally, we have our CAEP Consortium Manager Reports. A lot of you, I think, already have this login access. If you're a CAEP consortium manager and you don't have access, I'd obviously encourage you to do so. You can email golive-- that's G-O-L-I-V-E-- @CASAS.org. Indicate that you're the manager for CAEP consortium X and you want to obtain that consortium manager login. They'll take it from there. But that will allow you to access reports at the consortium manager level. Good news is you can view reports more globally, view the data for all the agencies in your consortium. Bad news is we do go out of our way to disallow drill-down. Privileges-- if you're at the consortium level, we feel like that's an agency decision, but you can take a look. And here's a sample where you can compare and contrast data performance for all the agencies in your consortium. This is really just focusing on a checkbox-- not really what I always like to do. But this checkbox called Aggregate Multiple Agencies is really important when you're at the consortium manager level. But if it's checked, then it will basically give you one report with all the data for your consortium. If you uncheck it, then you'll get multiple pieces of paper, one report for each agency. You say tomato, I say tomato, but you use that checkbox to govern which of those methods you like. And I'll just say, it's 1:59, and I just finished that Reports section. So from my point of view, I think that's perfect timing. So I'm going to stop right here, where we're supposed to stop. I'll just say, that was the point-- to really focus on the reports. (DESCRIPTION) Text Holly Clark. CAEP Tap. (SPEECH) HOLLY CLARK: Yes. Thank you, Jay. And someone asked, can we get slides? And that is the reason Jay just stopped right now. JAY WRIGHT: Yeah, I can do that now, and I'll close out. Harold, to your question, all of the information we talked about is available on the admin portal, though it may not necessarily be the same title. But if you get into the admin portal, those results from the CAEP Summary and CAEP Program Hours and so on are definitely included in the admin portal. But I'll point out, though, sometimes, the labeling and the window dressing is not done exactly the same as the way I showed straight from TE. Hopefully, that makes sense. As far as I can tell, that's the only-- oh, OK, the Parents/K12 Success, that's one of those CAEP programs that relates to parenting. But it's not parenting. I'll just say, when we started out as a BG, parenting was not included by the Legislature. Lots of people complained mightily about it. The Legislature stuck to its guns and did not allow parenting, but it did compromise and allow Parents/K12 Success. That is, we're not going to give you money to teach students how to be parents, but we will give you money if you want to train parents to be able to do a better job helping their students achieve K12 success. That's just one of those programs we have in CAEP land. So anyway, before I go, I will share the PowerPoint. I did close it. Hey, what do you know? I'm having trouble walking and chewing gum at the same time. So I guess I just need to stop talking here and dig around and find that file that I can-- HOLLY CLARK: Well, and let me tell you that when it comes to CAEP, you're going to be doing your Program Area Report that's due December 1. So in CASAS, it might be K12 success. However, in the Program Area Report, you're going to see it as CSS, which is Child School Success. So just make a mental note of that, or a paper note of that, now that CSS is the same thing as K12 Success. And we are working to try to align the terminologies. That is a goal of ours going forward with CAEP and the different reporting agencies. There have been some name changes. It's going to take a while to get us all aligned. And, Kimberly, we are going to remediate this webinar, hopefully. Please-- I dropped the link for the evaluation in the chat-- do fill that out. We use the data that we get back. If there are 44 of you-- there were 46 at one point. If we only get one or two evaluations, it may not get remediated. But if we can get more of you to fill out the evaluation, that is what will drive the remediation because there is a cost to do so, and then we'll post it to the site. Once we do post it to the site, we will email everyone that registered to attend and let you know that it is posted to the site for your viewing purposes. So that is all we have. I did drop a link also for upcoming trainings that we have. We do have on November 20 an overview of TOPSpro Enterprise. If any of your staff, if you yourself, or anyone at your site needs that training, it's a great one to attend. Jay, thank you so much for your willingness to present this training today and for your flexibility with the timing, with our mishap of the scheduling, and for the attendees. I see that we had all of our numbers stay on to the very end, which is unusual, but that just proves what a great presenter you are to keep them all engaged till the very end. So we appreciate you as always, and thank you, everyone, for giving us your time today. JAY WRIGHT: Thank you very much. HOLLY CLARK: Thank you. Have a great rest of your day. Bye-bye. (DESCRIPTION) Holly smiles.