[music playing]

Speaker 1: OTAN, Outreach and Technical Assistance Network.

Jay Wright: So look and feel included, we're probably starting out with something not so new here, not so new here today. My guess is a lot of you have gone through this a lot. About half of you are definitely not new administrators, you've been through this many, many, many, many times.

Some of you are new administrators, so a few of you, this might be new but I'm thinking most of you have been getting started. We've had a couple meetings for new agencies. We've had a couple of the regular WIOA II accountabilities online. So I think the majority of you have gone through this. This is just picking and choosing from the WIOA II accountability, picking and choosing some highlights. The agenda nonetheless almost word-for-word the same as what most of you have seen in WIOA II accountability in other similar situations.

We'll talk a little bit about NRS. I'm not sure if that's come across anybody's radar yet or yesterday but NRS is National Reporting System, that refers to the feds. We'll talk a little bit about some ways that we're applying the federal funding in California.

And then we'll move out of the agency and state level and more to the basic student level with basic data collection requirements. We won't get into all the gory details, but we'll talk fundamentally what do you collect at intake. What do you collect at update? And then what are some things you need to do for follow-up?

This is not a CASAS assessment session. We're not going to dig into the CASAS assessment system either but we will have a few slides that talks about how you apply that CASAS pre and post-testing to meet state and federal accountability requirements.

Everybody's heard about payment points but pretty much everybody wants to hear about it again so we will detail that. Some of the real in-the-weeds things we talk about in accountability we'll kind of stay out of that but we will definitely identify what all the payment points are.

And then a little bit on special programs and focus areas. We'll probably stay out of that one. That one's a little bit more for accountability but I think there are a couple slides, there are a couple things like selecting course codes, like designating things for Perkins, designating some other things for CTE that obviously have nothing to do with WIOA II reporting but are nonetheless state-level requirements. So there's enough of those kind of things now that it's at least worth talking about. And then a couple resources like the OTAN website, CASAS website, et cetera.

So a little bit of an overview. I left this here on purpose. For most of you, this is something you've heard a lot but if you really are new this is important, that is National Reporting System or NRS. Again when we use that acronym we're talking about the feds. To be fair to them, they're not the feds, they're contracted by the feds to provide a means of regular evaluation for adult education nationwide.

It started in the '90s, it started as a voluntary program. 1998 is when I say it became not so voluntary, that's when the Workforce Investment Act or WIA federal legislation was passed. WIA was a huge piece of legislation covering a lot of different things.

Title II was the component of it that pertained to us. We went through WIA for a long time, up until 2016, that's when the Innovation and Opportunity Act, or WIOA went into effect. WIOA covered a lot of the same things as WIA. It was passed explicitly to replace WIA. That third bullet on the slide shows how yes, the train did leave the station on time. We started WIOA July 1, 2016.

Since that time, WIOA has been the proverbial law of the adult education land. It covers lots of things too. I like to say it's the federal legislation that puts forth those valuable federal grant dollars in exchange for the detailed testing and detailed data collection we all know and love today.

One more thing about the federal side about the data, we also have the federal tables. That's a big job of the NRS. Those are some reports that a lot of you have used. NTE, those are the ones that the feds put out. That is there is a very specific format that California and all states must follow when reporting all the data to the federal government.

A little bit on WIOA, just kind of overview, WIOA wide overview, this covers all the areas of WIOA. For WIA and WIOA we have four title numbers, Title II for both is the one that pertains to us. Title II under WIA and WIOA, the section that covers adult ed and literacy.

Title I are the workforce programs, those are the ones that, quite frankly, have the lion's share of authority and lion's share of responsibility at the state and local levels. Why? Because the federal legislation says so.

Title III is short-term workforce. Those are the one-stop centers.

Title IV, voc rehab, that serves individuals with disabilities, individuals receiving workers compensation. That again, has been the same since 1998 for WIA and WIOA. Some of the little graphics at the bottom not really new anymore but what I like to say is where WIOA has been a little different from WIA that is all of that interaction going on.

In WIA, if we looked at a slide like this, most people would have no idea what we were talking about. In WIA, there really wasn't any reason to know what these other guys were doing but in WIOA, collaboration is required at the state level. It's required at the local level. So interacting with your WIOA partners, 100% mandatory.

I'm not going to get into the Table 4 or other federal tables' weeds, but I do just want to point out that the fed tables exist. The big ones in my mind are Table 4 and Table 5. Table 4 is the one that reports measurable skill gains, mostly but not limited to those gains that students make between pre and post-testing.

You can get a measurable skill gain through pre-post for us in Title II. That's by far the most common way students make gains. They can also get one by getting diploma or GED. There's also ways that they can get certain workforce outcomes if they're in integrated education and training. But keeping it simple for now, Table 4 calculates all that. It has all of those various levels, six levels for ABE, six levels for ESL, and it calculates at each of those federal levels what percentage of students achieved a measurable skill gain.

Usually, we're looking at it from a pre-post format, how many of our students at each level made a pre-post gain. The way the table's formatted is it looks at all of those measurable skill gains and just looks at the percentage at each level.

I make a big deal of this because I do think overall, this is the name of the federal reporting game. What the feds really want to know at each level, what percentage of students made enough gain on their testing and/or achieved some other acknowledged outcome to move up to the next level to get that outcome.

That's how they measure us in California. They do it the exact same way for all other states. They'll look at our performance, assign us goals based on this table, usually a couple years in advance, and then we're expected to meet or exceed those goals at each level when they look and say, hey, California, you're doing a great job or not so much. This is how they're determining that information.

The other big one, again, staying out of details but just pointing out highlights, is Table 5, this is the one for follow-up. By follow up what that means is it's outcomes we're looking for, for students that exit the program, not for students that are attending right now.

That's just a fed say-so issue as well. There's certain outcomes, You can see here employment, wages, entering college, that are very important outcomes but for federal reporting the feds are just more interested in those outcomes for the students that exit. They're much less interested in those kind of outcomes if students get them while they're attending your program. They measure it more on what happens after they exit. Long story short is that makes the data collection process very, very, very, very, very, very, very, very difficult. We have to figure out ways to get data on students that have exited our program.

So this crazy chart is showing Table 5, you can see we're looking at employment, wages, entering workforce training, entering college. A certain amount of time after exit, the way the feds have it designed is to use data match. On the slide, you can see there's the areas for college and post-secondary where we have one data match with the chancellor's office, another one for employment and wages where we have a data match with the EDD.

Long story short is the one on the bottom uses a fuzzy match, it incorporates demographics. It's not perfect but it gets enough matches to where it passes federal muster. As a lot of you have heard on the EDD side, they've always required Social Security number, as you know, that's a big issue. So that when we relied on that data match that guaranteed exceedingly lousy results.

Yes, the feds noticed. Yes, they did call us on it. Yes, they did demand that we do a lot better. That's what brought up that illustrious Employment and Earnings Survey. Since about the beginning of 2019, we've continued data matching but we've also done that follow-up survey to try to bolster our data with more results.

A lot of you have heard we now have goals. We have made a lot of progress but we definitely still have a long way to go based on the goals the feds have set for us and so on. But again, this is the culprit of why we have to do that survey. This is the culprit of why we have those goals and why you listen to this stuff incessantly is we have this follow-up table for which has always been a lot of difficulty for us in California.

So just wanted to give the overview there, lots more we could say. I don't see any Q&A. So I'll just keep on going here. If you have questions, feel free to ask, I'll just keep talking. I think I have till 10:15 so I better just keep talking, wind me up doll or we'll never make it.

Speaker 3: Jay, we do have a question.

Jay Wright: Yes.

Speaker 3: Please define fuzzy match.

Jay Wright: OK, specifically that means they use demographics not just Social Security number. So I think specifically but I'm less than 100% sure but I think they use first name, last name, gender, date of birth. Again, not perfect, but that does give us way more matches than we get if we rely on SSN. So there's still a lot of people we probably leave on the table with that, but it gets enough matches that we can report good enough outcomes to where the feds leave us alone on that one. But the one that relies on SSN, obviously, they don't.

OK. So this is what I like to call the transition slide. I also like to call this one the follow-the-money slide. So this shows how it kind of transitions from federal to state. I like to say the buck starts at the top here with the Office of Career Technical Adult Ed or OCTAE, that is the component of the US Department of Ed that's responsible for us, that is responsible for career tech Ed and responsible for adult Ed. So the buck starts there.

Before I get into that I'll point out there's the NRS that we talked about a couple of minutes ago, not the feds but contracted by the feds. From a data point of view and from a deliverable submission point of view, they're kind of running the show. When we submit data, which we just did less than a week ago, we report it to NRS. They have the website, do a lot of the same stuff on a national level like we do at CASAS for California.

So following the trail under WIA and WIOA, it's always been up to the state to decide who runs WIOA II. In California, it's been CDE all along. So CDE receives that money on behalf of the State of California disseminates it to you guys, the providers, and then at the same time does that federal monitoring to ensure it gets applied and used according to federal guidelines.

And then I'll just point out there's us, CASAS at the bottom, we're doing a lot of the same quote-unquote, 'regular' evaluation activities at the state level collecting the data, aggregating it on a state level, submitting it. That is a lot of similar functions to what NRS does at the national level.

I'll point out we're one of three leadership projects. I make less of a big deal about that here because you're hearing from them plenty but there's three leadership projects, CASAS, CALPRO, and OTAN, each responsible for different areas. You're obviously, hearing from CASAS this time. I think CALPRO is yesterday and OTAN is tomorrow if I'm not mistaken. But sticking with CASAS here now and moving along, these are our deliverables, again kind of sticking a little bit at 40,000 feet above for this one but we have quarterly and end-of-year data submission.

I feel in my mind of minds anyway, it's a lot simpler the last couple years than it used to be. That is, the last two or three years, we've had the Quarterly Data Submission Wizard in TE, which allows you to do WIOA II data and CAEP data in one fell swoop. It also includes-- it includes both data submissions and both DIRs every quarter.

So long story short, is you meet this requirement by using that Quarterly Data Submission Wizard in TE. Most people anyway say it is easy, a lot easier now than when you were sending reports and PDFs and all that good kind of stuff.

You basically repeat that process at end of the year, you run that same Quarterly Data Submission Wizard, you get all that done on or before July 15th. Once you do that, then we have that payment point certification letter based on your Payment Points Summary Report in TE that you submit.

Long story short, is we have that certification process at end of year, we don't do that at the quarterly but we do that at end of year, of course, because that's the data that affects your payment points results. So we have to be extra special careful and make sure that the total you think you're due to earn is exactly equal to the total we think you're due to earn. We have that process to go back and forth.

Maybe it's fixing data, maybe it's reconciling reporting, maybe it's something else but we'll keep doing that until we ensure that what you think you're slated to earn is the same as what we think. That's a big part of that letter that you sign showing that you agree that you're slated to, at least at the time of July 15th or August 1, that you're slated to earn x number of payment points.

There's also that Employment and Earnings Survey that we already mentioned. Those due dates are the same as the data. It's a different Wizard in TE. I think most people would say that Wizard, not that hard but certainly a lot more complicated than the Data Submission Wizard. A lot of decisions you make there but long story short is there's a separate wizard that you use to do the follow-up survey.

It basically uses all of those federal guidelines determining who the exiters are for a quarter, who's two quarters after exit, who's four quarters after exit, generating that list and then automatically sending that follow-up survey to those students, sometimes via email, sometimes via text. It allows the student to automatically respond. Some do, some don't. There's more process where you then can follow up after the follow-up, so to speak, to try to get more results.

OK. This is just a quick table of the due dates. The due dates are one month after the last day of the quarter. So a simple example, we just had the last day of the fourth-- or the first quarter, September 30th, first quarter July, August, September. Obviously, it ends on September 30th. The due date is one month after the last day of the quarter. In this example, that due date would be October 31st. You can see the due dates are set up in that fashion.

The one exception is the one everybody hates, July 15th end of year is a little earlier. I think most people figure that should be later than the others, not earlier. But because of that October 1 federal deadline, there's just no way we'd make it any other way. So that due date is a little earlier.

You can see we have other deliverables in addition to the follow-up survey. And I think I put that all on one slide here for time. So we have these additional deliverables due each quarter. If you look at that beginning of year letter and the accountability training, we detail all these other deliverables. So we just point out that you need to do them here without getting into all the gory details but first quarter there's that deliverable to do at least one EL Civics COAAPs. There's the CDE CASAS training deadline on or before January 31st, there's the Continuous Improvement Plan and IELCE report on or before April 30th. And then end of year, July 15th, there's some other-- the WIOA survey is due March 31, and there'll be anyway, more information on a couple of these.

There's more to it but this is a good summary quarter by quarter. Every quarter you've got the data, every quarter you've got the follow-up survey. Most quarters there's usually some little goodie in addition to that, that you need to get done on or before that same streamlined due date. I'll just say I still don't see-- everybody's question-less wonders here. I don't know what-- oh, here it comes.

Jim Shields: Jay, this is Jim. You do have a few questions in the Q&A. Do you want to take those now or wait till the end?

Jay Wright: OK, sure. Fire them up. I keep seeing nothing up there.

Jim Shields: OK. We have one, this refers back to when you were talking about Social Security numbers. What happens when a student doesn't provide a Social Security number?

Jay Wright: If they don't provide it?

Jim Shields: Yes.

Jay Wright: Well, no grave consequences to the student or to you, obviously. But there is incentive in the sense that employment and earnings survey that we described a couple of minutes ago, only goes to those students without a Social Security number. If they have one then they're good to go through data match. There's no reason to bother the student with that survey. So not a huge consequence but there is a positive consequence if you provide that Social Security number.

And then I'll just add, we've been talking about this a lot here the last year with federal goals and so on, is hey, it helps our data collection efforts. We do have some goals, one of them is for providing Social Security number. There is a slide with those exact numbers I think coming up here soon.

Jim Shields: Great. And we have another question, is the follow-up survey required when we have the student's Social Security number?

Jay Wright: You still need to run the Wizard either way but the Wizard will automatically look and see who has Social Security number and weed those students out of that survey Wizard process. So the only students that get it are the ones that did not provide it.

Jim Shields: OK. Well, a couple more rolled in.

Jay Wright: I see now. Yep. All right, now everybody's feeling a little safer there, huh?

Jim Shields: OK. OK, so will the IELCE report and the CIP be gone over in more detail in the data and accountability [ INAUDIBLE ]?

Jay Wright: Short answer there is-- yeah, I see them now. Thank you. The short answer is yes, at least for IELCE I would expect to probably see more information on that after the new year. There's usually more training and things like that.

Probably January, February as we're getting closer in the regular accountability trainings, there's more information in terms of what to submit and when. As far as gory details on how to reconcile it, we don't really go over that but for IELCE certainly early, and I would imagine for the CIP it'll be a similar sort of timeline.

Jim Shields: OK, and then do staff proctoring CASAS, they might be meaning assessments need to be retained every year. What does the CDE training refer to?

Jay Wright: OK, that's the-- well, there's one CASAS, this is one that's really better for accountability but yeah, one CASAS implementation and one WIOA II federal accountability per agency per year. Yeah, that has nothing to do with proctoring. The CDE requirement is at the agency level, not individual level.

For proctors, we don't really have a formal requirement to renew that. Although, if you are a proctor, we've always recommended every three years you should renew that certification.

Cheryl, I'll just say that if I answer it the way you wrote it that's an hour we don't have. Need to be way more specific, not sure what you mean specifically there. The Wizard is just for the end of year, you're indicating a number of teachers, number of administrators, there's an area for certifications, there's a set of tables you fill out once a year. You don't need to do it every quarter but you do need to do it annually. You just fill out those tables when you do that end-of-year data submission. OK, hopefully, that covers it.

OK. Thanks. Sorry, I kept on seeing zero up here. I guess everybody's dutifully looking at it. So whenever I look there's no little balloon there like the 99 plus for the regular chat. Anyway--

Jim Shields: Yeah, you can go ahead and continue.

Jay Wright: Thank you. No good. Thanks and yeah, the more questions the merrier. If we get zero questions all I'm doing is scratching my head. If we have to give slides short shrift for questions that's a good thing, not a bad thing as far as I'm concerned.

But going back to the slides hey, we've got the Quarterly Data Submission Wizard. I've already said I think it's pretty darn easy but hey, if you disagree here's a link that just gives you the TE help for running that Wizard inside the software.

Similarly, and a lot more people would say, hey, there's reason for needing more help with the Employment and Earnings Survey Wizard but here's a link to several help documents that help you go through those steps in that particular TE Wizard. Long story short, is we've got it organized by quarter because some of the steps do appear a little differently in TE depending on which quarter you're running that information.

OK, here are those goals. I knew what was coming up soon. I did know these percentages but I still shirked it because I knew we had the slide coming up. But I think this came out early November. So we've had these for almost a year now that is, we talked about how we've always had a lot of trouble with this yes, the feds have noticed, Yes, the feds have called us on it several times.

So you might say a year ago they threw down the gauntlet, hey, we're tired of talking in platitudes. So they gave some specific numbers a year ago. It's an either/or that is the goal is to get 45% Social Security number or 60% response rate on that survey.

To be clear, they're not looking for 60% getting a job, they're just saying, hey, we need to get at least 60% of the students to respond to the survey. If we can get either one of those they'll consider the goal met, which sounds good. Maybe a little more context makes it less good in that it is a big pole vault, not a small one. Again unofficial results but based on what we just submitted October 1, we're still at about 22%, 23% Social Security number.

We have bumped up to the high 30s now, and response rate I think we were kind of stuck at the low 30s for a good year. It does feel like we've made some good progress there. More on that maybe next week when we do that statewide meeting. We're still a long, long, long way from 60% but there's at least some good news that it really does look like the '22-'23 year overall was a lot better in terms of survey responses than '21-'22. So there's at least palpable growth here.

The other big update with this is ITIN or Taxpayer ID my guess is a lot of you have heard about this one but this will be allowed in addition to Social Security number. Back to the hey, we're always doing lousy in California narrative with this, the big reason again relying on Social Security for that data match so EDD is leaving the door ajar, they're not opening it wide open but they are at least letting it be ajar say, where they're going to allow Taxpayer ID in addition to SSN.

Not sure if that helps you or not some feel like it really will be a big game changer, others not so much. But either way, if you have students that do not have SSN but do have Taxpayer ID, it's still helpful to provide that. There is a new field in TE to put it in and it will help with that data match and it will exempt the student from the Employment and Earnings Survey the same way it exempts students that provide Social Security number.

The second bullet, it looks just like an SSN but it begins with the number 9. Yeah, as long as the student responds, yes. As long as we get those survey results it's counted by completed results that we can report to the feds. So yeah, if they tell you over the phone that's the same as if they complete the TE Wizard.

And there was another one that got cut off here.

Jim Shields: You answered that one, Jay. It was regarding if the ITIN number can be.

Jay Wright: Oh, OK. Got it Thank you.

Jim Shields: And Jay, when you are answering, can you read the question real quick?

Jay Wright: Oh, sure. Yeah, sorry.

Jim Shields: Because I'm not sure everybody has the Q&A open. Thanks.

Jay Wright: OK. So I will just say, Bev, I think I answered your question. I'll leave you kind of hanging if you beg to differ but anyway, I think we've got these goals in ITIN, so great.

Here's one I'm not quite sure why I slapped this one down here but this is just repeating what I just said that we realize that early deadline is unpopular but we nonetheless need to continue it as is.

It seems like it made a bad situation worse but I think it has alleviated a little bit by attaching CAEP to the same deadline, most people hate earlier deadlines but it does seem like having those two dates be the same has helped more people than it's hurt. I'll just say a few years ago, we did change that so it's all the same date. That was actually was by a popular demand.

So are the percentage based on enrollments or exit or is the percentage-- I think you just mean how do we do the denominator, it's going to be the exiters for that particular quarter. So like we've got-- I mean, so employment two quarters after exit every agency has a big bucket of students that exit and so it's looking at everybody who exited sometime in that quarterly time frame. So that's the denominator.

So yeah, if it's an enrollee that's still with you and hasn't exited yet, then they're not eligible for the survey, they're not eligible for data matching. So yeah, they're not counted in any of this data. I think that's what you're asking, Laura.

OK, moving-- good, great questions. I'll just say but I better move along. So now we're kind of in a transition, again I did cut out some of that because we're consolidating two hours into an hour and 15 minutes. We're just sort of jumping right in but I will note we're moving out of agency-level issues and state and federal-level issues and more into student-level issues here with this entry record slide.

I'll also point out I always like using those old-school bubble forms to convey this. Nobody really uses those anymore. If it was 2006 you'd all be using them but nobody's using them anymore but it really helps to convey this to say on the entry record on the update record just saying.

So on the entry record meaning what you're required to record for the student at enrollment you go ahead and complete that entry record, that records that enrollment in a federal program that is they're an ADE, ASE which is Adult Secondary or ESL. As you're recording that enrollment in the official federal program, as you're doing that please do collect demographics, gender, race and ethnicity, date of birth, all required. Please do collect education level, please do collect labor force status, please do collect barriers to employment. You can see some nifty little screenshots of what those fields mean on the slide.

Here's a little bit more. This was kind of a marquee item for 2022 because we made goals required again last year after 10 years of it not being required. I think long story short is at least 98% of you were collecting goals over that time anyway, even though it wasn't required. Now that it's required it really hasn't been that big of a deal.

Another long story short is back in 2012 for federal reporting it stopped mattering, goals didn't matter any for federal reporting. That's why we stopped requiring it because the feds didn't need it for anything but over time we realized feds or no feds, we really need that information. OK, Jim, did you answer it Jim or did you want me to answer?

Jim Shields: The person was just thanking you for answering her previous question.

Jay Wright: Thank you. Great. All right, so moving on to updates, I will sidestep here and say since 1999 for those of you that are new, we've always had a little bit of a difference in the way we present entries versus the way we present updates. For entries or enrollment we're really finicky we say do that as soon as the student enrolls. Day one, square one of your interaction with that student you're required to document that student's enrollment and that student's participation in your program. That's pretty rock solid that you need to do enrollment or entry records immediately.

For updates it's always been a little looser, there's no exact timeline, we just say complete update records after a substantial block of instruction and said for updates you can kind of tailor it to your own agency, maybe it's monthly, maybe it's quarterly, maybe it's a semester or trimester system how you like. So that's what we've said all along, that you need to do entries at this particular time updates more up to you.

The last six or so years though, that we've been in WIOA the exception is for attendance because we're in-- we've got a short timeline I won't be goofy quiz master and just say there is a federal reporting issue called Periods of Participation or PoPs, it's just one of those things we don't have time for today. You can go to another session to get it. It's 10 or 15 minutes we don't have right now.

But for federal reporting entry and exit is based on attendance. There's fields on the update that matter like not at all. In the old days, it was up to you to determine that. Now it's not. Now it's based exclusively on student attendance.

Bottom line is that means you need to have attendance and TE for all your students because TE needs to calculate who's exited and who's not. It's based on 90 days of no attendance. So we're saying yeah, at least once a month get your attendance in TE, maybe you're doing it the old school way like we're showing on the screen where you're doing bubbling and scanning, more likely you're either entering it directly in TE or you're doing a third party import to get that information in TE.

Either way by hook or by crook, you've got to get that attendance in at least once a month. TE needs to determine for every student whether or not they're an exiter, and furthermore, how long it's been since exit for all that crazy follow-up reporting I just got through explaining over the last 10 minutes or so.

If you do not do this, then you'll get false positives. There's nothing to undo, you just have to live with the fact that you bothered yourself and you bothered your student unnecessarily because you're not doing a good job getting your attendance in.

Here's one more on update. This refers a little bit to in recent years, there has been angst, people we've had that information out about PoPs for a while. We've had information out about how a lot of those update fields don't matter anymore for a long while but some people still fill it out. So I've really been trying to discourage people from spending a lot of time, teachers beating their heads against the wall trying to fill out fields that are 125% unnecessary. That's obviously a bad thing, not a good.

But not to say don't do the update. This is the one thing on the update that I say more important than ever, this is why you still need to do updates, is we have all these outcomes for CAEP, there's a lot of outcomes we need to mark.

And on the WIOA side, for things like IET, there's a lot of things now more important that we never used to care about that now we do. So marking those outcomes for every student more important than ever.

OK, don't see any questions. So I'm just going to switch gears here again talking about assessment policy. Usually, these are a few slides but this year it's a bigger issue with the long story short being we've got those new CASAS tests available this year that is listening and reading STEPS for ESL, Math GOALS 2 for ABE, so a lot of new tests come across everybody's bow here at the start of the year.

For any year that CDE statewide assessment policy is the official document giving you guidance on placement, giving you guidance on pre and post-testing, listing what CASAS assessments are allowed for state and federal reporting. It also gives you all those do's and don'ts around testing and it also provides those guidelines for local assessment.

The local assessment policy moving to the next slide is what I consider to be probably the most critical piece from an agency point of view, that is the statewide policy requires you as local agencies to complete a local assessment policy and revise it at the start of the program year. So I'll say, this is the last time I'll say this, tis the season to update it. If you're doing it right now, quite frankly, you're a month or two late but hey, better late than never but at the start of each year, every agency should update the assessment policy. Who are your proctors? Who are your coordinators? What's your agency-wide testing schedule? What are your preferred tests for ABE? What are your preferred tests for ESL? How is your agency handling important issues such as test security? All of those things are things you should put in writing and list in your local agency assessment policy.

At a bare minimum if you get monitored, a 100% chance that will be something that gets looked at. Likely, it's the very first thing. So you need to make sure that's in good order. I've made a-- I always make a big deal about this but this year more than usual because again we've got STEPS, we've got Math GOALS 2, not all of you are jumping head over heels on it but most of you are at least kicking the tires on it and incorporating it into your agency's testing.

If so there's a 100% chance you need to revise your agency's local assessment policy at a bare minimum to talk about how you're using STEPS Math GOALS 2, et cetera. Even if you're not doing that, of course, there's still 100% chance you need to update that local agency assessment policy. But again, I think the other slides are kind of detailing what I've already been talking about,

OK, how often are outcomes zeroed out for reporting? I don't know what you mean-- I don't know so I don't know what that question means at all. I'm confused about the two different types of outcomes. I guess I don't know what that question means at all.

[ INAUDIBLE ] slide 19 that teachers do? So I don't understand your question Lars, and I still don't understand your question, Aaron.

Mark, where does that policy live? It's on the CASAS California Accountability page where you get the CAEP data dictionary, the WIOA II data dictionary. We've got attachments A through X. That's a little crazy I admit. I believe the statewide policy is simply attachment A and that local assessment policy template is attachment B.

OK, hopefully that-- Mark I do think I answered your question. Sorry, Lars and Aaron, I don't understand either one of your questions. Yes, there-- yeah, OK is that-- so yeah, for WIOA II reporting, it's annual-- that's something I usually don't get into because it's 10 minutes we don't have.

But I'll just say for federal reporting, I mentioned those PoPs. So for federal reporting it's based on Periods of Participation. Yearly is not what they use, it's all based on those Periods of Participation based on that federal 90-day rule, which is exclusively based on student attendance. That's one big reason why we make such a big stinking deal about that.

I'll just say one, allow me to-- excuse me while I talk out of the other side of my mouth however, that for Payment Points it's the exact opposite. For years for WIA it was always July 1 to June 30. For California Payment Points it's continued that old-school way where it's still July 1 to June 30th.

Got to say it's way out of date now but for pay for performance. I think everybody feels like there needs to be something a little bit more finite. PoPs don't really allow that finite way to look at things so it's easier to use a date range.

We're getting there, Kristen. A couple slides. I'm going to wait until after we get through these. I don't think it will answer all of it but it will tackle some of it, that is I want to get into some of the things with assessments.

I'll skip to this slide and then backtrack, Kristen. This covers kind of what you're asking and accountability this is three or four slides but just to be clear, for ABE, ASE, you use reading or math. For ESL, you use reading or listening. To your question, it's either/or, you're not required to do both. You are required to do one or the other. It's agency-level choice on for ESL you can do all listening or all reading or you can do both. Same sort of choice available for ABE.

But back to your question, you're required to do one or the other. Reading works for both listening ESL only, Math ABE, ASE only. I think that's what you're asking. Speaking, writing optional, I'll just say reading, listening, and math are the ones that are used for state and federal reporting. You're welcome to use writing or speaking but they don't necessarily have a role in this federal reporting like reading, listening, and math do. I think that's your question, Kristen. I guess you can elbow me in the ribs if that doesn't answer it.

Here's another one, where I don't want to go into gory detail but I did leave this here because I do think it's just kind of hey, the topical issue, pink elephant. That is, hey, we've got a lot of new tests out this year, the most-- the biggest new test year that I can remember in 23 years of CASAS, that is we've got listening STEPS, reading STEPS, and Math GOALS 2, all at the same time. So obviously, lots of information out there on new tests, way more than we have time to talk about now.

So number one, if you have been trying to figure it all out, there's a link, there's a bunch of pages on our website with everything you want to know about these new tests and more. I present these charts, there's three charts, one for Listening, one for Reading, one for Math GOALS 2, I'm not going to go over all the details but I just want to point, well, here's some basic information like level and form number.

I like to point out more for experienced users I guess but that's about half of you here. If you're a new user, I guess you can start having it be this way anyway so it won't be that confusing perhaps. But with the old tests, there was a lot more uniformity in my opinion across the different forms and the same series across the different forms and the same modality and so on.

When you look at these charts for Math GOALS 2, when you look at these charts for Reading and Listening STEPS, you can see things like scale score alignment, lots of variation. Things like number of test items and test time, same variation. So it's not good, it's not bad but it does require a little bit more attention.

As you're bringing in students to test, you might say using the sensationalist example here for Listening where the test time's all over the map. Good if you want to do Listening but just warning, hey, you just need to be careful when you do Listening. And that is you need to set up different test sessions to make sure that the student doing the test has the appropriate test time.

If this the Math GOALS 2 the best example, a little oversimplified but you can see at the high levels for Math GOALS 2, you need 75 or 90 minutes, that's what you need to have it be statistically accurate. I think a lot of agencies are used to that nice round number of 60, maybe for some of you it's 65, for some of you it's 55 but your average agency I think has a pretty uniform test time.

The old series overall were pretty conducive to that kind of consistency. The new forms are not at all conducive to that kind of consistency. Quite frankly, yeah, you do need to bite the bullet. You do need to do that extra scheduling in this example, you do need to meet, do need to make sure if you're using Math GOALS 2 that the lower-level sessions are given 60 minutes and the higher-level sessions are giving 75 or 90 minutes, which A, means you got to separate them out, offer multiple test sessions. And B, yeah, that'll probably cramp your style. If you do this, you're going to need to schedule 90 minute sessions.

If you have somebody with an E-level Math GOALS 2, and you only give 60 minutes, yes, those test results will be skewed. Yes, in that example it is 125% your fault because you're scheduling 60 minutes when you should have scheduled 90. Yes, that is absolutely something where you're artificially causing problems with your agency's testing. Sorry, there's just no way around it. You've got to do that extra scheduling, just saying.

The appraisal and the locator, the appraisal is the full load, so to speak that is at some roughly the same length as an A, B, C, level test. The locator is the one that's designed to be able to identify that level more quickly. Prior to these new tests, the locator was an eTest but never had a paper copy. That is something a little different about the new test, there is a paper locator, there wasn't before but the locator is shorter. It's designed to more quickly put that student in a level.

Sometimes people don't think the locator does it exactly right. The recourse is you can do the appraisal instead, that works just as well. It just means you and your students are committing a little bit more time but that's always a good way to do it.

As an alternative if the appraisal isn't working, the other reason for appraisal is like Title Is or Workforce programs, you may be doing it just to determine basic skills rather than a precursor for pre-post. If so you might want to do that appraisal.

OK, the other thing is scale score ranges, that's 10 minutes or probably more like 25 minutes we don't have right now. But that's another thing that's been a big issue I'll just say to avoid sweeping the pink elephant under the carpet or whatever, that is scale score alignment, scale score alignment is the short answer to why a lot of you spent years of your life over the last few years doing all that CASAS field testing and why we at CASAS spend years of our life on the same thing. That is all of that data is so we can align our scale scores to those federal levels. If we don't do that right we have a 0% chance of getting federal approval. So we must align those CASAS scale scores to the A through E levels. We got to do it for every single series. We've got a large number of calisthenics we need to get through successfully to get federal approval.

So long story short is that's a lot more difficult now than it was 20 years ago. We need to do a lot more analysis and a lot more alignment, which means those scale score ranges work the same as before but it varies from level to level and modality to modality. There's no consistency with that either we end up needing to adapt it by series and by modality. So a lot of you have benchmarks like 180, or 200, or 236, or 246 that a lot of you are used to using a lot to your advantage that really aren't doing you much good anymore. So you just need to look up those scale score ranges whenever it's time to figure things like benchmarks or test sessions out.

OK, it looks like no good. So I went a little bit overboard with that one probably. So with payment points, I'm going to just rattle off what earns payment points and then dig into a few of them. So we have some payment points that we call benchmarks, not really a big deal but a common terminology issue.

For the most part, when we use the term benchmarks and payment points we're talking about the same thing. But a lot of times yahoos like me always insist on using the term payment points, never benchmarks. I think the technical distinction is in California we have that payment points pay for performance system. All of those different outcomes that relate to federal reporting that gives you that pay for performance payment point is called a payment point.

Benchmarks are those outcomes from federal Table 4 that comprise a big portion of those payment points. That is all benchmarks are payment points but all payment points aren't necessarily benchmarks you might say.

I don't know if it's on this slide deck but if you go to the website and look at those links there's all kinds of information on that. In the WIOA II accountability we've got Math GOALS 2, Listening STEPS, and Reading STEPS, those conversion charts just like you've known forever with the old series. I think I took some of that out of here though just because yeah, I needed to consolidate and I needed to trim somewhere.

But again back to here, the benchmarks are those same outcomes we talked about with Table 4. Complete a Level refers to those gain between pre and post, or you can also get it by getting high school equivalency or high school diploma, sorry about that.

OK, we also have student outcome data sets or SODS. That relates to EL Civics, we've got those three EL Civics focus areas. Not so coincidentally we've got payment points that relate to each of those focus areas. For civic participation, that's where we're using those additional assessments or COAAPs.

That's another 15 minute rabbit hole we don't have time for. I think there is going to be an hour somewhere that focuses on EL Civics so you'll be able to get in those weeds a little bit better with that presentation. But Civ part is the one where you're identifying locally which assessments to use and then getting CDE approval on the course's website.

Cit prep is using CASAS assessments specifically Citizenship Interview or Government and History. And then there's that 243 not really new anymore but new over the last five or six years, that's that integrated EL Civics where you're combining ESL and EL Civics with workforce training. From a payment points point of view, that's using COAAPs just like Civic Participation. The difference is for 243 there's a smaller number you can use.

Not sure of exact numbers but there's roughly 60 COAAP areas out there for which you can choose what to do with your students. Roughly 20 to 25 of those 60 are considered, quote-unquote, workforce appropriate. Those ones that are workforce appropriate are the ones that you're allowed to use for 243 IELCE.

OK, and then IET Outcomes, same thing as what we just talked about for the 243 SODS but it also is referring to those that are enrolled in IET or Integrated Education and Training, that's a checkbox you need to check at the class or student level. You're just going to have to live with, you're just going to know that has to happen.

But in any case, in IELCE, the goal is to have most of your students co-enrolled in some kind of workforce training, maybe locally, Maybe with the employer, maybe with Title I, maybe with somewhere else. In some cases, though, they may be doing the extra work on the ESL side but they may not be ready or may not be willing to enroll in workforce training.

When we started this we agreed the ones that are enrolled are doing more work and usually merit a higher dollar amount. So oversimplified here but the data for this one is exactly the same but potentially the dollar amount might be something different.

OK, this one kind of flaps in the wind here, probably should have moved this up a little earlier but this is the attestation. We started this two years ago, the last couple years we've done this on January 31. We're doing the same thing this year but it's moving up to December 15th. So around mid-November this year, because we're moving up this deadline, we send everybody attestation.

Trying to make that long story short, I think I guess I tried to get out of these we do that HSE data match. The HSE data match is what really pays off students or pays off agencies I should say, for getting GED or HiSET, we use that data match to verify the results you report in TE.

What we found a few years ago is a lot of discrepancies we'd go back to agencies and verify, yeah, the discrepancies are 100% truth, so we needed to go back and redo the data match for several years in a row.

So we brought up this attestation as a way to give everybody a fair shake, and review those totals after the data match and give you a chance to kind of wave your arms if you feel like the result was not what was right so you can look and verify that after the data match. Sometimes the agency's right, sometimes it isn't, but it gives you a chance to point out discrepancies.

The other one relates to high school diploma. For high school diploma, there is no data match. You're just marking in TE that they earned it but for auditing purposes, they're looking at that certified list you keep at the district level in order to guarantee all those high school diploma payment points are on the up and up.

At some districts, you've got that list available when you're submitting your data, it's not a big deal. But for some districts, they might be a little bit slow getting that information completed. So if they don't have their act together at your district by August 1st, you just have to rely on what you put in TE. But it also gives you a chance hey, our district does this but it always runs late. So you can use attestation to then go back and make sure that everybody you marked in TE is on that certified list.

OK, so I guess that was just listing it. I'll just point out again for HSE, it uses that data match. For high school diploma, you have to verify locally whether it's on that certified list. There is no data match.

OK, so moving to EL Civics, we've got this is just showing how we determine who the quote-unquote EL Civics students are. That is, it's looking for the presence of an entry record in a class that you've designated for that EL Civics focus area. If you're looking at updates pre and post-testing, it's just following the same guidelines as for a quote-unquote, regular ESL student.

And then for payment points purposes, for WIOA II reporting purposes EL Civics students are instructional program of ESL 100% of the time.

OK, detailing civic participation, again civic participation is using additional assessments or COAAPs. Don't have time to explain everything about those but again those are those 60 different areas.

There's a bunch of information about this on our CASAS website. This is where you select which areas to work with your students on locally and get CDE approval for each of those areas on our CASAS website.

The maximum at the student level for this is three, in order to get that payment point you need to achieve what we call a complete outcome data set, that is you need to complete an entry, update pre-test and post-test for that student in order to get that EL Civics payment point.

For Cit Prep, we'll use CASAS assessments, not COAAPs, specifically Citizenship Interview and Government and History. If the student passes one or both of those they can get Cit Prep payment points and Cit Prep it requires that same complete outcome data set entry, update, pre-test, post-test.

For 243, we're using additional assessments just like we are for Civ parts we're just operating from a smaller list. The max for IELCE is three. Just like Civic Participation, it has that same requirement for complete outcome data set.

This one I probably should have moved up a little bit, sorry, there's a little bit out of order. I rearranged this because we're setting it up for last time. I'm seeing this I should have moved this up a couple slides earlier but in the Class Instance Record, you're designating the program for that class or if it's multiple programs, designate multiple. This is where we're-- TE is determining at that class level, which ones relate to federal reporting for WIOA II, which ones don't. That is specifically, which classes relate to ABE, ASE, and ESL? That's important when running things like Payment Points Reports or NRS Reports.

At the same time at the class instance level, if you're doing EL Civics, you have that focus area drop box, you're selecting from one of three areas. We've talked a few times already those three focus areas, those are what appear on that drop box, you select one and only one focus area if that ESL class applies to EL Civics.

OK, a little bit getting in the weeds on a few things here, a little bit new. This was about a year ago this changed, where for a while we required you to select the focus area, of course. If you're doing IELCE and they're in workforce training, you mark it either the class or individual level Integrated Education and Training. This shows that they're not just an ordinary EL Civics student, they're an EL Civics student in 243 that's also doing that vocational-related workforce training.

OK, moving on here, sorry, cutting off we've got about four minutes. I think we're in good shape time-wise. So a few resource slides here. This is just a screenshot of the CASAS California Accountability page. Somebody asked about this earlier, so here you go proof in the pudding. It's that section of our CASAS website for California agencies.

I'll also highlight here that Training and Networking page. I'll say the beginning of year stuff is on the CASAS California Accountability page, Assessment Policy, Data Dictionary, all those attachments, et cetera.

The kind of up-to-the-minute, well, not up-to-the-minute but the more updated type documentation is on the Training and Networking page.

We have those regional meetings. We have statewide network meetings. As we generate documents from those kind of meetings we post them on the Training and Networking page. So lots of wonderful information there too, including that illustrious monthly PowerPoint that CDE and CASAS do every month.

Here is just another plug for that same page with a little bit more information just reiterating what I just said. Also, that order guide is on that same page. That's what you can download, and as a WIOA II agency, you can order CASAS materials free of charge, including test units. CDE might say, hey, wait a minute, it's not free of charge, it's courtesy of the California Department of Education.

Regional and statewide meetings, we've had those for a long time. In COVID years, we started doing stuff virtually for all the obvious reasons. It was kind of an unintended bonus, we did virtual meetings statewide once a month, hey, what do you know, lots of people showed up. So we've continued to do that statewide WIOA II, statewide TE.

Usually at the beginning of the month, WIOA II usually on Tuesday afternoons, TE usually on Friday mornings. We also have several regional meetings. We've tried to move more regional meetings in person. Lots of different meetings available. You can look on our website or that CA Adult Ed Training for the schedule.

Finally, we've got CASAS phone numbers, CASAS email address, and I think that's it. There were some things I'm used to talking about that I skipped. Sorry for the hemming and hawing, I'm used to this kind of being the full two-hour thing. So hopefully, I didn't lead you astray with some of that hemming and hawing at the end where I had to cut some things out.

It's 10:13 according to my phone. So a couple more minutes. So great, we've got a high school diploma question. No, you don't need to do that.

So sorry, Jim, you want me to-- so for high school diploma students, do you need to change their education level when they earn high school diploma? No, you don't. I would discourage against it.

That education level actually needs to stay the same quite frankly because it's documenting exactly what that student had achieved at the time of enrollment. If we change it, then we're misreporting because when they enrolled they didn't have it, we want to accurately report exactly where that student was at the time of enrollment. I gave you more than you asked for just to point out, hey, why you would never really want to change that.

OK, are there requirements for testing? So it's not required. The requirements-- good, you're catching me on something I don't think I explained very well. That is again, the testing is required for those in WIOA II. That is the ones in ABE, ASE, or ESL. So you're correct, that's not CTE. It's not required.

Obviously, you're listening to the CASAS person so sure, great idea if you want to do that. We certainly strongly endorse it. We certainly would say there's lots of good uses for it. But I will reiterate it's not required.

Where CTE might use it to their advantage that's different than what you might do for ABE or ESL, is you might use those placement forms. For CTE we would generally say you don't really need to do A, B, C, D progress testing but it is a really good idea to give a reading or math appraisal or both upon enrollment as a way to just generally determine basic skills. That is you don't need to measure progress but you may want to make sure they have the basic reading skills and/or the basic math skills necessary to succeed in that CTE class before enrolling them. That's where I think your average CTE program would say yeah, testing might be helpful.

You do not need to get the-- so this is a good both sides of my mouth questionnaire and is pre-post required for HSE, HS diploma? Short answer is yes, it's required. The policy requires it. However, if you're looking at those Payment Point Reports in TE it will not take that Payment Point away for equivalency or diploma if they don't happen to get it. Sorry, have to talk out of both sides of my mouth. Hopefully, you got the right answer there somewhere, Aaron.

I'll say that was not your question Aaron. That was 0% your question and a problem completely because of my answer, not your question but hopefully, you still got the right answer there.

OK, Kristen. Right. Yeah, CTE is not covered under federal WIOA II, so yes, they are rightfully completely excluded. The feds have no interest in including CTE and WIOA II so we rightfully exclude them 100%. Yeah, we just can't do that or the feds will rightfully bop us over the head many, many times. OK, great questions.