Neil Kelly: Thank you. Thanks, Mandilee. I'm just going to do a quick opener for everyone before I introduce Tim. So the reason we do these webinars is because there was budget bill language a few years ago that mandated that we track hours of instruction and leverage funds from a variety of sources that are called out in legislation. And so this is-- I think we're going into our third year of this.

And actually, starting last year, the legislature and the Legislative Analyst's Office started using this information. There was a bill on adult education charters, and they wanted to use this data to compare the hours of instruction, get an hourly rate. So this is critical to having good data quality.

And so Tim's going to go over what we're seeing the last few years, what we saw last year, and then recommendations going forward. So it is important that we maintain the integrity of this data, because it is being used.

And I could see down the road, although no one's told us this, that if they're going to create an average daily attendance model, this is definitely the data they're going to be looking at, because there's nowhere else that we get self-reported data on hours of instruction and expenses. So keep that in mind.

We have noticed some of the outliers that kind of put our data into a little worse quality, zero entries or entries that aren't supposed to be there. So we're trying to isolate those outliers. And starting this fall, we'll be working with consortia as we identify those outliers and trying to fix that with some technical assistance. And Tim will be working with us on that effort too.

So with that said, I'm going to turn it over to Tim Harmon. He's been working with us at least-- well, he worked with us on the program evaluation and now with this. So he's got a good idea of what we do in California with adult education. And so without further delay, I'm going to bring Tim Harmon in. Thank you.

Tim Harmon: Thanks, Neil. I appreciate your support and your help as we've worked through this project. And it's been a gratifying one for me to be involved with over time in getting familiarity with what you all-- the great work that you're doing there. Even though I'm not from California, I love that place, and I love all of the work that you guys are doing. And it's exciting to be involved in a project that I think really can help improve the quality of the work that you do as adult educators.

So let me kind of jump into this. Now, these slides are meant as a companion set of slides to work a separate report that has been produced. And that will be issued I'm assuming fairly soon, that provides a little bit more detail than what you're going to see here this morning.

As Neil mentioned, this is the third reporting cycle for this data collection requirement that applies to all adult ed member providers in the state. And so this reporting requirement has been through some evolution over the last couple of years.

And this next slide, which is a little busy, I apologize, but a lot has happened. A lot of changes have occurred in the policy. So I just wanted to highlight a few of these things as a way of setting the table, if you will, for the data that we're going to be looking at with you today.

This reporting requirement started out, as Neil mentioned, as a requirement implemented through the legislature that requires every adult ed provider to provide information on its hours of instruction and its expenditures in a companion reporting process.

And so when this started, it started out as a requirement for reporting expenditures for hours of instruction, and also for supportive services. And it started out as a requirement for expenditures for the seven key program areas that these have been retained. But also, that those seven program area expenditures were to be allocated across 16 available funding sources or fund types, as we called them.

So that was a very ambitious policy, and there's been some adaptation to it, some simplification to it in the second cycle. And that PY '18-'19 process is when a lot of additional-- a lot of changes were made to the initial policy. That included clarifying instructions to avoid duplication of our reporting between CT and workforce reentry, or which is now called workforce preparation, because there was a question about, how do we allocate hours? So there was some guidance provided on that.

We adopted a federal definition of what constituted an instructional hour and a contact hour. We eliminated the requirement to report services hours. So it just focused strictly on instructional hours.

For K-12 providers, guidance was provided regarding tracking hours of instruction for integrated courses. These of course then combined ESL and CTE, et cetera.

And then we altered the fund source expenditure allocation requirement into required fund sources and optional fund sources. So the required fund sources are the main program, California Adult Ed Program, the Workforce Innovation Opportunity Act Title II, the Noncredit Apportionment, CalWORKs, Perkins, Local Control Funding Formula, Jail Ed Funds, Fees, and In-Kind.

So those are the required funds sources. So if you are using those funds sources, you should be reporting on them. And then the other ones are all optional. I won't mention those.

And then we defined in-kind expenditures and fees, and those definitions were aligned with the definitions that are used in Title II. We all at Title II adult ed federal reporting requirements.

So there is a bunch of changes there that really simplified the reporting requirement for that year. And then for this most recent year, we didn't make a lot of changes, but there were some. We clarified that members with expenses but no hours of instruction or hours of instruction but no expenses should not be reporting in this requirement. So this is only a requirement for those members that are doing instruction basically and that have their four hours of instruction, and therefore costs of instruction. So if you're not doing instruction, this doesn't apply to you.

And so for that reason, we're saying we don't want to see zero entries for either hours or expenditures in any of the program categories. And that's prohibited by policy, but not technically. So you'll see we had some folks that snuck around the policy requirement, and I'll show you that in a little bit. So that's just an overview of some of the changes that were made to reporting requirements.

So jumping into how this works, basically, as a reminder, the guidance for this most recent reporting requirement was issued on the 27th of August of last year. And then the webinar was done in November of last year. And then the reports for last year were gathered in two phases. And this has been the process since the beginning.

An estimated hour and expenditure data was requested by the 1st of September. And then the final or certified-- so-called certified hour and expenditure data was requested by the 1st of December. Actually, we didn't really draw those down until the 7th of January, so we gave people a little extra time to get those data in before I took them down and started working with them.

And in fact, it's kind of a good thing that we did that, I think, because about 27% of the final reported transactions were certified a week or more after this December 1 due date, and 16% were certified two weeks or more after the due date. So we had some folks that kind of lagged in a little bit with the reporting. And by stretching out the due date just a little bit, we got more data. In fact, we got all 71 consortia and 408 member organizations, which I think is universal compliance. But I can't guarantee that that due date gets stretched out again like we did this past year, so word to the wise.

OK. So for the most recent data submission, my firm, Workforce Enterprise Services, was asked, again, to analyze the data to do five things. Look at the hours of instruction by program area; look at expenditures by program area and fund type; to compare those across provider types, particularly K-12 versus community colleges; and then compare all of that across the three years that we've been doing this so far; and then finally, look at data quality.

And that's very similar to what we had done in the prior two years, except of course, we're adding an additional year to the mix. And so like I said, this is the third time that I've been involved in doing this.

So to start with, hours of instruction. And when looking at the hours of instruction, we were trying to get a handle on, what programs have the greatest number of hours? How is this different across different types of providers? And how do the PY 20 results compare with prior years?

So Figure 1 shows the basic percentage distributions of hours of instruction by program area. And so as you can see from there-- and you're probably not at all surprised and note this-- this is very similar to what it was in the prior year. The ESL/EL Civics, short-term CTE, and ABE/ASE account for the vast majority. In fact, 87.7% of all instructional hours that were reported for PY 20 in those three categories. And the rest are relatively small by comparison.

Table 2, which you're looking at here, shows the totals and the percentages of hours of instruction by program area down the left-hand side, in the seven program areas, and by provider type across the top of the table. Basically, community colleges and K-12. And there's a tiny bit of other that aren't either one of those things.

But so for the '19-'20 program year, over 78 million hours of instruction were provided by CAEP members. And about 57% of those hours of instruction were provided by the K-12 school districts, which include elementary school districts, high school districts, unified high school districts, county offices of education, charter schools. What else? Joint powers associations, regional occupational programs. I think that's it.

And about 43% were of the hours of instruction that were provided by community colleges and community college districts. And the other providers, which include workforce development boards, libraries, and the like are kind of like a rounding error in this process. They barely registered. But so it's mostly K-12 and community college, as you know.

But here's something interesting. The reported hours of instruction, that 78 million hours of instruction that you're seeing there, that increased substantially from prior year. And it increased much greater than the increase in expenditures, which is only-- it was a 20% increase in hours of instruction from the prior year versus only a 1.2% increase in expenditures.

And almost all of that reported increase in hours was from community colleges. In fact, the total share of all hours of instruction reported to the state by community colleges increased from 32% in the prior year to 43% for the current year, as you see here. So 43%. That's a big increase.

So I'm thinking that what we had was either some pretty important changes in how reporting was being done or we captured some people in the reporting this year that we had missed in the prior year, or perhaps some combination of the two things. But that's a very big increase in the share of hours of instruction attributable to our community college members.

So Table 3 then compares the hours of instruction by program area across the three program years, from '18 to '20. Again, as I mentioned, there's been a substantial increase in reported a number of instructional hours for most program categories from '18 to '20, especially for adults with disabilities, which a 59% increase in hours of instruction for that second category, AWD, Adults With Disabilities.

Also, big increases for K-12 success, 55.6%. And workforce preparation, 58.9%. On the other hand, there's been substantial decrease only for pre-apprenticeship, which went down by 20%. And this is hours of instruction.

Now, I'll show you that chart. There's a separate table for expenditures, which I'll mention in a minute or show you in a minute. But those were-- there were increases in expenditures as well, but not as dramatic and somewhat different. So that's an interesting process, but we'll get to that in a sec. So anyway, that's the hours of instruction across the three years that we have data on so far. And you can see some pretty important changes over time.

Here's the pie chart for expenditures. Again, we're looking at similar things for expenditures as we looked at for hours of instruction. What programs have the highest expenditures? What fund sources have the highest expenditures? How do expenditures vary across different types of providers? How do those things compare across the years that we have?

And so Figure 2 shows the percentage distribution of expenditures by program area. And again, as was the case with hours of instruction-- and almost all of the expenditures fell into one of the three categories of ESL/EL Civics, short-term CTE, and ABE/ASE. The distribution of expenditures across program categories is very similar, in fact, to the distribution for hours of instruction, which I guess would make sense. One would imagine it would be the case, wouldn't it, that where you're spending money is also where you're doing instruction. One would hope that they would be fairly similar.

So in addition to reporting expenditures by program area, the seven program areas that we talked about, the state CAEP office reporting policy requires consortia to allocate expenditures for each program area across the various fund sources or fund types that contributed to these program expenditures so that CAEP can have a more complete picture of how locally available fund sources are being leveraged to support operation of the programs.

So Figure 3, which you're looking at now, shows the percentage distribution of those expenditures by fund source type. So the thing that stands out here is that nearly 85% of all expenditures fall into one of three fund types. California Adult Education Program, Noncredit apportionment, and WIOA Title II AEFL funds.

Reporting of expenditures associated with each of these fund types is required, as you know, as is reporting for fees, in-kind contributions, CalWORKs, LCFF, K-12, adult ed jail funds, and Perkins funds. So reporting expenses for the other categories are optional. But as you can see, it's almost all in those first three categories.

So it's not impossible for me to know, of course, whether or not this reflects the fact that those fund sources simply aren't available or leveraged locally, or whether it reflects that they're not being accurately reported. I can't really know that, because the only way to know that would be to have some independent verification of what funds are actually available locally. And we'll talk about that going forward, how that works in future policy maybe that we could look at to try to get a better handle on what's going on.

So Table 4, moving on. Table 4 shows the totals and percentages of expenditures by program area and provider type. So we have our seven program categories, and again, provider types. This is very similar to the table you saw earlier for hours. This one focuses on expenditures.

So for the '19-'20 program year, over $821 million were expended by CAEP providers. Nearly 3/4-- 73.9%-- of total spending was done by K-12 districts, and about 1/4 was spent by community colleges and community college districts.

So you can see there, there's a difference in provider type between what we saw with hours of instruction where community colleges were about 43% of the hours of instruction, but now here only 26% of expenditure. So there is a very different pattern across provider types when we look at expenditures. The pattern across the program categories, those categories down the left-hand side, is quite similar between hours and expenditures, but the pattern of expenditures versus hours across the two different main types of providers is quite different.

OK. So Table 5 looks at the expenditures by the three program years, '18, '19, and '20. So again, there we have substantial increases in expenditures for K-12 success, 30% over the two-year period. Adults with disabilities, 13% over the two-year period. And let's see. ABE/ASE, 11.5% of the two-year period. So those are the big increased ones.

There were substantial decreases, though, in expenditures for workforce preparation, 31%. Pre-apprenticeship, 29%. Overall expenditures were up just 7/10 of a percent, basically unchanged over the two-year-- over the-- well, across, actually, the-- yeah, across the two-year period. A little bit more up from '19 to '20, but basically not a huge change in expenditures. There's a lot of money being spent, but it's not increasing a lot. The hours, though, did increase a lot, so that's an interesting phenomena, and I think it's worth discussing.

OK. So that's the basic overview of hours and expenditures and who's spending what on what. And so now I want to talk a little bit about the data quality side of this, because as Neil mentioned, people aren't waiting around for this to get perfect before they start using it to do stuff. So it's all the more important for us to focus like a laser beam on what's actually going on with the data quality here so we can see if there's places where we can improve. So that's what I want to try to focus on here for the remainder of my time.

So this table looks at compliance with the reporting requirement. And all of these really kind of get at compliance, but this one has to do with the members reporting zeros or not submitting a report, right?

So I think one bottom line thing I'd like to say is that there has been overall improvement in compliance with the reporting requirement. So for instance, this year we had 71 consortia submitting certified reports, representing 408 member organizations. Compared to last year, 66 consortia and 392 members. So I think we got close, if not universal compliance with the basic requirement to submit a report.

And Table 6 shows the number and percent of CAEP members who either did not submit a report or submitted a report with zero total expenditures or zero total hours. Because if you give me a report with zero expenditures, that's kind of like [inaudible]. Same thing as zero hours.

So about 8% in 2018, 9% in 2019, and 7% in 2020 failed to report expenditures. So that's not hugely different. That hasn't really gotten-- it's got a little bit better. It got better from '19 to '20, but not hugely, right? Still about 7% that are not submitting data for expenditures.

And then for hours, though, it's a little bit better for hours. There's been a kind of steady improvement in hours, particularly between '19 and '20. So in '19, we had about 11% of providers that gave us reports with no hours of instruction in it, and that's down to 6.6%, partly I'm assuming owing to the fact that-- the policy that said don't do that. So a few still did, but mostly you didn't. So that's good. That's an improvement.

And the ones that were showing zeros in neither category, it's down to 6.62%. So improvement, but maybe not as much improvement as we'd like to see.

OK. Another way of looking at this data quality question is, what data categories are people reporting for? Now, this could be either-- this could be data quality, but also could just be these are-- it could be reflective of the fact that not everybody's running every program, which I know they are not.

But what you see here in Table 7 is the percent of members providing a non-zero response to a major report element. And I'm only showing you these categories where we had at least a third of members say something, report something.

So in other words, there's a lot more categories. If you look at-- when you get the full table, the full report and you look at Table 7, you'll see that there's a lot more categories out there where somebody's reporting something, but they're all really tiny except for these. These are the ones where most of the reporting is taking place, and they are the ones that you would imagine you would find. ABE/ASE, ESL, short-term CTE. On the expenditure side, same thing. Same categories. And then for the fund categories, really just the two. CAEP funds and WIOA Title II.

Most everybody-- all the other categories, in fact, have fewer than 30% of members telling us anything, right? So there's these categories, which is basically almost all the activity, and then there's everything else, which is kind of like the rounding error part of this.

So I don't know-- again, without having independent verification of who has what kind of money, it's not possible to know, for instance, whether funds categories are being accurately reported, right? Without independent knowledge of who's running which kind of program at the member level, it's not really possible to know exactly whether or not member 27 is telling us something that they should be or telling us something-- or failing to tell us something that they should be, right? So we'll get to that in the recommendations, but this is kind of the big picture.

OK. So another thing we looked at for data quality was the consistency within the reports, between the hours and expenditures, basically. So in other words, what do we think about the underlying values that would be used to create a cost per value? Are they related to each other in some predictable way?

So in other words-- and they should be, right? Because members with high reported hours of instruction should also have high reported expenditures. And those with low reported hours of instruction should have low expenditures. And sure as it turns out, they do.

In Figure 4, they're very highly correlated as you can see in this scatter plot. Basically, it's got an R-square of 0.73, which for these kinds of things, where we're really talking about 400 dots on a slide, that's a pretty high R-squared, actually.

So that relationship is quite strong, I would say. And it is a little bit-- it's a little bit stronger for K-12 than it is for community colleges. You can see there, community college is in blue on the left and the K-12 in red on the right. And the R-squares are reported, but they're actually pretty similar, really, and have gotten better over-- I mean more related over time. And that probably reflects the

Mandilee Gonzales: OK. It looks like we might be experiencing some technical difficulties with Tim, so please just stand by while we try and touch base with him and reconnect.

Neil Kelly: And Mandilee, while we're waiting for Tim, I'll just go back to the chat really quick.

Mandilee Gonzales: OK. Thanks, Neil.

Neil Kelly: OK, I tried to answer most of the questions, and let's see. I'm going to table Lonzi's question for when Tim gets back or at the end. I think we've covered most of the questions. If there is a question we missed, do you want to-- John was saying-- wondered what leverage funds by program-- by provider-- oh, OK. Yeah. It might be a lot worse for 2021. We're kind of anticipating that.

And I answered Kelly's question. Let's see. I think-- I think that's about it. So let's see. John said cross-referenced program expenditures. We would know if they spent money on program and then reported zero hour. So let's wait to ask Tim that one. And then John still has a question about dollars to hours.

So Kelly's question, is there a way to compare the annual program area reports in NOVA? Well, the way they're set up in NOVA is if you go into the NOVA screen, you go to CAEP, and you click on the program area reports, it'll bring it up by each consortia. So you could look up, if you're comparing, you have access to all 71 consortia reports.

So technically, Kelly, you could look-- oh, but you only see one year. Yeah, so they're not displayed three years. You have to go into each year and look at those reports, unfortunately.

So yeah. You'd have to do it that way, unfortunately. We don't have it set up. It would be nice. It's kind of like what it does in LaunchBoard, but we're not as sophisticated. But something to think about if we do some more programming, if we want to enhance that reporting effort.

And I think we are planning to make some enhancements to this process to not allow people to put certain numbers. Like if it's ESL and it's fee-based, you can't put a number in because it's not allowed by Ed Code to put fees in ESL. Things like that. So we require some programming to get that done.

But we can also, like Kelly mentioned, maybe have some reports. And if anybody's interested, next week, next Friday we're going to show you all the different reports that are available now in NOVA, and we could think about adding some more. But it'll show you how many people have done their annual plan, which members haven't done their Q4, or all kinds of different reports you can do. Any kind of deliverable in NOVA, we now have a report that you can see who's completed and who hasn't out of our 400 members that access NOVA.

So that'll be next Friday. I think the same time, 12:00 noon. And Veronica Parker's going to be leading that presentation. So let me just check in with Mandilee to see if we resolved our technical difficulties.

Mandilee Gonzales: Thank you. Unfortunately, we are unable to get in touch with Tim. So at this point, we can hold for maybe another minute if there's any other questions possibly, Neil, that you could help answer. I do know that there was-- he mentioned earlier that he was experiencing some erratic weather, which would interrupt his internet connection. Unfortunately, we can't reach him to identify exactly what it is.

Neil Kelly: Yeah, Tim's in Illinois. So if anybody's familiar with the weather back there, there are some extreme weather situations going on in Illinois. Midwest.

Mandilee Gonzales: Yeah.

Neil Kelly: OK. So I mean, if anybody has any other questions.

Mandilee Gonzales: Yeah.

Neil Kelly: Emma says, what about the expenses of salaries for director and consortia staff? In the past, I've asked about allocating a percentage to all programs, but none are tied to hours of instruction.

Yeah. So when you have-- that's your admin cost. And so what we've suggested in the past, you should prorate that across your admin, your overhead. Those kind of things can be prorated across the programs. And we've suggested you use the student enrollment percentages as a way of allocating that. But maybe your business or fiscal office has a better way of allocating overhead to program areas.

Probably not. That's probably not a drill they're asked to do. So you could always use your enrollment to allocate that overhead or admin across those program areas. Unless anybody else has a different way they did it that seems to work in their district or in their consortia. Feel free to share.

Mandilee Gonzales: OK. It looks like we have a question from Lori in the chat. And Lori, if you want, you can also come off mic, or you can read it off the chat, Neil.

Neil Kelly: OK. When Tim showed the required funding to report, does it mean that it's required to report something, even a zero? Or is it required to report an amount? As a basic agent, we don't get any LCFF.

Right. So in those funding categories, you can have zero categories if you don't use those funds. You're not required. I think what Tim was getting at is if you offer a program-- let's say you have ESL and you show hours for ESL, but then in the leverage fund, if you showed no expenses reported for ESL, that's a problem. So it's got to be consistent. So if you have funds listed in a program area, there should be hours that correspond with that.

And so that's one of the problems we had the last few years of people reporting one and not the other. And maybe even if their district was paying for it. I don't know what the situation-- I think John Werner had a couple of situations like that where the member wanted to report hours, but didn't want to report the expenses for some reason. And I'm not clear on why that would happen. But if anybody has a clue on that situation, I'd be interested to know.

But Lori, you don't have to report-- it's OK to have a zero, as long as there's nothing to report in that area. So if you're running an ESL class, somehow you're going to have to report those hours and actually show those expenditures. Doesn't have to be in every expenditure category or as far as leverage fund category. It can be just in one or a couple. Doesn't have to be all of them, if that makes sense.

OK. And then Shannon said, I thought we were not supposed to put anything if we had no instruction hours. Right. So yeah, let me double back on that.

So Emma, when you're dividing that up across program areas, you have to make sure you're offering that class to have that overhead listed. So if there are no instructional hours in a program area, you can't put overhead on it because you're not offering that class, so to speak. Does that make sense? And I think that kind of confirms what Shannon was saying.

Mandilee Gonzales: And it looks--

Neil Kelly: Oh, go ahead.

Mandilee Gonzales: Oh, sorry. I was just saying that it looks like Tim is back on with us.

Neil Kelly: Oh, good.

Tim Harmon: I am glad on. Yeah, can you hear me?

Neil Kelly: Yes.

Tim Harmon: OK. I wonder, Mandilee, if you could--

Mandilee Gonzales: Share a screen?

Tim Harmon: --do the slides, because apparently my crack internet providers decided to play with the lines today.

Mandilee Gonzales: Oh, no problem. Let me just go ahead and get into Presenter mode.

Tim Harmon: And I'm on my hotspot on my phone, so I don't think it's going to do video very well.

Mandilee Gonzales: No, no, no. No problem.

Tim Harmon: I apologize.

Mandilee Gonzales: No, you are good. Thank you. And what slide were you on?

Tim Harmon: Oh, pretty deep in there. Hang on. I was on the-- slide 19.

Mandilee Gonzales: OK. Thank you.

Tim Harmon: OK.

Mandilee Gonzales: There we go. Can everyone-- looks like everyone can see. OK.

Tim Harmon: OK. Sorry. I apologize. The-- whatever. The miracle of technology isn't always a miracle, I guess. All right. So Table 8 is looking at another aspect of data quality, which is extreme values. So these are arbitrary. I picked these. And no one said, these are your extreme values. I think going forward, we need to have a discussion about what constitutes an extreme value. But these seem reasonable to me.

In other words, that if you had less than $50,000 worth of expenditures, that that's something at least I'm interested in. It may be there are members who had very small grants that could fall into this category. But I just maybe wonder.

And then also, less than 1,000 total hours of instruction over for a year also seemed to be a pretty small operation. And again, we might have those, but I wasn't sure. They stood out to me. So I identified those in the table here and looked at how they changed from '19 to '20. And actually, we have more extreme values than in '20 than we did in '19 even.

So I'm not sure what's going on there. It's worth some discussion, I guess, to sort out what we really consider to be extreme values. And then maybe follow up with these members to see, is this really right? OK. Next slide.

So final thing I looked at for data quality was the year-to-year consistency. And I looked '19 to 20 and '18 to '19, the idea being we looked for-- we might have really big changes in one program or another from year to year. But it didn't strike me that it would be typical to have radical shifts in expenditures or hours of instruction total for a member from one year to the next.

Although again, I don't really know that. It could be that members are coming and going and having part years and things like that that I'm not seeing. And so this could be reflective of that, this data that you're looking at here. Like if a member starting at the beginning of the year. Then obviously, they're going to have very low expenditures for that first year.

But again, I'm only including members that were in both years. So if you didn't exist in one of the two years, you're not going to be on this table. So that's why we're 360- 370.

So the interesting thing to me was that there was a substantial portion of the members, like over half of the members had more than 25% variation in the hours of instruction from one year to the next, and about 30% of them had a 50% variation in hours of instruction. 111 members had a greater than 50% variation in the hours of instruction from one year to the next. Again, it's entirely possible that's exactly what's going on. But it does make you wonder about, what's actually happening there at the reporting activity level?

So that's why I thought these, again, might be some members that we might want to look at and say, what really is going on there? Is that really reflective of what actual levels of activity are? OK, next slide.

OK. So just to wrap this up then with some recommendations. And these are fairly similar recommendations for the prior year, frankly, but with some changes.

So the first one is that the CAEP Office should continue to investigate and where feasible implement technical improvements to NOVA reporting system to improve data quality. So I had three suggested changes that could be made to NOVA. I think they go from easy to hard, I guess. Easier to harder.

The first is prohibiting report certification of zero entries. And that's prohibited by policy now, but it's not prohibited by the reporting system. So we had 28 members that did that and 29 members-- did that for instruction and 29 that did it for expenditures. So that should be not possible to do. And that would be a technical change that probably wouldn't be all that difficult to make.

The second one there is requiring consistency between hour and expenditure entries at the program level. So in other words, it probably shouldn't be possible to certify a report that has hours of instruction for WIOA or for the main program. You know, CAEP. But no expenditures for that, or expenditures but not hours of instruction.

Again, that's prohibited by policy, but it's not prohibited by the reporting system. There's been improvement in this area, and we're much more consistent in this reporting cycle. But we still had 29-- in fact, all but one of the 29 members that submitted zero expenditure reports also submitted zero hour reports.

So it's not a huge discrepancy, but at the program level at least, we did have some people that were doing one but not the other. And even though overall they had both expenditures and hours in one category or another, they did it. So that's probably not a good thing.

And then finally, we could link report requirements to records of fund availability at the member level and require reports to provide hours and expenditure data for funded programs in order to be certified. So in other words, we'd have some separate sorts of data, which I'm sure we have, which says, this member is doing this.

They are providing instruction in ESL/EL Civics. It's a program that they're running. And so they should have spending and they should have hours. And if they don't, then there's a problem, because they're running the program. So what happened to the hours? What happened to the expenditures? So it just would provide an external source of information and allow you to make sure that the data-- that the report's being taking care of appropriately. Next slide. Hello?

Mandilee Gonzales: Yep. It's not going. There we go. Yep.

Tim Harmon: There we go. Sorry. Oh, there it is. OK. So this one is just that the state office should continue to support consortia and members with training and TA on the reporting policy and procedure, including best practices for allocating cost to leverage fund sources. I know this was a big question early on, but there hasn't been, to my mind at least, a lot of attention paid to it subsequently.

So that's something that's, I think, worth continuing to have as a TA process and to work with consortia members to gather input, making further improvements in the policy and procedure. In other words, just the TA and dialogue associated with that, obviously just keep that going. That makes sense. Next slide.

And then this third one is kind of a new idea. We felt that-- we recommend that the state office provide feedback to members on their reporting results, including unit cost estimates, prior to member submission reports for the next cycle. So that could include monitoring report entries for problems and notifying consortia of any issues prior to finalizing the reports in December. And then it's going to take some effort to do that, so we said we should augment that.

So I think the plan is to proceed with a model in which we have the process where we pass the September data through some additional checks and give some feedback to people and say, hey, this doesn't look right for whatever reason. It's either an extreme value or it's got missing things that we know should be present, et cetera. And give people a chance to take a look at that and get things cleaned up prior to the final reports being submitted. Next slide.

And then finally, we feel that the state office should consider options in addition to technical assistance to improve compliance with the reporting requirement once other recommendations have been implemented. In other words, we're going to do some technical things to try to make it easier to report. We're going to do some TA and training, and so forth.

But then we also feel that there should be some kind of review of the hour and expenditure reporting as part of periodic monitoring of members. And so that we fold this reporting requirement into all of the other things that you as members and consortia are required to do and comply with under the state programs, and that you be held responsible and monitored for that, just like you would any other requirement.

And frankly, in order for this to really get to the point where it has high quality and people have confidence in its use, we really need to do something to monitor it and give feedback to people in a more formal way if they're not properly complying with the reporting requirement.

So I apologize for the interruption, and that concludes my piece of this. If there are questions or discussion-- it sounded like you already were getting into that while I was trying to get back on the internet. So I'll turn it back over to Neil, Mandilee, and others, and you guys can ask questions if you have any.

Mandilee Gonzales: Thank you so much, Tim. And we appreciate that you were able to get back on with this and complete the session. I believe that Neil, unless you had anything to add, we can go through the closing, or if there's any additional questions.

Neil Kelly: I think there are a couple of questions that I tabled because I wanted to get Tim's opinion. One was by Lonzi. I'm trying to find-- OK. So here's a question for you, Tim.

Zero expenditures are not a problem, right? After all, providing services on a program area is optional, correct? And so what I think Lonzi is getting at-- and somebody else mentioned this too. How do we deal with-- and maybe, Lonzi, you can explain. So I'm just trying to unpack the question as far as providing services on the areas optional. But Tim-- and I think you're getting at this-- if we provide instructional hours, we need to show expenditures that correlate to those instructional hours, correct?

Tim Harmon: Right. That's what I was getting at. There's kind of three things. The first is just overall, if you're not spending something-- I mean, I understand individual programs, you may be or may not be running.

But if there is no instruction being provided of any kind, if you're like a non-- you have a program. You have money to do something but it's not instructional in nature, that's not a person that's reporting as I understand policy, right? So you're off the hook. So we don't want to see a bunch of spending for you but no hours of instruction, because that's not what we're doing this about. We're doing this about comparing expenditure, right? So that's one thing.

The other thing is, yes, obviously, if there's a program area that you're not operating in, we would not expect to see any hours or spending for that. But what we would expect to see is consistency, not some spending and no-- as long as they mesh up, right?

Neil Kelly: Yeah, Tim, quick clarification. So if you're doing outreach-- let's say you want to start ESL and you don't technically have ESL, but you're doing outreach and marketing. I mean, I think that's not going to be a lot of money, but we probably don't want to see that as an expenditure, because it doesn't correlate to the instructional hours. Maybe once they get up and running, we want to show that. How would you handle that?

Tim Harmon: Yeah, I would not commence hour reporting or expenditure reporting until I was providing instruction. OK. I mean, that's the cleanest thing, because it has to do with how this is meant to be used, as I understand the requirement. We're trying to get cost per, right?

Neil Kelly: And then in Lonzi's situation, if they're offering services in a program area, we're not asking for them to report that. I don't even know if it's optional, because when the program-- the services-- unless they're part of the instructional class. And that's kind of a tricky one, because you could put services into an instructional class. But if it's a standalone, then it's difficult to say it's really part of the instructional classroom, right?

Tim Harmon: It is. And that gets back to that issue that we joined early on in this process a couple years ago, which was, what are we doing about services? And as I understand it, we are saying, we're not asking you to track services, hours, or expenditures per se.

So we're only-- we're focusing in on instruction. So if we have money that's being spent on services, if it's not allocable back to the instructional hour or instructional time, then we're not capturing it. I think that's right. I mean, tell me if that's a mistake and I need to--

Neil Kelly: No, I mean, there's kind of a good point there, because yes, we are providing services, but this information is being used by the legislature for other purposes. So we do want to show the hours of instruction. And if they keep asking us, what's the hourly rate, or what does that look like, we can't conflate it with services, unless those services are embedded in the classroom. So it gets a little different-- gets a little difficult, because we want to display accurate information based on those hours of instruction.

Tim Harmon: Right. Yeah. And Neil, I think we thought that we would have liked to know about services, but I just think the conclusion was that it was just so fraught with my conceptual questions about how you really allocate that to the program categories. And that was where it kind of got so gnarly around how to allocate those costs appropriately and associate them with individual students and all that, and just became too difficult to manage. And I think that's why we kind of walked away from it.

But it does leave us with a little bit of fuzziness there around the cost equation, because we are expending funds on services presumably, and we're not really-- we're not really including that in this exercise. We may be capturing some other spending exercise, but not in this one.

Neil Kelly: Right.

Tim Harmon: I don't think you answered the question or not, but that's where we're coming from, I think.

Audience: Neil I just want to go offer another scenario. This is Lonzi. And basically, that is, for example, in programs for adults with disabilities, there are many expenditures that are not related to instruction. And how about supported services that are provided as well?

So we really need to do a better job on the definitions. I mean, we have program areas that are not really program areas. They are more like populations than program areas. And so they're not necessarily related to instruction. How do you differentiate CTE versus students re-entering the workforce if you're providing that CTE class for them to re-enter the workforce?

So those are issues that need to be probably addressed a little bit more closely for the reports to be meaningful, because otherwise there's just, as Tim indicated, there's a lot of fuzziness still. And I think that this information really should not be used for a decision making process, because it is very fuzzy in terms of how it's all being reported.

Neil Kelly: Agreed. We'll continue-- as Tim has shown, we'll continue to try to, you know, through the recommendations, try to make this data quality better or the quality of data better. Yeah, and you're right. There's certain program areas that are populations rather than programs. So it's kind of interesting how supportive services and things play.

So that's definitely something we'll have to think about, whether we get a field team together to look at that or-- because we do have an opportunity not for this round going into the fall, but for next year to do some changes on NOVA and things like that. So if we did want to make some changes, we do have that opportunity for next year to try to make some wholesale changes. So I appreciate the question and the comments, Lonzi. Thank you.