VERONICA PARKER: You seen the agenda. So today-- [COUGHS] excuse me. Today, we'll be covering, like I said, a number of topics, all of which we think will help inform you in your new role. So our first session will be with the CAEP office. We have Dr. Carolyn Zachry, Gary Adams, and Neil Kelly. 

And they will be presenting on CAEP welcome, as well as program guidance. So providing a brief overview of the purpose of adult education and update on our funding, a walk through procedures, guidance timelines, and reporting requirements, and answer any questions related to CAEP. 

And then next, we'll be moving into a CAEP technical assistance project Overview, as well as doing a high level overview of the Nova financial reporting system. And that will be by way of a live demonstration. And then later on this morning, we will be-- WestEd will join us and they will go over a CAEP data reporting for launch board session. 

So they'll be talking about launch board as well as community college MIS reporting and how MIS and TOPSpro fees, the LaunchBoard, as well as other topics. And then we'll do a little lunchtime networking. And then we'll have our last session of the afternoon, and that will be CAEP data reporting and TOPSpro Enterprise. And so that will be provided by classes. And that will be our day of work training. 

Again, this is an opportunity for you all to be informed about the CAEP as the CAEP deliverables and how the different systems work together to provide the California Adult Education Program. So I will stop sharing my screen in terms of the agenda. And I will add it into the chat so that you all have an opportunity to have it for yourself. 

We'll also be posting in the chat a Google folder which will serve as a repository of all of the materials that will be shared today. Some of which are available to be share today, but because of fiber way compliance, they are not all available to be shared today. However, as we receive materials, we will upload them to that particular folder and inform you all about when things have been uploaded. 

And that will be something that you can take away as a tool and use for future reference. And it also has a couple of things that we'll use during the lunchtime networking-- the lunchtime networking session. So before I pass it over to the CAEP office, a couple of other housekeeping items. So everyone has entered the Zoom room and-- [COUGHS] excuse me, hopefully, everyone's able to hear me. 

And if you have any questions throughout any of the sessions today, please be sure to post them in the chat. The presenters will be monitoring the chat as well as myself and my colleague Mandilee Comstock. We'll be monitoring the chat. And any questions that come up, we will alert the presenters whenever there's a natural pause. [COUGHS] Excuse me. 

And if there is an opportunity for you to come off mute, the presenter will let you know and so you'll be able to engage in actual dialogue with them. And this session is being recorded. Again, there are multiple sessions that will be going on today. So for any reason you need to step away or you have to end your day with us, the recording will be made available. We will be remediating it and we'll make it available to you all. 

However, remediation does take about three weeks, so it will not be as fast as we would like. But you will still have the recording for future reference. And we are taking attendance, this is a smaller group. And so I think we will be able to capture everyone. 

However, if there's any reason why we need to private message you to capture your attendance, we will do so. So now I will start sharing again and pass it over-- [COUGHS] excuse me, to the CAEP office who will get us started Dr. Zachary, I think you're going first. 

DR. CAROLYN ZACHRY: Yes, all right. Well, good morning, everyone. Thank you for being here. I really appreciate you taking the time out of your very busy schedules to onboard yourself with CAEP. So Veronica, if you're done coughing, you've got to go to the next slide. Thank you. 

When we think about CAEP, CAEP is really this joint venture between the community college, Chancellor's Office, and the CDE. And it's really important that when we learn how to work together-- I just love this quote, versus against each other, things might start getting better. And when we think about the adults in California and the goals that they have, it really is important that we're working together and that you're working together within your consortium. Next slide. 

And the other piece I think that's really important to as a new lead when you're looking at adult education is it's really all about equity. And we've talked a lot about the importance of equity over this last year. It's a focus of the Adult Education Office, a focus of the superintendent, a focus of the Chancellor, and I know a focus of Gary's as well. 

And when we think about how adult education can help break down barriers for students, that's an important piece that you are all contributing to in the work that you're doing within your consortium and leading your consortium. Next slide, and back to thinking about equity, adult education started in California-- I always like-- I always thought that California was the first state. No, we were the second. 

Massachusetts beat us out by a couple of years by starting-- with starting adult education. But in 1856 in California, adult classes were started in San Francisco. And they were for adults to learn English and vocational skills at that time. And now, between WIOA Title II funded programs and CAEP, we have well, well, over 400 agencies providing services for adults to better their education, to receive career training, and the opportunity to learn English. 

And just so if anyone out there is working on a doctoral dissertation, all of the history of California Adult Education is housed at Stanford University. And I'm also happy to say that we are working through OTAN, the CDE we've added on to a contract with OTAN, to take the orange history book and add on to that. And to bring in the next pieces of history, which really include everything related to CAEP. So next slide, please. 

Henry Ford said, "Coming together is a beginning, keeping together is progress, and working together is success." Again, this is what CAEP is all about. We came together under-- the next slide, please. Skip that one. Oh, wait, is that yours, Neil? 

NEIL KELLY: That's OK. 

DR. CAROLYN ZACHRY: OK. Go to the next slide. Next slide. I'm not sure these are different. I haven't seen the order now. OK, go back, I guess. So the idea of coming together was related to AB86. And we had-- AB86 started and the idea of coming together in a consortium model really added our adult education K-12 programs and our community college partners together looking at their region to help move adult education forward. 

And then when AEBD came into being and then renamed to the California Adult Education Program, the idea of keeping together was critical. And that working together is where you will all find that success for your programs and for the students in your programs. 

Next slide, I think. And then we'll skip that one. I think that's what Neil will cover. And the other piece that I think is really important, this one-- this session for me was just full of some quotes because I want to inspire all of you as you are embarking on your new leadership role. 

But Mother Teresa said, "You cannot do what I can-- you can do what I cannot do. I can do what you cannot do. Together, we can do great things." And again, that's the important piece about CAEP. Is how are you ensuring that the adults in your region that you're meeting your programs are meeting the needs? 

If one agency has a wait list for an ESL class, is there another agency nearby that can add an ESL class at the same time or a similar time frame so that those students can get the programming and the education that they desire so that they can move forward for their future rather than sitting on a waiting list and perhaps being discouraged, and then not enrolling once their name is called on that waiting list? Next slide. Are these next slide slides yours, Neil? 

NEIL KELLY: Yeah. 

DR. CAROLYN ZACHRY: All right. So go ahead, Neil. 

NEIL KELLY: But I will go into, like you said, the history of the adult education the California Adult Education Program from AB86 and how that foundation we built is what the program looks like today. So they're a little bit farther down on the slide. 

DR. CAROLYN ZACHRY: Yeah, right, OK. And so before I turn it over to you, Neil, I'd like to turn this over to Gary Adams for any words that he may have. 

GARY ADAMS: Thank you, Carolyn. I would just like to thank all the directors for joining us this morning in this effort and for all you do in your daily jobs as you face out into the next fiscal year and beyond. I strongly want to underscore what Carolyn has said, the importance of community colleges joining hands with our K-12 partners and addressing the needs of adults in our regions and in our communities. 

And the Chancellor's Office is committed to its strong partnership with the California Department of Education. And we hope that that is reflected in each of your regions and each of the work that you individually will do as leaders for your consortia. Thank you, Carolyn. And I will turn it to Neil. 

NEIL KELLY: Thanks, Gary. So Veronica, if you-- or Mandilee, if you could back up to the agenda. There we go. So I'll talk a little bit more about the history and background of CAEP, then we'll dive into the nitty gritty of the funding, we'll talk about the CAEP structure and the systems in governance at a high level, and then we'll talk about the deliverables as far as receipt of the state funds, what you're required. 

And then member effectiveness, how that impacts if you don't submit your deliverables. And then we'll leave a lot of time for questions. So you can ask any kind of questions that you have, and we can respond to that and hopefully satisfy all those questions you've had as new leaders or if you've been in this for a little bit of time, maybe something that you've been always wanting to ask the state, you can feel free to ask in this session, so. Next slide, please. 

Here we go. So the purpose of the adult education program, the state funding-- and I just wanted to reiterate. So when we designed the state funding, we also had in mind the federal program as well, that WIOA Title II program. So our metrics are really aligned with the WIOA Title II program, the pre and post testing, the transition of students. So we're really aligned with WIOA II. And so that's the framework that we use to build the state funding around, but we added things in the state funding that the federal program doesn't do. 

So it's a real nice collaboration there between the two federal and state funding. But for the state funding, when the legislature put this forward, they wanted to rethink and redesign the educational system, they wanted to establish those linkages for students across the adult education systems, like Gary said, between the K-12 adult and the community colleges, and provide adults with academic and career skills, develop plans for expanding and improving Adult Ed services by creating linkages between the two systems, and providing that capacity or seamless transition for students. So next slide, please. 

So really, as Caroline was talking about the history of the state funding from AB86 to AB104 and in the CAEP universe, because there was state funding before CAEP came along. But if you look at the history of adult education, there was flexibility around 2008, 2009. 

And for about five years or so, Adult Ed didn't have that. We went from like 738 million, maybe even more than that, down maybe 60% of that during the flexibility era. So with that, AB86 came along with a $25 million planning grant. And it was a two year planning and implementation grant started in 2013, went into 2014. 

The planning grant outlined the expectations for consortia development, expanded and improved upon that consortium via a regional model, and then prepared the consortia for planning and implementation that started as part of the E104, which is when the $500 million block grant came along in 2015 and 16. Next slide, please. 

So as part of that 25 million from AB86-- I think we already covered this. Next slide, please. What AB86 did and you see from eight years ago, we still have this foundation that we built as part of AB86. We established the AB86 cabinet, which included key members at the highest level from the Chancellor's Office as well as Department of Education. 

We developed a comprehensive certification of eligibility, which I'll go into a little bit more. We consulted with expert panels relevant to each issue and various organizations, had a stakeholder sounding board, used a transparent process to listen and inform the field through the development of-- let me say it again, of the certification of eligibility. Let's talk about that certification of eligibility in the next slide, please. 

So there was a cover sheet that the consortia had to sign as far as each one had to have a fiscal agent. And then each member had to sign a form saying they were going to participate in the consortia. And then they also had to come up with the project management plan that included a governance structure, which some of you are still using today, talk about membership and partnerships, shared leadership strategies, a communication plan and a budget, and then overall, a regional comprehensive plan. 

And so next slide, we'll talk a little bit about what that regional comprehensive plan looks like. So next slide, please. So the comprehensive plan included an evaluation of your current levels and types of services in the region, and that's still part of the planning process that we go through even to this day. If you're working on your three year plan for next year, you'll have to do an evaluation of current levels and types of services. 

We also did an evaluation of current needs in the region way back then, which you still are doing as part of the planning process. And then in this comprehensive plan, there were plans to integrate existing programs and create seamless transitions into post-secondary education or workforce. So that's still true today as transition has become a real strong suit for the California Adult Education Program. 

There's also, in this plan to address gaps in services, which you continue to do, how to accelerate student progress, how to leverage existing regional structures through your partners, and then collaborate with shared on professional development opportunities. So all this stuff is still-- even though it started back in 2013, it's still woven into the fabric and foundation of the adult education program that you're working on to this day. So next slide, please. 

So we're going to get into a little bit of the nuts and bolts of the funding. So if you're not familiar with how the funding is allocated, what happens every year in January around the first week-- towards the end of the first week of January, the governor announces the proposed budget for the coming year. 

So what we do with that information. We take it, we have a formula that we develop that will go into a little bit. We use that formula to determine an allocation. Because it's a proposed budget, we release by statute. We're required, by the end of February, to release preliminary allocations for each consortia. We also release that into NOVA. 

And so that allows the consortia, over the next two months, to figure out how much funding each member in their district-- in their consortia will receive. And so they use the months of March and April to have public meetings, get input, have decision making, and then by May 2nd, they are required to certify how much each member district will be receiving for the upcoming year. 

And then absent of any changes, like trailer bill additions subtractions or budget bill additions subtractions, then the CAEP money begins flowing from the state to the local district or fiscal agent by August in 11 monthly installments. And, of course, like I said, CAEP funding is based on a needs based formula. So if we can go to the next slide, please. I'll talk about that needs based formula. 

So education code 84911 listed out the factors that we're supposed to use in determining funding. And so these are the six factors we used in determining the CAEP allocation for the consortia. So no high school diploma-- this is all based on census or American Community Survey information. 

So no high school diploma, unemployed adults, adult population, which would be 18 years and older, household poverty, adult literacy at the seventh grade education level, and the ability to speak English. So all those were weighted and calculated to come up with a regional share ratio, and that's how the consortia got their funding for the consortia. 

So back in 2015 when we rolled out the funding, it was a twofold approach. We had the regional consortia funding, and then we also had a lion's share of funding going to the K-12s as what they call maintenance of effort, which gets confused because a lot of people use the term maintenance of effort. But about $337 million was earmarked for the K-12s based on a certification process using 2012, 2013 as a benchmark year. 

So what the legislature had the K-12 districts do is certify how much expenses they reported in 2012, 2013, they certified it, and then we were able to provide an allocation on that amount. And so together with the maintenance of effort plus the regional consortia formula, we got back to that $500 million mark. And that's what we've rolled forward to this day. 

Everyone gets set same amount as they did in the prior year, so it all goes back to this 2015 maintenance of effort and Regional Consortium formula and there are ways that members can be reduced, and we'll talk about that in a little bit. But there is education code that says you get no less than the prior year, so there is some stability there. Next slide, please. 

And so here is the funding. If you look at what goes out on the street, to either a K-12 County OFS entity versus a community college entity, the split's usually about 88% to the K-12 and 12% to the community college. And that's remained constant throughout the last five years. As you can see our increases in the funding, we've gone from 500 million starting in 1516, then we had three straight years of $500 million, and then in 1819, we got a bump. 

And then 1920, another bump. 2021, we had a bump but it was taken away, and so we stayed flat or even. And then this year, we got a sizable increase up to the current amount of $560,376,000. So let me pause there. I think we had a couple-- Veronica, do we have any questions in the chat or did anybody take care of that? 

VERONICA PARKER: So one, Victoria took care of. Beth wanted to know the err code for A4911. And then Janet wants to know, will CAEP allocations be affected/adjusted by the new census results? 

NEIL KELLY: So we're not-- we haven't had that discussion as far as the new census results. And that's up between the Chancellor's Office and CDE. But as of today, there have been no discussions about re-benching or incorporating the new census data. 

I know there are some discussions in the legislature about looking at different models, but nothing firm that has come out. But there is a lot of dialogue going on, so I won't deny that. But at this point there hasn't been any discussion between the two state agencies of incorporating that census data just yet. 

All right, appreciate the question. OK, so next slide. OK, this one got a little messed up. But this shows you-- so the last slide showed you what it is when it gets down to the street. But as you know, consortia have fiscal agents, some consortiums are direct funded. 

So when you look at how that is allocated based on the governance and fiscal structure of the consortia, it actually goes from an 8812 split down to a 7723 split. So if you look at all the K-12 fiscal agents and direct funded members, it's really $432 million or 77%. And the community college fiscal agents or direct funded members goes up to the $128 million or 23%. 

And what I mean by direct funded members. So if you're not familiar with this, when you-- as a consortia, you decide whether you have a fiscal agent that receives the funds from the state or you can opt out of a fiscal agent and have that money sent directly to your district. But you have to make that decision when you certify your allocations on May 2nd. 

And that's the certification-- the CFAD, which is, I think, it's Certification of Fiscal Administrative Declaration or no, it's Consortia Fiscal Administrative Declaration. That's it, CFAD. So you have to make that choice of what funding channel your consortia is going to use before the year starts. And you do that as part of the May 2nd certification process. 

And so I just wanted to show the difference between what actually gets to the street versus what goes through our fiscal structures that are built in around the state. So next slide, please. 

VERONICA PARKER: OK. And now before we move forward, there was a question. What are the pros/cons for the two funding channels? 

NEIL KELLY: OK. If you could just move to the next slide and then I'll answer that. Because this provides a little more information. So in that question, as far as pros and cons, so you can see from the fiscal agent standpoint, when we look at consortia with direct funded members, it's been steadily going up a little bit each year. 

And so if you look at K-12 County office fiscal agents, it's gone down a little bit, gone up, gone down. So it's holding steady. And then community colleges as a fiscal agent, that's gone down from 37 from five years ago to 28. And so that's where the change has really been. Is a lot of consortia started out with the community college fiscal agent and moved over to direct funded members. 

So the pros and cons. What I would say is depends on your fiscal agent. When we started this, a lot of fiscal agents wanted to see every single they treated it like a grant. Because in the beginning, it was a block grant. And so they wanted to see-- and they treated it like a reimbursable contract. 

So members of the consortium weren't able to spend funding until they provided justification and documentation. And so that caused a lot of slowdown in spending. And so that resulted in us changing from a block grant to apportionment, and changing the name from the Adult Ed education block grant to the California Adult Education program. 

And so it took a couple of years for the districts and the consortium fiscal agents to realize-- and we put other education code things in, like you have to get the money out within 45 days just because fiscal agents were holding on to the money. So that was a con when you had a fiscal agent. 

But once we resolved that, then it became the issue of getting the money to the local district. And so there was a problem at the state level of moving the money, it came to the Chancellor's Office, and then the State General Services required that we did a contract with the Department of Education-- California Department of Education to move that-- like you said, the 400-- you saw that 400 and-- what did they say? 

$27 million or so or whatever that amount was, that 77% had to be moved over to CDE to be dispersed to the K-12 County offices, which included some fiscal agents that had community colleges under that fiscal agent. And that process took like six months because it was a contract. But because it was such a large amount of money, it took forever to get through the state processes. 

And so that included getting the contract approved, getting it put into the state fiscal system. It just wasn't used to having such a large sum of money. And so that was a big con to being a direct funded K-12 district at that time because it took forever. Some people weren't getting their money until December or January. 

And it got to the point where districts were borrowing money from-- the adult school was borrowing money from the district and having to pay interest, and teachers were getting paid, so it was all kinds of bad things. And so this year, we have hopefully resolved that where we don't have to go through the contractual process to move those fundings over to CDE. 

And we're hoping that we've streamlined that process. And I think this week or next, that funding should be deposited in your local districts accounts if you're direct funded, or if you're a fiscal agent it should be deposited in their account. So remove that con. 

And so really, with all those barriers removed now, I guess the remaining pros and cons would be what do you-- what bang for the buck do you get from your fiscal agent? If you're getting the money timely, if the fiscal agent is not being punitive with asking for too much information. 

Some people like to use the fiscal agent to hold consortia funds. Some people like to use the fiscal agent to pay their regional director or their consortium lead. And so as you can see, there are still 38 fiscal agents versus 33 without a fiscal agent. 

And so I think we have posted on the website, and I don't know if Veronica or Mandilee can find it, but LA consortia did a survey of all the consortia in the state to look at their governance that they use, and that ties into their fiscal agent as well. 

And so there are a variety of governance structures, as well as fiscal structures that tie-in to that governance, and so it varies. And so to answer your question about pros and cons. I think it really depends on your structure and your governance of how your consortia is set up, and if you want to use the fiscal agent to hold money for regional projects. 

So there are some pluses and minuses with some other things. So what I would do is, before you have this discussion about what funding channel you want to go, I would find out how much is the fiscal agent charging you? You can only charge up to 5% administration on your consortia funds. So are they charging you the 5%? Are you getting what you think you bang for the buck for that 5%? 

I know some fiscal agents charge a lot less. Some charge like 1% or 2%, not a lot for their other services. So some people think it brings it together, some people like to get that money right away without going through a fiscal agent. So a lot of variation there. I think if they could find that report, it might be worthwhile looking at it. Although it's still-- it's probably about a year and a half old. It's still relevant to this discussion. 

So sorry I didn't give you a pointer on how to make that decision, but at least you know that there is some variation out there depending on the region and depending on the relationship with their fiscal agent. So let me just check the chat because I see a couple of other posts in there. 

Is there a current document that you can share with our consortia board with current pros and cons? Well, we'll share that report. It doesn't really have pros and cons on the fiscal agent situation. But what you could do is you could look at our program guidance and our fiscal guidance. 

And we lay out what is the consortium level administration supposed to do? What is indirect? What is a fiscal agents supposed to do? Those kind of things. It doesn't really give you the pros and cons, but it lays out the requirements. And I think those pros and cons are going to vary from each consortia depending on who's leading and who's following. 

Some consortia are K-12 strong, some are community college strong, some don't have a fiscal agent, some do. So I'm sorry I wasn't able to give you too strong of an answer there. And then I think there was one other question there. You just spoke of a director sometimes being paid by the fiscal agent. But is my case from a legal standpoint and my employee of my consortia and my employee of the college? I've asked because I feel like I have two masters. Yeah. 

So unless you're JPA-- Joint Powers Authority, which we only have one in the state in West Kern, there is no legal entity called a consortia. So the fiscal agent would be that district. So if your fiscal agent is a community college district, then you would-- and you're the consortia director or lead and you are being paid out of the fiscal agent pot, then technically, you are an employee of that district because that's who's paying you. 

And so you have to wear the two hats. And maybe that's where the governance could be strengthened. The role of the consortia director could be laid out in that governance or by law or charter, whatever you want to call it, however the consortium wants to put it forward. 

And it could lay out, in your case, the responsibilities of the consortium director so you would have some kind of document that you could go back to, the bylaws, which are how you govern your consortia. So if the fiscal agent district starts pushing you, you could go back to those bylaws and say, hey, I am required per my position with the consortia to do these things. So maybe that could help. 

But you are technically an employee of who's paying you. So I'm not sure if that helped, but that's what I would do. OK. So you got a little-- we dive down a little more rabbit hole, but it was a good rabbit hole. There were a lot of carrots at the bottom. So let's go on to the next slide. 

OK. So here's the CAEP structure. If you weren't familiar, Carolyn alluded to [INAUDIBLE]. And so that 400 breaks down. We have 71 consortia, but we have 72 college district. So back in 1516, we had two community college districts combined. And that's down in San Jose. So I think it was West Valley, Evergreen and Mission and I forgot the other one. They all combined to form-- I'm sorry, San Jose City, that's it. Thank you. 

VERONICA PARKER: Neil? 

NEIL KELLY: Yes. 

VERONICA PARKER: Your audio is breaking up. It's going in and out. 

NEIL KELLY: Oh, I'm sorry. 

VERONICA PARKER: OK, when you move closer to the computer, it gets better. 

All right. So I'll lean forward a little bit. Sorry if I'm scaring you with a bigger image of me in the screen, but So those four community college districts combined into-- or the two community college districts combined into one. So instead of having 72 regional consortia, we have 71. 

And as you know or if you didn't know, the Chancellor and the superintendent of instruction decided to use the commutative college district boundaries to form the regions in the state. So there were 72 regions. And then with the combination or consolidation in the San Jose area, now we have 71. 

So of that 71, there's 312 K-12 districts, and 72 community college districts, 52 county offices of education. And then JPAs of which we have one consortium, but we have 10 other JPAs in the form of regional occupational programs around the state that make up our CAEP structure. Next slide, please. 

And then you're going to be going through this today, but our key systems in the CAEP universe. We have Nova that we use for planning and fiscal systems, you'll be getting a demo of that. We also have TOPSpro, which collects all our K-12 and county office student data, and you'll be getting a deep dive on that this afternoon. 

And then we have the Chancellor's Office MIS called COMIS, which includes all the non-credit community college system data. And you'll be getting-- WestEd be covering that. And then those systems, TOPSpro and COMIS, are displayed on the LaunchBoard, which is the adult education dashboard. And I think WestEd is going to be covering that as well. 

And then our technical assistance provider is CAEP-- CAEP TAP, which is under the auspices of Sacramento County, Office of Education. And then JPA would be the Joint Powers Authority. And I think someone answered that, so you'll be getting a little deeper dive on all these systems today. So next slide, please. 

And so here's-- I'm just going to go through, at a high level, some key education code that is for the California Adult Education Program. So education code 84906 covers the planning requirements. And so it says, consortia shall have a consortium-approved three-year adult education plan that addresses a three year fiscal planning cycle. 

The plan shall be updated at least once each year based on available data. So what that means is, that's the three year plan, and the annual plan, and your member work plans. So if you're working on your three year plan, or you just approved your annual plan, or you're starting to submit-- or yeah, it's due, I think, next week, you remember work plan or budget, it all comes out of this education code 84906. So next slide, please. 

And then here are some other key education code. If you're looking for local governance like member requirements or decision making public meetings, that's all part of 84905. And then if you're looking for the seven program areas that are required for-- or restrict the CAEP funding to the seven program areas, those program areas are listed under 84913. 

And then if you also want to look at the funding guarantee of you get no less than the prior year or how you can reduce a member's funding, that's part of 84914. And then if you look at how we leverage funds, there are six funding sources that the legislature requires us to track and also allows members to join a consortium to get these funding sources and so that's listed in 84916. 

And then of course, all the data reporting and outcome metrics are all listed as part of 84920 so we won't do a deep dive but I think we did a prerecorded webinar. And I think it's part of the onboarding tools that we have on the website. So I don't know if Veronica or Mandilee want to post that pre-recorded webinar if it's still available. 

But I went through all the Ed code in that webinar. So if you really want to take a deep dive on the Ed code, you could listen to that pre-recorded webinar. Oh, Joint Powers Authority, sorry about that. OK. And do we have these local governance key education code in one place? 

Yeah, on the website under Administration, under Governance, under Legislation, and I think we'll put it in the chat. We have a link to all the CAEP related education code. And there it is. So I don't know, if-- Veronica, I'll pause there. Did we get any other questions in the chat that we haven't answered? 

VERONICA PARKER: No, it looks like you covered everything. 

NEIL KELLY: OK. 

VERONICA PARKER: The other question was about the presentation slides. So they will be available, Marina, once they are 508 compliant. So they are to come. 

NEIL KELLY: And Veronica, is that education code legislation webinar that we pre-recorded a while ago, is that still available on the website? 

VERONICA PARKER: That is still available on the website. We're actually pulling that one as well as the larrick report. But since I can't-- I'm sharing my screen, so I can't access the website. It's going to take us just a little bit longer to pull those links, but we'll-- 

NEIL KELLY: OK. 

VERONICA PARKER: --have them to you after. 

NEIL KELLY: Great, thank you. Next slide, please. OK. So we're going to get into the deliverables. And we have-- let's see. We have plenty of time. I think we're going to be able to answer questions as well. So for student data reporting, if you're a K-12 County office, your student data report or student data is due through courses through their TOPSpro system in October, January, April and August. 

And so I think it's usually at the end of the month, except for August, it's at the beginning of the month. And then if you're a community college, your data is automatically reported each semester through COMIS. And some people ask, well, why does we have to get reminded for courses reporting? Why isn't it automatically done? 

So if you were in the Cal pad system, like the K-12 districts, that's the state system for K-12 and that automatically gets uploaded. So the Cal pads are similar to COMIS. And so because adult education is not in Cal pads, we have courses to serve as the adult education Cal pads for the K-12 adult education system. 

Unfortunately, it doesn't automatically happen, so you have to report every quarter. And there are some districts that have to be reminded and steps to follow. And so that's why there's a difference in the level of effort, I would say. So if you're a community college, you've got it made because you're part of the state reporting system. 

If you're a K-12 County office, you have to do a little bit more of a heavy lift to working through costs. It's not a bad thing because we get a lot of good data out of the TOPSpro system. But it does require a little level of expertise and that's why you'll see a lot of good TOPSpro coordinators around the state that are vital to getting that data into the system. So any questions on that? I'll pause there. 

You're going to learn a lot more about this today, but I just wanted to share that these are the student data deliverables for CAEP. So next slide, please. And then for planning, your planning deliverables, like I alluded to in the education code. So CAEP annual plans are due in August. And then member work plans and budgets are due in September. 

And then those work plans and budgets are certified by the consortia at the end of October. So you see this logical process. Every three years, we have a three-year-plan, which some of you are working on right now, it'll be due in June of 2022. And then each year, you update that three-year-plan through an annual plan that's due in August 15. 

And then once you have your annual plan certified by August 15 by the consortia, then each member works through, in Nova, to develop their member work plan and their budget based on that annual plan, which is based on the three year plan. And then the consortia approves that by the end of October. 

And all this is done through Nova as part of the planning process. You can see this logical progression that we do on the planning cycle. Goes from big picture three-yearr-plan down to the nitty gritty district level work plan and budget. So with-- oh, thank you for posting that. So next slide, please. 

And then the allocation certification which we already talked about, but as it relates to in the NOVA system, the CAEP allocations are certified by May, that's May 2. If there is a late adjustment, like I said, like trailer bill or budget bill like we have this year, another certification would be due in September. 

And so that was due September 1 to certify that additional amount that you got when we got the budget bill at the end of June surprisingly, but it was welcome because it was a lot more money. So you have to do an additional, what we call allocation Amendment and certification by September 1st. 

And usually funds are, like I said when we ironed out that allocation disbursement issue between the Chancellor's Office and CDE, by statute funds are supposed to be disbursed in August on a monthly basis through June. And so we're hoping that we've streamlined that process so that no later than, I think, this year, it'll be sometime this month. 

I mean, it got disbursed in August, but then it goes to the controllers and then to County offices, and then County offices to either a fiscal agent or the local district, and so all those steps take a little more time. So we're hoping that CAEP/12 districts and community college districts and County offices will see that money in their district accounts sometime this week or no later than next week. 

Next slide, please. And then expenditure reporting once you have your allocations certified and your plans certified, you can start reporting your expenses based on those plans. And this is also through NOVA. So expenditures are due the first of the month at the district level. 

So districts report into NOVA in December, March, June, and September by the 1st, and then the consortia have 30 days to certify those expenditures. And then just a caveat, once the Q4 is certified, no revisions are allowed. So we're coming up on that September 30 date for the Q4 certification. 

So that's a critical certification date because once you certify that, no more changes. If you haven't certified Q4 yet, and Veronica and her team will go over this in the next presentation, but there is a way that you can unpack your previous quarters if you noticed an error in the previous three quarters, and you can go in and correct that before you certify Q4. 

But when do you certify Q4, no going back, and you have to fix that in the next year. So Veronica are you going to be covering kind of Q4 certification and how to go back and fix things before you certify Q4 in the next presentation? 

VERONICA PARKER: Yes. 

NEIL KELLY: OK, great. So next slide, please. So now that we've covered all your state level or state-required deliverables, we sent out a memo or a guidance memo, I think it was maybe end of June, but it went into effect July 1, stating that if a member fails to submit their CAEP deliverables, which we just went over timely, you'll be subject to a reallocation of funds by the consortia. 

And so what that means is the process is we have due dates that we just went over, and if you miss that due date, you'll get a reminder from CAEP tab, Veronica, Mandilee, Holly, and their team will remind you like, hey, you miss this. 

Or if it's student data and you're a CAEP/12 or County office, you will get a reminder from Costas and their team, and they'll say, Oh, you forgot to submit your deliverable. Do you need any help? 

And then you'll get a couple of those reminders, and then if you still haven't submitted your deliverable, the consortia director will be contacted and hopefully, they can work with the district to have those deliverables submitted. 

There will be offers to the consortia director and the district itself of professional development or technical assistance. If all that fails, the reminders, the technical assistance, the connection with the consortia director, and you still haven't submitted your deliverable, or we haven't identified a problem that prevents you from submitting your deliverable, then that will get elevated to the state. 

And then the state will determine if you're an ineffective member, and then they'll notify the consortia. If that district is ineffective, that it's up to the consortia to reallocate funds. So they have the ability to reallocate that funds per Ed Code of that member. 

It's still the state calls them out it's ineffective, but it's still the consortia's decision how much, and whether to reallocate funds for that member. And so that's the critical piece. Now let's say the consortia says, no, I'm not going to do it even though the state finds them ineffective. 

Let's say you have several members, or you have consistently members not submitting their deliverables, then the state will look at that consortia, and the state has recourse. They can maybe combine you with another consortia if they feel that you're not running your consortia effective, or they could talk to the district or offer more technical assistance. 

But there are some recourse if the consortia doesn't take care of their members or reallocate funds to send that message of, you've got to take care of these deliverables. So there I think I saw a couple of questions, and we're kind of at the end here. So I'll open it up to general questions. I don't know the next slide is-- I think it's just a wrap up in questions. So let me look at the chat. 

Please pause for CAEP. [CHUCKLE] We'd love to review it again, and also help ensure that our steering committee would request that every new member-- Oh, thank you. Well, I'm going to open it up. Feel free to ask any kind of questions, make a comment. I love talking about this stuff. We have 10 or 15 minutes, and so you can go off mic. You don't have to put it in the chat. 

I would like to meet-- let's see. We have some members here. If anybody wants to put in the chat, if you're a brand new consortia lead, which consortia are you leading? I've met a few virtually, and I've responded to a few questions from some of you. 

I don't know everyone. Although I know a lot of people in the universe, I don't know everyone. So it'd be good to put the name with the region with the consortia. So I'll stop talking now to see if there's any questions or comments, and I appreciate the feedback there from Victoria and Ryan and company. 

AUDIENCE: Neil, do you mean those directors that are less than a year? 

NEIL KELLY: Yeah, I think well you've been around for more than a year. 

AUDIENCE: Yes, so I'm not going to say anything. I am saying something now, sorry. 

NEIL KELLY: No, no, that's good. No, I think it's important to get a refresher because there's so many moving parts and we have leadership changes, not necessarily all the time at the consortia level, but at the district, the leadership at the district is constantly changing. So you have to re-inform, re-educate, familiarize these new people with the mission of adult education. 

And so sometimes you've worked many years establishing these relationships with districts and you developed articulation agreements, and we'll use transition specialists that are shared, counseling, and all of a sudden then you get a new leader which puts some of those relationships in jeopardy, if they're not familiar with adult education. 

And a lot of times, these people come from a different part of the district, and they aren't familiar with adult education, which is, I guess that's our job to get them with speed. And that's why we have CAEP Tap and all the professional development that we offer through CASAS and WestEd. 

And now we have OTAN as well as CALPRO from the other side. So we usually see about a 30% churn in leadership in our CAEP universe. So we're always having to get up to speed people that are coming in to the system. And I think we're going to see more churning of leadership as people begin to retire. 

We had a big glut of leadership retirement in the last year or so, a lot of long term on the CAEP/12 side, administrators that have been in adult education for years, major talent that have left, and we've got new talent now. So we're hoping to get them up to speed so they can hopefully stay in adult Ed and represent for a long time. 

So we do have some questions here. Our messages. So let's say I'm familiar-- is it Melissa or Mellise? I just want to make sure I get your name right. 

AUDIENCE: It's like a llise at the end. Mellise. 

NEIL KELLY: Oh, Mellise. I've been butchering it all these time. Sorry. 

AUDIENCE: It happens all the time. I think that was part of my parent's plan to keep people on their toes all the way to the end. NEIL KELLY: And then Jeanette. Is it Jeanette or am I pronouncing that correctly? 

AUDIENCE: Yes, that's correct, Neil, thank you. 

NEIL KELLY: So I know you guys but sometimes we don't get to meet, so sometimes I probably put your name. But for the most part, I think I've met most of you. I know Patricia is brand new with Capital Consortia, so welcome. 

AUDIENCE: Thank you. Hello 

NEIL KELLY: And then Greg remain responsive. So great you are with-- I think you put it in earlier, which just-- 

AUDIENCE: I'm with San Luis Coastal Adult School, and I'm brand new to you. You were just talking to me, and I also appreciated the historical summary you did because I am just immersing myself in it all. I am not the director I am considered a WestEd teacher on special assignment. 

So I've been 28 years in the special Ed field, teaching special day class. So this is all brand new to me. So I'm just trying to take it all in. I had a question in there about being responsibilities of a fiscal agent, if you could just kind of hit upon a few of those. 

NEIL KELLY: So fiscal agents, so their responsibility is to pass through that funding to the other districts. Now that's at a minimum, and so sometimes will charge you for that activity of passing through the money. But then if they manage the money, they can manage it at the consortia level. 

Let's say I know some consortia like to have regional projects that like maybe outreach or marketing or maybe a transition specialist or counselor that goes around to the various member districts and provides those seamless transitions and linkages, that sometimes is paid out of the consortium pot. 

And so along with the consortia director, and so it depends on the structure of the consortia of what those fiscal agent responsibilities are. So at a minimum, they're passing the money through to the districts. 

If you add a few levels of complexity on, they could be where the consortia director is housed, they could be taking on regional projects, they could also be taking on paying for other staff at the regional level that are shared by all members. 

Sometimes a fiscal agent can also help with getting those Q4s in and maybe helping the consortia director or lead review that before certification because as you know, at the consortia level, everything has to be certified. So the buck really stops with the consortia director or lead. So the district does the submission, but the consortia does the certification. 

So there has to be some level of review on the planning documents, on the allocation documents, and on the expenditure documents by the director. And usually, we say you're not going to be looking at it as a reimbursable, but you will be looking at did the plan and the expenditures match up? 

So if the member district submitted a work plan and budget, did that match up to those expenditures? And did the data that they're reporting align with what they're spending on, and the plan that they had developed for the year? 

So there is some level of planning and fiscal complexity that the regional director has to go through and should be responsible for because they are certifying. And sometimes the fiscal agent can help out with that, but it just depends on the structure. So I didn't give you the definitive grade, but there are possibilities there can be a minimum level, and there can be all kinds of different variations and complexities-- 

AUDIENCE: For sure, it's very helpful. I'm on your CAEP website right now, and I'm looking at there's-- a Q&A for the fiscal agent. So with your info and with this supplemental, that's very helpful. I appreciate it. 

NEIL KELLY: Oh, great. I'm glad you could find it because sometimes you can get lost on that CAEP website. There's so much material on there. 

AUDIENCE: Yeah, Neil, could I speak to that just as another model for Greg? 

NEIL KELLY: Yes. 

AUDIENCE: In Contra Costa County, we're a large county, so we have seven adult schools, we have three colleges and CBOs. And we had a fiscal agent, which was our County office of Ed who's a member because they also directly serve students. But we came to the conclusion when we started this was a heavy lift, figuring out all this structure building, and we spend a lot of time over the years doing that. 

So they had the fiscal keys not only to forward the funding, but manage it. We had at that time, which we don't have, a common cost pot which we used because funding, we couldn't kind of agree on how it should be used. 

But over time that changed and we are now member-funded directly, but we maintained the accountant at the County office of Ed because they were so good at keeping us on track with the CAEP and NOVA pieces for the deliverables. So that person at a much reduced amount, we were paying like $150,000, I think we're paying like $35,000 because she doesn't just work for us she works for the County office of Ed. 

And she connects with our budget and fiscal work group, and we are the ones who oversee the quality of what we are doing, and she helps to coordinate it. So that was just another kind of evolutionary process, but it's worked really, really well. We do have a consultant we hired ultimately for helping to facilitate and run other things, special summits, et cetera. But the members actually do the work, but with the help of these experts like an accountant, who can do certain things for us. 

NEIL KELLY: So Victoria, under your model the different districts pay. So you give money to the County office to pay for that accountant, and then you probably have another district that pays that consultant. So you kind of partial out who's responsible for what, and that brings you all together and the planning, and so that's why you probably don't need that fiscal agent any anymore because you're all working well together. 

AUDIENCE: Exactly. But it took us a few years to get there, and that was sort of the easiest thing which is why some of us weren't favorable towards that, frankly, at the beginning, but when you're starting out, it was the easiest way. And so it is an evolutionary process, the whole structure building you do. 

NEIL KELLY: And just like at the state level, we've evolved too. 

AUDIENCE: Yes. Sure. 

NEIL KELLY: Thank you so much, Victoria, for sharing. There's one more question I'll handle before-- I think we've got about four minutes, so Janice, is there a place to find County-specific data points showing lax or lack gaps in available teachers for program classes. This could be a barrier that many consortia faced in the three-year planning process. 

So the only place I know is where we teach captured teacher data, and maybe the other people on the line might know this, but for courses for the WIOA grant, we're required to, or the CAEP/12 districts and community colleges and County office, the grantees are required to report how many teachers they have. 

Now that doesn't answer your question about the gaps, but at least there is some information provided into TOPSpro of how many teachers are employed in the various programs. So that only covers the WIOA program. So that wouldn't include any non-WIOA of programs like CTE, those with disabilities, things like that. 

But I don't know if anybody else has any suggestions. There might be district level data that you could grab on to, but I know we do collect that in the TOPSpro system. Anybody else have anything to offer? Jennie. All right. Well, I guess there's no other magic bullets for you on your teacher gaps. So if anybody else has anything, we'll be sure to circle back with you. 

Kim's been around for a while, but as consorcia coordinator for four years, so welcome. So Kim says we get information from our own research department, and then from local economic data from the city, as well as the resources CAEP gives us, so thank you for sharing that. 

All right. I'll turn it back to Veronica with closing remarks, and getting you ready for the next presentation. I appreciate you all being on the call today and lively discussion and questions. So we'll continue our adulthood journey. Thank you. Oh, I think we got another comment in there from-- Victoria posted some stuff. 

She said please know that our professional organizations, CCAE and KAIA have been and continue to work with Carolyn and Neil and consortias together. So sometimes you can always rely on your professional organizations like KAIA, like CCAE if you're a community college ace, they might be able to get some of that data for you because they're working with their teachers and administrators on a regular basis. 

So Jannie don't forget to use those resources if you get a chance. They also have vital networking and ability to get the word out and gather information. So with that, I'll turn it back to Veronica. Thank you Gary, thank you, Caroline, and thank you, Tab for the presentation today and support. So look forward to you guys having a great day. Thank you. 

VERONICA PARKER: All right. Thank you, Neil, Gary, and Carolyn. She has left. So we will transition to our next session, which will be on the SAP Technical Assistance Project as well as a live demonstration in NOVA. So I will share my screen, and I have just a few slides that I will present to you all, and then we will go into the NOVA presentation. So let me find my own screen. All right. Here we are. 

So my name is Veronica Parker, for those of you who do not know, and I'm the Coordinator with the CAEP Technical Assistance Project, and we have a small but yet mighty team. Our director, Renee Collins, she is here with us today, as well as Mandalit Gonzalez, and then we also have Holly Clarke, who is not with us today, but she helps us with all that we do. 

So just going into this presentation. So the second slide just talks about our productivity and our projects. So we were established in 2017 as a program or project to be able to support the CAEP office as well as CAEP consortia across the state, all 71 consortia, and we provide technical assistance and professional development in the areas of CAEP. 

Some of the things that we have done is we host the annual CAEP summit, so that is coming up this October 26 through October 28. So if you have not registered, definitely take the opportunity to register. It's a great opportunity to continue to learn from your colleagues. 

This year we have 60 breakout sessions scheduled that are aligned with the CAEP state priorities, which are equity, learner transition, leadership, marketing, development, program evaluation, and technology and distance learning. 

And so there are a number of great presenters in presentations from your colleagues in adult education who will be talking about the practices and strategies they use within their agency, as well as their consortium. 

And then we also are working with a team of consortium leads, both newer as well as veteran consortium leads, to develop a consortium leadership academy, and that will be launched in the spring of next year. And so that leadership academy will be providing a number of modules on consortium management. 

So the topics include the building blocks of CAEP, to consensus building, to relationship building, managing CAEP deliverables, having hard conversations, fiscal responsibility, a number of topics that will be covered during that leadership academy. So more information to come on that. 

We definitely encourage all of you who are joining us today, especially if you are a newer consortium lead to participate in this leadership academy, as it will be of great value to you. We're working not only with advisory team, but we're also working with a facilitation team who will be designing our curriculum, as well as facilitating the individual modules within the consortium leadership academy. 

Another program or opportunity we provide is targeted technical assistance, where we work with previous consortium leadership or directors, as well as current agency-- excuse me, adult education practitioners, and they coach consortia on becoming more effective consortia by working through data issues, consortium management issues, developing a program improvement plan and action plan, and then providing a host of professional development to meet your goals and objectives. 

So that's something that we launched in spring of this year. We're currently working with five consortia who were identified by the state as needing just a little bit more independent support, and so they have been making strides in meeting their objectives. And then another opportunity that we are currently developing and we'll launch actually next week is our Peer Learning Circle's Facilitation Training. 

And so we are working with Jennie Mollica and Peter Simon of High Road Alliance, and they are going to be providing a facilitation training to adult education practitioners across the state in order to build your capacity to lead peer learning circles around targeted questions that have come up either at the regional level, the local level, even the state level, in working through conversations with adult Ed practitioners in order to find resolution to those particular questions. 

And so we have sent out an interest form. Up until this point there are, I want to say close to 50 who are interested and meet the commitment requirements. And then we're going to be opening up a second date, so the first date of September 30 we'll be opening up a second date on October 1. 

So we will be hosting two facilitation training, and more communication regarding that second date and the opportunity to join will be coming out later on today. And so this is just some of the things that we do. We also hosted a directors event, so that's an annual convening that we host every year and including the new consortia on board training. 

And we host other workshops for the CAEP office as well as for CAEP contractors, so CASAS in WestEnd and then we partner with OTAN and CALPRO or AIR to provide other professional development opportunities. So that's just a little bit about what the Technical Assistance Project does. 

And I'm going to be moving through these slide so we can get to the live demonstration. But of course, our project partners, we work with CASAS, OTAN, AIR, and WestEd to provide professional development and technical assistance. 

So if you haven't taken advantage of their professional development opportunities, you can visit the California Adult Education training page, and you can see all of the offerings that they are currently offering and be able to sign up to participate in those courses and training. 

And then we recently launched the advancing California Adult Education website, and this website is a relaunch of the practices with Promise website that was out a couple of years ago. And so it's an opportunity to have a repository of model programs that consortia as well as agencies could visit to learn about other programs that are currently going on in the state that have been deemed model programs. 

And so the model programs are organized by the state priorities, so as I mentioned, equity, learning, transition, leadership, program development, et cetera and so you can filter and learn about particular programs. So they state their problem and practice, how they responded to the problem or practice, the unique features of their program, any outcomes that they may have, as well as an overview of the model program. 

And with this new site, we also implemented the research and practice page, as well as the connect page, and those two pages provide research and current practices and an opportunity to connect with either the authors or the organization of those research and practices. 

And then the connect page is an opportunity for practitioners to connect with one another, and on a particular topic or area of focus. And so on that page, again categorized by the state priorities, you're able to see practitioners across the state, their area of focus, and if that aligns with your area of focus, then you're able to connect with them. So if you have not visited this website, we definitely welcome you to visit the website and learn more about the different model programs across the state. 

We just recently completed a nomination process to be recognized at the CAEP summit, and so there are two model programs where you will be able to see a highlight video of their actual program, which you'll be able to see all of the CAEP model programs that were submitted and learn more about them. And so we'll have an exhibitor booth specifically for the model programs that were submitted during this last submission cycle. 

And then we've been talking about the CAEP website, and so the CAEP website, as Neil said, there are a lot of great resources and tools on this website. The training this month page is really important, if you're looking to find out information about what's kind of important or on the horizon with CAEP. 

So for example, we just posted the CAEP fact sheets on there, we produced a CAEP Summit 2021 publication in partnership with News and Review where News and Review interviewed programs across the state that align to the CAEP state priorities. And so this publication is a great publication to learn more about those model programs, c and about the CAEP state priorities. 

And then we also have a page on-- the last page talks about superintendents of CAEP/12 and community college districts, and how they support adult education as well as the needs of adult education. So if you haven't checked out that publication, we definitely invite you to do so, and we are highlighting stories from that publication each week in the newsletter. So you'll be able to individually see each of the stories that were included in that publication. 

And then just more about the website, so I talked about training this month, the quick links as well as peer to peer networking and publicity. We also have a find a school feature that is utilized. It includes all of the adult education agencies across the state that are a part of CAEP, and so you can use the search feature and search by zip code or address to find schools in the area. 

We definitely let consortia leads as well as member representatives to continuously check this to make sure that all the information that we have remains current and relevant, because we send sometimes the CAEP Technical Assistance Project. 

We receive emails from students, for example, and they're looking for adult education programs within a program area and within their region. And so we refer them to this find a school locator, and someone makes sure that all the information here is current and relevant. 

We have a translator on here so we use Google Translate, so you can select the language and the entire website, not including actual documents, but the text on the website will translate into the particular language you select. 

And then also other features, so the administrator page-- excuse me, this is the educator page. So we highlight as I mentioned before the California Adult Education Professional development page. That is a one stop repository of all of the professional development offered by the California Adult Education leadership projects, so CASAS, often WestEd, as well as CALPRO/AIR. 

So again, visit this page if you want to learn more about their professional development opportunities. And then we archive webinars by audience type. We have other resources on here, such as the Practice for Promise or the new model programs, professional development and program areas. And again, the leadership projects, course outlines, and standards are on the educator page. 

And then the administrator page, which is definitely relevant to you all as it talks about a lot of the CAEP guidance, so on this page, you can find the CAEP newsletters. So the newsletters come out every Wednesday around noon. If you are not currently subscribed, we will drop the link in the chat so that you can subscribe to the newsletter. 

The newsletter contains a lot of great up to date information, things that you need to as a consortium lead. So if you are not subscribed to that, definitely subscribe. We post all of the archives of the newsletter on this page, so you'll be able to find if you're looking for something in particular. You can search on this page, and the newsletter will pull up, and you'll be able to review the archive newsletter. 

And then we post all of the guidance documents here, so whether it has to do with data reporting, fiscal guidance, program guidance, all of the guidance documents that govern the CAEP are on this section of the website. So if you haven't looked at, for example, the fiscal management guide, we just recently updated that this year, so we definitely invite you to read that through and through because it contains a wealth of information about CAEP fiscal management. 

And then on this page we also archive all of the webinars that are hosted, and we categorize them by type. Some of the webinars that will be most useful to you all, of course, as we're going through the three-year planning cycle. Those webinars, fiscal data reporting webinars, so whenever we have a webinar on NOVA or program area reporting, any of the other CAEP fiscal deliverables, those webinars are located there. 

And then student data reporting webinars as well as policy related webinars will be of great information for you. So if you want some more historical content, please view those past webinars as they will provide those pieces for you. 

And then for technical assistance, so we have communicated with pretty much all of you who are on the line to some capacity, but if you ever need technical assistance, either myself, Mandilee, or Holly are available to assist you with whatever it is that you need. If it's policy-related, we do consult with the CAEP Office of the Chancellor's Office as well as the CDE on those policy related issues. 

So we are the first stop for communication and then if we have to seek further advice or answers from the CAEP office, we will do the work for you. And then also, if we need to contact let's say OTAN for something or CASAS or AIR, we are kind of that intermediary in terms of communication. But we also provide a lot of direct support, especially as it relates to the mobile fiscal system. 

So if you're ever in need of any technical assistance you would use the button up top on the website to request support. If you are a user on the website, you will simply log in, and then submit your support request. 

If you are not currently a registered user on the website, I would advise you to create an account and become a user because that'll be your way of having that historical context or content for support requests that you have already submitted. And if you need to reference it for any reason, you will have that information there. And there are other benefits to registering on the site, so we welcome you to do that as well. 

So that was definitely a very quick overview of our website, as well as the services that we provide. I'm going to check the chat as I see there are quite a few messages in here, and see if there's anything that is relevant to me that I need to answer. 

MANDILEE: There are no questions for you. But there are accolades from Shannon, letting us know that we've helped her many times. 

VERONICA PARKER: OK. Great. Thank you, yes, Shannon, we are definitely here to support you all in any way we can. So I'm going to go to the NOVA fiscal system, and we'll begin the live demonstration there. And again, if there are any questions, definitely feel free to type them in the chat and I will get to them as soon as I can. So let me log back in. 

So I am on the protected site or the safe environment. So anything that I do or touch here will not impact any of our consortia information. So it is a safe environment. Mandilee has posted the URL for NOVA. By show of hands, are by using the Raise hand feature, does everyone who's on the line today have access to NOVA? 

Either you should have access as a primary contact because you are the consortium lead, but you may have a certifying authority who have that permission. So I'm seeing some show of hands. Not everyone, but I hope everyone on this call has access to NOVA. So quite a few of you. If you do not have access to NOVA, c you can contact us at the CAEP-- excuse me, I have a horrible cough. Please excuse me. 

You can contact us at the Technical Assistance Project or you can contact whomever is listed as the primary contact within your consortium, and they can invite you to become a member in NOVA. But like I said, all of you should have access to the NOVA system as a primary contact since you are the consortium lead. 

So Malisse, I know that you are with us today, so if you don't mind, I'm going to use your consortium just for demonstration purposes. Again, I'm in the safe environment, so I will not make any alterations to your consortium information that's going to hinder you by any means. 

So I've logged in. I have access the CAEP program by going to All Programs, clicking on CAEP, and then I click on Consortia & Members. And when I clicked on Consortia & Members, I was taken to a list of all of the consortium members, and I just simply search for the consortium in which I wanted to pull up. So I pulled up Butte Glenn Adult Education Consortium. 

Now here consortium details, it is just simple consortium information, so the consortium name, the short name, the address, their website, their funding channels. So they're currently fiscal agent, c and then their CAEP funds for 2021 through 2022, down through 2019 to '20. So these are their total CAEP allocations for the program year. And then they're pretty logo is here as well. 

And so scrolling down consortium contacts, so this is actually a very good example of what the consortium contacts could and probably should look like, although we understand that there are other consortium contacts that may be included, for example, the director of business services or the budget analyst who works with the primary contact on fiscal related matters. But the primary contact, Mellise is listed here. 

She is the certifying authority for the consortium, and so she's listed first on this page. Some consortium may use the consortium contacts as kind of a one-stop kind of catch-all contacts list. We advise you not to do that, but only keep the consortium contacts who are at the consortium level and need to be in the consortium at the consortium level. 

Anyone else, especially member representatives, we want them to be aligned to their agency and not necessarily included in the consortium contacts, as at the consortium level, they may not have the certifications needed to certify the reports. Through I see your hand is raised do you have a question? 

AUDIENCE: No, it was from when you ask the question. I apologize. I'll turn that off. 

VERONICA PARKER: OK. No worries. So moving forward, the next section is Consortium Fiscal Administration Declarations. So as Neil covered in his portion, the CFAD is due annually by May 2, and so that's where consortia vote on their member allocations for the year based on the total consortium allocation. 

So I'm going to click on construction fiscal administration declaration for the fund year 2021 through 2022. And it has been consortium approved. So the consortium met that May 2 deadline. And then if you ever want to see any of the past effects, they are listed here. You can click on View PDF or click on the link, and you can look at all of that historical information about how consortia divided their allocations in prior years. 

So I'm going to click on those CFAD that here. And again, this is going to be a very high-level overview since we are on a time schedule. However, I just want to pause and let you all know after the session if you need further training, or you want to see other areas with the NOVA, we will be happy to schedule one-on-one with you all to walk you through the system. 

So back to the CFAD. So again, this was consortium-approved. Their funding channel is fiscal agent. So Neil went into the particulars about the funding channel, fiscal agent versus direct funded. And just to reiterate what he said, this could be done on an annual basis. 

So the consortium decides to vote that they want to change let's say to direct funded before they submit this effort on May 2. That's their opportunity to make that change for the upcoming program year, and that's the only time that you're able to make that change. So just keep that in mind as you are approaching the CFAD cycle next year. 

They provide a narrative about how they plan to use the funding, and then the changes that they made, so you're able to highlight those pieces of information when you complete the CFAD. Then we move into Member Allocations. Now here, things get a little tricky, especially as we go from program year to program year. 

So when consortia, let's say, come next spring, when they come to look at their CFADs, they're going to see these red, the total remaining numbers. And some consortia think that they have not done something because they see these red total remaining numbers. 

These red total remaining numbers is because we only complete the CFAD once a year. And once we submit it and approved by the consortium, we are unable to make any changes. And so these total remains just highlight the changes that were made to the total allocation for the consortium after that May 2 deadline to submit the CFAD. 

So as Neil had talked about, at the end of June, we received an increase in the total CAEP funding, which meant an increase for each consortium. And so that's highlighted here. So the change was an increase of $57,077 for this particular consortium. 

Now in prior years, as the governor signs the budget, there was a reduction in funding by a certain amount. And so that's why you see this reduced number here. Now again, this is just to highlight the changes that occurred after this was submitted. However, everything that you all needed to do as a consortium has been completed. 

So when you come next spring and you see these red highlighted total remaining amounts, it doesn't mean that you didn't do anything, it just means that there were changes in the total allocation after the governor signed budget. And you all took care of these changes whether it was additions or subtractions to the total allocation. It was taken care of in the Allocation Amendment section, and I'll briefly go over that in just a moment. 

So here are the member agencies for this particular consortium, and annually you are able to add members or even remove members depending on if a member decides to leave the consortium. One thing to note here is that there are some members who are voting members of the consortium but they are not funded members by the consortium, but they're still a member. 

And so here is where you designate if the member is funded or not. So if the member is not funded, there's a mark that you're able to check, and it will indicate that this member is a non-funded voting member only. And so this particular member has a responsibility to approve certain CAEP deliverables. 

However, they do not receive an allocation so there are certain CAEP deliverables such has a budget and work plan and fiscal reports. This particular member do not have to complete those deliverables because they do not receive any CAEP funding. So that's just something to note as you all are evaluating your members in the next year. 

And then Certification and Assurances. So here are all of the assurances and certifications that each member within the consortium has to abide by. Neil talked about the member effectiveness process. And so when the CAEP office goes to evaluate whether a member is effective or not, they're going to be looking at these certifications or these assurances to see if a particular member has met the assurances. 

So definitely, make sure that you become very familiar with these assurances because again, it's one way that the CAEP office-- they make sure that you adhere to the assurances of being a member of the CAEP program. So these are all the assurances. 

And then underneath here are all of the member representatives who are part of the consortium, and then it highlights which member approved the CFAD and then it gives a date and time stamp of when the CFAD was approved. So you'll have this information here as well. So that's the CFAD in a nutshell. And I'm not seeing any questions, so I am going to move forward. 

You go on to the next section. The next section is our Allocations. So this section is where a consortium will be able to complete an allocation amendment. Allocation amendments, you are able to complete at any point throughout the program year, as long as the consortium has voted to complete this allocation amendment, whether it be reduced in a member, adding funding to a member, et cetera. 

And allocation amendments do not impact the baseline funding. So for example, let's say Oroville Union High decides that they would like to give moneys to, let's say, Butte Glen Planned Community College District for a special consortium project. The impact of that allocation amendment will not impact or bill Union High's base funding within the next program year so. You're able to move monies around, and it will not impact your base funding if you do it within this Allocation Amendment section. 

Another way we use the Allocation Amendment section is as I talked about the changes in funding, so whenever there are changes in funding past that May 2 CFAD deadline, those changes will be made here. The consortium receive an additional 57,000 and some change, and so they came here and they completed an allocation amendment to account for that increase, and they spread the increase across the members within the consortium. 

And so we can look at their amendment history and hopefully, we'll be able to see what they did with that 57,000. Now please note that this safe environment is not always up to date with the real environment, and so we may or may not see those changes, but I think we can because they completed an allocation amendment on August 10, 2021, and that was shortly after we had that increase. 

So I think this allocation amendment that they did was based on that, and it looks like based on the amounts, that it was an increase to each of the members, and without using a calculator, I will say that this is close to $57,000. So as you can see here, the amended allocations, so they increase each member's allocation by a certain amount. 

So you can see these increases here, and then their total allocated to members is a little more than $2 million, and our total CAEP fund matches, and they have a total remaining of zero. So they completed that exercise to account for the increase in total allocation. And then you can see all of their member agencies who are listed, certifications and assurances are always listed, and then you can see the approvals. 

So each of the members who approved this allocation amendment, and when they approved it. Now there are instances where a consortium completes an allocation amendment and they're consortium members, not all of them approve, they may reject it. 

And when they do reject it, the consortium primary contact will have to come back in and either cancel the allocation amendment or start new, and they'll have to change the amounts at least by $1 in order to resubmit. So that is the allocation Amendment in a nutshell. 

I'm going to click Start just to quickly go over what the process could look like. So it's going to ask you-- it's going to tell you that you'll have to complete the workflow steps, and we'll submit it. The change will be made once they are approved by the members. I'm going to press Continue. It's going to take me to all where I can see all of my members. 

Now you can't make any changes to the members in this window, but you can see all of the members. And if you need to add a new member agency, be able to do so here. So I'm going to select Next, and then here's where you make all of the adjustments. 

You'll add or subtract each member's allocation, and then the adjustments will be shown here. And then at the end in order to submit this, the total allocated to members in the total CAEP funds have to match one another, and your total remaining must be zero, in order for you to move forward with this process. 

So I'm not going to have any changes, and so it's going to take me to a preview window. So if I had made changes, those adjustments will be made here. So you'll be able to see them like I showed you in that PDF, and then once you have made all of the adjustments, you would press Submit, and then they'll say changes to agencies and member allocations must be made because I didn't make any changes. So I'm just going to cancel this it was just for demonstration purposes. 

And then we're going to go back to the consortium. Scroll down. Again, just like in other sections, if you want to view amendment history, that information is provided to you here. So you can see all of the amendments that have been made by the consortium, and when they were completed. So the date and time is there in which they were completed. 

The next section is the Annual Plan. Now the annual plan is based on the current three-year plan. So the current three year plan, it was completed in 2019, and it goes through 2022 when you all submit a new three-year plan. And so this is, in theory, the third year of that three-year plan. 

And so here, the consortium provides an executive summary about their plans for the year, and then they have a regional planning overview, and then they identify the different strategies that they are going to work towards within this program year, and they are categorized by the objectives that the CAEP office identifies. 

So Meeting Regional Needs. And so they identify the regional needs, they identify the gaps in service or the regional needs. How do they know that this is a regional need? What resources they use to identify these gaps? And then they talk about how they will measure effectiveness or progress towards meeting that particular need. 

And so they have identified several regional needs. I think there are four that they have identified. And then we'll talk about gaps in service, that's another objective that they are supposed to address. 

So they talk about the strategies that they are going to use to address the gaps and service, seamless transition, they identified the strategies that will adjust seamless transitions, student acceleration, professional development, fiscal management, and then they certified this particular annual plan. 

And so here you will see all of the members who have certified this annual plan and when they or-- excuse me, approved. When they approved the annual plan. And this is a deliverable, just like the allocation amendment. Those are deliverables that even your non-funded members will have to approve in order for them to take effect. 

Now at any point throughout the year, if the consortium wants to amend their annual plan, they are able to do so by clicking on this Amend button, and then they will amend the annual plan, resubmit it, and then all members of the consortium, even non-voting members will have to re-approve their annual plan. So that's the annual plan in a nutshell. 

And the annual plan, just like the allocation amendments, are completed by the primary contact. So Mellise is the only one who can complete those two particular deliverables, and then her members within the consortium approves those deliverables allocation amendments as well as the annual plan. 

And again you can see prior year annual plans by clicking on either PDF or clicking on the link, and then you can see when they were completed as well. So if you're ever looking for, especially as a new consortium week, if you're ever looking at the historical context within the consortium and you want to learn more about what was done in prior years, all of that information is actually at your fingertips within the NOVA system. 

The next section is the Supporting Documents section. Now the supporting documents section serves two purposes. It is an export of prior year fiscal reports. So as you can see, quarterly fiscal reports were exported to the section, and then you also have-- this particular consortium does not have any per se, but you have the opportunity to add documents. 

So some consortia use this section. For example, if they have made, let's say, a large investment in capital outlay, and so they've had that investment approved or informed by the CAEP office, and they want to upload, let's say for example, their minutes to show that the consortium approved this particular investment, consortia have the opportunity to add that particular document here. 

Some consortium for their annual plan, of course, we provide that template in the section prior to where you input just simple text in there, but some consortia like to produce well-formatted documents with charts and graphs, et cetera. Since you can't include those that type of formatting in the annual plan template that's provided in NOVA, you could add your formatted document here in this section. And that will be a part of the historical content for the consortium. 

So those are the ways in which we use this particular section. But again, all of the fiscal reports after the program year has passed is added to this section. So you can see that information here too. 

Then we have this Three Year Plan section. This was prior to the new three year plan that you all will be working on and submit in June of 2022. But prior to this particular three year plan, three year plans had been uploaded to the system. And so just by adding the document here, it was uploaded to this section within NOVA. So if you ever want to see prior year three year plans, they are listed here. 

But moving forward for next year's three year plan, there will be a separate section where you all enter the information directly into the NOVA system. However, if you have that formatted document that's really comprehensive that you would like to add, you're able to do so here, but you'll also have to complete the template within NOVA. 

Then here are Member Agencies. So again, these are all of the voting members of the consortium. And when you click on any of these sections, you'll be able to see the member representative-- excuse me, the member agency details. 

So the member agency itself, their annual allocation, minus carryover their website home the member representative is, their budget and work plans, as well as prior year budgeting work plans. And then you can also see supporting documents that this particular agency have added to NOVA. 

We're going to go back to the consortium, and go through the last sections on the consortium details page, which is the budget and workplan. So here, you see two budgeting work plans for two program years. You see the 2020-21 budget and work plan, and then the 2021-2022 budget and work plan. And then you see all of the information that is included in this particular budget and workplan. 

Now one thing to note that because the 2020-21 program year has not been closed out the 2021-22 program year available funds are inflated. They are inflated because the system has not calculated the carryover from prior years, and so what we advise is close out the prior year, 2020-21 before you complete the budget and work plan for '21-'22. 

Some consortium members they see these inflated numbers, and think that there is something going on with the system like a system glitch. It's not a system glitch. It's just that the system has not calculated or reduced your available funds for the next program year by your carryover, or it hasn't reduced it down to just your carryover in the current year allocation. 

So complete 2020-21 first, and then complete the '21-'22 budget workplan. So again I'm not going to look at the 2021-'22 budget and workplan, but we're going to go to the 2020-'21 budget and workplan to look at some of the things that were happening here. 

So I'm going to click on the Butte County Office of Education, and look at their budget and work plan. It is currently certified, so the primary contact, so you all as primary contacts, you all have the opportunity to uncertify the budget and work plan at any point during a program year if your member has to make changes or make revisions to their budget and work plan. 

One thing that Neil talked about is making those revisions prior to the certification of quarter 4 because once you certify that quarter 4 expenditure report or fiscal report, you are no longer able to assess your budget and work plan to make changes, and you are no longer able to access the fiscal report to make changes. And so we advise consortia to make those changes prior to that certification, so that you do not have to do any reconciliation in the next program year. 

So I want to uncertify just so we can open it up and be able to view some of the details here. And we're going to start with the work plan. So the work plan is a direct input from the annual plan. So all of the strategies by objective are automatically included in each member's workplan. Now it's up to the member to check which of the strategies they plan to address. 

Members do not have to address all strategies that the consortium has approved, but identify the strategies in which they will work towards throughout the program year, and ensure that their budget aligns with these strategies. So here are all of the strategies that the Butte County Office of Education identify. And so as you can see, they will not be working towards each of the strategies, but most of them. 

So once you check all of the strategies that you will work on, you will select Next, and then you'll start working on your budget. So your total available funds in your budget are your total annual allocation for the program year. So this is 2020-21. So this number includes their annual allocation as well as carryover from prior years. So that's calculated here. 

So this number in most times will be different from the annual allocation that you'll see on the member agency details page. Just keep that in mind that this number includes carryover from prior years. Then you have your total budget. And your budget remaining should be zero once you're done completing your budget and workplan. 

And then over here, you have your indirect costs or your direct costs. So your direct costs will be all of your costs minus your indirect costs. This particular agency did not take any indirect, so that is zero, so their direct costs is all of their available funds. Now, this consortium fiscal admin expenses, basically they're saying that each of their budgeted items by object code are related to conservation fiscal/admin expenses. 

So if we scroll down, we'll be able to look at their budget by object code. So their first object code is 1,000 Instructional Salaries, and so this is the budget item title they identified as certificated, a brief description includes salary for teachers, the budgeted amount is $80,000, so $80,000 of their $230,000, and then they check that this is the consortium fiscal admin expense. 

And then moving down, the Cumulative Quarterly Expenditure Forecasts. So this is a forecast that you use to forecast the year to date spending by entering cumulative percentages for each quarter. Now unfortunately, this member did not complete this exercise correctly because if they had, if they are truly going to be spending 25% each year, at the end, their total amount of 80,000 will be expended by the end of the year. 

Now we give an example. Because this is a cumulative forecast, you start out with let's say 15% for quarter 1. Again, this is just an example. 15% for quarter 1. And since it's cumulative, in quarter 2, the member would have expended 30%, and then in quarter 3, 45%. 

So as you can see, it's continuously adding on or it's cumulative, and then at the end of quarter 4, using this example, the member would have expended 60% of this $80,000 and then they would be carrying over 40% at the end of the program year using this example. 

Now this member, they probably were attempting to do that by saying each quarter they would have expended 25% of that $80,000. And how they should have identified this is they should have started with 25% in quarter 1, but in quarter 2, this should have been 50%, this should have been 75%, and then their total should have been 100% since this is cumulative. 

So as you can see, they started out with spending $20,000, in quarter 2 $40,000, quarter 3 $60,000, quarter 4 $80,000 to have expended the entire amount in this object code. But again without having spoken to this particular agency, I don't know if that's what they were attempting to do, but if I'm thinking correctly, I think that's what they were attempting to do. 

So we're still working on that with all consortia and all members, and hopefully we'll get to a point where we're actually using the cumulative system as it was designed. And then again, you would just do the same process for your next object code, which they have 2000 which is non-instructional salaries. 

And then again, if they are completing this exercise the way I think they are, we would accumulate each of these percentages, and then at the end of quarter 4, they would have expended 100%. So we're just going to go next here. Again, at the end of this exercise or completing your budget your total remaining should be zero, and your total budget should be equal to your total available funds. 

We're going to go next here, then you're able to preview all of the information that was presented here. I have made some changes to their budget in terms of that exercise, but it's not going to impact because we are in the safe environment. So I'm going to go ahead and press Submit. 

And when I press Submit, I have the opportunity to enter any comments that I would like to share with the consortium lead, who after you submit or after you press OK, it has now been transferred to the consortium leader or the consortium director, and they review this information, and they either certify your budget workplan or they reject it. 

And if they reject it, then you have to go back and make some changes, et cetera, and then resubmit. But here is what the budget looks like. And so I'm going to certify just for practice. And that is the budget plan process in a very quick nutshell. So I see that there is something in the chat, so I'm going to check. 

This year's budget only has one cumulative target section for the total budget. So it's all, I think so. I have not studied that just yet, but I think that's what's happening with the programming this year, is that it's one cumulative section. But since that is new, I'm thinking that we want to have a webinar on that so everyone understands that. But thank you for that, Shannon. 

All right. I'm not seeing any additional questions. So we're going to move forward with fiscal reports, and then I will also quickly cover the dashboards that will support you all in monitoring and managing your consortium. So I'm going to go over to fiscal reports, and we are looking at view plan. So I'm going to take a peek at their fiscal reports. 

And so I just clicked on fiscal reports and I'm typing in here, and then when I press Search, their fiscal contortions should appear. So I'm going to click on their fiscal reports, certification of the Q4 fiscal reports is due September, 30 so there is still time. I'm in the safe environment. 

So in the live site or on the live site they may have already submitted and certified their expenditure reports, but here some members have not yet submitted. But again, it's a safe environment that is not always up to date with the actual NOVA system. So you're each of the members, and you can see the certification authority is Mellise as the primary contact and the director of the consortium. 

And then here's a little note that says all member agencies must submit their fiscal reports for the quarter before they can be considered for certification. So since not all of the Q4 expenditure reports or fiscal reports have been submitted, we don't see that certification button here. So I'm going to go down to Butte County Office of Ed since they are the first ones listed. 

Now the fiscal reports are also cumulative, starting with Q1 and they accumulate through Q4, and so you can see, for example instructional salaries, $20,000 was spent in Q1, then if I go to Q2, it looks like it's $20,000, maybe they had not expended all of their instructional salaries, then in Q3 again still 20%. So they have only spent 20% of their $80,000-- excuse me, $20,000 of their 80,000 for the year. 

So they still have 60,000 remaining for the program year. And then we're at quarter 4. So at quarter 4, if there are additional money spent for instructional salaries though, that amount will be added to this 20%. Now the NOVA system or the expenditure report is on a cumulative basis. However, the system does not automatically calculate your totals. 

So you would have to add the current quarter's expenditure amount for the prior quarters expenditure amount in order to create that cumulative system. So keep that in mind. 

When you're entering your expenditures, the system will not automatically accumulate for you. And also here-- what was I going to say? Oh, the next column is-- Oh, one thing I want to say, it just came back to me, before you could submit this, if for any reason you are reporting under what was reported in the last quarter, you can do one of two things. 

One, you may not be accumulating your total, so that could be why you will receive an error message. The second thing is for some agencies, their expenses in let's say quarter 4 would somehow became less than what was reported in quarter 3. Maybe they received a refund on an expenditure, or they had a purchase order and didn't expend all of it, et cetera. So their expenses become less than what was reported in the prior quarter. 

If that happens, the consortium primary contact will have to open up your prior quarter fiscal report, make those changes, and then come back to quarter four so that your quarter four expenditures are greater than what was reported in the prior quarter. This system will not allow you to move forward unless that happens. 

Another thing, if for some reason you under report or over report for whatever reason, we have heard of all types of scenarios that have happened, you want to make sure that you do that before the quarter 4 certification because, again, once we certify quarter 4, we are not able to go back and undo anything that has been done. You will have to reconcile in quarter one of the next program year. 

And to create a clean system of accounting, we want to make sure that we take care of all of those things in the current program year. So the next column, after year to date expenditures is the year to date percent of a project budget. 

So those are those percentages that you entered in for your expected forecast, and so here because I changed what was inputted before, it now says by quarter 4 they will have expended 100% of this budget object code. and then the next column-- and it gives you the total amount as well as the percentage. 

And then the next column shows you the percent expended of the forecast. So this tells you how much you have expended based on your forecast. So because this member has only expended $20,000 of the $80,000 for the project budget, they have only expended 25% of their year to date forecasts, because they said by quarter four they would have expended 100% but they've only expended $20,000. 

And then it gives you the project budget. So these are all of your budget totals by object code, and then the expenditure of overall budget is here. So again, they have only expended 25% of their overall total budget, and they have $60,000 left of this $80,000. So that's your expenditure report. 

Again, year to date expenditures are calculated here. Your year to date forecast, so where you said you would be at the end of quarter 4, for example, and then where you currently are, your project budget, the overall fiscal percent expended of overall budget, and then how much is remaining. 

This budget remaining is particularly important. Some agencies, for whatever reason, may over-expend in a particular object code. And when that happens, we do advise that you complete a budget revision so that you do not continue to run on a deficit for a particular object code. 

And in some cases, we've seen where a consortium or an agency did not take care of that budget deficit for object code, and then it continues to accumulate, accumulate, and then by the end of the program year they had a negative total remaining, and move them to the next program year with a negative amount which when that happens, the system takes off that negative number of the annual allocation. So you go into the next program year with less than what you had in the prior program year. 

So again, just something to be mindful of and to monitor, if you have over-expended in a particular object code and there is an opportunity for you to complete a budget revision, and maybe you're under expanding and another object code, completing a budget revision to move those monies around so that a particular object code does not go into the negative, and then you move forward into the next program year and potentially can have a negative amount. 

So I want to go to the chart. I see that there are some things happening there. So since budget changes cannot be done on fiscal reports, can you please demonstrate how to make those changes? It can be a bit confusing. So yes, you are correct. Problem, you cannot make budget changes on the fiscal report, but you can go back to the budget and work plan and uncertify the budget and work plan and make those revisions there. 

So I will quickly go back. Before then, I'm going to look at Jennie. So what happens if agencies are not meeting projected expenditures in any object code? So if agency is not meeting their projected expenditures in any object code, what will happen is this total year to date forecasts-- excuse me, this percent expenditure of year to date forecasts, if this total amount is accumulative under 90%, a corrective action plan will appear. 

So as you can see, they're at 60%. So this corrective action plan appears. And they will have to indicate the action steps that will be taken to address the expenditures falling below target. 

And so you as a consortium lead would help the agency with that, not necessarily help them execute the action steps, but monitor the action steps to ensure that they are meeting the action steps that they said they would meet in order to ensure that the next quarter or the next program year that they are meeting those expenditure targets. And then if that still is not happening, then you start to evaluate the member effectiveness process, and then underneath here, you'll complete the summary of the activities. 

So consortia primary contact-- consortia primary contacts or consortia leads, when your member submits their expenditure report or fiscal report, the CAEP office asks that you evaluate the expenditures but then also, more importantly, if there's a corrective action plan, evaluate that to see what action steps are in place, and then help your members to meet those action steps, and then also look at the CAEP summary of the activities. So that's just a tool that we can use. 

Can you share that we need to click the box to certify? Yes. So you click this box to certify that the agency did not exceed the allowed indirect rate as directed by the Adult Education Program Guidance. So you would click here to say that this indirect cost is not more than 5%. So this particular agency did not have indirect-- let's look at other agencies to see if they had indirect costs. 

So this agency, Butte-Glenn CCB, just looking at their expenditure reports, and I don't know if this is the final expenditure report, but here it looks like-- no they had a project budget of $3,809 in indirect costs. And so they did not exceed the allowed rate because you will look at their budget and work plan to make sure that this $3,809 is not more than 5%. 

And if that is true, you look at their expenditures, was $2,856. That's less than this and in theory, this should be 5%. And so you would check this box to say that they did not exceed the allowed indirect. So this amount is 5%. Hopefully, without me looking at it, I'm guessing that this is 5%, and then they did not over-expend. So you would check this to say that they did not exceed the allowed indirect rate for this agency. 

And then we have, Veronica, is that projected budget? Yes, this is the budget amount that they have indicated by object code. I hope that answers your question. So I'm going to quickly go back to the consortium as they require similar activities today. 

They have a model that agencies can follow to make sure-- so the state does not have a model for agencies to follow. The state just ask that you include explicit details in the summary of the activities of exactly what took place during that particular quarter. But at this time, there is not a model. 

And I'm going to go back to the fiscal report just to show Prue-- excuse me, budget and work plan to show Prue where she would complete the budget revision. And then we are going to share-- actually, she's not live yet, because we did a webinar on the new dashboards because we're running out of time. But I can talk about them just quickly. Someone got a view? 

And then Prue, if you go scroll down to the budget and work plan, you would go to the budget and workplan, let's say, for 2020-21, you would click on Butte Office of Education and this is the budget that we want to make revisions to. You as a primary contact, Prue, would uncertify here. That's going to open up the budget of workplan. 

Once you do that, the member would then come in, and let's say they have to make a budget revision or a particular object code to move monies around, they would click on their budget, and then they would make the changes to their amounts in their budget to move those monies around. 

And what they would just have to be mindful of that at the end of the changes that they make, the total available funds and the total budget have to equal one another. The remaining amount has to be zero. And that's how you complete a budget and workplan. 

They would then go next, and then they will submit, and then you, Prue, will receive an email saying that this particular member has resubmitted their budget and workplan, and then you go and verify the information, and then you would certify, and then their budget and workplan will be in certified status once again. So I hope that answers your question. So yes, you are not able to make those changes on the fiscal report, you will have to go back to the budget workplan. 

So I'm going to go to monetary reports. Just to look at these quickly, these are individual dashboards based on the CAEP deliverables that you can look at the consortium level, you can look at it at the member level or the entire CAEP level as we're looking at here. 

And it talks about the CFAD status, the annual plans status, how many approved member plans have been submitted, fiscal reports, you can look at the certified-- how many fiscal reports are in draft status, how many have been submitted. You can look at budget carryover, how many total expenditures, and we're looking at the CAEP total. But you can look at total remaining budget. You can also look at expenditures by object code in different categories. 

So we completed a webinar on this is currently being re-mediated, but as soon as it's available, we will share with you all, and then again, because we have run out of time and we want you all to have a break, I can do an individual session if you send me an email at Tap@caladulted.org Either myself or another member of the team can do a walkthrough of these dashboards because they are very important, and it will help you in managing the consortium. 

Unfortunately, we just ran out of time today, but something to consider in the future if you want that individualized session. So it's 10:50 now, and we have about 10 minutes before the next session starts, so I will go ahead and stop talking so that you all can take a 10 minute break. 

Please be back at 11 o'clock, and we will move forward with our next session with WestEd. And if you have any questions, feel free to type them in the chat, and we will address them. All right. Thank you all so much. I'm going to stop sharing my screen 

SPEAKER 1: Veronica, I'm just testing to see if you can see my screen and hear me. 

VERONICA PARKER: Yes, I can see your screen, and I can hear you. 

SPEAKER 1: OK. Super. I'm going to stop share. And when Randy joins us, I believe that he'll want to screen share himself, but I'll go ahead. There he is. 

VERONICA PARKER: Yes, he's here. 

SPEAKER 1: Randy, I'm assuming that you're going to screen share for your portion, or do you want me to just run that PowerPoint? 

RANDY TILLERY: Well, it's probably easier if we don't try to go out and back in, so you can run since you're finishing. 

SPEAKER 1: Since I'm so good at it. 

RANDY TILLERY: OK. Now don't make me nervous. 

SPEAKER 1: You just have to be clear. All right. That's great. You want me just to leave it up, Veronica, or did you have something you wanted to share prior to that? 

RANDY TILLERY: Are people already in the room? It looks like there's a lot of people already in the room. 

VERONICA PARKER: Yes, there are a number of people in the room. They've been with us since about 8:30 this morning. We are on a break and it ends at 11:00, and that's when you all will start. 

SPEAKER 1: All right. 

RANDY TILLERY: Sounds good. 

VERONICA PARKER: All right. 

RANDY TILLERY: Are you checking out the live view? 

SPEAKER 1: What were you saying? 

RANDY TILLERY: I said were you checking out-- do you have it open for the demo? 

SPEAKER 1: Yes. So can you see my screen? Can you see the PowerPoint? 

RANDY TILLERY: Yes, I can. 

SPEAKER 1: OK perfect. I'm going to run get a glass of water. I'll be right back. 

MANDILEE GONZALES: Hi, there. Hey, Blaire. Hey, Randy. 

RANDY TILLERY: Hey. 

MANDILEE GONZALES: We are just coming back from our break. So we're just going to give everyone a minute. We ran over just a bit. 

RANDY TILLERY: Hey, Blaire, are you there? 

BLAIRE TOSO: I am. 

RANDY TILLERY: I'm not sure this is the last version of the deck. Not that we can't futz our way through it, but-- 

BLAIRE TOSO: Let me stop sharing and make sure that I've got the right one. Hang on. 

RANDY TILLERY: Because I've moved the CAEP metrics up. 

VERONICA PARKER: All right, everyone. While Randy and Blaire work through their PowerPoint, you hopefully should be coming back. Things happen. I totally get it. You all should be coming back. And so we'll get started in just a moment. 

And this is the next section of our on board training and it's with westhead. And they'll be covering the LaunchBoard and student data reporting. So we'll get started shortly. 

Thank you all very much for hanging in there we know that we have presented a lot of information this first-half. And we still have quite a bit to go, but it's all great information that we think will be very useful to you all, especially for those of you who are newer in your roles. And for those of you who are more seasoned, I'm sure this is a great refresher for you, or at least I hope. 

RANDY TILLERY: Hey, Blaire, I think it was my mistake. 

BLAIRE TOSO: OK, OK. Hang on. Let me just-- 

RANDY TILLERY: I got confused on whether I was looking at the Zoom window or the PowerPoint window on my computer. 

BLAIRE TOSO: Oh. [LAUGHS] 

RANDY TILLERY: Sorry. But we will be right with you, and we will be much more focused in about 30 seconds. 

BLAIRE TOSO: OK. 

RANDY TILLERY: All right, good. So Veronica, should I just start or do you want to-- 

VERONICA PARKER: Yes, go right ahead. 

RANDY TILLERY: OK. Hi, everyone. Good to all see you again. We were with you yesterday, doing a session on the CAEP fact sheets for the three year planning and program evaluation and planning. So it's very exciting to be back here to talk a little more in-depth about the LaunchBoard and about data reporting processes for CAEP in general. 

So my name is Randy Tillery. I am an area director for post secondary education and workforce development at WestEd. Most of you know me because I've been with the CAEP program since 2017 since I helped lead the measuring our success process for identifying the key metrics for adult education and for the Cape program, then the ABG program. So Blaire-- 

BLAIRE TOSO: And I'm Blaire Toso. I work with Randy on the same team and out of the adult education field. I started out as an ESL teacher and then continued on until I actually have a PhD in adult education and have done evaluation and research a lot of product development. And I'm really intrigued with the idea of using data to inform best practices and using data to explore and create new and better programming for adult education within the educator's field as well as for learners. 

RANDY TILLERY: All right, next slide. That transitioning OK? There we go. So this is just a brief description of our group at WestEd. Our main focus is really on issues around economic mobility particularly at populations who lack access to adequate earnings employment or access to education. We work both in the K-12 space, adult education space, and higher education as well as the workforce system. Next slide. 

And here are just some of our lines of work. Many of us from our work on the LaunchBoard since 2012. We do a lot of work with data tools and analysis for different educational and workforce systems. 

Not as many people know that Blaire and I both have a background in pathway development and planning and the development of workforce programs and strategies, particularly on a regional basis involving lots and lots of different systems. We have a whole team dedicated to helping colleges with the implementation of Guided Pathways and we'll be running a whole series of institutes for the Guided Pathways colleges in California over the next year and a half. We do labor market and economic analysis to support regional pathway development, and we lead a lot of conversations on the use of data to increase educational success and support economic mobility. 

And here are our lovely headshots. Next slide. So this is generally the content we're going to be covering today. I'm just going to be completely honest and tell you that it's way too much information. I tend to talk a little bit too fast. 

But please feel free to use the chat to post questions. And I, Veronica, and others will be looking to raise those questions with us as we go along. We don't mind answering questions as we go. Next slide. 

Origin stories-- so I don't actually know how many of you are brand new to CAEP, the program formerly known as AEBG. But I do like to do just a little bit of background about how we got to where we are with the data and accountability systems since it has been really a four year journey. So next slide. 

So this all goes back to AB83 in 2013. The LAO had produced a report asking why we have so many different systems serving adult learners between adult education, adult non-credit, and adult credit programs in the community college system. And AB86 was designed to really force the community colleges and adult education and non-credit education to come together to create regional systems to better serve adult learners. 

And that was when the first three year plans were created. There was a lot of talk in the legislation around acceleration, contextualized basic skills, really joint or integrated programming strategies, and really kind of shared career development platforms. It's worth noting that people have been really working hard at this for four or five years. And I feel like, really, even just now, there are places where we're really starting to get traction. 

I've worked with consortia in the last year where they actually have started to have those conversations about how can we can't offer non-credit at an Adult School, how come we can't integrate the Adult School offerings into the Credit Pathway, that I'm seeing deeper conversations in particular consortia around the state that's really exciting. Next slide. 

And then shortly thereafter, AB104 in 2015 that really created ABG and created the funding mechanism for it now called CAEP. And that was where all the consortium processes for governance membership, funding allocation, annual plans really kind of redefine the three year planning cycles. It also identified, very salient for our discussion here, requirements for measuring the effectiveness, data and accountability reporting, and common assessment for placing students into courses. And it identified the seven general program areas that we still live with pretty much intact today. Next slide. 

Here are some specific language from AB104 from that time. So one of it was identifying common measures for determining the effectiveness of members of consortia in meeting the educational needs of adults to as much as possible to align the data used for reporting with data reported by local agencies for other purposes such as WIOA, also Perkins, Strong Workforce, and establishing common assessments and policies regarding placement of adults into the adult education program, measuring the needs of adults, and the effectiveness of providers. And then it established twice yearly reporting cycle on October 30th and March 1st after the end of the previous program year. 

So in response particularly to this question around identifying common measures is really getting community colleges and these adult education providers in the same page about how were we going to measure ourselves. The legislation also had seven general kind of required reporting elements that are actually reflected in the AEP scorecard part of the tab that we'll review a little later on. But we actually ran a process with adult educators, community college, credit or non-credit, and the state agencies. Next slide. 

There's animations. Blaire, you can just go ahead two more and it'll pull up the text. Thank you. So we actually ran a process over eight weeks in 2017. Many people on this call I'm sure remember that. 

We had 35 K-12 college and workforce practitioners, we had 40 hours of meetings over eight weeks, we had three separate committees-- a Data and Accountability Committee, basic skills, and CTE committees. And out of that, we produced a report to the California legislature. And it included definitions for what are the populations that we're saying are part of adult education, what are the program areas, and what are the primary accountability metrics that everyone could agree on. Also identified processes for tracking and reporting, and we identified a process for aligning the basic skills rubrics for adult education with community college. 

And initially, that process was meant to be a across walk between the two different systems. But partly, as a result of that, two years ago, we actually rewrote the community college developmental education framework and rubrics to match the federal EFLs. And now there's a 1 to 1 correlation of levels across both systems. You can see to the right the various data definitions, programs and populations, the difference between adult serve-- someone who shows up for an Adult School maybe just one thing and leaves as opposed to someone who's a participant who gets 12 or more instructional contact hours. We define measurable skills gain metrics for ABE, adult secondary education, and ESL. 

We defined a number of transition metrics transition from ABE or ESL to adult secondary Ed or transition to post-secondary education, high school diplomas, post-secondary credentials, occupational skills gains. It was really a ton of work and people worked really, really hard and it was really nice to see the conversations in that process. Next. 

So I'mma talk a little bit now just about the metric definitions and just some baselines. And this is sort of at a higher level so that you understand when you start seeing things in the adult education pipeline dashboard, kind of they don't come from nowhere. These are definitions that basically came out of that eight week process that we led in 2017. So I'm just going to review a few of them so that people understand the basics term structure and frameworks for anyone who isn't aware of how we're defining things. 

So it's important to say that if you haven't been to the adult education pipeline dashboard, there is a tab or a tile there that takes you to the CAEP report card metrics. And these basically are aligned to the metrics that we agreed to in 2017 that we would report annually to the legislature. And really, it's for you to be able to see a snapshot of how many reportable individuals or adult serve are actually in your consortium, how many students are participants with 12 or more instructional contact hours, how many students completed EFLs. 

Gary's pointing out in the chat that we're trying to eliminate the preliminary report. Like I said, this is history. This isn't necessarily the way it's going to be next year. 

So this is just core metrics that you can have a chance to review. And we'll be talking a little bit more about some of these as we go. Blaire. 

So CAEP students-- so one of the big items of discussion in the process of 2017 is who is an adult education student. And what the committee came to was that basically well-- so this applies differently to WIOA where you can serve students as low as 16. But for CAEP funding, it's any student 18 or over served at a K-12 Adult School or a community college non-credit program with at least one instructional hour in a CAEP program area who receives services. So that's the total denominator of students that actually count, the purpose of the CAEP. And then we have-- just give me one second. My screen [INAUDIBLE]. 

So while the PowerPoint becomes back up on my computer, I think the other two definitions here are the difference between a reportable individual or the adult served metric and then participants. And so basically, the total denominator of adult education non-credit and credit students who get one instructional hour or get services are part of the adult served definition. And then anyone with 12 or more instructional contact hours for the community college folks-- that would be 12 or more positive attendance hours then counts as a participant. 

The distinction is important because we only really count outcomes-- there we go [INAUDIBLE]. We only really count outcomes for students who hit that 12 or more instructional hour threshold, which means that when you see the number of EFLs and there's the denominator, the total number of students that that's measured against for calculating a percentage that it's only for participants. Students enrolled in credit programs are not considered adult education students unless they are also enrolled in a K-12 Adult School or a community college non-credit program. 

So if a student starts in non-credit or at the Adult School and start taking credit courses, they're still an adult education student. If they're enrolled in credit courses only, they're not unless they started at a K-12 school community college. And then they're a student who's transitioned to post-secondary education. Next. 

So as we said, we tracked a number of kinds of students. So we have adult serve with 1 to 11 instructional credit hours or adult services only students. Actually, all students are considered part of the adult serve denominator. 

And then of participants, we have two types of students that we count. We actually allow you to disaggregate for first time students. We have students who just started in that year or continuing students. So students had been enrolled in a previous year. And then you're able to look at the characteristics of your students by gender, by age, by race, by ethnicity, or by the WIOA barriers to employment categories which are both captured and TOPSpro Enterprise and in MIS and in MIS. 

Like I said, we do capture all of the WIOA barriers to employment. There are two categories of those. And this is sort of a common practice in the community college system that if a student has ever been flagged as having a cultural barrier as ELL, as ex-offender, foster youth, low income, or low literacy, they're always flagged that way. 

So you have that at your disposal to understand the students started with a cultural barrier at some point in time in the past. Could be the current year or previous year. And then some things are just flagged in the selected year. So just within that program year. Displaced homemaker, homeless, long term unemployed, migrant farm workers, seasonal farm workers, exhausted TANF for two years, or single parents. 

These are the primary program areas. So participants in English as a second language, ESL students, adult secondary education, adult basic education, career technical education, and participants and programs for adults with disabilities, and then adult participants training to support child schools success. Within CTE, there are three subcategories. Blaire, if you click, they come up. 

So short term vocational training-- that generally means vocational training designed to lead to a job that is one year or less in duration. Workforce preparation, which is the WIOA category of what people call kind of workforce readiness skills. Some people call them soft skills. Now we're calling them 21st century skills. And then pre apprenticeship students-- so that students participating in a program to help them get the skills and to be ready to enter into a registered apprenticeship program. Pre-apprenticeship programs are meant to have a direct connection to some form of registered apprenticeship. So they can actually have access, and they're supposed to be a formal agreement for that as well. 

One second. So these are just the definitions that I shared with you. Next slide. 

Educational functioning levels which probably many of you are aware of if you've spent any time in adult education, particularly in a WIOA title to program. And in CAEP, unlike for WIOA, there are more than one way that a student can achieve an educational functioning level partially based on whether they're at a K-12 Adult School or WIOA Title II agency or whether they're in a community college that is not a WIOA Title II agency. 

So for WIOA agencies, including K-12 adult schools, the way a student achieves an educational functioning level gain is through pre and post testing using cost of standardized assessment instruments. And this aligns to the methodology for the WIOA measurable skills gained in WIOA. And then community colleges-- there are two additional ways a student can achieve an EFL. If they're a WIOA Title II agency, they are also pre and post testing. 

Most, if not all, of their ABE, ASE, and ESL students using concepts instruments. And so we capture it that way through the top score Enterprise data, the cost of submits to help populate the LaunchBoard. It can also be through course progression. So based on the CB21 competency rubric-- which really is just-- it's just like the EFLs. 

There are six levels and it's used to code courses. So if they move from a course, which is CB 21 equals C, and they finish that course and then enroll in the next higher level course, they're basically counted as having achieved a level gain. And as I said, CB21 in the community college system is now aligned to the federal NRS levels. And then there are new data elements that we're not going to go into tremendous detail today, but which can be used by CASAS or other NRS approved assessments to enter students' EFLs based on the cut scores for that instrument. Next. 

This is just a sample of what CB21 looks like. And you can see that at the top, it's aligned to the levels for CB21. And there's one of these for each kind of course level and complexity. Next. 

So student transitions-- we measure two types of transitions. Actually, we measure more than this, but these are the main ones that are part of the report card measures that we report to the legislature. And so transition to adult secondary education, which is a student who begins in adult basic education or ESL program at a K-12 Adult School or a college non-credit program and transition to an adult secondary education program, adult high school, or adult GED prep program at any K-12 Adult School or college non-credit program. 

What that means is if they start as an adult basic education student at the Adult School, and then they enroll in an adult secondary education course in the college non-credit program. We actually capture that as a transition as well because we get the full TOPSpro Enterprise data set from CASAS, and then we have the MIS set. So we can actually track this between institutions. And it's the same thing with transition to post-secondary education which, in this case, this is an area of great discussion with the committees. 

Post-secondary education for the purposes of CAEP is considered to be CTE as well as college credit coursework because a number of adult schools in California are actually accredited as post-secondary vocational institutions and have highly developed and structured CTE programs. So if a student who transitions from adult basic Ed, secondary Ed, or ESL to an Adult School or a college non-credit program into a CTE program, then they're considered a post-secondary transition. If they transition into a college credit program, they're also considered a post-secondary transition. 

I will also say we have a separate metric that actually just looks at students transitioning into college credit coursework as well. And that's actually part of the dashboard. Next. 

The transition is coming up really slow. 

BLAIRE TOSO: It's the one on post-secondary credentials, Randy, is the following one. 

RANDY TILLERY: Oh, here it comes. OK. So post-secondary credentials are not necessarily considered college credentials. The college credentials count for post-secondary credentials. So college certificate or a college degree is considered a post-secondary credential. 

But if you look at the actual definitions in WIOA and across the different systems, there are a number of kinds of credentials that can also count as being post-secondary. So these include things like locally approved certificates eligible for the eligible training provider list or ETPL list. And these are just examples. So this isn't necessarily a complete list. So obviously, there's some fungibility in terms of how you choose to count something or not. 

A career development or college preparation or other local CTE certificates with more than 48 instructional contact hours, certificates that meet the minimum threshold for inclusion under Perkins, certificates that meet the threshold for Title IV federal student aid, completion of any degree or for-credit college certificate over six units. What's important to note is that workforce preparation, work readiness, or things like OSHA or Safeserve are not counted for post-secondary credentials. Then so if a student goes from an adult basic Ed program into a work readiness program, that's also not a post-secondary transition. 

And as I said, because there are a lot of CTE programs that are actually accredited in adult schools and there are many adult schools that actually administer federal financial aid that we're counting any CTE program as a part of this. Next. 

So employment and wage gains-- so we include both the WIOA employment metrics and we include additional metrics related to employment and wage gains. So that's employment 2 and 4 quarters after exit-- that's aligned to WIOA-- the median annual wage, two quarters after exit-- that's also a part of WIOA. We look at wage gain after exit compared to their wage before their entry into adult education, and we look at wage after exit compared to the regional living wage. 

Now one of the challenges with employment metrics in CAEP is that the preferred methodology is a match to the California Employment Development Department unemployment insurance wage file. However, as I think many of us know only a percentage of students in adult education actually provide a social security numbers, which is what you need to match that data. Historically, it's been about 10% of K-12 adult Ed students and about 50% of non-credit students who provide their social security numbers. 

We have had discussions among the CAEP contractors about including both the CAEP wage survey administered by CASAS and the community college CTE outcome survey data in potential future builds of the LaunchBoard. I will say these would all be partial fixes, that none of them would absolutely capture 100% of employment. But they would give you different views understanding the limitations of what generally is happening to at least some of your students after they leave your program. 

GARY W ADAMS: Randy. 

RANDY TILLERY: Yes. 

GARY W ADAMS: Sorry to interrupt your flow. We are exploring a memorandum of understanding a data sharing agreement with the franchise Tax Board which may give us access to wage data that is a better source than a survey method. So I just mentioned that. That's something that is in the works and hopefully more to report later down the road. Thanks. 

RANDY TILLERY: Yeah, and obviously, the franchise Tax Board data would have its own sort of window because we know that many immigrants work under the table with unreported income. So employment's just a little bit of a murky field when it comes to adult education. But Gary, that'd be great to have that as an additional data source to work with. And I look forward to that. Right, next slide. 

So we also have a metric now for completing an immigrant integration milestone. So there's three ways that that's measured for the purposes of CAEP. On the MIS side, it's if someone's completed a community college EL Civics course. And then it also includes if they've met the immigrant integration indicator outcomes related to the EL Civics COAAPs for CASAS. 

And the denominator for this includes all participants rather than just those identified as ESL. This is an area where we look forward to continuing to refine the way we measure this over time. Next slide. 

So this is just to understand a little bit about the data reporting flow. And it's quirky. So all K-12 adult schools report their data through CASAS TOPSpro Enterprise as do community college program, not non-credit programs, who are WIOA Title II funded. And I believe that's currently about 19 colleges statewide. And then community colleges also all report all of their data, and this is part of their regular reporting through their submissions, their management information system submissions, to the community college Chancellor's Office. 

So what that means is we have some students we both we both see through MIS and we see them through TOPSpro Enterprise for community college non-credit programs. What I will say is though was we deduplicate those students so we're actually not reporting the same student twice. But if you are a college non-credit program reporting ESL gains up through TOPSpro Enterprise, we do capture them there if they're not captured through one of the other methodologies that are available for community colleges to actually report that. We use what's called a derived key-- we call it a derkey-- which is a last name first name data beta gender to deduplicate students. 

We don't use general use socials because there aren't enough socials actually to do it adequately. We get a pretty high hit rate for students who show up in both systems. So we will find students who are enrolled in the K-12 Adult School that show up in TOPSpro Enterprise, and we'll find those same students at the community college in a non-credit or credit program and the data analysis, which is how we're able to do things like report students who are co enrolled or students who have transitioned from the Adult School to the college. Next. 

And one of the things-- and this is really-- I mean, this applies to TOPSpro Enterprise as well. But one of the things we learned a lot and that we have spent a ton of time in the field, in particular with the non-credit programs-- discovering coding issues that were messing with the data. 

And one of these was that it was an example that lots and lots of colleges-- we didn't really have non-credit programs. We're funding their tutoring and study skills students through non-credit courses that they had created. And those students were showing up as adult secondary education students, I think, in most cases in the data. 

So we excluded those two top codes in MIS from the denominator. So the numbers are now very, very close. So there was a college in particular that had like 3,000 non-credit students, and they didn't have a single non-credit program other than the fact that they were funding their tutoring center with the courses in that top code. Blaire? 

So I'mma talk a little bit about the LaunchBoard. And then I will hand over to Blaire to talk about the adult Ed pipeline. So first slide. So this is the LaunchBoard. And if you just Google Cal-PASS LaunchBoard. It'll take you right to it. And we often encourage people to go ahead and do that while we're doing these presentations. 

Basically, it's a suite of data tools, it's six different dashboards. There's going to be another one added, a K-12 strong workforce dashboard coming up. There, basically, you can track any student in any program at a community college in California and look at outcomes including employment and the course taking patterns. 

The adult education pipeline is the one we'll be covering today, specifically for CAEP, the college strong workforce program K-14 CTE transitions, which is an old tab that I actually love, and then Guided Pathways, which is a major reform effort in the community college system. The LaunchBoard development team is really a partnership. So it's a Chancellor's Office program. But historically, there have either been entities who have provided support for it. This is a little dated, but-- so Delta College which is the fiscal agent for the education results partnership in the Cal-PASS plus program. WestEd has worked on this since the beginning. And at different times, the CRP group or the Community College Centers of Excellence have also provided support for this effort as well. Next. 

Like I said, it's a suite of web-based dashboards. But really, it was designed-- this all came up literally at a CCCAOE conference meeting in 2012 and it started with CTE practitioners in the college. It said, if we just have better access to our data and what happens to our students, we could tell better stories about what our programs do, and we could figure out ways to improve our programs to help students attain better outcomes. And so it's really been primarily meant to be-- it's becoming more of an accountability tool. 

It really was meant to be to improve practice, to provide data that could be sliced and diced by practitioners in the field to figure out what's happening to my students and how can I help them do better. So how many and what types of students are in programs, are they earning a living wage for the region, how do transition and completion rates compare across different student populations of programs. You can disaggregate almost any metric by race or ethnicity or gender which allows you to really look at sort of equity issues within your own programs. 

Like I said, some of the tools are more accountability tools [MUFFLED AUDIO] student success, strong workforce. Guided Pathways is really a reform tool tied to that reform effort. And then really, some of them are organized around student journeys. So the community college pipeline, the adult Ed pipeline, and K-14 transitions are looking at how many students connect, how many students progress, how many students complete, how many students get work, as kind of can really see where along the student journey things are going well or where things might be breaking down. Next. 

So some of the key characteristics-- it was really driven by practitioners originally. I've talked about the different types of displays. It is aligned to the legislative and CO priorities. That's why we have the AEP scorecard view and the adult Ed pipeline. 

It uses public data sets for the most part. The emphasis is on administrative data sets. So the state MIS records, Cal-PASS for the K-12 strong workforce, those data sets, and uses the direct student enrollment records in most cases. When CASAS sends us the data file for TOPSpro Enterprise, it's a unitary data files for every student. 

So we're able to run the calculations based on the individual student level data that sits inside that data set. It matches data across systems. So we can look at adult and community college transitions, K-12 the community college transitions, community college transitions into the labor market, adult education into the labor market, which is why the matching algorithms are so important. 

It has views at an institutional level, a regional level, you can look at the state level. So you'll be able to see data for your consortium, and then you'll be able to see data for every member of your consortium. And then it allows for disaggregation and it really help you with doing various kinds of equity analysis for your data. 

It uses multiple data sources. So the MIS data, employment earnings from the California employment department wage file, in many of the tabs, we do include data from the CTE outcome survey, we look at transfer outcomes from community college through a match to the National Student Clearinghouse and the CSU and UC systems. For some of the dashboards like the students success dashboard, we use the CCC apply data. And we have the LMI, and then we have the CASAS TOPSpro Enterprise data. And the idea is to be able to merge data into one place. 

Now for those of you who have followed the California cradle to career efforts, it's going to operate in much the same way. So it's going to be based on unitary administrative data sets from Cal-PASS, from community college MIS, and from the CSU and the UC systems to really look at student journeys longitudinally. And eventually, it will include things like social services data and health data is really learn much more about who's being served across all these systems and how they're being helped. Next. 

Finally, what makes it important-- the community college system-- and I really laud the leadership at the time in 2012 to 2015 when we started working on this is really to increase the stakes for how we serve or how to make ourselves really responsible for our data and what happens to our students. So it's about owning our weaknesses as well as our strengths. There are things in our data that we're not crazy about, that we know could be better. 

Linking outcomes to practice, really helping colleges focus on continuous improvement. A lot of the PD that Kathy Booth and I and Blaire and others in our team do is around this issue of continuous improvement and using data to drive changes and outcomes. I'm just connecting the dots about what's happening to students in different systems, what's happening to them in adult Ed as opposed to community college, as opposed to the labor market. 

This may be my last slide. It is my last slide. So I'm going to pass over to Blaire now. And she's going to start walking you through the adult education pipeline. Take it away, Blaire. 

BLAIRE TOSO: Thanks, Randy. We've covered a lot of the context for the adult education pipeline. So we're going to get a little bit more in depth with the way it's structured. And then we'll have time for a live walk through at the end. 

So Randy really talked about the purpose. But I just wanted to reiterate that for the adult education pipeline, these are the three purposes that the pipeline was designed for. It was to improve educational practice and economic mobility for adult learners. And that ties in to being able to see what their journey is, what their outcomes are, and how those are tying into the goals that have been set. 

It's used by consortia to develop and track progress on their plans and annual updates. And it's also used by CAEP contractors and technical assistance projects to identify which countries who might be struggling or pinpoint a topic across the state that needs to be just a little extra PD or we need to explore a little bit further. 

And as a quick review, the pipeline is a resource for adult educators, colleges, and consortia to improve student outcomes. So the audience for using the adult education pipeline is very wide. And we also use it to report outcomes to the legislature. 

The metrics are aligned to the student journey. It includes those wide array of data to populate the dashboard. It is one of the only complete sources of college non-credit and K-12 adult education student data and outcomes. 

I've done a lot of national work, and I really think that this is something that California should promote themselves for. This is a really unique board where you are representing a full journey for adult education learners. And then also, the consortia and institutions are expected to use this adult education pipeline as the data source for the development of their three year plans in NOVA. And I tuned in a little bit towards the end of the last session. So you all are well familiar with that. 

So some of the key features that we'll cover more in depth is that the adult education pipeline offers some nice ways to visualize your data by region, consortium, institution. There are six high level live metrics for you to easily navigate and target your interest area. There's also another tile, that very first one, which is the AEP scorecard which Randy reviewed that gives you really access to some of the key points that are reported to let the legislature. There are these summary infographics, detailed data charts, and these disaggregation pieces and a comparison view that we'll walk through. 

So we've designed the dashboard to follow a learner journey. And under them, we have put all the different metrics. These are the high level, the key progress metrics. 

But what we really like to point out on this is that the outcomes are reported and counted on the dashboard regardless of whether a student does all those steps or whether they just complete a portion of it or have they made it to the 12 hour threshold. So this charts all of these regardless of where a student might enter or exit. And as I said, this also identifies the key metrics along the journey as they're structured on the adult education pipeline. 

So what it is to note is that the dashboard is updated yearly. And it's important that we do this so that we can align both with the other dashboards on the LaunchBoard which are updated regularly on a yearly basis as well. And part of it is just so there's an equivalent for the continued learner journey through the credit world should they choose to make the transition further as Randy said. 

And sometimes, you have dual enrolled learners. So we want to be able to capture the full array of outcomes and the full journey for a student and adult education learner. The other reason is we are responsive to feedback from the field. So feel free to email us that so such as-- like for example, last year, we added a code we have been after it for a couple of years. The field had really been asking and requesting for an IMIS coding element that would document a high school equivalency outcome so that that could really be flagged and seen. 

As we all know, that's one of the main outcomes that's measured in WIOA. So this was an update that we made based on feedback from the field. Sometimes, it's to identify a coding error or we need to identify or refine a definition to better align with MIS or TOPSpro categories such as-- we recently updated the immigrant integration outcome to better align with the COAPPs and to include a broader array of learners who weren't just ESL, realizing and identifying that immigrants may not necessarily be English as a second language learner. 

And while we do use the AEP to document learner journeys, it's also used in a variety of ways. We talked to a lot of consortium leaders last year to find out how we might want to improve the app or gather some data about how people were using that. And these are some of the ways that we were told that people use the adult education pipeline. So it's for planning, evaluation, program improvement, and then definitely the three year planning, which you're hearing a lot about. 

So let's go ahead and dig into some of the key features on the adult education pipeline. The pipeline allows for you to filter for region or institution. And you'll see this up at the top. 

We're going to take a close look at this because as we do the live walkthrough, it goes fairly quickly. And this, you can return to these slides and take a look at how these layout in segments on the pipeline. So you can filter them by a statewide region consortia or community college districts. 

You can also then look at them by institution. You can sort it out by institution and by academic year. Each time though as you select these, you're going to want to click on the view so that it will refresh the screen to show you the data for what you are looking for. So you can go in through here as you see. That we'll take you there. 

You can also use a detailed comparison feature should you want to review data side by side with either state wide or another consortium or even by institutions. So you have a comparison feature. The screen will pull this up, and you'll be able to look at them side by side on your viewfinder on your screen. What you will want to remember to do is to set the academic year for each one of those as they are-- just because you are looking at 2019 on one of those, it will not necessarily auto populate for the second one. 

So Randy mentioned the data tiles. When you go to the home page of the adult education pipeline, it provides you with this overarching general information regarding the key metrics. And the tiles are organized along the student journey. 

Once you click on that title, you will go first to a summary infographic which offers you an initial visualization of the key information. So for example, when you go to the students and programs, this will bring up the participants in the CAEP program, race, ethnicity, gender, and just gives you a brief glimpse at what you can look at. And then as you dig into that under student programs, would be able to get detailed charts. And each chart will provide you with some different aggregations. 

And here's what the disaggregation will look like. They'll be at the top of the chart. The first population will always be the time trend, which is really interesting for your overview. But as you want to look maybe at some of your equity issues or you want to look at which programs are being enrolled and whether you're serving returning or first time students, clicking on each one of these will bring you into a more refined look. 

So for example, what we see on the screen right now is you have the gender, and then you have just for the program type of adult secondary education, and these are the returning or continuing students. So you can drill down and really parse out the information that you would like. And most of the trend of the metrics will have this disaggregation. There will be a few that are currently not, but that's on the list to do for our next build. We'll be releasing adult education pipeline 5.0-- we're currently in 4.1-- 5.0 next spring. 

A new ad for this year was the top five charts which allows you to view a comparison or the highest achieving in each of these categories. We don't have them for everything, but it will give you an idea. And we suggest that you use them to look for people who might have a similar program that you do and collaborate with them or just understand better where your program is placed in the state. 

So you're probably thinking, OK, this is a lot of information, Randy's covered a lot, you're covering a lot. But we do have some really excellent adult education pipeline resources, and we're continually producing some more. So I just wanted to point these out. These are all-- if they are linked, that they will be live in the PowerPoint when you receive it. 

So there are always the CAEP Webinars. CAEP does a, SCOE TAP wonderful job of supporting the field through a variety of webinars. We produce how to guides, we've got the data dictionary, and we also have some infographics that will help you move through and explain some of what we're saying. 

Then as the WestEd staff, we have a team who works on the CAEP dashboard. And we are very accessible. You can email us, and we are willing to do institutional or regional trainings. We also do check-ins. So feel free to reach out to us, and we'll happily either point you to some of the resources or willingly set up a call with you to discuss some of the issues and see what we might be able to provide for you. 

The resource library which I'll show you when we do the live walkthrough. There's a link on the adult education pipeline. There are some excellent resources. 

We have a newly produced the adult education pipeline dashboard coding guide. It's a long document, but it will offer you-- it's done with tips and it's very accessible. You want to go in, and you can search it and find what you need in there. We highly recommend going back to Randy's point about accurate coding. This will really help you do that. 

The AEP dashboard overview is an infographic that will walk you through-- as you can see, there's a screenshot of it. It will tell you exactly what the tiles are, it will help you navigate. There's a piece on how do you dig into equity and use disaggregation, what's the difference between my MIS and CASAS TOPSpro reports. So dig into that one as well. 

And then there's also the adult education pipeline FAQ which will answer some of your basic questions. Like I said, there is a host of other resources. These are just a few that I'm pulling out and highlighting. If you're new to the dashboard, they'll be really useful for you. 

We also have a feature-- the tooltips on the dashboard. It's that question mark up there. If you click on this, it will give you further information about the element, how it's calculated, what numbers are used to inform the analysis, et cetera. It just sort of breaks it down and provides you with a little more information if you're interested. 

We also know that people are interested in digging into their own data and organizing it or sorting it in a way that might better answer a question that is not represented on the dashboard. So we do have an export feature. And that is one of the tiles that is on the AEP dashboard. And I'll show you how that works as well. 

So the metric definition dictionary is a resource that a lot of people don't use. But it should be your best friend. Honestly, it's a wonderful way, whether you're using MIS to report, COMIS to report, or you're using TOPSpro to report, if you dig into that, it will answer any of your questions. 

It's a little bit complicated from when you first looked at it. But once you get to know it, I promise you'll love it and you'll respect it because it gives you all the information you need. We'll happily walk through it with you if you would like to set up a time with us to explain it. 

But as you can see, it gives you all of your data definitions. And that includes how the displays the limitations the caveats will tell you your key source or documents that you can look at and it includes all of your metrics, the definitions, any related metrics. Each one will display how is this calculated in COMIS [INAUDIBLE], how is this calculated for TOPSpro. And what it really does is it makes it transparent how we're looking at the two different data sets and showing them how they're being reported on the AEP dashboard. And then there's also any notes of you should know them. 

So the other thing is when we're walking into this, the AEP-- it's really complex. It's beautifully set up. I have to say, I don't do anything with the design aspect other than to maybe give some input. So I can say, it's beautifully designed, it's very easy to move through, the infographics are easy to read, but there are a few key concepts that we think that you should when you're looking at your data. 

So Randy talked about the de-duplication or student matching. This is something that is done. We get asked this repeatedly. 

Students are not represented twice. They appear once in the dashboard. And we do that through the derived key which is the last name, first name, date of birth, and gender. And that's how we identify and make sure that if students are reported twice or through two different systems, we can match them up and report them as one individual. 

As we're talking about coding, it's really important if you try and be consistent in the way you name students if they are how you put their names in, their date of birth, and verify as much as possible if they're using a nickname or if they use their full name at different iterations in your program. I know you can't look across institutions necessarily. But within your institution, you can certainly do that. 

The other thing that's important-- people oftentimes ask us about how do we calculate. And so we use the denominator and the numerator to represent the outcomes. And just for those of you who haven't experienced this, the denominator is all of the students who can be identified in a metric. So that maybe all who identify as female and enrolled in ESL. 

And then the numerator is the total number of students who meet the criteria of the metric. So for example, in the denominator, you must be a participant. That means for the educational functioning level, you must meet that 12 hour threshold, be enrolled and then be enrolled in ESL, ABE, ASE program. So that's the base number that represents the field, the numerator, the number that you're seeing that's posted on the dashboard. It will be out of the number of people who are included in the denominator. 

And the numerator is only those who need it. So for example, in this example, you have to be a participant, you have to be enrolled in ESL, ABE, or ASE program, and you must have completed an EFL level either using a pre-test or a post-test or through the course progression in the same program area. And if you click on that little question mark that I talked about, the tooltip, it will tell you what's being used as the denominator and the numerator for that. 

We see a questiom-- when will 2021 data be in the LaunchBoard. That usually will be when we release it in spring of 2022. Oh, Randy's on it. He's on it. Thanks, Randy. 

So another important feature about this that people ask about is the lagging metric. The lagging metric is where a learner may need more than one year to meet the criteria that's required for us to register it as an outcome. For example, in a transition, it may need one more than a year to complete a transition. So we wait until the following year. And then we'll go back and retroactively look and see if there's been a completion in the following year and it posts to the other year. 

So if we pick it up, for example, in 2021-22 data, that will be posted back and show up and counted and repopulated on the 2021 dashboard. This is also used-- many of the employment and earnings are lagging metrics because we need to receive verification from EDD UI wage files to be able to document the student outcome. And also maybe we need to wait until a student fully exits a program. So we have to wait a full year to make sure that they haven't registered so we know that they've transitioned out of adult education. 

These will appear as length on the pipeline for the current year. But you can go into it select an earlier year and the previous years that data will display. All right, so we're going to go ahead and do a live walkthrough. I'm just going to close this and screen share to the next. 

Sorry, I've got to get my dashboard up. So as Randy said, you can Google Cal-PASS plus or you can Google adult education pipeline, California, any sort of combination of words and it will still get you to the LaunchBoard. Or you can bookmark it, which is probably the easiest way to do that. 

When you hit the LaunchBoard, you're going to come into this page with all the tiles on it. And the California Adult Education-- if you just click on that, it will bring you to that home page that we talked about. And here are the tiles. Up here is where you select your region. We've got statewide all, 2019, '20, and then I would click view and that will repopulate this. 

Here is where you'll find the detailed data comparison that we talked about. And you can compare it to-- if we change the region, I can go to East Bay, close all the institutions, and click the view. And that will show up when we go into the detail more accurately. 

We do that and then if we click on this first tile, you can view the AEP scorecard. As you said, these are all the metrics that are reported to the legislature. You click on time trend. So again, you drill down, you can look and disaggregate the data by gender, race, ethnicity, and age group. You click on one of these, it'll give you a dropdown. 

Any time there's a time trend, this is what the time trend will look like. And because we've got this comparable view, we have their information displayed as well. Let me go back to my. And same thing, if you go down-- sorry. 

You go into the instructional contact hours, that will show up as well. If I want to switch it out, and I want to see what's going on by age group, then this will populate with a graph such as this. One of the thing is for all of the graphs, if you hover over them, they will offer you just the number, but also out of the number and the percentage. So always feature your hover view on this. 

So then you can click back home. And say we want to go into students and programs, we talked so there is that summary graph that we talked about. If you want to see a little bit more information or a lot more information, you click onto your detailed data. And this comes up with a page and it will offer you, on the left hand side, a full menu of different metrics that you can identify. 

And so if we're looking at participants in a second ESL of the language, again, here are your features, your drill down features, inclusive of gender, race, ethnicity, and age group. And again, your first time returning or continuing. The other piece that you can look at is if you drop further down, you can click on the technical definition. That will take you out to the metric definition dictionary and tell you exactly how this is being calculated. You can click on for a little less information. 

There's the tooltip that we talked about. And then underneath there is a graph that you can look at, and it will provide you with the information. And so we'll just take a look at what it looks like with race, ethnicity, and there it goes and it populates. 

One thing that we should note is when you hover over one of the lines on the graph, they are taken out of not the full denominator. It would be out of the learners who participate and are in this characteristic or category. So for example, this would be all of the 152,000 roughly. That would be out of only the Hispanic or Latino students as opposed to the full piece. Just allows you a bit better to see how within a particular characteristic a learner is doing. 

And you can see we've got all of the different metrics. And it's a dropdown, then you will have subprograms identified here. So that means that they are a portion of the data from another metric. You have your demographics, barriers to employment, et cetera, et cetera. 

One of the ways that you'll move you can easily move through your tiles is using these arrows at the top. So you can click through, and now we're on the progress which otherwise, when I first started working, I would always go back to the home page. And I could easily organize my thoughts this way. 

If you look at progress, go to progress again. It will pop up. And you'll see here that now we've got a third disaggregation. So let's do gender, and we want to see it, as I said, down by ESL. And we want to just check out the first time students. There you have it. Again, here's your hover feature. They all function pretty well the same way. And then you've got your technical definition, your tooltip, and your chart here below that breaks it out in a different way. 

So I'm going to go back to the home. And I'll show you about what we talked about with the lagging metric. And I said that they will appear as a blank and it's explicit. It will tell you to select an earlier year. 

So let's go up here. We choose 2018, 2019. We want to view again. And now you can see that these two tiles which were previously blank are now populate. We go in, get that, go into the detailed data, and you can look into access any of these. 

This is the top five institutions chart that I was talking about. As I said, they don't show up for all metrics partly because the metrics just don't function that way. There's always some back work about how outcomes are represented. And sometimes, they're too complex to represent in a top five charts. 

But we did get great feedback. So we're looking at them, refining them, and we'll probably add a couple more. And there you can see. It populates up, and you can go in, and you can look at-- you can dig into the 25 and see exactly what's informing those transitions to post-secondary. 

One of the things I will say upfront we've been looking at is that smaller institutions, when we're just doing percentages, are overrepresented. So we're looking at having two different displays, one by number count and then the other by percentage. Marina, yes, we are able to meet with a consortium and help identify some of the data caps at the individual institutions. 

All right, I'm just trying to remember if there were other pieces. Oh, the export picture? Let's go back to the home. 

You have two ways of exporting your data. Here's one up here, which is the export data CSV. They come out in CSV files. And then here is the other one here where you can export your information. Let's see if I can just-- I'm going to do a consortium because when you do it by state, it exports an awful lot of information. [INAUDIBLE] 

And so if I wanted to do a consortium-- sorry. Riverside you always pop up first, so you tend to be my go to. And then you can select an institution or not. And then you would just click on your view your export and see how it turns out. And you would hear [INAUDIBLE]. 

So you would go up to all metrics or just the scorecard metrics, and I would export my data. And it just downloads. May not be able to see that. But then it downloads [INAUDIBLE]. It will download as a file that you can then manipulate to answer any of your questions that you need. 

Then if I go back, takes me back here. I wanted to show you where the metric definition dictionary is. That would be up here. And you would just click on that, and it'll take you out. 

And then as well the click to view resources. I just noticed I'm over time. My apologies. But if you just click here to view the resources, it will take you to the adult education pipeline resource library and it's searchable. It takes a little bit. It's a little slow on my computer to come up. 

And so you can go through here, and you can browse all of your resources. Oh, thank you. I'm not out of time. Super. 

So this would be the how to guide. And then there's an how to CASAS, the questions that we always are requested. People ask us. We went ahead and made some resources on that. We also did some webinars on them. 

The CAEP archive webinars are a great resource. If you want to go back and revisit them, we did two webinars on the how to CASAS COMIS data align. The first one was a fairly formal one and then the second one was more of what we called our fireside chat where we just answered questions and had conversation with people in the field who attended. And you can flip through there, or you can browse the resources, or you can search resources as well. 

So I'm going to go back to the PowerPoint. And then we can-- just wanted to have some time for questions or discussions. Or if you would like us to go back-- and I can show you anything you want in the pipeline live if you are curious about that. 

RANDY TILLERY: In response to Marina's question-- so one of the things that we do a lot with the data is we do provide targeted technical assistance or kind of rapid response work with consortia or the consortium members just answering questions. Two years ago, before, well, I guess BC, before COVID, we did a whole series of training institutes across the state where we sat with consortia from across the region and actually had them do exploration of their own data. 

And obviously, there's always limitations of time and energy related to those kinds of things. But these are the kinds of things we love to try and accommodate whenever possible. We love helping people doing data walks, doing exploration, doing equity analysis. We've had requests to actually do some dedicated consortium work with the fact sheet dashboards that we actually unveiled yesterday. So please, feel free to reach out to Blaire and I if you have questions or you want us to actually spend a little more time with you to explore your data. 

BLAIRE TOSO: Yeah, thanks, Randy. I mean, an example of this-- a lot of times, we get a question that appears to be relatively simple. And then because it also requires us to go in and look at your data, we'll often end up setting up a time to meet with you virtually, obviously right now, but we'll meet with you, and we'll walk through some of your reports or questions on your data or how you might want to think about coding particular pieces, how you might want to work across your consortium and tie different outcomes together. 

So we provide a vast array of resources and support to the field. It's really a matter of reaching out to us. And we'll be happy to work with you to meet your needs. 

We have time. As Veronica probably know is I am usually over time. So Randy and I were very attentive to making sure we scaled within the time frame that we have. So you have a good 15 minutes if you have other questions. 

RANDY TILLERY: Although, we do know some of you have been on Zoom since like 8 o'clock this morning. So we totally understand that people are overwhelmed, after this presentation in particular. 

BLAIRE TOSO: Well, they were in NOVA earlier. So I imagine that it all feels a bit over the top with the information overload. And we will be doing another walkthrough on the AEP much more detailed into how you navigate and some additional features and how you might want to use it for program improvement. That will be coming up at the summit. So there are some other opportunities as well. 

RANDY TILLERY: Right, so Beth Carter has a question. And it's about the ABE ESL transitions to post-secondary, whether it's Adult School GED or diploma program. What we do is we look for an initial enrollment in ABE or ESL, and then we look for a subsequent enrollment into the GED or diploma program which we should be able to see in the data that we receive. 

Whether it's from coughs or whether it's a subsequent enrollment actually is from the MIS file. So we automatically compare both records, looking for an initial enrollment and then a secondary enrollment. So we do or should capture that. If, for any reason, you look at your data and you're not seeing it, then we would invite you to contact us so we can sort of talk it through and figure out if there's an issue. 

BLAIRE TOSO: All right, then I'm going to move to the thank you slide. Our contact information-- yeah? 

RANDY TILLERY: I know I have friends out there. I'm really tempted to call on one. But I probably won't do that to anybody. Go ahead, Blaire. 

BLAIRE TOSO: No, I was just going to say, our contact information is there. As Randy said, please email us. And yes, to when I was talking about why we update the dashboard every year, and there were the coding errors or there might be a glitch or we haven't refined a definition-- those are the things that we find out oftentimes via you all in the field. 

We do a lot of user testing, we do testing before the data goes out. But as you can probably guess, there is a lot of data here and a lot of metrics. And so any time you will notice something that looks a little odd, feel free to call us or to touch base with us because we will definitely investigate it with you and either correct it if we can or make a note of it. 

And it will go into our file on things that need to be updated, or we will have discussion with other people-- the CEO, CDE-- and figure out how we might want to move forward with this. It is not a sole process, it is not something that westhead makes decisions about. This is really a collaborative effort. 

We work with CASAS, we work with COMIS, we work with the Chancellor's Office, we work with CDE, and we work with you all in the field. It's a really broad collective of people who inform the dashboard, as it should be. It's meant to serve multiple purposes, and we want to be responsive to you all in the field and responsive to legislative needs and identifying where adult education can both better serve adult learners and then to really push forward on those pieces that are being successful. And the adult education pipeline is one of the ways to do that. 

RANDY TILLERY: So Veronica, if there aren't-- go ahead, Blaire. 

BLAIRE TOSO: I was just say, Veronica, if there are no other questions or if you all are good, I think that people are probably ready for lunch. And we can give you 13 minutes extra to 100 times. 

VERONICA PARKER: Yay! Thank you. And thank you, Randy and Blaire. We definitely appreciate your presentation and your participation in today's session as well as yesterday's session. So thank you so much. 

Yes, that is correct. So we will now take a break. We are scheduled to take a break from until 1 o'clock. And then we'll come back at 1 o'clock for some lunchtime networking. And we will break you all into breakout rooms so that you all can have discussions amongst yourselves. 

We have compiled a series of questions. We always take an opportunity to identify what your needs are, especially for those of you who are newer consortium leads. We always want to stay in tune with what your needs are so that as we are preparing and planning our upcoming professional learning opportunities, they are aligned to your needs. 

So at this time, we will break and come back at 1 o'clock. And then we'll carry on with that lunchtime networking opportunities. So please, get up, take a break, move around, replenish water and food, et cetera, and then we'll see you all back here at 1 o'clock. So see you all soon. 

BLAIRE TOSO: Thanks for the opportunity, Veronica. Appreciate it. 

VERONICA PARKER: You're welcome. Thank you. 

BLAIRE TOSO: Bye-bye. 

VERONICA PARKER: Bye. All right, see you all at 1:00. Oh, and feel free to stay logged in. You do not have to log out. You can just, as you all are, off camera, just step away and come back at 1:00. 

JAY: Like CTE, it's long term. Like CTE, it's occupation specific. It's just a different delivery model. However, CTE obviously tends to be more classroom or lab oriented whereas pre-apprenticeship, of course, is more on the job training directly through an employee or through a union where all the learning is taking place at the workplace. 

And then workforce prep-- this is one that I'll get to set that workforce prep-- this is one that's kind of morphed over the years. It wasn't official a CAEP program until it was. But that's usually short term and not occupation specific. So if you're doing things like Microsoft Office skills or business skills or maybe basic first aid and CPR, sometimes if it's more than just a one hour training or something, sometimes those kind of things fit under Workforce prep. 

OK, that's a good question, Jennie. The official answer is there really isn't a good answer. Nobody really does a very good job of clarifying that. The one number I'll give you though is 48 hours. 

48 hours, to be clear, is absolutely not the official answer. Though I will say, that's what the colleges use, I think, in there-- I don't know if it's MIS or just the overall system. But colleges use that to distinguish CTE and short term CTE. Again, that's just a number they use. It's definitely not official across the board. 

At the federal level, they do a notoriously lousy job of that kind of stuff, I got to say. But 48 hours is what I'll throw out there just because it's the one number I know. But I make a big fuss about why nobody says numbers is because like if at your agency you have CTE and short term CTE, and you were to push back and say, hey, wait a minute, we've always used 70 hours, that's just the way we roll at our college or our school, I'd go way out of my way to say, no, you're not wrong, you have a lot of flexibility and really determining that locally. 48 is just the one hour I know. 

So I'll stop here. Thank you for that question, Jennie. Otherwise, I haven't heard a lot from people here. So let's just sanity check-- if nothing else, sanity check with the group here just because I'm not sure I have the warm and cozy that I'm doing much other than talking to a wall at this point. So do you mind humoring me and just reassuring me that that's not what's happening? 

OK, all good. Thank you, Jennie. You, of course, were the one person that I could count on that wasn't in that category because you asked the savvy question and so on. OK, then there go. Thank you. 

Just want to see. Yeah, so that's a good question, Carla, is 7 is what TE says, but 5 is what we say for official state reporting to the legislature. And this is also where I'm grazing a little bit here. As some of you might remember, when Neil and I did the trainings in 2019, that is not the first round that we did. 

But we were two or three years into CAEP and had that reset button that we hit that started at the Summer Institute there in Garden Grove there a couple of years ago. And we did around there in 2019 where there were kind of some updates and kind of some tightening up. This was part of that tightening up, Carla, is from a state level point of view, we tightened it up to 5 programs. 

Well, I think it's for all families. Certainly, if your consortium says we really want to make it about underserved families, my hunch is most people would say, great, by all means, that sounds like a noble effort. But I don't think there's anything in the legislation that requires that. 

So I'll back up to, again, the title does go out of its way to say supporting K-12 success. So that's a program that I have to admit has really undergone very little state level scrutiny or encouragement. You decide whether the glass is half empty or half full there. But the scrutiny has been very low, so has the encouragement, so to speak. 

The legislation purposefully angles it to working on helping students with their work in K-12. That's an on purpose. So somehow, it should relate to helping students succeed in K-12, but it can be any students. It doesn't necessarily have to be students in any particular economic or ethnic category. 

Does that answer your question, Greg? Sorry, I went a little overboard there. I will say, though, that if this is part of your question, there was a deliberate decision back there five, six years ago when they did all this to not just make it parenting. 

There was a feeling that it should be about education and not teaching, quote unquote, parenting skills. I will say I do remember there was quite a bit of discussion about that back in 2014, '15, when this was all laid out. But I don't think it was necessarily targeting a population. Go ahead. Go ahead, please. 

SPEAKER 1: I'm sorry, Jim. I'm coming on. Yeah, I'm all brand new to our Adult School here. But this conversation has come up-- one of the issues was, how do we do the intake testing. It's not necessarily CASAS. So how would we measure our parent's growth over the coursework? 

JAY: Well, when you say-- so are you talking about it for this specific program or kind of for all K programs? 

SPEAKER 1: No, no, no, just for the parent and a child's success program-- 

JAY: Wow, OK. That's an outstanding question. I mean it positively-- did you do that on purpose? Because you kind of hit the nail on the head with the sort of things, I would say, has received very little state scrutiny and/or very little state support depending on whether you want the glass to be half full or half empty, so to speak. 

So with this particular program, I don't think there's any official answers, period. My default answer would be the students that are in this program frequently tend to be similar to students that are in things like ESL. Not always, but frequently. 

So my default answer is to probably treat these students like ABE ESL and do CASAS testing and measure literacy skills. And with that, to the maximum extent possible, a lot of students in this program are likely to also be enrolled in ABE or ESL. I'm not saying that's a requirement or necessarily true at all places. But I do believe-- 

Yeah, Carla. I think that's kind of feeding into what I'm saying that anecdotally, not officially, but anecdotally, a lot of K-12 success students are also in ESL. So a lot of that would be served by doing the same classes testing for that program as you're doing for ESL. 

So that said, I will reiterate-- you pointed out CASAS testing is absolutely not required for K-12 success. But there are no tests at the state level or anywhere else that I know about that really serve as the, quote unquote, "Test for that program." If you feel like you're developing your K-12 success program to be something related to workforce, you might do something related to CASAS testing kind of like what CTE programs do where you do that CASAS appraisal, but not necessarily pre and post testing. That would be another suggestion I might give. 

I'm not sure how that relates to K-12 success. But if you're feeling like you want to be able to get a read on that student's basic level of literacy, you could give the appraisal, which is the CASAS or locator, which is the CASAS form that has some hard items and easy items and everything in between. It gives you that ballpark figure, but not necessarily the detailed score like a pre-test or post-test. 

Sorry, that was an overly long answer. But it was a good question. You made me think a little bit with that answer. 

SPEAKER 1: I appreciate it. Thank you. I'll stay on later with you. 

JAY: OK. 

SPEAKER 1: OK. 

JAY: But if you're interested, I definitely would be happy to talk it out with you. Back when Neil and I did the trainings, we did, to some extent, sometimes kind of search out people that maybe we're implementing the program. I got to say, we never really got takers. 

Not that it's really not all you guys, but that program really, in my opinion, never really did get all that fleshed out. Here we are five years later. I would still have to say that's true. 

OK, so a little bit more on structure. Workforce reentry used to be a program up to a couple of years ago. Now it's workforce prep. When it was called reentry, it tended to focus on older adults. It doesn't anymore. 

But workforce prep-- a lot of you have said over the last five years that sometimes you have workforce related courses that are not long term occupation specific CTE. You're doing a lot of that workforce readiness that doesn't quite rise to the level of CTE. But hey, those students are doing something, not nothing. So you really ought to document the good work they're doing. 

So over time, that's morphed. A couple of years ago, we made it official, so to speak, and we just call it the workforce preparation program. Adult served includes three categories. So a couple of years ago, we started talking about it. That was the part of the presentation that nobody paid attention to. 

I'll just say now, that's the part that everybody wants to pay a lot of attention to. You've probably heard from people like Veronica and Neal over the last couple of days about all the work you're going to do in NOVA. And you have those three buckets, that is you've got the participants with 12 or more hours, you've got the reportable individuals that have at least one hour, and you've got the adult served which can be 0 hours. 

So there's lots of issues now where you're looking at how many students fall in each of those three buckets. Obviously, you're looking at maximizing participants and minimizing adults served. But that, we talked about a few years ago is what's coming. Now with all that stuff you've heard about in NOVA, I would say that it's arrived. 

A little bit on learners and multiple programs that's probably a little bit dated-- we track services. But if they're doing services only, that's not instructional hours. A little TMI, but a few years ago, that was a big CAEP conundrum. Some consortia were counting those that service time as hours, others weren't. It was about a 50-50 split. 

For a summer there, we were kind of arguing back and forth for a while. The final answer was services are a really good thing to be providing, but they do not count as instructional hours. Instructional hours have to be actual instruction. 

Record barriers that intake-- that's the field called barriers to employment. That's something that I think everybody understands as second nature now. But it took a few years to get everybody conditioned to always do barriers to employment. I think it is kind of second nature now. 

And then a little bit on those federal measurable skill gains. Over the last couple of years, the feds have made changes. We've really tried hard in CAEP land to have our outcomes match what the feds have for WIOA reporting. Not all but many of you are at WIOA Title II agencies. 

So the federal program is obviously very well laid out. It's been that way for a very long time. So when California came up with an adult Ed program, kind of the obvious no brainer was, hey, we've got requirements at the federal level that everybody's already kind of needs to comply with. So if we're going to do our own program, we're not going to, quote unquote, "reinvent the wheel." We're going to structure it the same way the feds require everybody to structure their programs already obviously because we wanted the alignment here to be within the real world and have it be relate to what everybody else was doing and so on. 

OK, so hours by program. I think you've probably heard a lot about this today and probably the last two or three days. Again, in NOVA, you've been talking about how you're going to need to do those hours. This slide actually relates to divvying it up. 

If you've got learners in more than one program, I don't think this is quite as big a deal with the new reporting as the old. But we have lots of learners that are in more than one program. So what we said a couple of years ago is if you've got students that will just say, hey, they're enrolled in ESL, they're also in CTE, the default answer is it will be equally split 50-50. 

So at some consortia, that issue really doesn't matter at all. But at some local areas, that's of great importance. Sometimes it's because you're really being careful to allocate between college and K-12 or maybe among K-12. Or maybe it's not among different agencies at all, it's just making sure that your ESL and CTE and ABE are all divvied equally. 

So just know in TE, If you have a student in more than one program and you just leave it that way, TE is always just going to assume it's an equitable 50-50 split. If that's what you intend, then you need to do nothing. TE will take care of it for you. 

If you know in reality, it's not so even, maybe in reality, you know it's really 75% ESL and 25% Career Tac Ed, that's fine. But then just know you need to go out of your way to create two separate classes, make sure you allot those hours, 25% CTE, 75% ESL in my example. You've got to make sure you do that extra work to keep it all separated class by class and so on. 

OK, and then this is what we talked about with hours, sticking with instructional hours. Again, this was a big issue a couple of years ago. I think the dust has settled on this one. But a couple of years ago, there was a big debate about what to do with those short term services. 

The short answer is we've doubled down, and we strongly encourage you to report short term services. They'll be a little bit on this here as we get later in the presentation. So yes, please do record short term services. But please refrain from counting any of that as hours. 

The hours should be instructional hours, that is, hours that actually takes place in CTE or in ESL and so on, not with the short term services. So again, when you're doing the hours in NOVA and so on, you're looking at the hours they spent in the classroom or-- obviously not literally the classroom. We do a lot of distance learning. So of course, it can be virtual or face to face, but it's going to be the hours that are in those K programs, not in the short term services. 

And then this is a new report. If you were in the training on Tuesday, we mentioned this briefly. There's some fairly new reports in TE. These are reports we added back in May. One is called enrollees by hours, one is called services enrollees by hours. 

We develop these specifically thinking they would be useful for some of that reporting you need to do in NOVA. One of those big goals everybody has, whether you like it or not, is, again, maximizing the number with 12 or more hours, minimizing the number with less than 12 hours. So these reports are basically looking at those three buckets. 

The enrollees by hours is kind of doing it the way that I anyway think about first and foremost when I think of the concept. It's looking at the number who qualified there in that federal reporting section, the number who qualified for outcomes, the number who qualified for the services, and just looking at that total number and allocating, OK, of the learners you qualify for each of those three areas of CAEP reporting, here's how many with 12 hours, here's how many with 1 to 11 hours, here's how many with 0 hours. You can double click and drill down and get the list of students that appear in each one of these buckets. So this is the simple way when you get to figuring out where all your students are falling in all this, you can use this CAEP enrollees by hours to dig into more details and figure out how your students are doing. 

There's another one called service enrollees by hours. I think I have a year later that looks at it. But it breaks it down into those different types of services. So you can kind of look at it from a services only student's point of view. 

And then this one's getting a little bit wonky here, but this is outcome. We'll talk a little bit more about outcomes. This might be a little bit cart before the horse. But there's an outcome we have in CAEP land called occupational skills gain. 

I know we're all new. So if you've never heard about that outcome, that's fine. Hey, you're new. That's why you're hanging around and all that. But but that said, how many of you have heard about that before, how many of you are aware of CAEP outcome that we call occupational skills gain? I'm not asking you to give a detailed definition, but you just know it exists. Have any of you heard of occupational skills gain before-- yes or no? 

OK, no. Thank you for being honest. I know you're there, Jennie. How about everybody else? Just curious. 

So I'm playing musical slides because for whatever reason, that little sidebar keeps kind of going out on me for whatever reason. So mostly no. So anyway, I'm going to get into stuff here that's not necessarily on the slide. So just be forewarned. 

But occupational skills gain is one that we kind of carved out for CAEP reporting. It was set up for CTE, but it might apply to things like workforce prep. It could potentially apply to things like K-12 success as well. But it's for those CTE related programs. 

But the idea is your question, Greg, about testing, hey, some programs require CASAS testing. Others don't. So you might say, for the ones that do, that is ABE, ASE, and ESL, that partial progress is easy to measure. Hey, that's what we have CASAS testing for. 

But for the CTE related programs, that is, CTE and apprenticeship and workforce prep, there isn't an obvious way to do that. So backing up a step, when we're looking at these meaningful outcomes, we have what are called performance indicators and measurable skill gain. Performance indicators are what you might call the tangible really obvious to measure outcomes. Those are things like get a job, get a pay raise, get an occupational license, earn your diploma, earn your GED and so on. That is tangible outcomes that you can easily hang your hat on, you've got something that really shows that-- hey, we know that adult Ed was worthwhile because we got our GED, we know that adult Ed was worthwhile because I got my automotive license et cetera et cetera. 

But we know that if we rely on those tangible outcomes to measure our students, we're way more likely to fail than we are to succeed because it's going to make our students look bad. We get a lot of students that earn a GED every year. But probably the number that still have progress is always going to outnumber the number that finishes diplomas, same thing, occupational license, same thing, get a job, same thing, where a lot of times, it takes time to get the job or it takes time to finish the diploma, whatever. 

So we want more incremental ways of being able to measure progress, we want more incremental ways of being able to say strongly for sure why is that our students are doing a really good job. So back to ABE, ASE, ESL and those programs-- hey, you're hearing from the CASAS guys of course I'm going to say, we have a really, really, really great foolproof way of reporting that incremental progress for ESL, ABE, ASE. It's called CASAS pre and post testing. 

That's what we've been doing for years, that's what allows us to show that kind of information, hey, the student still has a long way to go before they finish their diploma. However, they did a bang up job on their costs pre and post. So we know for sure that our student is learning, we know for sure that our student is doing a really good job. 

So I'm going to pause here because I went crazy there. Is everybody with me or was that 5 minutes you'll never get back with that crazy explanation? Sorry, I got to know before I move on. OK, thank you. A few of you are humoring me anyway. OK, thank you. 

So I'll just say back to occupational skills gain. That was what we looked at five years ago when we started CAEP is we needed to have ways to measure that incremental progress for things like CTE and workforce prep just like we do for ABE, ASE, ESL. We came up with these outcomes called workforce prep milestone and occupational skills gain that is ways where you can show partial completion of a workforce program without necessarily getting to the finish line, so to speak. 

So we aligned it to these other measurable skills gains that they use in WIOA Title I or WIOA Title III. We really try to show that alignment so we could basically have something to hang our hat on and making these official outcomes and making it as part of our official reporting. 

Hold on to that, Annabelle. I'm going to get into this here in a couple of minutes. If it doesn't help, then we can kind of pick it up again. 

So I'm kind of over viewing, and then I'm going to be at more specific outcomes in about 5 minutes. So if you can hold it for a minute. But one last topic I wanted to cover here in the overview-- we've got some targets here. 

So I'm not sure these were presented either. Or I talked to Neal about this earlier today. He didn't cover this in his presentation. So we agreed it would probably be good to at least have this. And somebody say, you're getting it in mind. 

So you've been hearing about these goal setting targets. These are what you're going to need to do. There's consortium level metrics that everybody's going to have to do that's going to be based on how your whole consortium is doing. 

You've got mandatory metrics for all members, and then you've got the longer lists that are optional ones that you can do as members. I'm not sure. Maybe, Veronica, you know. I know that you're going to have to do some of those optional metrics, but you're definitely not going to do all of them. 

Has that been decided yet how many each agency is going to have to do? That part, I'm not really sure about. But anyway, there's going to be mandatory consortium level one's mandatory member level. And then most of them are on this bigger list that will be optional. 

So consortium level-- you're going to be reporting student barriers. There's four student barriers. Low income is one, English language, low literacy-- there's another one that I'm forgetting. I think it's long term unemployed. But anyway, you're going to get a list of like four barriers, and you're going to basically show that you're serving students in those barriers. 

And then you're also going to be looking at your overall number of reportable individuals, that is everybody that basically makes it in CAEP plan, you're going to be looking to expanding the pool of reportable individuals. And then at each member level, you'll be looking at moving those reportable individuals into becoming participants. I'm not quite sure why the first one is at the consortium level and this one is at the member level. But again, we've got a large number of students that might qualify somehow, some way, for CAEP. 

But what we're looking at is moving them as much as possible into that participant's bucket, that is, those with 12 or more hours, that is, those that actually enroll in a class and are enrolled in one of CAEP programs. And then there's a metric that I'm guessing Neal talked about. But I'm not sure where he's come up with the little metric, that is, the amount of money per student that it costs at the school or at the college you'll be basically looking at that percentage of funds that you're expending year by year. 

And then here's the optional ones. This is where yahoos like me get a little bit more involved probably. But you can report out on number of diplomas or HSC, number that are in a post-secondary, that is, that occupational license or occupational certificate. You can do the ones who made the pre post-games, transition to CTE, transition to 4 credit coursework, the employment metrics. 

It's looking at it from that Federal Table Point of view, which is going to make it pretty impossible to measure. But that's just the way it is for now. And then the median change and earnings-- again that's aligned to table five, which is probably more to our detriment. And then that immigrant integration milestone, that is, that I3 outcome-- there is an option for that as well. 

So here, I'm getting back to alignment. I'm going to stop right there. I kind of breezed through that. So everybody hanging in there? Any questions at all? 

I still have your question in mind, Annabelle. I haven't lost sight of it. We're transitioning to that section. No news is good news. I'm just going to move on. 

So anyway, this is a little bit about that long winded explanation I gave 5 or 10 minutes ago. Again, we've got what the feds call performance indicators and what the feds call measurable skill gains. So this is just proof in the pudding that when we developed our outcomes for CAEP there around 2014, 2015 or so, this is showing how we didn't just make them up, we didn't just throw darts at a wall, we went way out of our way to have our outcomes be directly things that the feds had already defined as a priority, that the feds had already kind of gone out of their way to require California and all other states to be reporting anyway. 

Some of them they call performance indicators which, if you ask me, are those ones that are kind of, again, outcomes that you can kind of hang your hat on. So you can see the two that we emphasize-- not surprisingly, employment and wages. The others are those measurable skill gains that is very important outcomes but not always things we can hang our hat on. That is things like literacy gains, that are gains that are achieved through things pre and post testing. 

And then those bottom two, that occupational skills gain concept, that is, the training milestone and skills progression. To be clear, the number 4 and five or the federal terms to this concept. But again, we've kind of retrofitted them for our own CAEP purposes. 

So over to the right, what happens simultaneous to this is on the California legislature had this legislation called AB104, no relation to the legislation you've heard about more recently. But the California State legislature went out of their way about six years ago to pass specific legislation that said, these will be the areas that all California Adult Ed agencies collect. You can see not so coincidentally it matches exactly what the feds had already laid out and WIOA legislation. The one exception I'll point out is that number six where the state legislature went out of its way to say, we need to report out on transitions. 

So here is our system here where we have these six categories. I won't rattle them all off. But we've got lots of different outcomes that we need to report in CAEP for all the different areas. So we're not going to go outcome by outcome because, again, that's an hour so. We don't have, but I will refer everybody to that CAEP Summit. 

I'm sure you've heard a ton about it, but there is going to be a session where I get a little bit more into the weeds here. So if you want to do that, you can sign up for that session at the CAEP Summit. Lots of really good sessions there, obviously. 

So here's the slide on occupational skills gain. So when you do that, back to your question, Annabelle, you can do it in a lot of different ways. What we say is for occupational skills gain, we're just saying somehow, some way, you're showing the student has accomplished a portion of a longer term program. If you're calling it that workforce prep milestone, that usually suggests full completion of something shorter term. 

So an example for occupational skills gain-- we'll just say they're in a welding program. I'm just making it up. We'll just say, it's five semesters long. And so in this case, we're saying, hey, the student finish or went in module 1, they did a great job, they're finishing module 1. That passage of an exam part, we're saying, yeah, somehow, some way they should pass some exam for them to get an occupational skills gain. 

Back to your question, if you're doing CASAS testing, well, you're hearing from the CASAS guy. So obviously, I'll say, yes, that's not the required way to do it, but that's certainly a good way to do it. So one way to make the passage of an exam valid would be to just CASAS test the person. You can do pre and post testing or you could just do that appraisal to show that they have the basic literacy skills needed to move on to the next module. 

You could do your own homespun written test if you want. Some people in career tech and might say, doing written tests is not the best way to go, doing skills checks makes a lot more sense. That too is fine. You can do whatever you need to do. For the exam part, you have flexibility as long as you do some kind of exam and have some way to show that the student did pass something in order for you to record that as that outcome. 

So that was half answering your question and half otherwise. But I'll just say, everyone, but especially Annabelle, I'll just go out of my way to point out in my mind to minds, that was the answer to your question right there. Well, in MIS, I don't know. In TE, if you were in that session-- and I'll send this again to Veronica and Mandele, but there's a PowerPoint that we just don't have time to do that I've called the Bubble Boy PowerPoint where I have some of those detailed PowerPoints that shows. 

I think I included it here. So here's that slide. So in here, we've got this. This is the color coding for TE So the 1 4 occupational skills gain is skills progression or that one called acquired workforce readiness skills. Either one of those directly translates into occupational skills gain. I'll just say there's a more detailed PowerPoint that I just don't have time to present that gets into that obnoxious level of detail where it goes bubble by bubble and kind of shows exactly what is what. 

So sticking around here again-- occupational skills gain, partial completion of a longer program, workforce prep for completion of something shorter term-- that falls into what we call post-secondary outcomes. So we would say, market is a post-secondary credential. That is, they complete their CNA, they complete their automotive specialists, they complete their welding license, whatever. That would be a completion of a longer term program, that would be what we would call postsecondary, that's what we mean when we say postsecondary outcomes. 

With under that though, we have those what we're calling literacy gains, quote unquote, that again would be that partial completion of a longer term program or full completion of something shorter term. It could be either one. But again, we're structuring CTE. So I'm just making this up on the fly. 

I'll stick with that welding example. I don't know if it's appropriate or not. But again, we'll just say our welding program is 5 semesters long. So when they finish module 5 of 5, that's when we want to mark that occupational licensure or certificate and credit the student with the post-secondary outcome. That's going to be that one that we can kind of hang our hat on. 

As they're moving toward that final licensure or final certificate though, we want to be able to record the fact that the student is doing a great job every step along the way. So that's where we might want to carve out and figure out ways to measure them for things like workforce prep milestone or occupational skills gain. So again, summarizing this up on the WIOA 2 program side or the academic program side because you're doing that testing. 

We find these incremental gains to be built in and real easy to measure and record. But on the CTE side, there really isn't anything systematic for that. That's why we've presented these slides as we did kind of didn't really make it up. We've retrofitted, but we felt like we needed to put something in place so we could record this for CTE students just like we do for ABE, ESL. 

The last outcome I wanted to get into a little bit here is that transition. This is the one that doesn't directly relate to WIOA. But what the legislature says is we're also tracking students that transition from adult Ed to CTE or ones that transition from adult Ed to 4 credit college. 

So for a while, we used to rattle it off with bullets and nobody understood it. When we moved to this little graphic, people tended to understand it a little bit better. So on the left, those two blue boxes represent the two possible starting points, the red boxes to the right represent the three possible finishing points. So the starting points are either K-12 adult Ed, ABE, ASE, ESL, or non-credit community college ABE, ASE, or ESL. 

To do the transition to college, they start either one of those places and move to 4 credit community college. Again, that 4 credit is mandatory. If they go to K-12 adult Ed to non-credit college, that would be considered a, quote unquote, "Lateral transfer." So that doesn't really count for this outcome. 

The second one would be if they're in K-12 adult Ed or non-credit community college, and they transition to adult Ed CTE or community college CTE. That would be that transition to CTE. So that's another one. 

I'll just say, there are way more outcomes in what I've described. But over time, the ones that have been the steepest learning curve are the occupational skills gain in these transition outcomes. So these are the ones that I always go way out of my way to detail a little bit more than the others. 

Back to a couple of the questions. This is the infamous bubble boy slide. We've been showing this for five years. This is a screenshot of our CASAS update record. Not that many people use it anymore, but this is what you use for that bubbling and scanning. 

A few of you do that, not many. But we like to use this just to show what all the outcomes are because for training and for visual graphics, it works great still. So you can see we color-coded those six areas of AB104, the six types of cap outcomes. And we just color coded it to the update record just for simplicity's sake. 

You can see we color coded all the employment outcomes in blue. So if you look there to the left, not so surprisingly, get a job, retain a job, and/or military-- those are the three bubbles we have on the form that directly translate to employment outcomes. You can see again, we use the five other colors to equate to those five other areas of report. 

If you're more of a hand entering TE person, this next slide has the exact same information as the one I just showed you, only it's looking at a screenshot inside TE software instead of the update form and it's providing letter codes instead of color codes. But again, it's taking those same six categories, providing a letter for each category. The purpose is exactly the same. It's just showing you which of these outcomes in TE directly correspond to a CAEP outcome and obviously more specifically, which particular outcome does it equate. 

OK, so out of outcomes and into services. Here is another TE screenshot of those short term services. In TE, we just have the same categories they have for WIOA reporting. In WIOA land, they have supportive services, they have training services, and they have transition services. So we just organize it in those same three categories that the feds have. 

This is a screenshot straight from TE. Supportive services, broadly speaking, are the short term services that work at helping an individual with their personal issues-- legal counseling, fiscal counseling, transportation, child care-- stuff like that. So the idea of supportive services is providing those short term counseling opportunities or other services that address the student's personal needs with the idea that sometimes, students might have personal issues that are getting in the way of them succeeding an instruction. 

Until we are able to work with the student and solving their personal issues, we really have no business expecting that student to succeed in class. So we provide the supportive services to help the student address personal needs so they are better equipped to excel in the classroom and so on. The other hand, the training and transition services relate more to the short term trainings or counseling opportunities. 

You might be providing at the Adult School or college training. You can see all kinds of different types of short term training you might provide. Transition services are the same sort of thing, but they tend to directly relate to getting the student into college and/or directly getting the student into a vocational program. 

So this might be a little dated, but we had a lot in the last year about a lot of the short term services we've been providing under COVID. So here's just a list, a lot of additional health care needs, job search needs that we've been providing in COVID. Last year, there were a lot of questions about how we might mark it. This is just for reference for that. 

Here's another example or another list of examples of services that a lot of agencies and consortia have been reporting on lately. And then here's a summary of those three different categories of services with some examples of the sort of things you might be doing at your agency that might fit under those three different categories of short term services. So getting in to some of the basics here-- we have the CAEP data dictionary. 

So how many of you have actually seen the data dictionary on our CASAS website? Have any of you-- OK, Kim has. OK, a lot of you have. OK, great. So you've been there before. 

You can see at the top, that gives you the link that's to our CASAS California accountability page. We have our WIOA 2 data dictionary and our data dictionary in the same place. A lot of you have probably been to WIOA trainings. We have that summary of changes, we have a detailed document on short term services, a detailed document on barriers to employment, we have our statewide and local assessment policy, a lot of information on things like virtual learning and remote testing. So everything you wanted to learn and more on this page. 

Moving into TE-- we've got that Quarterly Data Submission Wizard. Last year, we had a lot of fanfare where we moved out of you actually saving your data, we moved out of you submitting that Data Integrity Report, you use that simple wizard, every quarter you run that Wizard and TE, it does your WIOA 2 and your CAEP reporting all in one fell swoop, it includes that Data Integrity Report with your actual data. 

Well, it depends a little bit. You can self report them as getting a job in TE. So for showing up on the CAEP summary, I would just say, yeah, report they got a job and that will show up. I'll just say in TE, there is a bubble called got a job. It's totally reliant upon self report. But we did decide that five years ago or so to say. 

Yeah, it's not verifiable. But we do feel like it's definitely in the something is always better than nothing. So if your student says, yippee, I got a job, yes, please do market. And it will show up on the CAEP summary. 

That said, it will not necessarily show up until you do that data matching for federal reporting when we're doing things like that employment and earnings survey. So for WIOA 2, you need to do that follow up. But for the simple CAEP reporting, you can self report and it will count. 

Anyway, back to this. Use that data Submission Wizard every quarter and that will take care of your CAEP and WIOA 2 quarterly reporting. And then starting last year, we started with the employment and earnings survey. So it's that same Wizard we have for WIOA 2 agencies. 

I'll just say, you run this every quarter. It automatically sends student surveys by email, by text, or both. We've had a lot of training in the last year on how you can customize it. So for now, I'm just giving you the link. It basically goes to a different page on our website. 

We've got help documents for each quarter. We've made four different documents instead of one uniform document with the idea that the directions each quarter are slightly different, we've found that if we just have separate directions for each quarter, that works way better than having one uniform set of directions. So if you go to this page, depending on the quarter it is, you can download one simple set of directions and just follow that and follow the Wizard click by click. If you use the directions on this web page, it's a lot easier. 

I'm not sure what you mean by that. I'm not sure what-- you mean do you need to have documentation that they really and truly got a job? I mean, that might be good. 

I got to say, the chances of the CAEP data police sniffing that out are somewhere between slim and none. So if you're not doing that, I got to say highly, highly unlikely that you're ever going to get in trouble. I mean, that's great if you're doing that. But insert all your jokes about the CAEP data police here. That's a really good example of one that the CAEP data police probably are unlikely to find. 

Just like Annabelle's question about are we going to get you in trouble if we find you did it CASAS test for occupational skills gain, that's another one where yeah, the data police probably ain't going to get you. But if you have something uniform where you're doing that, I'd say certainly all the power to you. So here's the data submission calendar. The CAEP calendar matches WIOA 2 in every area except one, that is, the end of Year. 

Four the quarterly deadlines, CAEP and WIOA 2 are the same. CAEP gives you a little bit more time than WIOA does. WIOA, require you to get it in by July 15th. CAEP doesn't require you to finish any of that up until August 1. And then remote testing-- this seems like so 2020 now, but we still have lots of information on our CASAS website about that. 

So here's a link to our remote testing website if you're still using remote testing in response to COVID and all that. Immigrant integration-- we've got that new I3 column. We also have those I3 reports. I want to move to CAEP summary. I'm going to talk about this on the DIR, and then I'll whisk through the other stuff because as usual, I had way too much. I thought I was going to be simpler today, but I drifted off as usual. 

But anyway, the CAEP summary-- have you all used the CAEP summary in TE before-- yes or no? This is the one thing that you-- OK, you're all saying yes. So a lot of you have probably heard this. But this is the basic report. 

It does just what it says. It summarizes all of your CAEP outcomes and CAEP services on one handy dandy report. I like to say left-hand section, middle section, right-hand section, high bar to the left, middle bar in the middle, low bar to the right. Literacy gains is directly from the NRS federal reports. It requires everything federal reporting does, it requires all the demographics, it requires 12 hours of instruction, it requires that qualifying pre-test. 

The middle section are all those other outcomes. It's basically all of the outcomes you have under CAEP reporting except for pre and post testing. So to get any CAEP outcomes, that requires almost everything that literacy gains section does, except that CAEP outcomes goes out of its way to exclude pre and post testing. So for that reason, it requires everything the federal reports require, except it goes out of its way to not require pre and post testing. 

It does require 12 hours, it does require all those demographics, but it does not require anything related to CASAS testing. That right-hand section for services basically just requires a valid student ID and nothing else. So if you have a student that maybe has some data in there and the data relates to CAEP somehow, some way, but they don't necessarily have the hours or they don't necessarily have all the demographics or maybe they've got data, but you never really enrolled them in a class or program, they might show up in that right-handed services section. 

So a little bit here on the CAEP DIR, and then I'll kind of wrap it up as we have that CAEP-- well, it depends on-- OK, so hang on. So I would say, in general, you're probably looking at column E, that is, your official enrollment are going to be those with at least 12 hours. 

I'll just say, though, back to those three buckets-- for some of those deliverables, we're looking at adult serve. So that would mean column M. For some, we're looking at participants. That would be column E. 

So I've got to say, not really what you want to hear. But the official answer, I got to say, it depends. And it depends specifically on which of those items in NOVA you're measuring yourself on. Some of that stuff in NOVA looks like column E to me. There are some things though where they are looking at column M where they are specifically looking at adult served. 

Sorry, that's not the answer you want. But I do think that's kind of it. The good news, though, is that prior to this year without all that in NOVA, probably the official CAEP answer was something like, yeah, that's up to you as a consortium. And yeah, a lot of your consortia have the same sort of arguments you're talking about. 

Annabelle, you're in a big club, obviously facilitating those sorts of arguments. Right, it's a services only. So M is going to be people that may not have any class enrollment, they may not have any official CAEP program enrollment, obviously they don't have the 12 hours, they are frequently not going to have all those required demographics. So M is going to be all your dirty data, so to speak. If you want a cleaner read, you definitely look at E where it's requiring the class and program enrollment, it's requiring the 12 or more hours, and it's requiring things like gender and race and ethnicity and date of birth and all that. 

B and E-- again, B is directly from the NRS tables. So it's requiring that qualifying pre-test just like NRS table 4. Cape outcomes, on the other hand, are all those outcomes outside of pre post testing. So for obvious reasons, it kind of goes out of its way to exclude the part about pre and post testing. 

So CAEP out comes as almost like the nurse tables, except, like I just said, it goes out of its way to not require anything related to classes pre and post testing, okeydoke? All right, so good questions. So back to the CAEP DIR. 

I'll just say, again, it starts at that services section with the universe of everybody. But then it's scaling back and eliminating those that do not have the enrolled and CAEP programs. So what it's doing at the top is it's coming up with that uniform denominator for all 27 items on the report. 

So I just say, use that summary information at the top. Make sure that that looks good to you. If you've got everything good at the top of the CAEP DIR, then moving on with these individual items is really easy to do. If everything, on the other hand, is messed up at the top, then your data down below is probably going to be met. 

Yeah, you need the student ID no matter what. No student ID means no nothing. Yeah, you need the student ID, but that's really all you need. I'll just say, to really to get into TE, you're not going very far at all with TE without a student ID, even more super TE basics here. But the student ID in short is the hub of the TE universe. You really can't have a student record at all without that student ID. 

So we've got the 27 different items. We don't have time to dig into the nuance of the CAEP DIR. We'll schedule some trainings on CAEP DIR later where we'll give an hour for just this. But again, it's looking at demographics, it's looking at CASAS testing, it's looking at hours of instruction, it's looking at a little bit of everything to look at how well you're doing in terms of all these reporting elements. 

And then again, we've got these new enrollees by hours reports that looks at those buckets that should be super helpful for NOVA reporting. This is just a more detailed look so you can see what it is we're talking about. Here's that service's enrollees where it's doing the same thing I just said, but it's breaking it down into those services categories. 

So a lot of times, when you're trying to manage the 3 buckets, in most scenarios, you're probably micromanaging that section of the CAEP summary that relates to services like a lot of you have been correctly suggesting. So this is looking at it from the services section point of view where it's saying, all right, we have way too many in the services section. So let's just drill down in that services section and look at what specific services they're getting. 

This is a real good way for you to figure out, OK, in the services section, how many of those students really are services only students where they're there because all they did is receive services versus other students that might have been there because they have incomplete data, and we really need to fix some things related to their data so they get more demographics or get more hours and get them into the participants section so to speak. Past I3 outcome-- this is the last thing I'll talk about. This was new at the beginning of last year. This is that immigrant immigration that you've heard a little bit about. 

But the long story short is a lot of you are doing EL Civics that when you have students pass those EL Civics co APS, we've got a lot of data that shows that directly relates to outcomes and the area of immigrant immigration. Yes, you can. That really is up to you. We do have guidance that kind of relates what types of counseling we think fall into those three types. But yeah, ultimately, those are definitions you can decide not CAEP. 

The best guidance is that's a long winded. There is no official guidance. But I'll refer you to attachment S on that same page on our website that has the CAEP and WIOA 2 data dictionaries. There's an attachment S that gets into a little bit more detail and defines all of the services we have listed in TE. 

So last thing-- back to this past I3 that suggests that they passed to COAPPs the rules for CAEP are a lot looser than they are for WIOA 2 EL Civics. But this is just to show you that if you're doing El Civics, that's a good way to get some outcomes. You can show outcomes related to I3. 

And we've got the proof in the pudding here with that new past I3 outcome. It's now column F on the CAEP summary. And I'm not going to detail this either, but we've got those I3 reports that we released about a year ago that connect EL Civics co APS with different areas of immigrant integration. This is just a snippet here on the screen. 

We also have those consortium manager reports. If you haven't done that, you can email golive@casas.org. Here, I'll type that in. I should have put that on the slide. 

So that's a email where if you're a consortium manager, and you do not have access to the consortium manager reports, email that address, identify your name and give the full name of your consortium. If you think that nobody at your consortium has ever done this, you can indicate that in the email. Patricia, I think you're the person who replaced Branka at CAREC I'm picking on you just because I know that's a consortium where there is one that we've established for long time. 

Then you would email the same address, but just go out of your way to say, I know that our consortium has this. But the issue is I'm a new person, not necessarily new to this consortium level reporting overall. But what we have are consortium level reports. 

The good news is you can look at data for all agencies aggregated with your entire consortium. The bad news, which is a purposeful bad news-- you can drill down at the consortium level. But I'll just point out-- we have a few of these reports that you can access at the consortium level and look at it kind of 40,000 feet up if you think that would be helpful. A lot of you have done that and said, yes, it really is helpful. 

So again, I'll just show you. You can look at demographics, you can look at the CAEP summary, you can look at barriers to employment, you can also look at the Data Integrity. And again, I've run over as usual. So I'm going to start now because I'm already 10 minutes over. 

Hopefully, that gave you something instead of nothing. As usual, I'm just shoveling way too much on like normal. So sorry, I ran over. Sorry, I'm shoveling too much at. But that said, I'll be quiet. So I can be done. Rescue them from me, Mandilee, or Veronica, or they'll never get out of here. So cut me off at the knees so they can get out of here. 

VERONICA PARKER: You are all good, Jay. And actually, we were supposed to go till 3 o'clock. So you were-- 

JAY: Oh, I thought I was supposed to-- I thought I only had an hour. 

VERONICA PARKER: No, it's an hour and a half. 

JAY: Because I thought you said an hour. I went way out of my way to say that, actually. 

VERONICA PARKER: Oh, I'm so sorry. 

JAY: OK, because I thought I had an hour and a half Tuesday, but only an hour today. 

JAY: Yes, it was probably an error on my part. And I apologize for that. 

JAY: All right, everybody needs to break away. OK, sorry about that. I thought I needed to do that in an hour. 

VERONICA PARKER: No worries. Thank you so much, Jay. 

JAY: That'll be my excuse though on why I ran over. 

VERONICA PARKER: Yes, you are all good. But thank you. And thank you, everyone, for hanging out with us today. We really hope that this day of training, although was a lot of information, we hope that it was beneficial and that you all have learned something and are taking away some new understandings or increased knowledge of the different areas of CAP. 

We have shared several things in the chat, and one being the evaluation. Again, we're always looking for opportunities to improve and/or identify those areas where we could provide more professional learning opportunities to you all. So definitely feel free to complete that evaluation and let us know what you think. 

You do have access to that Google folder. And as I mentioned before, we will be adding more content to that Google folder so that you all will have the presentations from today. Once the recording is remediated, you'll have that as well and then other resources that we think will be beneficial to you. So you can expect to receive a follow up email as we're receiving those pieces of information. 

Thank you, Amanda Lee, for helping us today with facilitating today's workshop, and all of the other facilitators, and, again, you as attendees. So yes, if you do not have any other questions, we will prepare to close the room. 

And if you are registered for tomorrow's workshops, we have one at 8:30 AM. And it will be on goal setting and targets to continue to help you plan for the three year planning process. And then at 10:30, we will have the Community Asset MAP webinar. And that will conclude day five of the CAEP directors event week. 

So thank you all very much for your time and your participation this afternoon. And we look forward to seeing you all tomorrow. Have a great day. Thank you.