Randal Tillery: Hi, everyone. We're very excited to be here today with you. I'm sure the very large number of people who registered for this, that has nothing to do with any anxiety or feeling around your three-year plan, and only because you love spending so much time with us. I'm the area director of Post-secondary Education Workforce Development at WestEd. We are really excited that you are also interested in coming today.

You are not obviously going to hear the words "three-year plan" use correctly in a sense, only today. This is going to be part of a continuing body of work. All of the contractors working with CAEP that includes WestEd, CASAS, and the Sacramento County Office of Education, TAP, are planning a whole series of professional development around your planning process and a whole support structure for next year as to go through the planning process. So thanks for being here, Blaire.

This is a quick nod to our team-- we could not do this work without a really strong team. Blaire Willson Toso, our senior program manager adult education workforce development. Jessica Chittaphong who is the adult education pipeline build manager and a coordinator on our team, and I've already introduced myself. So with that, I'm going to turn it over to Blaire who's going to take us through what's going to happen today.

Blaire Willson Toso: Super thank you, Randy. I'm delighted to be here with all of you mainly because we're very excited that we have a wonderful roster of guest presenters that we're going to start with-- Dr. Carolyn Zachry and Neil Kelly to provide an overview of the three-year planning process. And then we'll provide a little bit more information about the AEP dashboard, and how you might be able to use that information as well as then having four presenters who will be providing us insights to aspects of their three-year planning process, which will then be followed by an opportunity for you all to ask questions and engage in discussion with them.

We also hope during this time you'll be able to share any of your best practices as well. And that is, we actually just wanted to note that we have moved number four-- data collection and validation-- after the panel presentation because we're going to roll through some of these slides to make sure that we get to the panel presentation and allow everyone to have an opportunity to present their information and have some robust discussion after that. So now I'm going to turn it over to Dr. Carolyn Zachry, the California CDE Adult-- from the CDE Adult Education Office and Neil Kelly from the state CAEP office.

DR. CAROLYN ZACHRY: Thanks, Blaire and everyone. And welcome to the three-year planning webinar. I'm really happy that you're all here, I know that we're going to have a full house today. And the thought about starting out planning again, I'm sure might be daunting for all of you because it feels like we just did three-year planning. And for those of you that are also we often did, you just completed your continuous improvement plan.

So there's this thing your mind going, wait a minute, can I get to relax and not think about planning? No. So we're going to start you on this road to working on your three-year plan early so that you don't feel rushed at the end.

When you think about this three-year plan, it's really about looking at data, which you certainly-- we're going to hear a lot about today, but also looking at your region and your consortium, and looking at your labor market and the needs of the workforce. And pulling all of that information together so that as a consortium you can be forward-thinking for those three years to then develop this plan on how you're going to be nimble and address the needs in your area.

And so for me, three-year planning is all about where do you want to take your students? And how are your students going to benefit from the programs that you're offering so that they can get a job, help their children be more successful in school, improve their work environment perhaps by getting a promotion, getting citizenship? It's all about those pieces on that path for your students.

And so with this three-year planning, gives you that opportunity to look back-- what did you do over the last three years, and then either move that needle forward or shift to something else that you need to shift to. So again, I appreciate you taking your time today to attend this webinar, and I know that you're going to get a great deal of substance from it. Neil.

Neil Kelly: Thanks, Carolyn. Yeah, I just wanted to add just a couple of reassurances because we have so many people on the line-- was interesting-- this last few weeks I heard from people around the state saying we never really got going on my last three-year plan. We were in the process of implementing it, and then COVID hit, and then we had to shut things down.

So if you're worried about this three-year plan, just think of it as a refresh of your last three-year plan. I mean, you have to do your needs assessment and look at your data and stuff like that. But you might already have done a lot of work, and you do have to look at the landscape of what adult education looks like now, after the pandemic is slowly winding down, we hope.

And then the other thing was, there was some concern in the field about the launch word and all the dashboard information we're displaying. We're going to head to some paper for performance or performance-based funding model. And that is not being discussed at the state level, so I don't know where that's coming from. We are still working out data bugs, so to speak.

We have CASAS and WestEd working together, so data sets are shared. The community college data set, the K12 data set, and TOPSpro, the EDD base wage file. All those coming together. So that's quite a monumental task, and they're still working out some minor bugs.

We're a lot better than we were a few years ago and we still have some little ways to go to get to a perfect system, but bear with this. And we hope you enjoy this webinar today. And we look forward to you coming on to more of these webinars related to using your student data, and long-range planning, and looking at strategies to help transition students to work in a post-secondary.

OK, so just a quick purpose statement here, I think we use this in the guidance from the last three-year plan, but it still holds. Three-year planning process is designed to provide consortia and consortium members a chance to collectively assess the impact of services provided over the previous period and to identify new approaches to meeting educational workforce needs of adult learners in a diverse ever-changing economic landscape.

In this way, it also provides an opportunity to reexamine commonly held assumptions about beneficiaries and providers, which taken together, can promote stronger collaboration among agencies and deeper connection to students and the communities they serve.

So let's get into a little bit of the state level guidance and what we plan to roll out later this summer. So as the first bullet says, we'll be rolling out guidance on the three-year plan later this summer. We'll be having webinars addressing the template, the guidance, what's going to be required, what tools, what resources, and we'll have additional webinars-- how to evaluate, how to create actionable steps, all that stuff. And you have over a year to submit your plan-- it'll be due in probably mid-June of 2022.

And then, of course the three-year plans will be entered into NOVA, and it'll align with your consortia annual plan, your member work plan, and the members budgets. So I think the last three-year plan we had you upload it to Nova in a document, Word document. This time, you'll be able to enter all that information into NOVA and then it'll align with your annual plan so you won't have to do something vastly different-- cascade down to that member level and their work plan and their budgets.

So we're hoping to make this a lot more-- oh, can you go back to the last slide. OK, and then also we'll include, and we're going to get to this a little bit more in detail, member level goal setting and targets. And we have a couple of slides on that coming up. So next slide, please.

And so here's, we don't have the prototype, yet we have programmers working on it in NOVA, but here's what it's going to look like. You'll be required to do an executive summary, then there'll be a section calling for a needs assessment, then you'll do an evaluation of current levels and types of education and workforce services for adults in the region.

And what we'll be doing for that evaluation will be bringing in over student data, say like one hour or more of instruction, 12 hours or more of instruction, and by program area. And we'll create tables for each consortia, so they can evaluate the current levels and types of education workforce services. So that'll be available in NOVA.

Same thing with the evaluation of funds available to members of the consortium entities. As you know, in December, you're required to certify your hours of instruction in any leverage funds from a variety of fund sources. We'll bring that information over from NOVA or from that part of NOVA into the three-year plan-- I guess process or screen-- and you'll have a table there that you can speak to and evaluate the funds available and what you're doing with that. Next slide, please.

And then the plan will get into those goals and strategies. These are listed in the legislation. But if you look into the legislation, there's a few things under this but pretty much the first one actions that the members of the conspiracy will take to address learner educational need, so action items around addressing need, and then actions to improve integration of services, and improve transition of postsecondary and workforce.

And so I think there's a subcategory in that that you'll see in the guidance that talks about placement into jobs, any kind of strategies you're using to improve those transitions. And we know lots of strategies that we're using, and so we'll probably reference those in the guidance as a helpful hint. And we have lots of them-- anything from IT, immigrant integration, to enrollment school enrollment, credit for prior learning, competency based education-- you name it. There's tons of stuff out there to help with that transition piece.

And then finally, we'll have a section on actions that members of the Conservancy will take to improve the effectiveness of their services. So the effectiveness ties into the goal-setting in target shall be required to do. And so you'll be able to list actions around that goal-setting that you went through and what targets that you've chosen for your consortia and for your member districts.

And then if you look on the left side of the screen, this is kind of a snapshot of what the prototype looks like. You can see this is where the action or objectives and outcomes are listed. And we'll get into this a little bit. Next slide, please.

OK, and then just a few other comments. If you're planning to do maybe a more elaborate three-year plan than what's in NOVA, and you want to put in pictures, and charts, and tables, and graphs, and have this available for stakeholders or board members or partners, you will be able to download this from NOVA and HTML. It will allow for customization for those local presentations or regional presentations.

We'll also, when I mentioned the metrics and the targets, we'll be updating those on an annual basis as part of the annual planning process you can see how you're doing based on those projected targets. Also we'll include, as part of the CFAD process, a government certification process that will be due at the same time in May of 2022.

And then-- I think this-- we're going to get into what we talked about metrics and targets. So at the conservatory level, when you go into NOVA-- oop, previous slide, please. Thank you.

So when you go into NOVA, you'll see you'll have to pick-- there's two mandatory consortia level metrics or targets. One is number of adults served. And so what you'll see is the last three years of data and then you're going to be projecting out, at the consortia level, number of adults served. So when we mean number of adults served, we mean those students that are one hour or more, what the feds call reportable individuals.

And then the other mandatory metric are target at the consortia level is student barriers. So we're listing the four student barriers that tie to the funding formula. And so if you look at English language barrier that equates to what we call in the formula ESL English spoken-- that's spoken well at home, or low literacy equates to low literacy and the funding formula. Low income-- same thing. Long-term unemployed is unemployed and the funding formula.

So we'll be asking you to select at least one of these student barriers and to project out the percentage of students that you'll be serving that will have these barriers . So you can select more than one, but you have to select at least one. And so that'll be mandated at the consortia level. OK, next slide, please.

OK, and then here's a little more background. And then they'll be aside from the consortia level metrics, we'll also have member level metrics and these will be mandated. We'll be tracking the percentage funds spent. So we'll be looking by year how much the percentage of your CAEP funds you've spent because carryover has become a big issue. And these will be calculated for you and they will be put into NOVA. But we'll get to your questions in a little bit.

And then we'll also be tracking, at the member level, we'll mandate to project out the number of participants-- those are students reaching the 12 hours or more mark. And so you'll have to project out over the next three years at the member level how many students you'll be serving that are participants.

And then we have a list of 10 optional metrics that you can choose from. You can choose all 10, and this is at the member level or you can choose one or none, and just go with the mandatory member level metrics listed in number two. But if you want to take a look at those optional metrics-- next slide, please. And so-- can we shrink that a little bit because I can't see that last metric, or maybe scroll down now.

OK, so we'll get to that last one-- it's kind of cut off. So here are the 10, and we're counting educational functioning level gains for ESL, ABE, or ASE as three separate optional metrics because we don't want to lump EFLs all into one. So starting with high school diploma, number of post-secondary credentials, of course, the EFLs that set an additional three transitions in CTE, transition to post-secondary, employment second quarter after exit because we find that we have-- it's easier to find out about employment second quarter after exit than it is after fourth quarter. And the medium engineering.

And then that last one-- if it's related to immigrant integration-- is there a way you can--

Blaire Willson Toso: Sorry, I don't know what happened. I guess it's showing on my screen and it says it's the immigrant integration milestone is--

Neil Kelly: OK. So those are the COAPPs from EL civics part of that WIOA II grant, but we're also using that in CAEP, and so that could be another optional metric for you to use and that'll be also available this year on the launchpad and they're working with CASAS to further define those COAPPs and how it relates to immigrant integration. So we did want to include that would be intentional about that include that as an optional metric if you chose to project that out over the next three years.

And is that the last slide, Blair or? OK, just a couple of things. So things that SCOE is going to be working with this on. Of course will have the planning guidance available probably this summer, hopefully early this summer. We'll also have a planning template because everyone likes to see the planning template rather than going to NOVA, they like a hard copy, so we'll get that running. We'll also do webinars, on the guidance, on the template, on planning, on evaluation, and so much more tons of technical assistance and webinars coming up.

We'll also update the fact sheets that we had last time by region so you can use those as a resource. Also provide other resources that will be elaborated in the guidance as far as what those additional resources are. We'll do regional network meetings starting probably in August as check-ins and allow for some collaboration, maybe some communities of practice. We'll do presentations about this at the Summer Institute, and then of course, in October at the director's event.

And then we'll continue to-- let's see there-- OK, there, had to clear my screen there. Targeted technical assistance, we'll continue to offer that to rural remote consortia to help them with their three-year plan, and then we'll also have SCOE facilitated connection, if you need to connect with planning experts or need technical assistance that will connect you with those experts in those fields. And next slide, please.

OK, so let's see-- were there any questions that we could cover right now, or in the interest of time, did you want to move on? Not sure, Blair, what you wanted to do.

Blaire Willson Toso: Yeah, the one it was one-- that says they're asking if they're tracking K-12 success, and I don't want to speak to that now or if that wants to be offered offline. The other thing we can do is download the chat and then send out an email afterwards that responds to questions, whatever works for you.

Neil Kelly: So there's 10 metric we're tracking. So related to K-12 success, it could be part of tracking participants. Could be related to conspiracy level reportable individuals if someone in that class had a student barrier, if the student was going towards a diploma or a degree or had educational functioning level gain, or had a COAPP. I know it's been difficult to qualify those students in K-12 success.

But as far as WestEd and CASAS working together, they looked at all the different metrics we had out there, and those 10 optional metrics were probably the most solid. And now we didn't want to make a mandated metrics, that's why we chose the two at the consortia level and the two at the member level because those were probably the safest ones to mandate. So I don't know if that answered the question, but based on the experts and the input, that's how we picked those 10 metrics. Were there any other questions?

Blaire Willson Toso: No, just more-- there is a suggestion that the template for the plant through your planning cycle should be streamlined and maybe a little bit simpler than the last version that was used.

Neil Kelly: Right, that's why we're putting it into NOVA and aligning it with the annual plan. So your three-year plan becomes your annual plan which becomes your member work plan which becomes your budget. So it's a cascading effect all in one system. You don't have to upload standalone documents anymore.

So this is about as streamlined as it gets. Because we have to follow the legislation. So all those items that I listed, those are in legislation. They're codified in Ed Code, so we have to follow them. And so this is probably the easiest way to answer the questions that the legislature wants us to do with the money they give us. So it is pretty much streamlined, given the parameters set up by the legislature.

Blaire Toso: Super, thanks. There are a couple other questions. What about the other literacy gains, like Workforce Prep, and is there a set date the template will be released?

Neil Kelly: So we're hoping the template will be released probably early next month. And then the guidance would follow over the summer, because the guidance is a lot more of a deeper dive, a little more detail. And so but we can release the template, and probably allow you to go into NOVA.

We'll see. But at least, we'll get a template out there, so you can look at it, and get a look and feel of it. And maybe be able to test drive it in Nova. And then what was the other question for workforce prep?

Blaire Toso: Yeah, for other literacy gains.

Neil Kelly: When we looked at Workforce Prep, and passage of an exam, and Workforce Milestone, we weren't comfortable with that yet. We're still working on that metric as far as alignment. And so that's why they didn't make the top ten. And so we're still trying to work on the quality and integrity of those metrics. So that's why they didn't make the top ten list.

Blaire Toso: Thanks, Neil. Thanks for the questions too. We're going to go ahead-- and I just wanted to reiterate just to support what Neil was saying that, I'm sure there are a lot of questions, and that SCOE TAP going to have is rolling out a lot of resources. WestEd is going to also provide some of the technical assistance.

So this is really just to whet your appetite, s and sort get you into the mindset of planning. And we'll continue to answer questions as they come, and then feel free to reach out to Neil, and his team, or to WestEd as we move along, and Carolyn as well.

So I just wanted to say, this portion of the presentation is going to be really just a quick framing of the planning process, and the key decision points. So the slides have been developed so that you can return to them for fuller information, and as a springboard into the planning process. But we really want to move into the panel presenters, because they will bring this information on the slides to life via their s life experiences.

And we're going to be sending out some more tailored resources that will help you to dig into the different kinds of data, how that informs decision making, and setting goals, targets, and developing your plans, and entering them into NOVA. And this slide is really just to sort of begin to frame the adult education pipeline roll in here, as Carolyn and Neil identified, the required and optional metrics that you'll need to be acquainted with for planning and reporting.

And we are highlighting them, so that you can easily find them on the AEP dashboard. Each step represented here is also a dashboard tile, which Jessica will demonstrate in a minute. And just as a note, we're doing this really quickly. If you're unfamiliar with this, or you would like further information, just feel free to reach out to us.

This slide takes exactly that. These are your tiles, and then highlighted, you can see where the different metrics that Neil talked about are going to be placed in within the adult education pipeline.

And before we move into a demonstration of where these are found, I just wanted to give you a quick reminder about how to read the numbers on the dashboard. Here, the denominator identifies all of the possible students who are considered in the count for outcomes. Whereas, the numerator is those who achieve the outcome.

So for example participants, those who have achieved 12 or more instructional hours, so the denominator for this would be-- or here on the example on the slide-- is the participants, and then whether they're enrolled in ESL, ABE, ASE programs. Whereas, the numerator will only take those who have actually completed the EFL level by pretest, versus the post-test, or of course progression in the same area, which are the definitions of how you meet that criteria.

So basically, it's just like a fraction. And we wanted to make sure that you also knew that the denominator changes based on the criteria of the metric. So Jessica will show you where to find this information on the dashboard. But we just want to be aware that the numbers, how to read those numbers, as they're important as you begin to delve into the data.

But at this point, I'm just going to hand it over to Jessica, who will do a quick live walk-through of the metrics, trends, and outcomes, and the drill downs.

Jessica Chittaphong: Thank you, Blaire. Let me take over. And you can access the site by going to this link that I've just added to the chat. You can also just Google Launch Forward Adult Education Pipeline, if you're unfamiliar with how to get to this pipeline. It is an open access dashboard, so anyone can access it.

You don't need to log in. You don't need a criteria. So just feel free to use the link, and hop on and explore. So I will be providing a quick demonstration of where to find those metrics that Neil and Blaire had just presented on.

If you want a longer demonstration, feel free to go back to the caladulted.org website, and there should be a recording of the webinar we did back on the 30th, where we did a much slower walk through of the pipeline. And so for those who who are unfamiliar with the dashboard, that would be a good resource for you.

So jumping right in, this is the home page. And as Blaire mentioned, we have the metrics organized by the different steps in the student journey. So we have the students programs bucket. And we have the progress metric bucket. And all of the metrics will be grouped by those particular categories.

To find your institution, feel free to jump to the second box, and just start typing your name. It could be just start typing, and the list underneath will update automatically. You can do the same for your consortia. If you could consortia and start typing, your consortia name should slowly populate here.

I'm going to stick with the statewide view for now. And any time you make a selection, please remember to click View, because that's going to update the data that's populated underneath here. A quick note for folks, just because it's useful to remind everyone, is that we do have something called lagging indicators on the Adult Education Pipeline.

So for transition and employment earnings metrics, we don't have data for the most recent year, because we either gave the student an extra year to meet that outcome, meaning, we wait a year to see if they fully transitioned. And we also wait a year to see if they end up in the employment and workforce cohort.

So that's why these tiles-- they select an earlier year. And if I go back a year, the data will update. And I'll show you some-- it'll will show you the updated charts here. All right. So for the first metric that is mandatory for consortium, we have the reportable individuals, or the adult service metrics.

This is a good place to remind everyone that there is a couple of different ways to access the different levels of data on the Adult Education Pipeline. You can go to the AEP scorecard, which is what I dubbed the executive summary of the Adult Education Pipeline.

It actually holds a lot of important metrics that have been identified by the state. Most of the metrics that we talk about, and that have been identified to be included in NOVA will be on this page, but not all of them. So on this page, you can get a quick list of the metric, and then the number associated with it.

By clicking the name, it actually expands, and shows you a time trend view of that particular metric. On the scorecard, you can also drill down all of the charts by demographics. So you can look at gender, race, ethnicity, and age group by clicking one of these. It will display a bar chart that displays the counts for that selected year, for the different subgroups in that drill down.

If I go back home, I can also find the report of individuals, or adult service metrics in the students and programs tile. When I click here, it'll take me to something called the Summary Page. And this page is just an infographic of some key metrics that we've selected. This is for people who don't really want to engage with the metrics themselves, but just want a quick bird's eye view of what's going on.

But I want to focus really on this detailed data page. So the detailed data pages is actually where you'll find all of the data metrics that we have available for the Adult Education Pipeline. They're grouped on the left hand side. And you can actually extend and figure out what metrics are where.

If I click on enrollment, that's where you'll find the total reportable individuals, or adult serves chart. Here, you'll again see the time trend view, and you'll see the demographic breakdown drill downs, but you'll also see the student type drill down, which is only accessible on the detailed data view.

Underneath will also be a data table that shows accounts for the selected years. If I make a selection on the drill downs, the data table will update with those particular data. Also, on this page, is where you'll find the barriers to employment metrics. So we have the barriers to employment for if ever flagged.

And that's where you'll see the English as a language learner. You'll see the low income and the low literacy rate. The low literacy barriers to employment, they're posted here. You can also drill this down by program type, as well as by the student type for sign in returning. Below again, is you'll find the data table.

And the difference between this table and this next table, barriers to employment, is that early on in the design process, folks recognized that some barrier to employment are long term, and can affect students throughout their educational career, and so they should be counted if they were ever flagged as having that particular barrier. So that's what this chart shows.

The second chart flagged in the selected year are for those barriers that may change, or are temporary. And so they are counted. They are recounted each year for this particular metric. And that's where you'll find the long term unemployed barriers to employment.

Since we're here, we want to stay in the detailed data pages. But we might want to jump to the next page, which is the next student journey bucket. So I can quickly do that by clicking this arrow to the right, and it will take me to the progress page. And this is where you'll find the next two metrics that will be on the NOVA.

These are I think, part of the optional metrics list. So you'll see here the completed one, or more educational function levels, as well as the immigrant integration milestone. And you'll see here there's a big jump because the COAAPs of the I3 report were first introduced in 1920, and so that's when we included the data. And that's why there's a big jump here.

If I go back to the educational function levels, you'll see now that I have three possible ways to drill down this data set. I can look at demographics. I can look at program type, as well as student type. Now, these three all play together.

So if I click gender, and I click ESL, this will give me account of gender by program of those who gained one or more educational function. Levels so we play around with this, particularly, if you're planning and looking into equity issues, this is a great tool. And it's available to you on the detailed data page.

If I jump to the next page, it'll take you to transitions. And that's the next couple of optional metrics that'll show up, is the transition to CTE, and transition to credit college courses. These two are actually subsets of the transition to post-secondary metrics.

So the transition to post-secondary metrics has traditionally included transition to CTE, as well as transition to credit. But we for this particular build, we broke that up to maybe support institutions who maybe focus on one over the other.

Going to the next page, you'll see what we call the Success Page. And that's where you'll find all of the award counts. So we're counting here students who earned a high school diploma, GED, or high school equivalency. And this is where you'll also find students who completed a full secondary credential.

Jumping to the next page, which is our last page on the Adult Education Pipeline, is the employment and earnings section. And so you'll see here employment two quarters after exit, as well as change in earnings. And those are also the last of the two optional metrics that are going to be available on Nova.

I'm going to keep it there. Because as Blaire mentioned, we want to focus on our panelists. But I'm happy to answer questions on the chat, and we can circle back to this towards the end if there's time.

Blaire Toso: Jessica, sorry. I missed something. Someone was asking-- there are two questions that definitely need to be asked, or one that he wants to know how colleges are recording reportable individual, and then there was also one from John about showing you the technical definition link from the data set page.

Jessica Chittaphong: Yeah, so let me see. So reportable individuals for the college side, there's a couple of criteria that we count. So the first criteria is whether or not they had one, a noncredit enrollment, and one hour of positive attendance. So that's one criteria.

The other criteria is whether or not they received a noncredit student service. And then we looked at both criteria, and found statements from there for the college side. What was the second question, Blaire?

Blaire Toso: It was about looking at where the technical-- sorry. It's way up there from John. Was asking about the technical definition from the page.

And you know what? I'm just going to go ahead, John, not to deflect your question, but I think we're going to have to move on, so I can be sure that everybody on the panel has an opportunity to present. Maybe Jessica, if you can scroll through the chat, there are a couple of questions that still remain.

Jessica Chittaphong: OK. I will do that.

Blaire Toso: So thank you. We know that we've really packed the agenda. We just sort of wanted to orient people. And again, there are some really great webinars that have been reported about using AEP. And contact us also if you'd like us to do some direct technical assistance, or do a walk-through on the AEP. We can certainly do that as well.

So we just want to really make sure that we have the chance to get to talk to the people who are really the ones who are in the field, and who you all will relate to the most as far as what planning process looks through. I'm going to fly through these slides, because we're just a tiny bit behind.

As I said, they're all written so you can come back and look at them, and we'll be doing more work on this later in the summer, and in the fall. But we just wanted to really emphasize that in setting your goals and targets, that, you ask very good questions. And we have a series of these. And I'm going to leave it at that, because one of a couple of the presenters will talk about that.

But really, you're looking at what is sustainable? What is feasible? And where are the gaps and trends that you want to address. And then also, we really wanted to emphasize that once you begin to look at these, you look at them both in the way that Neil and Carolyn have identified the metrics, that there are consortium level. And then there are member levels.

And that it's important to consider and talk as a consortium about whether the goals and targets are tangible, how you can support one another, and how do each of you as an individual member want to set individual goals, and how they work together to support your consortium as a whole to provide the services. We also know and suggest that you use a lot of other different resources to collect data. The Census Data, Labor Market Information, Centers of Excellence, have some great pieces. The Healthy Living Index, Poverty Index.

And I know that the panelists will talk about more. If you have a moment and you're thinking about it, and you have a favorite data source that you use to inform your planning, please go ahead and put that in the chat, so it can be shared with your colleagues who are also on this presentation.

And that is probably the quickest overview of setting goals and targets. But we're going to lead into that. And each one of your panel presentation members will be able to give you more in-depth information about this through their contextualized examples of their work.

And we're going to start with Dulce from North Orange. And then we'll move Ute from East Region, Maryann from Glendale, and John from Sequoias. And we would like to just take a moment and thank them again for sharing their expertise with all of you to prompt discussion, and thinking, and provide strategies that might work for you in your context. So Dulce, I'm going to go ahead and hand it over to you.

Dulce Delgadillo: All right. Thank you, Blaire. So I'm going to go a little bit about the work that we've been doing at North Orange Continuing Education, and the North Orange County Regional Consortium for Adult Education. So next slide.

So we've been doing a lot of work on a variety of areas. And I'm going to go through the story of what NOCRC and NOCE has done. But really, we started this work a couple of years ago really trying to understand what is our local data. What does it look like, right, our consortium data at a local level.

What does it look like within our community level, so that census data. And what does it look like within our institutions, and within our systems that we are collecting this data in. And does all that data live in the systems that we need it to live in, right?

In addition, we also did some work around understanding the CAEP outcomes, so really drilling down into the technical definitions, looking at the coding, really looking at how things overlap with what we currently have within our institution, and through other initiatives. And just really getting a good grasp on what those CAEP outcomes are, and how our data are displayed through those outcomes.

Then I'll go over into our CAEP evaluation report, which we just finalized this last past year, which was our first year of CAEP evaluation report. We hope to do that on an annual basis. And then finally, I'll do a quick wrap up on how we're just continuing to build capacity around data literacy, really helping our community and the consortium to feel comfortable around data, to feel comfortable around digging into the data, and how we can use data to inform the decision making, and the planing for the next couple of years for our consortium. Next slide, please.

All right, so the first thing I'm going to talk about is really diving in. We did this infographic a couple of years ago to really look at just our community profile. So we provide a community profile in our institutional effectiveness report every year. And we pull our census data according to our census tracts within our service area.

And so really, what we're focusing on is, are we addressing the gaps, or what are the gaps that exist within our community, right. So how many students actually have a high school diploma, or need a high school diploma. What is our income levels? What is the distribution of the age population within our community, service area, and race, ethnicity. And so those are the types of components that we really wanted to be able to present alongside our institutional data as well. Next slide, please.

In addition, we wanted to see kind of how do each of the students that we're serving kind of break up into the number of students served by CAEP area, right. So one of the first things that we looked at is, what is the type of census data that we need to look at, and how does it connect to this particular CAEP strategy?

So the activities, we refer to them within our consortium as CAEP strategies. And these are really the activities that we're implementing through each of those CAEP areas. And so we looked at-- you can see we looked at our institutional effectiveness data of, this was how many students were served in this academic year by CTE. These are the certificates that we awarded.

Just to give our consortium members really a broad overview of, what are those trends typically look like within our institutional effectiveness report within our institution. And how do they kind of fit into those CAEP buckets that now we're going to have to report on, right. In addition, we also looked at just building capacity around understanding the data flow.

So you can see in the middle of that second data, or the infographic, that middle box, that was something that was provided to us. I think it was from Neil actually. And to really understand where are all these data sources that are coming from? What is the data flow? Where does it live within our consortium?

What do we need to input into these data systems in order to really get, again, a better understanding of how is this data going to be. As we enter the data into the systems, how is it going to shoot out on the other end in the LaunchBoard, right. And really, just already at the bottom piece, setting those expectations around these data trainings, and alignment with CAEP goals. Next slide.

So again, now that we have a big picture of our community data. We have a picture of our just overall institutional data. Now, we have to get a better understanding of what are these CAEP outcomes. So again, this is a visualization that our team produced in terms of just breaking down those CAEP outcomes.

And this has served as a really good just training tool. We even include this I believe, in our on-boarding package for the consortium, just to give members a sense of, this is exactly what this metric holds. This is what are the things that are being calculated, and this is how your students are going to be thrown into the denominator, or in the numerator, right. Because all of those things are going to matter once you start looking at those outcomes, and those metrics on those launch boards. Next slide.

All right. And then we also wanted to see what does this look like. What is CAEP in our overall institution look like, right? So last year in our institutional effectiveness report, we actually included state wide initiative metrics in our IER. And we revised our institutional effectiveness process for it to actually have a component of initiative planning around our process, right.

So we wanted to see, OK, when we look at our local metrics as an institution for institutional effectiveness, how are we defining students served, and how does that compare when we're looking at these metrics for strong workforce program? How does that compare when we're looking at metrics for adult ed? And how does that compare when we're looking at student success metrics?

So that we can, again, have an understanding of where are some overlap on these metrics, and the students, right. Do students fall into all three of these buckets? Maybe we're capturing students in one initiative, and not another initiative, and what is the reason? And how are we facilitating those conversations within our consortium, and with our partners in terms of maximizing the students that we're reporting. Next slide, please.

So that took us to our annual AEP evaluation report. So this was something that we produced in October of 2020. And we were really focusing on these three key valuation questions. So what types of instructional and supportive services are being provided to our students through the consortium? What data elements is the consortium capturing through the implementation of these strategies that are being funded?

So please note that when we produce this evaluation report, it wasn't the whole universe of NOCE, right. It was an NNOCRC. It was really focused on how are we spending our money, and what are the activities that we are spending on. And how many participants, how many students, how many outcomes are being produced with those activities and strategies that we are implementing with our CAEP funding?

So it was very much a narrow scope tied to how is the money being spent within our consortium, and what are the outcomes that are related to the money that is being spent. And we only produced outcomes in terms of-- or did some reporting on the evaluation based off of those that would continue, right.

So even the evaluation report isn't including all of the strategies that we included. It was really a planning document to help with the conversation of what should we continue next year. What do we know is effective in creating an impact from what this preliminary data has. And we do plan on doing this.

Another great opportunity here was to get those student voices, right, so have that quantitative and that qualitative data, so we know what the numbers are, but what our students experience is. So being able to incorporate that into a document like this was really helpful in facilitating the conversation. Next slide.