(DESCRIPTION) Consortium Planning Using the A E P Dashboard (SPEECH) RANDAL TILLERY: Hi, everyone. We're very excited to be here today with you. I'm sure the very large number of people who registered for this, that has nothing to do with any anxiety or feeling around your three-year plan, and only because you love spending so much time with us. I'm the area director of Post-secondary Education Workforce Development at WestEd. We are really excited that you are also interested in coming today. You are not obviously going to hear the words "three-year plan" use correctly in a sense, only today. This is going to be part of a continuing body of work. All of the contractors working with CAEP that includes WestEd, CASAS, and the Sacramento County Office of Education, TAP, are planning a whole series of professional development around your planning process and a whole support structure for next year as to go through the planning process. So thanks for being here, Blaire. (DESCRIPTION) Title, Today's Presenters (SPEECH) This is a quick nod to our team-- we could not do this work without a really strong team. Blaire Willson Toso, our senior program manager adult education workforce development. Jessica Chittaphong who is the adult education pipeline build manager and a coordinator on our team, and I've already introduced myself. So with that, I'm going to turn it over to Blaire who's going to take us through what's going to happen today. (DESCRIPTION) Webinar Agenda (SPEECH) BLAIRE WILLSON TOSO: Super thank you, Randy. I'm delighted to be here with all of you mainly because we're very excited that we have a wonderful roster of guest presenters that we're going to start with-- Dr. Carolyn Zachry and Neil Kelly to provide an overview of the three-year planning process. And then we'll provide a little bit more information about the AEP dashboard, and how you might be able to use that information as well as then having four presenters who will be providing us insights to aspects of their three-year planning process, which will then be followed by an opportunity for you all to ask questions and engage in discussion with them. We also hope during this time you'll be able to share any of your best practices as well. And that is, we actually just wanted to note that we have moved number four-- data collection and validation-- after the panel presentation because we're going to roll through some of these slides to make sure that we get to the panel presentation and allow everyone to have an opportunity to present their information and have some robust discussion after that. So now I'm going to turn it over to Dr. Carolyn Zachry, the California CDE Adult-- from the CDE Adult Education Office and Neil Kelly from the state CAEP office. (DESCRIPTION) Welcome (SPEECH) DR. CAROLYN ZACHRY: Thanks, Blaire and everyone. And welcome to the three-year planning webinar. I'm really happy that you're all here, I know that we're going to have a full house today. And the thought about starting out planning again, I'm sure might be daunting for all of you because it feels like we just did three-year planning. And for those of you that are also we often did, you just completed your continuous improvement plan. So there's this thing your mind going, wait a minute, can I get to relax and not think about planning? No. So we're going to start you on this road to working on your three-year plan early so that you don't feel rushed at the end. When you think about this three-year plan, it's really about looking at data, which you certainly-- we're going to hear a lot about today, but also looking at your region and your consortium, and looking at your labor market and the needs of the workforce. And pulling all of that information together so that as a consortium you can be forward-thinking for those three years to then develop this plan on how you're going to be nimble and address the needs in your area. And so for me, three-year planning is all about where do you want to take your students? And how are your students going to benefit from the programs that you're offering so that they can get a job, help their children be more successful in school, improve their work environment perhaps by getting a promotion, getting citizenship? It's all about those pieces on that path for your students. And so with this three-year planning, gives you that opportunity to look back-- what did you do over the last three years, and then either move that needle forward or shift to something else that you need to shift to. So again, I appreciate you taking your time today to attend this webinar, and I know that you're going to get a great deal of substance from it. Neil. (DESCRIPTION) Neil Kelly (SPEECH) NEIL KELLY: Thanks, Carolyn. Yeah, I just wanted to add just a couple of reassurances because we have so many people on the line-- was interesting-- this last few weeks I heard from people around the state saying we never really got going on my last three-year plan. We were in the process of implementing it, and then COVID hit, and then we had to shut things down. So if you're worried about this three-year plan, just think of it as a refresh of your last three-year plan. I mean, you have to do your needs assessment and look at your data and stuff like that. But you might already have done a lot of work, and you do have to look at the landscape of what adult education looks like now, after the pandemic is slowly winding down, we hope. And then the other thing was, there was some concern in the field about the launch word and all the dashboard information we're displaying. We're going to head to some paper for performance or performance-based funding model. And that is not being discussed at the state level, so I don't know where that's coming from. We are still working out data bugs, so to speak. We have CASAS and WestEd working together, so data sets are shared. The community college data set, the K12 data set, and TOPSpro, the EDD base wage file. All those coming together. So that's quite a monumental task, and they're still working out some minor bugs. We're a lot better than we were a few years ago and we still have some little ways to go to get to a perfect system, but bear with this. And we hope you enjoy this webinar today. And we look forward to you coming on to more of these webinars related to using your student data, and long-range planning, and looking at strategies to help transition students to work in a post-secondary. (DESCRIPTION) Title, Three-Year Planning Process - Purpose. (SPEECH) OK, so just a quick purpose statement here, I think we use this in the guidance from the last three-year plan, but it still holds. Three-year planning process is designed to provide consortia and consortium members a chance to collectively assess the impact of services provided over the previous period and to identify new approaches to meeting educational workforce needs of adult learners in a diverse ever-changing economic landscape. In this way, it also provides an opportunity to reexamine commonly held assumptions about beneficiaries and providers, which taken together, can promote stronger collaboration among agencies and deeper connection to students and the communities they serve. (DESCRIPTION) Title, Three-Year Planning (SPEECH) So let's get into a little bit of the state level guidance and what we plan to roll out later this summer. So as the first bullet says, we'll be rolling out guidance on the three-year plan later this summer. We'll be having webinars addressing the template, the guidance, what's going to be required, what tools, what resources, and we'll have additional webinars-- how to evaluate, how to create actionable steps, all that stuff. And you have over a year to submit your plan-- it'll be due in probably mid-June of 2022. And then, of course the three-year plans will be entered into NOVA, and it'll align with your consortia annual plan, your member work plan, and the members budgets. So I think the last three-year plan we had you upload it to Nova in a document, Word document. This time, you'll be able to enter all that information into NOVA and then it'll align with your annual plan so you won't have to do something vastly different-- cascade down to that member level and their work plan and their budgets. So we're hoping to make this a lot more-- oh, can you go back to the last slide. OK, and then also we'll include, and we're going to get to this a little bit more in detail, member level goal setting and targets. And we have a couple of slides on that coming up. So next slide, please. (DESCRIPTION) 3-Year Planning - Fresh Look. (SPEECH) And so here's, we don't have the prototype, yet we have programmers working on it in NOVA, but here's what it's going to look like. You'll be required to do an executive summary, then there'll be a section calling for a needs assessment, then you'll do an evaluation of current levels and types of education and workforce services for adults in the region. And what we'll be doing for that evaluation will be bringing in over student data, say like one hour or more of instruction, 12 hours or more of instruction, and by program area. And we'll create tables for each consortia, so they can evaluate the current levels and types of education workforce services. So that'll be available in NOVA. Same thing with the evaluation of funds available to members of the consortium entities. As you know, in December, you're required to certify your hours of instruction in any leverage funds from a variety of fund sources. We'll bring that information over from NOVA or from that part of NOVA into the three-year plan-- I guess process or screen-- and you'll have a table there that you can speak to and evaluate the funds available and what you're doing with that. Next slide, please. (DESCRIPTION) Goals and Strategies (SPEECH) And then the plan will get into those goals and strategies. These are listed in the legislation. But if you look into the legislation, there's a few things under this but pretty much the first one actions that the members of the conspiracy will take to address learner educational need, so action items around addressing need, and then actions to improve integration of services, and improve transition of postsecondary and workforce. And so I think there's a subcategory in that that you'll see in the guidance that talks about placement into jobs, any kind of strategies you're using to improve those transitions. And we know lots of strategies that we're using, and so we'll probably reference those in the guidance as a helpful hint. And we have lots of them-- anything from IT, immigrant integration, to enrollment school enrollment, credit for prior learning, competency based education-- you name it. There's tons of stuff out there to help with that transition piece. And then finally, we'll have a section on actions that members of the Conservancy will take to improve the effectiveness of their services. So the effectiveness ties into the goal-setting in target shall be required to do. And so you'll be able to list actions around that goal-setting that you went through and what targets that you've chosen for your consortia and for your member districts. And then if you look on the left side of the screen, this is kind of a snapshot of what the prototype looks like. You can see this is where the action or objectives and outcomes are listed. And we'll get into this a little bit. Next slide, please. (DESCRIPTION) Title, 3-Year Planning (SPEECH) OK, and then just a few other comments. If you're planning to do maybe a more elaborate three-year plan than what's in NOVA, and you want to put in pictures, and charts, and tables, and graphs, and have this available for stakeholders or board members or partners, you will be able to download this from NOVA and HTML. It will allow for customization for those local presentations or regional presentations. We'll also, when I mentioned the metrics and the targets, we'll be updating those on an annual basis as part of the annual planning process you can see how you're doing based on those projected targets. Also we'll include, as part of the CFAD process, a government certification process that will be due at the same time in May of 2022. And then-- I think this-- we're going to get into what we talked about metrics and targets. So at the conservatory level, when you go into NOVA-- oop, previous slide, please. Thank you. So when you go into NOVA, you'll see you'll have to pick-- there's two mandatory consortia level metrics or targets. One is number of adults served. And so what you'll see is the last three years of data and then you're going to be projecting out, at the consortia level, number of adults served. So when we mean number of adults served, we mean those students that are one hour or more, what the feds call reportable individuals. And then the other mandatory metric are target at the consortia level is student barriers. So we're listing the four student barriers that tie to the funding formula. And so if you look at English language barrier that equates to what we call in the formula ESL English spoken-- that's spoken well at home, or low literacy equates to low literacy and the funding formula. Low income-- same thing. Long-term unemployed is unemployed and the funding formula. So we'll be asking you to select at least one of these student barriers and to project out the percentage of students that you'll be serving that will have these barriers . So you can select more than one, but you have to select at least one. And so that'll be mandated at the consortia level. OK, next slide, please. (DESCRIPTION) Title, Goal Settings and Targets. (SPEECH) OK, and then here's a little more background. And then they'll be aside from the consortia level metrics, we'll also have member level metrics and these will be mandated. We'll be tracking the percentage funds spent. So we'll be looking by year how much the percentage of your CAEP funds you've spent because carryover has become a big issue. And these will be calculated for you and they will be put into NOVA. But we'll get to your questions in a little bit. And then we'll also be tracking, at the member level, we'll mandate to project out the number of participants-- those are students reaching the 12 hours or more mark. And so you'll have to project out over the next three years at the member level how many students you'll be serving that are participants. And then we have a list of 10 optional metrics that you can choose from. You can choose all 10, and this is at the member level or you can choose one or none, and just go with the mandatory member level metrics listed in number two. But if you want to take a look at those optional metrics-- next slide, please. (DESCRIPTION) Optional Metrics (SPEECH) And so-- can we shrink that a little bit because I can't see that last metric, or maybe scroll down now. OK, so we'll get to that last one-- it's kind of cut off. So here are the 10, and we're counting educational functioning level gains for ESL, ABE, or ASE as three separate optional metrics because we don't want to lump EFLs all into one. So starting with high school diploma, number of post-secondary credentials, of course, the EFLs that set an additional three transitions in CTE, transition to post-secondary, employment second quarter after exit because we find that we have-- it's easier to find out about employment second quarter after exit than it is after fourth quarter. And the medium engineering. And then that last one-- if it's related to immigrant integration-- is there a way you can-- BLAIRE WILLSON TOSO: Sorry, I don't know what happened. I guess it's showing on my screen and it says it's the immigrant integration milestone is-- NEIL KELLY: OK. So those are the COAPPs from EL civics part of that WIOA II grant, but we're also using that in CAEP, and so that could be another optional metric for you to use and that'll be also available this year on the launchpad and they're working with CASAS to further define those COAPPs and how it relates to immigrant integration. So we did want to include that would be intentional about that include that as an optional metric if you chose to project that out over the next three years. And is that the last slide, Blair or? OK, just a couple of things. (DESCRIPTION) 3-Year Planning: Guidance and Resources (SPEECH) So things that SCOE is going to be working with this on. Of course will have the planning guidance available probably this summer, hopefully early this summer. We'll also have a planning template because everyone likes to see the planning template rather than going to NOVA, they like a hard copy, so we'll get that running. We'll also do webinars, on the guidance, on the template, on planning, on evaluation, and so much more tons of technical assistance and webinars coming up. We'll also update the fact sheets that we had last time by region so you can use those as a resource. Also provide other resources that will be elaborated in the guidance as far as what those additional resources are. We'll do regional network meetings starting probably in August as check-ins and allow for some collaboration, maybe some communities of practice. We'll do presentations about this at the Summer Institute, and then of course, in October at the director's event. And then we'll continue to-- let's see there-- OK, there, had to clear my screen there. Targeted technical assistance, we'll continue to offer that to rural remote consortia to help them with their three-year plan, and then we'll also have SCOE facilitated connection, if you need to connect with planning experts or need technical assistance that will connect you with those experts in those fields. And next slide, please. (DESCRIPTION) WestEd Technical Assistance (SPEECH) OK, so let's see-- were there any questions that we could cover right now, or in the interest of time, did you want to move on? Not sure, Blair, what you wanted to do. BLAIRE WILLSON TOSO: Yeah, the one it was one-- that says they're asking if they're tracking K-12 success, and I don't want to speak to that now or if that wants to be offered offline. The other thing we can do is download the chat and then send out an email afterwards that responds to questions, whatever works for you. NEIL KELLY: So there's 10 metric we're tracking. So related to K-12 success, it could be part of tracking participants. Could be related to conspiracy level reportable individuals if someone in that class had a student barrier, if the student was going towards a diploma or a degree or had educational functioning level gain, or had a COAPP. I know it's been difficult to qualify those students in K-12 success. But as far as WestEd and CASAS working together, they looked at all the different metrics we had out there, and those 10 optional metrics were probably the most solid. And now we didn't want to make a mandated metrics, that's why we chose the two at the consortia level and the two at the member level because those were probably the safest ones to mandate. So I don't know if that answered the question, but based on the experts and the input, that's how we picked those 10 metrics. Were there any other questions? BLAIRE WILLSON TOSO: No, just more-- there is a suggestion that the template for the plant through your planning cycle should be streamlined and maybe a little bit simpler than the last version that was used. NEIL KELLY: Right, that's why we're putting it into NOVA and aligning it with the annual plan. So your three-year plan becomes your annual plan which becomes your member work plan which becomes your budget. So it's a cascading effect all in one system. You don't have to upload standalone documents anymore. So this is about as streamlined as it gets. Because we have to follow the legislation. So all those items that I listed, those are in legislation. They're codified in Ed Code, so we have to follow them. And so this is probably the easiest way to answer the questions that the legislature wants us to do with the money they give us. So it is pretty much streamlined, given the parameters set up by the legislature. (DESCRIPTION) Blaire Toso (SPEECH) BLAIRE TOSO: Super, thanks. There are a couple other questions. What about the other literacy gains, like Workforce Prep, and is there a set date the template will be released? NEIL KELLY: So we're hoping the template will be released probably early next month. And then the guidance would follow over the summer, because the guidance is a lot more of a deeper dive, a little more detail. And so but we can release the template, and probably allow you to go into NOVA. We'll see. But at least, we'll get a template out there, so you can look at it, and get a look and feel of it. And maybe be able to test drive it in Nova. And then what was the other question for workforce prep? BLAIRE TOSO: Yeah, for other literacy gains. NEIL KELLY: When we looked at Workforce Prep, and passage of an exam, and Workforce Milestone, we weren't comfortable with that yet. We're still working on that metric as far as alignment. And so that's why they didn't make the top ten. And so we're still trying to work on the quality and integrity of those metrics. So that's why they didn't make the top ten list. BLAIRE TOSO: Thanks, Neil. Thanks for the questions too. We're going to go ahead-- and I just wanted to reiterate just to support what Neil was saying that, I'm sure there are a lot of questions, and that SCOE TAP going to have is rolling out a lot of resources. WestEd is going to also provide some of the technical assistance. So this is really just to whet your appetite, s and sort get you into the mindset of planning. And we'll continue to answer questions as they come, and then feel free to reach out to Neil, and his team, or to WestEd as we move along, and Carolyn as well. (DESCRIPTION) Learner Journey: Key Metrics (SPEECH) So I just wanted to say, this portion of the presentation is going to be really just a quick framing of the planning process, and the key decision points. So the slides have been developed so that you can return to them for fuller information, and as a springboard into the planning process. But we really want to move into the panel presenters, because they will bring this information on the slides to life via their s life experiences. And we're going to be sending out some more tailored resources that will help you to dig into the different kinds of data, how that informs decision making, and setting goals, targets, and developing your plans, and entering them into NOVA. And this slide is really just to sort of begin to frame the adult education pipeline roll in here, as Carolyn and Neil identified, the required and optional metrics that you'll need to be acquainted with for planning and reporting. And we are highlighting them, so that you can easily find them on the AEP dashboard. Each step represented here is also a dashboard tile, which Jessica will demonstrate in a minute. And just as a note, we're doing this really quickly. If you're unfamiliar with this, or you would like further information, just feel free to reach out to us. (DESCRIPTION) Title, Learner Journey (SPEECH) This slide takes exactly that. These are your tiles, and then highlighted, you can see where the different metrics that Neil talked about are going to be placed in within the adult education pipeline. (DESCRIPTION) Reading Your Numbers (SPEECH) And before we move into a demonstration of where these are found, I just wanted to give you a quick reminder about how to read the numbers on the dashboard. Here, the denominator identifies all of the possible students who are considered in the count for outcomes. Whereas, the numerator is those who achieve the outcome. So for example participants, those who have achieved 12 or more instructional hours, so the denominator for this would be-- or here on the example on the slide-- is the participants, and then whether they're enrolled in ESL, ABE, ASE programs. Whereas, the numerator will only take those who have actually completed the EFL level by pretest, versus the post-test, or of course progression in the same area, which are the definitions of how you meet that criteria. So basically, it's just like a fraction. And we wanted to make sure that you also knew that the denominator changes based on the criteria of the metric. So Jessica will show you where to find this information on the dashboard. But we just want to be aware that the numbers, how to read those numbers, as they're important as you begin to delve into the data. But at this point, I'm just going to hand it over to Jessica, who will do a quick live walk-through of the metrics, trends, and outcomes, and the drill downs. (DESCRIPTION) Title, Metrics on the Dashboard. A E P 4.0 Live Walkthrough. The metrics, trends and outcomes, and drilldowns. (SPEECH) JESSICA CHITTAPHONG: Thank you, Blaire. Let me take over. And you can access the site by going to this link that I've just added to the chat. You can also just Google Launch Forward Adult Education Pipeline, if you're unfamiliar with how to get to this pipeline. It is an open access dashboard, so anyone can access it. You don't need to log in. You don't need a criteria. So just feel free to use the link, and hop on and explore. So I will be providing a quick demonstration of where to find those metrics that Neil and Blaire had just presented on. If you want a longer demonstration, feel free to go back to the caladulted.org website, and there should be a recording of the webinar we did back on the 30th, where we did a much slower walk through of the pipeline. And so for those who who are unfamiliar with the dashboard, that would be a good resource for you. So jumping right in, this is the home page. And as Blaire mentioned, we have the metrics organized by the different steps in the student journey. So we have the students programs bucket. And we have the progress metric bucket. And all of the metrics will be grouped by those particular categories. To find your institution, feel free to jump to the second box, and just start typing your name. It could be just start typing, and the list underneath will update automatically. You can do the same for your consortia. If you could consortia and start typing, your consortia name should slowly populate here. I'm going to stick with the statewide view for now. And any time you make a selection, please remember to click View, because that's going to update the data that's populated underneath here. A quick note for folks, just because it's useful to remind everyone, is that we do have something called lagging indicators on the Adult Education Pipeline. So for transition and employment earnings metrics, we don't have data for the most recent year, because we either gave the student an extra year to meet that outcome, meaning, we wait a year to see if they fully transitioned. And we also wait a year to see if they end up in the employment and workforce cohort. So that's why these tiles-- they select an earlier year. And if I go back a year, the data will update. And I'll show you some-- it'll will show you the updated charts here. All right. So for the first metric that is mandatory for consortium, we have the reportable individuals, or the adult service metrics. This is a good place to remind everyone that there is a couple of different ways to access the different levels of data on the Adult Education Pipeline. You can go to the AEP scorecard, which is what I dubbed the executive summary of the Adult Education Pipeline. It actually holds a lot of important metrics that have been identified by the state. Most of the metrics that we talk about, and that have been identified to be included in NOVA will be on this page, but not all of them. So on this page, you can get a quick list of the metric, and then the number associated with it. By clicking the name, it actually expands, and shows you a time trend view of that particular metric. On the scorecard, you can also drill down all of the charts by demographics. So you can look at gender, race, ethnicity, and age group by clicking one of these. It will display a bar chart that displays the counts for that selected year, for the different subgroups in that drill down. If I go back home, I can also find the report of individuals, or adult service metrics in the students and programs tile. When I click here, it'll take me to something called the Summary Page. And this page is just an infographic of some key metrics that we've selected. This is for people who don't really want to engage with the metrics themselves, but just want a quick bird's eye view of what's going on. (DESCRIPTION) Clicks Detailed Data (SPEECH) But I want to focus really on this detailed data page. So the detailed data pages is actually where you'll find all of the data metrics that we have available for the Adult Education Pipeline. They're grouped on the left hand side. And you can actually extend and figure out what metrics are where. If I click on enrollment, that's where you'll find the total reportable individuals, or adult serves chart. Here, you'll again see the time trend view, and you'll see the demographic breakdown drill downs, but you'll also see the student type drill down, which is only accessible on the detailed data view. Underneath will also be a data table that shows accounts for the selected years. If I make a selection on the drill downs, the data table will update with those particular data. Also, on this page, is where you'll find the barriers to employment metrics. So we have the barriers to employment for if ever flagged. And that's where you'll see the English as a language learner. You'll see the low income and the low literacy rate. The low literacy barriers to employment, they're posted here. You can also drill this down by program type, as well as by the student type for sign in returning. Below again, is you'll find the data table. And the difference between this table and this next table, barriers to employment, is that early on in the design process, folks recognized that some barrier to employment are long term, and can affect students throughout their educational career, and so they should be counted if they were ever flagged as having that particular barrier. So that's what this chart shows. The second chart flagged in the selected year are for those barriers that may change, or are temporary. And so they are counted. They are recounted each year for this particular metric. And that's where you'll find the long term unemployed barriers to employment. Since we're here, we want to stay in the detailed data pages. But we might want to jump to the next page, which is the next student journey bucket. So I can quickly do that by clicking this arrow to the right, and it will take me to the progress page. And this is where you'll find the next two metrics that will be on the NOVA. These are I think, part of the optional metrics list. So you'll see here the completed one, or more educational function levels, as well as the immigrant integration milestone. And you'll see here there's a big jump because the COAAPs of the I3 report were first introduced in 1920, and so that's when we included the data. And that's why there's a big jump here. If I go back to the educational function levels, you'll see now that I have three possible ways to drill down this data set. I can look at demographics. I can look at program type, as well as student type. Now, these three all play together. So if I click gender, and I click ESL, this will give me account of gender by program of those who gained one or more educational function. Levels so we play around with this, particularly, if you're planning and looking into equity issues, this is a great tool. And it's available to you on the detailed data page. If I jump to the next page, it'll take you to transitions. And that's the next couple of optional metrics that'll show up, is the transition to CTE, and transition to credit college courses. These two are actually subsets of the transition to post-secondary metrics. So the transition to post-secondary metrics has traditionally included transition to CTE, as well as transition to credit. But we for this particular build, we broke that up to maybe support institutions who maybe focus on one over the other. Going to the next page, you'll see what we call the Success Page. And that's where you'll find all of the award counts. So we're counting here students who earned a high school diploma, GED, or high school equivalency. And this is where you'll also find students who completed a full secondary credential. Jumping to the next page, which is our last page on the Adult Education Pipeline, is the employment and earnings section. And so you'll see here employment two quarters after exit, as well as change in earnings. And those are also the last of the two optional metrics that are going to be available on Nova. I'm going to keep it there. Because as Blaire mentioned, we want to focus on our panelists. But I'm happy to answer questions on the chat, and we can circle back to this towards the end if there's time. (DESCRIPTION) Blaire Toso (SPEECH) BLAIRE TOSO: Jessica, sorry. I missed something. Someone was asking-- there are two questions that definitely need to be asked, or one that he wants to know how colleges are recording reportable individual, and then there was also one from John about showing you the technical definition link from the data set page. JESSICA CHITTAPHONG: Yeah, so let me see. So reportable individuals for the college side, there's a couple of criteria that we count. So the first criteria is whether or not they had one, a noncredit enrollment, and one hour of positive attendance. So that's one criteria. The other criteria is whether or not they received a noncredit student service. And then we looked at both criteria, and found statements from there for the college side. What was the second question, Blaire? BLAIRE TOSO: It was about looking at where the technical-- sorry. It's way up there from John. Was asking about the technical definition from the page. And you know what? I'm just going to go ahead, John, not to deflect your question, but I think we're going to have to move on, so I can be sure that everybody on the panel has an opportunity to present. Maybe Jessica, if you can scroll through the chat, there are a couple of questions that still remain. JESSICA CHITTAPHONG: OK. I will do that. (DESCRIPTION) Title, Setting Goals and Targets. Ask good questions. (SPEECH) BLAIRE TOSO: So thank you. We know that we've really packed the agenda. We just sort of wanted to orient people. And again, there are some really great webinars that have been reported about using AEP. And contact us also if you'd like us to do some direct technical assistance, or do a walk-through on the AEP. We can certainly do that as well. So we just want to really make sure that we have the chance to get to talk to the people who are really the ones who are in the field, and who you all will relate to the most as far as what planning process looks through. I'm going to fly through these slides, because we're just a tiny bit behind. As I said, they're all written so you can come back and look at them, and we'll be doing more work on this later in the summer, and in the fall. But we just wanted to really emphasize that in setting your goals and targets, that, you ask very good questions. And we have a series of these. And I'm going to leave it at that, because one of a couple of the presenters will talk about that. But really, you're looking at what is sustainable? What is feasible? And where are the gaps and trends that you want to address. (DESCRIPTION) Text, Talk as a consortium (SPEECH) And then also, we really wanted to emphasize that once you begin to look at these, you look at them both in the way that Neil and Carolyn have identified the metrics, that there are consortium level. And then there are member levels. And that it's important to consider and talk as a consortium about whether the goals and targets are tangible, how you can support one another, and how do each of you as an individual member want to set individual goals, and how they work together to support your consortium as a whole to provide the services. (DESCRIPTION) Title, Additional Data Services. (SPEECH) We also know and suggest that you use a lot of other different resources to collect data. The Census Data, Labor Market Information, Centers of Excellence, have some great pieces. The Healthy Living Index, Poverty Index. And I know that the panelists will talk about more. If you have a moment and you're thinking about it, and you have a favorite data source that you use to inform your planning, please go ahead and put that in the chat, so it can be shared with your colleagues who are also on this presentation. (DESCRIPTION) Panel Presentation. (SPEECH) And that is probably the quickest overview of setting goals and targets. But we're going to lead into that. And each one of your panel presentation members will be able to give you more in-depth information about this through their contextualized examples of their work. And we're going to start with Dulce from North Orange. And then we'll move Ute from East Region, Maryann from Glendale, and John from Sequoias. And we would like to just take a moment and thank them again for sharing their expertise with all of you to prompt discussion, and thinking, and provide strategies that might work for you in your context. So Dulce, I'm going to go ahead and hand it over to you. (DESCRIPTION) Dulce Delgadillo (SPEECH) DULCE DELGADILLO: All right. Thank you, Blaire. So I'm going to go a little bit about the work that we've been doing at North Orange Continuing Education, and the North Orange County Regional Consortium for Adult Education. So next slide. (DESCRIPTION) Title, Data Informed Consortium Planning (SPEECH) So we've been doing a lot of work on a variety of areas. And I'm going to go through the story of what NOCRC and NOCE has done. But really, we started this work a couple of years ago really trying to understand what is our local data. What does it look like, right, our consortium data at a local level. What does it look like within our community level, so that census data. And what does it look like within our institutions, and within our systems that we are collecting this data in. And does all that data live in the systems that we need it to live in, right? In addition, we also did some work around understanding the CAEP outcomes, so really drilling down into the technical definitions, looking at the coding, really looking at how things overlap with what we currently have within our institution, and through other initiatives. And just really getting a good grasp on what those CAEP outcomes are, and how our data are displayed through those outcomes. Then I'll go over into our CAEP evaluation report, which we just finalized this last past year, which was our first year of CAEP evaluation report. We hope to do that on an annual basis. And then finally, I'll do a quick wrap up on how we're just continuing to build capacity around data literacy, really helping our community and the consortium to feel comfortable around data, to feel comfortable around digging into the data, and how we can use data to inform the decision making, and the planing for the next couple of years for our consortium. Next slide, please. (DESCRIPTION) What does NOCRC data look like? (SPEECH) All right, so the first thing I'm going to talk about is really diving in. We did this infographic a couple of years ago to really look at just our community profile. So we provide a community profile in our institutional effectiveness report every year. And we pull our census data according to our census tracts within our service area. And so really, what we're focusing on is, are we addressing the gaps, or what are the gaps that exist within our community, right. So how many students actually have a high school diploma, or need a high school diploma. What is our income levels? What is the distribution of the age population within our community, service area, and race, ethnicity. And so those are the types of components that we really wanted to be able to present alongside our institutional data as well. Next slide, please. (DESCRIPTION) Community Data and CAEP Strategies. (SPEECH) In addition, we wanted to see kind of how do each of the students that we're serving kind of break up into the number of students served by CAEP area, right. So one of the first things that we looked at is, what is the type of census data that we need to look at, and how does it connect to this particular CAEP strategy? So the activities, we refer to them within our consortium as CAEP strategies. And these are really the activities that we're implementing through each of those CAEP areas. And so we looked at-- you can see we looked at our institutional effectiveness data of, this was how many students were served in this academic year by CTE. These are the certificates that we awarded. Just to give our consortium members really a broad overview of, what are those trends typically look like within our institutional effectiveness report within our institution. And how do they kind of fit into those CAEP buckets that now we're going to have to report on, right. In addition, we also looked at just building capacity around understanding the data flow. So you can see in the middle of that second data, or the infographic, that middle box, that was something that was provided to us. I think it was from Neil actually. And to really understand where are all these data sources that are coming from? What is the data flow? Where does it live within our consortium? What do we need to input into these data systems in order to really get, again, a better understanding of how is this data going to be. As we enter the data into the systems, how is it going to shoot out on the other end in the LaunchBoard, right. And really, just already at the bottom piece, setting those expectations around these data trainings, and alignment with CAEP goals. Next slide. (DESCRIPTION) Understanding CAEP Outcomes (SPEECH) So again, now that we have a big picture of our community data. We have a picture of our just overall institutional data. Now, we have to get a better understanding of what are these CAEP outcomes. So again, this is a visualization that our team produced in terms of just breaking down those CAEP outcomes. And this has served as a really good just training tool. We even include this I believe, in our on-boarding package for the consortium, just to give members a sense of, this is exactly what this metric holds. This is what are the things that are being calculated, and this is how your students are going to be thrown into the denominator, or in the numerator, right. Because all of those things are going to matter once you start looking at those outcomes, and those metrics on those launch boards. Next slide. (DESCRIPTION) Institutional Effectiveness Process (SPEECH) All right. And then we also wanted to see what does this look like. What is CAEP in our overall institution look like, right? So last year in our institutional effectiveness report, we actually included state wide initiative metrics in our IER. And we revised our institutional effectiveness process for it to actually have a component of initiative planning around our process, right. So we wanted to see, OK, when we look at our local metrics as an institution for institutional effectiveness, how are we defining students served, and how does that compare when we're looking at these metrics for strong workforce program? How does that compare when we're looking at metrics for adult ed? And how does that compare when we're looking at student success metrics? So that we can, again, have an understanding of where are some overlap on these metrics, and the students, right. Do students fall into all three of these buckets? Maybe we're capturing students in one initiative, and not another initiative, and what is the reason? And how are we facilitating those conversations within our consortium, and with our partners in terms of maximizing the students that we're reporting. Next slide, please. (DESCRIPTION) Annual CAEP Evaluation Report (SPEECH) So that took us to our annual AEP evaluation report. So this was something that we produced in October of 2020. And we were really focusing on these three key valuation questions. So what types of instructional and supportive services are being provided to our students through the consortium? What data elements is the consortium capturing through the implementation of these strategies that are being funded? So please note that when we produce this evaluation report, it wasn't the whole universe of NOCE, right. It was an NNOCRC. It was really focused on how are we spending our money, and what are the activities that we are spending on. And how many participants, how many students, how many outcomes are being produced with those activities and strategies that we are implementing with our CAEP funding? So it was very much a narrow scope tied to how is the money being spent within our consortium, and what are the outcomes that are related to the money that is being spent. And we only produced outcomes in terms of-- or did some reporting on the evaluation based off of those that would continue, right. So even the evaluation report isn't including all of the strategies that we included. It was really a planning document to help with the conversation of what should we continue next year. What do we know is effective in creating an impact from what this preliminary data has. And we do plan on doing this. Another great opportunity here was to get those student voices, right, so have that quantitative and that qualitative data, so we know what the numbers are, but what our students experience is. So being able to incorporate that into a document like this was really helpful in facilitating the conversation. Next slide. (DESCRIPTION) Title, Using Data to Inform Planning Conversation. Student Outcomes. (SPEECH) So this is just one of the tables, or summary tables that we actually provided for our consortium members through the evaluation report. And really, what we wanted to get across this is, again, you can see that these are much smaller numbers than what our consortium actually produces. But again, tying it back to what that funding was, and how can we inform planning in the future. And so for example, as you can see for our students served by basic skills for that 19-20 academic year, we served through the funded activities 218 students. But if you go down to under the headcount of students enrolled in basic skills and CTE, you can see that of those basic skills, students, 13% enrolled in CTE. 1% enrolled in DSS. 11% enrolled in ESL, and none co-enrolled in parenting classes. So really, the purpose of this type of table was to say, basic skills students aren't just being served by basic skills students, right. Our students aren't just being served within one program. It's actually they're taking classes in a variety of programs. And that's really what we wanted to get across with this table in terms of facilitating that type of conversation. Next slide. (DESCRIPTION) Building Consortium Capacity (SPEECH) So the last piece is really about building capacity around data literacy. So one of the first things that we started doing, we've been doing something called Data & Donuts within our institution. And we really wanted to focus on CAEP metrics. So this was one of the first trainings that we did around CAEP metrics, just a broad understanding of what are CAEP metrics, how do they relate to each of the strategies that you're implementing, or thinking about implementing, and how can you assure that the students that are being served through your activities in your consortium are being reported through the MIS system, or through the TOPSPro system, depending on the type of institution that you are. And then again, just building capacity around how to utilize the LaunchBoard. Next slide. (DESCRIPTION) Class Coding (SPEECH) And then our latest and greatest was on class coding. So in November we actually held a Data & Donuts directly on class coding. And I know this is something that's been a hot topic for the last couple of webinars and so we really went into our class coding, our CB file, what is CB21. I know those who are in the weeds of MIS know exactly what I'm talking about. Can you tap it one more time, Blaire? And we even walked through exactly how do you go and find what your CB coding is, right. So giving the resources and the tools for consortium members to go in COCI, which is our curriculum inventory system, it's a public facing. You can pull your coding in there. And we, as technical assistance can be able to assist you in kind of diving into your course coding, and how that course coding is actually being reflected in how those numbers are displayed on that LaunchBoard. Last slide. (DESCRIPTION) What's coding got to do with it? (SPEECH) And so the next thing was, we provided tools for consortium members to really understand, again, what is that CB21 coding. So this is the data elements dictionary straight from the chancellors office, MIS. So just what is the core coding. We even brought in screenshots from Banner to say this is exactly in Banner where the coding goes, right. And how does it connect to the bigger picture of how these students are being placed into these buckets for the CAEP launchpad. So that's really, what is it, why is it important, and the implications. And then one more tap on there, Blaire. And so this was a sample of the Excel file that we provided them. It was a massive Excel file, because it actually has all of our CB coding, but we wanted to make sure that our consortium members knew exactly when you are looking at your class, so we know what is your CB coding. Does this make sense? And what are the implications of that CB coding moving forward? Something that I wanted to end on really quickly is that now, we are very much in the weeds of what that coding looks like. And so we are now in conversations with curriculum committee to have a better understanding before they even get that coding, right. So as they're going through the curriculum committee and they're getting approved, it is our intention to start having those conversations with faculty, with program directors to say, hey, if you code your class this way, these are going to be the implications at the other end on LaunchBoard. So that's just a little bit about what we're doing at our consortium. Thank you. (DESCRIPTION) Blaire Toso (SPEECH) BLAIRE TOSO: Thanks, Dulce. And what a great overview from that first step of making sure you have clean data. You have data that has coding, and then how do you understand it. And someone asked if you would be willing to share your template of the infographic about the CAEP outcomes. That would be lovely. Thank you. There was actually more than one request. So now, we're going to move to East Region Adult Education. And Ute. UTE MASCHKE: Yeah, I wanted to pick up there. We stepped into the weeds of our three year planning process with a summit a month ago. And at that summit, we wanted to take next steps toward thinking system wide. As Neil mentioned earlier, how can we align metrics across systems becomes more and more important. So for our region, that matters quite a bit. We cover a large geographic area, but only three members in our consortium. And one member actually serves about 97% of our programs and students. Next slide. So we wanted to use, pick up the pipeline, the Adult Educational Pipeline and use it as a tool for starting to build the collective impact ecosystem. And use the data provided, the visualization provided, the summaries provided to establish a common framework of reference. This frame of reference has to be one for us that brings community college partners on board, that brings our San Diego workforce partnership on board to create that ecosystem, in which our consortium plays a significant role, but not the only. So the pipeline is actually the tool we are using to share agreed upon metrics, to share common language, to build out that language that will then allow us to explore structural issues together. We started this conversation with actually reviewing where are we at, and what is it that we're talking about. Next slide. And that led to a review of, and agreement on a student success metrics that we decided to align with the Guided Pathways Initiative, in which we find the most productive way to move forward in our communication, which then leads to coordination and collaboration. As you can see, and you're probably all familiar with these pillars of the Guided Pathways Initiative. We aligned there with the connections entry, progress, and transitions. And agreed as an ecosystem, that we want to build this together. These are the metrics we want to focus on. So we have now a tool, if you will, and guidelines for coming together on a regular basis to look at data points, and decide on next steps together. There was one key issue for us. And it came up earlier already a little bit. We needed to articulate good questions. Next slide. So for each of these metrics, we designed guiding questions, where you on this slide you see the guiding questions for connections, and for entry. These guiding questions are now shared across partners. So I can turn to for example our Regional Career Center. And we can use the same types of question to dive deeper into data. For us, this is incredibly important, because we are just this very small consortium. And have other career training providers who play a significant role in our areas. Next slide. (DESCRIPTION) Guiding Questions (SPEECH) We did the same. We developed the same guiding questions for our progress, and for our completion and transition reports that help us to define next steps in our planning process. The tool, the visualization that the pipeline provides is turning out to be very helpful, because it allows us to share out using one voice what we want to measure. And it is for us, also, the tool explain to become more visible in the region, and actually help us to explain to someone who might not be a stakeholder, but an actor in the region, to understand what adult education actually is about. Where our language might not always be the one-- the language that our actors in the region understand, by using the pipeline, we actually initiated the data dialogue that is now reaching beyond the consortium, and into other stakeholders. And with that, I want to leave it at that, because we don't have much time for the other presenters. BLAIRE TOSO: Thank you, Ute. There's thought to this, where we were talking about the data, and now, we've moved into the guiding questions. I appreciate you giving that context. And now, we're going to move on to Maryann. (DESCRIPTION) Maryann Pranke, Glendale Learns Coordinator. (SPEECH) MARYANN PRANKE: Thank you. And we can just go ahead and go into the next slide. Just a little bit about our consortium, there's three members to our consortium. And Glendale Community Colleges are our Adult Education Provider, and is also our administrative entity. We also include the Verdugo Workforce Development Board as a member, as well as our k-12 school district. Now, my role is a little bit unique in that I am the Glendale Learns Coordinator. But I'm also staff to the workforce board. So throughout the years that we've been partnering in this way, we've really tried to capitalize on having this integrated partnership. And we are working to integrate the workforce and education system. And one area where we have learned that it is an advantage is in planning. Because on the workforce board side, I am responsible for developing the four year local workforce development plan, and contributing to the development of the regional plan for the Los Angeles basin. And then on the CAEP side, I'm responsible for writing the three year plan, and the annual plans. Now, of course, I'm saying that I'm writing these plans. However, as you all know, it takes a lot of input, a lot of data and information, and a lot of input from partners in order to write those plans. So I kind of see myself as the journalist that kind of puts a story together. In the next slide, when I start planning the facilitation of the planning sessions, what we do is convert our monthly meetings into working meetings and planning sessions. So at each meeting, we bring in the data sets, including our environmental scans that include the economic landscape, and the demographic data, and so forth. And we use the information that is developed from our workforce board. We could develop our own, however, it's already there. They already did it. They already aggregated and analyze the data, and put it into graphs. So I can just pull it and use it in our planning. However, to bring it home, I use the LaunchBoard graphs, the LaunchBoard data. I rely on that to help me facilitate the strategic planning sessions. And I'm not just saying that because I'm on this panel. But I really do. What I do is when I'm coordinating and planning my session, as I go through the LaunchBoard data and look at all of the graphs, and then I go in and pull the ones that I want to use for facilitation to generate input, as well as identify specific issues. And the LaunchBoard data provides us with that overview of what's happening in our program. So I just go in and use my snip tool, and snip the graphics, and pull them into slides. So just the fact that if it wasn't for a LaunchBoard. I would have to be aggregating this data, and developing these graphs. So just the fact that I don't have to do that saves a lot of time and effort, And one of the reasons why I'm a big fan of LaunchBoard. Also, they're very user friendly. So as you can see from the slide, very basic graphs. And you don't have to be in the college system or in the education system to understand what's happening here. So with our monthly meetings that we've converted into planning sessions, we have about 15 partners that participate in all of our meetings and planning sessions. And the majority of them are not from the college system. They're not even from education. They're either workforce or community organizations. So it's important to have these the slides that are overviews that can spark input, and they can contribute to the information. They can contribute to the discussion. And you don't have to be from the college system in order to understand what's happening. For purposes of this presentation, I'm just going to focus on the ESL. Because in this graph, the ESL-- and by the way this is 18-19, because this is the graph that we used-- or a slide that we used in a previous planning session. So of course, now, you have access to 19-20 in LaunchBoard. So the ESL graph shows that-- or the bar-- shows that 70% of the students enrolled in our adult education program are enrolled in ESL. Which looks good, that means it's in demand. Yes, there's other opportunities for improvement. We can always outreach and increase that number. However, this looks like our ESL does have a demand. Now, going on into the next slide, this is another data set that I pulled right from LaunchBoard. (DESCRIPTION) English as a Second Language (SPEECH) And this shows the trend over time, which shows that our ESL enrollment has been increasing over time, except in 18-19 it dropped a little bit. It dropped about four percentage points. So well, that's interesting. And if once I do this, and include 19-20, we're going to see an even bigger drop because of the pandemic. However, this is pre-pandemic. And now, this is where the data on the right hand side is really important. Because if you look at 18-19, yes, there's a four percentage point drop. However, if you look at 17-18, you can see that you can't really tell from the graph, but you can see that the dip actually started in 17-18. So this is important in Glendale, because as partners, we have been expanding access to ESL. We have been expanding the offerings of the times that they're offered, the locations where they're offered, trying to increase that accessibility, and make it really easy for people to enroll in ESL. And it seemed to have been working for some time. And then all of a sudden, there's a dip. So what is happening? What's going on? On this planning session, if you go on to the next slide, we wanted to look at, what is our target population? What does it look like? And what are we really looking for in Glendale? And this is where I start pulling in data from the local workforce plan of our workforce development board. Now, it's easy for me to access that data, because I've created that data. So it's in my computer. However, just to let you know, and save you time and hopefully, you won't have to create your own environmental scan, the local workforce boards are required to post their local workforce plans on their website. And they're also required to include the regional plan. So many local workforce boards won't include their environmental scams in their local plan, but it is in their regional plan. We just completed those plans and submitted them to the state on April 30. So they are available now. They're public documents. And you can go in and pull information from that. So looking at our information here, and the Verdugo Workforce Development Board, they are a tri-city consortium. So they have Burbank, and locking out at Flint Ridge. So you'll see that in these particular tables. But we're focused on Glendale. And on Glendale, to understand what the target population looks like, we look at the information on the number of residents that identified as speaking English less than very well. And in Glendale, there's over 65,000 people who identified in that characteristic. Probably about 20% of this 65,000 are children between the ages of five and 17, inclusive. So if we pull out that 20%, we are still looking at about 52,000 residents that identified as speaking English less than very well. So that's still a pretty good sized population that we can target in order to recruit and enroll in our programs. We use other data sets to also get a picture, and kind of paint the profile of our target population. And so one of the data sets we used was poverty level. And this is characteristics of people who speak a language other than English at home. And what we find is, that our English language learner population, a significant portion of them are at, or below the poverty level. So this is definitely evidence of a link between English language learners and low income. So when we're looking at barriers, we're not just looking at the English language as a barrier, but low income. And if when we pull in our education levels, we'll find that there's also a link between English language learners, and the population that has less than a ninth grade education as well. So literacy, low levels of education also barriers. OK. So now that we have a little bit of a profile of our target population, we want to look at, OK, how are we going to address these needs? And how are we going to translate this into strategic goals, and objectives, or activities. And so on the next slide, I'm just giving you an example of how these in particular were addressed in our logic model taking our activities. So first of all, we thought, OK, these are low income individuals. We want to help them develop the technical skills that they need in order to create the opportunities for them to earn higher salaries. And hopefully, start them on the path to break that cycle of poverty. So we want to focus on career pathways, specifically for our priority populations. English language learners is one of them, as well as individuals with disabilities, in particular, those with intellectual disabilities and developmental disabilities. So that is one of our activities. Looking down at the 2.6, we also have where we want to integrate ESL and basic ed into our career pathways, in order to bridge that gap. So they're learning that technical skill at the same time that we're addressing the ESL and literacy gap. And then in order to motivate them, in order to help motivate to recruit them and enroll them, we want to tie them to workforce. And we want to tie them to paid work based learning, that's on the job training and work experience, which could be paid internships, and paid externships. And we use our workforce board to help us with that. So those are the activities that were in our plan, and now, translating that into a program. In the next slide, you'll see an example of one of our programs that we developed, which is our medical assistants program. It has a dual credential and administrative medical assistant, and clinical medical assistant. We incorporated the vocational ESL, so that we could target English language learners for this particular program. And then we work with the workforce board to offer the work-based learning, and this is paid work based learning. So they're using their connections with employers to create the opportunities for internships and externships. And they also run the payroll. And we're able to co-enroll with WE 001 from this program, so they can access those services. And then our workforce board also provides case management and job development, as well as the supportive services, which were absolutely critical during this pandemic. Many of our students were able to access just assistance with living expenses. Some of them were able to get reimbursements for their rent, their mortgage, even health insurance premiums. So with that co-enrollment came so much benefit to our students, and they were able to stay in the program and complete the program. So that's just a brief walk through and one example of how we use LaunchBoard data, integrating the data from our environmental scan, and then translating it to goals, and objectives, and into program. So at this point, I think that's it for me. At this point, let me go ahead and turn it over to John. (DESCRIPTION) Blaire Toso (SPEECH) BLAIRE TOSO: Thanks Maryann, for demonstrating how the data informs decisions and setting targets. And yes, John is going to put a bow on this. And I just want to say that we're right up against our deadline. I think we'll be able to keep the line open. So we'll make sure we have time for discussion. But we also understand that people may need to get off at 2:30. Thank you for your patience. And thanks Maryann. John, on to you. (DESCRIPTION) John Werner (SPEECH) JOHN WERNER: Thank you, Blaire. So I don't have a bunch of slides to show. I think the field here has gotten perfect visuals throughout the panel presentation from Dulce, and Ute, and then Maryann here. I think we all want a copy of North Oranges planning document that they used to work with their teams. I'll leave you with a few recommendations here, I think. One is, know what the data means. Dive deep into those data, the pearls, those data dictionaries. And the adult ed source that we were looking at earlier does have some technical definition links in it that can help you out with that. And that will help you do-- I think I heard Dulce talk about that, kind of knowing where the data resides in your system, so you know how you're structuring your strategies to influence that. And that's a pretty important piece. Now, that's getting really technical. That's kind of beyond the philosophical questions that you have as a region in planning that gets down to really that data expert, technical hat that you need to wear. And in order to help support you with that, there's a couple of places I go. One, I go to my friends. I go to my region's super consortium members, and I work with them. And then I also spend a lot of time working with TAP, and participating in the technical webinars to understand that. So I want to leave that kind of idea. Know what your data means, and what it's worth deriving from. You got to know what that denominator is, and what that numerator is, and know the story behind it. Otherwise, you just got a lot of good charts and graphs. And there are resources out there to help you with that. If you haven't started planning on your three year plan yet, I'll be honest. We haven't formally waived our start of the race flag here in my consortium. We have started with a lot of these same activities where we're looking at the online data sets. We're looking at our own local data sets. We're having our own local community meetings and conversations. But if you haven't started that, now is a perfect time to do it. You still have a whole year to get it done. I think it's got a due date. We saw earlier of June 2022. So there's lots of good time to get participation from your stakeholders, and make sure you give everyone plenty of time to weigh in, including your local workforce development friend. And I'll keep it short at that, Blaire, because I know we're short on time. BLAIRE TOSO: You did a beautiful job of summing everything up, John. You really did put the bow on. And I want to thank all of our presenters for really speaking to the issue, for taking those sort of theoretical issues when we talk about a numerator or denominator, or good questions, and using data to inform decision making. Thank you for putting them right into place. I'd like to offer a few moments if there are people who would like to pose some questions, or asked to some provide some additional information, or some suggestions about how they do planning for the three year plan. Feel free to open up your mic, or put it in the chat. And we'll address those now. (DESCRIPTION) Title, Discussion. Questions and Answers (SPEECH) All right. Well, it looks like all questions were answered. So thank you. But please keep Jessica posted in the chat, or email that you can reach out to us. And I'm just going to walk through the slides very quickly, what remains. I'm not going to go through them. I just want people to know that they are there. Again, they're reference points and they support the general ideas under which the panel presenters gave us really excellent, concrete examples of how this works. And this is really talking about reviewing and evaluating your data, that you ensure you have a data collection entry and review plan, that you go don't just stop once you have the plan. But there's something where you can review and provide your ongoing continuous improvement, so that you review your data. But then there's also these different ways that you can check your own data, and what's being produced if you're a community college, your COMIS Datamart will help you on that. And for institutions reporting into CASAS TOPSPro, you have your CAEP quarterly reports. And then also, check your assumptions, and your records, and your reports to verify your outcomes, right. And then create that plan to rectify issues, revise and submit data in the timeline that's provided for you. (DESCRIPTION) Tips for Good Data Collection (SPEECH) And then tips for good data collection. We really want to ensure that your participants appear in the data. So a couple of those really key pieces are just entering your four key pieces of student information, first name, last name, date of birth, and gender. I can't reiterate that enough. And then there's some other tips about how you ensure that those are correct as well, such as confirming the names across. Because we track across institutions to try to confirm that they are using the same name that they used at other schools. We know that each of these are problematic, and you probably already have strategies. But this is really key for getting your students represented in the data sets. (DESCRIPTION) Title, Data Tips (SPEECH) And then we addressed a little bit of this. I think Dulce really pointed it out about coming from becoming familiar with the metrics, and the naming conventions. Jessica posted one of our last webinars about what's coding got to do with it. Check that. We really went into that in some depth. And again, also reach out to us. We provide some very tailored technical co-assistance to help you navigate some of this work. And so again, just a few more tips that will really help you understand, and make sure that your learners and their outcomes are being captured. And we've identified just the ones that seem to appear most persistently in the questions that we receive. And I did not think we're going to do this. Thank you everyone for being concise and presenting the information that we had agreed upon. We do have a few minutes for questions, or discussion points that you would like to make, any unanswered questions that went into the chat that you would like to bring forward. Presenters, you all were very short. Is there anything that you wanted to say that didn't that might resonate with the audience? NEIL KELLY: Blaire, this is Neil. BLAIRE TOSO: Yes, thank you. NEIL KELLY: I just wanted to add that if anybody had anything-- so we're still developing the guidance. So if there's something really special, or important I don't know-- we can't wave the magic wand, but if there's something you think we should put in that guidance, or something that wasn't clear the last time, feel free to put it in the chat, and we will examine it. So you're getting a freebie today. BLAIRE TOSO: Thank you. We're getting some thank you's in the chat. And yes, I think this is a great kickoff to a year of planning ahead of us. And I wanted to reiterate that as Neil laid out, there's a lot more information to come. And we'll keep you in touch with. SCOE TAP will be sending out the information there, your conduit to what's upcoming for TA. And then feel free to reach out to the WestEd team if you want some different, or additional more in-depth information. Neil, did you have any last words that you wanted to say to spur people forward? NEIL KELLY: I think Carolyn had to go to another meeting. But from the state perspective, we want to make this as easy as possible. We're introducing goal setting and targets, something we haven't done. So we want to work with you on that. We don't want to make this sound like punitive, or it's going to come back to haunt you. We want you to use this as a tool to get better. And so that whole mantra of continuous improvement, we'll continue to stress that, and not make this something that people are afraid of, or shying away from that. It's all about moving forward, improving your processes, and your student delivery systems, and stuff like that. So look for a lot more information, webinars, workshops, on strategies. Look for communities of practice. Look for a lot of these data dives from CASAS and WestEd. So we look forward to this upcoming year, and this new beginning of three year planning, and goal setting, and targets. BLAIRE TOSO: All right. Well, I think Neil summed it up for the good of all. And yeah, it's exciting to move forward. And I see a question about whether there was the share from Dulce. I will see if we can get that sent on, and then we will send it on to SCOE TAP and they can post it to the website if that's all right. (DESCRIPTION) Jessica Chittaphong (SPEECH) JESSICA CHITTAPHONG: Sounds good. OK, well I think with that, we will just thank everyone who has stayed. We have posted the evaluation for this session multiple times. So if everyone will please take the opportunity to fill that out. A recording of today's session will be shared tomorrow, along with the handout. If we have Dulce's handout at that time, we'll go ahead and share that forward as well. And be on the lookout. We have a couple more webinars coming up. This Friday, May 7th, Utilizing SB554, presented by Matthew Morron with Chaffey Adult School. Next week on the 12th, we are going to have WestEd back again. I believe, Why Do My Data Reports Look Different, AEP dashboard, and CASAS TOPSPro Enterprise. And then we've got multiple more coming up, May 19th, What's It All About, CB21, The NRS Educational Functioning Levels and Curriculum Alignment. So again, we thank you all for coming and staying with us today. If there's anything else that we can help you with, you can always email us. And unless there's anything else, we'll go ahead and close it out. All right. Have a great afternoon, everyone. Thank you.