(DESCRIPTION) On a side, a presenter speaks from a video call tile in the upper right corner ablated, Will Neddersen. On the slide, three presenters smile in headshots above their names and titles. A QR code sits in the left side of the slide. Text. South OC Education for Adults YOUR GOALS , YOUR WAY. Presenters. Karima Feldhus, Saddleback College Extended Learning, k feldhus at saddleback dot EDU. Nancy Miller, Miller Education Consulting, nancy ewban kmiller at comcast dot net. Will Neddersen, Tustin Adult School/ Tustin USD, w neddersen at tustin dot k12 dot C A dot US. (SPEECH) WILL NEDDERSEN: Hello, everyone. We're glad you're here with us today. Very quickly we're going to introduce ourselves. And I would like to start with my colleague, Karima, if you could introduce yourself. (DESCRIPTION) Karima appears in the video call tile. She wears a red top, glasses, and has short brown hair. (SPEECH) KARIMA FELDHUS: Good afternoon, everyone. Thanks for joining us. I'm Karima Feldhus, and I'm the Executive Dean of Extended Learning at Saddleback College and I'm also the Co-Chair of the SOCRC. WILL NEDDERSEN: Nancy, would you please introduce yourself? (DESCRIPTION) Nancy appears, with short blonde hair and a black sweater. (SPEECH) NANCY MILLER: Hi, I'm Nancy Miller. I am the consultant with the South Orange County Regional Consortium. And prior to retirement, I was the Director of Sonoma County Adult Education and SRJC Adult Education. (DESCRIPTION) Will reappears. He has glasses, headphones, and a plaid shirt. (SPEECH) WILL NEDDERSEN: And then I am Will Neddersen. I am the coordinator for Tustin Adult School and Tustin Unified School District and also a Co-Chair with Karima for the South Orange County Regional Consortium District, or if we say it, SOCRC. So we are, again, glad you're here. Forgive me, I'm just reminding housekeeping one more time. (DESCRIPTION) Text, Housekeeping: Please mute microphone. Please use the chat for questions. There will be check in points to ask and answer questions. Please feel free to share how your consortium assesses member effectiveness or agency performance during share out time. A photo on the slide shows a group of five people posing behind a tented booth with a red and gold Saddleback College Adult Education banner over the table in front to them. (SPEECH) We will be checking in with the chat. So if you'll use that for questions and hopefully getting to hear from you or ask you some questions towards the end. So please, again, be ready for that. And with that, our focus for today-- again, this is part 2, so our focus for this part 2 is on SOCRC's member effectiveness journey. (DESCRIPTION) In a photo on the focus slide, a large group of people pose in dark blue nursing scrubs, some holding pill bottles in the air. (SPEECH) We're going to be honest with you. It's been a journey, not something quick and easy, but good work to be put in there. So you'll hear more of that. Then we're going to share with you some tools to monitor member effectiveness that we've created for our consortium. And then last but certainly not least, we're going to talk about incorporating member effectiveness into bylaws, which really is a key and important piece to understand is you need that to be a part of your bylaws. (DESCRIPTION) Text, SOCRC Shared Resources during this presentation: 1. Quarterly Fiscal Report. 2. Directions for Quarterly Fiscal Reporting Template. 3. Facilities, Classroom, Technology and Equipment Assessment. 4. Self-Evaluation Tool for Agency Effectiveness. 5. Evaluation Measurement Metric Document. 6. SOCRC Bylaws 2024. QR Code with resources at end of presentation. In a photo, Will smiles in a tented booth. A blue banner on the table in front of him bears the Tustin Unified District Adult Education seal. (SPEECH) And the other part of this is I don't know about you, but when I come to presentations, I love receiving things to be able to use as tools. And that's our goal today is to share with you some tools. So you will be seeing links shared in the chat as well as a QR code at the end if you don't want to click on links because it gets a little bit confusing. But we're going to share with you in the sequential order. You'll see it 1 through 6. The folder that's being shared will have documents numbered 1 through 7, 7 being this presentation. But we will share these items with you as we get to them, discuss them. And know that they're in a Word form or a Google Doc form that you can then take, alter if you would like to use. Again, totally optional, but we'll give you that insight. (DESCRIPTION) Text, South Orange County Regional Consortium. Funded Members: College and Career Advantage, Irvine Unified School District, Irvine Valley College, Laguna Beach Unified School District, Saddleback College, Tustin Unified School, District. Non-Funded Members: Coastline ROP, Capistrano Unified School District, Orange County Department of Education, Saddleback Valley Unified, School District. (SPEECH) With that, I am going to hand this over to Karima. KARIMA FELDHUS: Well, thank you, Will. Those of you who joined us last week, we covered this and we thought we'll repeat this information in case we have new members joining us today. So quickly, the South Orange Regional Consortium consists of 10 voting members, six of whom are funded members and 4 are non-funded members. As part of AB86-- and you have the list. Just for the sake of time, I'm not going to read the list of all 10 members. But as part of AB86 or AB86 planning process, Saddleback Valley Unified and Capistrano Valley Unified agreed to turn over their adult schools to Saddleback College. So as a result, Saddleback College has a little over 50% of the funding for the consortium and that's because it took over the two K through 12 feeder schools funding that came with this adult schools. With that, I'll turn it over to Will to discuss the specifics of the funding. (DESCRIPTION) Text, Member Funding. How is funding being used? Changing Fiscal Administration, Reallocation of Fiscal Agent funds, Practice of collaborative funds and sharing carryover, Funding Advocacy. On the slide, two women sit behind a booth table with a white Irvine Adult School banner. (SPEECH) WILL NEDDERSEN: So funding and connecting member effectiveness, the two important pieces, right? We need the funds to be able to implement programs, but we also have to understand if the funds are being used effectively. And so that's where our consortium really started in trying to discuss member effectiveness was looking at a vote about funding in those pieces. So we originally were a consortium that was a fiscal agency where our group agreed to put the money into one area with our community college district and then have the members funded from them. In our journey of trying to receive funds and be effective with our funds when we receive them, we went through a journey of changing from a fiscal agency to direct funded agencies. So know that that change in us brought out a component of the money that was set aside for our fiscal agency, the community college district, to manage and operate what we were going to do with those funds. We as a consortium do not have a director position. That's why you hear us talk about co-chairs. So how do we manage those funds? And the conversation in that funding was how do we redistribute those funds or our one primary agency that has a little bit of extra funds added to them to help with some of the management components, as well as working with the co-chairs. With that, also brought the understanding of, hey, there's a pot of funds that are a little extra. And what are we going to do with those funds, and how will we collaboratively share those funds? And that was something that we've created in process that we'll talk a little bit more about. And then advocating for our funds is really an important piece of dialogues that come into our general sessions and work together. So with that, I'm going to hand this over to Karima to talk about member effectiveness. (DESCRIPTION) Text, 2019 SOCRC Bylaws: Member Effectiveness. Article IV-Executive Committee. The voting structure will be reviewed annually in the spring. Maintaining one vote per member is contingent upon effective and consistent participation in the work of the Consortium. The Executive Committee will evaluate the effectiveness of member participation. Member effectiveness will be defined as 1) regular attendance and participation at planning and quarterly meetings; 2) participation and contribution to the development of the annual and three year plan; 3) financial expenditures are aligned to annual and three year plan goals and CAEP guidelines; 4) funds are expended in a timely manner in accordance with CAEP guidelines. A process will be developed, as needed, to determine member ineffectiveness and interventions in accordance with AEP guidelines and EC 84914. (SPEECH) KARIMA FELDHUS: Well, thank you, Will. We thought it would be good for us to review quickly parts of our 2019 SOCRC bylaws member effectiveness before we share with you our newest bylaws from 2024. So we have a screenshot of this language here because with that to point out some of the even negative language, right? I'm not going to read this for you. You have the language in front of you, but I'd like to draw your attention to the word "ineffectiveness" at the bottom, the last two lines. So it was a little vague and it was negative. So that's what we had in 2019, and you see the big difference where we are. Is it five years or 4 and 1/2 years later? With that, I would like to invite Nancy to dive deep into what we're doing now and what we've done the last several years. (DESCRIPTION) In a photo on the Member Effectiveness slide, a man and woman smile at an outdoor booth with a CCA banner. Three silhouetted figures join under a graduation cap in the CCA logo. (SPEECH) NANCY MILLER: Thank you, Karima. Because we knew that we wanted to develop the tool that was outlined in the 2019 bylaws, again, we went to the use of subcommittee so that any member who wanted to participate, whether they were a voting member or a member that regularly attends the general membership meetings, we wanted to have their input on, what does member effectiveness look like? Were the four criteria enough? Did we need more? And then, how are we going to measure each one of those criteria? We did not do this in general meetings. Subcommittee work probably took 20 hours at least to develop the shells for the tools, and then we went back to polish them up. But this really needed more attention than we could give it in the general meeting. So that's why we used a subcommittee, similarly to the process to develop the bylaws. We developed three specific tools. We're going to go in order of how we started using the tools, but the three tools that you'll get today along with directions for the quarterly fiscal reporting are the member self-assessment. That was the first one that we actually developed, but the first one we started using was our quarterly fiscal reporting. And then as part of member self-assessment, we were utilizing the facilities classroom equipment and technology assessment that will show up in a couple of different ways. And we'll discuss all of these tools later. Again, in linking the tools to the annual and three-year plan, before we even started the development of the member effectiveness tools, we actually came up with a list of metrics that we would be using in each of these tools. The metrics for evaluation are included in your folder, and I think that they are document 5 or 6, number 5, I think. So you can find and look in detail about the metrics. They consisted of the bylaws, the annual plan, and the three-year plan metrics that were set out by the state but selected by the consortium. So let's see. Karima, it is back to you. (DESCRIPTION) Text, Quarterly Fiscal Reporting. Completed each quarter when agency submits in NOVA. All funded agencies share in General. Consortium Meeting prior to consortium certification in NOVA. Student Enrollment numbers and sections allow for big picture of all served by consortium. Tying everything into the Annual and Three-year Plans. A screenshot on the slide shows a NOVA page with the heading, 62 South Orange County Regional Consortium (SOCRC). A table titled, Allocations, shows allocations in dollar amounts for 2023 to 2024 for six member agencies. Text under the table reads, Total Allocated to Members, $5,918,322. (SPEECH) KARIMA FELDHUS: Thank you, Nancy. So the first tool-- this is actually a three-part fiscal reporting template that is filled out quarterly. Funded consortium member present this information at our monthly consortium meetings prior to the consortium certification. As you know, there is a 30-day window between the member quarterly report due date and the NOVA-- in NOVA, rather, and the consortium certification. And we hold our meetings sometime in between and we make sure that all agencies, funded agencies, present their information before we as a consortium certify the quarterly report. And we've had actually to schedule a special meeting to accommodate agencies that miss the presentation meeting. It didn't happen overnight. Of course, it happened very gradually. We were more forgiving at the beginning. And now we're at the point where if an agency does not present their data at a meeting, then we as a consortium will not certify it, certify the quarterly report. So part 1 is self explanatory. Funding members enter their amounts as they submit in NOVA. Part 2 asks about the staffing in place to support adult education programs. This is work in progress and we continue to refine it. Some agencies actually only reported permanent CAEP positions, for example, and didn't enter all the staffing that support the program. So again, we're refining it. (DESCRIPTION) A slide shows the part 1 and part 2 and 3 pages of the 2023/24 Quarterly Fiscal Reporting Template. Part 1 has the heading, PART I: Expenses to Date. Numbers in your expenses to date should come directly from NOVA. A table on part 1 has columns titled, Category, Total Allocation, QTR 1 Expenses, QTR 2 Expenses, QTR 3 Expenses, QTR 4 Expenses, and Percent Expense of Category Allocation. The Category column is filled in with values from 1,000 to 5,000, with increments of 1,000. Rows below 5,00 say, Subtotal, Indirect costs, 6,000, Totals, and Targets. In the Targets row, 20% is filled in for QTR 1, 40 for QTR 2, 60% for QTR 3, and 81% for QTR 4. Text below the table reads, If percent of cumulative expenditures for the reporting period to total budget (allocation) is less than expected target, please provide an explanation of how your agency will expend funds to reach the target in the next quarter. Part 2 reads, PART II: Adult Education Staff Members. Please list all staff members working in adult education regardless of funding source. Staff member percentage of salary funds in CAP should correlate to the NOVA budget. A table below has columns titled, Number of Employees, Summer 2023, Fall 2023, Spring 2024, Totals, and Percent of Expense charged to CAEP funds. A blank column sits to the right. The number of employees column is filled in with rows that say, Admin (1000), Classified (2000), Instructional FT (1000), Instructional PT (1000), Temp or Student Employees (2000), and Totals. The totals row for Percent of Expense charged to CAEP funds is blacked out. Part 3, reads, PART III: Sections and Enrollment Please refer to the directions page for description of categories. Students may be duplicated between programs. Below, a table has subdivided columns for sections and enrollment in Summer 2023, Fall 2023, Spring 2024, and Total. A column on the left titled, Categories, is filled in with rows that say, in part, ESL/Civics, ESL specialty, IELCE, A B E, and A S E. The bottom of the table is cut off. (SPEECH) Part 3 asks about classes offered by programs as well as the enrollments. So the challenge for some of the agencies in filling this out-- and these are six funded agencies. So the challenge in filling this out is that our classes are based on a semester, for example, at a college and not a quarter. Another challenge is that some agencies use TOPSpro to report the data and others use MIS. At Saddleback College, we use both because we're a WIOA funded school. And of course, student enrollment numbers and sections allow us as a consortium to have a higher picture of all students served by the consortium, culminating into a discussion about outcomes as we work on the annual and three-year plans. This information also brings a discussion of how much it may cost per student to run the agency's program. CTE programs tend to be more costly, for example, than language classes, ESL classes. I'd like also to note that not everyone participated at the beginning. So it took us years to get where we are now. And the change, as I said earlier, didn't happen overnight and certainly AB 1491 helped us in this journey. It took us several years, again, to get where we are. With that, I'd like to invite Will. WILL NEDDERSEN: Karima, I just want to add here as well that as you talk about our journey, some of it was a change in administration in some of our agencies in understanding how we shared out that information, not just uploading into NOVA where we vote on it to say, OK, yes, we approve it, but really being able to process what we approve. So that really is where part of this journey came. And I think for all of these documents, and this being one of them that started up first for us, was because we were trying to process that sense of effectiveness and finding numbers when we couldn't get numbers from everybody at one point. So I think that is an important piece to remind ourselves. As you go in that journey, what is it that you want to understand and process together as a consortium? Especially for us, we've put a big emphasis on completing this and sharing it out before we approve our quarterly components in NOVA as well. So it's really an expectation that we've set. So thank you, Karima. (DESCRIPTION) Text, Facilities, Classroom, Technology and Equipment Assessment. Completed once a year in February. Process for agencies to assess the content and condition of classrooms and equipment. Can be used to determine one-time funding needs. Tying everything into the Annual and Three-year Plans. In a picture, adult students sit at desks in a classroom, arranged in groups of four. (SPEECH) In moving into another component for us, when we looked at the fiscal side of it, you heard me share about changes to the way we were funded from being a fiscal agent to a directly funded agencies. It brought a conversation about an excess of funds or members who might have had some carryover that had been carryover for a few years. What are we going to do and discuss about looking at those resources, and can we support the consortium as a whole and linking those resources? And one thing that came up was the understanding of what are we using as a consortium in the sense of facilities, classrooms, the technology and equipment that's being used in classrooms to really truly understand the needs as well as what is being used so that we as agencies could share together some of the technology updates that other agencies had done. But this also brought out the idea of being able to look at, when talking about needs, we could represent it in an assessment of what we have. So we as a consortium is set up where we complete this every February so that way it gives us a sense as we're looking at funding, hey, if we're reporting, there's needs in classroom or facilities or the need to upgrade technology. This report is already done to be able to look at and reflect together. Also, we look at it as an opportunity to really say what areas need a component. One of these areas was-- one of our agencies needed classrooms. And so funding was shifted through looking at this report, funding was shifted to them to be able to purchase some portables to place on that and have that conversation. And then it really helps because again, the funding isn't something that we always have carryover on, but it's that idea of a one time. So that gives us that quick view that we can look at the six agencies. And then we tie it in as part of member effectiveness to hold it with the next tool that we share. So if you look at it, it's a pretty simple yes/no kind of question/answer. (DESCRIPTION) The document is titled, Facilities, Classrooms, Technology and Equipment Assessment. It has blank spaces for Agency and Date of Assessment. Facilities questions with yes and no listed next to them say, 1. Facilities are located centrally and easily accessible to A student population. 2. Facilities and surrounding area have adequate lighting for evening classes. 3. Facilities have adequate parking or public transportation accessibility. 4. Facilities are in good repair. 5. Facilities are able to house both classrooms and administrative offices. 6. There are bathroom facilities easily accessible to A student population. 7. Facilities have HVAC systems and they are in good working order. Text, If no to any of the above, please describe agency needs. Below, a list of Classroom yes or no questions reads in part, 1. Adequate number of classrooms to support AE programs. 2. Adequate number of desks/tables/chairs for students in program. 3. Adequate whiteboard space to conduct A lessons. 4. Adequate space to house technology and equipment needed for programs. (SPEECH) And then it gets into the sense, do you have needs from the above places? So it's just that sense for a self-reflection of what is owned by an agency and possible needs. So that is the tool that we use for really evaluating needs through facility and equipment that maybe others can support or just allows us if we're needing to advocate as a consortium and looking for future things that's that dialogue as well. So with that, I'm going to pass this over to Nancy. (DESCRIPTION) Text, Self-Evaluation Tool for Agency Effectiveness. Completed once a year by each agency after Q4 submission. Assessment results discussed at Consortium General Membership Meeting. Links to the Annual and Three-year Plans. A picture on the slide shows a teacher gesturing to a smart board in a classroom of students. (SPEECH) NANCY MILLER: OK, so the self-evaluation tool, that one is overarching. So we use both the quarterly fiscal reporting and the classrooms facilities technology, as Will mentioned, as part of the self-evaluation tool. This one is completed once a year after the submission of quarter 4. So once you have all your information, everything's in NOVA, all of your fiscal information, everything is in TOPSpro if you're a TOPSpro user or submitted through MIS, so MIS, then this particular tool is filled out and we'll go through the specifics on it. This year was the first year that we did this self-evaluation tool. So it was a learning experience. We did compile all of the results from each agency to do an overview document that was presented at the meeting, but definitely had feedback from the members. (DESCRIPTION) A rubric with blank spaces for Agency and Date is titled, SOCRC Self-Evaluation Tool for Agency Effectiveness. The rubric has columns titled, Fully Executed, Partially Executed, Some Progress, Not Started, and Not Applicable, with descriptions for each rubric score filled in for five different Part 1 SOCRC Bylaws. The bylaws sit in a lefthand column titled, Description of Metric. The first description of metric reads, Regular attendance and participation at monthly Consortium General Membership meetings. The full executed score for this metric reads, Representative or designee attended at least 80 percent of monthly general meetings of the SOCRC. (SPEECH) So what you're seeing is page 1. There are three parts to the self-assessment, again, what is listed in the bylaws. The bylaws is part 1. So it may be a little bit hard to see the description of the metric. I'll read the first one to you. It's regular attendance and participation at monthly consortium general membership meetings. So was this fully executed, partially executed? Have we had some progress or have we not started this? And while that might not specifically apply to attendance at a meeting, we did this as a rubrics, something that we're all used to using for assessment. And then each agency would rate themselves on how well they did on this particular metric. So again, this first section goes through the bylaws. So anything that was in the bylaws as a member effectiveness metric was used here and the agencies rate themselves on that. So we can go on to the next slide, Will. (DESCRIPTION) The next slide shows a section further down the rubric, with a comments or explanation section at the top. The Part 2, Annual Plan part of the rubric has descriptions of metrics that say, Recruit and Retain Students, and, Evaluate classrooms, facilities and equipment, with three rows for each one. The rubric is filled in with descriptions in each of the rows for the two metrics. Four rubric cells are highlighted, all in the Fully Executed column. For Recruit and Retain Students, the highlighted sections say, Administrators, instructors and staff members met at least 4 times during the year to discuss and evaluate instructional and support programs, and, Agency provided a full range of digital literacy courses for students and training for staff. For Evaluate classrooms, facilities and equipment, the highlighted sections say, Evaluation completed on all classrooms, Evaluation of facilities completed, and, Evaluation of technology and equipment completed. (SPEECH) Part 2 is the annual plan. So again, this information is pulled directly out of the annual plan. This is a document that's going to get updated every year because it does have to match the annual plan, and then the three-year plan metrics when they come out. We did just update bylaws and pass them in January. So there were some new member effectiveness criteria that were adopted with that particular set of bylaws. So we'll be updating the bylaws section as well. So this isn't we're going to do it once and then we never have to look at it again until we pull it out once a year to fill it out. This is a living, growing, changing document, specifically related to our three main member effectiveness areas. WILL NEDDERSEN: Nancy, may interrupt you before we move to the next slide? NANCY MILLER: Absolutely. WILL NEDDERSEN: Can I ask you why did we make this an important component as a self-evaluation tool and not somebody ranking that's not a part of that agency? NANCY MILLER: Number 1, when we were looking-- and again, we may hit on this a little bit earlier, but we didn't want this to have a negative connotation. Really, we wanted members to be involved in the process of evaluation. This is not a punitive document. This is to assess how we are doing as providers of adult education. This also gives members the opportunity, again, comments, explanation, provide feedback. So as we're doing the annual plan, you can see on the third line there was nothing highlighted. This is how the agency will select whether it was fully executed, partially executed. So they select or highlight their rating of themselves. The other thing that we were very specific on not wanting is we didn't want to attach a number to it. So you had a numerical grade at the end of this. Like we see here on the third one, if no one filled out that third one, is that a necessary part of our evaluation tool? Should we even have it in our annual plan if no one's doing this? So those are some of the things that we took into consideration as we were, excuse me, as we were developing and putting in the different levels of participation or completion of the various metrics in this. Again, for annual plan, as we're going through and writing it, we have a whole section on recruiting and retaining students. Here we are evaluating our classrooms, facilities, and equipment. That was written into our annual plan because we had two agencies that we knew either needed to upgrade facilities or get new facilities. And so this was an important part. You can't spend money on it if it isn't in your annual plan, in three-year plan. So as we were going through the development of the tools and as we were going through the launch of the tools for self-assessment, we went through this with our agency members. Not well enough, not well enough. WILL NEDDERSEN: Nancy, may I interrupt? I see in the chat the question is, did the members complete the assessment, the self-evaluation assessment during a consortium meeting or in preparation for the meeting? And Karima, thanks for responding, but want to make sure we respond that. This was done ahead of time, which allowed it to be turned into somebody could compile the rankings. We didn't show everybody's plan itself. We just took the highlight of they said, this member said they were fully executed or partially executed to be able to give that sense of general overview so we could be reflective. Especially if there's a conversation about a member, I'm going to speak as a chair now, that we are wondering, hey, how do you think you're performing? And if they think fully executed, it allows for conversation. Just like the fiscal tool, the quarterly fiscal tool that's reported, it starts allowing a conversation more at the quarterly instead of just saving it for the end. But we knew we needed that total summary, as Nancy shared. I'm sorry that I keep interrupting you, Nancy, but I want to make sure we address that question-- NANCY MILLER: Oh, that's OK. This is a teamwork thing. KARIMA FELDHUS: I'd like to interrupt, too. Thank you, thank you, Nancy and Will. Just to piggyback on what Tim in the chat asked, actually he made me think that we could consider doing a retreat where we work together on these tools instead of just submitting them. And our meetings are just an hour and a half once a month. That doesn't really allow for a lot of discussions. I just wanted to acknowledge that we would consider that. Thank you, Tim. OK. NANCY MILLER: And Tim, we actually gave each member two months to fill this out. There were questions once teams got together to fill it out, and it was interesting because I was the one that actually did the compilation of all the responses from the agencies. So you could tell when the agency had a team that was working on the responses or when they had a single person that was filling this out to turn it back in. And that's because of the responses that were said or the way it was filled out, any questions that came back into the co-chairs or myself regarding this. And definitely Will and Karima can speak more to this because we were back and face-to-face and didn't get to do a Zoom meeting. So I did not participate when these were presented to the general membership meeting. But we are planning to do two to three-hour professional development session regarding the member self-assessment to go through it to make sure everybody understands, to have a document that would be completed so they have a template similarly to the quarterly fiscal reporting. Definitely update information in this and have more things that would be filled out or an information sheet to give each member more information to be able to fully participate. One (DESCRIPTION) Another section of the rubric is titled Part 3, Three Year Plan. In this section, cells for the rubric scores are not filled in with text. The description of metrics column is divided into three parts with the headings Consortium Level Metrics, Agency Level Metrics, and another heading that reads Agency Level Metrics. These heading cells have cells that say, Target 2022/23 to the right of them, and cells that say, Actual, to the right of those. The metric, Overall Number of Students Served, is filled in with 9,105 under the Target cell. The metric, Overall students in ESL, is filled in with 5,000 under the Target cell. The rest of the cells on this section of the rubric are blank. A section titled, Comments or Explanation sits below. (SPEECH) of the areas that people did get held up is in the annual plan. We definitely have some consortium related information for marketing or for a consortium transition specialist type work. And agencies didn't necessarily understand how to fill out that information if it wasn't specific to their agency and it was generalized to the consortium. So those are some of the areas that we have in the evaluation tool. Again, you have this tool to look at in full. But those were just some of the things that when you're creating it, you think, oh, we're being really thorough and this will be great. And then it comes to putting pencil to paper or highlighter to computer screen and it doesn't really go the way you think it might have gone. So again, it's a labor of love and definitely something that will continue to be updated so that members can get the most use out of it. WILL NEDDERSEN: As being one of the members that was part of the subcommittee for this work, it was interesting because we thought we had hit all these areas and talked with people, but it was bringing the group together. So there is the pros and cons for both. Do we fill this out together collectively as a group in that professional development or retreat situation, or having the time to let teams process together and then bringing everybody together to then share it? It was rather interesting this third page that you're seeing looking at the numbers. We really wanted to be able to share numbers, but that even brought confusion within our agency. So really needing to articulate that together. But it shows for us where we need to process. How are we gaining that data of enrollment from systems that are TOPSpro and systems that are MIS, or MIS? Forgive me, I'm not sure how you describe it. I come from the K-12 side of the world. But really, how do we share that information so we can get that collective number? Waiting two years for LaunchBoard to be able to show and represent it together, that's too long, right? And not to look negative on that, it's great to be reflective with that. But moving us forward, how do we take those numbers and celebrate our successes as the annual plan and three-year plan are asking us to look at and do as well? So this third page we'll be honest is something that we're going to work on and reflect together on still, knowing that we'll bring some subcommittee work back together on it. And then we shared it before, but a reminder in our subcommittee-- and I believe Nancy said it too, I think. And I just want to make sure clarity. We would always bring subcommittee work back into the general session so that way it was heard what was being done. So if an agency couldn't send a member to represent, they still heard the information of the progress that was going forward. NANCY MILLER: And this last one, one of the things that we discussed after the presentation was these are the metrics that are out of the three-year plan. What we'll do differently this year is to actually fill these out for each agency so they don't have to look. The one difficulty is that if you aren't uploading your information into TOPSpro, it is harder to get it back out to fill out this information. So again, our quarterly fiscal reporting tool hopefully will help us to get that information quicker and have that for agencies during that fourth quarter submission, and then we can get this tool out and they can have two months to fill it out and work with their teams on this. Again, it's an activity that the agencies use to determine how well they are completing each of their metrics in adult education. OK, I think I'm handing it over to you, Will. WILL NEDDERSEN: Yes, I just realized I talk a lot. I don't want to talk over you, Nancy. And I do apologize for that. NANCY MILLER: Not at all. WILL NEDDERSEN: The interesting piece I'm seeing in the comments right now-- somebody is sharing. I think it's Molly saying, I'm thinking through what to adopt and personalize to help your consortium understand your work better together. That's our hope is the tools that we're sharing it was helping us understand each other and be able to move us forward, especially in the idea of being effective as a consortium for our students. And that's the hope in these tools that we're sharing here is that you can look at them and say, hey, I don't want to address it or I do want to look at it, but AB 1491 is really bringing that dialogue of member effectiveness forward. So we do hope that that's why we're sharing is that we hope that you make it your own. With that, we start getting into the conversation of bylaws now. (DESCRIPTION) Text, Member Effectiveness in Bylaws. Policies and Practices to encourage transparency and information sharing. Clear definition of member effectiveness. Clear definition of technical assistance. Policies to address funding and performance under ABI491. Text in a screenshot reads, ARTICLE V - MEMBER EFFECTIVENESS. Section 1: Consortium Effectiveness Criteria. AB 1491 amends Sections 84901 and 84914 of the California Education Code AB104, Article 9, relating to Adult Education. It outlines consortium effectiveness criteria including that a consortium must not have a collective carryover balance that exceeds twenty (20) percent annually. a. Consortium carryover reflects the carryover of all funded members of the SOCRC. b. Any carryover funding that exceeds the twenty percent threshold in one or more prior fiscal years shall be determined as excessive carryover. c. A consortium that has been deemed by the Chancellor and Superintendent to have excessive carryover is required to submit a written expenditure plan to the State. d. A consortium with excessive carryover shall be subject to Technical Assistance from the State as outlined in Section 3 Technical Assistance. Section 2: Member Effectiveness Criteria Member effectiveness shall be defined as: All members: 1. Regular attendance and participation at monthly Consortium General Membership. SOCRC Bylaws 2024. (SPEECH) So if you just give a tool without a connection to why we need it, sometimes we don't do it. I don't know about you as administrators. We get a lot of different things. But here we want to talk about the tools supporting member effectiveness bylaws. And the bylaws really are drivers of the work that we do. And with that, we had to look at, how do we insert what we expect of our policies and practice to be transparent and make sure that we're collectively sharing information for the betterment of our consortium and our students? And so we wanted that dialogue to be transparent that way and really bring it out in by laws to give us guidance. And to that, looking at adding these tools that were created into the bylaws so that way it kept a constant dialogue forward that we didn't have to feel like we were in our own silos, but collectively working together. It also helps us in defining certain things as well. I go back to why we shared the 2019 bylaws. There was punitive, as Karima shared, and a general vagueness to it. And when we're vague, we're unclear where we want to be able to celebrate and then say, how do we support if support is needed as well? And then that's why that clear definition of technical assistance, really reading AB 1491 and understanding technical assistance, I had always understood it that's CAEP TAP will help support that. But AB 1491 really brought it back to the consortium and saying, how are you going to provide technical assistance to your members if they're not meeting those expectations? And then really trying to address if our funding, again, if we have that excessive carryover, what are we going to do about that to keep us as an effective group working together? So that's where we go to start looking at creating a complete article. And I know Nancy will go more into this, but really looking at member effectiveness. So Nancy, I believe I'm handing this over to you. (DESCRIPTION) Text, A table on the slide compares Approved 2019 and Approved 2024. 2019 reads, Member Effectiveness is mentioned in Article IV Executive Committee and relates to ability for members to retail their vote. 2024 reads, Member Effectiveness became its own Article (V). 2019 reads, Four measures for member effectiveness. No distinction between measures for funded and non-funded members. 2024 reads, Defined consortium and member effectiveness. Defined measures for funded and non-funded members. 2019 reads, Indicated that a process to determine member effectiveness and interventions would be developed as needed. 2024 reads, Clearly defined process for member technical assistance and member non-performance. Three photos on the slide show students posing in graduation caps and gowns. (SPEECH) Nancy, I apologize. I think you were kind to me to be on mute, but I'm not hearing you now. NANCY MILLER: I was. And I was trying to be really kind to you being on mute because my dog has a different idea about inside voice. For member effectiveness, last week if you were with us, we are comparing our approved 2019 pre-AB 1491 or the consideration of AB 1491 and approved 2024, which happened in January. So the little glimpse you got earlier it was-- member effectiveness was two paragraphs or two paragraphs and the second one was really a two-sentence paragraph about member effectiveness. And now member effectiveness is its own article. I think it's two or three pages now. There were four measures of member effectiveness. There was no distinction between measures for funded members and non-funded. Now, there's a definition of consortium effectiveness that comes directly from the state, but we thought it was important to define the difference between consortium and member. And then we defined measures for funded members and non-funded members. The reason we did that is because non-funded members did not think that they had a role in member effectiveness. They didn't think that part of the bylaws applied to them. So we were saying, oh, no, we want you at the table, we want you to be there. There is definitely a piece that you have as a member of this consortium. And so effectiveness applies in that instance. And prior in the 2019, we indicated that there was a process to determine member effectiveness and interventions would be developed if they were needed. And now there's a clearly defined process. It's not ambiguous. There's member technical assistance and we'll go into later member non-performance. The other thing about this was prior to the 2024 member effectiveness was really related to whether or not a member could vote at the meetings, and you had to be a member in good standing. But we didn't really give a tool to say, how do we know you're in good standing? And now member effectiveness relates specifically to AB 1491 and we do write that language into the bylaws. And it also still links back to the retention of votes since that is a measurement. OK, I think we're on to Karima. (DESCRIPTION) Text, Section 1: Consortium Effectiveness Criteria. AB 1491 amends Sections 84901 and 84914 of the California Education Code AB104, Article 9, relating to Adult Education. It outlines consortium effectiveness criteria including that a consortium must not have a collective carryover balance that exceeds twenty (20) percent annually. a. Consortium carryover reflects the carryover of all funded members of the SOCRC. b. Any carryover funding that exceeds the twenty percent threshold in one or more prior fiscal years shall be determined as excessive carryover. c. A consortium that has been deemed by the Chancellor and Superintendent to have excessive carryover is required to submit a written expenditure plan to the State. d. A consortium with excessive carryover shall be subject to Technical Assistance from the State as outlined in Section 3 Technical Assistance. (SPEECH) WILL NEDDERSEN: Karima, my kind, fellow colleague, you are awesome. You still are. KARIMA FELDHUS: I am so sorry, so sorry. We have office noise in the background. So thank you. I didn't think this would happen to me after so many years. Anyway, so this takes us to the current bylaws consortium effectiveness criteria in Article 5. So we have Section 1, and you see a screenshot now. And that deals with Consortium Effective Criteria, what makes the consortium effective. Of course, we-- and again, I'm not going to read everything, we took the language from the code and AB 1491. So consortium carryover reflects the carryover of all funded members of the consortium. And any carryover funding that exceeds the 20% threshold in one or more prior fiscal years shall be determined as excessive carryover. Consortium that has been deemed by the chancellor or superintendent to have excessive carryover is required to submit a written plan to the state. And then consortium with excessive carryover shall be subject to technical assistance from the state as outlined in Section 3 Technical Assistance. My colleague, Will, be covering the details of Technical Assistance in Section 3, and then Nancy will follow-- or rather Consortium Section 3 and Member Technical Assistance is Section 4. Next slide, please. (DESCRIPTION) Text, Section 2: Member Effectiveness Criteria Member effectiveness shall be defined as: All members: 1. Regular attendance and participation at monthly Consortium General Membership meetings. 2. Participation in and contribution to the development of the Annual and Three-Year plans. 3. Timely submission and approval of all CAP and SOCRC deliverables per CAP published calendar and SOCRC approved meetings and timelines document. Funded members: 4. Meeting the student success goals as defined by the consortium in the Annual and Three-Year plans. 5. Financial expenditures aligned to Annual and Three-Year plan goals and CAP guidelines. 6. Funds expended in a timely manner in accordance with AB 1491. 7. Member carryover shall not exceed 19.9%. Any portion of the member's carryover that exceeds 19.9% shall be deemed excessive carryover. (SPEECH) Section 2 deals with Member Effectiveness Criteria. And of course, we have criteria applying to all members and criteria applying to funded members. So regular attendance for all members and participation at monthly consortium general membership is the first criteria. We have to be honest with you. We haven't started applying this yet. However, we're in the process of checking the last 12 months attendance. If we have members who are voting members but who missed many voting meetings, which we do have currently, then we'll start the conversation, right? And number 2 here, participation in and contribution to the development of the annual and three-year plan. So we also require them to be active or actively participate. As we heard before, we do have committees. So like all agencies, we have 10 voting agencies to participate. Timely, of course, submission of approval and approval of all CAEP and consortium deliverables in NOVA in particular. I have to admit, we've had agencies who did not meet the deadlines. So now we have some teeth. It's in the bylaws, right? If the first of the month is the deadline, we have a deadline tomorrow Friday coming up, we sent a reminder. And tomorrow we'll go in NOVA prior to 5 o'clock and send more nudges to those agencies. So the message really here is by having this in the bylaws, it is not an option to be late submitting your reports, it is not an option to not participate, it is not an option to attend meetings, especially when we need quorums or we hold votes. Funded members, we have additional criteria. Of course, it has to do with student success goals as defined in our annual and three-year plans, financial expenditures aligned with the annual three-year plan. And you saw the tool earlier. That's why we have part 2 with staffing and part 3 in that fiscal quarterly report dealing with classes and enrollments. So everything lines up. And here number 6-- funds expended in a timely manner in accordance 1491. So now we look at that. And last but not least, we defined as a consortium that member carryover is going not to exceed 19.9. We didn't keep it at 20. As you know, we were asked to define that percentage excessive carryover and our consortium decided that 19.9% per member is our carryover excessive. Anything beyond 19.9% will be considered excessive carryover. Now, next slide, please. So what happens if an agency is not being effective? So we're not even saying ineffective. So it's more positive. What if they're not being effective? So you don't have in front of you. You just have summary bullets. But I'm going to go ahead and read you the paragraph from the bylaws. And you have that in your folder. I believe it's number 6 bylaws 2024. So if members are unable to meet at least one of the criteria of member effectiveness, they may need the support and mentoring of the executive committee and/or other members of the consortium to come back into and remain in compliance with consortium and state requirements for California Adult Education Program funding. Members who are out of compliance with the member effectiveness standards may be subject to technical assistance through SOCRC and placed into monitoring for a period of at least 12 months or until areas of deficiency have been remediated. And my colleagues, as I said a moment ago, will be covering the technical assistance part, both for consortium and members. Next slide, please. I think we're covered. Let's see. WILL NEDDERSEN: I think that goes to me, Karima, if that's OK. [INTERPOSING VOICES] KARIMA FELDHUS: Yes, we're covered (DESCRIPTION) Text, Section 3: Technical Assistance. a. Consortium Technical Assistance If a consortium has excessive carryover for at least one prior fiscal year, they shall be assigned to technical assistance by the California Community Colleges Chancellor and the Superintendent of California Public Schools. The technical assistance provider is selected by the State, and their purpose is to ensure the effective use of funds as specified by California Education Code and to ensure that the consortium can provide adequate adult education services while not exceeding the twenty percent carryover threshold. (SPEECH) WILL NEDDERSEN: So when we start talking about technical assistance-- and you heard me say, oh, I thought technical assistance was just at the state level. We need to clearly identify as AB 1491 has. There's a consortium technical assistance and there's a member technical assistance. And we felt that it was important in the bylaws to include the definition of the consortium technical assistance so that way all agencies understood this is at the consortium level, this will take place. And then as Nancy will go into, there's that member technical assistance that's also needing to be embedded in what we do. With that, I want to bring it up, as a new member to the adult Ed world at one point, the bylaws were a little vague for me. So reading these bylaws, we just thought if there is an admin change at any of our agencies, the bylaws might give clarity as well. That's why we were focused on adding definitions where definitions needed to be added or to give the assembly bill or the law to give guidance as well. So I don't want to read this here, but if a consortium has excessive carryover for at least one prior fiscal year, they shall be assigned to technical assistance by the chancellor or the superintendent of California Public Schools. We took that directly from AB 1491. I will be honest I'm not going to read the rest of it, but it gives us clarity on that. And Nancy, I know member effectiveness is where we had to work and move forward to. So I'm going to advance it on so you can start talking about member effectiveness. (DESCRIPTION) Text, b. Member Technical Assistance. Members may be subject to technical assistance through SOCRC and placed into a monitoring period for a minimum of twelve months if they have not met one or more of the member effectiveness criteria outlined above. Technical assistance may include but is not limited to any of the following steps and supports: 1. Notice sent to agency representative and Superintendent outlining area(s) of noncompliance. 2. Measures necessary to regain compliance with SOCRC and CAP regulations. 3. Regular monitoring of member effectiveness areas that are out of compliance and were included in the notice to the Superintendent. 4. Develop and implement an Excessive Carryover Reduction Plan to meet member and consortium excessive carryover limits. If a member who is in technical assistance is unable to meet the obligations of SOCRC and CAP funding requirements after a twelve-month monitoring period, additional actions may be taken including the following: 1. Extension of technical assistance monitoring period for an additional twelve months or until agency remediates areas of deficiency. 2. Reduction in funding not to exceed the amount of excessive carryover from the prior fiscal year. 3. Removal from being a service provider. (SPEECH) NANCY MILLER: OK, for member technical assistance, we really needed to lay out a process for members to go through. Again, we wanted this to not be a punitive process. We value all of our members and we want our members to be successful. So this is the process that we felt would best address how members could get technical assistance. And one of the really nice things is that this technical assistance process came out of the fact that we had members who weren't performing and these are some of the things that the co-chairs did to work with members to help them get into-- I wouldn't say compliance. Effectiveness wasn't an out of compliance. It was more non-transparency issue. And so in having conversations regarding operations or sharing of information, this is part of the process. So the co-chairs if after the 12 months technical assistance is needed by a member, then the co-chairs would send a notice to the agency and their superintendent that specifically outlines the areas of non-compliance. So those are going to be identified in the bylaws. Agencies already have a copy of the bylaws. So we would just note the section or the area that they're out of compliance. What are the measures that are necessary to regain compliance if it's an SOCRC rule or if it's a CAEP regulation? What do they have to do to come back into compliance? Regular monitoring of member effectiveness of the areas that are out of compliance. I'll give you an example of why or how this was developed. If a member is consistently going over in their carryover, then we would want to have them report back to the co-chairs on a monthly basis with their financials so that we can make sure they're keeping on track. Maybe every three months is too long to go before having a discussion about fiscal responsibility or how an agency is spending down their funds. It also helps the consortium better say, hey, is this something that you can share this year? And then next year, if you need money, we'll hopefully have money to share out next year. To develop and implement an excessive carryover reduction plan so that member may be required to do that, where if it's a consortium, the consortium would decide together what that excessive carryover reduction plan would be. So if a member who is in technical assistance, and technical assistance is going to go at least 12 months, if they haven't remediated their areas that are out of compliance, then they may have an extension of technical assistance monitoring for an additional 12 months. Or again, if it only takes three more months to get back into compliance, then they're only going to go three more months. A reduction in funding, again, this is right out of AB 1491, not to exceed the amount of excessive carryover from the prior fiscal year. So if carryover is your issue, then we may have to reduce so that you aren't in that excessive carryover whirlwind that you can't get out of. And then the final one-- again, this is language that is supported by education code, it would be removal of being a service provider. Because now you're two years into this, if you are really unsuccessful in meeting CAEP and consortium regulations, specifically around being a service provider, if you can't spend the money, if you aren't spending it in accordance with plans, then we're going to have a discussion about potentially being a service provider isn't the best thing for your agency. So it's kind of like, when you get a job evaluation, it always says that clause "up to and including termination." Well, this is you're up to and including termination clause in member technical assistance. So Will, next slide. (DESCRIPTION) Text, Section 4: Member Non-Performance. The Executive Committee may recommend that a member be removed as a service provider for nonperformance of duties after the member has gone through at least eighteen (18) mönths of technical assistance without remediation in the deficient areas. A decision to remove a member for nonperformance requires a two-thirds vote (7 out of the 10 members). The Executive Committee will send notification of potential action to the agency representative and governing board/Superintendent outlining the actions taken by SOCRC technical assistance, areas of non-compliance, and the date of the Consortium General Membership meeting at which the removal of the member from service provision will be addressed. (SPEECH) So again, this is member non-performance. The agency is gone through at least 18 months. Most likely, they've gone through 24 months of technical assistance without remediation. The decision to remove a member for non-performance, again, number 3 in Section 3 above, its 2/3 vote. Normally, we have a majority vote rule. This is one of the areas where it's a 2/3 vote. The executive committee is going to send-- executive committee being the co-chairs, so Will and Karima would send a notification of potential action to the agency representative their governing board or the superintendent, if it's the county, outlining actions taken by SOCRC technical assistance for non-compliance areas and the date of the consortium general membership meeting at which the member being removed from service provision will be addressed. So again, when we were going through this and had a retreat with the member agencies, we had a long discussion about this. And once we really looked at this, by the time it would come to a member non-performance action, we're about four years into the state technical assistance, the consortium technical assistance, additional remediation and follow-up with the agencies. And 36 to 48 months later, it's a non-performance issue. So we did write this in case it ever came to that. We don't think that we'll need it, but it's there just in case. OK. WILL NEDDERSEN: And Nancy, I think if we make that link you just shared when we get into technical assistance. But if you think about the other tools in their design that we shared with you today, they're there to bring that dialogue, that transparency and understanding of each agency together annually or quarterly so we have that process of understanding. And just like, if you go back to the fiscal tool, you'll see just like it says in NOVA as well. What's your plan if you have a carryover and really making sure that we dialogue about that together? I know for myself in Tustin, I've had to share. We didn't meet this mark, but I see spending going this way as I'm adding two more sections here of an advanced ESL class or those elements. So that was the other in the subcommittee work as well as with the general session. Like you said, Nancy, understanding it's three to four years out. But even before that, that's why all the other tools make an impact into the work that we do to be effective together as a consortium. NANCY MILLER: Yes. And again, your technical assistance is provided just with the voting member and whoever from their team they want to bring and the co-chairs. It's done at an executive committee. It is not done in front of everybody at a general session meeting. We really want this to be personalized approach to ensuring that the agencies are doing what they're supposed to do according to education code and SOCRC bylaws. We tried to make it a constructive process and not a punitive process. OK, Karima. (DESCRIPTION) A questions slide appears. (SPEECH) KARIMA FELDHUS: All right, thank you. We thought we'll take a brief moment to ask you to share your practices or about your consortium or consortia. So we'd like to know, who has established member effectiveness criteria already? Who has established a member technical assistance plan? Would like to also learn from you as well. As you see, this is in progress for us. And prior to AB 1491 guidelines, how did you address non-performing members? If you would please share with us in the chat would appreciate it. (DESCRIPTION) Karima's image takes up the body of the slide. (SPEECH) And at the same time, we'll ask you, do you have any questions? We do have a few minutes. We're at 1:54. CAEP TAP will need a couple of minutes to-- NANCY MILLER: Karima, we do have a question and it is, will you be posting these slides on the CAEP website? I'm going to send that one over to Mandilee or Holly. But in the meantime, you do have access to this presentation. It's number 7 in your folder. So if you clicked on the link to the folder, you do have the presentation. You will have all of the tools. You have a copy of the quarterly fiscal report, you have a copy of the self-assessment, you have a copy of the directions to fill out the quarterly report, the facilities, the metrics that we use to develop all of these tools. Thank you, Holly. Holly just put the resource link back in there today. So in the meantime, until that gets back up, Holly or Mandilee can answer. You will have access. (DESCRIPTION) A woman in headphones and a white sweatshirt appears alongside the text, CALIFORNIA ADULT EDUCATION PROGRAM Technical Assistance Project. Mandilee Gonzales; CAEP TAP. (SPEECH) MANDILEE GONZALES: Yes, so we are going to be posting and sharing all-- well, both recording. So this is part 2. We'll share part one as soon as it's completed the remediation process along with this video and all 508 compliant resources. I would highly encourage if you are in this setting today absolutely grab that link. It is a Google folder and it does speak to all of the tools and resources outlined today. NANCY MILLER: We have kind of a compound question here from Tim. And Tim's asking-- oops, I have to slide back up, criteria outlined on the self-assessment slides 15 and 16 were developed by the subcommittee. Did the general session members vote to approve this criteria for the assessment, and did they have input once the assessment tool was developed? So the answer to both of those questions, Tim, is yes. Will and Karima, if you want to jump in on this, again, please feel free. Each of those criteria-- we have the bylaws, we have the annual plan, and we have the three-year plan. And each one of those, to even develop the metrics that we were using, all of our members had input in all of those and voted to approve all of those because we vote to approve our annual plan or three-year plan before it's submitted to the state. So they all had input opportunities and voting opportunities on all of those. And like we said before, any time we use these tools, we get back together. The whole group has input on, hey, did that work? Did it not? What would you do differently? Should we have a subcommittee meeting so we don't have to take up the general session meeting? Anyone want to join us? And we set a date for those so that we can work-- anything we do in subcommittee is always reported back the next general session meeting after the subcommittee meetings. WILL NEDDERSEN: So Nancy, I just want to give clarity on that, even Tim. We were transparent with the name of the subcommittee and the purpose of the subcommittee. There one that was specifically for a member effectiveness tool development so that way everybody knew the dialogue that would go in that subcommittee. And then be brought back to the general session as well when we looked at those. So that was part of the subcommittee's work in determining how do we look at that self-evaluation to say that, are we being effective for the work of the consortium? So that was an important piece for us in that dialogue. And I'm sorry. I'm seeing Rebecca ask a question about slides. Our slides, that's what we presented today is exactly what that link is there. So I believe it is there. I know that we are hitting the 1:58 mark. Is there any other question that we can answer? Or I am appreciative to a response in here talking about a consortium who developed a member report card to determine member effectiveness. I'd love to hear that name or process. That's interesting to me. But saying that maybe it focused a little bit on the punitive rather than help and maybe that readjustment. But I even like that for me being reflective of that member report card, I'd be curious to reach out to you and hear and learn more about that as well. KARIMA FELDHUS: Thank you, Will. And given that it's 1:59, do you mind sharing your screen again please? Thank you. (DESCRIPTION) The beginning slide with the speakers and their titles appears. The QR code sits in the left side. The speakers' email addresses are, k feldhus at saddleback college dot edu, nancy embank miller at comcast dot net, and w neddersen at Tustin dot k12 dot C A dot US. (SPEECH) So here is the QR code again to the folder, and I believe CAEP TAP has a slide after that. Thank you, everyone. And please do feel free to reach out to any of us three. You have the email addresses here. And we love sharing what we do. This is work in progress, as we mentioned. So we're progressing, progressing in the right path or positively. However, we're not 100% there yet and we love to learn from you as well. So we thank you. (DESCRIPTION) The next slide shows a photo of an ad for an Adult Education Pathways Fair above a scoreboard. The date of the fair is listed as October 15th. A QR code reads, SCAN ME, in the right side of the slide. A link below the QR code reads, https colon backslash backslash register dot cal adult ed dot org slash Eval slash Home slash Evaluation slash 664. Text, Thank you for joining us. Please use the QR code for evaluation. (SPEECH) MANDILEE GONZALES: All right, well, thank you so much, Nancy and Karima and Will. I know that all of these tools I'm sure are going to be-- people are going to grab them and use them. We appreciate you sharing all of the hard work that you've done. And I know it's taken a long time to get here. So thanks for propping up everyone else and helping everyone learn and grow together. Holly has popped into the chat a couple of things for you all. So not only do you have access to the Google folder that is shared out with all of the resources. We also have an evaluation link. If you could take a few moments to fill out that evaluation link, it helps inform us how we develop our professional development calendar as well as our facilitators. We do have some PDs coming up right around the corner. We have one on Monday-- update adult education as a gateway to college, which West Ed will be facilitating. We also have one coming up on the 12th. It's the CAEP allocation fiscal year '24/25 having a discussion and overview of some essential budget implications. So please take time to make sure that you do a few things and register for some upcoming events. Give us that feedback. And with that, I just want to thank you all for spending your afternoon with us. We greatly appreciate your time. And again, Will, Karima, and Nancy, thank you, again. Everyone, have a great afternoon. WILL NEDDERSEN: Thank you, everyone. MANDILEE GONZALES: Bye.