(SPEECH) DEBI PEZZUTO: Thank you. There we go. Thank you, Holly, and thank you, SCOE CAEP TAP and the Chancellor's Office. My name is Debi Pezzuto, I'm the Senior Program Associate for Adult Education, Workforce Development, and Post-secondary Education at WestEd, and I'm joined by John Brauer and Meghan McBride. We'll do intros in just a second. Thank you for joining us for the Employer-Led Industry Engagement webinar, Best Practices and Steps Model, A Framework for Community College, Adult Education, and Workforce Development Partnerships with Industry. In this webinar-- This is a part of the CAEP professional development project, and this new contract for this New Year is focused on the increasing pressure to align Adult Education programs with the labor market and policies, and what we want to see, and how we want to see better outcomes for participants in the form of labor market participation and earnings. Therefore, a central theme in this new contract and the TA contract is alignment with the labor market and measuring effectiveness. One of the activities for this project involves creating a dashboard that compares the targets that consortia and individual members set in their three-year and annual plans with outcomes data, and another activity is conducting a non-credit inventory of all non-credit CTE programs across the state of California to help us do a gap analysis of your regional labor markets, ensuring that your non-credit programs are aligned with employer demand. And finally, these webinars are designed to help you improve your employer engagement and determine if you have quality employer engagement, where to get started, and then the next steps to take it. We are going to be looking at aligning our strategies with our curriculum, making sure that we're really incorporating employers into the programs that we're designing and the curriculum that we're building for our students so that we can have better outcomes for all. I'm going to go ahead and move this slide over to the next one here. As I said, I'm Debi Pezzuto, I'm the Senior Program Associate. I'm going to pass it over to the ever-lovely Meghan McBride to introduce herself. (DESCRIPTION) Debi looks to her left, but the screen does not change. (SPEECH) If she's not frozen, which, of course, she immediately is frozen. Meghan McBride is the director of Adult Education Projects. She and I work very closely together and we are the PIs for the CAEP contract. So you'll be seeing a lot of us in the next several months, and specifically in the next two webinars that'll be happening in May. We'll have more information at the end of the webinar. So now I'll pass it over to the always-lovely John Brauer. JOHN BRAUER: Afternoon, everybody. I'm John Brauer. I've been at WestEd for a couple of years, but I have been largely a practitioner in relationship to industry and education partnerships up and down the state. For 11 years, I was the Workforce and Economic Development director at the California Labor Federation. So I worked on partnerships up and down the state of California in just about every sector, creating labor management, training partnerships, apprenticeships and other High Road partnerships. And before that, I actually for 11 years ran a nonprofit in the city of Oakland that did sector work with Oakland Adult Ed and the Peralta Community College Districts around construction trades and trade and logistics. So I come more from a practitioner than a education background in relationship to this stuff. So go ahead, Deb. Next slide, please. DEBI PEZZUTO: I also wanted to mention, if you have questions and you want to interrupt, you can put it in the chat or you can raise your hand. Meghan and I will be monitoring the chat throughout the presentation. So you can interrupt if you'd like. We got a pretty small group. Or you can just raise your hand or put it in the chat and I'll call you out. JOHN BRAUER: Yeah. So I've put a lot into these slides, and my hope is you probably aren't going to get a lot of this as you intake it over this hour, but there's a lot to take away and certainly at the end, feel free to reach out through Debi and us to ask for more questions or assistance in addition to during the presentation today. One of the big things that I want to emphasize in this and my orientation to this work, and hope that you think about this, is I always start with the mantra of starting with the jobs. And that engaging industry, and by industry, I mean employers and labor, that the goal of industry-led partnerships is to have them viewing industry as a partner, not just as a customer from your educational institution. So the goal is to put industry in a co-leadership role with you while still preserving your academic authority, your instructional equity expertise, and equity commitments that you as educational institutions bring. The goal of this is you're going to be-- in addition, is that the partnership is going to become a problem-solving, one for industry. That they view you as somebody who becomes to understand them, or that there's a return on investment for them in terms of their time and resources, and for their workers in that for them, and the work leads to having a talent pipeline and a talent pool, if you will. Next slide. Debi. I just want to state that there's a lot of research out there that suggest great success or big success when an industry is a true partner with you in these kinds of endeavors. The US Department of Labor has found that programs with high levels of industry involvement and design and delivery produce measurably better employment and learning outcomes for participants. There is a really good model of business and industry leadership team that's been developed by the National Science Foundation and Collin College that has demonstrated in over 100 colleges around the country that employer co-leadership and curriculum development directly increases graduate employability and employer hiring commitment. A lot of that work for them is in the area of stem, and particularly in manufacturing, and areas like that. And then the third is an area that I was involved in around California's High Road training Partnerships, the California Workforce Development Board that involve both management and labor, and led to real problem-solving partnerships, achieving both scale and systematic input. I was part of an entity of transit agencies and transit unions and a number of different community colleges and some Adult Ed programs that have started with colleges down in Santa Clara. And right now we're at about, I believe, 12 different partnerships around the state. So there is a lot you can take to scale and be effective on in these kind of partnerships. Next slide. Debi. (DESCRIPTION) Text: What the Evidence Tells Us. (SPEECH) So some of the things that different research entities or different evaluations have pointed out, that supporting that industry-led model, if you will, and there are different variations on it, you end up-- generally, some of the tact evaluation has shown that there are higher average wages, that the employment rate gain has been higher, higher average annual earnings. That project quest is particularly in relationship to health care occupations in the Southwest, particularly in Texas and in Colorado and elsewhere. Greater retention rates for learners and workers who come through them. There's a lower cost per hire, meaning that there's less turnover for the employer, so they see a real source of talent as part of that. And again, some scale, as I said earlier, with BILT, where you've got Department of Labor and Department of Ed recognizing industry-led as a really successful model moving forward. Next one. Debi. (DESCRIPTION) Text: Four Pillars of This Model. (SPEECH) I start with-- In the work that I've been doing probably for the last 20 years, and some things for you to think about, some of which I've stated earlier, but that industry leads the problem solving. So you're thinking about this not just in terms of your students and your workers and learners, but from industry standpoint, you're in partnership with them to help to define the demand, identify gaps in curriculum, skills gaps, work processes, or helping them meet needs, or validating those competencies and setting standards so that they become co-architects with you. Not just something that you present to them. And I think that that's really important. It's sort of like, if they're helping to make the meal, they're more likely to buy into what's getting baked, if you will. The partnership is a sustained priority. So even whether you're starting with a pilot or something else, you're thinking about this long-term. The engagement isn't just transactional or evidence-based. It's ongoing. You're thinking about the structure and the two-way relationship that you define with the governance. A third one is, I think, another area that folks often don't think about, which is that learners and workers also have really valuable input on the job quality, barriers, and pathways, and getting their voice and having their input can design you better programs through the partnership, and that the training solutions are, again, industry-driven. You all are obviously doing the curriculum approval and that kind of thing, but industry can help you on that design, the credential alignment, helping create work-based learning opportunities, as well as being part of the quality improvement, are all grounded in the process and what industry partners determine that they need. Next slide. Debi, are there any questions at this point? I know we're kind of rolling, rocking and rolling. DEBI PEZZUTO: No questions in the chat yet. But does anyone want to shout anything out? (DESCRIPTION) Text: The Six-Phase Model at a Glance. (SPEECH) I think we're good, John I'll keep watching, though. JOHN BRAUER: OK. Thank you. So I have thought about this work and some of the research of some of the entities that I've cited already as kind of a six-phase model, but it's not linear. A lot of this is really iterative and ongoing. So some of these phases will repeat annually, others are visited quarterly or regularly around it. But the first is just building that foundation and partnership. And you can do some of these concurrently, obviously. They're not steps, if you will. The second is industry-led labor market analysis and moving beyond just basic LMI research. Working with industry around curriculum design and credentials, work-based learning infrastructure, and establishing that to meet the needs of your workers and learners in that equation. Putting some thought into faculty and instructor industry currency to give them an idea of what's actually going on in industry that you're involved in. And then thinking about the data improvements as well as another aspect of even from the start thinking about taking what you're doing to scale and moving from there. Next slide. Deb. I just wanted to start with this before I jump into each of the individual phases that I'm going to talk about, which is that I think the single most important structural investment that you can make in your CAEP work and in your partnerships is having a dedicated human whose job is to make the employer participation easier. And I don't know what that looks like for most of you, but somebody who plays that Partnership Navigator/Liaison operates as the primary friction-reduction mechanism in the partnership, particularly in relationship to industry, that there's somebody who can answer questions, that they know that there's a single point of contact, that they've got the bigger picture in mind in terms of the MOUs that you may create and execute, meeting logistics, matching, work-based learning, coordination, data collection. They're really the relationship manager in some of the models like BILT that have been out there. And a lot of these are frankly models that, really, you should think about from a high touch in terms of the outreach and responsiveness that you have. And finally, I don't know that you can teach this, but it's somebody who understands, particularly for employer partners, the what's in it for me aspect of it for partners and working with them to tailor recruitment and retention and messaging to employers in particular over time. And that's really important. So somebody-- I don't know if I would call it customer service, but at least relational skills is the way I really think about this, and being responsive to industry. That's a priority for them. In addition to learners and workers. Next slide. Debi. DEBI PEZZUTO: Before we move on, John, I'd love to jump in with just an example from my experience working in Adult Education. So before I worked at WestEd, I was the Student Services Coordinator for an Adult Ed program at a community college out here in Reno, Nevada, and in my role, I was the bookends part of the program, so I did intake recruitment outreach. And then I also was building the transition programs, ITs, getting students into college, and working with employers to build Workplace ESL classes, or build the pre-apprenticeships or apprenticeships. So it was one of those does it all gals. I didn't know how to do any of this. I was not trained to do any of this. I got a degree in English, but my role kind of melted into this over time by talking to employers and going to the workforce boards and hearing what they had to say. And one of the best experiences was doing tours of facilities or workplaces so that I could see what they were doing in the actual site, and then sat down with them to say, where are your weak spots? Where do you need help with? What's your bottom line? And I tried to always say-- find out what their needs were first before I talked about mine, mostly because I had no idea what I was doing. But it taught me a lot over time, and it opened up the relationship so that we could design the programs around what they really needed while still meeting the learners' needs, because I had experience teaching and tutoring and counseling, so I knew what the students' needs were. JOHN BRAUER: Next slide, Debi. Thanks. (DESCRIPTION) Text: Foundation and Partnership Infrastructure. (SPEECH) Here we go. So in terms of the first phase and Foundation and Partnership Infrastructure, and a lot of this is food for thought, and am happy to reach back out if folks have questions or comments or needs after this to get back to you and talk about some of this stuff more. But some things to think about are defining the industry co-leadership model that you want to have, what is their role between being just advisory participation in a committee that meets once a year or a little bit more often versus being a co-leader, maybe they can co-chair. What it means to the industry that exists, and also understanding how success is defined for the partnership itself, that that's a really valuable role for the co-leadership model. I think industry will also help you figure out what's the right model for the context for you regionally, for the sector or sectors that you're involved in, or the number of colleges that are involved or adult schools in the equation. And so working with them to figure out what that configuration looked like. Also important to think about the industry's role within your partnership. So I have to be honest, HR professionals alone are kind of insufficient. Their job is to fill a hole. And sometimes it's not necessarily to fill a hole for somebody who's going to stay, it's to fill a hole. And they also don't always bring what I think are that industry knowledge that is going to help you in your curriculum design or adaptation or changes over time, if you will. And so there's a role for them. There may be a role relative to high-level executives or strategists. So furure-oriented folks from the sector, there may be folks who, for instance, are bringing ideas about what's going to happen with AI in their particular sector, or as folks move to greenhouse gas reduction and green aspects of the work, those kinds of things. A lot of times I'm a big fan of frontline hiring managers or floor managers because they really understand the knowledge, skills, and abilities, the KSAa on the floor, what's there, what's missing, what's going to change, those kinds of things. So both distinguishing and maybe having a variety of folks in there. Again, if there's a place to have learner and worker voice, they can also give you feedback in the partnership. What's working both for pathways and resources, how the learning is delivered and how it's received, those kinds of things as well. I also want to take a moment as somebody who came out of the labor movement for 11 years is to also have you think about labor as part of the demand side, not just part of the supply side. They can also give you real industry knowledge as to what's going on and what's the future of an industry and sector. And there's something there for you from that side as well. They can also be advocates for you too. And then the other thing that I would suggest is creating a formalized governance structure. Putting yourself in a place where you have an MOU or some other document that lays this out, that is not just based on individual relationships, and that after folks are gone and/or after a particular funding piece is gone, that can still continue in some fashion, if you will. Or as challenges arise, you know how to deal with them. Next slide, Debi. (DESCRIPTION) Text: Formalize the Partnership Governance Structure. (SPEECH) So in terms of that formalizing the partnership governance structures, things to think about in that is executing a partnership charter and MOU. Some of those things would be shared goals and vision, defined roles, and decision-making authority. What your data sharing protocols may be or need to be. I'm involved in a project with about 20 community colleges and Home Care Worker Union in San Diego, and we've had to do data sharing agreements between all 20 and the Chancellor's Office and the Union as well. So that has taken some time to work out, as well as the reporting needs and requirements that everybody has and meeting around it. What the role of your industry advisory board looks like, and what's the meeting cadence, and how those structures, the agendas get created, and who chairs those, and who participates. And then from that participant governance structure as well is figuring out the engagement levels of participation of industry, if you will, so that you've got clear expectations, they've got clear expectations around time and benefits, each of those. And then the other really big thing I would put in here, besides having a point-person on the employer side, on the coordinating your partnership side, is you have workforce boards, chambers, economic development agencies, industry associations, others who can help hold and move and bring resources to the partnership. And I would really encourage you, if you don't already, to really move towards having a partnership, particularly with your local workforce boards and your AJCCs and your chambers and industry associations. They have some of this as their traditional role and they really can, in terms of sustainability and responsiveness, really bring a lot to the table. Go ahead, Debi. (DESCRIPTION) Text: Build the Employer Value Proposition. (SPEECH) Also as part of this, and part of getting industry engagement, is helping to build, in conversation with them, an employer value proposition. So that they, in conversation with you, understand what the clear specific value to them is going to be. Just kind of a generic partnership, come and partner with me and with my CAEP consortia, is not going to be nearly as successful. If there's an understanding-- if you're again, problem solving. So when we did our problem solving, for instance, with the Transit Union and Mission College and Santa Clara Valley Transit Authority, we found out that half of their bus coach operators could retire today. So they had a real silver tsunami taking place. I've had other situations where they need folks to be retrained or trained in a particular use of a new technology. In the electricians end, it was automated demand response or advanced lighting controls or those kinds of things, or there's health and wellness issues they're trying to address, or they need to diversify their workforce and meet community commitments in there. So that's part of this, is understanding what that specific value of the partnership may mean to them. And that starts with real conversation. The BILT model has an elevator pitch that they have folks walk through as part of it so that the program has them understand why subject matter experts are needed, what the time commitment is, and what the employer receives out of it. So doing that kind of thing, even at a basic level, is really wonderful. And then just, again, DOL research addresses negative employer perceptions proactively. So thinking about cost, time, skepticism about program quality, which is often overcome by having them be part of that curriculum development and affirmation are really foundational engagement strategies with industry. And I guess I'll pause again, if anybody has questions at this point or thoughts. OK. I guess I'll keep going. [JOHN LAUGHS] (DESCRIPTION) Text: Industry-Led Labor Market Analysis, Phase 2. (SPEECH) Let's see. Here we go. So the next piece of this is thinking about the partnership as a way to go beyond standard labor market data that LMI, whether that's EDD, or BLS data, or even Lightcast, those kind of folks, the data that you're working with is often about trends. But it's not really that specific for what industry needs in the moment in terms of knowledge, skills, and abilities or program design for you, if you will. But starting with that Lightcast or state projection for trends is a way to help identify for the partnership's specific job titles, maybe pathway gaps, near-term hiring projections that partner employers. It's even a way-- frankly, we've used it particularly with Lightcast to have an occupation and understand the trend of hiring in a specific occupation and/or identify, say, the top 50 employers. So I could use Lightcast to identify the top 50 medical assistant hiring entities in Orange County. And that may be one place that I start for my recruitment and engagement with industry, if you will, is just an example. Accounting for demographics and retirement is another reason for going beyond this equation. Again, the Transit Union example applied with Delta College for CAI Grant to the Chancellor's Office, and that aging doesn't show up in the LMI data, and so they didn't get it initially. They've had to apply it a number of different times. But it's that kind of thing where you start to understand from your locoregional employers and industry what they may be facing in terms of demographics and retirement and other pieces. You'll also need to be thinking about specific policy and regulatory drivers. So a lot of times I think about regulatory stuff as opportunities. So, for instance, the reason that we also started those transit partnerships was the state of California had mandated that every single bus authority in the state of California has to transfer from compressed natural gas to zero emission buses by 2040. And that change is pretty significant. When transit agencies went from diesel to compressed natural gas, it's kind of the equivalent of they knew Spanish and they had to learn Portuguese. Pretty easy to do. To go from compressed natural gas to a zero emission electric bus, high-voltage bus is kind of going from Portuguese to Mandarin. You can't get from here to there, even for, frankly, a journey-level mechanic. So it really spoke to a need and an a desire, then, by the employers to engage with education and other institutions to build in that kind of training in a new way. So understanding those things or licensing changes in health care, that's another area where you'll see a lot of change, but that will also drive the opportunities that you may see. And then again, as I said earlier, there's obviously a lot of industry disruptors going on right now. What's the role that automation and AI and robotics are going to play in a field like manufacturing and different sectors of manufacturing, different business models that are taking place and just generally market changes and trying to understand those. Next slide. And so this is obviously part of Phase 2 The other piece that I want to suggest and am happy to work with folks, with us here, is around conducting an industry needs assessment. And I view this as a process and a way to really engage industry. It's not just a end product itself. It's one of those ways that you can really start to have a conversation with industry and the rest of the partnership to really talk about what needs to be put forward as actionable recommendations. And so this is something, actually, I recommend doing an initial industry needs assessment as part of that LMI, going beyond the LMI, and something that you do, frankly, with industry and your other partners annually. So that it's used as a way to affirm what's happening in terms of your curriculum and your pathway design. So you can, in terms of assessing current and projected job openings by title, and employers, what pathways may be there or what gaps. A lot of times you'll have employers who say they want entry-level folks but are only hiring, frankly, at a second or third tier. What skills and competencies, affirming what their requirements are for the targeted jobs. Are there cross-occupational skills? And that will help you get sort of an aggregate of employers, maybe, thinking about this. Are there barriers to fulfilling the demand in terms of a hiring process, issues, or training, pipeline gaps, scheduling, and compatibility? So if you've got students and learners who can't attend during the day, you may need to do some different kinds of learning delivery, if you will. Are there barriers for the workers and learners in that? And then are there equity gaps? So these are just examples of some of the things that you can work on as part of an industry needs assessment. Next slide, Debi. (DESCRIPTION) Text: Industry Needs Assessment. (SPEECH) And there's different ways to gather it. You can use forums, roundtables, employer convenings. You could do individual interviews, you could do surveys and focus groups. You may want to use Lightcast or Indeed or others and look at job descriptions and posting analysis. You may also want to go through your existing inventory of training programs and their outcomes of all of you and your CAEP consortia, those kinds of things. And then use it, frankly, as a way to get specific, prioritized recommendations, not just generalized guidance. And it will help you get agreed upon by in from partners if they're part of conducting it and being involved in it, and presenting it to your advisory committee or the partnership for validation and kind of action planning. And again, treating it as a living document that's refreshed annually. It's not just a one time activity, if you will. Next, Debi. Kind of this third phase, and I just-- This is the piece you already know and live, but I just want to acknowledge it and recommend it because this is not a place where you're being disempowered. This is where you're taking what you've learned in those first two phases of setting up the structure and doing the going beyond the LMI, and then working with industry on the co-design and credential alignment. So BILT uses-- they actually, once a year, regionally and locally, they will sit down for a structured two, two-and-a-half-hour knowledge, skills, and ability analysis. They will actually come up with upwards of 50 to 70 knowledge, skills, and abilities for the particular sector or occupations that they're talking about with their education partners. Shortly thereafter, they actually meet with the faculty and the program to prioritize those KSAs in their existing courses, and then the faculty is affirming and using that to make changes in modification. So industry is telling faculty what's needed, and faculty is determining how and when to address it. That's the part of the model and partnership that exists in there. Next one, Debi. (DESCRIPTION) Text: Curriculum Co-Design and Credential Alignment. (SPEECH) Another interesting activity that I really thought folks may want to think about and do as part of that curriculum co-design and credential alignment is a number of the BILT folks also do what are called in other entities, do what are called trend meetings. So they may be quarterly or two or three shorter kind of virtual meetings during the year, where they provide forward-looking intelligence, that they're thinking about curriculum and staying ahead of the industry, not just reacting to current needs. So that may be the place where you have what's going to happen in the next while with AI, or other technology changes, those kinds of things. If you've got industry partners, you can focus on industry innovation three, five years down the line. You can see the discussion. They have their participating industry members basically prepped to participate around what they're going to see and do. A number of them has set-up structured checklists that they use each time they meet to discuss the trend, the implication for educators, assess the timing and adoption, identify skills that you can put together and project risk for being able to do it or not do it. And again, they are very clear, too, about including faculty in that first trend meeting, particularly after the KSA meetings. The other is what can you do in terms of work-based learning and other engagement opportunities based on that, incorporating those trend meetings. And it also may be a way for those trend meetings to have an initial recruitment of potentially new employers and industry into your partnership as well. (DESCRIPTION) Text: Work-Based Learning Infrastructure, Phase 4. (SPEECH) And this fourth phase, this is just a structured way for folks to think about as you're building out your work-based learning continuum, and the partnership, is both for yourselves and for your other industry partners about different ways to incorporate work-based learning along the continuum. And so I've thought about this. Obviously, a of folks involve them in career panels, site tours, informational interviews under awareness that exploration may be providing opportunities for workers, learners to do job shadowing, have folks do guest lectures or project briefs with your class and workers, preparation in terms of project-based learning, simulated workplace or capstone programs for your workers and learners. And industry should be part of helping you create some of these. Then, obviously, the training piece. I'm a big fan of paid internships. Those cooperative education is also really valuable. I have a nephew back East who is in Ohio and got to do a cooperative editor experience with General Electric in Cincinnati, and it just really gave him a grounding of what to think about. Pre-apprenticeships are obviously another place for this kind of opportunity. Keep in mind that you need to explore what the state's requirements are around hiring and the state FLSA wages versus an internship and those kinds of things. And then, obviously, deep training, and that's particularly registered apprenticeship or other opportunities that you may have. And using, if you've got, apprenticeship programs in part of your consortia thinking about OJT and in related and technical instruction, and how those fit in to federal and state registration, if you will. (DESCRIPTION) Text: Faculty and Instructor Industry Currency, Phase 5. (SPEECH) I'll maybe pause for a second. I'm still wondering whether there's folks there, if folks have questions or comments at this point. And we've got two more phases I'm going to walk through. But I just want to make sure folks have an opportunity to ask. DEBI PEZZUTO: Well, we have one comment from Van that says, great slide that illustrates the spectrum of commitment levels. JOHN BRAUER: Thank you, Van. DEBI PEZZUTO: Thanks, Van. [JOHN LAUGHS] JOHN BRAUER: The other thing I-- and again, I think I really am a believer in, helping faculty and all of us be adaptive that industry and its needs and priorities and changes are constant. So an ability to keep faculty and instructors in your departments and programs connected and learning what current industry practice is in some structured, ongoing employer engagement is to your benefit. In just some ways, if that's possible at all, is to implement structured faculty externships. You can obviously bring in folks to do that. Offering industry guest lecture series, developing practitioners in residence arrangements, so you have part-time industry professionals teaching co-courses with you, those kinds of things. I also think figuring out ways to ensure access to industry-training platforms. And some folks, some I've got here are like Cisco NetAcad, and your employer partners, or other industry partners may be able to give you access to those. I think we are also TA for ELL Healthcare Pathways. And I know some of the partners in there, they've had an ability to access things like Epic, which is the medical coding apps for that through their partnership, those kinds of things. And then also thinking about creating professional learning communities that span your consortia, or a number of consortia, which is, I guess, is what ELL Healthcare Pathways is doing there. But there may be other sectors or things to do in that. So across college and adult school faculty networks meeting to align with the regional employer partnerships. And I think investing in the folks who are delivering is really important. So this is, again, some food for thought for you all in that partnership. (DESCRIPTION) Text: Data, Continuous Improvement and Scale. (SPEECH) And I will say, before I get to this next slide, it also indicates industry that you want to remain current and meet their needs. And so I think enacting some of those, I think is a really good way of showing them that you want to stay current with what's happening in the sector. This last part, so Phase 6, these are just sample outcomes for you to be thinking about. They are used by BILT in the hybrid training partnerships and in DOL and others. So some of these, really, again, are going back to both meeting the needs that you may have as educational institutions for your grants and contracts and funding at the state, as well as meeting the needs of employers. So looking at enrollment and completion, credential attainment, particularly industry-recognized credential completion rates, not just your certificates and degrees, but also those that may be of real value to industry. Your placement rate. So a good indicator of are you meeting industry's needs, frankly, is are they hiring? Are folks coming through our programs and participating the starting wage? I know you all have living wage and other requirements that you're shooting for. So being really looking at that. The retention rate. And I think that's both necessary for you in terms of meeting needs that you may have, but also, that retention rate, I think is also an indicator of the industry's practice in the partnership. That they're not just running students through and workers and exploiting them, and then getting the next set of folks, that they're also really invested in the folks that you're sending them, which really speaks to job quality and importance. Employer satisfaction. And doing it in relationship to the graduate preparedness, that the folks that you're putting out into the work environment are folks that are meeting their needs, not just are they happy or not with the partnership. Looking at end of placement surveys for your work-based learning, and the quality and learning of that, you all, maybe more so than industry may have some real need to be looking at outcomes in relationship to race and gender and age and other things. And then some engagement metrics you may want to create. And these are some that BILT has also incorporated in there, if you will. Go ahead to next slide, Debi? (DESCRIPTION) Text: Plan for Scale from the Start. (SPEECH) Just some food for thought for you all. We've kind of walked through those six areas, or six phases, if you will. One of the things that the research also indicates is that, for all of the different areas, is that even when you're launching a pilot or designing it with scale in mind, think about it, of trying to go to scale, to big, that they're not designed just to-- pilots don't just inform a broader practice of valuable learning, and they have kind of a finite impact, if you will. And so thinking about linking the learners to existing high-quality training providers or programs you already have or employer opportunities, thinking about aggregating, either across your programs or colleges or across a number of different employers, maybe through an industry association that is not just a new program creation that you're working on. You may want to focus it, initially, on, say, underserved meeting, underserved training needs, where those quality options don't exist, but that you're looking to have a higher investment and a longer lead time to get yourself there. It may take you a while, initially. And then aggregating. Again, I said a moment ago, aggregate that pool learner demand across multiple employers. So if you just lose an employer that you've started with, you're not out of luck. I'm going to not use bad words here, Meghan. SOL. And just figuring out the most scalable model when employer demand is fragmented and what that looks like at the start. And next slide, Debi, I think I have another slide relative to the scale piece. Some additional strategies that folks have thought about in terms of starting from scale. So HRTP and the BILT folks. So starting with one program at one college, let success inspire adoption in addition-- adjacent programs. Sorry, I've been speaking here for a while. So they've kind of moved, they've learned in one thing and moved to replicate it or expanded it. They've also learned to recruit internal influencer champions. So that faculty lead or dean or VIP who can clear institutional barriers and inspire their peers. I think you guys probably know this better than I do, that that can, at times, be really important. There are different models that folks have done in terms of having subsectors or subdisciplines within a particular sector or industry, maybe just one way to take an initial bite and then grow it farther. And then in another area, and again, happy to talk to folks outside of this, is leveraging sector intermediaries that may already exist in your region. That could be the local workforce board, it could be an industry association, it could be an existing partnership, industry partnership, that you want to become part of or have them fold into you and build off of. So rather than just creating the wheel for something that's already existing, you're going to places to identify champions and build that employer-to-employer recruitment, if you will. So next slide. I think we're getting near the end here. And again, I just wanted to reiterate of those phases, if I could, that this is all cyclical, not linear. So again, the industry-led labor market analysis and curriculum design, that you're doing those annually at a minimum. You're looking at data and doing continuous improvement. You're thinking about going to scale. And those things feed back into helping you in your labor market analysis and going beyond just basic LMI, if you will. The work-based learning infrastructure you may create can give you real-time intelligence about what's going on in the workplace, both your learners and your industry partners to help you down the line, revisit and make changes, needed changes to your curriculum design and credential alignment. And just that it's a continuous improvement loop, and not a one-time launch, if you will, around here. And some challenges. I'll just share a couple of these. But I've got two slides here of some things, just where folks have seen challenges in setting this up, if you will. Again, employers view participation is too time-consuming. So presenting precise time commitments so they know what they're getting into, what their role is. Again, for them, time is money, obviously, in the work that they do. Faculty fear losing control to employers, again, making sure that they are part of that, obviously, the central role on the curriculum design and affirmation and incorporation based on industry need. There may be institutional resistance to entering into some partnership model overall at your adult school or college. So maybe just starting with a program and documenting the outcomes. Let the success build the case for the expansion, if you will. A big challenge is making sure that your advisory boards move beyond being just a rubber stamp committee. I know industry folks who have said, well, the advisory committee asked me to meet once a year to tell me how great the program is, as opposed to asking for my input and partnership, if you will. So those are those. Another slide, Debi. And just some others to share around finding committed SME employers, having different cultures between education and industry program, misalignment with actual labor market demand. There may be equity goals for you all that are in tension with employer. Preferences or needs. The data collection may be a burden relative to employers and reporting. And then obviously, the sustainability of funding and engagement over time. But one of the things that I would encourage you all to do is look for a way, as part of that, maybe I need to put this into the earlier slide on the engagement piece, like the work-based learning, is having your employers put some skin in the game. They will be more invested and that will grow over time. That's the significant piece around, for instance, the largest apprentice programs in the state of California are in construction and in the firefighting. A number of you are the LEA for. Those employers are putting DOL a sense on the hour into those training programs. They have invested in them. They are making them more sustainable. They don't cover everything, but they make them a lot more sustainable than other training and education models up and down the state. Great, Janice, thank you for your good example. Debi, I think that's it. Yeah DEBI PEZZUTO: That is, and thank you so much, John. We've got a lot of compliments in the chat. No questions this time. But Van did say he'd reach out. They'd reach out. Janice has got some comments about CTE, overcome outcomes are not physically compensated in the CAEP funding model. Those funds could also be used for hiring transition specialist counselors, right? Question mark. In the recent past, staffing up to increase WIOA II outcomes in ASEN and ESL has produced a stronger fiscal model. If you've got rollover funds, I would suggest hiring a transition specialist as I was one. [DEBI LAUGHS] I see how vital the role is in employer partnerships. If you have more questions or you want to chat with us, you can send me an email and I'll get it out to Meghan and John, and set up a time for us all to chat, or to some of us to chat, depending on what you need. Holly already put in some links in the chat to check out the evaluation, and then the two upcoming webinars that are part of this three-webinar series. And then look out in the fall, we will also have two virtual sessions, again, focused on employer engagement. So, yeah, if there's no more comments, questions, concerns. Meghan, did you want to say anything? MEGHAN MCBRIDE: I just wanted to reiterate, you all can reach out to us if you would like us to help you identify employers in high-demand industries who are hiring or what those high-demand industries are. We also offer one-on-one TA looking at your data and data vista. And again, that data should drop next Thursday. And we're happy to look at that with you one-on-one. That's all. DEBI PEZZUTO: And soon we will also have a webinar with the data vista team just highlighting the changes or the updates to data vista for CAEP. So you should see that relatively soon as well. All right. Thank you. HOLLY: Perfect. And thank you to the WestEd team. Thank you, Jennifer, with the Chancellor's Office for being here. I did drop those links in the chat. I know they're a little chunky with the URL, but I do that on purpose because if you click the three dots in the top-right corner of the chat, it will allow you, if you don't have time to click and register for the events now, you can actually save our chats, here at CAEP TAP, to your computer, which will allow you to then grab the URLs later and register for those events. So that's why we post the chunky URL. Thank you WestEd team. Thank you, John. Thank you, everyone, for giving up an hour of your afternoon to be here with us. We do hope to see you on the 12th and on the 19th for the continuation of this series, and we will see you at the next event. Thank you, everyone. Have a great rest of your day. DEBI PEZZUTO: Thanks, Holly. Bye, bye. HOLLY: Bye, bye.