Blaire Toso: Hi. Thanks Mandilee and Holly for accommodating us and supporting this webinar. I always forget towards the end when we get busy. So I wanted to give a shout out to them. They're always the most gracious of hosts for our webinars

So thank you very much. My name is Blaire Toso and I am from WestEd and we are looking at Data for Continuous Improvement and 3-Year Planning today. Next slide, please. So I am joined here by my colleagues Jessica Keach and Ayanna Smith, and you'll hear from each of them later today. We are the mighty CAEP team over at WestEd and then we're also supported by many other people at WestEd on building the dashboard and working on data and professional development.

So we are here today to support you and hopefully give you some information and really position this work so that you can hear from some of your colleagues in that field. Next slide. So, as Mandilee mentioned, we are joined by our guest presenters, Ute Maschke, Ilse Pollet, Rick Abare, and Jenna Cestone. And we're really delighted that they're here. I'll start us out with some background information on continuous improvement, but they'll really bring us home by giving us concrete examples of how they're using data and investigating data to prompt questions and think about their programming and engage in this continuous improvement cycle.

Next slide, please. We'd also like to welcome Mayra Diaz and Lindsay Williams from the Chancellor's Office. As usual, they're here supporting our work, which we really appreciate.

They support the dashboard and make sense of the work. And we, in turn, help support the CAEP work that goes on across the state. But I'd like to turn it over to Mayra now to say a few words.

Mayra Diaz: Thank you, Blaire. Just want to say welcome everyone. Thank you for participating in today's webinar presentation. We are excited that you can join us and excited to hear from today's group of presenters. So thank you. Thank you for having us here.

Blaire Toso: So, as I said, we're going to go-- excuse me. We're going to go quickly over the continuous improvement process. We'll have a couple of activities. And we'll move pretty quickly from one to the other because we want to make sure that we stay on schedule and have a chance to learn from the field, as I mentioned. And then we'll have a little bit of time for discussion and questions and answers.

Next slide, please. Our goals are really to be able to learn about for continuous improvement phases, use the five whys to identify and refine the issue, and think about using data to explore and identify solutions. Next slide. So continuous improvement is really about-- it's an ongoing commitment to quality improvement efforts that are evidence based, integrated into the daily work of your program, and contextualized as well as iterative, which is why you see those arrows that go around in a circle. Just because you complete one cycle does not mean that you're finished.

It's continually looking at your learners, your programming, your stakeholders, and seeing what works best for them. Continuous improvement has been shown to increase reaching target outcomes. It helps to decrease inefficiencies. It also helps to really identify those solutions that work because it allows for realistic planning and testing. And, again, going back and forth and not thinking that you found the solution the very first time.

It also allows for incremental and, over time, fixes. As I said, you can take off a small bite and begin to work with it there. And then increasingly explore an issue over time as you would like. So you can do a small quick fix, but it isn't intended as a quick fix to your full system. Continuous improvement implies that you take little pieces of it, examine it, and come back, and we'll hear about those different efforts from your colleagues later on.

It does help you focus on a single issue and allows you to go deep so that you can identify what's going on and then come to possibly explore and come to a solution. Also, I did want to-- there are many different ways of naming the phases of the continuous improvement process. This is the one, the identify, plan, execute, and review is the one that I have chosen to use. But many of you are probably also familiar with plan, do, study, and act. And I use this one because I like to be really intentional about identifying what you think is going on and possible solutions.

But you can choose whichever one best fits you and your context and using language that resonates with you and indicates a process that you're willing to take up. Next slide, please. So really when you look at identify, you are doing this by thinking about how you're going to use observation, data, and inquiry. You're asking about what is the problem. What's really going on, how might we change it, and how might we address the issue?

So in identify, you're not just talking about the problem, but you're also thinking about, oh, what do we need in order to solve this? Is there an intervention or a solution or a strategy that we might be able to use? And I just want to say, for anybody who joined us last week when we were looking at the equity walk, it's very similar to how one starts the equity walk, which is by using data to spur deep thinking about an issue. Next slide, please.

Jessica Keach: All right. Thanks, Blaire. Now we're going to put some of this thinking into practice by looking at some data and walking through a reflection activity. And you're looking at data from the most recent release of the adult education pipeline. On the left, you see the number of participants by race and ethnicity that earned a diploma, GED, or high school equivalency in a pre-pandemic year, which is the 2018-19 year compared to 2020-21.

And on the right, you're looking at the percent change, or really the percentage decline, in students meeting the same outcome by race/ethnicity during the time frame. So on the left, you have the counts. On the right, you're really looking at the percentage decline by race/ethnicity, and comparing that to the state decline, which was about 43%. So the line represents the statewide change and then each bar represents the change among that particular demographic group. So, for example, between 2018-19, and 2020-21, the number of Asian students that attained a diploma, a high school diploma, GED, or high school equivalency declined by 52% compared to a 43% decline statewide.

I'm going to pause just for a minute to allow you to look at this data and take it in. All right. Now I'm going to turn it over to Ayanna, and she's going to kick off an activity where you are going to be able to reflect on what you're seeing in this data.

Ayanna Smith: Thanks, Jessica. So we have a link to a Jamboard. And, Jessica or Blaire, if either of you could paste that in the chat for me, please. And on the Jamboard we want you to reflect on that data that Jessica just went over with you. So go ahead and click on the link. And then the data is there on the first slide in the Jamboard.

So you don't have to worry about remembering what was there. You can definitely continue looking over that data and finding what stands out to you. And our discussion question is just to define the problem. What problems do you see when you look at those numbers? And so you can go ahead and create a post-it note, a sticky note, to add your comments there onto the Jamboard.

OK, thank you for those of you who are already adding your comments.

Blaire Toso: Ayanna, can you maybe share the screen so people can see what's on the Jamboard?

Ayanna Smith: Yes, I wasn't sure what happened. Let's see. Can you see the Jamboard?

Blaire Toso: Yeah, thank you.

Ayanna Smith: OK.

Blaire Toso: This is great. I also see a couple of comments on the previous slide as well where it talks about disproportionate declines for some groups. And then yeah, was there a shift in delivery online that impacted some groups more than others?

And yeah, it looks like when I'm sorting through these that there are challenges people are identifying, a problem that maybe some groups experienced greater challenges and then what impacted them. Definitely about-- somebody is asking, is the problem declining enrollment or attrition?

Are folks leaving the state or are they leaving adult ed? Are we not able to serve the students or do we not have enough teachers to serve a greater need not represented by this data? That the number of participants declined by 17% but the number of completers declined by 43%, interesting, yeah. So what's the problem? Is it-- that goes back to somebody asking about enrollment or attrition.

And then the numbers of students completing this outcome was low even before the pandemic. And then, however, the pandemic impacted some race, ethnicity groups more than others for this.

Were Asians fearful to come to campus due to the racial threats they were experiencing during COVID? Exactly. So that's great. Let's go ahead and go back to the discussion, the slide. And keep that bitly link or if you're in the board, keep it open because we'll go back into that.

So we wanted-- once you identify that problem, one of the things that we talk about in the continuous improvement process is to dig into those issues, right? And there's a process of doing this called the Five Whys. I'm just curious, has anybody used this in trying to solution around problems that they've identified to really dig into this?

Can't see if any hands are going up. I saw a couple of nods, for those of you who have your cameras on. That's great. Thank you. Thanks, Jenna. So the Five Whys really helps to get you to what's really at the heart of the issue you're discussing. So for example, sometimes what we think is the issue masks the real issue.

We might say, why students aren't enrolling in IELCS courses because the IET is not of interest to them. They don't want-- they're not interested in that occupation. However, if we look closer and we ask why aren't they interested in that occupation, it might be that we've begun to think about, well, there are a lot of women in our classes. Maybe that is a nontraditional occupation for women. Or it might be that they are not aware of what the job is.

Speaker 1: You have entered the waiting room. You are muted.

Blaire Toso: Thank you. Or it might be that they feel that they won't. We've even heard from students who say, well, we were interested. But we didn't feel like we had enough English skills to be successful or we didn't even know who to ask or where to go for it. And that each time you ask a successive why, like, why isn't a student interested?

Oh, well, because we think it's primarily women. Well, why do you think women aren't interested? Well, it's a nontraditional job. Why do you think it's a what-- it's not a nontraditional job? Well, there are societal perceptions about what this job is. Keep asking those whys. It helps you dig into it.

And the other piece, when you're digging into these whys, is to take it off of being inherent in the student. We talked about this in the equity as well-- is a lot of times it's about the system or our programming. So when we're asking those questions, digging into, so why aren't they?

And then the next question as opposed to being an attribute of the student might want to be, like, what is it about our society that creates that? What is it-- or the way our program is structured. Why don't they feel comfortable coming and talking to us as opposed to, well, they just don't have enough English to understand it.

So this is also a technique that's used to get at root causes, which is an important process for us as well, even if we're not looking at the root cause, even if we're just trying to dig into what we want a solution around-- is because it will help you refine your question and target the problem and identify solutions that address the issue.

So what we're going to do now-- next slide, please-- is we're going to do a quick breakout group about practicing those whys because sometimes those whys feel a little funny after you get past the second, when you're like, hmm, what is that next why going to be? But if you practice it, you'll begin to see how it just digs in and helps you refine into not just the problem, but then thinking about the strategy that you might want to employ or test out.

So we're going to go into random breakout groups. We're going to go back to the jamboard. And then there will be-- you'll look at your group number. And go in and post on one of the jamboards. They're labeled by group one, two, three, four. Five go ahead and use them. And then post your whys there. And Ayanna, can you just quickly share the screen of that those slides so people can see what it looks like?

So there, Ayanna will show you. If you click on the top carrot to the right, you will see how you have gotten-- you'll each have a jamboard that has the different whys. And just use your post-its to ask. Print your first question. Dig into your second question, your third-- until the fifth. And see how you are able to refine that question. Are there any questions before we go into the breakout rooms? Excellent. So we'll see you in a breakout room.

Looks like we're all back. And we have some great questions that popped up. And it is interesting to see where people got stuck and where they were able to populate further. So I just wanted to quickly touch base. If you flip through them, does anybody want to share what kinds of questions you came up with? And what new ideas or clarifications did you experience through this process? How did it refine your thinking?

Does anybody from group three-- we can see you're on the screen. You've got a lot of questions.

Speaker 2: Yeah, hi, everyone. This is Dulce.

Blaire Toso: Hi.

Speaker 2: I think for us-- I was part of group three. And one of the things-- the first thing that came to mind, at least for some of us, was pandemic, pandemic, pandemic. Hence our little post-it here, pandemic. Because that's been kind of on the tip of our tongues for all of these enrollments, right?

And then so it was hard to say, OK, how do we go from pandemic to another why from a pandemic, right? So that was kind of hard. But then we kind of talked about other whys including this-- we had already seen that there was an existing drop in K through 12 enrollments historically, even prior to the pandemic.

We talked about external factors that were impacting our students to access their courses, right? So lack of internet, a lot of job loss, food and housing insecurities, all of those pieces. So that's kind of where we ended up going.

But I have to say, it was hard once you hit-- when you throw out a why and answer it with a pandemic, this is the first time we've had to deal with this at least since the last 100 years. So it is hard to understand it in the context of a very unique outlier circumstance.

Blaire Toso: Yeah, thanks for pushing on that one a little bit, Dulce. I think that's a really valid point. But also to the point of saying, OK, so this is the pandemic. It also makes us look at other factors, right? I think that one of the things the pandemic has-- why has the pandemic hit us so hard? Part of it is where you get to in that third one about being affected by factors such as digital access or to take care of family.

So you're digging into not just that it's the pandemic, but why? And what structures within our system affected the fact that the pandemic not just sent us home and people had so many other things going in their lives. But what within our system, had it been in place, might have been able to better support adult learners and keep them engaged?

So I think that that is a really interesting example of yes, this is where we get stuck. And we know the answer. But digging into it helps us identify some places to go with it. Right, we're going to go ahead and move along so that we can make sure we have time. But that's exactly where it's supposed to push. Those kinds of discussions, it doesn't necessarily answer. But it does push discussions and make us think a bit more deeply about the issues.

So then we have the continuing improvement process. We move into the plan. And this is one that you all, I'm sure, are well familiar with-- identifying what data do you need to collect, what activities do you need to do to collect the data, what's your timeline who needs to be involved, right? And as you do this, you're going to want to continue to explore what intervention would you want to test if you look at Dulce.

And many of you went this route once you identified digital literacy, technology. People were building up boot camps in response and then figuring out if that was working. Provide-- getting technology into learners' hands, access to broadband-- so all of those pieces were things that you all identified. And it would be very interesting to see how you all landed on what you all thought the effectiveness of those strategies were.

So next slide, please. We're going to do a chat. But I think we're going to go ahead and continue on. And so it would have just been, like, essential to that planning process is identifying the data that you would like to collect. And then there's the execute, which is where you're really implementing your plan.

And throughout this, you want to make sure you're taking really concrete and copious notes because that's part of your data collection is understanding the process that you're going through. You're going to want to stay true to your plan unless, when you're revisiting it in the midstream, that you as a collective are already able to say, this strategy is not working.

And so we need to go back to that identify and plan section. And then you would swerve back. But you would have that mini review in there. And you can do that pretty quickly if you're executing and something isn't working.

You want to make sure that you're engaging your stakeholders. And then even before you get to the review process, you want to begin to see if any patterns or trends are emerging from that process. Next slide.

And then the review is really about, what are you learning? You'll analyze your data and determine whether your intervention or strategy has worked. And that can be simply, yes, let's go forward and continue looking at it and implementing it and maybe turn our eyes to a different issue or it might also be-- a question is, OK, it's working. But does it need to be tweaked? And you'll see that within your data.

And you'll be asking questions about what action steps you want to take and why and then identifying those next steps. And just because you've hit the review space, that doesn't mean it's over. You may want to test the strategy or intervention again, beginning the cycle there, or you want to abandon your strategy and revisit and start with something anew, or you may just want to say, this is great.

Let's move on to something else and begin that identify, plan, execute, and review process again. Regardless, you really don't want to abandon your work. You'll want to take a look back at what you've done, even if it doesn't work because that's a historical look at what you have been doing and what you may not want to consider again.

And the other piece that you can learn about this is establishing a process for going through these continuous improvements as you identify issues in your work. You can use the same process and procedures that you have already identified if they have worked for you and your team.

And then you'll always want to be continually monitoring your solution, even if you've identified that it works and you're moving on to a new issue. Keep those pieces in place so you can monitor and then make adjustments as maybe something in your consortium changes in your program changes, whether that's learners, new labor market information, or just new stakeholders.

Does anybody have any questions before we go on to some concrete examples of how this gets applied? Super. We're going to hear from South Bay Consortium for Adult Education first. Then we'll move to East Region.

Can you go back one more sec? Thanks. To East Region Adult Education Consortium and then we'll move to the Silicon Valley. And they'll each have been looking at a different issue and how they worked through that.

If you can either post your questions in the chat or hold them into the end so that we can make sure everybody has a chance to participate. And then we'll have a discussion time at the end. Thank you. I'm going to hand it over to Ilse.

Ilse Pollet: Thank you. Good afternoon, everyone. And thank you to the WestEd team and CAEP TAP team for inviting us from the South Bay Consortium for Adult Ed to share a little bit about our experience with continuous improvement and using data for continuous improvement. We wanted to touch on two broad areas where this applies in our consortium.

And one is in our day-to-day consortium work as a consortium as a whole. How do we approach data? And how do we commit to using it as a continuous improvement opportunity? And then of course, we're in a three-year planning year. So we also want to share a little bit more about how we have used data in our three-year planning process and how that is really informing and shaping our strategic plan for the next three years.

So I'll start by talking a bit more about, as a consortium overall, how do we look at data? And then my colleague, Rick Abare, will talk about how we approach the three-year planning process and using data in that one. So as a consortium overall, we made a commitment three years ago in our active three-year plan to be a data-informed consortium.

And I was happy to see that what was shared today was a cycle and was not a linear process. So this whole cycle of review, identify, plan, execute, it's an iterative process. And you go back and forth. And you can make small changes or bigger changes and then just always reassess and go back to your-- back to the drawing board. And that's really how it has been for us as a consortium.

And when I look at that circle, that cycle of continuous improvements, I would say as a consortium, we are somewhere in between the review and identify stages of continuous improvement. And that may be surprising because we've been at this for about three years or even more now. But it really was important for us, in order to be able to review our data, to be able to trust our data.

So we actually spend a long time and a good amount of effort in really making sure that we trust our data, that everybody in our consortium understands where the data is coming from, how it flows from the student information system into TE, into NOVA, into LaunchBoard. It's a complex landscape that we're operating in.

We are the only consortium in the state that has two community college districts. So it makes it a little more complicated. And we have five adult schools who don't all use the same student information system. So it was really a puzzle to understand how everything flows and how everything comes together.

And then the other piece of that is making sure that there is uniformity in how we collect and report data across our members so that when we look at data at the consortium level, we are comparing apples to apples. And we're talking about the same things. So we really built our capacity to understand our data, to talk about our data, to develop that data literacy among our members, to even have the conversations.

So we do that in Quarterly Data Study sessions where we look at our enrollment data overall and enrollment data broken out by program area. And that has just been a very helpful process and really engaging to be looking at our data at a consortium level. And then the questions, the whys, come up almost automatically because you'll look at numbers.

You see numbers coming down. And someone will raise their hand and say, oh, why is that? I see this ESL go up here but ESL go down there with another member. So we do have those why questions. In the next three years, I think we are going to move as a consortium into that planning and executing phase and really responding to what we're seeing in the data.

I feel like we have a strong foundation now, as to being able to talk about our data, to trust the data that we're looking at. We have the space in our Quarterly Data study sessions to take the time to really dig into the data with all of our members.

And so the next step will be for us to not only identify the problems and talk about the whys, but also ask that question as to what are we going to do? Which strategies are we going to develop and employ to address some of these areas?

And the area in which we think we'll see the most activity is in the transitions piece. So I think that's where it really comes down to consortium-level work to look at, globally, across our nine members, how our students moving in between our schools and our colleges. And are the pathways that we developed, are they really working? Are people trying positioning where we would expect them to transition?

And if they're not, what kind of support do they need? What kind of programmatic changes can we make to increase those transitions? So that's going to be a big focus in our next three-year plan. So I think that's globally as a consortium where we are. And then I'm going to turn it over to Rick to talk more specifically about how we use data in our three-year planning process.

Rick Abare: Cool. Thank you, Ilse. And echoing my thanks to everyone from the WestEd and CAEP TAP teams for having us. This is a lot of fun to get to hear from you and from other people about how they're using data for these types of really important initiatives because-- Ilse mentioned that we do our Quarterly Data study sessions.

That, in conjunction with using the CAEP fact sheets and LaunchBoard and several other tools, which I'll get into a little bit more in a second, allowed us to almost get meta with our own process and decide how we were going to improve the way that we approach the sharing of data internally and aligning ourselves to the three-year plan.

It also created a really great plan for us to start-- I think it was back in August, by doing, essentially following, the playbook of the requirements of the three-year plan, which was to identify our gaps to start with. So we looked at the fact sheets to do a, make sure we were on top of our regional need, which helps inform those three-year plan activities.

We built-- we call it a CTE matrix. But it's functionally a big list of all the CTE programs that we offer so that we could help identify gaps in conjunction with folks from our local work2future folding in labor market data to see, hey, what do we offer? What's in demand that we don't offer? What's in demand that we do offer but we may need to create a pathway for?

So that type of activity was basically step one. Step two was to leverage the Data Study session space to engage the stakeholders from the rest from our members because it's one thing if us as a consortium team just sit around and think about all this.

But we need to hear from everybody. We need to get them bought in on why we're thinking what we're thinking, especially when we want them to sign up for activities that they say they're going to do for the next three years.

So we took that space. And we did a few interesting activities that I thought were really valuable to hear back from. So we looked at-- there's a great online mapping interface called the Healthy Places Index that's active in California. And it's a zip code by zip code breakdown of, I think it's eight-- it's an index that they create at a zip code level of eight to 10 education, health-type measures where they rank in percentiles each of those measures.

And then they create an index out of that. That is a great way for us to stay in touch with, hey, we all think we know what the need is like around us. But does that actually match up? How does that match up with actual data?

Following that, we basically just created a map of our region and loaded all of the student zip codes into a density map to see, are we serving the students who we need to be serving-- and tried to spend some time thinking about-- understanding that in the context of the CAEP fact sheet's view on regional need, which is very top-down.

Related to the exercise we sort of just did, our biggest change here in our region has been our ESL enrollment. And we wanted to take a deeper look about halfway through last year as we were-- I'm sorry. I think it was in September, maybe November. We were trying to figure out, well, who's coming back? And how is that different?

And one of the great exercises that we did was to look at the-- race ethnicity is a really tough proxy. We have a lot of folks who are identified by these tools as Asian. But they come from all over the world and have to get lumped into that category. So to try to ameliorate that, we actually looked at language spoken at home, which gave us a better indication about-- we have a large Vietnamese population here.

It gave us a better indication who's returning to school in 2021? What does that even look like? So that kind of helped us dig a little bit deeper into some of the ways that different groups of people felt more or less comfortable coming back and what we could potentially do about that. We followed that up with diving deep into LaunchBoard to make sure we were as clear as we can be.

And it's still a work in progress on our data definitions-- to ensure that, as Ilse talked about, do we even know what we're talking about? To make sure that when we look at data that it's aligned to the way that we're defining these adult ed metrics in LaunchBoard so that we can populate goal setting trackers in preparation for this three-year plan.

That's been the most recent work we've done. We've gone around and done one-on-ones with all of our schools and looked at what we have for their last three years and had individual conversations about what strategies they were comfortable taking and what goals they wanted to set around which optional metrics that aligned to our plan.

So it has been a journey over the course of the year of taking our Data Study sessions, which were often just-- it was a place for us to look at our enrollment. It was a place for us to discuss data-related issues that we needed to have a consortium-level understanding on and actually use it to do the three-year plan.

But then going from that, to improve on this, is to align the Quarterly Data Study sessions to these metrics and monitor progress on this stuff so that we're fully aligned as a consortium with the way that we're reporting or the way that we're goal-setting based on our three-year plan. So I think I wanted to do that faster. If I do have one more minute, I did want to just give a shout out to that LaunchBoard for Equity Analysis session.

That was a really great session. I think there's some really valuable tools that are immediately turnkey to working at a member level. I actually, as part of two years ago's Regional Need Analysis for that annual plan, we went into LaunchBoard and we examined outcomes and divided those out by demographics and look to see if we could find any equity issues that were very obvious between percentages of groups achieving outcomes.

So I think the tool is really good. And there's a lot of value to that. I think that's pretty much-- did I forget anything, Ilse?

Ilse Pollet: No, I think you covered everything. Thank you, Rick. And yeah, we're looking forward to any questions or comments. And please feel free to contact us if you have any questions about this.

Blaire Toso: Thanks, Ilse and Rick. We will we should have time at the end for some questions I know I have questions. So hopefully we'll get to that point. But at this point, can we go ahead and switch over to Ute? And Ute, talk a little bit about your process.

Ute Maschke: Yes, good afternoon. Thank you for having me. And WestEd and CAEP for having me here. I want to share the screen in a moment to share a little bit, I think along the same lines that Ilse and Rick brought up, consortium-wide, but then also trust building internally for next steps within that concept of continuous improvement.

So the first, more consortium-wide, our consortium only has three members, one of which takes about 95% of all the programming. And when we moved into the strategic planning process for the next three years, we ran into this issue, how can we ever compare our work with one partner when one partner has 95% of the students and the other one has a handful?

So we needed to find a way to compare, as Ilse already said, apples to apples. And we needed to-- per request, we wanted to simplify it, make it more easy to work with without dumbing it down. So we decided together-- and that's what you see on my screen now. We decided together on the key points we want to measure. And similar to South Bay, transitions is what matters.

How do we measure successful transitions across very different partners? So we created together this spreadsheet as a preliminary first draft of the metrics that we could use when one member has 20 students and the other one has, say, 8,000. We can still measure transitions. We started looking at the data in LaunchBoard. And then compared it to the data we had internally.

And we discovered, just by identifying those descriptors, some discrepancies that allowed us to start a conversation. As you can see in column A, the categories that will look familiar to most of you. It's outcomes reports, right? And sometimes even summary reports. We discovered discrepancies that allowed us to have a really insightful conversation that also led to that trust building that Ilse was talking about.

So our welding class is one of the most successful ones. We routinely, every single term, have 24 completers. When I look in LaunchBoard, I have five. We then did sort of a backward design. If it's only five in LaunchBoard, what's happening in TopScroll? What's happening in our internal data tracking system? Where's the hiccup?

And we included not just program directors in our data tab but then also the teachers. And I think, after all these years, it was the first time that a teacher had this aha moment and understood what data entry means and the consequences, the lack thereof, has. So it was a refreshing conversation because I don't think the teacher ever connected the dots that way.

This measuring transitions also, I think for the very first time, allows us to include our transition services in the metrics. There is not really a space yet in our reporting to the states where we can report on our work in transition and student support services. At the colleges' side, there are different mechanisms to do so. But at k-12 and our schools, we usually don't have a space other than in the update record, those three boxes.

But we don't really have a space where we can report our work in student support services. So we created these descriptors for our transition services, for example, leveraging transition services for co-enrollment with colleges, to finally have a tool to report our work. That is so crucial with the adult learners-- the support system around classes, around training programs.

So we now also have internally a tool that we can use with teachers, with our governing board, to make a case for solid transition services. And I think we also finally have the beginnings of a metrics that allows us to make a case to the state that we need to invest in these transition and support services way more than we do. The program outcomes we now measure usually don't include these support services outside of the classroom.

And we have also been working on a better, more insightful, more trusting data dialogue. When we talked about improvements internally, we wanted to measure what happens between orientation and initial pre-testing and start of classes. And we had quite a high number of students dropping out between-- they took the pre-test. But they wouldn't show up for classes. And we wanted to figure out what's going on there.

So what we decided on was we looked at scores and placement data just following the NRS guidelines. And this is color-coded here. And I wanted to share this one for two reasons. Number one, not too surprising, we actually discovered that we had students in our previous classes who didn't belong there.

We also decided actually, there is no pre-literacy. This is beginning years. We want to modify and revise our curriculum to better address the needs of those students who test in the so-called lower scores. And just looking at these scores and these data, we decided together to revise our pre-testing and welcoming sessions for students.

And we actually introduced a more extensive welcoming session for our students to help them understand how assessments, how pre-testing works. That already has impact we are measuring now-- the attrition rates between testing and starting classes, participating in classes. And I think we can already see a correlation between a solid introduction to how assessments work and why we do them and students staying with us, starting classes, staying in classes.

So these two tools or conversations about data connect because transitions are only possible when the student starts classes. So by sort of aligning a larger conversation across consortium partners, with the conversation almost one-on-one with teachers and our proctors, that adds to improvements for both, for the students, but also curriculum review and the creation of a new welcome session for our students.

This whole conversation came up because we're using LaunchBoard a lot more nowadays than we ever did before. And of course, these discrepancies we saw didn't really sit well with us. So we dug a little bit deeper in our own internal data. So that's the broadest overview. A lot of it mirrors what Ilse and Rick described.

Blaire Toso: Thanks, Ute. Yeah, it's really interesting to begin to see the trends come out. And maybe we can talk a little bit about that afterwards because it's interesting how this continuous improvement process loans itself to particular pieces and pieces that you've pulled out that are really important. But before we go to that, it would be great if we can go on to Jenna. Welcome, Jenna. You're on mute.

Jenna Cestone: Hi, everyone. I'm Jenna Cestone from Silicon Valley. I'm going to go ahead and share my screen. Let's see. OK. Here we go. So we did a similar data dive and a cycle of inquiry when we saw similar things to what everybody is talking about, all the speakers who have talked so far, where we saw our high school enrollment dropping significantly.

And this is above and beyond what can be attributed to the pandemic. We just didn't see our students returning. And it wasn't necessarily attributable to a demographic, any kind of specific demographic group. And when we did our five whys and we kept asking the teachers and student what they were hearing so we could complete our TOPS report, we would hear similar things to what you all hear.

We've spoken about this at our consortium meeting ad nauseum about no transportation, lack of accessibility to technology, could not participate in the Zoom, did not understand the curriculum, could not get-- the biggest sticking point, and we've talked about this, is the math. The math was too rigorous or it was just not accessible.

And so we kept trying to think, what could we do? Where could we go? How could we figure out another plan? Well, we queried our consortium. And it was varied. Schools were doing different things. One solution was independent study. And that seems to work well for many of our adult schools. Our school in particular does not have independent study. So that wasn't even an option for us.

We also were given directive by our superintendent to be in-person. We had to be in-person. So that was very troublesome for us because many of our students kept saying, well, you have a program that's in-person five days a week for three hours a day with mandatory attendance. And I just can't do that. I'm working. I'm living on the gig economy. My schedule is fluctuating.

So from week to week, some folks would be on a night schedule, some would be in a day schedule. So we were constantly changing students around. It just wasn't working for us. So through that, we reached out and found, through CASAS, the problem-- how could we find a program that would fit our students and our students' needs?

OK, so we polled our students. And we polled our teachers what they were seeing. And that data suggested, yes, they don't have child care. But we have CalWORKS on site. Some students didn't qualify for that-- no transportation. We got them through our transition program passes. But even though we're on the VAT line, the transit takes two hours from their location to get to us. It's very laborious. So they miss class anyway.

We don't have independent study. We don't have a laptop rental program anymore. And we really didn't meet the DEI equity requirements. The students who were coming in, we were having a really hard time servicing them.

The consortium was very gracious to us. And they actually contracted with an outside person qualified to test. And I understand that Dr. Williams goes around to different adult eds. And so we're lucky to have him one day a week.

But we really need a lot more for our AWD that we're seeing coming through. And so in addition, our falling enrollment needed another thing. So what did the students want? We queried them. What do you want? They all want online, hybrid classes. They all want it. But they want to come to class, too. They want access to a real human being. They want access to teachers. They want flexible schedules.

So we came up-- this is the data. It's very clear that all of our ESL folks, they want five days in-person. ESL was drastically different than our high school diploma folks-- drastically different. And so all of these, you can see, hey, I want flexible schedule. I want one day a week with my teacher. I want two evenings, on, off. I want to be able to come in the morning and come at night depending on my work schedule.

So high school diploma was all over the place. The data bore it out. And we said, what can we do? So out of our students, we queried, would you take more classes if the schedules changed? 42% said yes. That's real data from our high school. And so I took that to my superintendent. And we chose the NEDP, the National External Diploma Program mostly because it's well vetted.

I'm New York resident-- former New York resident. I know this program well. I said, hey, I'll take this on. It's been around since 1975. It's not that hard. It's very successful, well vetted. Most of us are using pre and post-testing CASAS anyway. So I started this part of the program and got it through our board here.

It's very good for folks who fit all of this criteria that kept bubbling up organically. Too much anxiety, timed tests, from pandemics or otherwise. They have work or other obligations. They struggle with higher math. Some of them are non-native English speakers. Some of them just wanted to work independently. Sorry about that.

OK, so we had student testimonials. And teachers could see the relevance of the curriculum. So how do we identify students? What do we actually do in our-- or we pre-test them. They need to meet that threshold of 230 in math and 236 in reading as a gatekeeper. In orientation, we have this conversation with them. People are coming to us from all over the world, OK? So in our hour orientation, we have this conversation.