Jay Wright: OK, hello. I'll go ahead and blow up the PowerPoint. I reckon, while I do that I'll just say he. Everybody hear me OK, see the screen OK, all that good kind of stuff?

Audience: Yes.

Jay Wright: Are we A-OK on that sort of thing? So hey, here's your proof in the pudding. Hopefully, this title and date matches your expectations. So a lot of you-- you look like you're certainly, mostly the coalition of the willing, usual suspects. You decide which cliche you'd like to be.

But a lot of people I've seen before in these webinars, so as such, you know kind of [humming while flicking tongue] like my cliches, whatever. So anyway, and I also like my silly questions to start. So the silly question to start this one is, how many of you were in the workshop we did a week ago on March 31st? We kind of put this one together as a little bit of a two-part series.

I mean, of course, it's really just two parts of a bigger than that series that a lot of you have been doing for the last three years. I admit that. But we did set this up with CAEP TAP to have two sessions specific to CAEP and so on. So maybe I should have answered it the other way-- maybe I should have asked it the other way.

It's not a big deal either way. But how many of you were not at that one? Maybe just about all of you were. I haven't had any of you say no yet. Anybody here not, maybe? Not a big deal either way, but just curious because of these first couple of bullets-- OK, thank you for being brave, Angela. So we'll start by recapping a couple of key points from the last one.

OK, we're splitting up now. All right, so anyway, last week we did bring up some new CAEP reports. Hopefully tomorrow morning at the network meeting, I'll be able to speak to this a little bit. But in any case, we talked a little bit about, hey, here are some new reports we either have available now or will be available very soon because I hadn't done anything with CAEP TAP in a while as of last week.

And then, the bulk of it, of course, was that second bullet where we brought up again those CAEP goals, the metrics for CAEP goals. That is the two mandatory metrics for consortia, the two mandatory metrics for agencies, and then the 10 optional metrics for agencies. So we went over all that. We've done that many times before, I think.

But what we did is we went over all of those goals and outlined, OK, here are some suggestions. In TE reports, you could use other features. you could use to directly address that goal and also some ways that you might measure it. So we'll recap that at first. And then another form of recap, this we didn't really do a week ago.

But again, you're a coalition of the willing sort of group it looks like. So my guess is most of you have heard this in other workshops but will relate a little bit. What we have going on for CAEP to those NRS performance goals that we've talked lots about the last two or three years, if you're not NRS, it's not important that you know table 4 like the back of your hand or anything, but there is a basic concept of looking at what goals we have statewide, looking at like averages we have statewide, and using that as kind of a process to get started with being able to identify your own strengths as well as, of course, identifying your own needs.

And there's that next bullet right afterwards. It's anticipating what I'm going to say, I guess, but we'll talk about how we can do that using data. And we'll get really into brass tacks with improving persistence and performance.

What's not on the agenda that I also kind of want to introduce is, last time, again, a lot of you that were there, we talked about a lot of quantitative methods, some of my own goofy terms, just some general processes you might use when you're in your data, in TE, in other systems.

What are some just general processes you might want to use? As a data analyzer, so to speak, what are questions you want to be asking yourself every step of the way? A lot of real basic Sesame Street-like concepts of hey, which of these is not like the others? I thought about this over the last week, and I think a lot of that is really-- the underlying issue is a lot of the things you're doing are really the exact opposite of rocket science.

Sometimes data gets falsely perceived that way. But a lot of ways, if you not only dismiss all the rocket-science aspects of it, but take it that extra step and say, not only is this not rocket science, but it's the exact opposite, it's really Sesame Street-- which of these are not like the others, who do we need to talk to you to find out what's different, and all that-- that's the sort of questions, that's what we're trying to set it up as, is it's really just asking basic questions.

So we'll really hone in on persistence and performance this time. And then the other thing is we'll kind of start introducing, by agency and student-level solutions, some suggestions for how you might want to engage your teachers and other staff, how you might want to engage your students as we get more data.

We already know we have data in TE, but we know that there is probably a 99.9% chance TE is not going to solve all our problems for us. We also, of course, need to engage our students more and engage our staff more, and simultaneous to all this data digging, there's a lot of stuff in our interactions that we also need to consider. And then we'll kind of tie it up with some of these CAEP reports issues and some even more detailed suggestions at the end.

So sorry, that's way too much time on the agenda. I guess I'll just sew it up by saying, hopefully, that's more or less like what you were expecting to hear today. If not, I encourage you to bring it up in chat or verbally, hey, there's still enough time to adjust if there's something we're missing that really needs to not be missed

OK, so here this slide is definitely from last week. We kind of gave an update because it's been a while since we've done so, at least from a CAEP TAP point of view. So we do have some new reports out, and others coming soon. But we showed this consortium manager menu because that really has kind of been the bulk of our new efforts, at least as far as CAEP.

So old news, but just to summarize if you haven't heard that I-3 summary that is those reports that relate to EL Civics. And we have now for CAEP agencies related to immigration indicators or I-3. So we now have I-3 information at the consortium manager level in addition to just the agency level like we did.

We also have two new reports, one called student enrollment by ZIP, the other, by ZIP and demographics. That is for the consortium manager ways of looking at data by ZIP code. So student enrollment, I think is, the one that will be really helpful for everybody. And it's kind of matching what was requested that is, hey, as a manager, we know that we have some communities that are well represented, others that are less represented.

So maybe we want to be able to see where our students are coming from, partially to help marketing efforts and get more students in our doorway, partially more for the reason of this workshop to evaluate performance try to figure out perhaps where our positive outcomes are coming from and where they're not.

So for marketing and performance-related reasons, it was thought, hey, if we can really figure out where our students are coming from and what's represented, that will really help us so that student enrollment by zip gives you enrollment counts by zip code and by city, and then that second one here on, the list a little bit gorier detail.

So for some of you, it will be about the most wonderful thing you can possibly imagine. For some of you, it may be way TMI. But it basically takes all of those factors I just described and adds the third dimension of demographics, so you can also-- OK, I'm not sure. I'm not quite sure about that one.

If it was the one, the second one with demographics-- you know what? Send me something that I can check out. I will say the second one is super busy. It's got three dimensions, not two. It does the same thing enrollment by zip code, but further organizes it by male, female, race, ethnicity, et cetera. So maybe-- I mean, I wonder maybe it's a too many pages issue.

I'll just say I'll show you a goofy screenshot, if you want, where I can get it to work but it does give me gazillions of pages, and you end up needing to put it at like 10% to really see everything because it's so detailed. I'll just say that's a known issue, then, where-- and I'll just say, if you have any logic suggestions, we're all ears.

But it's literally so much information that we've had trouble really getting a handle on it and getting it to fit within one page. So to date, we figured out a way to get all that wonderful data there, but we haven't figured out a way to get the data in a neat and tidy arrangement, so that could be it.

But anyway, two new reports there, and then there's the butterfly hours, which is very similar to those other newer hours reports we have, but it uses those stages of a butterfly. Some of you love that, others not so much. But that's the one that's not there yet. It should be available in the next couple of weeks.

OK, and then this is just a reset. I'm not going to dig into this as much as last week. But we showed this and then we gave the longer list. That is the two mandatory consortium metrics, two mandatory agency metrics, and 10 optional level agency metrics. Bottom line is-- well, obviously, the ones that are mandatory are mandatory.

For the optional ones, CAEP has kind of gone way out of its way to say, it's optional, which ones you choose. It's also optional, which data system you use. So I think it was intended to be LaunchBoard, but lots of things have come up there so it's now optional. You can use LaunchBoard if you want.

If you'd rather use TE, go ahead, use TE. That was kind of a point of last week's workshop, was to give you more options using TE. If you'd rather use your K-12 system or your college system or whatever, that, too, is also fine.

OK, and then now, I'm kind of sidestepping here and making it a little too much like NRS, perhaps. But I can see most of you are WIOA ll agencies, but not all of you. But are you all kind of familiar with the Data Portal?

I'm not saying you know all the Table 4 stuff or you know everything about the Data Portal, but are you all at least familiar with the fact that it exists and what it does? As I'm meeting the request here in the chat, I'm going to let you input in the chat on whether you feel like you know what we're talking about here with the Data Portal, yes or no.

So most of you are saying yes as far as I can tell. OK, thank you for being honest, Karen. OK, a couple of you are no. I'm not going to get into it. I'll say, first off, if you don't know much about this and you really want to know the mechanics, I don't have time to do that today, but there is a workshop I have scheduled for 11:00 AM Monday. It's kind of a repeat of sessions I did a few times in March. I'll do it again Monday, the 11th at 11:00 AM.

And that will be one where we'll go through the mechanics of how to access the link I have on the slide. And we'll go into the Data Portal. We'll talk a little bit more about Table 4 and 4B so you know that part of it a little better and we'll also go through all these mechanical steps of describing these three dropboxes and kind of, quote, unquote, teach you how to get through and follow the steps and drum up your data using the Data Portal.

For now, I'll just say, here's a screenshot. You can access this information at the link on this slide. And this is what we've had for the federally funded agencies for a very long time to look at annual data. We've made a big stinking deal about this recently because we've been a little late, but better late than never.

We've put up the most recent 2020-21 data up about a month ago. We got up the Table 4 performance info. And as Chris notes, just a few days ago, we got the data from the Employment and Earnings survey, so you can look at that.

And I've already promised that when we do this workshop on Monday, I will spend a few minutes making sure we go through all the steps of the E&E survey data given that that's new, and it does seem like there's definitely people that feel pretty good about the Table 4 side, but less good about the survey data. So we'll definitely try to give it some good airplay on Monday.

But that said, this is the Portal we have for NRS data. If you are a federally funded agency, here's where you can get really good information. It's not required, but if you ask us, at the CASAS, what's the best way to get this information and use previous performance information as a way to establish goals, we, at CASAS, would definitely say the Data Portal is your best bet, more terra firma probably than watch more a little easier to work with and going into TE or into other data systems.

This is where you can get that info. The catch though, of course, is that it's part of our WIOA ll contracts. So that means it really necessarily needs to cover just the federal reporting agencies. So if you don't have that federal grant, your agency may not be in here. I admit that. But it will be usable.

If you're not WIOA, I will go on to say I am going to continue along with this now is that a bare minimum. When it comes to making things like screenshots for PowerPoint slides, the Data Portal tends to be a lot more user friendly than TE or any other data system because, as you can see, we can get the year-wide results.

We can get several years of information on the same screen. So it's super duper better for things like screenshots. So just know, I'm using this for several screenshots. From this point forward is a way to show you data in a more comprehensive way to where anybody can kind of look and say, yeah, I can see some strengths in this example. I can also see some weaknesses in this example.

Sorry, I'll pause here and just say, does that make any sense at all? I'm fearing that this might be two or three minutes you'll never get back I'm just listening to myself talk, and I'm confusing myself. Maybe everybody else is in a lot better shape than me, though, from this level of response. I'll take that as a, hey, don't worry about it. We're OK. Thank you for the feedback

OK, so moving right along here. So here is an example from the Data Portal. Again, skipping steps, but the point here is we have our NRS levels that we use for federal reporting, six levels for ABE, six levels for ESL, like we do for the Fed tables. And we can drum up our agency-level data and compare it to the state average, compare it to the state goals.

So we're kind of making it easy and cutting to the chase by giving you some fresh out of the box examples. So here, we're using a fictitious agency from a few years ago that has some real obvious strengths and real obvious weaknesses.

And cutting to the chase, where we've got red arrow, green arrow, the red arrow to the left is giving us an example where we're not doing well, ASE Low. Hopefully, everybody can see this clearly from the screenshot. Clearly, ASE Low is an area where we're way below the state average and we're way below the state goal, so that's an area where we may need improvement. I'll stop again and just say, does everybody see, in this fictitious example, where we really need to make a lot of improvement in ASE Low based on what I'm showing you?

Thanks, Isabelle, Connie, OK, everybody's playing ball. Thank you for doing so. OK, so on the other hand, over there to the right, we have this green arrow, beginning-level ESL. That's an area where we're above the average and above the goal. So hey, congratulations, that's an area where we can safely say we're doing well, pat ourselves on the back, whatever.

Does everybody else, though, see where I'm coming from, where I'm saying that's an area that looks good? So we've got one area of strength, one area of weakness.

And so I'll move along here. And so if you were in the workshop last week, we had a bunch of slides that looked like this, Jay just being goofy Jay, I guess, trying to mix it up a little bit. But I'll just say, hey, we now know the basics. I cut to the chase, obviously, in saying, hey, we identified an area of strength, we identified an area of weakness, so we really need to start focusing on those areas of improvement. So we need to target our data to the areas of need, so in this case, we're looking at ASE Low to start.

Once we kind of start focusing on those areas where we really need to improve, we need to start looking big picture for agency-level strategies, and maybe more detailed picture to find specific student-level strategies to the areas we identify as being areas in which we need to improve.

So I'm going to double back before moving forward. We looked at that small detailed screenshot that showed a ASE Low as an area for improvement and beginning level ESL as an area strength. I'm just going to take a step back and show, hey, here's that same ASE Low. It's on the right-hand side this time.

But we'll reiterate that ASE Low is a weak area. And oh, just try to find another area. So we're just going to quickly cut to the chase again and identify ABE Intermediate High as another area where we're not doing so well. I'll just point out-- here's a little broader screenshot, but it gives the same news for ABE intermediate high that we already shared for ASE Low. That is another area where we're well below the state average, another area where we're well below the state goal and might represent an area in which we need to improve.

OK, so we're still hanging in there right. Did everybody agree or see how those were two areas that we clearly were way below average. Sorry, I keep asking all these innocuous questions. But I am shuffleboarding on you a lot here. So good to make sure-- all right, thank you. Everybody is being cooperative.

So now, we've got another blown-up slide. To be clear, you can see by the header. It says Evaluating Persistence. So this screenshot looks a lot like a couple of these others we've shown the last couple of minutes. But it's different data. We're still on the CASAS Data Portal. But we're using the persistent section.

So we're comparing persistence data, not performance, that is, we're just looking to see how well we're doing in terms of getting pre/post spans. So if you haven't heard this a million times-- some of you have, some of you haven't-- what we're looking at is performance. That is, we're evaluating our high-performing and not so high-performing levels.

When we use the term, performance, we're talking about what percentage of students do the test-- not only do the testing, but make enough of a gain between the pre-test and post-test to make a gain on the federal charts. That's what all these percentages are concerned with.

When we move out of performance and into persistence, now we're not talking about their scores, we're just talking simply about the percentage of students that, at least, stick around long enough or quote, unquote, persist long enough, to, at least, complete a pre-test and the post-test.

So what I'll just say-- I've said for a really long time doing Tech Assist and doing all these trainings, whatever, is once we start looking at these federal performance levels, we can quickly identify areas of strength and areas of need, a really, really good first step that doesn't solve our problem, but at least allows us to start prioritizing and start making decisions about ways in which we can start effecting improvement is comparing those low- performing areas with our persistence rate, and kind of take this as a first step.

So that's what we're doing here. So from the previous slide, we already show ABE Intermediate High is an area of weakness. ASE Low is another area of weakness, So. We're looking at those same two areas that we already know, for sure, are areas of weakness and comparing it to that persistence rate.

So you can see another red arrow and another green arrow. Over there to the left, ABE Intermediate High red arrow, you might say, is confirming our suspicion. This area of low performance is also an area of low persistence. That is an ABE Intermediate High. We're not doing a good job getting pairs.

You can see that our persistence rate is below the statewide average, and below that average by a pretty large margin. On the other hand, we have a green arrow here for ASE low that is kind of surprisingly the persistence rate is OK or actually pretty good. You can see for ASE Low that persistence rate quite a bit above the state average.

So in this case, that persistence rate probably not the explanation we're looking for why our performance is low. We already know our performance is low, but we'll need to look somewhere else other than the persistence rate to find the cause because we know performance is low. But by that green arrow, we know persistence is not really the explanation in the case of ASE Low.

So I'll stop yet again. Does everybody see why pre-, post-persistence is explaining away things for ABE Intermediate High, but it's not explaining things for ASE Low? We can see that by looking at the persistence rate and kind of measuring that off this exam.

OK, everybody looks good on this one. So I'm going to sidestep here. What I just did is reiterate that NRS federal reporting explanation we've gone over many, many times. So this is a CAEP training, so we know we need to move this over to CAEP. So I'll just say a lot of these things that we talk about related to Table 4 and NRS Persister, we can use that CAEP summary if we're a CAEP-only and use some fields on the CAEP summary to get this information.

It may not be exactly the same, but there's definitely ways we can use the CAEP summary and get basically the same information. Certainly, we can get enough information from the CAEP summary, more than enough information, to at least make that basic determination as to whether persistence is a problem, yes or no, weather performance is a problem, yes or no.

And so in particular, we're talking about using the pre/post section over to the left back to NRS reporting. A little bit to your comment, Janae, we talked about performance and persistence. But I did kind of conveniently skip over the part that says when we're looking at persistence from an NRS federal reporting point of view, we're always looking at pre/post testing.

When we use the word "persistence," we're using that to say, is the student sticking around? There's definitely metrics you can use beyond pre- and post-testing to successfully measure persistence. That said, for NRS reporting, they're always defining persistence by pre/post pairs, not by attendance hours or other measures we've talked about a lot.

So again, we want to make sure it's apples to apples with federal reporting. We use the pre/post section of our Literacy Gains section of the CAEP summary. That is these left-hand three columns we're showing on the screen. So again, that column B, Enrollees-- again, those are the number of enrollees on our CAEP summary that also qualify for the NRS tables.

That is, they have demographics. They have 12 hours. And they also have that qualifying pre-test. And then, we've got the enrollees with pre/post column, C. That's the column that speaking the persistence, again, of those enrollees that make Table 4. These are those enrollees that have a pre-test and also have a post-test.

And then over there in column D, that's the smaller subset. Those also have a pre and post. But those also have not only that, but scores on their pre- and post-test that suggest they made that gain based on an NRS reporting. Those are the ones that successfully made that level.

So to measure persistence, we can simply compare column C on the CAEP summary to column B, that is, of those enrollees qualified for federal reporting with the hours, demographics, and qualifying pre-test. How many of those completed a pre/post, we can calculate that easily simply by doing column C divided by column B. That gets us our persistence rate.

We can do it by program. We can do it for our whole agency. We can do it lots of different ways. I'll just say this is kind of the pre-COVID feedback, but I'll leave it on here, depending on where you might be setting goals. Before COVID, I used to always say there's not really a number I can give you for performance because that performance rate really is all over the map.

And it's not even all over the map for ABE and ESL. All of those different levels of ESL have different rates. All of those different levels of ABE, ASE also have different rates. So there is no number I can give you. I just don't give you one because it's got 100% chance of being misleading.

Persistence though is way more straightforward. I've said, yeah, good ballpark figure. 70% persistence, that's our California rate for almost 10 years now. So if you have 70% of your students or better completing that pre/post pair, in general, that's what I, anyway, would say means you're on the right track. So if that helps, there you go.

And then for CAEP performance, same math here. But we're looking at column D rather than C that is the EFL Gains Achieved, that is we're looking at not only those that complete a pre/post pair, but those that make a good enough score between the pre-test and post-test to move up to the next level. So that column D, as in delta, divided by B, as in bravo, would give us our performance.

So I'll just stop right here. If you are NRS-funded, you could use the Data Portal probably easier. But I bring all this up to say, hey, if you're not an NRS-funded, or maybe you are, but you really want to incorporate non-NRS programs like CTE or Workforce Prep or whatever, then you can use this CAEP report and get to the bottom of it using these equations.

Again, it's the CAEP Summary, so that's the one I showed you here at first. I showed the screenshot of the whole report. This is the one you'd use. And then the next subsequent slides were blown-up screenshots of that same report. Okey-doke.

All right, so we've identified the issue with persistence. We've identified the issue with performance. And now, we've also kind of correlated it to some CAEP-specific suggestions. So here's what we go crazy with all these questions. So this is what we looked at a week ago, as I think, we had one related to getting enough testing, where we kind of talked generically about, what if we're low on hours, what if we're low on demographics, et cetera.

What are some things we should start looking at? So we looked at that for testing, but I think the one we looked at last week was just general what if we're not doing a good job of getting students tested. This one is a little more specific to pre/post performance. So we were talking about some of those questions, look at it longitudinally.

That is, when we look at our data and we see a problem, we should be-- this is doubling back to a little bit of review from last week-- that is, if we know we've got a problem, one question we always want to ask is, look at that longitudinal data issue. That is, we know we have a problem right now in 2022.

You always want to ask yourself, is this a new problem where we've always done well, and all of a sudden in 2022, we went from good to bad? Or is this an age-old problem, where we've always kind of run lousy, and it's been a problem for 5 or 10 or 20 straight years? I'm not saying one is good and one is bad.

I'm just saying knowing the answer to that question is very important. If you know you have a problem, you want to know whether it's an ongoing problem or a newly developing problem. That's what this beige question is. Hey, is this something where we've always done well? Or is it something that's always been a problem?

This next one-- and to also reiterate, we talked about you always want to look for those kind of outliers, back to those Sesame Street questions, which of these are not like the others? So if we know we have a problem with performance, hey, we know we've got a problem specifically with high level ABE and ASE.

We want to look and focus our data just on those higher levels of ABE, like this example, just our classes related to ASE in this example. And then, look for outliers in that small focus group. So we're looking at our ASE data. We know ASE overall is way closer to lousy than good. So we expect most of our ASE data to be lousy because we've already figured out that's our problem area.

ASE is the area where we're lousy. If we narrow it and look at ASE, what we're looking for is, hey, where are our big fires coming from? We expect ASE to be below average. But they'll probably be a few areas that are way below average. So we want to find those areas that are not just bad but are really overwhelmingly stinking bad and put a lot of focus on that.

At the same time, we'll have those diamonds in the rough. That is, even though we know ASE is lousy, here are some specific classes, or specific teachers, or specific groups of students that are doing really well, even though they're an area where most others are doing lousy. Look for those diamonds in the rough that are doing well when others are doing lousy.

Look for those really bad areas that are doing way worse than others. So that's another question you want to look at. You also want to look at some of those specific levels, hey, are there some specific areas where we know there's some specific levels in ABE or ESL or worse? And then, the green are kind of those what-ifs.

So the what-if here is, hey, what if every student just made one more point of gain on the post-test? Would that make a big difference in our performance data? Or hey, it's way worse than that. We need to make big gains, not just little gains. But some of those things in the margins sometimes can surprise you and make a much bigger difference than you would expect. So these are the sort of things we're talking about to be thinking about when you're looking at good data as well as when you're looking at bad data.

I'll just stop here. Is this making any sense at all with all these captions coming out? Yes or no? Sorry, I got to be a little bit of a fussbudget on this one to check. OK, and I see some new yeses, so that's encouraging.

OK, so these are good questions for performance. So here's another set of questions, very similar process, but different specific questions. These are questions you'd ask if you know your persistent state is weak. So just like we did with performance, well, hey, what's our historic track record? Hey, we're doing really lousy capturing pairs now. So an example might be the COVID question. In this case, maybe we were great in 2016, '17, '18.

And now we're way closer to lousy. If so, we'd obviously want to say, hey, maybe that's a COVID issue. We were testing everybody great in 2018. COVID hit. And now we're not doing so well. Maybe it's simply a COVID casualty. Ask yourself that question. Maybe it's something else. Either way, start figuring out what it is that's made you do worse now where you are doing great before and move forward.

There's no right or wrong answer, just knowing what correctly explains your situation, just like we talked about with performance. Maybe there's that random class or random teacher that's doing a spectacularly good job with this amidst everybody else doing spectacularly lousy. What is that one teacher or one class or one group of students doing that clearly everybody else isn't? Figure that sort of stuff out.

And then the what-ifs, what about those low-performing areas that we know we need to fix, are the areas that we know are also the ones that have low pre/post test persistence? If it's a match-- what I'm really getting at is this question in bold at the bottom. That is, can we improve our performance if we really just put our focus in improving persistence?

I'll stop right there. Does everybody understand that question? Because I feel like it paraphrases a lot of what we talk about when we talk about this goal set. Sure, so what does it mean? Anybody want to dig in and explain? Anybody want to take care of my 'splainin' for me?

What do we mean by that? Anybody brave want to reiterate that verbally or by chat? Yeah, get them to the test. Right. So I'll-- OK, nobody wants to be brave .

Audience: Yeah, I can do it.

Jay Wright: Do it. Please do.

Audience: I want to try. So if you make sure that the students are coming constantly and they're focusing on the classes, you can assure, in my point of view, that the performance is going to be better because they are getting into classes. So if the persistence is good, the students are going to improve. And it's going to show in the test.

Jay Wright: Right. OK, that's good. And I see what you're saying, Connie. Yeah, that's a good part of it. And I'll just say beyond just that, it's kind of a little bit of, hey, I'm not looking-- but a little bit of-- you see the motion I'm making where, hey, we're just sort of dismissing the problem because it's mission accomplished.

Can we be like that with our performance if we just say, you know what, we kind of feel like our student learning is fine, but we're not necessarily getting everybody tested? So if we sort of ignore the learning part but just focus on our data cleaning and our pre- and post-testing, will that alone kind of bump up our performance, too, because we know our persistence rate is low?

So can we really just safely ignore bothering our teachers for now and just focus on our data and our testing rate? Is it safe to say our persistence is such a big part of this problem that if we just focus on the grunt work like that, we'll probably, by osmosis, improve our performance because our data and our testing rate is so bad that that really is where the bulk of our low performance is coming from?

So back to the, hey, if we just focus on data and getting more pre/post pairs, we'll probably automatically improve performance because we've already figured out that that's such a big part of explaining why we're performing low in the first place. What I'll say is sometimes we have it easy, and we can kind of safely brush it off and say, if we just do that, we probably will spike up.

Other times, we'll be more like that example in ASE Low, where we know we're exceedingly and overwhelmingly lousy in terms of performance. But what do you know? We're actually doing a good job getting everybody tested. So we know that's not really going to help in that area. So sometimes it really will do this, and other times not. Go ahead, Dana.

Dana Galloway: No, you're probably going to say this. But once you rule that out, then you need to go on and figure out what's going on in your classrooms or what's going on with your curriculum. And I'm sure that's what you are going to say next. But you know that's--

Jay Wright: What do you think? What does everybody else think? You think I'm going to say that? Yes or no?

Dana Galloway: Yeah.

Veronica Parker: You think that's on the itinerary soon?

Dana Galloway: Yeah, so that's the hard part.

Jay Wright: OK, sorry, I had to get other people involved just to be cute, but yeah, sure enough. That's it. Yes, we will be talking about that. And this is kind of a segue to say, yeah, sometimes we can focus on fundamentals. And a lot of times just that focus on fundamentals alone, as long as you grip or use elbow grease, we'll probably be successful.

Other times, that might be more nuanced, then we need to look a little deeper, et cetera. So now, we'll double back. Here's a slide that we just showed about 10, 15 minutes ago-- back to the, hey, we know where we're weak. So let's compare it to persistence-- one comparison where, yeah, persistence is definitely the problem, another one where it really is not a problem, we can filter those other issues, absolutely.

And so a couple of people have been asking me about this. I'm not sure if it's anybody here, but that little goofy grid. So this is where that grid comes in, where this is just, for me, where I like-- you know, hey, we know performance is going to be good or bad.

We know that persistence is also either going to be good or bad. So that allows us to oversimplify things a little bit and put ourselves into one of these four quadrants, where we're good in both, bad in both, or perhaps good in one, and not so good in the other. I'll just say what I like to focus on here more is the left side of the equation.

If we're in the upper right, and we're good in both, well, hey, congratulations. Maybe we don't need any of this. If we're in the lower right, where performance is good, but persistence isn't, I'll just say, hey, maybe it's mathematically possible. But it's very, very rare. So rare that I don't really talk about it much because it's just unusual to be a high-performing agency and have low persistence at the same time.

So we'll focus on the other two. The first one is obviously, hey, we're low on both. That one's self-explanatory. Of course, we need help, then. So that's that example where we know we're low-performing. We cross-checked with persistence. And yeah, our pre/post rate is also low. So that's the one where we'd stay focus on the fundamentals.

Where if we're low in both that suggests for now, we really just need to work on those basic issues of, hey, if we've got data that needs cleaning up, demographics hours of instruction, whatever, clean up those demographics and clean up those hours of instruction. And then at the same time, of course, really put the focus on getting all of the students overall or in that specific level pre- and post-tested.

So these are kind of, in my opinion-- you're hearing from CASAS, so, of course, I'm going to say this. But what I would consider the two most basic fundamentals of data cleanup, pre- and post-test rate. Focus on that. I mean, you can certainly look at other things, but this theory suggests you really need to focus on fundamentals, get those fundamentals up to snuff.

And then, cliches I've used in other workshops, once you get the fundamentals up to snuff, that's where you can get a chance to rinse and repeat, house of mirrors, all that stuff we've talked about in other workshops, where, hey, we know our data is clean. We fixed those missing demographics. We've added our hours.

So most people are over 12 hours of instruction or whatever. We've got 80-plus pre- and post-tested. So let's run the reports again and see what performance looks like now that we've cleaned up our data and got more students pre- and post-tested. The idea is, hey, we do another evaluation, hey, all of a sudden, what do you know, our performance is overwhelmingly and exponentially better. Well, hey, a little elbow grease fixed the problem. Hey, we did this and it's a little better, but it's still way closer to lousy than good, then we need to look at those other areas.

OK, then the other one is, hey, our performance is low, but our pre/post performance is actually pretty good. So hey, that suggests we're already doing well with the basic data collection requirements. You literally can't do very well with persistence and have fundamentals be bad. If you're fundamentals are weak, by definition, that means your persistence is going to be low. So by determining your persistence is good.

That is suggesting you're doing well with these fundamentals. But our persistence, like you can see from the graphic, is still low. So we need to address other areas. Usually, that means we need to focus on instruction and the class. So now, that brings up another series of questions, where we looked at our levels. We've identified strengths. We've identified weaknesses.

We've kind of rinsed and repeat and taken it the second step, where we've evaluated our weak levels with persistence. We've identified which of those weak levels have weak persistence as well. We've identified which of those weak levels have good persistence. We've kind of split that up and kind of divided it accordingly.

We've done another round. Now, we've kind of honed in on those with good persistence and not so good persistence. So now we need to, A, use those data, bring in those data questions we talked about, and B, start looking at our staff, specifically teachers, of course, and the students that are involved in those weaker areas we've identified, get their feedback, figure out what's going on from their point of view is kind of what we necessarily need to do if we really want to roll up our sleeves and get this fixed.

So we're going to step back. This is the data side. This looks suspiciously familiar from a week ago. In fact, I think this was a slide from a week ago, maybe minus a few word changes or whatever. But this is what we talked about with the word, "troubleshoot," where we're in our data troubleshooting things with TE reports or other data submissions.

These are our kind of, quote, unquote, "quantitative data" related suggestions of ways to troubleshoot our problem areas. To reiterate again, longitudinal data, good idea, look at the history. It's not always going to say the same thing for you. But just knowing what your history is, at the bare minimum, puts you in a much better position to effectively solve these problems.

That is, there's no exact formula I can give you for, quote, unquote, "good history or bad history." But just knowing, hey, this is an area where we've been superstars for 20 years. Now, all of a sudden in 2019, 2021, we're a basket case. What the heck happened? Was it COVID? Was it some other development?

That's a whole different story, in my opinion, than, hey, yeah, we've been terrible about this every year since 1999. We can never figure out our elbow from our earhole in this area for whatever stinking reason. I'll just say scenario A and scenario B, neither one is better or worse than the other.

But you might suggest a completely separate set of questions for the, hey, we've been bad at this since forever, versus, hey, we've been great at this. Why are we suddenly bad instead of good? Does everybody at least follow what I'm talking about with these type of questions?

OK, thank you, Connie. I knew I could count on you to [inaudible]. OK, so there you go. Then these are my goofy terms. I didn't invent them. But I make a big deal about saying these are my terms just because I really don't want you to feel like artificially or falsely giving you definitions that everybody knows or artificially projecting my terms as the terms you need to use.

I don't want to misrepresent myself, so that's why I say that. But these are terms that I like, so that's why I'm giving them to you. So by hotspots, I just mean those areas-- hey, we know that we're really weak in ABE. What are some hotspots within our ABE data, where we know ABE is a problem?