NICOLE CLARK: Good morning or afternoon, everyone. My name is Nicole Jordan Clark. I work, obviously, at CASAS. And I am the Senior Data Analyst here. My primary role is to work with both CAEP and WIOA Title II funded agencies, particularly, on their data submission to either the state, and then as it comes into TE. 

So I review what we'll go over, the Data Integrity Report. And I also do a lot of training on employment and earnings surveys. So it's really nice to see everyone here. I hope no one was expecting Jay too much. Sorry if I'm the surprise, but hello. 

So let's get into our objectives for today. So what we'll be going over today is an overview of the CAEP summary report, and then we'll go into an overview of the CAEP Data Integrity Report. We'll also define how TE determines which students qualify for the Data Integrity Report. And then we'll identify specific items and how to address those data issues. And then we'll also go over Drill Down options, including some listers and reports that pertain to the DIR. 

And towards the end of this session, we'll hopefully get to go into our Rolling Hills database. And I can show you how to correct and Drill Down into a certain student's demographic records lister and how to correct some of the items that you may encounter and that you probably will want to change. 

So, again, welcome everyone. And I like to start these Data Integrity Report trainings with a common list of acronyms. Once we get the PDF given to you all, there is a link to common acronyms for us. But these are the ones that I'll majority go through in this presentation. 

So we'll talk about the DIR. So when we talk about the Data Integrity Report, a lot of us like to shorten it to say just DIR. There might be some references to PoPs, measurable skill gains, which we also talk-- which we'll refer to as MSGs. 

And then we'll talk about TE a lot. So if this is your first time using or generating the CAEP Data Integrity report, TOPSpro Enterprise, or for short, we say TE. So just common acronyms, but the big list will be in a link provided on the slide. 

So we can't talk about the Data Integrity Report without talking about the CAEP summary report. And to generate that in TOPSpro Enterprise, you would go to Reports, State Reports, California, and then CAEP Tables. And once you get into CAEP tables, there are about-- how many-- 17 different reports that are able to generate when you generate the CAEP tables. But that first one at the top of the report is going to be that CAEP summary report. And that's just the one we're going to focus on today. 

So in this CAEP summary report, there are three major sections in the report. The first one on the far left covers literacy gains pre/post, the ones in the middle cover outcomes, and then the ones on the right will review services. Now the way that I like to think about this is that the section on the left has the most rules, requirements. And then we give a little leeway in the CAEP outcomes. And then our services, that's the biggest bucket we have as far as CAEP enrollment, CAEP learners go. 

So let's start with the CAEP literacy-- or the literacy gains section. That's columns B, C, and D of the summary report. And these are learners that qualify for NRS Table 4. How many of you, and if you can share in the chat, how many of you are CAEP only agencies and do not know what I'm talking about when I refer to our NRS, or National Reporting System, Table 4? Can I get either a thumbs up or say me, or is everybody fairly familiar with our table 4? 

OK, somewhat. Got a thumbs up. So the NRS Table 4 are tables that were created by the OCTAE And our WIOA Title II agencies are reporting these outcomes and results to the NRS Table 4. And that gets puts up to the feds. 

So there are requirements that will get a learner into NRS Table 4. And I have those, too, so that they took a pretest, and that they have those 12 hours of instruction. So if you have any CAEP students that took a pretest, and then in column B or C took a post-test, and then have those 12 hours, they'll show up in this section. 

So then in the middle section, we have our CAEP outcomes. And these outcomes are other outcomes that are reported by AB 104. And they still use those WIOA Title II reporting requirements but not the pre/post-test requirement. So we have our outcomes listed here in columns F through L. And if you've attended the CAEP accountability training, you might be familiar with this slide here. 

However, I did Nicole it up a little bit from what maybe you've seen with Jay. So here we have our list of outcomes and then some of the results that you would enter in order for that student in order for them to achieve that outcome. So things like retained a job or entered the military would fall under our entered employment outcome. And these are outcomes or results that you would mark in a student's update record after a significant amount of time at your agency. 

So once you've updated attendance reports, you have some students that pass high school diploma in that update record, then you would be marking these. And that's when they would show up in this CAEP outcome section. 

So we have a question in the chat. And it's from Millie Vang. The question is, what is considered significant amount of time? Or what would be best practice to update this? 

So it really depends on your agency and how your agency is set up when you are reporting these update records. So if you have a program that works on quarters, or if you have a month-long program or semesters, then you would create that update record. I would say the only one that you would want to update more frequently is your attendance hours. But maybe at the end of your two month, three month program, then you're entering in these outcomes. So that really depends on your agency. 

And then we have another question in the chat from Joanne Miller. What NRS tables CAEP outcomes show up? What NRS table-- yeah, so what NRS tables CAEP outcomes show up? 

So when we look here, we have the High School Diploma, passed GED, passed HiSET. And that will show in the CAEP outcomes of the CAEP summary report. And then under Literacy Gains, other literacy gain outcomes would be our occupational skills gain workforce prep. But this pre/post level completion would show up in that first couple of columns instead of this middle section here 

and then another question in the chat. Outcomes are based off of self-reporting from updates. Correct. So if a student says they got a job, we don't verify that information. Yes, so it's what you would put into your update record, if you are updating that in TE. 

I see Jay writes here, so thank you, Jay. Jay answered that question. So tables and CAEP are a little bit of apples and oranges comparison, as CAEP includes workforce prep, CTE, NRS tables only So thank you, Jay. 

So now we'll move on to the final section of the CAEP summary report. And this covers services. So the services received are services that don't impose any WIOA Title II reporting requirements. So they don't require that pre/post test. Maybe they don't have any hours in your agency. 

And on our CASAS website, we have definitions for short-term services. This is our Attachment S. So there's a link to that document too, if you want to see detailed guidelines about those services, definitions, and so on and so forth. 

Something I wanted to point out that I didn't point out earlier about the CAEP summary is we have this bottom section here that has a total number of students. Then it has students within two or more programs, and then total unduplicated students. When we look at this total line here, and in this example, we have 870 total students, this number would be a duplicated number. 

If you have a student that is in ESL and CTE, they would show up in the ESL row, and then they would also show up in the CTE row as well. So then if you add up all these numbers, you'll get the total. Now we've unduplicated those and identified the students that have more than two programs. So we have 96, in this example, that have two or more programs. And then our unduplicated student count is 740. 

Like all of our reports in TOPSpro Enterprise, the CAEP summary report does have Drill Down Lister features. Here is a screenshot of what the Drill Down features we have. So there's listers, which have that grid, it looks like little Excel grid, next to it. And then we have a magnifying glass, and then it says Drill Down. Those drill down to reports. 

So in order to do this, and we can go over this at the very end, you would hover over a cell, it would highlight red, and then you would right-click in order to get this dropdown. Student program year population, if you wanted to get the exact number of students in a cell, you can just double-click that cell or right-click into that student program year population, and you would get that many students. 

So in this example, other literacy gains, we have four students. If I were to click on that Lister, I would get four students. We also have a student program population. So this will start giving you a duplicated list of students and all their instructional programs. In my previous example, we had our ESL students and our CTE students. So you would see a record for that student in ESL and then a record for that student in CTE. 

One of our other listers are student record population. And this will generate all the records for that student. If throughout the program year, you've done an update record monthly, you will see 12 update records for that student, depending on when you look at it, under this Lister. And it'll give you all those entry update records. And in the entry update records, you have your results of that student. So you'll see if they got a job, and if they'll then follow in line with that enter employment column in the CAEP summary. 

So that's the listers. And now under our Drill Down reports-- and I have a screenshots at the very end of what's a lot of these reports look like. So the first one, we have our Drill Down to NRS Monitor. And that's a view to the federal tables monitor report, and includes and has those requirements for NRS Table 4. So it'll show what would be their pretest, what would be their post-test, if they had 12 hours of instruction, if they dropped for whatever reason, if they didn't have a birth date and ethnicity, it'll show on that table. 

The next one we have is Drill Down to Outcomes and Drill Down to Services monitor. They work very similar. So one focuses on the outcomes, one focuses on the services. 

So if I continue with my example, an outcome is to enter employment. In that drill down report, it would say, oh, they retain a job. So John Clark retained a job. And then that'll be under entered employment column. Very similar to the Drill Down to Services. So you'll see the services and what exact service that student had. 

Then we have a Drill Down to Outcomes report, which is different from the monitors. And this breaks down each result by program area. So you'll have retained a job, I think, and then up would be that program area, if they're in CTE, if they're in workforce prep. And again, this is at the very end of the presentation. 

And then lastly, we have our Drill Down to Assessments Audit. And this will report the pre or post-test pairs for those students. And you'll see what's the top bracket of that pretest, what the bottom bracket is, what modality is, what their score was, so a very detailed report about the assessments. 

Let me see. And the last one that I didn't cover, if you're looking at the dropdown, is our Drill Down to CAEP DIR because that is in our next section. So before we move on to the next section, does anybody have any questions? I'll look through the chat to see if there is any unanswered questions. 

OK, so I'll let that one-- I do see a question from Millie, but I'll let that one simmer in the chat. So I have another question from Laura Ochoa. So the question is, the NRS monitor is like the payment points monitor? It's very similar to the payment points monitor. But the payment points monitor would only take into account the California reporting requirements and not exactly the NRS reporting requirements. 

Finally, we'll move on to our overview of the data intake, the CAEP Data Integrity Report. There are multiple data integrity reports in TOPSpro Enterprise, but the CAEP one obviously focuses on CAEP outcomes and goals. We also have our Data Integrity Report that follows our reporting requirements and that our WIOA Title II agencies will submit every quarter. 

So our data integrity reports all work very similar to each other. But I'll be able to point out here what the specific ones are to this CAEP Data Integrity Report. So in order to generate that in TOPSpro Enterprise, you would go to Reports, State Reports, California, CAEP Data Integrity. 

And once you generate that report-- and you want to generate it as is. You don't really need to futz around with all the filters. Just go ahead and select your program year, and click Generate. And once your report generates, you'll get the summary information table at the very top. 

Selected students will be all the students that you have in your database in that program year. So if you were to go to Rolling Hills-- and I'll be able to show you this later-- the report selected 808 students. And if I were to go into my Demographics Lister, I would be able to see 808 students. 

Now from here, we would take out any students that are not enrolled in any of the CAEP programs. And students would fall into this bracket maybe if they had got exported in from your third-party vendor as other program. So you might want to look at those students, drill down onto those students, and see if they accidentally got migrated over as other program and you would need to correct that maybe to CTE. Or maybe you included a group of students that maybe shouldn't have been pushed over into TE. Those are very common reasons why you would have students that are not enrolled in a CAEP program. 

And we have other outcomes. So let's see, Marked HSD, Marked HSE Outcomes but did not have a CAEP program, that could be a flag for you to look deeper in that because they got a high school diploma. They achieved their high school secondary, so should they be in a CAEP program? And maybe it's easier to identify, yes, they should have been in your ASC program. You now need to go in and make sure that their instructional program is correct. 

And we also have Students Enrolled. So once we subtract the selected students from anything that shows up here, we'll get the students that are enrolled in those CAEP programs, we subtract out those students that are concurrently enrolled in high school and K through 12, and then we have our total of students that are eligible for the Data Integrity Report. In this example, we took the 808 subtracted the 66, and we're working with 742 students that are eligible for the report. 

So that's the top table of the report. And in the bottom section, we have an item description. So here, we have an item number and then a description of what that item number is, the item count, and then the item percentage. And there are over 33 because we have 2, 2A, 2B, 9CX. So we have multiple items and their descriptions on this chart. And they contribute to our CAEP outcomes. And we'll identify those in a couple slides. 

So again, we have our item counts here, and then we have our item percentages. So in order to calculate these item percentages, what you would do is you would take that item, and in my example, we're going to go with 02, less than 12 hours of instruction. So we'll take that 12 hours of instruction item, and then we'll divide that by the students eligible for the Data Integrity. And in this case, that's 273 divided by 742. And that's how we get our 36.79%. 

And like the CAEP Summary, we are also able to drill down on any number in the Data Integrity Report. There are a couple listers that are unique to this one. So that would be the student demographic history population. And that will show a longitudinal enrollment that updates that demographic population. So if there's any issues identifying where the birth date went wrong, you can look at that. 

And then we have the student assessment population. And that will list out all the assessments, so whether or not they took 185R, 186R, and then how to calculate it for NRS there too. 

So now we'll finally get into our items here. So the first items, 1 through 110, these are key requirements for state, federal reporting. And they could be drop reasons into why they're not qualifying into certain areas of the CAEP summary report. So I guess I should also add here that reviewing your Data Integrity Report is a way to help clean, review your data in an aggregate way in order for you to see what issues can arise and why certain items are not showing up into maybe your CAEP summary report or your NRS reports as well. 

So here are some of our items, 1 through 1b. And we'll start going through them. And here's where we'll start talking about what are common issues and how to resolve them. 

So first one, looking at 2, since 1 is self-explanatory, right? Missing birth date or their outside ages 18 through 110. If they're missing a birth date, then that's when you would drill down onto those students. You could select that student if you have their birth date and quickly update that in your database. 

Less than 12 hours of instruction, zero, empty hours of instruction, total hours, so these are all very much related to each other. Those that show up in the 0 or empty hours will also show up in this two as well. So if you have students with less than 12 hours of instruction, those could be your students that only receive services, and maybe they didn't take the pretest, didn't take a post-test. You don't have outcomes. 

With the Data Integrity Report, when you're looking at it at any given time, you should always take into account what time of year it is, where you're at in your program. So if I just started yesterday and my school came up yesterday, yesterday was the first day of class, that's when we got everybody to come in and pretest, if I have all 100 students with less than 12 hours of instruction and I run my DIR, I would likely see all my students falling under this 02. And that's OK. 

But let's say now it's the end of the program year and I still have 100 students, and I only have 100 students with less than 12 hours of instruction, then that's where my red flags start to pop up because I know, from my program, that everybody has more than 12 hours of instruction. So why are people still showing up in this item? That's when you would drill down, look at maybe update records to see the attendance. Maybe you need to go back and correct an export or an import that came in from a third party database with their attendance hours. And that's when you would identify that issue and you can correct it on the back end through their update records and so on and so forth. 

Let's see. So a lot of these right here, from 1 all the way to 6, are demographic data. If you don't have a gender, but you're able to gather that from your student, then those are easy corrections that can be made in their student record. When we look at 9 through 9a, 9b, and we have this 9ax, 9b, we'll just focus on the 9's for now. 

These would relate to those CAEP out-- or let's say, those literacy gain section or maybe the CAEP outcome section. So no valid pretest. Those are your students-- were you supposed to test all the students that came into your agency, so they all should have a valid pretest. That would be something to note. Does that number equal what you see at your agency? 

And valid pretest with no post-test. Should all your students have no post-test? Are you an agency that's looking to post-test all your students, and now it's May, and it looks like no one has a post-test. Well, now, you need to start thinking about how you're going to get those students post-tested before the end of the program year. 

And then we have valid pretest with no post-test, excluding those that earn high school diploma. So a lot of these could be more FYI. And then we have our 10a, so these are our students with a pre/post test pair. And if we remember back to what we were looking at the CAEP summary report, in order to get into that literacy gains section, those first couple columns, those are the students with a pre/post test pair. So it should fairly match what you see in your Data Integrity Report to what you see in your CAEP summary. 

But maybe you have students that have a pre/post test pair but they don't have 12 hours of instruction, then they wouldn't fit under that literacy gains section in the CAEP summary report, because 12 hours would be required for NRS reporting. And that's a requirement for those columns in the CAEP summary. It might not be a requirement for the CAEP outcome section of the CAEP summary, but it would be a requirement for the literacy gains section in that. 

So a question in the chat from Laura Ochoa. I hope I'm pronouncing that right. What if those students are-- what if those are students who never showed up, will they show up in 02? So if they were enrolled in a CAEP program and they never showed up, then they would fall in, probably, that 2a section. And that would be your-- that would be up to you to figure out if that is correct, or that could be the group of students that you want to reach out to again, right? 

So you want to get them back into your program. Maybe there's a way to do some outreach to get them to that 12 hours of instruction. I heard earlier this week, you want to get them going. And then once you get them up in the air, it's like flying a kite and they're willing to go after you get them to that 12 hours of instruction. So this could be an item that your agency looks at to do further reach out to get them to come back to your agency. And you would be able to identify those students, again, by double clicking or drilling down on the item count in the Data Integrity Report. 

So we have items 11a through 16b. And these detail out specific NRS conditions. And these will talk about outcomes like, obviously, you can see, past HSE, earned high school diploma. And then it goes over periods of participation, which is an NRS thing, but will apply here, too, and applies to those that need to take an Employment and Earnings Survey. And then at the very end, it covers IET. 

So starting with 11a, b, c, and d, again, these are table 4, when we're talking about NRS table 4 factors, when we're talking about whether or not they retrieve that Measurable Skill Gain, or MSG. So they achieved a education functional level gain. So you have a student that came in, tested at a certain level, and then, let's say, like a level A. And then they came back and they were post-tested, and they got a significant gain. And maybe now their a level B, and they've changed that functioning level gain. 

With their pre or post testing, they would fall into this category. There are certain other ones for high school credits and all that. That would be 11b and c. 

For 12a and, let's say, 13a, again, these are items that would be marked on your student update record. So when they come in, there's an item to mark on that update record that says GED, passed HiSET. And once you mark that student as passed, then they would fall into this column. Passed HSE and passed high school diploma, again, are in the CAEP outcomes section of the summary report. 

If we see that there's 14 people in our Data Integrity Report under passed HSE and earned high school diploma, but only 10 of our students are showing up in our CAEP summary report as passed HSE, high school diploma, then I might need a drill down on these students, figure out why. There might be something that's not qualifying them for that section. So it could be the hours. It could be that they're missing a birth date. It could be that they're missing ethnicity, and that's why they're not showing up on your summary report. 

And then we have 12b, 13b, again, those are more of FYI items that can be corrected again. And then 12c, again, is just another FYI item there. Once we get into TOPSpro Enterprise, I can show you a little bit more about these and how to maybe make those corrections. 

So moving on to item 15a and 16b-- 

AMANDA LEE: Nicole? 

NICOLE CLARK: Yes? 

AMANDA LEE: Sorry, just before you move on, a question just popped in as you moved on. Harold asked if you can clarify 12b, please. 

NICOLE CLARK: Yes. So 12b is past HSE, but instructional program, not HSE. I think the best example I can give to you on that is that you had a student that came in and is registered as ESL, and then they were marked, at some point, as passed high school, or they passed their GED, passed HiSET. So they're not enrolled, instructional program, not HSE. So they don't have anything in their records that say that they're part of that HSE program. Does that make sense? 

SPEAKER 2: Have a brilliant lunch. Starting Sheila's transcript because she graduated. Amazing. Yeah. 

AMANDA LEE: So, Nicole, there is another question-- 

NICOLE CLARK: Oh, I see that. 

AMANDA LEE: --that looks like we missed from Erica. And it asks if you could please describe the difference between 9 and 9x 

NICOLE CLARK: Yes, so 9 and 9x-- so here, I've gone back a couple slides. 9 is valid pretest but no post-tests. 9x is valid pretest with no post-test in any area. 

For one of our qualifications for NRS reporting is that you have that pretest or that first test. And that would mean that you would have-- you might not have a post-test in that specific pretest. So if you came in and your first test was listening, and you don't have a post-test in listening, then you would show up as having that pretest with no post-test. But let's say the student came in and was listening, and then eventually took math, and then now has math again, then they would be able to have a pre/post test in a different area, but just maybe not that first test. 

So that's what the x is. It really just marks you in with that pretest, that first test. But then the x, we know that students take multiple modalities throughout the entire program year. So you could have multiple pre/post tests and a gain in different areas, but just maybe not in some other ones. Or you could be in different periods of participation, and that could help-- that could be a reason why you fall into those columns as well. 

So going back to the CAEP Data Integrity. So moving on to items 15a, 16b. So this, and I think it was up in the chat a little earlier, we were talking about employment and earnings survey. 15a and 15b identify those students that have within 83 to 90 days for 15b, and then 90 to 97 days of service. 

When we talk about an exiter or why a student needs a Employment and Earnings Survey or they become an exiter, in TOPSpro Enterprise, a student will become an exiter once they have 90 days without attendance, then they've exited your program. Now if they don't have a Social Security number, an ITIN, then you'll need to send them an Employment and Earnings Survey. So identifying students that are about to hit that mark, that are about to exit your program might be a good reason to reach out to those students to see if they need to come back. It's identifying that students have been gone for way too long-- or have been gone for long enough to become an exiter, and then they're going to come back to you. 

Identifies student learning. So again, any time a student reaches that 90 days without attendance and they don't have an SSN or don't have an ITIN, you will need to send them an Employment and Earnings Survey because the Employment and Earnings Survey requirement is not just for WIOA, it is also for CAEP agencies. And that is something that you'll need to do quarterly. Every quarter, send those students those surveys and look to get some responses too. 

And then 16 identifies those students marked as IET. So if you are running that IET program, and I'm sure a majority of you are WIOA Title II agencies, those students to identify an IET learner, they'll need to be marked either at the class instance level or the student level as IET. So they'll have to have a special instructional program, a special focus area, and then a special program too, as IET, to fall in those categories. So if you have this large IET program and then no one's showing up in that section, then that would be a reason for you to maybe look at your class instances, or maybe look at your student records as to why these students are not falling into this IET section here. 

So moving on to 17 to 33-- items 17 through 33. These are our items that don't directly contribute to the CAEP summary or tables. But again, they're state and federal priorities. 

I'm sorry that 30 here is before this one. It's just the way that the table is set up. But you'll look at this first, and then here. 

So starting with items 17 through 22cx-- and I do see that question in the chat. We'll get to that soon. Or actually, there is a question in chat, Patricia Garcia, I see it. And I think we'll cover it in a couple slides. So I have that on one of our slides. We'll get to it. And then I'll reread the question when we get to that part. 

So 17 through 18, documenting primary and secondary goals. So this is a requirement for your learners to identify a primary goal, that would be a goal that is set within the program year. And then the secondary goal would be more of a long-term goal. 

We have an attachment posted on the CASAS website, Attachment X, that goes over goal setting. And then we'll give you examples of what a student's primary or secondary goal may be, and then how to identify that and how to mark that in TOPSpro Enterprise. And here, you would mark that probably in their entry record. So as they enter into your program and you're getting down their basic demographic information, you'll also collect their primary and secondary goal as well. 

Now 19 through 20, these are barriers to employment. So again, as they're entering into your program, you're getting their goals, demographic information, and then all their barriers to employment. They can have multiple barriers to employment, and then they would show here. So if you have a ton of students with no barriers to employment, you can drill down to those and work out a way for your agency to gather those barriers to employment, or even goals. 

22a, 22b, 22c, hours between tests. So we're thinking about when to start pre/post-testing students, if that's what your agency is doing. We have pretests in any area, as long as it's in the conservative range. Remember, it needs to be accurate. They can't score too low in the assessment because then that won't qualify. It'll have to be out of that. 

And then pre/post-test pairs in any area, but less than 40 hours of instruction. If there's a ton of students that are falling in this category, it could be a reason that your agency is testing too frequently and you're not giving enough time-- you're not giving the 70 to 100 hours of recommendation between pre and post-testing. If it's no pre/post-test pair, but the student has over 70 hours of instruction, that could be a reason for you to reach out to those students and start getting them post-tested. 

So you would be able to click on that list. You can send that off to maybe an assessment coordinator or your test administrator and work out a way to get those students in to get them post-tested. 

So items 22a, and these couple of items here, they're specific to this CAEP Data Integrity Report. 23a, 24a, 25a, 26a, 27a, they're all the same-- or fairly the same. So achieved an outcome of HSE, or they have the outcome of post-secondary, employment, increased wages, transition to post-secondary. A couple slides ago we had that chart where it had the outcome and then what fell in that line. 

So here, you would see, OK, I should have the same number of students with HSD/HSE in this line of my Data Integrity Report, that same number should be in my CAEP summary report. If not, you'll have students that are marked in HSE outcome but didn't qualify for CAEP. And you could drill down and see why they're not qualifying for CAEP. 

Again, some reasons that they may not qualify, again, if they have maybe a missing birth date is just the basic one that I can think of right now, or maybe they have a missing ethnicity. And I can show you that soon when we get into TE. And then so 23a focuses on HSE/HSD. Post-secondary is 24. 25, employment. So you would drill down and figure out reasons why-- through drop reasons and maybe using one of our drill downs to identify why that students not getting in there. 

So back here is the section that I was talking about, the items 30 through 33. And these focus on our Employment and Earnings Survey. So item 30 would be those that have an SSN not provided. And those would be the students that you would eventually need to survey at some point two quarters after they exit. 

If your agency is able to collect Social Security numbers, then maybe looking at a list of these students and seeing who you have Social Security numbers for, maybe there's some corrections you can make, because the smaller this list is the smaller number of students you'll have to survey, or you'll have to send a survey follow-up with. Now, items 30d, 30e, 30g here, they discuss the ITIN. And currently, we're just recommending that you follow the same guidelines as SSN as you do with ITIN. 

So this goes back to Patricia's question. When a student provides their ITIN instead of SSN, do you need to mark signed consent? When I do, it shows up under 30a, SSN not provided with signed consent. So I think the first part, though, we can really address is that those students, you should just continue to mark those students the same way and gather consent in the same way that you're doing with SSN as ITIN. 

And then there's another question in the chat from Rosas Monica. Will a student with S-- again, the same thing that-- and the question is, will a student with SSN but no consent need to be surveyed? A student with SSN or ITIN will not need-- 

SPEAKER 3: If they say no in their consent-- they gave us their Social Security number, but they said we could not use it. 

NICOLE CLARK: If it's in the system, if it's in TOPSpro Enterprise, I believe it will check-- it will not require them to get a SSN. So I'd, probably in that case, recommend removing it from our system since they didn't have consent. 

Yeah, so the consent field isn't going to affect SSN. If there's an SSN or ITIN in the SSN field, ITIN field, then that's what's going to affect if they need to be surveyed or not. 

SPEAKER 3: They're entering it at the demographics collection through e-tests. They're punching in their Social Security number. But when we're giving them the consent form, sometimes they just don't want-- they don't want to mark yet. 

NICOLE CLARK: Got it. Jay, I know, sorry, I'm going to call on you for that one. Do you have any comments about that one? 

JAY WRIGHT: Looking. 

NICOLE CLARK: OK, we'll get back to that. OK, so there's another-- 

JAY WRIGHT: Sorry, I'll speak now. Sorry, a second split second too late, just to get it out. I don't think it would. I don't know if I would say it won't matter, but it won't trip you up for ENE or anything with state and federal reporting. What Nicole said on the slide is just a short discussion to date, because a lot of people are asking about whether the same rules for consent apply to ITIN as SSN and so on. 

For now, the official answer is just follow the same guidelines for ITIN as you already do for SSN. That said, I don't think it's an official infraction if they don't have it. It's just kind of, for your info, if you are really dutiful about consent and you know your method is spotless, well, hey, this is really more to catch any that might have slipped through the cracks so you know that you're doing it according to your own district's policy. Hopefully that makes sense. 

NICOLE CLARK: Thank you, Jay. I think that covers a Kevin Sam's question too. Does the ITIN require consent? And if so-- it does-- does CASAS have a sample ITIN consent form? 

Currently, so the SSN form is what we have posted on the CASAS website. So it would be a very similar form for ITIN. But I don't believe we have a specific ITIN consent form available or linked to. Purpose of-- Dana Carter in the chat. Purpose of 30a, just to document that consent form signed, since no consent provided. And then I think that relates to the comment underneath. So it won't matter if we have students in 30, providing they have their ITIN. 

So SSN not provided with signed consent, yes, so you have that signed consent but the SSN wasn't provided. So maybe that's something that needs to get updated in TOPSpro Enterprise, since you're able to enter in that data. 

OK, so I think we have 40 minutes left. Kristen's question in the chat. So to clarify, we put the ITIN number in the SSN field, right? No, there is separate fields for ITIN and SSN. And I can show you that when we get into our Enterprise. 

So now, additional TE reports. So I referred to some of these reports earlier. But these are our Drill Down Reports. So again, we're hovering over cell, we're right-clicking, and then we have these drop down reports. 

The first one is our NRS Monitor. Again, this follows specific NRS requirements, whether they have a pretest, a post-test. So it'll identify if they have a pretest, if they have a post-test. If they achieved a level gain, how many hours do they have, what's their program area. You'll be able to see if those students fall into that line. 

If one of your goals is to make everybody NRS fulfill-- meet the NRS requirements, then you would be able to see, OK, Kat doesn't have a post-test, but we want to get them a post-test. You would be able to see that on the report. 

Drill down to Assessment Audit, this shows, again, the assessments for each student. It'll show you what that upper bracket is, what their pretest is, and then what their post-test is, what their score is, what the modality, what the exact form they took. It'll detail that all out in this report. It'll even show what level they got, what gain they got. 

And now we have our outcome services and monitors. And you'll probably get a better view once I get into TE of what this looks like. But we have our outcomes here, our services here. And in this example here, we have this entered employment. And then down here, the item result that was marked was retained job for our Alexandra. 

And it'll be very similar for those other literacy gains. And again, that refers to that slide that I showed earlier, and probably a bunch of slides that Jay referred to in his CAEP accountability training. And the same thing that works for outcomes is the same thing that happens with service monitor. 

Now there's another drill down here is students in multiple programs. So again, we showed in the CAEP summary report that there was that line for total. And that's a duplicated number of students because you might have a student in ESL and a student in CTE. And then we show a line for how many students were in two or more programs. And then we have that unduplicated count. 

So in here, it'll show out in a grid form, OK, you have a student that's enrolled in ABE and CTE. So I can go over and say, OK, ABE, CTE, there's 26-- or probably in this case, it's ASC and CTE, there's 26 students that go on both. 

And then I'm just going to quickly touch on this before we get into TE again. So just quickly, quarterly data submission-- when we're talking about the CAEP Data Integrity Report and cleaning your data, updating your data, making sure that it's more accurate, for CAEP requirements, you're required to submit the CAEP Data Integrity Report quarterly. And we do have our quarterly data submission wizard instructions and step-by-step guidelines on how to complete that. If you're both WIOA Title II and CAEP funded, you can do it all at once. 

But there are FAQs in here. It'll tell you which buttons to press step-by-step on how to submit it quarterly. So I just wanted to throw that in there. And then now we can get into TOPSpro Enterprise. 

So here is our TOPSpro Enterprise. And I'm just logging into our Rolling Hills simulation. I'll first go over how to generate your CAEP summary report. And so to do that, you'll go to Reports, then we'll go to State Reports, California, CAEP Tables, and then that same area is where CAEP Data Integrity is. But we'll start with CAEP summary. 

So we'll go to CAEP tables first. And for the most part, you'll leave this really untouched. Maybe you want to select different program years, maybe you want to compare last year to where you're at this year. But for the most part, you'll keep it all the same. 

Under this left report setup navigator, you'll go to report selection here at the bottom. And here are all those reports we were talking about that you can generate with CAEP tables. So we're only just going to cover this CAEP summary report here. I'm just going to deselect all at the bottom and only select CAEP summary report. And I can generate it from here. And we'll wait for it to generate. 

I'm seeing some conversation in the chat. Cynthia, thank you for clarifying. So if you're using ASAP and transferring to ET, [AUDIO OUT]. Keep that in mind. 

So here's our summary report. Sorry, let me go back to TE. And here we have this section again, which we identified as literacy gains. So they have their pre and post-tests. They fit our NRS requirements. These enrollees are NRS requirements, there's a total of 474. If I were to add up all these, it would equal 474. 

78 of those students are in two programs. So I'm sure there's some in ESL and some in CTE, ABE, CTE. And our total unduplicated students is 366. 

We have our enrollees with a pre/post test because you might have students that qualify for NRS Table 4, that qualify for that NRS guidelines, they might only have a pretest, but they don't have the post-test. So you can determine at your agency if this looks like an accurate number to you. And then 85 of the 124 students in ESL were able to achieve an entering EFL level gain. 

Then our little less restrictive section here is our CAEP outcomes, passed I3 literacy gains, HSD/HSE, achieve post-secondary, so on and so forth. Here are our total enrollees here. There's 221 that fit into this particular section. Four of those students have achieved other literacy gains. 3 got either passed GED, passed HiSET, or gained a high school diploma. And that works the same for all the other ones. 

And then we have the services section here. So 339 of our students were fit under to the service section. Of those 339, this agency or our fictitious database agency has only marked services received for four enrollees. So two have that supportive services received, transition services received, one. So there's four students, but there's some duplicated because students can receive training and transitional services too. But this is our number here. 

And again, with the drill down, so I'm hovering over and it's highlighting red. I'm right-clicking, and I have our first four listers here. And then I have our drill downs to monitor here. So I have the four students. And if I want to know the names of those four students, I can either double-click or right-click, and then go to Student in Program Year Population. 

And again, I had four students in that cell. So up here at the top where it gives me my count of students, it says four. So I have Judith, Yarly, Abdullah, and Eliya. And those are my four students that had a service received for-- and I'm going to-- and if I click back on this orange, it will take me back to-- so those are my four students that had any services. 

Then I can identify who received the support service. Again, I can right-click, get those students. These are the two that receive report services. So it works like that for all of the different cells here. 

Now we can drill down to, let's say, services monitor. And again, this is going to show us what the transition-- what services it was and then what the exact item was marked. So for Abdullah, Abdullah received support services, and what was checked off in their entry update record was other support services. And then under transition services received, counseling career development was what was checked off. And then we have family child care for Judith. 

And I would assume here-- let's just go a, b, e. Right-click on this 34, HSE/HSD. And then I'll go to my Outcomes Monitor. So see here, I have the outcomes. And these are the HSD/HSE received. All these students received HSD, High School Diploma, and that was marked somewhere in their update record. 

Let's focus on Samantha really quick. Samantha Del Rosario Andre. OK, so HSD/HSE received. I can go into our student record population. And remember, this is the records for all the students. And it'll show all their results here. 

So if I were to search here, Samantha. Hey, maybe-- is this the same person? Yes, so Samantha-- Andre. Under results, this was, at some point, and probably an update record made in the future, in June 24. And it looks like they earned a result of earned high school diploma. And that is why that Samantha is falling into that specific cell. So those are examples of how you can drill down, look at a student's update record, look at some of the monitors, and figure out who's in that cell, who's not in the cell, and where those results were marked. 

So now we'll go into our CAEP Data Integrity Report. I'm actually going to keep this generating. But I'll go to Reports, State Reports, California, CAEP Data Integrity Report. 

And like the CAEP Summary report, you'll want to keep this all by default. Everything's pre-checked for you. You can see that all the program areas have been pre-checked. You just might want to change your program year. 

So I'll click Generate. And there was a-- So here you go. So here is our CAEP Data Integrity Report. Here's our selected students. We have 804 students selected. 

And my guess, I have 804, so that should be all the students in my database. And I can double-check that. I can go to Records, Students, Demographics. And up here where it gives me my total list on this lister, 804. So you can see that 804 matches the 804 on my Data Integrity Report. 

And again, we don't have any students that are not enrolled in CAEP program. But if we did, we can correct that here. We have 66 students that are concurrently enrolled in high school, K-12. And so the total eligible for my Data Integrity Report is 738. 

So now let's get down into the counts. So item one, missing birth date or outside 18 to 110 years old, we have four students. And that is 0.54% of all the students that are eligible for the Data Integrity Report. Now let's see who those students are, because, let's say, I know everybody at my agency. My agency is small. I know everybody. And everybody provided the birth date to us. 

So let's look. So let's look at Tony. Oh, Tony does have a birth date, 2006, 21. I don't know if Tony is-- I don't know if Ruben is exactly 18 yet. But I'm thinking Tony is not-- or Ruben is not 18, so that's why Tony fits in that category-- or Ruben fits in that category, sorry. 

But Tony doesn't have a birth date. And let's say I know Tony's birth date. So I can double-click on Tony, and it would take me to Tony's program year information. I can see in my navigator, I have Tony's records, class enrollments, tests, program records. I can look up anything I need about Tony. 

But I actually need to go into Tony's demographic records. So I would just click here on Tony's name up here. And then here's where it's going to take me to Tony's student information. And I can see here that Tony doesn't have a birth date. 

So I'll click the pen. And I'm going to change Tony's birth date. Tony is going to be born in 1967. Tony is a, let's see, we'll let Tony be a July Leo. So Tony's born on July 25, 1967. I can save that. 

And this is where we have the question about the SSN and the ITIN. So here in TOPSpro Enterprise, we have our SSN up here. And then we have a field for ITIN down here. I can write in-- I don't think that's a valid one because I think it needs to start with 9. Let's see. So we can do that, we can save, and there you go. 

So we've saved that. Tony now has a birth date. I'm going to click out. And this is, obviously, just like a-- oh, and I got this error. So duplicates are not allowed. So OK, let me remove Tony's ITIN. Make sure that's going to fix it. 

No, well, we're getting rid of Tony's SSN. OK, so now Tony doesn't have an SSN because that was done by mistake. But Tony has a birth date now, so I'll click out of here. And what I'm going to do is I'm going to rerun my CAEP Data Integrity Report. 

And while that goes, I will answer a question in the chat by Laura Ochoa. I notice when you generate the CAEP Data Integrity Report, it only has a ASC CAEP program area selected. Should we select all? So I would keep it as is. But there are ways that you might want to break it down. 

It would just depend. But I would keep it as is. I believe this falls in line when it gets ported over, but just leave it as ASC, HSD/HSE. past OK so we'll get to that one. So here we go. Did this work? 

So missing birth date, outside 18, we have an item count of three. Now Tony's not in there. So that's just a very simple example of how you could drill down and change the records if you know to update your Data Integrity Report. 

I see another comment in the chat. Joanne Miller, regarding the CAEP summary column, how is passed identified? What fields in the update record are counted as passed? 

So like the CAEP Summary report, and I'm going to say passed HSE, this is the total number of students that have passed HSE. So I will click on these four students that passed HSE. And this is where we're going to go into their update record. 

We have, let's say, Roxanne. We can double-- there was multiple ways to get here, so I'm taking a little bit of the long way. But we've double-clicked into Roxanne student and program year information. What I'm going to do is I'm going to look at the student record. 

Remember, that Roxanne did pass HSE. So we can see here that there was an update record made in the future on January 28, where a result of passed HSE was entered. I can continue to drill down here, and this is what a update entry record would look like. 

And if I go under Results, here are all these results. If you did take Jay's CAEP accountability, he bubbled in which results fit into what category, which CAEP outcome. So here, passed HiSET was marked as passed. And so because it was marked passed here, then it will show up in that item as passed HSE. 

Well, let's say, I know for a fact Roxanne actually didn't pass HSE, or didn't pass HiSET. That was marked incorrectly. Now I can uncheck pass HiSET, and I'll save that record from the first. And then I would regenerate the CAEP Data Integrity Report. And now Roxanne would not have that record-- would not show up in that column for that item. 

And then let's say, now we need to give Roxanne another student record. So now it's time to provide an update record. I can add an entry update record here. I can go down, I can say I know Roxanne has a couple goals nowadays. And let's say, so work, education results, passed HiSET. So I could check that off again. 

And then Roxanne got a job, which is another CAEP outcome. And the record date, let's just-- it's a weird example because these are made in the future. But we'll just put today's date. Let's say, today is the day we do it. And then I'll click Save. 

Obviously, this is something easier done if your agency is smaller. But if you have some agencies where you're importing data from, then you would probably need to take your data source to see if there's a reason why it's not coming over. 

OK, hold on. Something is preventing me from moving on. Hold on. All right. 

I'm not sure what happened, so hold on just a moment. Oh, OK. So data was saved. OK, so I'll refresh. So on 9/27, that's today, here's my entry record, I put that Roxanne had a secondary goals and passed the job. So now that record has been updated. 

Obviously, you would probably go in progression. But this is kind of the example for that one. 

OK, let's go back to the Data Integrity Report. So earned high school diploma is another example. So we have 32 of our students here. 32 plus 4 is 36. So if I go to my CAEP table, passed HSE, 34. Did I add that correctly? 

Let's see. 30-- oh, OK. So earned high school diploma, 32 plus 30 passed HSE, OK, that's 36. But then on my CAEP outcomes table, I only have 34. So now I need to investigate-- I would need to investigate why these students are not showing up in that column, right? 

My guess is that one of these students didn't qualify for NRS table-- or didn't qualify for the NRS requirements for some reason. And what I can do is I can drill down, right-click, NRS monitor. Let's look at these students. So here's what an NRS monitor looks like. 

It looks like these students are pretty good. But I can see that there's level gains here. Looks like Roxanne doesn't have a post-test. They all achieved a high school diploma. They have some hours. OK, so that one looks fairly good. 

I'm going to go back and look at my earned high school diploma, right-click NRS. OK, I have these 32 students. But right here, this D will identify drop reasons, so why that student was included. So, again, those two columns added to 36. But I have two students that dropped for a reason. And so 36 minus 2 is 34, so that's why these two students are not showing up on my CAEP summary report. 

Drop reason one, if I scroll down, drop reason one is missing birth date or outside of 18, 110. So Ruben is only 17, so that's why Ruben has been dropped from the table. Gilbert, drop reason number five, no ethnicity or race. If I had that for Gilbert, I can then drill down into Gilbert, and then update Gilbert's ethnicity and race down here. So ethnicity, race, and language. 

Let's say Gilbert is not Hispanic or Latino. They speak-- language is Tagalog, and then they're Filipino. So I'll save that. When I rerun my Data Integrity Report, then now, Gilbert should be off that drop. I guess for fun, we can do that. I'll generate that again. 

Here, so we have 34 students unduplicated that received HSD/HSE. I'm just going to, for funsies again, generate the CAEP summary report. And I think we have about 15 minutes left. 

OK, so here's our CAEP summary report. Now it's 35 unduplicated students. Let's see, Gilbert. And now I'm looking up Gilbert. I drilled down, Gilbert. And then there's Gilbert, so Gilbert's here. Gilbert's race was identified as Filipino, which you saw me update here. 

So again, these are examples of drilling down entering into a student's record and correcting that data, if you can, in TOPSpro Enterprise. We'll go to the CAEP Data Integrity Report. And sorry if I'm flipping between a bunch of them. 

Earned high school diploma, we'll right-click. We'll go to the NRS monitor. We'll wait for that to go. Looked at the CAEP summary sheet, found my answer. Thank you. And Gilbert Borja no longer has that drop reason of five. And that's why now all the reports are showing as updated. 

So let's scroll down. So, again, no primary goal here. We have 10 students without a primary goal. Again, you could just drill down to these students, provide them with a primary goal if you know it. Maybe I just need to know who these students are. 

I need somebody to give them a call, so I've right-clicked at the top. And then I put in this column, cell phone. And so now I can be the person that gives them a call, and then I can update their record here for what their primary goals are. 

And here again, we have our students with SSN not provided. Here are the students that don't have any missing phone numbers, mobile, email addresses. Again, you would drill down into that student's record and update it from there. 

OK, let's just do one more here that we can identify. So we'll take this section here. Let's do achieved-- oh. So, again, here, when we talked about updating Gilbert, so achieved a outcome of HSD, we have 35. And then they're marked as HSD/HSE outcome again. 

So I guess I took it the long way. I could have just looked at 23b. But now we have one student. I can drill down on that one student. And that one student should be that student Ruben, who is 17. And that's why they do not qualify. 

OK, is there anything else? There is a question in the chat. And maybe this is more for Amanda Lee, but what web page will be able to find the PowerPoint and recording once it's uploaded? 

AMANDA LEE: Yeah, I was just responding in the chat. But I can come off mic and answer that question. So thank you for the question. We will send this recording off for a third-party remediation, which does take about two weeks. 

You will find that recording on the website under Webinars. I'll pop the link in the chat. All attendees who have registered will also receive a follow-up email with the link and the attachment for your convenience. 

NICOLE CLARK: Thank you. And that pretty much covers this presentation. I'll leave you here with our CASAS resources. 

So we have our CASAS technology and support team. We have five amazing people on our CASAS tech support team. You can give them a call. Here are their hours, and talk with them. We also have our techsupport@casas.org. 

As you are getting ready to submit your CAEP Data Integrity Report for the first quarter data submission, do not wait till October 29 to call them. They will probably be a little busy. So if you can give them a call sooner than later, we can work out some specific issues at your agency. You can also give them an email and ask them to give you a call back. That's fine too. 

And then we have other resources here. So if you have questions about CAEP, we have caep@casas.org. Most likely, you'll hear from me. And if you don't hear from me, you might hear from one of our CASAS program specialists. And [AUDIO OUT] own email as well. 

But if you have-- if you choose one or the other, that's fine. We can put them in the inbox that's needed at the time. But that ends this presentation. If there's any other questions, I'd be happy to answer those for you. And thank you all. 

AMANDA LEE: Thank you so much, Nicole. Thank you everyone for joining us today for your lunch hour. I have popped in the chat an opportunity to provide feedback. If you can just take a few moments and fill that out, we would greatly appreciate it. It truly does inform how we plan our professional development as well as improve. 

We send all of the evaluations on to our presenters so we can all learn and grow. Again, thank you so much for joining us for your lunch hour today. We look forward to seeing you in our next-- and we hope to see most of you at our summit convening coming up in October. So thank you, and have a great afternoon. 

JAY WRIGHT: OK, it's just finishing.