[MUSIC PLAYING] SPEAKER: OTAN-- Outreach and Technical Assistance Network. 

DAVID STANG: Good morning, and welcome once again. As the slide indicates, I am David Stang. I'm just going to adjust this moment. OK, so the bad news is that you just have me today, not Abby, unfortunately, to do this presentation. 

But the good news is that according to the agenda, we're scheduled for lunch, I think, around 12:00. But my presentation will not go nearly that long. So you may be excused for lunch quite a bit early. Again, I'm David Stang, and I'm going to provide a brief overview today of Federal Program Monitoring. Let's see if I can go to the next slide. 

MELINDA: Use your arrow keys on your keyboard. 

DAVID STANG: Got it. They weren't working, Melinda. So I'll just use my mouse. OK, so our agenda for the presentation is to describe or tell you a little bit about what Federal Program Monitoring is. I'll talk a little bit about purpose, why we do it, and I'll talk a little bit about the FPM Office. 

As you may know, there are many different offices here at the Department of Education, the Adult Education Office, us, for example, and the FPM Office. We'll talk a little bit about their role and how they coordinate Federal Program Monitoring for other program offices here at CDE. 

And we'll talk a little bit about Federal Program Monitoring program instruments, more specifically the Adult Ed Instrument, and then we'll talk about instrument items or categories which are found in the instrument, and evidence requests or documents which you will upload into a management information system here at CDE called CMT, the California Monitoring Tool when you get scheduled for a review. 

And then lastly during our presentation this morning, we'll talk a little bit about your role as AEFLA administrators during the course of a Federal Program Monitoring review. 

OK, so AEFLA, or the Adult Education and Family Literacy Act, which is why we are all here today, require CDE to conduct on-site and online monitoring of all adult education grant recipients, OK? So anyone who receives federal funding court documents at some point will be scheduled for an FPM review. 

The purpose of the review is to ensure agencies are in compliance with federal and where applicable state law, OK? And the current monitoring cycle is for this program year 2023/24. 

OK, CDE is required to monitor the implementation or administration of federal and state funding. These are the types of agencies who receive AEFLA funding-- local education agencies or K-12 school districts, community colleges, coalitions, community-based organizations, or CBOs, libraries, and other state agencies, like the Department of Ed. 

OK, I'm going to provide you with a couple of tips during this presentation today. The first of which is please do not wait until your agency is scheduled for a review to familiarize yourself with the Adult Ed Instrument, OK? 

In a couple subsequent slides from this one, there will be links, several links, for you to access, one of which will have the Adult Education Instrument among other resources for you to review prior to your [ INAUDIBLE ]. 

Typically, we do our very best not to schedule program reviews for agencies with new administrators. Within the first or second year, you have somewhat of a grace period. But we do really encourage you to at least get a hold of this year's instrument, download it to your computer and look through it so you can begin to have an idea of what will be asked of you during a program review. 

And if you're up to it and have time, we also would encourage you to even begin prepping for it in small ways that make sense before you're scheduled for review. 

OK, so I mentioned we talk about purpose. And as I alluded to before, part of the purpose of conducting Federal Program Monitoring is to ensure agencies who receive federal funding meet compliance, OK? 

It's also for agencies that receive program funding who are responsible for creating and maintaining programs to ensure they meet minimum fiscal and programmatic requirements. We refer to this as minimum compliance, OK? 

During the course of a federal program review, we are not trying to take back money from you. It's not in any way, shape, or form a got you type of moment. We work together with you and help to the best of our ability to make sure that you get to minimum compliance during your review. 

The purpose is also for the AEO to learn more about your programs and the services provided, and to identify highly functioning agencies. There are many different characteristics of highly functioning agencies and those are the types of characteristics that we look for so that we can share those with other agencies and so that we can learn more about the types of services being provided throughout the state. 

OK, so what does the FPM Office, what is their role? The FPM Office here at CDE, they coordinate most of the reviews that are conducted through the Department. I say most because the FPM Office, the vast majority of reviews are conducted for K-12 school districts, and the FPM Office coordinates those reviews. 

If you are a CBO, a library, or a community college or another state agency, our office, the AEO, coordinates those reviews. That's a little bit different. But in either case, agencies can get selected for review every two years aside from the grace period that you get in your first and second year as a new administrators. 

There are on-site reviews where we actually go on site and do classroom observations. We often review staff and just get a better sense of the program and services that you offer. And then the other type of review is an online review, also referred to as a telemonitoring review, conducted much the same way as an on-site review. We don't physically go to where you're located. 

But both types of reviews involve a 30-day free review period where we review documents that you've uploaded into the management information system I mentioned earlier. We review those documents for 30 days and make sure they meet compliance. And if they do not, then we let you know so that we can work towards reaching minimum compliance. 

And then the final actual week of the review is when we work towards finalizing the review. And if there are any issues, issuing findings and coming up with a resolution process. 

And then regional team leaders, or RTOs, work with the CDE program reviewers, those of us who conduct reviews, and agency coordinators typically at the district level or leadership at community colleges or libraries to conduct on-site or online review. 

Here is a-- from my understanding, you'll have access to these slides. But here is the link to the compliance monitoring web page. This is the FPM Office resources that you have available to you where you can learn a little bit more about the process and what it encompasses. 

There are frequently asked questions. There are monitoring cycles. All K-12 agencies are on a cycle, a two-year cycle. And then there's also information about the criteria that is used to identify which agencies will be scheduled for review, and then there are also links to additional training. 

I don't think you need to access those at this point. Probably a good idea to wait until you're actually scheduled to access the additional training. But just as an FYI, these are the types of resources that are available to you at this link. 

All right, a repeat of the first tip for success, which should convey to you the importance of familiarizing yourself with the Adult Ed Instrument, right? Here's a link. Again, you'll have access. These slides will be shared with you later. You do have a minute to jot this down. 

But please do familiarize yourself with the Adult Education Instrument. And if you have any questions, feel free to email me. I'll put that in the chat once my presentation is done, or to ask your regional consultant. 

Again, you will know well in advance when you will be scheduled for a review. But as I say, it's a really good heads up to go through the instrument beforehand to get an idea of the kinds of things you're expected to know and also to get an idea of documents which, in some case, there can be many, many documents that you will need to upload into CMT. 

So if possible, please take a look at that and get an idea and even begin organizing a little bit, if that's possible, some of the documents that are included in the instrument. 

OK, so our program offices, like adult education, like English learners-- before and after school, these are the types of examples of other program offices here at CDE, fiscal monitoring. 

All of these program offices use program instruments to guide their reviews. You don't need to worry about the other offices. Just adult education. But all program instruments have items, instrument items. They're essentially categories that the reviewer refers to during a review. 

Our instrument and your instrument has 10 items, all right? And once again, here is the link to the Adult Ed Instrument. This is just an overview today, folks. Trust me. The instrument is a detailed instrument and it includes these categories and many of the documents that you'll need to become familiar with and to upload when you are scheduled for a review. 

Just quickly and to give you an idea of the categories, there are 10 in our instrument, 10 items. And just to give you an idea of what items we have in our instrument, hopefully you're familiar enough with the Adult Ed Program by this point that these categories or items will resonate with you in some way. 

The first category or item is collaboration, alignment, and support services, right? So we will ask you to provide documentation in this area that demonstrates that you are collaborating with partners, that you have support services, that you're aligning some of your services and programs with other partners, OK? 

AE 02 is financial accountability. Makes sense, right? As a federal award, there would be some aspect of financial accountability. Data collection and program effectiveness, right? Hopefully, the fact that those two are together gives you some idea of the significance of data collection for program effectiveness. To have an effective program, data collection is a key element. 

And then also staff qualifications. Are your teachers properly credentialed, duty statements? These are the types of things that we will ask you to provide to us when you're scheduled for review. 

Professional development-- we work with a contractor, a Cal Pro who provides exceptional professional development, which is available to you. So we look to see that you are making professional development available to your staff, to teachers that's much needed and that will benefit your students. 

Number 5 is needs assessment. I think Diana alluded to her in the previous presentation. But we want to make sure that you are doing needs assessments on a regular basis of the students and to make sure that the program and services that you are offering is in alignment with the needs of your area, right? 

And then the next group of categories, the second half of our instrument is serving individuals with disabilities, E06. Intensity, duration, flexible scheduling, OK? 

We ask you to upload class schedules so that we can see that courses are being offered at appropriate times and not just at one time during the day. And that they're being offered at a rate and that the hours of instruction are commensurate with when you are pre- and post-testing. 

And then this is one that obviously would need to be here in the instrument, right? Evidence-based instructional practices and reading instruction, OK? Effective use of technology and distance learning, and then finally IET, integrated education and training. So these are the 10 items or categories you'll find in our instrument. And now we'll talk a little bit more about evidence requests. 

So within each item or category, the 10 that I mentioned, there are evidence requests. I think we have approximately 35 evidence requests for our instrument. And here on this slide are just a few examples of the types of evidence or documents that you'll have to upload. 

Under AE 01, remember it was collaboration, alignment, and support services. We ask you to upload an umbrella MOU with the local workforce development board. So hopefully-- well, I'll say this. If any of you are wondering what that is, find out soon that you are required to have a current umbrella MOU with the LWDB in your workforce development board area. 

And it outlines the services that are provided under the MOU and the cost sharing agreements that all of the partners agree to. So this is why we say, please make sure you at least go through the instrument because this is an example of something that you should be familiar with now, OK? 

AE 02-- something else that you should be aware of and make sure that you are doing is to make sure that you have a time and effort policy, right? Your agency, because you receive federal funding-- and many agencies use that federal funding to compensate staff and employees for their time and effort, you are required to complete or to have a policy which reflects what you are actually doing. 

That is staff are completing timesheets, you're doing some kind of semiannual certification. It depends on the agency and what your policies are how you pay employees. But whatever your policy is, it needs to match what you're actually doing. And that is what we check for during an FPM interview. 

Under AE 03, data collection and program effectiveness, you need to upload a local assessment policy. And that policy you can access-- if you're not familiar, you should be. You can access at the CASAS website. Your regional consultant can provide you a template. 

I imagine most of you are familiar with this or there's one somewhere in your previous administration or person who held your role prior to you. I'm sure more than likely they had a local assessment policy, which you can update or refer to. If you cannot find it, as I mentioned, there's one on the CASAS website or you can ask your regional consultant. 

Under AE 04, which was staff credentialing and professional development, duty statements, we want to take a look at all of the duty statements for the staff that you work with, OK? 

And then under AE 06, individuals with disabilities, we ask for your ADA and IDEA policy, right? How is your program, how does your program, how does your agency, what is your process for making sure that students and staff have access-- there's a process for when they need accommodations, all right? 

So just having a policy that says that you do not discriminate against individuals with disabilities is not sufficient, that would not meet minimum compliance. You need a process for when a student has an IEP, for example. What is your process for how you make accommodations for that student? How do you follow-up with that student's IEP? If staff need an accommodation, what's your referral process? 

So again, you can begin to see the importance of looking at the instrument to give you an idea of the types of evidence that will be required to meet minimum compliance. And the sooner you do that, the [ INAUDIBLE ] make your life much easier and your staff if you get a little jump on that now. 

And again, these are just a few examples. There are, as I mentioned, approximately 35 requests. But the last one here under AE 10, if you have an IET program, you're receiving 243 funding, then evidence of co-enrollment, right? 

We want to see evidence that students are enrolled in both and ELL class, English language learners class, as well as some sort of training component, right? That's what that evidence of co-enrollment is. 

OK, final tip for success. Make sure your agency has the correct documentation and/or an example for each of the requests. What a lot of agencies do is they look at the Adult Ed Instrument-- which you'll have links to in this presentation. It won't take you long to just go to the link and then bring up the instrument. 

And then what they do is create folders for each of the categories and evidence requests. So you can begin to just-- when you become a little bit more familiar with the documents as new administrators that are reflected in the instrument, you can begin to just put those into the folders you've created. 

And that way when you are scheduled for a review, you'll at least have an idea of what they are and you can share those with your regional consulting to see if that is, in fact, what we're looking for. But by doing this, you get a head start on the process because it is just a matter of time. If an agency has not been reviewed in three or four years, they will get scheduled for a review. 

OK, so what's your role as AEFLA administrators? As we've talked about and I've emphasized probably too many times this morning, becoming familiar with the instrument is obviously important. 

Ensure you and all additional users-- so when you are scheduled for a review, you're going to get access to CMT CDE monitoring tool, some management information system here. Once you get access to that, whoever you delegate amongst your staff to help you with this process-- and you will want to delegate because it's a big process. It takes time. 

As I mentioned, there's a 30-day prereview period. But even leading up to the prereview period, there are things that you can be doing. And even your reviewer can even look at documents prior to the 30-day period. So make sure everyone has access is the point here to CMT that you want to be helping with this process when you're scheduled for a review. 

You can assign additional users. So others can have access to CMT. So you are not the only person who is going into CMT and uploading what can be many, many documents. So you want some help with that. 

And then communicate with district personnel or your agency's leadership about who's being assigned to coordinate the review, right? If you're a K-12 school district, as the adult ed director or principal, you really are the most appropriate person to lead the adult ed portion of the review. 

Sometimes school districts will assign someone from the district, which we do not-- we cannot tell schools or any agency who to appoint as a coordinator. But oftentimes when school districts do that, many of those people they assign at the district are not familiar with adult ed. It just slows up the process. 

So reach out to district staff. Make sure you have that rapport or relationship so that you can communicate the importance of you having access to CMT. The process works much better when your reviewer, who are your regional consultants, typically work with you and not somebody at the district to get all of the evidence uploaded into CMT. You are much more familiar with your program and how it runs and what forms of evidence would be appropriate to provide to the CDE. 

And then coordinate on-site scheduling and work with program reviewer during the course of the review. So if you're being scheduled for an on-site review, whether you're a K-12 school district or community college, library, whatever type of agency you are, if you're being scheduled for an on-site review, make sure you work with your reviewer, your program reviewer to put together a schedule so you have an idea, that your staff have an idea what day 1 will look like, day 2, day 3, and day 4 if necessary. 

OK, so looks like my slides-- I apologize. It looks like my slides are little out of order. That's OK. We'll just go through the review together. I had a little review for you here, but that's OK. 

Which AEFLA agencies get selected for program reviews? So you all are representatives of AEFLA agencies. You receive AEFLA funding. So which of you will get selected for reviews? All of you at some point will be scheduled for review, all right? 

Which office at CDE coordinates most federal program reviews? Again, if you're a K-12 school district, the CDE-- the FPM Office at CDE will coordinate your review. If you are any other type of agency, a library, another state agency, community college, or CBO, the Adult Ed Office will coordinate that review. We will. 

And then, how often may agencies be selected for review? You can be selected. It doesn't necessarily mean you will be, but you can be selected for a review either online or on-site every two years. Typically, we do not review agencies every two years unless there are issues or concerns that we have. We usually like to review agencies every three, four years, not two. 

OK, I see here the problem. Again, my apology for the slides being out of order. And then, what is the name of a document that we really want you to take a look at as soon as possible? That's the Adult Education Instrument. And last question, what is another name for the documents agencies provide during a program review? Those are evidence requests, OK? 

Again, as soon as we end here, I'll put my email in the chat. But that's the end of my overview of Federal Program Monitoring today. Do access the instrument when you get a chance. 

And please, email me or any of your-- I'm one of the FPM leads in our office along with Abby Medina Lewis. So if you have specific questions, feel free to email us or your regional consultant. Carolyn or Jim, I don't know if we have questions or if we want to need to move on. 

JIM SHIELDS: Hold up. Dave, you do have a couple of questions. Do you want me to call those up for you? 

DAVID STANG: Yes, please. 

JIM SHIELDS: OK, OK. We have one question. Where can we find the document containing characteristics of highly functioning agencies? 

DAVID STANG: We do not have a document that reflects anything or the characteristics of highly functioning agencies. That's an interesting question, though. But the purpose, one of the purposes of FPM is for us, as I mentioned, to identify those agencies for myriad reasons when we want to know more about how they provide certain types of services, how some agencies-- 

Adult ed has a lot of moving parts, and there's a lot to it. The learning curve for people who are new to adult ed can be pretty steep. And so we try to identify those agencies to do presentations when appropriate or in a certain areas, to provide support to agencies that could use some additional support, and to let you know who those agencies are. 

So if you have questions, if you're not sure about something or you need support, reach out to your consultant and they may be able to put you in touch with some of those highly functioning agencies. But again, that's an interesting concept that you raise. We do not have anything specifically that indicates the characteristics of highly functioning agencies. 

JIM SHIELDS: Cool. OK, and we have another question. To clarify, new administrators have a grace period of two years and will have their review in the third year. 

DAVID STANG: Good question. Not necessarily. We do our best to make sure that-- we know that you have a lot on your plate. And for those of you who are new to adult ed, as I mentioned, the learning curve can be pretty steep. We try not to burden you with a review because it can be a lot. It's doable, especially if you're well-organized and you start well in advance. 

And we have many tips. When you are scheduled for a review, we conduct training. You will be well-prepared come the time you have your review. But you have that grace period. It does not mean that you'll be scheduled in that very next year, but typically, we really try to review agencies no more than every four years. 

Agencies that are highly functioning and typically don't have issues, we may give them a little bit more time. We may allow for an additional year. But we really do our best not to allow agents or not to go more than five years, four or five years for a review. 

JIM SHIELDS: OK, and it looks like our last question. When will we know if we are selected for a review? In the previous school year or the summer before the review. 

DAVID STANG: Good question. Typically, the notification process begins in May and June and we will first initially let you know that the scheduling or identification process has begun. 

And I mentioned for K-12 school districts that there are monitoring cycles on the FPM or CDE monitoring website. If you take a look at those cycles, it will give you an indication of when you were last reviewed. 

But otherwise, we give notification to agencies typically in May and June well in advance of when you'll be scheduled for-- reviews are scheduled throughout the year. But usually, by the middle of July, you should have a good idea if you've been scheduled for the subsequent year. 

JIM SHIELDS: OK, I think that's it for the questions. 

DAVID STANG: OK. All right, thank you everyone. Again, I'll put my email in the chat. And if you have any additional questions or you need help accessing those links, let me know. Thank you.