ANTHONY BURIK: This is Anthony from OTAN. Welcome to this afternoon's webinar, Considerations When Implementing Distance Learning Part II - Local Assessment Policy. The presentation slides are actually available right now on the OTAN website. So if you go to the OTAN website, otan.us, on the right hand side if you're on a desktop or laptop, you want to look for that COVID-19 Field Support button on the right hand side. If you're on a mobile device, you might have to scroll down a little bit. 

The COVID-19 Field Support button will take you to the COVID-19 Field Support page. And the slides for the presentation today are listed at the very top of that table. So you can go ahead and download those slides right now. And Jay Wright, I'm going to turn it over to you for today's presentation. Go ahead, Jay. 

JAY WRIGHT: OK, great. So looking-- there we go. OK, there's the sharing. So here's our cover slide. We've got too many to worry about straw polls and so on. But this is what we're calling Part II. Because there's that Part I that we were just simply calling Distance Learning Considerations that we've done collaboratively with the OTAN CASAS team four or five times now. 

So we did that a bunch starting there in late March when everybody was in quarantine mode. Obviously, lots, and lots, and lots, and lots, and lots of questions have come out of that to the point where we agreed, yeah, it would be good to piece some of that together and make a Part II of this to start covering a lot of the things that came up during Part I. A lot of that was related to the local assessment policy. I won't say it was restricted to that, but that was definitely the number one source of questions related to Part I. So for the last few weeks, we've been sort of taking note of that and putting together this Part II. 

You might say another funny thing happened on the way to the forum with this. Just like all things COVID-19 since mid-March, you might say, where things keep changing, in this case, the same basic approach is holding fast. But, as we were saying amongst us CASAS OTAN folks before this, this has turned into a little bit of a transition from nice to do to need to do, meaning that we really developed this over the last couple of weeks with the idea that you had lots of really good questions, in particular questions about how to address your local assessment policy. So knowing this was coming and knowing it was definitely best practice, we were crafting this presentation to provide tips and more, in particular, lots of specific examples of ways you can craft it in your policy. 

The reason why I say need to do is a lot of you, I'm sure, have been to those CASAS workshops on remote testing and so on. You've heard a lot of those updates on the federal memos and so on. So it's kind of gone more from nice to do to need to do. That is, the Feds now require states to submit an addendum to the statewide assessment policy if they wish to do remote testing. To be clear, California is definitely opting with yes on that. So thereby, as a state, we're now required to do an addendum to our statewide policy. 

Because we're selecting yes, now US local agencies that decide to do remote testing will need to do an addendum to your local assessment policy. So that's why I say we transitioned this, literally in the last two or three days, from what I would call nice to do to need to do. So it now includes some of those specific issues related to how we're going to deal with this as a state. 

I'll just say, hopefully that makes at least a little bit of sense. I know it doesn't make a lot a bit of sense, but hopefully a little bit. So in any case, the agenda, that's what I mean by updates. We'll get into a few updates that have occurred. Again, we're moving fast at the state level. We're moving fast at the federal level. It's only been a couple weeks since we've done this presentation. But that's all it's taken for things to have changed in terms of rules, and regs, and all that wonderful stuff. 

Because of that, I'll recap a little bit on Part I, sort of talk about the updates and how that's affected things here. Part of the recap is a big part of Part I, of course, was looking at those different ways in which we can assign instructional hours to distance learning. So we'll recap that and how that will affect what we need to do with the local assessment policy. 

Because most people, quite frankly-- even if you say you know all about the local assessment policy, sorry, I know too well that the overwhelming majority of folks know little or nothing about the local assessment policy. Lots of things we've understood about that that's just reinforced that opinion, so we're going to go ahead and provide a brief overview on exactly what those things are to make sure everybody knows what we're talking about when we talk about modifying the policy. If you don't know what the statewide policy is, you don't know what the local assessment policy is, obviously, expending a lot of hot air on how you need to modify those things is not going to be much help. 

Related to this, there's an Appendix C, Distance Learning. No, that is not new. Appendix C has existed for at least 12 years, since at least the '08-'09 program year. I think it dates back to '06, if I'm not mistaken. But we've had a special Appendix C in our statewide assessment policy specifically to address distance learning for that long. If you didn't know that, don't worry. You're in a very big club. Just about everybody didn't know that. But there is an Appendix C that's been around for a while that's collected a lot of dust over the years, quite frankly. We've spent a lot of time unearthing it the last couple of months. It's worth pointing out that it does exist, and it does cover all those things we've been talking about. 

Now that the Feds have moved on with these new memos and all these issues related to distance learning, there's a new addendum that hasn't existed yet that will need to begin to exist starting here in 2019-2020. We'll talk about that as well. And then at the end, we probably won't have time to go into all these gory details. But when you download the PowerPoint, when you start working on this on your own, the thinking is there's lots of examples we added here. I want to say there's like eight or nine examples at the end. Again, not enough time to cover each one, but it should be pretty self-explanatory when you download the PowerPoint and review the slides. 

There's different examples based on what learning model you're using. A lot of those examples that Penny gave in her presentation are elaborated upon here for accountability purposes and specified in terms of how you would do it in terms of local assessment policy. So I'll just pause. Hopefully that sounds more or less like what everybody was expecting to get themselves into here this afternoon. Next, please. 

OK, so here's the first slide of Part I. Because it's so darn important, it's also the first slide of Part II. COVID-19's effect on agencies, of course, that's comprehensive. We've all been affected by COVID-19. Everybody's closed. We're all in quarantine mode. 

We've talked about the resource page that OTAN has. That's been our official COVID-19 hub since about Friday the 13th of March. It's got all kinds of FAQ documents, official proclamations from CDE that were all affected by this, that we're all looking at the same age, FAQ documents from CASAS, FAQ documents from CALPRO and OTAN, all kinds of help documents, all kinds of information on all things COVID-19. I think we know about that already. Here's the link at the bottom of the slide. We'll move on to the next. 

So here's a little bit of an update. We've talked extensively about memo 20-3. That is the big memo that came out at the end of March. Again, we went into quarantine mode mid-March. You know, at the end of March, the Feds came out with that memo where they went out of their way to provide all of the additional guidance that all states were looking for, given that we were all in COVID-19 quarantine mode. 

That basically proclaimed from on high that, yes, COVID-19 is a really, really big deal. But that doesn't mean we're going to stop collecting data. We're still collecting data. We're still reporting at the state level. We're still reporting at the federal level. Just because COVID-19 is a really big deal doesn't mean that we're changing that. But it did at the same time give states a lot more increased flexibility on how to handle testing, how to handle outcomes, on whether to-- you know, on how to handle instruction. We all did at a slightly different way. It basically loosened things up so all states could kind of find their own way and not worry about whether that's in violation of federal policy. 

So 20-4 came out a few weeks later. It basically just followed up on (inaudible) 20-4 simply upped the ante on 20-3. Again, we're still collecting data. But we know we're doing it in a very modified way. It increased the flexibility to the states to, number one, implement remote testing. Because it seems like eons ago, but when we started this, our answer was we can't do remote testing, because the Feds say so. The federal guidelines were really clear that all pre/post testing required a face-to-face Proctor. 

So they've loosened that twice now. They've loosened it up so each state can decide on its own whether to do remote testing. But it kind of upped the ante on it. You know, can you hear me better now? Here, I'll turn it-- I'll turn my head differently. 

So any case, it upped the ante, more flexibility to states. It basically said, if you're doing remote testing, however, that means you need to submit an addendum to your statewide assessment policy detailing how your approach is going to be. At the same time, that also means agencies doing remote testing in turn need to update its local assessment policy. 

So to be clear, each state gets to decide whether it is or is not doing remote testing. I'm sure a lot of you were in those webinars yesterday where it was very clear that CDE has selected the box labeled yes for California, meaning yes, California is going in on remote testing. As you heard yesterday, we're not requiring all agencies, all students to do remote testing. But we are encouraging to the maximum extent possible agencies and students that are able to do so to the maximum extent possible do so. So we've chosen that option. 

I'll just add one last thing, that if we're exempting students from testing, we have to have a detailed timeline for how we're going about moving away from that if we chose to do it that way. And we also have added that Force Majeure check box in TE to identify those students that, either on the state level or the agency level, we're exempting from testing and/or may be impacted somehow, some way by COVID-19. 

All right, so moving to remote testing, again, this is not the workshop where we're doing that. We had two very large workshops yesterday that detailed that. This is more for policy and accountability. But we did have two big webinars yesterday on remote testing. 

So, again, CDE has decided that California has selected the box labeled yes. We are implementing remote testing. So to be clear, CDE is drafting a document. We referenced it several times in yesterday's webinars. It's not ready yet. It's not posted. But it will be posted later this week where there will be an official CDE document that will basically serve as the addendum to the statewide policy on how we're handling distance learning, more specifically how we're handling remote testing in California for the '19-'20 year. We're doing that because the Fed said so. 

So in that CDE document, they're basically approving remote testing. But it's saying if agencies choose this, they must have procedures in place to ensure, number one, the student can be identified, number two, any approved test that's administered is properly secured, and number three, the remote proctor can properly administer the test. That seems simple. I'll just say that's the oversimplified way of basically accounting for what we spent an hour two times yesterday detailing in that remote testing workshop. 

That's just a simplified account of our hour-long account that we provided yesterday where we got into details on how you can make it secure, how you can identify that student, and so on. This is just simply stating that we, as the state, are saying it's OK as long as you can follow these steps. So we provided that webinar at CASAS to give you the details on how we think everybody should be able to do that. 

OK, here's the details. I'm not going to read this word for word. But again, here's the more detailed guidelines that you'll see in that CDE policy. In this case, I really just did a copy and paste. So this is not anywhere near the whole document. But, for what it's worth, yes, this is a copy and paste from that official CDE document on remote testing. 

So right now, it's just saying CASAS, GED, and HI-SET are the only ones that you're allowed to do this way. You need to complete that remote testing authorization so there is an agreement. Again, that's something that we at CASAS are still working on. We will make it available no later than May 11th. It's still in process now. If it's available a little earlier, we might post it a little earlier. But may 11th is the date we're promising. So I'm not going to give a date any earlier than that. 

But this will be probably a short little one or two page document where each agency is basically throwing its hat in the ring indicating that it's doing remote testing, indicating that you're following all those additional security guidelines, indicating that you've got the technology at the agency and student level, specifying which CASAS assessments you're doing, all these things that you're signing your life away, saying you're taking these precautions, signing off on the copy, submitting a copy to CDE, submitting a copy to CASAS to basically have you be good to go. 

To be clear, this is not a training you're completing. This is a two or three page document you're completing, basically signing off that you're following on all the guidelines, you'll have a copy locally that you'll have handy if you're monitored for any reason. At least for this year, because it's such a big deal, for this year, you'll submit a copy to your CDE consultant. You'll also submit a copy to your CASAS specialist. 

That's for this year only. I'm not sure that that will be a requirement every year. But for '19-'20, while we're still in quarantine mode, you've got to submit not one but two separate copies to make sure we're all aware that you're doing this. Because we're doing the full court press on getting everybody up and running. 

In the future, my guess is you'll continue to need to do this form. But you may not need to submit copies every year. It'll probably settle into being something that you'll need to retain locally like the rest of your local assessment policy. But if you're monitored, you ought to have it available and ready for show. 

OK, and then again, you need to have the training records. Bottom line is there's not a special training that you're going to be required to complete for this. But you do need to have anybody that does this be somebody who's already completed that proctor training. And you need to have the documentation that shows that person already has completed that proctor training. 

And then, again, you need to submit a written standard, or you need to incorporate a written standard procedure for how you're doing remote testing and add it as an addendum to your local assessment policy. To be clear, that item under five is not something that you need to submit to CASAS or CDE. But it is something you need to complete before getting involved in remote testing. It is something you should have on hand if you're ever monitored for any reason. 

So I won't get into all this. But this is a slide from yesterday. My guess is most of you recognize it, because you were in one of those trainings yesterday. If not though, we had Phase 1 that started a couple of weeks ago. It ended last Friday. We started Phase 2 this Monday. We started very cautious. We only had like five or six agencies nationwide. We limited it to just one on one student to staff testing. 

So again, by one to one testing, we mean we did it-- by doing it remotely, it's just one staff test proctor and one student completing testing. It's a one-to-one ratio. That Phase 1 was restricted to one-to-one ratio. Now that we're in Phase 2, we opened it up to a larger number of agencies. And as you heard yesterday, we opened it up to where you can continue to do that one-to-one ratio. But in Phase 2, you also have that option to do the one-to-five ratio where that test proctor is monitoring five students at a time instead of just one. 

Again, we're staying pretty cautious with this. But again, it's-- lots of issues have come up. So we need to keep it very cautious, keep it slow. That's been our approach on this. We know that we need to go that way. So we're staying one-to-one and one-to-five. I'll add, we also added government and history for EL Civics. We started adding new kinds of technology. 

We're going to stay with Phase 2. Eventually, we'll hit Phase 3. And we'll start looking at additional options. But again, we kind of fell off the grid with that, because a lot of those things really depend on how things proceed with Phase 2. 

So this one, I don't need to talk about a lot. But this is just a screenshot on how it looks. Again, student and staff are on camera on Zoom. Student is able to see staff. Staff is able to see student. That's a big part of the test security on how we feel like we can get away with doing it remotely. We can do it remotely, but only if we can view what all the students are doing and vice versa. 

Here's a copy of that agreement. Again, this is still a work in progress. It should be available the next few days, again, no later than May 11th. But again, all agencies that opt to do remote testing will need to complete this remote testing agreement. To be clear, this is the one that you've got to do, and you've got to complete, and you've got to submit to both CDE and CASAS before getting started. 

You can see here, it's pretty basic. It's not rocket science at all. You're just giving the name of your agency, who's giving consent, who the person is signing their name, signing their life away, specifying which assessments you're using and which ones you're doing remotely, specifying which remote techniques you're doing. Are you doing all one-to-one testing? Are you dabbling into one-to-five testing? Which specific technology platforms are you using? Obviously, you need to have this thought out before you just dive in. 

So by forcing you to do the agreement first and then jump into remote testing, it ensures that you're giving thought to the remote testing procedure, the technology, the assessments, all of those things you need to think through before you do remote testing. By addressing these basic decisions first, it guarantees that you did exactly that before doing it. 

OK, so I think that's all I wanted to cover on remote testing. Again, the details related to security, and ID, and the whole pilot testing process was obviously covered in more detail yesterday. So we're kind of moving on. But sticking with the topic that was covered yesterday, that's that Force Majeure. So again, by Force Majeure, that's the term the Feds are using to refer to those students that were impacted by COVID-19. 

In particular, it's been talking more and more about students that are exempt from pre and post testing because they're unable to do so due to COVID-19. So you've heard about this a little bit, because it's been in TE for a while. We programmed it in TE back on April 15. So we've got the checkbox in a few different places in TE. And again, it's been there for a few weeks. 

But we've gone way out of our way at CASAS, and same for CDE, to say don't mark anything related to this yet. The CDE is still completing its policy on when to mark things for Force Majeure. So some of this has been as advertised for a few weeks. 

What we've been saying is there's been different interpretations on exactly what that means. Some would say, well, of course everybody's been affected by COVID-19. So if we're just saying, who's affected by COVID-19, by definition, that means everybody. So we need to check everybody. Others are saying it more specifically needs to be just the people that got kicked out of class. Others are saying it's more specific to testing. So again, CDE is making a very specific policy for California that will clarify exactly what needs to happen at the agency or student level in order for you to check that box. So that will also be available as part of the statewide policy here in a few days. 

So again, that's what we mean. There's a checkbox in TE already. Another bullet that I've put here-- I did this with some trepidation by putting this bullet it on the slide. But there really was no better place to it. Way TMI, but I'm saying all this TMI to hopefully use that as a way to circumvent the confusion between the Force Majeure checkbox that already exists in TE And the 125,000% separate checkbox-- I'll repeat again, because I know even though I say this, people are still going to get confused. 

Even though I say this in as heavy-handed way as possible, completely separate checkbox from Force Majeure-- Force Majeure exists in TE. The second checkbox I'm talking about totally does not exist in TE. So is everybody exceedingly clear? I know I'm making a fool of myself here to avoid all these silly questions. But I'm going to get all the silly questions. But I'm huffing and puffing to try to avoid the silly questions-- two separate checkboxes in TE, totally not the same thing. Does everybody follow that? 

OK, because you know everybody is going to ask this question anyway, right, and get the checkbox confused as the same thing, even though I'm making a fool of myself right now, right? So anyway, two separate checkboxes. Force Majeure in TE for those who are exempt from testing because of COVID-19 or potentially exempt from classes or exempt from services for that same reason. The second checkbox relates more specifically to remote testing. And it will be the box that you check for those students that do complete remote testing. 

So there's a checkbox that's there that you mark for those that are exempt, a separate one you'll mark specifically for those students that complete remote testing. Most likely the second one will be in the test record. But it will be something that you check for those specific students that did remote testing. Most of you have pointed out a million completely verifiable, understandable, good reasons for why there's no way on God's green earth you'll be able to remote test everybody. 

But, hey, as long as it's understood you're just doing it for some and not all, hey, you'll get on board with this. You're all for it. So again, that means that we're going to need to specify which specific tests, which specific students those were that did remote testing. So that's a checkbox that does not exist in TE yet. So don't go looking for it. Because there's 125% chance you won't find it. Because again, it ain't there. 

But it will be coming very shortly where you'll mark those specific students that do remote testing. So-- 

JANICE FERA: Hey, Jay-- 

JAY WRIGHT: --huffing and puffing on that issue. Go ahead, Janice. 

JANICE FERA: Sure, I've got two things. One to share is this morning I got a phone call from a consortium manager. And she said that she was looking at some of her data. And she noticed that one of her agencies had very aggressively gone in there and marked Force Majeure for a whole bunch of their students who were impacted. 

So the advice back to her was, please invite your agency. Go into the student in program use records. You can grab them, do a bulk batch edit, and go ahead and remove that flag. 

JAY WRIGHT: Right. 

JANICE FERA: Because-- 

JAY WRIGHT: No problem. 

JANICE FERA: Right. The Q3 assessment-- 

JAY WRIGHT: If you're doing that, I'll just say no harm, no foul. It's not a capital offense if you mark it anyway. We're just trying to have some standardization. So yes, you can uncheck it just like you checked it. I'll just say no harm, no foul. 

JANICE FERA: Good. And then Jill had a question. I left it open in the question box. I wanted you to kind of look at that. Who said-- she says, did Jay say that we also have to submit the local assessment policy with addendum to CDE and CASAS as well as the remote testing agreement? 

JAY WRIGHT: No, OK, good question. I saw that coming. I thought I cleared it, but maybe not. So yes, that agreement that's just-- you know, where you're checking off your technology and checking off which CASAS assessments, that one you do need to submit to both CDE and CASAS. The addendum to your local assessment policy, you don't need to submit at all. Yes, you should do it. But no, you don't need to submit it. I'll just say the policy for that, of course, is exactly the same as it is for the local assessment policy in general. That is, of course you need to do it. But no, you don't need to submit it anywhere. 

Definition of distance learning, this is straight from Penny's presentation. Again, any learning activity where students and teachers are separated by geography, time, or both for the majority of the period. There's all the details. I'm not going to read that. But if you want it, there' the page in the NRS implementation guidelines that gives you that information. 

So here's more review from Part I again. And generally speaking, DL learners have had the same requirements as everybody else. Since 2009, we've followed the 50% rule, simply meaning more than 50% just-- you know, in a distance environment mark distance learning. Less than 50% in a distance environment, don't mark distance learning. And then again, there's the new feature that tracks activities affected by COVID-19, called Force Majeure. I just talked about that. 

OK, more review from Part I. Here's the three models. I went over it a little bit. Penny went over it a lot of bit. But those three models in the NRS guidelines-- clock time, teacher verification, learner mastery, and review. Clock time is simply measuring hours based on the clock. That is, if they're spending 30 minutes in the regular classroom, that means exactly the same thing as 30 minutes of distance learning. If you're using that approach, that's using the clock time model. 

The other two, teacher verification, learner mastery, that's more for asynchronous, where you've got things like software, educational software, or videos, or Zoom recordings, or web sites, or whatever. You're not necessarily with a facilitator. You're having the student do it on their own. So teacher verification is marking hours by having the teacher be directly engaged and directly observing each student and having the teacher determine on her own or his own exactly how many hours of instruction each student deserves. 

Learner mastery is the one we've recommended in California since about 2008-2009. That's suggesting that we go ahead of time, look at that software, look at that video, or website, or whatever, assign a specific number of hours to that application, assign it ahead of time, and then that way you can mark the same amount of instruction equally to all students based on the fact that they completed that particular module. 

In Penny's presentation, she talked about testing. I've gotta say, to be clear in our California assessment policy, having a formal test is not necessarily required. But that is one way you can do it. The more common way is in some of the software applications you're using, usually they have a built-in test already done module for module. So most agencies simply use that test that the educational software provides for you by the student completing those exercises or that test in the educational software. 

Each unit or lesson, by completing that, that demonstrates mastery. You can use that as long as you record it as such and have students complete what's already built in in the software to serve as the way that they demonstrate that mastery. 

OK, then this is more review. Facilitated/synchronous, we're saying use clock time. Generally speaking, asynchronous, again, recorded sessions, videos, educational software. Historically, we said use learner mastery. And then this last bullet is kind of what we've done on the fly, where most of you are using software. We've come up with those three general approaches. 

So again, if they-- with the software, this is all in review. Some software are higher end. They're a little fancier. They have the amount of time programmed into that back end. If you're using software that programs the amount of hours into the software already, well, then the obvious thing is use what the software gives you. Then you don't need to do any of these models. Because the software does it for you. Take what the software publisher gives you. 

If you've got software where it's not programmed, but sometimes it won't be any fancy programming, but they will give you recommended time frames, then again, give what the software publisher-- take what the software publisher gives you. You use the recommendation provided by that software publisher. Again, by doing that, you're on firm ground by accepting what the publisher says. Then you don't need to have any extra responsibility on the teacher or the management team to do those extra steps. 

Number three is where you're kind of stuck where you might need to take those extra steps. If you have nothing in terms of recommendations or programming from the software, then that's where we're most likely to recommend learner mastery. That's where you need to look at it as a team and kind of plan that out ahead of time and specify how many hours you're giving the student for each module. 

OK, here's the general guidelines. The first one, hey, if you don't get anything from that publisher, a lot of times a really good idea to do it anyway. Usually they're going to have some information. No matter what they have, good to get whatever information you can from that publisher. If you have to do it on your own-- again, fewer and fewer are having to do this. But if you have to, again, don't have it be one person. It doesn't have to be anything that excruciatingly detailed. 

But you should at least have more than one person be on the team. So it can at least be an agreement and not be the opinion of just one person. Another one that's come up a lot with the questions, trial and error is allowed. I think I've used this cliche already today. But again, when you're doing this as an agency, totally understood you're not going to get it perfect the first time. So don't worry about getting it perfect. 

Something is better than nothing. If you need to tweak it later, tweak it later. Again, trial and error is totally allowed. If you decide learner mastery is not your bag and you'd rather put it on the teacher, that's 100% permissible. But if so, be sure to document it in your policy. If so, be sure to-- you know, that means the teacher has a lot more on her or his plate where that does suggest that the teacher is being very vigilant, student by student is able-- and is so vigilant that she or he is able to differentiate how much time student x put into this versus how much time student y did on this, and by observation is able to make that level of granularity. And then, again, if so, describe that in your policy. Next slide. 

OK, so getting in to the assessment policy here, again, over, and over, and over again already, you've discerned this was true in Part I. The same tired answer for a lot of this-- document it in your agency's local assessment policy. I put the link here to that attachment B in our admin manual. Attachment B, for many years, it's been the local assessment policy. 

I'll just say, if you don't know what I'm talking about when I use that term, here's the link to Attachment B that's been the local assessment policy template every year since 2005. 

Here's what the local-- here's-- so again, we've got statewide policy and local assessment policy. Just a couple things about the CDE statewide policy, if you have no idea what the local assessment policy is, sorry, but highly likely you have no idea what the statewide policy is either. So just to make sure everybody's on the same page, the CDE statewide assessment policy, 20 or 30 page document, existed every year since 2005, statewide policies on pre and post testing, again, recommendations for PoPs, placement testing, 40 hours rule, requiring you to do the local assessment, and so on, all those things you know. And then the down ballot issue that's always been there that nobody really paid much attention to is, again, that policy for assessing and reporting distance learning. 

For the local assessment, again, just canned description here as well. Again, part of the statewide policy for 15 years now is we've got the statewide policy. All local WIOA II agencies are required to develop its own local assessment policy and update it at the start of each program year. So again, it's always been flexible on exactly what the do's and don'ts are as long as you address all those issues specifically addressed in that CDE statewide policy. 

And then, again, you've never had to submit this anywhere. The policy has always been you need to do it every year, but the way you report it is through your FPM visits. There's no formal annual reporting or deliverable you necessarily need to complete. 

So I won't get into this. This is going to be for a future when we do where we really roll up our sleeves on local assessment policy. But a big long laundry list of things a good local assessment policy should entail-- so that last slide is just telling you, yeah, as long as you address what, because CDE says so, you're basically fine. But over the years, we've obviously said a lot more things that a good policy really includes. But it's going to get into more detail on how you handle placement, pre and post testing. 

If you're a good agency, you're maintaining a calendar for the whole year that includes testing dates and strategies for makeup dates for those who are answered. You get into details on test security, specify which CASAS tests you prefer, get into how you're addressing regional priorities. 

Get more into hours between test, 40 hours rule for training and so on. And then you're also addressing some-- you know, again, it's kind of-- gets more in the weeds, the part that you probably think about last, but we're starting to think about also how do we provide accommodation for disabilities. And there it is again, what are our procedures for learners and distance learning? 

So here's the sections. I won't read off on this. But again, here in a week or two, we'll follow up and get more into local assessment policy and detail it. For now, these are the sections that we have in our local assessment policy template. These are the sections you would address in any good local assessment policy. 

I'll just add there's the part of this that nobody knows about, Appendix C for distance learning. And then there's the part that you know about even less, because it doesn't exist yet, where there'll be that new appendix for program year 2019-'20 on how you're going to do remote testing. 

I'm also going to use this as a little bit of a sanity check. Did everybody follow that transition from generic stuff to more specific stuff as it relates to COVID-19 and distance learning? Just making sure. OK, that's your story and you're sticking to it. Thank you. 

OK, so again, other issues here. Again, these are not necessarily required. It kind of depends on what you're doing at your agency. But again, accommodations, EL Civics, any security that might be special to your agency, differentiation between eTests and pencil-paper, if you've got agreements with your WIOA I partner and consortia, whatever. And then, again, here's, in red, that additional issue of distance learning, which is another thing that you might hypothetically need to add. It was hypothetical for most of us up through end of February. Now, obviously, for all of us, it's not so hypothetical. 

OK, so here's Appendix C. I probably should have shown a screenshot of this. I'm not sure if I have one or not. So sorry if I don't. But if you download that statewide assessment policy on that link I provided a few slides ago, you'll get that statewide assessment policy template. It's been at the same location since 2005. Yes, we do update it every year. But by and large, year to year, the similarities are much greater than the differences. 

But as part of that policy, again, the part of this that nobody ever knew about, including CDE and CASAS, you might say, we've been doing a lot of dusting here the last couple months. Since 2008-9, at least, there's been a section of this statewide policy called Appendix C, distance learning, that provides our statewide policy related to distance learning. 

Nobody realizes we had this. But the bottom line is, yes, we did. We already had very specific statewide policies on distance learning. The fact is, however, that nobody realizes they existed yet. 

So just to get into this a little bit, because we're all worried about distance learning, Appendix C is the part of the statewide policy that addresses distance learning issues. It really and truly does address these issues. And then it has three-- or four sections, not three, one, two, three, four bullets, not three. Definition of distance learning, testing, curricula, instructional hours. 

So definition of distance learning is what we've said already. It talks about the 50% rule. Now in this year's policy, you're going to want to describe how you've transitioned to distance learning. Again, we're talking you really need to make it more like an Appendix D for that. But the definition of distance learners is the same. 

And then for testing, again, this is kind of wavering. This is where it's changed a little bit. But it's talking about distance learning must complete the same requirements as everybody else. You specify the process for how you do it. Now we're going to have that addendum. For now, we're probably going to leave this alone in Appendix C. 

But we will have that addendum. Again, at the state level, they're coming up with an addendum for this for '19-'20. At the local level, you'll come up with an addendum where you're specifying specifically what you're doing. Are you doing pre/post? Are you doing EL Civics, et cetera, et cetera? 

For '19-'20, I think what we're saying is, don't worry about writing a book. Just write what you need to do at a bare minimum to move forward with remote testing. If you've thought further than this, yes, that's a really good idea. Yes, no, I don't know what I'm talking about. But yes, I really see this as being something we'll need to do every year from this point forward for all the ridiculously obvious reasons. 

Curricula, again, this is where you're specifying by curricula. Are you using educational software, specific videos, specific websites, specific platforms? Here's where you're specifying which technology you're using successfully in order to implement distance learning. 

And then here's where we're talking about hours. This is the pointed area where we've really been talking about it. We've been spending most of the time talking about this, I really think, because this is where most of the questions are. The basics on testing and curricula, especially when we didn't really have remote testing, pretty straightforward, really not generating that many questions. But the part on hours has always been the trickiest part. So that's what we've talked about the most, because it's the most confusing. 

But again, here's the quote where, again, this is old, but it's still basically the policy. The CDE promotes a rigorous, outcome-based learner mastery model. I'll just say, here's why I always say you could use any model, but learner mastery is the one we've always recommended in California, if nothing else, because Appendix C of the statewide policy says so. And it's said so since at least 2008. 

And so again, with designated levels for program completion verified by a process that may include teacher or external certification, I'll just say, that's why I say the test, or the percentage they get correct, if you wanted to use that, great. But is it required? Sorry, I'm going out of my way to say no. Because it's so explicit in the policy that it may include that. It definitely doesn't require that. Quite frankly, it goes out of their way to make that an option on the part of the agency. 

OK, and then for hours again, it's describing methods for delivering distance learning, models used to record hours, designated a group from the appropriating area, that's that team discussion for learner mastery we've been describing. Any additional activities included, so that's been a big one where A, maybe the idea is, mostly, we're going into the software and completing modules. 

But we know that there's a little bit of instruction that's taking place outside of that. We really want to record that. We really want to give the student credit for that. Again, that's fine, but you do need to identify what that is in your policy. And then you can use multiple methods. The policy has always said that's fine. Again, that's fine if you combine them, use them separately, whatever you think is best. All of those answers are yes, yes, yes, yes, yes, yes, yes, yes, yes as long as you dutifully document them in your local policy. 

OK, then again here in '19-'20, here's a little bit of need to do here. Again, for '19-'20, we'll have that addendum. So to be clear, here's where I'm giving you TMI. I'm telling you policies that don't exist yet. We've had Appendix C for many, many years. Appendix C is what we already-- what we're going to do is we're going to make an addendum to Appendix C that basically has a paragraph or so stating the obvious that, so anyway, for '19-'20, we're going to include that extra information. And then there'll be an Appendix D that's specifically for '19-'20 and specifically for how you're undergoing remote testing. 

If I were a betting man, I would say it will probably become a permanent part of everybody's policy and everybody's statewide policy that will need to have that specific issue related to remote testing. For the sake of argument, I'm going to call that Appendix D. So again, we've got Appendix C that addresses all those basics of how we're delivering distance learning. And then Appendix D is the more specific component that addresses remote testing in particular. OK, next slide, please. 

So in the addendum, we're going to-- here's the-- this is straight from the CDE policy. Incorporate a written procedure on remote testing, security, how do you identify the student, how are you pre-screening the student to verify identity, again, lots of questions came up yesterday. It'll be a little bit different at every agency. What are your technology tools, technology platforms, and so on? What publisher guidance is there for the platforms? 

And then, in particular, this is-- we kind of addressed this yesterday. I think all of this should be clear if you were in the trainings yesterday. But also, you'll need to address what happens if somebody undergoes technical difficulty. There needs to be a plan where the student and the staff both have a step they know they can do in the event of technical difficulty. 

If you haven't looked at that, that's the part you really need to look at. Because the state is very concerned about that. They're definitely going to have to add that as a required step for any distance learning addendum related to the local assessment policy. 

OK, and then the remote addendum should address these issues. Again, this is kind of the same thing, security, technology, tools, orientation. Again, plan to respond to technical issues. Next slide. 

Sorry, that was unnecessary. So security and integrity, process for registering computers, testing and maintaining a secure environment, ensuring the platforms you're using are secure, pre-screening steps for how you're verifying student identity, what IDs you're checking, how are you putting the camera on the student? Are you taking pictures? Are you just putting the video camera on it? What are you putting in place to make sure you know that student is the correct student? Certification or trainings to administer, all that is is put listing, just like you would in the regular policy, which staff have completed that proctor training, which staff have you designated for doing remote testing? And then orientation, what's the process for orienting your students on how to do remote test delivery? 

You know, obviously, you can't just throw the student in there and expect him or her to do it. You've got to have a way where you're working with the student a little bit so the student, like you, knows what to expect. The student, like you, is comfortable in that situation. You'll need to orientate each student a little bit to expect that to happen in any way, shape, or form. Next slide. 

Technology, so again, here you're just specifying which specific technology platforms you're using. We're a little bit limited now. But we're going to be expanded, even with our limited list. Everybody is going to be a little bit different probably. And then how to respond to technical issues, I'll just repeat again, I feel like all of these are ones we've talked about a lot. But this last one I pause. I'm less sure about this last one, that we haven't talked a lot about this. 

So I'll go out of my way to point that out, to say if you haven't thought about what it is you're going to do in the event of technical difficulty, the state is saying, yeah, you absolutely need to start thinking about it like right now. Before you implement testing, yes, you do need to think about that and outline what your steps are. If you get any sort of technical problems, what are you going to do in response to that? What is the student going to do in response to that? And make sure, before any staff or student sets foot in a remote testing doorway, that you already know what the procedure is for any sort of technical difficulty. Next slide. 

And then general guidance, again, models, how are you going to employ? I'll just say, this looks the same, written a little bit differently. This is the more nice to do that we included when we were really looking at this training being more of a nice to do related to all those issues we presented in Part I. I left it here because these are good steps. But they're not the official state guidelines, so probably a little less important. 

But here is another series of things you should consider. It has nothing to do with remote testing. But it does relate to all those issues we talked about in Part I related to class setup, class creation, proxy hours, distance learning hours models, and so on, all the different things we talked about in Part I. Still really good idea to the maximum extent possible to put that in your policy. Next slide. 

I think we're here moving to the transition. I think that's the last one. Next slide, please. Yes, good, all right. 

So it's 2:01, and we're here into the examples. If you can just slowly, if you don't mind, going slide by slide, I want to show as kind of a proof in the pudding that these do exist. But here's a clock time model example. Just stop here for a minute. So we've got example one, example two. So we've got, I think, three different clock time model examples. I won't read it. We're about at the end. But again, we provided very specific word for word examples that you can use, and lift, and copy and paste into your local assessment policy. So we have not one, not two, but, next slide, please-- oh, well, I thought we had a third one for clock time. Oh well, my bad. 

So we've got two clock time model examples and one teacher verification examples. I think if you go to the next slide, we've got not one, not two, but three examples on the learner mastery example. Yeah, there are. I was actually right. 

So yeah, let's start with example one, because I made a big fuss about it. So again, two clock time examples, one teacher verification. Again, this is a little bit reading from the slide. So I've got to say, if we didn't read from the slide, I wouldn't really have seen it as that big a deal. Because really, you're going to have a lot more success reading and doing this on your own than just reading it off or whatever. 

But again, here's a clock time model. We're just saying everybody's in the same ESL class. They attend for an hour on Tuesday and Thursday. This is one of those where an hour via distance learning is exactly the same as an hour in the regular classroom, same bat time, different bat channel, I guess you might say. But same teacher and same students, same bat time anyway. So that hour that they're spending on Tuesday morning, same as the hour they spent on Tuesday morning when they went to your school. They're just doing it on Zoom instead of in person. So everything's basically done the same. 

So I'm still not going to read word for word in the example. Sorry, not going to do that, but here's your word for word example on how you can type. You all can read, so no, I won't read it for you. You can all read word for word and copy and paste it into your local assessment policy. I've just left it blank here where you can populate it with Google Chat, or Zoom, or Canvas, or whatever it is you're using as the software platform to gather all your students. Next slide. 

So here's a second example. OK, let's back up again. We've got time. So let's look at example number two. There we go. So they're using software that records the correct number of hours for each student, so totally different scenario, but another example where we might recommend clock time. So the most obvious is that one in example one where you're just having students meet the Zoom or whatever for the same hour as they would if they met in person. 

But here's a second situation where you'd use clock time. So it's by model. So the first one is the synchronous model. Here's an asynchronous where you've got the fancy software that records hours. That would be another one where you might use clock time. Because, again, the software records the amount of time the student has put in the software. So this would be another one where we'd use clock time model where the software is automatically recording the time the student spends on the software recording it in that software program. So again, the easy way is to use clock time for this example where you're just taking what the software gives you and recording what the software gives you and directly into TE. Next slide, please. 

Here's teacher verification. Where here, for teacher verification, we're suggesting 20 students are using software that does not record hours and does not include any sort of additional exercises at the end of each module to confirm understanding. So this might be a scenario where you want to use that teacher verification. 

Again, this is the one where through teacher observation you're using-- you're putting it all on the teacher to decide. Student x gets this amount of time. Student y gets a different amount of time. Student z gets yet another amount of time-- where the teacher is dutifully and diligently observing each student in a distance learning environment. So that's kind of why we're skeptical of Scotty on why we don't think this is really going to work that well. But again, if you know the teacher is able to view all the students, and observe all the students, and come up with good answers for each student in terms of hours, then go ahead, knock yourself out, use the teacher verification model. And this would be the example for that, where based on teacher observation, she or he is recording the appropriate number of hours for each student. Again, I'm not going to read the example. But here you go below that in italics. 

Learner mastery number one, here again, 20 students using software. It does not record hours. But it does include some of those exercises for understanding at the end of each module. So again, in this case, we're saying, once again, you're going to need to use learner mastery. Because it's not recording hours. 

So yes, you will need to meet as a team and identify the number of hours. But no, you won't necessarily need to create any kind of activity for them to do. Because the software is dutifully doing that for you. So here in your assessment policy, you are coming up, or you did come up with the method for recording hours. So you can apply the same method for each student equally. But you allow the software publisher to provide those exercises at the end of the lesson. And your policy is just pointing your agency, pointing the teachers and the students to those module exercises and suggesting if the student successfully gets through the exercises that the publisher provided for you, that's sufficient for the student demonstrating learner mastery, and moving on to the next module, and getting credit for those hours in that module the student just completed. Next slide. 

Throwing a monkey wrench, here's another example using learner mastery. Example number two, where 20 students are enrolled in an online ESL class for an hour Tuesday, Thursday via Zoom-- again, totally different example. Most students attend as scheduled. But a few students who are unable to attend access the recorded session. So in this example, we're using clock time for the majority of the students. But a few students couldn't make it Tuesday at 10:00 AM. So they're viewing the recording. 

For those that made it Tuesday at 10:00 AM, we're basically going to use clock time. But we had a few students that didn't make it Tuesday at 10:00 AM. So for those students, we're going to need to use learner mastery. So we add in our policy the classes that meet two hours a week use the clock time. For students that can't make it at the advertised time, we give them the option of accessing the recording. 

If they successfully do that, we provide a few questions they need to answer at the end. If they get those questions right, we'll credit them with the x hours of instruction. So again, you simply specify what we want to do for those that make it at the right time. We have different specifications for those that don't make it at the right time. 

I'll just add, in Part I, we went way out of our way to say it's up to you as an agency to decide whether to allow that. Some of you say, yeah, we really need to allow it or we'll never make it. Our students will never make it. Others have said, have noted, that can be kind of a slippery slope to allow that. So if you've decided that's too much of a slippery slope, go ahead, knock yourself out, and disallow it. Next slide, please. 

So here's one more learner mastery example. This example, they're in independent study classes. And they're watching assigned YouTube videos as part of the required instruction. So in this case, again, we're detailing how they do that. 

So here's one where we're saying, yeah, there's not really any kind of confirmation for understanding. There's no way we can really use the YouTube videos at all and know whether they've watched them or not. So this is one where we really need to do a little extra effort on the agency's part. And we need to specify exactly which videos they're completing. And then once they complete the videos, we give them a quiz. And they need to pass the quiz with a certain percentage. Because there's no other way we have any way of knowing whether they did or did not watch said videos. Next slide, please. 

And then, finally, a lot of you have brought up, can we use mixed delivery methods? The answer is yes. But yes, you do need to specify if you've used mixed methods. It does make it a little more complicated, but just a little bit. So here's an example using mixed methods where we're saying they're using software. It doesn't record hours, doesn't include exercises. It's real basic, bare bones. 

So we say some participate in a live session where we're all working on the software as one, big, happy family. But similar to that other example, if you aren't able to make it at the right time, so they do it more on her own or his own, so here we're saying for some students that make the live session, we're going to just give them clock time. Hey, we don't care if they did a lot of work or a little work. They sat in. And they were there for the whole hour while the teacher was training. So they get an hour of instruction, just like they would for regular. 

But if they miss it, then we're going to apply learner mastery. Because the teacher isn't there. And we can't really verify what the student is or is not doing. So if they don't make it at that time, we're going to apply learner mastery. So here's an example where, for the same students and same class, we're employing two different models. And we're specifying which circumstances use one model versus the other set of circumstances that use the other model. Next slide, please. 

And then here's another mixed delivery method. We're using a software for up to six hours a week. They're logging in. There are some times when we're going to use teacher verification. So same sort of thing as the other example where some students are-- they're online at the same time as the teacher. So the teacher is able to observe and evaluate each student. Other times, students might be online at a different time from the teacher. So thereby, of course, the teacher is unable to make those evaluations. 

So in this example, we're saying up to the point that the teacher is able, the teacher is going to rely on teacher verification to determine this. But when the teacher is not there, then we have this learner mastery process in place that we can still record hours for students that are unable to attend the online sessions at the exact same time as the teacher. Next slide, please. 

OK, and then here's kind of not a specific model. This is the last one. But this is kind of putting it together, kind of the overriding learner mastery, where if you want to do it for a whole software module, which my understanding, a lot of you are doing, where you've got a CD with like 12 different modules, and the students are going through all 12 throughout the semester or whatever, here's another way where you can do it where you can start with putting the name of the software up at the top, what does it address, list the modules, give the titles and the projected amount of time for each one of those modules. 

So here we've got module one through four with a targeted amount of time for each of those modules. Again, just follow, listing all 12 modules. Record the number of hours to each module. Maybe it's what the publisher tells you. Maybe it's what you determine through learner mastery, whatever. But put the hours there. And then again, when students complete each module, you have that little plan that you already put in place. So whenever the student finishes the corresponding module, you simply record that amount of time for the student depending on which module they complete. And that's it. 

JANICE FERA: I just wanted to add that the Q&A, once we get a chance to go through it and shake out any of the weirdness, we will be posting it on otan.us alongside of the PowerPoint. 

ANTHONY BURIK: OK, Janice, thank you so much for that reminder.