[MUSIC PLAYING] NARRATOR: OTAN, Outreach and Technical Assistance Network. CHRISTINA HYATT: We're from Corona Norco Adult School. And today, we're going to share our journey into developing a successful remote program at our school for testing, and CASAS, and civics. So I'm Christina Hyatt and I teach adult basic education math. And under the direction of our Dean of Students, I run the remote testing for the CASAS schools for adults secondary education students. MARCI ENGLAND: Hi, everyone. I'm Marci England. And I am like the DJ today. So I just need to double check. Can you see-- I'm sharing my screen, right? I know it's like nobody likes hearing that at my screen. But I wish there was some way we could just ask Zoom-- am I sharing it, without asking everyone. OK, and then I'm also going to manage the chat. And I will watch the questions and just make sure that this presentation continues to flow. So that's my role, a little bit. But also, I'm here to share the EL Civics side of remote assessments. I'm Marci England. Let's see, I'm the ESL program coordinator at Corona Norco Adult School. I'm excited to talk to you about remote testing and what we've been doing. So based on that, I'm going to go to our next slide. Maybe, there we go. All right. So it's always fun to start a presentation with some fun facts. So did you know that in 2020, puzzles sales increased by 300% to 370% over the same time in 2019? Ravensburger, which is like the gold standard of puzzles, it's the German puzzle company, they were selling puzzles at a rate of 20 puzzles per minute. Penny Gold, a pediatric neurologist said, when you are working on a puzzle, you can't worry about anything else. You can't let other things intrude, other thoughts intrude. It's all about meditation for me. Another puzzle enthusiast said-- it's my chill time after working all day. It's something I look forward to. So the puzzle fad of 2020 is something that I think we can all relate to in some way, shape, or form. And I know Christina, she completed well over 20 puzzles last year, you guys. And she's even sent me pictures of her completed puzzles. And I have to say that I have contributed to a few or several assists, although I cannot brag and say I completed any. But I am the one that walks by and says-- maybe this piece would go really well right there. So when Christina and I were talking about implementing remote testing, we really thought that this had a good connection to solving a puzzle. And we love an analogy in how looking at things from a different perspective helps you connect pieces. And so we're really going to hang onto this puzzle analogy and talk about how remote testing has been an enjoyable experience of focus and meditate, maybe not as far as meditation, but it's been an enjoyable process as we've solved problems and created a better picture. So today, we're going to discuss the critical first steps we took to implementation. We're going to talk about administration of the assessment. We'll also discuss ways that we've worked to maximize learner outcomes, and different ways that we measure learner outcomes, and measure our successes. We're looking at things differently than just testing scores. We're looking at a lot of other positive things. We're also going to give you some troubleshooting ideas. And we're going to share our pitfalls. We will tell you the things that you can do to avoid, like, we've been through some bumps and so maybe if we tell you where they are, you will not hit them as well. So the format, Christina is going to guide us through remote CASAS testing first. And then we'll have Q&A with Christina. And then I'll guide you through remote Civics testing and we'll have Q&A for me or in general. So that's what it's going to look like today. So just to give you a bit of background on our school. Let me move this out of the way. OK, so to give you a bit of background on our school, just so you know where we're coming from. In 2019-2020, we served over about 3,600 students. 28 had significant literacy gains based on CASAS testing. So we were successful and we had a robust CASAS testing program. Our programs include ABE, GED, high school diploma, ESL, we had 1,300 students, CTE, concurrently enrolled, and fee-based classes. Our technology accessibility, to start off with, in 2019-2020, we didn't have a loaning program, we just had laptops on school sites available for students to use, but not really a program where students took laptops home. Thankfully, due to grant funding and our consortium, we are now currently operating under a loaner program where we have enough laptops and Wi-Fi to provide students, anyone that needs it with a laptop and Wi-Fi. So understand that that's where we're coming from. Our etesting history, in 2019-2020, only adult secondary education was doing etesting. ESL and CTE did paper and pencil for Civics and CASAS. All right, Christina. CHRISTINA HYATT: So when you're getting ready to do a puzzle, everybody has their first steps. The first things they do. Like for example, do you dump all the pieces and sort them first? Maybe you sort them by shape or by color. Maybe you're one of the people that turns them all face up. I personally like to start with the frame. Everybody's got a different way of getting started. But there's some critical first steps to getting going. And so likewise with our remote testing puzzle, we looked at what were these first critical steps, and we called them the corners. The most important parts of the puzzle is finding those corners and building around them. So the first corner we looked at was the who. Who is this going to affect when we test in a remote situation? Who needs to be trained? How do we-- who do we need to get on board with this program? For us, we needed to train our staff and increase their understanding of why we give the CASAS test even in the middle of a pandemic. Right? So we worked to shift the mindset from pay points to student growth. Our CASAS test trigger digital badges. They trigger promotions. And we use them for recognition. In our meetings and in our PLCs, we shared about the importance of CASAS test for showing the student growth. And once teachers began to understand this, they became our partners and making sure every student got tested. And I do remember, when I first heard about remote testing, I think it was back in May, I thought-- you want me to ask students to do what? And what are we offering them in return for this? I needed a mindset shift too. So by shifting my mindset to student growth, I don't even think about it now. Testing from home, that's just what we do. The second corner piece of our puzzle for testing was the training. We trained staff on the testing protocol. We got them the necessary certifications. Both EL Civics and CASAS have requirements to implement remote testing. So we worked closely with CASAS staff to make sure we were within their framework. CASAS has videos, literature, trainings, and helps to get your remote testing program going. My favorite thing is they have a remote testing checklist. And who doesn't love a good checklist, right? We also train students. Some of our students have never used Zoom. Some of them needed to learn how to use a computer. We wanted to make sure that the students felt confident, that they had the necessary technology skills to take the test in a remote environment. A third corner piece was that technology. Getting the right type of device has been very challenging. And in fact, one of our pitfalls at times. But luckily, as Marci mentioned before, we were able to get Chromebooks and Wi-Fi hotspots into the hands of students who needed them. And then we also worked with the district IT department to make sure that those Chromebooks could work in kiosk mode so the students could take their tests online. The students need the right type of technology. And they need to know how to use it. And they need to feel confident when they go into a testing situation that the technology is not going to be what trips them up. So we continue to work with the staff to get them trained on helping the students on making the students feel comfortable with those pieces. And another important part of this corner of technology is making sure that the staff has the right technology and the right abilities. We all were nudism a year ago. And so learning what Zoom can and can't do, learning to make Zoom work for you when you're a proctor was really important for me. And I'll talk about some of those things later. Another thing is when our campus went completely remote and we were only testing students from home, I needed enough computers and enough bandwidth to be able to do that from home. So these are all things that we had to consider as our critical first steps in developing our program. And then of course, the fourth piece is the support system. So when I do puzzles with my family, I have a sister who she likes to start with the water. It kind of blows my mind. Like, why? But she always starts with the water. And I have a brother who likes to look at the shape instead of the color. He's a little color blind. So it makes more sense to look for the piece with that little arm or something. Whereas my brother-in-law, he cannot stop looking for that missing piece in an almost finished section. That bothers him to no end. And for me, I have to start with the structure, the house, or the boat, whatever it be. So when working on your system, you want to look for a team that has different strengths and different points of view, much like my family does when we approach the puzzle. Having that strong support system helped us work our way through a lot of the wrinkles that came our way. As questions came up, we had people we could go to for answers. Sometimes the pieces weren't fitting the way we thought they should, but getting a new perspective, or having somebody else look at it, or looking at it from a different angle, we're always-- that we were able to come up with solutions. So consider who's going to be on your team. Would it be your CASAS rep, your tops coordinator, the principal, your consortium, the experienced teachers on the campus, the techie people on campus, the district IT department. And for us, it was every single one of them. Another important thing to consider is the cost. So I kind of think of the cost as like the frame of the puzzle. It's going to determine the size of your testing program. And remote testing can be costly. I'm not going to sugarcoat that. But we looked at ways that we could minimize the cost and maximize our resources. For example, consider what do you have. And for us, we had lots of computers sitting in remote locations that we were able to bring back and implement in our testing program. You can also look at who do you have on campus. We had paraeducators and substitute teachers who were willing to step up and be part of the team. Look for the emergent leaders in your campus, those who can, and will, and are able to make that strong team to complete the puzzle. And last of all, look at how it could be better. It's easy to just look at a system. And go-- that's it. But we really need to constantly be thinking of improving the process and perfecting it as a way of reducing the cost and maximizing what you have available. And so for me, recently-- I started looking at I realized I was spending a lot more time on communication than I normally do. And I had to step back and think about not only what had changed so I could reverse that change, but go even further and cut into what was being done before. How can I make it so I communicate less frequently and yet still get the message out to the people who need it? And Marci, you want to talk about equity? And you're on mute. Yeah, yep. MARCI ENGLAND: What's a Zoom meeting without someone talking on mute? OK. Glad I could do that for us all. So at our school, equity means everyone has access to learning. We work to ensure that the most vulnerable students are supported. This means bringing students on campus for testing when we can, but also offering remote testing for students that are not comfortable or cannot come to campus. It means creating systems that support all of our learners. For us, it means getting our Chromebook books to work and having staff to support technology problems. It means troubleshooting different technology types. Equity is listening to teachers as they advocate for individual students and preparing for student needs. Specifically with a remote assessments, equity means access, accommodation, and flexibility within the testing parameters. MODERATOR: Oh, Christina, you're on mute. CHRISTINA HYATT: It's my turn. Thank you. MODERATOR: I checked some problems. CHRISTINA HYATT: Well, I wanted to be like Marci. She is my bestie. OK, so I'm going to start by going over the CASAS goals in life and work program and how that was developed. And I'll stop for questions at the end of that. And then Marci will jump in and share what she did with Civics. So as we developed the plan, it really-- I think about it is now getting all the pieces to fit together. You've got this 500,000 piece puzzle and now we've got to figure out how the pieces are going to fit. And so here's this key questions that we looked at. What needs to be done? When does it need to be done? Who needs to know about it? Who's going to do it? I honestly didn't think I was going to be the one doing it, but here I am. Because that decision was made somewhere along the line. And what training, technology, skills are required? And what are the CASAS policies that need to be followed. And right here on this slide, for those of you who want later, I'm going to be providing a template for developing your own plan. So that link is there when you want it later. MARCI ENGLAND: And everything-- we have our slides in a Google slide presentation. No, in a Google folder in Google Slides. So we'll be sharing the folder with you that has access. So any link you see in our presentation, any template, whatever, it's all inside that folder. And I'll put it in the chat here in just a little bit. CHRISTINA HYATT: Perfect. All right, so on the next slide, I'm going to go over what our school did, or what we do for our testing program. I'm not give you every single detail because we'd be here for four hours and that would be boring. But I'm going to give you some of the highlights. So for me, one week before, I start e-mailing students to confirm that they have the required technology and they know how to use Zoom. In this email, I even includes some Zoom tutorials. Because the number of times you've had a student come on and this is their first time using Zoom, and they don't know how to unmute, or they don't know how to mute, were all things that we needed to deal with. So the trickiest part of the process is getting students to show up with the right kind of technology. And so one of the things I do to combat that is first I send them an email. And I tell them-- here's the kind of technology you need. But then I created a survey because it's not enough to send an email and expect them to read it. They're busy. It's more than two lines long. And so Marci, if you'll show the survey. I asked them instead of just reading, I asked them to do something, to respond. This way I know that they read and they understand. And it's a very short survey. It doesn't take them very long. That I want to know exactly what kind of device they're bringing to the testing session. And I want to know if they understand that they have to be able to have a webcam and a microphone. I want to make sure they understand the amount of time it's going to take and the testing environment that they need to be in. So this survey has been fabulous. Because one, I now know that if they're showing up with an iPad, and I can send them the tutorial so that they can download the app and get ready. So the night they come, they're like-- yep, I've got it. It's all on screen. And we're not dealing with-- oh, let me send you that. Let me walk you through the process. And then the other thing, Marci, if you'll go back. I got lost. But give me just a minute. OK. So when they-- oh, and then the other thing is there's no surprises when they show up. Whatever technology they have, we're kind of expecting it and we're prepared for it. Because different kinds of technology require different conditions. So then the day before the test, I send that Zoom link. I don't send it in the first email. I'm waiting for them to respond to my email. And there's a reason because I need them to think I have to do something before I get this invitation. When I send the Zoom link, I assign them a time, and I stagger those times. The first time I did it, I had all five students show up at the same time. And it was not a good experience. Because I couldn't help five students on board at the same time, especially when each one had a different technology ability and a different technology platform. So now, we give 10 to 15 minutes so we can work with the student one on one and bring them in individually. So then on the day of the test, the first thing I do, is I open a breakout room. And I have one room for each student that's going to be testing. Immediately after I open those breakout rooms, and this is way before I expect the students to come online, I start bringing on computers in my office. And I bring on one computer for each breakout room I intend on running. So that I can see every breakout room all at once. I'm not having to scroll through and look at each breakout room individually. I can see all students. And if one student is struggling with something or needs help, I can hear and see them immediately. One of the things, though, that we run into, and I'm sure all of you can relate to this, whether you're running a program or not, is the internet. Does your internet over go out? It's just me? So a couple things that I have a suggestion, these are a couple tips I have. When you do the breakout rooms, there's a setting that says automatically move participants into their breakout room. And there's another setting that says manually at the students. So I manually add the students and the devices that are in my possession into the breakout room. But if the internet goes down, which it often does, it's lovely because as soon as it comes back up, Zoom knows to automatically move them back into the breakout room. And I'm not scrambling or having to move everyone back where they were. Zoom automatically just puts them back where they were. And a lot of times students don't even realize it ever happened. And then my second tip that revolves around the internet is because I'm using sometimes six computers to do this, I need more bandwidth. And one of the things I now do is I turn on a hotspot for my main computer, the one that I'm the host on. And that one runs on one Wi-Fi connection, and then the rest of them will run on the second one. So I just use my phone usually, I'm fortunate enough to have enough hotspot on my phone that I can do that. And it's made all the difference. Because if one drops the connection, I still have the other one up and running. So then the last couple of things or tips is after the test, sometimes students have a lot of questions. And maybe you're in a hurry, or maybe you have a couple other students, but make sure you give them the time and answer their questions and provide them the resources. Direct them where they're going next. I made this mistake early on. I said-- great, thank you so much for taking the test. And we'll get back to you. And that was not right. There was a process in place that we needed to make sure students knew what to do next and how to do it. And usually, in that end of the conversation after the test, I will tell them, three times, the next step. And later on, I'll show you another thing we've done to make that even smoother. Marci, if you'll go two slide-- two slide up, I think. Sorry. So this is my workflow. Please don't look at this and think-- oh, my goodness. Right? I'm kind of OCD when I write my steps, when I work on stuff. And I write things like turn on the computer in here. It's pretty intensive. But I wrote this down because I'm training my staff, or my team, I should say. My team to do this without me. I don't want this system to be my system. I want it to be the school system. And by writing it down, also I could look at the places where I can perfect it or make it better. So I'm not going to give you a copy of this. Some of you are probably saying-- but why, but why? And the reason, of course, is because this was our puzzle. This is what works for our schools. So instead, I'm going to help you. You guys have your own puzzle, your own way you're going to need to look at things. And I'm going to give you a document that's going to help you develop those skills. You can see, I've got all of the timing set up in here. And I've added even more to it than this picture shows. I've given you some of the tips, some of the resources we're using. They may not work for your school. We have Zoom in Spanish because we have a lot of Spanish speakers. But maybe you need Zoom in another language. But I've provided some questions or some things for you to consider as you go through the process. Who needs to be involved? Where is it going to happen? How is it going to happen? How is the student going to know about these things? How do we make sure we're staying within the parameters provided by CASAS? So these are all-- this tool is designed as questions to get you talking and get you thinking about how to get your program going. All right, and next slide, Marci. Oh, yeah, you're there. I'm not. Thank you. OK. All right, communication. So for me, communication is like assembling the vast blue sky of a puzzle. It's difficult. And you usually spend a lot of time on it. And you're constantly trying different pieces that are all the same color. This is where that brother who's colorblind comes in really handy. So communication is something you have to constantly work at. And it's where I spend most of my time when it comes to the testing process. You just need to make sure that whatever your message is, that it's clear, it's concise, it's consistent, it's continuous, and cohesive. We recently had a student that went through the testing process and I still chuckle to this day because Marci calls her a good little communicator, which is exactly what she was. When she got your email, then you got three to four emails a day, detailing exactly where she was, and what she needed to know, and how you were going to be able to help her. And it's easy to think about-- ugh, that student. But I'm actually really grateful for students like that because they allow me to look at what's missing. What do we need to relook at, right? How can we deliver the message even better? Because while this student is communicating her needs, there were lots of students who probably weren't going to have the same questions. Oh, go back, one more time. I was just going to share my quick tip. So a couple more things. So the quick tip. One of the things we run into is we thought we were running a remote testing program, and our first couple go rounds, we had tons of students show up on campus. And we realized we couldn't use the phrase from home, stay at home, do not come to campus, and thus. So we looked-- we went back and it turns out our message for students were signing up was conflicting with the message we were trying to give them. So we were able to look at fixing that message, making sure the message was consistent throughout anything that the student was seeing. And making sure that every email I sent over and over again told them-- you will connect online. You will not come to campus. We will see you on Zoom. And so my quick tip is make sure when you're communicating to students that they know exactly how this process is going to go. Communication and miscommunication is one of the biggest opportunities for pitfalls. So when you're developing a good remote testing system, make sure you look carefully at what the message is, who needs to get the message, and how it's going to be delivered. So one of the ways we did that is by creating a Google document or a Google site, excuse me. And we outline the steps that students need for getting enrolled. It has the necessary steps, the forms, the tutorials, the contact information for each step of the process. And right there on step 6 is our CASAS assessment team. Just this little thing of telling them-- hey, you're going to get a message from Christina Hyatt, has really made a difference in the number of calls that go to the office. Because now, instead of the student calling and saying-- I never heard from the administrator. They know first to search for an email from Christina Hyatt. Inevitably, it is there. All right. So the learner outcomes. So we talked about earlier that we weren't focused on pay points. We weren't focused on how this was going to look for our school. We were focused on how we can show the student growth. But there was a happy side note that we saw the pay points. Even though we saw fewer enrollments this year, we saw a higher percentage of students with paired scores. We saw higher percentages of students leveling up. And I also love that now, with a remote system in place, we can capture students who weren't available to come to campus. Even before the pandemic, there were always those students who-- oh, I'm in Mexico. Well, guess what? We tested somebody in Mexico. We got that post-test because of this remote system. And so it just allows the flexibility to test students where they're at and meet them if they don't feel like they can come to campus. We also are giving students digital literacy skills. For many students, these tests online are maybe the first test they've taken online. But now, HiSETs being administered, online, GEDs being online, all from home. And now, they've got this opportunity to practice. As they enter college or into a job environment, they're going to need to know how to navigate some of these online experiences. And by having this program, we're giving them the first opportunity to try it. And when they come around the second time, it's a lot easier. And so we can see that they are increasing in those skills. So the process was slow. We started right when CASAS allowed us to begin back in May. But we weren't that great at it back in May. And we're just getting better, and better, and better the more we practice. It's not that it's easy. That communicating with students, or that moment when you're trying to onboard a student with technology, it's still nerve racking. My team and I, we always talk about that being the most nerve wracking part of the process. But each time we put more pieces of the puzzle in, it gets easier and easier to put the puzzle together. So I am going to pause right here and answer some questions. MARCI ENGLAND: So Christina, we have a question from Peg, she says-- how do you make sure the students read and understand the Google site or other information about your program/testing? CHRISTINA HYATT: So the first thing a student does when they're getting enrolled is they have an orientation with our Dean of Students. And he takes some time to go over the steps and the process, and he also gives them the website so that they have it. And everybody on campus knows, if a student's lost in the process, here's the site to give them. And it became a one stop shop of-- if a student calls and says-- hey, I haven't heard from the testing coordinator. They could go-- well, it looks like you haven't done step five. Or you still need to do step four. And so it comes from him, but at each step in the process, the office knows about it, the Dean of Students knows about it, the testing team knows about it. And when we finish testing with the student, we don't just tell them, we show them. OK, here's where you're at in the process. Have you completed these five steps? And there's just a couple more steps after they meet with us. And we tell them, at least three times in the phone-- in the Zoom session at the end, and then we show them. MARCI ENGLAND: Thank you. OK. Oh, good question. Kathleen asks-- is there any translation? CHRISTINA HYATT: Can you be more specific Kathleen? MARCI ENGLAND: I believe, I'm thinking a translation on the Google site. CHRISTINA HYATT: Oh, I think that Google does translate. Right? Am I wrong about that? MARCI ENGLAND: I know for sure the forms, you have the translation option on the forms. And then Google translation really is just if the student is tech savvy enough to have that set up in their Chrome browser. But that's a good point, is that we could have a little mini tutorial, like click here to translate this site. CHRISTINA HYATT: Yeah, that is great. See? Another piece of the puzzle in. Thank you. MARCI ENGLAND: Diana says-- how did you find the students who need tech equipment? And then she says maybe Chrome extension for the translation. I'm sure she's talking about. So how do you find the students who need tech equipment? CHRISTINA HYATT: So usually in the orientation, our Dean of Students does go over-- if you're going to be testing from home, you're going to need some technology. But usually, it's that first email I send a week out. Here's what you need. Here's the options you have for testing. And if they don't have it, there's an other option on that quiz. And I did provide you guys a link to that quiz as well, that you can duplicate it make it your own if you wanted. And so when that other comes back, then we work with our team to either check out the device then, or maybe find another device. And we were lucky enough that not just our Chromebooks, but every Chromebook issued by the district, can now run the eTest. And it's just built right into the Chromebooks. So a lot of times that solution is-- hey, do you have a K-12 student in your house with a Chromebook? And then we're able to do that. MARCI ENGLAND: So that means, Christina, that you would use the K-12 device before they're issued a device through our school, right? CHRISTINA HYATT: Yes. And we've also enrolled students, issued a device, and then tested after as well. MARCI ENGLAND: OK, good. So do CASAS test students with different devices in the same session? CHRISTINA HYATT: So I know that CASAS, in their training, they recommended not doing that. And so I tried it the first time. But when you're only testing five students in a remote session, and this was early on, I just felt like it was too hard to find five students who all had a laptop, or five students. And it's actually easier for me if I don't have five Macbooks at the same time. Because that's a whole new dimension of working through the testing. So for me, I give them the options. My staff and I are well trained on what the options are and what they can do, and how to get the student on board with those options. And that's all part of our plan. I have all the devices written out and the specific steps that need to be performed for any particular device. MARCI ENGLAND: Let me show that really quick. So that's here on the right. Right, Christina? Where you have if it's a Windows, a Mac, a Chromebook, an iPad, and what to do there. OK, so we have-- Kathleen says-- with all the materials with remote testing steps, can we use native language up to begin the test? I think that probably means like to explain the test, like up to that point. CHRISTINA HYATT: I'm going to refer you back to your CASAS rep on that question. Because I don't want to say what's wrong. But my understanding, I could be wrong, was that yes, you can right up until the beginning of the test. But I could be wrong. So make sure you check with your CASAS rep on that. MARCI ENGLAND: And then here's a question from Peg. Do students ever lie about which device they have? If they're scheduled for a test or daytime-- so two questions. Do they ever lie about what device they have, or tell you the wrong device? And if they're scheduled for a test and they don't show up, what happens? CHRISTINA HYATT: So we have had students that tell us the wrong thing, and especially before I created the survey. Because they go-- well, I have a Lenovo. And then they showed up with a Lenovo Chromebook. And then I learned, another puzzle in the piece-- another puzzle piece placed, that a lot of people make Chromebooks, not just Google. I don't know, mind blown, or whatever. So lots of people make Chromebooks. Students don't always understand what they have. But they've never intentionally lied about it. So sometimes students come thinking they're bringing one device and bringing another, we work with them. And again, that staggered time to give us the 15 minutes to work with them to see if they can figure out a device that works within our parameters. If not, luckily our district was allowing us to bring small cohorts of groups in the beginning. And then when we went completely remote, we still would occasionally bring one or two students, usually one student in a room to do a test. And through using that, if a student absolutely did not have the technology, or the technology skills, that was our workaround. MARCI ENGLAND: So there's actually a lot of questions popping up here, which is great. We have time built in for this. Is there a specific-- oh, hang on, I missed one. So the question-- do you mean the school will provide a Chromebook to the students? So yes, our school does. We have the ability to provide Chromebooks to the students. Is there specific training for proctoring the CASAS test virtually? CHRISTINA HYATT: There absolutely is. And somewhere in here, I think, back two slides maybe or one, Marci. MARCI ENGLAND: Well, I'm managing the chat. Let's see if I can-- CHRISTINA HYATT: Oh, I'm sorry. That's OK. MARCI ENGLAND: Where do you want me to be? CHRISTINA HYATT: That one. That's the link. Right there is the link to the page where you're going to find that checklist I talked about, going remote checklist. And it also goes through all the training. And I don't know how recently they redid their training thing, but it's really lovely now because it says if this is what you're doing, then you should do these three trainings. If this is what you're doing, you should do these two trainings. And it's very specific and laid out. The training is not and very difficult. It doesn't take too long to become an eTest proctor. I think it takes about 30 minutes. And then we went-- we were part of a special group. And so we did a little bit more in depth on the remote training process. So I can't exactly say how long that is because I haven't done it. I did the more extensive one. But I can imagine it would take too much more time than the rest of them. AUDIENCE: Great, thank you. MARCI ENGLAND: OK, so I'm rounding the end of this. How are we able to get a link to your testing document? I'll put that in the chat towards the end of our presentation. And then Christina, where do they all leave after we're done presenting? CHRISTINA HYATT: I don't think they-- MODERATOR: OK. So I'll just make sure I get that in the chat sooner than later. And then I think there might be someone here from CASAS that if we want that can answer the question about using native language up to the beginning of the test. BARBARA LEHMAN: This is Barbara Lehman and I'm a Program Specialist for CASAS. And yeah, you can use native language up till the test in terms of getting them started and getting them practicing on the sample, the two sample questions at first. But in an eTest, you can get-- you can use native language to get them set up. MARCI ENGLAND: Perfect. Thank you for being here and for letting us call on you. BARBARA LEHMAN: Thank you for sharing. These are-- you guys have done a really nice job. And I know you were part of the pilot program. And so I really appreciate your willingness to share all of this. Because it is daunting for many agencies. MARCI ENGLAND: Yes, yes. It's a big puzzle. There's a lot of pieces in this box but the word of the picture at the end. And I don't know, Christina, I'm wondering if while I'm presenting, you could put the link to our slides and everything in the chat. CHRISTINA HYATT: Have you put him in the Google? I don't know that you shared it with me. MARCI ENGLAND: They're on slide one-- on the first slide and the last slide. NARRATOR: OK, perfect, yes. Then I'll take care of it. MARCI ENGLAND: OK. So thank you for those questions. And yes, Diana, we have really supportive admin and that is a big piece of the successes. We have the opportunity to make mistakes, to try things out, pilot. OK, so now, I'm going to talk about implementation of Civics. And I'll review the resources that were provided, the key steps. First, I'm going to talk about implementation and key steps, which include reviewing all the resources provided by CASAS for California remote testing, determining your testing team, and then identifying the COAAP. So I'm going to hop into that section and then we'll move on to my next section. So really, the essential first step of implementation for CASAS remote testing is understanding the CASAS guidelines. So make sure that you've reviewed the parameters of these guidelines and how you see that working at your school with the systems that you're creating. You need to check that constantly to make sure that you're still staying in those parameters. I feel like I'm constantly just making sure-- OK, am I still on track? Am I still on track? Because you want to stay in the lanes that you're provided with. And I appreciate the guidance, but sometimes I start getting a little farther and farther off track. So check that constantly. It was also helpful for me once I reviewed the instructions and went through it all, to make like a cheat sheet for teachers and staff that don't need every single detail, but they need to know the parameters as well. And those that are administering the assessment for quick reference. And again, I'm talking about Civics assessments. There we go. So the next step in implementation is really this step of taking inventory. This is about really looking at the school resources and identifying who can help. Is it teachers, classified staff, paraeducators? This is a step where you take the self-assessment of your school to identify what you have to work with. Also, you need to look at what students have to work with at this step. What devices do they have at home? For Civics, are they using-- are they doing most of their Zooms on their cell phone? Do they need to come by and pick up a laptop? What's going on? Are you using Zoom? Are you using Google Meet? What platforms technology wise are students familiar with and using? So this is about taking inventory and getting people on board. Some examples of questions you would ask at this point are-- do I have a technology team? Or do we have technology support? What will students be using for the test? Computer lab hours, could we increase lab hours to open accessibility for students that have poor Wi-Fi or distractions at home that make testing difficult? Classified staff with flexible hours can be great at supporting this system. There's always that teacher that's eager, willing to innovate. And Christina said look for emergent leaders. I love that term and I love the people that are just-- oh, how can I help? The key is to look at the pieces you have and identify what you need. So just really kind of taking inventory. Then the next step is to identify the COAAPs and to teach and assess, to reflect on the needs assessment, students' needs, the school resources, and ask is this doable. So one of the key things I'm going to say here is don't get tricked into picking COAAPs before you have done step one and step two. You'll end up with a COAAP that you're using and that doesn't fit in with the guidelines or the parameters for remote testing, or doesn't fit into the guidelines or the parameters of the resources that you're working with at your school. And that really varies from school to school. So make sure you do these steps one and two first. This will-- if you don't, it'll lead you down a path where you'll end up committed to a COAAP that you can't actually implement. So the third step of implementation is choosing the COAAP. Check back frequently again to the CASAS website and look at the COAAPs because they're constantly-- right now, they're reviewing them and they're updating them with that lens on remote testing. And they're adding bits of language that include orally or electronically. And they're updating the COAAPs so that they fit better within the parameters of remote testing. The second-- the first you need to review-- when you get to this step of identifying the COAAP you need to look back at the needs assessment. What did your students say they were interested and need to learn about? And make sure that the COAAP you picked are also relevant to student needs. Then look at the school's resources to make sure that the curriculum that you have supports the COAAP that you're teaching. And that you have those resources available, whether it's BurlingtonEnglish, which BurlingtonEnglish made a wonderful crosswalk between the Civics COAAPs and their lessons that they have. So if you're using BurlingtonEnglish, ask your rep to get that for you. It's really useful. I use mine all the time. And then you need to ask yourself-- is this doable? So it's important to look and visualize it. And the reason why I have a little snippet of-- this is COAAP 47.2, internet safety, is because I just wanted to point out looking at task three and task four. What I do is I look at these tasks and I say-- I notice that it says here on task three, students can respond orally or in writing. This gives you some freedom. If there's a problem with the online testing and a student quickly needs to move to oral and meet in a Zoom one-on-one with a teacher, it gives you that freedom or that ability to be able to do this. And then in four, it says-- you'll see this task says-- optimally, the writing task will be created on a computer and sent to the assessor electronically. So before, when we were really heavily paper and pencil, we almost kind of scoffed at this because we were like-- oh, our students don't have computers. This was maybe two years ago. Oh, they don't have computers. Or they don't have time to be online. And I think we were really underestimating our students. And I think this forward thinking was there from CASAS from the beginning. And now, I'm so glad it's there because it's working better for us and better for your students. So for us, we did find that role playing was difficult because it's very time consuming and it requires breakout rooms. So it's like one on one or appointments, one on one with the assessor and the student. So that's something that we learned at our school was difficult. It can be done, but it's time consuming. And then we also found that portfolio for us, we've shied away from portfolio because it's difficult for us to verify the work is student work, and not like family work, or group work. We want that portfolio to represent everything that the student created which would happen in a classroom but maybe not when students are working at home. So assessors have found that via Zoom they're able to monitor about 10 to 12 students effectively while they complete their tasks on Google Forms. And Google Forms has become our go-to tool for remote Civics assessments. And most students complete the assessment using on a cell phone while Zooming on their laptops, then the teacher can watch what they're doing. They can swipe the room and go through all those steps. And the link to the assessment is usually sent to students via email or remind during the class period. OK, so assessment creation. So what's the first thing people do when they start a puzzle? You dump the puzzle out and you turn all the pieces face up. This is how I see the assessment creation. We dump everything out and we start turning things over so that we can prepare. When creating the assessment, I have found a few best practices. And backwards mapping is now more important than ever. We use the platform and tools that students are familiar with, and we must create and innovate when looking at the assessment and objective tasks. We've got to be creative and innovative. Test learning objectives, not digital literacy. That's a big key here and I'm going to talk about that on one of our next slides. OK, so backwards mapping. So have you ever completed a puzzle without looking at the picture on the box? It's virtually impossible, right? So we all take the puzzle box and we set it up so that it's sitting right there for us. Or I think we all do. I don't know. That's how I've seen it done. So that's really how this backwards mapping works. We have to create the picture-- we have to have the picture for the reference with the Civics assessments. The first thing I do then is outline the objective for the team in a familiar format. So I go straight to the COAAP and then I outline it. And this is an example of what I do so that it's clear and written out in a way that everyone that's involved can understand. I use this format for every objective we do. I link the resources and assessment tasks to this format. This ensures that we're all on the same page. Next, I create the tasks. I use Google Forms as much as possible and create collaborative scoresheet so that I can view work as it's completed. This is a new function in remote testing, instead of paper and pencil testing, where you can watch the scores and have the score so much faster. Often I will take tasks from previous assessments and recycle them to fit within the next objectives tasks. So the format is always similar. We're not learning a new format. We're testing the objectives or the learning gains here. It's important to be innovative and think of creative ways to adapt the tasks to remote formats. Consider all the ways you're tasks-- consider all the ways you're testing tools. So all the different ways that your testing tool, so for us, it's Google Forms can be used. So Google Forms has a video. It has images. It has charts and graphs. So you could use a video for respond when you see queues in the tasks that say respond. You could use pictures where it says identify or list. You can use tables for create a table. Be creative. And when in doubt, if you're not sure, if you have any doubt, always ask your CASAS representative. They're there to support you and they want to see what you're doing and happily, they get back to you really quick and say-- yeah, this is great. This is great. This is within the parameters. Or they'll say-- oh, maybe do this or that. So I've had good experiences with that. OK, so when creating the assessment, I cannot emphasize enough, you are testing the understanding of the objective not digital literacy. And I know Cristina talked about this too. But there's a lot of great technology out there. There's tons of great resources. But if your students are not familiar with the testing tool, it's going to get in the way of the assessment. So pick one or two platforms and stick with that, and help students get familiar with the platforms, and then stay there. Here's a hypothetical example. And I'm using hypothetical because it is hypothetical, but it was almost a real example. So your principal or administrator comes back from a consortium meeting and says-- hey, this school's been really successful with, and I chose Flipgrid here, for the speaking portions of Civics. We should try that out. So absolutely, you should consider and be innovative. But it should be utilized in low stakes, low stakes in class assignments and activities long before it's ever used in an assessment tool. Keep things as simple as possible. This doesn't mean don't innovate, it means we're selective and you first teach the method before we use it as a testing platform. So there would be a lot of assignments and a lot of activities happening on Flipgrid before we utilize that or brought that in as a testing tool. OK, when it comes to the actual administration of the test assessment, I have found it is important to be available and ready to support. Have a calendar with your schedule of availability. Have Zoom links ready so you can hop on and help when needed. Be the calm when assessors are nervous. This sometimes feels like-- oh, it's so high stakes. And what if this, or what if that, and everyone questions technology. You represent the calmness. Be available just to say-- it's going to be OK. That's all right. I can email the printed version. Or we can do it another time. Or I can schedule something different. So that everyone understands-- yes, this is a test and it's important. But at the same time, we have that flexibility and that accommodation. Plan for technology problems. I've had assessors call me in the middle of an assessment and say-- it's not working. It won't let the post. And I just say, I have a backup, no problem. Or I can send you the slides. So just be-- as much as you can think through the questions, Christina has that great list of questions to ask. I don't have that for you, but I do it mentally. Like I try and think like, what could happen, and what can I do to avoid that to prepare if that happened. So the next thing I wanted to say is, Google Forms has helped tremendously for us. And with this-- as I have been able to immediately look at real time test results and share with assessors. And so I wanted to show you, this is what it looks like. OK, so it pops up. If you go to Google Forms and you click on responses, it pops up and looks like this. And then you click on the spreadsheet and this is what the spreadsheet looks like. I redacted, of course, all the student names teacher first names. But before I started the assessments, I would go to sessions like this and conferences and just be like-- where does all the information go? That's what I'm the most confused about and I couldn't figure it out. Well, it actually is so easy to sort. And I just sort everything by teacher name. And then I look at the scores. And I usually move this over to a written scoresheet because a lot of what the results are, I try and leave it as open ended as possible because we really need to measure the students learning. And so multiple choice is something you're going to want to avoid. So I need to look at this and grade all of this individually and then I can grab the scores. And it's just really beautiful. And I wanted you to just take a look at how easy and clear it is instead of stacks of papers and spreadsheets that you're creating. It just happens automatically. OK, so maximizing learner outcomes. Some key ways you can maximize learning outcomes with remote Civics testing here. So some things that you need to make sure you do is that everyone involved has the proper professional development. We have been so fortunate at our school to just have a lot of support from administration and a lot of extra hours devoted to professional development as we've moved into remote testing. This can be official PLCs, or even shared YouTube videos with FAQs. I do-- I'm like the queen of YouTube videos. And sometimes at the end of the video I say-- send me an email that says something so that I like some words. So that I know you watch this and that I know you made it through. Give students plenty of opportunities to practice the digital skills necessary before the assessment, which already talked about. Use resources that are already available. And I'm going to show you the some really great resources that are provided through OTAN and CASAS. And then focus on synchronous learning and Civics. And then find a partner to navigate with. It's really important that you have someone that you can say-- I was thinking about doing this. What do you think? Or I was reading about task four and it says such and such, what do you think? How could I do this? And if you don't have someone, I can be that person for you. My email is at the end of the presentation and I love talking about and exploring the ideas solving this problem, and exploring ideas on how we can do that within the parameters. So here's some examples of maximizing learner outcomes. OTAN has created some great resources. On the left here is a picture of curriculum resources. In the middle, we have technology resources you can use for the assessments. And then on the right, our ideas for how you can assess specific test instructions. So if it says create a list, OTAN has provided ideas on how you can do that. And the link to this is-- it's on the course's website but the link is also here for you. CASAS has digital resources on their website. Start here. They've posted one of my favorite OTAN resources that helps you with task creation. Let's see if I can get it to pop up here. OK, so this helps you with task creation. So maybe you need to create a sample identification card. OTAN has found a link where-- like on the internet, where you can do this. Maybe you need to create fake text messages for some of your curriculum, or social media posts, or newspaper articles. So they've created a list of places where you can just go here and get recipe cards or whatever. So all of these can and should be used when creating the assessment or building the curriculum. Because they're realistic and they're applicable in today's world. And they're already curated. They're already here for you in a nice list so you don't have to reinvent the wheel. So then this is an example of how I wanted to show you getting students use to the assessment tool before the actual assessment. So in a PLC with teachers, I was trying to show them the same thing, like please increase your use of the assessment tool. And we use Google Forms mainly. I created a PLC exit ticket and it just was like which best-- which picture best describes your winter break? And you just pick one. But instantly, you're having an encounter with the same format of the assessment. If you're going to have students email, then you need to make sure that they're emailing-- they know how to use email. Their first time emailing should not be emailing a high stakes Civics assessment or piece to that. So just make sure you're using-- you could use these as exit tickets, emails, quizzes, games, whatever. But you'll see, this practice test here looks a lot like this PLC exit ticket. And so then when they get the assessment, it feels familiar and that creates increased learner outcomes. OK, so here's an example of Civics materials streamlined and stored on clear onestop repositories. We use Padlet for curriculum. And I created a Google site for all of the objective information. When we keep the process as simple as possible, it allows teachers and staff to focus on the actual assessment and curriculum to build student learning, instead of searching through old emails, trying to find links to bits and pieces. This was a pitfall that we realized I was sending out-- here's the great resource. Here's a great resource. Here's this, here's that. And we realized that having everything in one location really helped have a clear process that ultimately has an impact on maximizing these learning outcomes. OK, so for my last slide, I just wanted to talk about how do you find support and just jump in. So at Corona Norco Adult School remote testing and online testing, it's not going away. We have a mindset of we're here and we're here to do this. And we're here to create systems that are not temporary. We're going to embrace this. When our students are back in the classroom, they will come to use-- they will continue to use these methods to test. They reduce paper waste. We have increased test taking response time. Scores and feedback have increased. What started out as a quick fix to a problem, actually has become a beautiful system. And it's important that we have a mindset to create smarter testing opportunities for students, and just really thinking about longevity and not band-aids. That we're here to create good long lasting systems. OK, so I think that's about it for me. CHRISTINA HYATT: So Marci, I was going to say, I must have the outdated slides. So can you grab that link and post it in the chat. And while you do that, I think Marci answered the question about-- is this around to stay? And maybe Barbara wants to chime in. But I don't see remote testing going away. It's equity. It used to be you have to come to campus. You have to do this. But what about the person that can't come to campus? What about the person who is homebound? We've provided an equitable situation for everyone who wants to be educated, to receive the education where they're able. And then let's see. And then the question about testing a student that's out of state, I'm not sure, and I'm not going to speak on whether or not you're allowed to test out of state. But I know I have a student who lives in California and her husband's work has taken them to Texas for a small time. And so we do that all through Zoom. She signs up for the appropriate time. I connect with her on Zoom. She downloads the platform onto her device using the instructions provided by OTAN. And then on our end, we're able to activate that test and let her take it while we proctor her and make sure that she's not cheating, not sharing answers. And those are all part of the testing process that CASAS provides the questions, the restrictions, asking to do a visual scan of the room, making sure they're in a room alone, and watching them rip up their paper at the end because they're allowed to use paper for the math. Only being able to use a calculator that's on screen when they're testing from home. These are all parts of the puzzle that CASAS training is going to help you fit in. And then where the student is doesn't matter from there. You can test them wherever they're at as long as they have a device that fits the parameters set by CASAS. All right, and Marcia, are you ready for some questions? MARCI ENGLAND: Yes. CHRISTINA HYATT: OK. So-- BARBARA LEHMAN: There's one question that says-- CASAS person, my understanding-- MARCI ENGLAND: I like that. BARBARA LEHMAN: --is that we can share practice test-- practices for task but not actually assessments. That is true. The sample tests, there are sample tests or assessments on the CASAS website, you can work with those. If you're talking about EL Civics, they are not as high stake as the goals or life and work series because they're developed by each individual agency. So the limitations on those are much less stringent than the NRS approved ones. So did that answer your question? CHRISTINA HYATT: In this environment, we're going to assume she said yes. If not, there's Barbara. MARCI ENGLAND: She says yes. She going to do that El Civics. And yes, yeah. And for the most part, I mean, every school is going to be different. And so it's really better to get to know your students and teachers. I'm talking from the Civics perspective. And then make sure that-- I think that that's the beauty of Civics is that it is tailored to your school, your environment, your needs assessment, what your students need. And so they really are unique to each school. CHRISTINA HYATT: And then another question was-- can you automatically-- this sounds like Melinda's influence-- get the scores in the spreadsheet, or do you set the form as a quiz? MARCI ENGLAND: Yeah, thank you, Diana. That's such a good question. And that really, for me, was like just the gray areas. How do you do this? And so yes, the form-- I set the form up as a quiz. And then what you do, because I like the questions to be as open ended as possible, what I do then is go through and you score each student's sheet individually. And then that'll populate the form automatically with their total score. Now, sometimes there's a piece like task 3 needs to be emailed, and there's a rubric. And so I still have the score sheets as a rubric. I keep those as their Google. Docs and I have each one scored individually is still a lot less time consuming than the previous way with scoring everything manually with a paper pencil. But I hope that answers your question. Yeah, play around with it because it's fabulous how it works out. CHRISTINA HYATT: And then another question-- is there a clearing house of EL Civics with practices, tasks, and assessments that have already been administered? MARCI ENGLAND: So there's not from us. There might be something-- this might be a question again for our-- BARBARA LEHMAN: This is Barbara, and I-- MARCI ENGLAND: Barbara, yeah. BARBARA LEHMAN: I put my name or my email in the chat. But there is an EL Civics exchange that we are doing in collaboration with OTAN. And it's in the works. So we're beta testing some things right now. And I don't know the expected live date. But when it's up and running, agencies will be able to download any of their materials that they're feeling comfortable sharing. They probably-- and I'm not for sure about the tests specifically, because you want to do your own tests for your own agency. You can-- there's samples on the website. So you can look at those. But each agency should develop their own assessments for a COAAP by looking at the rubric and working with their teachers. But that was one thing. Another thing I wanted to reiterate that you alluded to earlier was every meeting I've ever been that Dr. Zachary is in, she will say remote testing is a wave of the future. So it's not going away. And I believe that that was your point. And I was just going to share that. The other point is testing out of state students, I think if they are your students and they have indicated to you that they are going away with their husband, but they will be back, I think that's fine. I do believe that if you don't want to just accept somebody that is searching the internet and saying-- no, I want to be part of your school. I believe and I can do some more research on it, but it seems to me I was in a meeting and Dr. Zachery discouraged that. MARCI ENGLAND: Yeah, I know for us at our school we look to make sure they have a local address that they're using. BARBARA LEHMAN: Correct. MARCI ENGLAND: Verify that. So I see Neda, thank you for that, Barbara. And Neda, how's your hand? Is your hand raised, Neda? NEDA: It is. It's raised. And it was just to confirm Barbara's comment on the EL Civics exchange. And the big reveal for that is at Summer Institute, so it's at CASAS Summer Institute. They've been working on this for a little over a year. Programmers are really working hard on it. So look for it if you're going to be are you planning on-- if you're planning on attending CASAS Summer Institute, look for it there. If not, we will have a series of webinars after that that will reveal the EL Civics exchange, very excited. And you're right, not the test, but the actual 30 hours of instruction, the material, that's what you'll find on the EL Civics exchange. MARCI ENGLAND: That's awesome. Thank you. BARBARA LEHMAN: It really is. Because all these agencies have spent countless hours developing material. And why reinvent the wheel? MARCI ENGLAND: Right. BARBARA LEHMAN: Why redo the old puzzle? MARCI ENGLAND: Right, right. I agree. OK, so I think I've gotten to most of these-- CHRISTINA HYATT: It says-- for job application, do you use a Google form too? MARCI ENGLAND: I haven't thought about that one yet. So I don't know how I would do that. I don't know. I haven't thought about that. I think I probably would try and make it as realistic as possible. And probably a job application as a fillable PDF is what most employers would be using. I think it's really important to understand that when we're doing EL Civics, we are teaching Civics skills, Civics life skills. And so we want things to be as realistic as possible. So I think I would probably use a PDF, a fillable PDF form for that. OK, Christina. CHRISTINA HYATT: Yeah, so what do you guys do with your puzzle when it's complete? Maybe you glue it and hang it on the wall, like my aunt does with every puzzle they've ever done. I know, for me, you heard I take a picture of it, send it off to Marci, or whoever I want to see that I've done another puzzle. But the time is going to come when we can greet our students back on campus. But as we said before, and Barbara reiterated, and Carolyn Zachary is saying, remote testing is here to stay. And so this masterpiece we've created is going to become a system that our school can use to meet our students where they're at. MARCI ENGLAND: Thank you, Christina. I think at this point I'm supposed to really include a really cool quote from the beginning. But it's just really about having that open mindset and having expressed to your school and administrators, we need to try this, but we're going to make mistakes. There's going to be bumps. But let's support each other and just grow and figure this out.