[audio logo]

OTAN Outreach and Technical Assistance Network.

Cory Chitwood: My name is Corey. Some of you-- I've been working with OTAN for several years now, so I might have had the pleasure of working with some of you already before.

I was a middle school choir teacher for 21 years who used Canvas in my classroom. I was also a Canvas administrator for my school district. And I am a principal learning consultant now for Instructure. So if you like Clarissa, I'm sure you'll like me too, because I'm the one who trained her. So there it is.

So before we begin, there's a few things I would like to get your-- I need your input. So before we even get into what's on the screen right now, last week with Clarissa, the agenda was to learn how to create outcomes, organize outcomes into outcome groups, uploading outcomes with CSV files.

The question that I have is, did you at all talk about rubrics? That's one of our bigger goals for today. So can someone just put a yes or no in the chat with me that was here last week. Yes meaning we did talk about rubrics or no meaning I don't think we did talk about rubrics.

I just want to make sure that I'm not doubling up. Good. Perfect. Thank you very much. I just want to make sure I wasn't going over things that Clarissa might have found time to go over herself. So now that we've established that we're on the same page, now, you might already know about your beta account if you want to practice. I'm sure you've been told this every single week.

So if you feel-- if you're not already logged in, if you want to log in to your beta with outcomes. I really don't feel like it's all that essential to log in to your beta account. No one's really going to see them or use them or find them until you even tell them they're there. So you can always delete what you're doing.

So today, you've already done bullet point one, which is create and upload outcomes and manage them in their accounts. So today, our focus is now that you have outcomes in your account, how will teachers assess those outcomes?

Well, there are two different options. Number one is adding outcomes to your rubrics and attaching those rubrics to different types of assessments. Another option is aligning outcomes to our quiz questions.

So we'll take a look at all of that today. I'm just like, what I'm sure is that been the case for the past four sessions. I'll give you time to try everything out today. Please don't feel pressured to flip between screens and do everything as I'm doing it. I will give you plenty of time to try out everything that I'm showing you today.

But please don't hesitate to ask any questions you have in the chat, please make sure that your chat is set so that everyone can see your questions. And if you ever need more time on anything, please let me know that as well.

So as I was saying, now that you have outcomes in your account, the way that those outcomes are assessed are through rubrics and quizzes. Teachers can create their own rubrics and use those account level outcomes in their rubrics.

Or if-- and I don't really know that this will apply to anyone here, but if there are any common assessments in multiple courses that are going to use the same rubric, you can create those rubrics at the account level and then teachers can just when they build their assignments, find that rubric and use it.

So if there are outcomes in that account level rubric and then teachers find that rubric and use it, then you will have outcomes reporting for every single course that use that rubric and assess those outcomes.

So again, really the only benefit of that is if you have any common assessments that is going to be delivered across multiple courses. Otherwise, again, teachers will create their own rubrics, but then add those account level outcomes to their rubrics. And you still get reporting for everywhere those outcomes were used. You're just not doing the work in creating that consistency for teachers.

I'm going to skip through all of these so that we can just talk about it. So as I was saying, we have two options. Here, I am at the account level. I am-- I've logged into my admin menu and I've gone to rubrics in my admin menu. Over here, I'm in a course. And when I go to rubrics in my course, it looks exactly the same.

So again, either I can create rubrics at the account level that any teacher at my site can use or teachers can create their own rubric, but still use those account level outcomes. Let's talk about both.

And then you can practice wherever you want. If you practice at the account level or practice inside of a course, it's the exact same process. Everything looks exactly the same. The only difference is if you create the rubric in a course and you want to add outcomes to that rubric, you do have to import your outcomes into that course first.

So again, I don't know exactly what it looks like at your site and I don't know exactly if there would be a need for account level rubrics due to, again, multiple courses, multiple teachers, giving the same assessment and using the same rubric.

So I'm going to actually work here at the course level. So here I'm in a course, I'm in my Sandbox. But it could be one of my real courses. And I'm going to first import the outcomes that I am going to assess.

So I'm actually going to go to outcomes in this course, first. So this is how teachers are going to assess those outcomes that you created last week with Clarissa. Just trying to clean up a little bit. Sorry about that.

So if I'm a teacher in a course and I went to assess some of those outcomes that you all created last week or have created since, I'm going to come up here to find. And then I see the account standards. Metropolis University, that is the name of my institution. So that's where I would go. Of course, you would probably see either OTAN or the name of your site. And then here, I have the California College and career readiness standards. Looks like I only have one. Anyhow.

So when teachers go in to import the outcomes or standards that they want to assess, it's important that they only import the outcomes that they're actually going to assess. You don't want a whole library of standards or outcomes, whichever verbiage you like to use, that you're going to have to constantly search through and sift through to find the outcomes that you're actually assessing.

So over here, I have the South Carolina State standards. I wouldn't want to pull at a too high of a level. So that again, I'm not assessing hundreds and hundreds or I don't have hundreds and hundreds of outcomes.

And if you do try to import a large amount of outcomes, it will give you a warning, saying, hey, you're importing a lot of outcomes. Are you sure you want to do that? Only import the ones that you're actually going to assess. So I'll use this California, even though it didn't, I only seem to get one. I want to add that outcome.

Now that outcome now exists in my course lives in my course and I can add that outcome to any rubric that I want or align that outcome to any quiz question and new quizzes. And sometimes you might depending on how all of your outcomes are grouped and organized, you might have to come to one group and add the outcomes that you're assessing.

Then go to another group and add the outcomes that you're assessing. Then go to another group and add the outcomes you're assessing. It really just that's something that you really want to be thoughtful of as you're building and organizing your outcomes.

So in there, I have that one outcome from leadership and digital learning that I imported from my California career readiness. This is not a real California career and college readiness outcome. It's something that I just made up last time I was training.

So that's step one. First, teachers are going to come to outcomes and find the account level outcomes that they're going to be assessing in their course. And then the next step is to create the rubric that I'm going to put the outcomes in.

So there are two ways that you can create rubrics. You can go to rubrics in your course menu or you can create an assignment and then create the rubric there with the assignment.

To be honest with you, my personal preference is to create the rubric with along with an assignment. But because normally I don't think that far ahead. I don't think oh, I have an assignment come up. I haven't even created the assignment yet. But let me think about the rubric. That's just not the way that I do things.

But, again, if you with your, especially, with your California career and college readiness standards, you might have some rubrics that you reuse on a pretty regular basis. And so creating that rubric ahead of time might make perfect sense. But I'm actually going to do it the way that I like it. And that is by creating a rubric attached to an assignment.

So I'm going to go to Assignments. And I'm going to create a new assignment, I believe, I don't think I have a practice one here that I want to use. So I'm going to create a new assignment and I'm just going to call this ESOL oral proficiency assignment with rubric. Just so that I can identify it.

And then I'm going to fill out my assignment like I normally would. Choose what assignment it's going to be. And then save it or save and publish it. If I'm not ready for students to see it yet, then I'm just going to save it. If I am ready for students to see it, I'm going to click Save and Publish.

And then when you save your assignment, the plus rubric button is in the bottom left corner. So I'm going to pause for just a moment to let everyone get caught up. I'm going to ask you in a course, whether it's your Sandbox course or whether it is one of your real courses, I want you to first go to outcomes and find those account level outcomes that maybe-- this is just for practice.

So if you made some practice outcomes last week with Clarissa, then you can import those. If you were not here last week and you have no outcomes in your account, then today just create a rubric without outcomes for practice and then you can go back and check out last week's video and create those outcomes and try it, again.

So step one, again, come to outcomes. Find those outcomes that you created last week. Then go to assignments and create a practice assignment that you want to practice with the rubric and save it. Now, I don't actually have that text on my screen. Let me see if I can unhide some slides I probably do up here. So this is not identical but it'll work.

So again, I'm going to-- regardless of the text on my screen, I'm going to ask you to import your outcomes or find your outcomes from last week into a course. Of course, you don't have to name it and edit and all that stuff. Then you are going to create a fake assignment to add a rubric, too. So find an outcome, create a practice assignment maybe online file upload type of assignment, and then save it.

Again, if you ever feel like you need more time on any of this, please just let me know either in the chat publicly or privately, whatever, so that we can make sure that you feel good about this process.

After we have created this assignment, we're going to click this Plus Rubric down here in the bottom left. Now if I created the rubric in my rubrics menu first or if I am looking for an account level rubric that we've discussed for the past 20 minutes, then I'm going to click Find a Rubric.

When you click-- so whenever you create a rubric as a teacher, that rubric is associated with you the user not associated necessarily with the course. So any rubric that you create in any course is always available to you by clicking on Find a rubric.

Here I see all of the courses where that I've created rubrics at-- 65 rubrics Oh my goodness that's a lot. It shows me how many rubrics are in each course. As you saw a moment ago, my institution is called Metropolis University. So there are my root account rubrics that, again, maybe the Canvas administrators made for me to use for those common assessments.

So once you create a rubric, you never have to create it again. And whether I created it in my rubrics menu or if I created it right here along with this assignment or if it is a count level rubric, no matter what, I'm always going to be able to click Find a rubric.

So this is our ESOL oral proficiency rubric. And then when you create a rubric, you have two options of criteria. You can create your own criteria. That's different than the outcomes that you're assessing.

For example, let's say that I had my students create a slide presentation and I am assessing-- some of our outcomes or standards are being assessed with the information and skills that they're learning by creating this slide presentation.

But I also want to score them on the details of the slide presentation. Maybe the rules were you have to have a minimum of five slides. Maybe the rules were you have to include these bullet points of information in your slide presentation. Maybe the rules were you have to present it to the class and it has to last at least seven minutes long.

Those are specific criteria for this assignment that has nothing to do with our outcomes or standards. Again, the outcomes or standards are what did they learn or what did they show proof of learning or what skills did they get better at? Whereas, again, showing that they used at least five slides and presented for seven minutes is really just following the rules.

So if I do want to give them a specific grade for their presentation for the class and then I will say the presentation followed all of the required criteria, which are five slides or less or more, seven minute presentation, And required criteria listed in the assignment instructions.

So maybe this is worth 50 points. And that's going to update my top rating of 50 points. But then to edit this and to explain to my learners what they did to earn all 50 points, I'm going to click on that Edit Pencil.

I can change the title from full marks to something else if I don't like full marks. And then I will say these student included all of the REQUIRED Information with no errors. That's pretty generic, but no errors.

And then I'm going to use this plus sign to create maybe a 40 point rating. And I'll call this meets expectations. And I will say the presentation included all of the required criteria, except for the length of the class presentation. Maybe if it was five minutes, I'm not going to be too mad about it.

And then I'll add a 30 pointer here. I'll call this below expectations. And then I'm going to explain that the presentation was missing at least two important elements. And then I will do a 20 pointer. We'll call this remediation required. And then I will say the presentation was missing at least three required elements.

And then I'll do a 10 pointer and I will maybe we'll call this reassigned and we will say the student must do the presentation over, again. Again, I'd probably want to come up with better verbiage than that. But nonetheless. And if I want to talk about what 0 means, I can do that too.

Then I want to talk about-- I want to also assess those skills that they showed that they've gotten better at or that they've learned from my outcomes. So then I click plus outcome. And then as I mentioned, I'm only going to see the outcomes that exist in my course. That's why I had to import them first. So here is that leadership and digital learning outcome that I had earlier.

Now outcomes cannot be edited. So however-- on whatever point value and whatever scale you created that outcome at the account level, that's exactly how it's going to look at the course level. And it's not editable.

That way we get consistent data. If multiple teachers use the same outcome and everyone rates it on a different point value and a different scale, then we're not getting very good data. We're getting very scrambled data. So the reason why outcomes are not editable is so that we're getting consistent reporting, no matter who is assessing this outcome.

By default, this check mark is checked. And what that means is that when I assess this outcome in the rubric, in addition to feeding their learning mastery score, they're also going to earn up to 5 points on this assignment for achieving different levels of mastery. So they can earn up to 5 points, depending on how well they've shown mastery of this outcome.

So again, if I uncheck it it doesn't give them 5 points on their assignment. But I'm still tracking learning mastery of these outcomes and I can still see the reporting in my Learning Mastery Gradebook, which we'll talk about and in your account level reports, which we will talk about.

But again, with it being checked, it means not only am I checking, am I giving them a mastery score, I'm also giving them up to 5 points on this assignment. And again, notice that outcomes can-- they don't have the edit pencils, they can't be edited at all. And you can add as many outcomes as you want. And you could add as many criteria as you would like.

Now here is why I prefer to create my rubrics with assignments. When you create a rubric with an assignment, you see these assignment settings. When you create your rubric in your rubrics menu, you won't see these settings until you attach it to an assignment and go into Edit mode of that rubric.

A lot of extra steps. So I think it's just easier if I haven't already created the rubric to create the rubric with the assignment, so that I automatically have these rubric settings. Now these rubric settings, the reason why we don't see them over here in the rubrics menu is because these rubric settings mean, how do you want to use this rubric with this specific assignment?

Because sometimes we want to use rubrics in different ways. Sometimes we want to give our learners a grade with our rubric. Sometimes we want to just give feedback with our rubric. Sometimes we want to track learning mastery of our outcomes with our rubric. Sometimes we want to do any combination of those three.

So these settings do not transfer when you use this reuse this rubric. Because the next time I reuse this rubric, I might only want to use it to leave feedback. And then the time after that when I reuse this rubric, I might want to use the leave feedback and give them a grade. And then the time after that I might want to leave-- use it to leave feedback, give them a grade and track learning mastery of some outcomes. So that's why.

So here are our options. First option is I'll write free form comments when assessing students. And so that removes all of the criterion ratings. And instead you would type in your feedback and manually type in points.

If you want to use this rubric just to leave feedback and not to award any type of grade, you can remove the points from the rubric. If you have outcomes or standards in the rubric, but you don't want this particular assignment to count towards their learning mastery score, then you can choose, don't post the results to the Learning Mastery Gradebook.

If-- this is very important. If you are multitasking right now, please stop what you're doing so you don't miss this. Very important. Alarm, alarm, alarm. If you want to use this rubric in SpeedGrader, which all teachers should, if you want to use this rubric and SpeedGrader and award a grade in SpeedGrader, check this box.

If you don't check this box, you can still use the rubric in SpeedGrader, but you have to take the extra step of typing in their grade. But if you check the box, all you do is click in the rubric criterion ratings and then it will automatically give them a grade as well. So don't forget that people tend to forget that. And then don't understand why they have to still type in a grade and SpeedGrader.

So I'm going to select Create rubric. It's going to tell me that my assignment is worth 100 points. My rubric is only worth 55 points. What do I want to do about it? So I'm going to change my assignment point value to match my rubric point value.

So in the future, if you-- if I'm going to reuse this rubric on another assignment. So let me create a second assignment really quickly. I'll do it real fast.

So when I went to reuse that rubric on another assignment, I'll click on Find a rubric. I'll look in this course I'm in my Sandbox right now. Where is my sandbox? It's probably-- this one right here, quarries.

Yes. There's my ESOL oral proficiency rubric. I'll just select Use this rubric. But I'm going to have to go back in, click the Edit pencil, and notice, again, it did not keep my settings. So with this second assignment, I still want to, again, click on the Edit pencil and then choose use this rubric for assignment grading.

So now, I'm going to give you a chance to get caught up. So again, if you want to work at the account level, if you want to work in the rubrics menu of your course level, or if you want to create a rubric right there with an assignment, those are your three options right now. You can do all three-- any of those three.

And then, we will take a look at adding rubrics to quizzes. And then we'll finish up with learning mastery data with the Learning Mastery Gradebook and with your account level reports. Again, if you would like me to show you something, again, or if you have any questions, please let me know.

The second way that we assess learning mastery of our outcomes is through quiz questions, and that is through new quizzes. So when you go to a course and go to quizzes-- and you can also-- I mean, let me rewind a little bit actually.

So again, there is the option. If you don't want teachers, again, if you're going to have common assessments where multiple courses with different teachers are going to be offering the same assessment, you can create a count level item banks. So here I am at the admin level.

You can create a account level item banks. And then as you are creating those questions, you can align those questions to your standards and you can share those item banks with all of your teachers. And those that are assessing those questions can import them into their quizzes. And the admin has done all of the work for them.

But, again, if you don't have any common assessments or if you're just going to let teachers deal with create their own, when you create a quiz, you do-- you can only align quiz questions to New Quizzes. So if you're given the option of New Quizzes versus Classic Quizzes, it does have to be New Quizzes.

And then you're going to give it a name. And then at the very bottom-- I should see a plus rubric button down here. I don't know why I don't. Maybe they moved it. So when you create a quiz and go into the build menu--

When you create new questions here with this plus sign, no matter what question type you choose, you can align outcomes that are already in-- and, again, they have to already be inputted in your course. But all question types have a section that says align to outcomes. This is multiple choice. This is formula. All question types have a section that says align to outcomes, and it really is just as easy as clicking in this section.

There I want to find that outcome, again. Just check it and select Confirm alignments. You can align multiple outcomes. Here I have many of them. I could align multiple outcomes to the same question as well.

I don't have an activity about that planned, but if you have any questions or want to show me that, again, or want me to show you something, again, please let me know and I'll be glad to do that. Let me delete that one I just made.

Now, where do we see all of the information? So at the course level, teachers will go to their Gradebook-- actually, let me pull up a slide. I don't know if I have one in here. So teachers will go to their Gradebook-- let me go to a different course. There we go.

And they will turn on-- oh I don't have the Learning Mastery Gradebook. Hold on just a minute. Wrong course. Sorry about that. There we go. To grades. And then to the Learning Mastery Gradebook and the dropdown menu.

Now, all of my students right now are overachievers. They're all exceeding mastery. So here is what it would look like more normally. As you hover over each column, it'll show you a pie chart of how well students have achieved mastery of those of the different outcomes that have been assessed in this course.

I love the Learning Mastery Gradebook because it allows you to toggle on and off different mastery levels. So that you can focus on students who are at different levels of mastery. You can also hide the outcomes or standards that you've imported into your course, but you haven't assessed yet, and you can export your Learning Mastery Gradebook into a spreadsheet.

So again, you just go to your traditional Gradebook and in the dropdown menu you should see Learning Mastery Gradebook. Also with quizzes, I forgot to mention with quizzes, if you want to see how well students achieve mastery of the outcomes that were assessed just in that quiz-- I'm going to open up a quiz for you right now. So I can show you.

So in quizzes, as I was saying in the Reports tab, there is an outcomes analysis report that shows learning mastery of just the outcomes that were assessed in that quiz. Let me see if I have a screenshot of that. If not, I'll go back and I can show you one. I'm going to pull this screenshot up for you real quick. There it is.

So there's the outcomes report and New Quizzes. So real quick and easy to see. Britain is doing very well. This outcome itself is doing pretty well. This one down here is doing pretty well. Got a lot of Black in here that we need to maybe do some remediation.

So that is so far we've got learning mastery scores for the quiz, we've got learning mastery scores for the course, and the Learning Mastery Gradebook. And then finally, you have learning mastery-- well actually not finally, learning mastery scores for individual students.

If you go to people in a course or if you go to your admin menu and go to people in the Admin menu, either way, if I'm in my admin menu, I can find that student. Click on their name and view their outcome results right here. If I'm a teacher of a course, I can go to people in my course. And then down here at the three dots go to User details. And I have their outcomes report there.

So let me repeat that. If I'm an admin, I can go to people in my admin menu, search for anyone with the student role, and see their outcome results here. For all of the courses, every outcome they've ever been assessed on.

If I'm in a course, I can go to people, go to user details, and then view the outcome results report for just this course. So I can see how many times they were assessed on a different outcome, I can see what their most recent mastery score was, and I can see what their overall score is.

If I want, I can hide the outcomes that have not yet been assessed. And if I want to see where they were assessed, I can click on Show All artifacts. And that will show me-- it shows me the assignment and what their score was.

So again, at the course level I'm only going to see outcomes that were assessed in this course. If the account level I want to see all of the outcomes for that student for all of their courses, whether they've been assessed or not. If the outcome exists in that course, it'll look like this.

And then as an admin, the last thing that you have. And this really is the last thing. I think, let me double check. Let me make sure I didn't leave anything out. We got the Learning Mastery Gradebook. Yes. We've got the outcomes report and New Quizzes report. So yes, this is the last thing.

So in your admin menu, if you go to Settings, which is at the bottom of your admin menu, and go to Reports in your Admin Settings, there are three reports related to outcomes. Outcome export, outcome results, and student competency report.

If you click on any of these, it'll give you a description of what that report is and an example of how that report could look. But the outcome export literally just gives you a spreadsheet of whatever outcomes exist in your account.

It doesn't give you any learning mastery scores or anything like that. It's just a spreadsheet version of the outcomes that exist in your account. And last week with Clarissa, if you were there, you learned how to upload a spreadsheet of outcomes. So this is how to export a spreadsheet of outcomes.

The outcome results report is really the report that you're going to use. It's going to show you all of the outcomes in your account and how well learners are achieving mastery of that outcome.

Per student, you'll have a column, per student to show them-- show you their scores for how well they're achieving mastery on all of the outcomes, where the assignment was, all of that great stuff. Of course, it's just going to give you a spreadsheet that you can filter out and manipulate however you want.

The student competency report. To be honest with you, I have been working with Canvas since 2018. I've been using Canvas since about 2016 and I have never understood the value of the student competency report, and no one has ever been able to give me an explanation that makes sense to me.

The reason why is because the student competency report does not include data from learning mastery data from quizzes. It only includes learning mastery data from rubrics. So if rubrics are the only way that you are assessing learning mastery, if you're not using quizzes at all, then this is a great report to you. But again, if outcomes are being assessed in quizzes, the student competency report will not include that.

So a real quick review of what we did today and then we're going to head out. We started off-- you built your outcomes last week. This week, I showed you how to import outcomes into a course.

This week we talked about you can create account level rubrics that if teachers are giving common assessments, you can create them at the account level. If not, then teachers are going to first import their outcomes and then create their rubrics and their course.

You can create rubrics in your rubrics menu or you can create a rubric directly attached to an assignment. If you create-- either way, you want to make sure to always check your rubric settings and to make sure you use this rubric for assignment grading as checked.

We looked at our learning mastery data in the Learning Mastery Gradebook and the quiz outcomes report and the student outcomes report and then these three account level reports.

So that's our review of what we've accomplished and a little bit over an hour. Before we head out, are there any questions or was there anything that you would like for me to show you again?

If not, if you want to earn your badge for attending today's training, you can use the QR code on the left. If you feel like leaving a survey about your experience through all of these cohorts, don't forget to give Clarissa a shout out. You can use the QR code on the right or use the link that short bitly link on the right.

Thank you, both of you, for hanging out with me the whole time. I am-- if you by chance come back for cohort two, it'll be the exact situation, but it'll be me instead of Clarissa. So if you feel like you need to just go over some of the information, again, I'm pretty sure it's OK for you to come back for any of the second cohort sessions. Thank you both so much. I hope you have a great day and hopefully I'll see you, again.