[audio logo]

Speaker 1: OTAN, Outreach and Technical Assistance Network.

Speaker 2: Thank you, everyone, for being here this morning. We're going to be talking about, really, revolutionizing our data driven decision making with AI tools. And over the last year or so, I hope that you've begun really exploring and dabbling with AI.

And, really, what we're going to be doing is taking the data that we utilize in our assessments from summative assessments and formative assessments, as well as other school data that we can essentially export and import into AI tools where we can essentially take that so that we can use it to help for program development and work on instruction as well as student outcomes.

So I'm going to, in the chat right now, give everyone the link to the presentation. If you already have it in-person, please feel free to scan the QR code if you have not done that already. And it's in the chat right now, the link to the slideshow. But let's jump into it.

So I'm the EdTech integrationist. It's a shared position at the Education Career Network of North San Diego County Consortium. And I work on all things instruction, EdTech for the consortium. We have about seven schools.

And then I am also a Professor at San Diego State University and Concordia University, Irvine. I work with new teachers and dual language program, as well as oversee doctoral research in all things EdTech instruction. So got my hands in a number of projects. And I'm an author, podcaster, and from time to time I consult in schools. And I hope to connect with you on social media, on X, on Twitter, Instagram, and LinkedIn. I hope to see you all there and we can further learn from each other.

So AI can do many things, so I'm not sure if to play with image generator. I took two images of my dogs, Bailey's to the left, Boogie Bear's on the right. And I uploaded them into an AI tool and I asked them to play in a park in San Diego, and that's what it generated for me. It's pretty incredible how we can take images and upload them into AI and they can do quite interesting things.

Similarly with data, we can take data-- for example, I took trends in ESL courses data throughout an entire school year for an entire site. I made sure all the data was anonymous. I stripped all of-- any sort of identifiable information. May prompt the AI to look at particularly the trends of scores over the course of an entire school year. And it made essentially a trend line, as you can see. And we can use this information that support us in our programs.

And we're going to see a lot of these types of examples today, and I'm to show you how to do it. And there are a number of different ways that we can do it but let's, first before we jump into how to do it-- and I'm doing a demo of how to do it-- let's jump into the why because the why is really important as to really ensuring that we have a purpose behind what we're doing.

And with that in mind, we need to also have some prerequisites in mind because in order for us to do this type of data analysis with AI tools, educators need to have a degree of digital and AI literacy and data literacy so that I can, essentially, understand what exactly I'm doing. And I did get a comment where people want that QR code again, so let me go back.

So, essentially, going back to those prerequisites while a number of educators scan that QR code, is that we need to have background knowledge in what we're trying to do because if we don't have that background knowledge, then it's very difficult for us to essentially understand what we're doing. And also the AI sometimes is wrong. It hallucinates. So we have to, at times, go back and do the calculations ourselves.

And also part of this story is that there's a qualitative side. So qualitative data is textual data related to the experiences of those in our organization and our students. So that's also important to put along with our quantitative data, the numbers, so that we can triangulate. So we have to understand that we need to include both these elements. And this data analysis is available on a wide range of tools, including ChatGPT, Claude, and Gemini.

I didn't change Bard to Gemini quite yet on this presentation since I've last given it in November, but Bard is now Gemini. So understanding prerequisites and our why is really important. And our why, essentially, is that we want to take really important data points and really understand what's happening with our students' experiences in our school, what's happening in structurally, what's happening in terms of accountability and also programmatically so that we can make focus on continuous improvement.

And depending on your role in context in education is that a lot of what I'm going to show you can really change your ability to really create opportunities for you where you can focus on the actions of doing policy change or changes within your program versus really crunching numbers, writing reports, and whatnot.

I think that we live now in a time where we can spend a lot less time focused on writing reports and crunching numbers and focusing on really the actions happening in our schools. So these tools definitely give us an opportunity.

So what I'm going to show you is ChatGPT, but I'm going to show you the advanced data analytic piece of this. And I'm also going to share with you some add-ons with consensus, which is a research database where we can look at millions of research articles that we can connect our data analysis to, which is really exciting, where we can actually take our analyzes and triangulate with research and create policy from that.

And then additionally, I'm going to show you that Google Gemini does have some elements of what the data analytics piece of ChatGPT has, but I could say that it's not where we would like it yet. But also just noting is that within Google Sheets as well as Microsoft Excel, if you do pay $20 per month for Microsoft Copilot or Google Gemini, you do get within those programs that generative AI piece where you can prompt to create functions, to create graphs, to do a wide range of things within those tools.

So since we have people in-person and online, go ahead and click on this magnifying glass. And-- actually, don't do that. I believe that from last time so don't. If anyone can on the chat, at least, Zoom tell us who your contact is and education. And then I'll be sure at the very end to provide an opportunity for people to tell us where they're from and how they're using data. So I'll focus more on the demo today than spending time on this.

OK, so I think that there's four major quadrants of why we're trying to do this and why we can use AI to support us in doing so. We want to be evidence informed, and data can lead us to being evidence informed. And we can use the AI to really clean the data quickly and transform data and analyze data within minutes.

We can include textual qualitative data from our students' experiences and our experiences in our organizations, and utilize that within our analysis. And also, I think, really, a lot of my research in my doctorate was on data efficacy of school leaders and teachers is that, really, the biggest impediment of doing really any sort of data dialogues, data analysis was time.

So what this does is it frees up more time to do this as well as being more efficient. So that's, essentially, the four elements I think they're really important to consider here. OK, so I'm going to show you a demo now where we're going to look at the courses, test scores from an entire school year, from the '22, '23 school year. And then we're going to look at payment point data from a consortium.

And all the identifiable information has been stripped but I'm going to show you how to do this in the prompt. All right, so here's the prompt that I'm going to play with. So I'm going to copy and paste this data, this prompt into GPT4. So this is GPT4, everyone. And the code interpreter, the data analytics piece has been embedded directly into GPT4.

And what I'm going to do now, I put in the prompt, you first always want to give it a role for what the AI is going to do, and then you're going to tell it specifically of what you're uploading. I'm going to upload this ESL CASA's data. And I would like it to do some descriptive analysis in several visualizations, picking that data along with the data dashboard.

Last, I would like to have a short analysis in the form of a number of key themes and bullet points that I can discuss with my approved supplier the end of the year conference. So maybe I'm a teacher in this scenario where I want to look at my ESL data and provide my principal with a report of what that data looked like over the course of the year.

And then here is the data itself from this particular class, just the dates of the assessments. And the scale score and raw score. So I'm going to upload it. So I'm going to put in the test score data. Here it goes. All right, my prompt is there, my data is there, and then let's go and send a message. And now starting to analyze the data based on the prompt, and let's see what the output is.

So, first, what I like about this when it's giving its output is it's going to tell me exactly what it's going to do. And this right here while it's computing, it gives me exactly what it's going to do. It's going to do descriptive statistical analysis and it's going to do it-- calculate the mean, median, standard deviation, and range for both the scaled score and raw score.

Then it's going to give me a number of histograms in terms of these visualizations, a line plot showing the progression of the average scores over time. And then it's going to create a data dashboard and then it's going to do some key analysis.

So at times just know that if the data is not fully clean, it's going to tell me there's going to be an issue and it's going to correct that. So it says that there's going to be a mixed data types within this, but it's going to correct that, which it does.

So here's this here's the scaled score column. It's giving me the count, the mean score, 222, the minimum score, the 25th percentile, the median score. So it gives me all that for that particular data point and then the raw score.

And then it says, this analysis provides a broad view of the performance levels of students on these assessments, including the variability of both the scale and raw scores. So now it's proceeding with the visualizations to depict the data. So here's the histograms of the average scores over time. So it's giving me these as well. And then it's going to talk about these scores with the histograms.

It gives me a short little description of what exactly they are depicting. And now it's going to consolidate this into a concise analysis with key themes with the principal or the principal discussion. And then here's the key themes, diverse language proficiency levels, suggest a wide variety of language proficiency levels among students because the potential targeted for interventions, the identification of students scoring at the lower end of distribution, especially those shown as outliers.

So we can see, for example, what outliers are going to be. And it gives me, I think, some valuable insight with a lot of adult educators is that there seasonal trends. We have generally more students at the beginning of the year, trickles off a little bit more at the holidays and then it comes back up. So that's some interesting pieces.

And then instructional strategy effectiveness, the distribution of trends that can serve as indicators for possibly instructional interventions. So now it's given me some discussion points related to instruction, intervention, and curriculum review and professional development. And just know that this is just a small little fraction of scores that it provided for one single class. It can be an entire class, it can be your entire program, it can be a wide variety of different things.

But I also have a follow up prompt. I'm going to say creative visualization of the scores throughout an entire school year because the histograms they gave me, I wasn't not exactly super happy with it so I'm going to do a follow up prompt.

And you can do further analysis based on that data that you've uploaded with think of it as a living and breathing document as you're working with the AI. So if it doesn't give you what you initially wanted, you can prompt it further with that data uploaded. So it's going to correct this column, so that's the cleaning piece that I was talking about earlier.

Speaker 3: Hey, Matt.

Speaker 2: Yes.

Speaker 3: Question for you. If I had a question about one of those specific graphs, could I just have it reference that graph or can I-- will it analyze the graph if I put it back in or do I need to do that?

Speaker 2: Yes, so you can reference the graph itself or you could even take a picture of the graph and then upload the picture of that particular graph and ask it to further analyze that graph. So this shows the scaled score and the raw score trend throughout the entire school year here.

And we can make the graph itself-- we can change the Y and X-axis as well, scale it as well. We can prompt it. So there's a lot of different customization pieces that you can do here with this tool, it's just a matter of just continuing to ask it to revise as we go.

Speaker 3: One other question for you, since your data has been stripped, at some point can you just even share a picture of-- I'm just curious, I want to see the data that went in.

Speaker 2: Yeah, I'll show you right here-- I'll show you right here of this particular-- this is the data that went in.

Speaker 3: We can't see it.

Speaker 2: Oh, you can't. Hold on. Let me go ahead and reshare.

Speaker 3: That's the question.

Speaker 2: Now can you see it?

Speaker 3: Yes.

Speaker 4: Yeah, yeah.

Speaker 5: Pretty small.

Speaker 4: Make it bigger.

Speaker 2: Just let me see if I can make it bigger.

Speaker 3: Yeah, there you go.

Speaker 4: Thank you.

Speaker 2: So this is the data that I inputted in. So this was all the cost of assessments for an entire school year.

Speaker 3: That's good.

Speaker 2: That's just one example of me doing this. Now, let's say that you're a leadership team. So you're in adult education leadership team analyzing payment points for the consortium of adult schools. The data I'm uploading contains ABE, ESL, ASC, civic participation, citizenship prep, ELC, totals and various-- and I want it to do totals in these various areas of enrollees, total enrollee paired scores, completed NRS, level percentage.

So this is far more complex as you read this prompt. And I would like it to conduct some descriptive analysis of the data, followed by visualizations depicting that data, along with the data dashboard. I'm going to show you exactly what that data looks like as well. So as you can see, it's a lot more botched. It's not like it's clean, as you notice here. So the AI is going to have to do some more work. So let's go ahead and demo this now.

So what we've done in our consortium is during these data dialogues, we are using the AI as kind of like the third objective person in the room that's analyzing the data along with us. And, really, the goal at the end is to triangulate our data dialogue with what the AI has come up with. So that's something new that we've been doing.

So here's the prompt. And then I'm going to press this right here. And then I'm going to go ahead and click on payment points. And it's up. Here it goes. So now it's going to analyze it and it's going to tell us exactly what it's going to do in its evaluation. It's going to do a lot of cleaning up.

And it does a lot of thinking as to what it's identifying, which is cool to see. And you can verify what exactly it's identifying versus what it's not identifying. So now it's going to provide a descriptive statistical analysis here of this data as it's working.

All right, here it goes. It gives me the total enrollees and the range of enrollees, the average impaired scores, completed NRS levels. So let's go up here real quick. So it's giving me the average number of enrollees across all schools for these particular programs. It's given me the total enrollees with paired scores.

On average, schools have around 275 paired scores. So that's really great because we know that the total number of enrollees for these programs was around this. Completed NRS level, we were able to see that, 164, and then some schools have as high as 985. And then passed citizenship, 12.7 students per class passed citizenship with a maximum of 35 in some schools. And this is a year. This is for the year, by the way.

And now it says the next steps it's going to do a visualizations, the data dashboard, and then key themes and action plan. And see how another error came up, it's cleaning the data. As you saw on the example I gave you, it's not particularly clean, but here's the scatter plots of total enrollees versus achievements.

So this visualization depicts the relationship between the total enrollees and the total achievement across the various adult ed programs. And then I can go ahead-- I'm going to ask it to-- in a moment to do something else, but it's developing some key pieces of insight here, the variability of program outcomes, success in high school equivalency and diploma, citizenship and civic engagement.

It came up with some points there regarding the high pass rates, the academic achievement beyond basic education. And then I asked it to develop an action plan here. So beyond just the analysis, I asked it to, let's create an action plan based upon the data that we are already seeing.

So this draft it's giving me, it's giving me benchmarking and goal setting, it's giving me some ideas here that we could utilize. And we could further add more data to really build this out. So it's given me a number of these ideas here.

So I'm going to ask it real quickly to go back to this graph here. I'm going to ask it for the graph you've created. Create a new draft with a line of best fit. Let's see what it does. And then there's a question in chat. Hold on a second. Looking at it now. Here it is. Here's the best fit line. See how you can do that right there.

So I do this a lot with students that are doctoral students when we're doing correlational research as well as looking at regression. So looking at a variety of different variables and seeing if we can predict one or more variables in the future. So this is very nice here, as we can see.

So that's just one-- that's another example of what we can do with this type of data, with this data analytics piece. So there's also some other data that we can look at as well.

We can also look at coaching data and coaching cycle data, we can also look at attendance data, we can also look at wide variety of different data markers. So give me a moment, I'm going to now upload PDFs. So I'm going to pull this up for you right now.

Speaker 5: Matt.

Speaker 2: Yes.

Speaker 5: Is this is Claude? Is this a different--

Speaker 2: No, no, this is-- this is ChatGPT.

Speaker 5: It is ChatGPT. Is it data using Claude?

Speaker 3: I think the previous one was Chat.

Speaker 2: All this has been ChatGPT. I've not used Claude or anything like that. You can upload data, you can upload PDFs to ChatGPT4.

Speaker 5: Did you use the free version or paid version?

Speaker 2: So this is the paid version to do the advanced data analytics piece. But if you would like to use the free version, you can basically copy and paste the data on the Excel Spreadsheets into 3.5, and it will do some of the data analytics for you. But if the data is very not-- if it's not clean, then it's going to be a little bit more problematic to you.

OK, so here's some-- I do coaching with teachers, and this is just kind of like our overarching data, just relating to different things that I'm doing within our consortia. So this is from our coaching software here. I'm going to upload two pieces from our coaching software here. There's no names here or schools names, I just have it for entire consortia.

So I've got these bits of coaching pieces for our teachers based on interactions and what are we working on, what are we not working on, coaching cycles and whatnot. So I'm going to show you next what this looks like.

So now say you're an instructional leader or instructional coach, you're analyzing coaching data from the 2022, '23 school year, as well as coaching cycle data. Your job is to complete a comparative analysis of both PDFs of the data I'm providing you. First, do a descriptive analysis. Next, I would like you to compare the coaching data from the year with coaching cycles, determine on your analysis whether you see a relationship between the completion of coaching cycles and the yearly coaching data.

So let's see if this works. Sometimes with these big PDFS there are some problems, but let's go ahead and try and then click on my paper clip here. Let's see if these work. All right, success. Let's see. So similarly to how it analyzes the data-- here we go. Let me go down to the bottom, everyone. All right, here it goes.

So similarly to how it does with the data from a CSV file or excel, it can read these PDFs. So you can take a long, long report and you can have it-- or book, for example, or responses from a survey that are just textual, and it will read those for you and it will essentially give you the themes that you're looking for, finding that piece of information that you're looking for.

Let's see what it says. All right, so it's going to extract the data. It's given me exactly what it wants. I'm going to say, go ahead. Sometimes with some of these AI tools, it essentially-- it's going to give you a choice if you want to move forward or not. So I said it's going to go ahead and it's going to do the following pieces that are being listed above here regarding the comparative analysis.

So what happened here is-- and you'll notice this sometimes with the PDFs-- it wasn't able to read these coaching cycle PDFs well because above it stated essentially that-- I need to give it numbers. Within those PDFs it couldn't read those particularly completed coaching cycles. So I'm going to go back to my data and I'm going to see how many particular coaching cycles I did. Let's see.

So I did 271 and then I did 57 coaching cycles. So I'm going to say I did 271 coaching hours and interactions, and I completed, believe it was, 57 coaching cycles. 57 coaching cycles, now use that for your analysis as that is the information you needed.

Now it's going to be able to provide that information. Sometimes it's going to ask you for things that it cannot read so you got to make sure you go in and verify what exactly your data is saying here. So now it's creating a nice little graph here. And now it's kind of talk about the areas that it reviewed.

So it's saying maybe this nice little table here, gave me some ideas what it was reading. Noticed that there was a high engagement and structured support, potential for impact, 57 coaching cycles completed. And then it's going to give me some recommendations. It says structured coaching is key. We can do some further analysis to fully understand the impact, and then recommendations.

And what you could also do is you could take this type of coaching data and compare it to testing data or other metrics. So you can look at a variety of different data sets when you're working with this type of analysis. So you can upload more than one, just make sure in your prompt you discuss it.

So as you've seen, these visualizations are really helpful for us to see the trends in the data. And we can be utilizing them for presentations to stakeholders in our community. I think with the visualizations we're able to tell a story better than we are with just the sheer hard numbers. So I think this is one really great element of these tools is we can create these visualizations rather quickly and it can tell a story.

So we can utilize these types of prompts for visualization. So, say, after your initial review maybe you didn't want visualizations at first, you could say based upon your review of the data, can you create a visualization of the data in a visual X format. So graph, data dashboard, scatterplot, et cetera. You could develop a visual-based upon your previous data analysis.

So, really, you can include the visuals after the initial analysis or within that initial prompt. So I think that's something really important to note. And this is just an example of what I created, and you saw that early on.

And we can look at-- for this particular visualization here, this could tell a story of many different parts of a program if we also include and triangulate the qualitative data. So it's important to note that we can use these along with the human side of our organizations to tell a story to our stakeholders and use this data to help us in our programs and for our students.

So I did it a monthly count of scores, these tell a story. So these are some of the examples of visualizations. And just this past week we did a equity survey and I did something very similar to this for a particular school that I'm working with. And we created similar graphs like this to be placed on a stakeholder presentation based on our equity data. Here are some examples of some of these coaching visualizations that you can create with that similar data that I uploaded.

Now what we can do is we could, for example, integrate our findings with research. I really like that integration with our AI tools. So I'm going to show you how to do that. OK, so I'm going to copy this prompt and I'm going to go back to my prompt about the ESL data. So let me go down here, ESL data discussion.

And I'm going to, actually, create another chat with Consensus. Consensus is a GPT here. Consensus is that GPT which is essentially, it aligns to a database or someone's created a GPT to behave in certain ways. Consensus is essentially a database of 250 million research articles.

So I'm going to input the data here from one of our analysis. So give me a moment. I'm going to be pumping this in here. So if you are an educational researcher, take the X data analysis. So take the data analysis results, and then below-- so I'm going to punch in my data analysis results below this initial prompt here because that's what I'm asking it to do.

Let me go down here, and I'm going to include that analysis I received from my key themes from the data. So I'm putting in these particular themes from the ESL data that I did earlier.

And what I'm asking it to do in the initial prompt is to essentially look for research related to these themes. And let's see if we can find some research in these areas that we can triangulate here. So says, based upon the analysis and the focus on instructional strategies for adult ESL learners, the following studies offer some evidence and recommendations for practical interventions.

And also we can click on these particular studies as well, and we can look them up and read them if you would like. So if I click on this particular study here, it gives me the full abstract of this particular study and it gives me the key takeaway. And then I can also go click on the full text here of that particular article.

And look it here, this is an old article from 1985. We shouldn't be using research from 1985, so this could be an article that we throw away. But what you can do is if you want to take the PDF here, we can download this PDF. Sometimes they're free, sometimes they're not. This one looks like it's not free, so we're not going to download it. But for some of these they are free, and you could upload them into the AI tool and have it read those particular PDFs as well.

But it's giving me a wide variety of articles here that I can further look at. And we can also go deeper and ask it for more if we would like. So this one's talking about culturally responsive teaching to increase the participation of adult ESL classrooms. Intervention programs emphasize differentiated instruction, and highlighting the role of teacher self-efficacy beliefs in targeted interventions, curriculum review, discuss the challenges of implementing differing instruction for online ESL teaching.

So this kind of discussed that maybe if you had some online programs, you would need to focus on coaching and professional development related to online teaching, especially with differentiated instruction, monitoring evaluation.

So let's go ahead and see if there's more. Can you narrow your focus on research articles related to instructional strategies as we are wanting to work with a number of newer teachers to develop their instructional strategy toolkit, teaching ESL students. Ensure the articles are from the last five years.

Which strategies have yielded the most success in relation to student achievement scores? So you can further refine. So now it's searching within consensus, and it's going to give me some more outputs to look at. So for--

Speaker 5: Hey, Matt, consensus is that part of the paid version or that's part of the free version?

Speaker 2: It's part of the paid version.

Speaker 5: Yeah.

Speaker 6: This is an extension of the ChatGPT?

Speaker 7: Yeah, it's like an app or the GPT. They have all these add-ons.

Speaker 6: They knew what they were doing--

[interposing voices]

Speaker 7: I think the monthly on the GPT is $20 a month, yeah.

Speaker 2: And speaking of the cost for this, I recommend any school leader to be paying $20 a month for a license for this as it will-- it's the best $20 per month that you'll be spending to do this.

Speaker 6: In my district we're having teacher evaluations for life modeling--

Speaker 2: So it gives me a wide variety of studies to look at, as I just demonstrated early on. So you can further refine this if you would like. So this is kind of like the pyramid workflow that I like to use. So number one, you want to collect your data and export it from the tools and strip it of all identifiable information. Then you do your data analysis using AI, develop reports and action plans to progress monitor.

For example, we're working a lot of on our continuous improvement plan. So we've utilized this methodology to create our SMART goal and our ability to monitor them. So this process here is very helpful for not only doing the analysis, but creating goals and ensuring that we are using the right data to make sure that we're keeping track of those goals. And we can do it quite easily.

So we can use-- based on the research that I provided in the previous prompting, say, for example, you narrowed it down on a number of themes and research, you could then develop an action plan using this type of prompt here. You're now a group of adult school teachers and leaders developing an action plan based upon maybe the results of the ESL assessment, as well as the further dive in the research.

And we would like to develop our first progress monitoring system where we're collecting these metrics monthly. We would like to create an overarching goal for this action plan and designate six of our group member roles and responsibility to ensure the action plan essentially is followed.

Last provided analysis and presentation outline to present the teachers and the school community related to this action plan. So you have a ton of information that you've already delved into and have created that output, then you can follow it up with a prompt like this to develop that action plan, those goals, as well as a draft outline to present.

And, obviously, as we go through this, this is just a draft. It's a working draft. So everything that you are creating, take it with a grain of salt as you need to thoroughly review it for accuracy and hallucinations. So it's really important that, as you go through this, even though it's really quick, just know that be sure to review very thoroughly when you're working on this type of work.

All righty, so in person, please, if you can on a piece of paper, I'd like you to write down three key ideas, what are your main takeaways from today's presentation? And then for those on Zoom, write this all in the chat, please. Write down three key ideas related to what we discussed today.

And then I would like you to also write down what are two extensions you're going to take from what we've learned from this presentation and put it into practice. So basically like action items for you. How are you going to further extend what we talked about today and put into practice? And then, lastly, What are questions, what questions do you have to support your practice? Thank you, everyone.