Oby: From the campus of Harvard Medical School, this is Think Research, a podcast devoted to the stories behind clinical research. I'm Oby.

Brendan: And I'm Brendan. And we are your hosts. Think Research is brought to you by Harvard Catalyst, Harvard University's Clinical and Translational Science Center.

Oby: And by NCATS, the National Center for Advancing Translational Sciences.

[music playing]

An important element of learning, reading, and watching, is the accessibility of a given piece of material. As we look back on the history of our world, most of it is not easily accessible in the modern age. Considering historical newspapers, letters, and other archive documents-- how accessible can these documents really be to today's diverse world of researchers, educators, and readers?

World renowned economist Dr. Melissa Dell of Harvard University has developed an open source package that can detect objects, analyze layouts, and more, to uncover massive amounts of data trapped within various documents. Welcome to the show, Dr. Dell, great to have you.

Melissa Dell: It's great to be here. Thank you.

Oby: So you received a Translational Innovator pilot grant award to develop methods for detecting complex text layouts and create a database of newspaper content from American communities. This is using advances in deep learning, but can we take a step back/ What is deep learning?

Melissa Dell: So essentially, deep learning is a series of methods to take raw information that's in a form it's not computable, and to turn it into representations that you can use to do whatever you want to do downstream. So in this case, we have image files of newspapers which comes in the form of image data. And we use deep learning, to first of all, to recognize the structures. So that would be like headlines, articles, captions, photographs.

And then we further use deep learning to extract the text, so to do optical character recognition. And then the end result of that is a text database. You could take that one step further and create additional measures from that, that can be used to do analyses and stuff. But we're essentially taking this kind of form of information that is, in its raw form, you can't really do much with it. I mean, it's an image. You can read what the image says, but that's not something that is machine readable right. And we're turning it into text, structured text.

Oby: The tool you used-- developed to detect complex document layouts is called Layout Parser. Can you talk about Layout Parser and what makes it unique.

Melissa Dell: Yeah, so it's a Python package for recognizing document layouts.

By which I mean, in the case of newspapers, the layouts are like, headlines, articles, captions, images, and headers-- like, what are the coordinates that determine where those different objects are located on a page. But it could be all sorts of different things. It could be different types of information in a table. It could be table headers, table columns. Essentially, what your layout objects are is going to depend on what your document is.

And so the way that Layout Parser works is, first of all, if you have a document that looks like one of the models that is included with it-- I think we have maybe about seven different models-- then you can just use that off the shelf with four lines of Python code. It will run the layout analysis on your document. It will send individual layout regions to OCR. And then you can visualize and export the results-- again, with just a few lines of Python code.

Of course, you might have documents that don't look like things that we've processed, because there's a vast array of documents. In that case, it's not going to be that satisfactory to use just without any further modification on one of the models we have, because that model hasn't yet learned what your documents look like. And so what you need to do is to give the model some more labeled examples from your set of documents. That means images where you have drawn on the boxes that show the content regions.

So in order to make that more efficient, included with Layout Parser is a software interface for creating those labels, that we've built tools into to try to help it to be as efficient as possible to do that, because obviously, that can be a very tedious process. And so with Layout Parser comes this tool to be able to draw the boxes onto your images so that you can tell the model what you want the model to recognize.

And then you can also use Layout Parser to train your own deep learning model with those data you've annotated, potentially using one of the models that comes with it as a starting point, which is going to be better than starting from scratch. And then once you've trained your custom model, then you can, again, use it to send to OCR, to visualize the results, and to export the results to a desired format that you can then take and do whatever you want with them.

And so that's the basic idea. You can get the code on GitHub. As I mentioned, it's all open source, so maybe you want to add something to it or build it into something else. You can do that, or just download it and use it off the shelf. If you're-- I think if you're a researcher and you kind of want to do something similar to us, you may just want to kind of download it and use it as is.

If you're-- I mean, we've been contacted by some companies that are developing commercial applications and stuff. In that case, you might want to say, like, oh, we can build part of this into our existing code base.

Oby: With so many newspapers out there that aren't digitized, how do you decide where to start?

Melissa Dell: We essentially use content from an off copyright database of newspapers. And we started with front pages and editorial pages. There's something like over half a billion pages in the entire database, which is more data than we could-- that would cost, probably, like $1 million just to process it alone. It's more data than we can deal with, but half of that's ads. So I think by looking at what's on the front page and what's on the editorial page, you see what people thought was important, and you see kind of what their opinions on important issues are.

We are essentially limited by having things that are off copyright, but that is the vast majority of papers prior to 1978, which is how far forward we go, because there's a change in copyright law in 1978 that makes everything automatically copy written. And then that leads to an orphan works problem, where by-- automatically, without having to register, a newspaper is copyright. That newspaper went out of business in 1985. There's no way to access that sort of copyright.

But before 1978, like, the papers that we don't have are in the vein of The New York Times, the Chicago Tribune. It's really, like, the major ones that bother to copyright. So obviously, some stuff was not preserved or was never scanned, but that's just a problem that can't really be dealt with. So I wouldn't want to claim this is 100% representative of all, you know, 200,000 newspapers that have ever been published in the US, but it is over 10,000 newspapers, which is a lot better than having just a few.

Oby: Why is deep learning needed to detect the layouts of documents like newspapers?

Melissa Dell: The alternative would be to try to use some form of rules. Like, you tell the computer what the layout is. But that just tends to be very brittle, to give very poor results, because there are so many different ways that a newspaper could be laid out. And on top of that, it could be skewed when it was scanned. There's just all kinds of noises that enter. So if you try to sit down and hard code all the different ways that newspaper could be laid out and detect that, it's just going to lead to kind of very poor results.

Whereas with deep learning, we give the computer some labeled examples where we've taken images and drawn the layouts on them. And from those labeled examples, it learns how to recognize layouts. And this ends up being a much more kind of robust result that's sort of true-- been true across the board with deep learning. Like in other applications, people may try to use deep learning methods to detect disease for medical imaging.

And again, kind of in the world pre-deep learning where you just have kind of rules to say, OK, this looks like it could be a tumor. That tends not to be very accurate. From using annotated examples, actually, the computer is able to learn how to recognize things in a way that's much more robust than me trying to tell it ex ante how to do it.

Oby: Right. That makes sense. So you talked about maybe different areas it could be used in. So why focus on newspapers? And why are they a valuable resource?

Melissa Dell: Yeah, so I think a couple of reasons for that-- they're a fascinating set of documents in and of themselves. And so you can go and look at The New York Times or The Wall Street Journal, going way back in time. But for historical newspapers that come from different communities, different kind of towns across the US, they're generally just available in these images. That's not accessible to the visually impaired. Researchers can't use that for analysis.

And if we only look at The New York Times and The Wall Street Journal, I mean, that's one particular view of society. That's probably not really representative of the media consumed by a majority of Americans, or kind of their thoughts and their views. And so from the point of the substance, there's just a lot of substance there. We can learn a lot about how communities were thinking about major issues, policy issues, social, cultural issues, like across time.

On top of that, it's just there's a huge amount of data there. We've extracted hundreds of millions of articles and headlines and image caption pairs. And that's also really useful from the standpoint of the methods right, because now we have this very, very large data set of information that you can use to train models and do other things. We essentially use content from an off copyright database of newspapers. And we started with front pages and editorial pages.

There's something like over half a billion pages in the entire database, which is more data than we could-- that would cost probably like $1 million just to process it alone. It's more data than we can deal with. But half of that's ads, so I think by looking at what's on the front page and what's on the editorial page, you see what people thought was important. And you see kind of what their opinions on important issues are.

We are essentially limited by having things that are off copyright, but that is the vast majority of papers prior to 1978, which is how far forward we go, because there's a change in copyright law in 1978 that makes everything automatically copy written. And then that leads to an orphan works problem, where by automatically, without having to register, a newspaper's copyright. That newspaper went out of business in 1985. There's no way to access that sort of copyright.

But before 1978, like, the papers that we don't have are in the vein of The New York Times, The Chicago Tribune. It's really, like, the major ones that bothered to copyright. And obviously, some stuff was not preserved or was never scanned, but that's just a problem that can't really be dealt with. So I wouldn't want to claim this is 100% of all 200,000 newspapers that have ever been published in the US. But it is over 10,000 newspapers, which is a lot better than having just a few.

Oby: So we started off talking about the pilot grant award project you're working on, and the two aims to develop this type of technology and to make a database of newspaper content available. How will people access this database?

Melissa Dell: Yeah, and so we kind of have three distinct parts of the project. And so the first part is a Python package that contains tools for doing learned analysis. And so people can use it to do their own analysis on their own documents. They can use the models that we've pre-trained off the shelf, but also, it has tools for training kind of your own sets of models. We're also working to make the text available to the visually impaired, and kind of trying to figure out the logistics of doing that.

And then finally, we'd like to make some of the majors that we extract from this text more kind of generally available to researchers. And so, like, the full text itself, while the newspapers are off copyright, at least for post-1920, it gets a bit murky, because they could have reproduced something that was copy written. So just putting the full text on the internet, you don't quite know the copyright implications of that. Even though we got-- the scans are off copyright, but Library of Congress had this entire complicated issue with it as well, where it's just kind of messy.

But certainly, like, making majors to research available that we've calculated-- like, what's the sentiment on these policy issues and things like that, is well within copyright regulations.

Oby: So I heard you talking about making the text accessible to the visually impaired. What other applications are there for this technology?

Melissa Dell: I think-- to sort of any type of document where you want to be able to recognize the structures in it. So if you're to go and just put a document into a commercial OCR software, like Google's software, Amazon software, Tesseract, which is an open source software. Unless it has a very simple layout-- like, it's essentially, like, a single column book, those softwares aren't capable of recognizing the document layout.

So like, in the case of our newspapers, you know, they'll tend to have six, seven columns. It'll read it like it's a single column book. The headlines will get interspersed with the articles, with the advertisements, with the captions. And-- but that's true with a lot of other things. Even web readers oftentimes don't work that well if the web page has a complex layout. Researchers often want to digitize tables. Just-- there's a whole vast array of documents out there that exist in the non-machine readable format.

And we need to extract kind of what those structures are. And so that's a big part of our motivation for having made this Python package that's totally open source. Anybody can go to GitHub and they can use it as is, or they can modify it for their own purposes. Right, so if there was a company that's developing a web reader for the visually impaired, they can go and take that code and use-- modify it to exactly their needs. But it's open source, anybody can use it for whatever they want.

If there's a researcher working on a project and they have this firm level records from the 1970s that they need to digitize, or administrative records from the government, or medical records, or whatever it may be, generally, like, the technology of extracting that is the same. But they just may need some labeled examples from their context, and the software is set up to make it efficient to create those and to train a model on them.

Oby: Nice. So you just talked about the technology being open source, meaning the code is freely available. And anyone can add to it and improve it. How would you like to see people using it in the future?

Melissa Dell: I think really in kind of whatever way is useful to them. So certainly, I'd like to get all the research community involved. There's just a ton of applications there. And would really-- like, people, when they-- let's say they have a distinct data set. They have administrative records from the government that are in hard copy, and they're digitizing them. Contribute that model-- not necessarily the data, but the model itself.

And then when other people want to digitize something similar, they can use that, and they won't have to reinvent the wheel. I think, though, it also-- it's also, as I said, potentially useful to people who are developing commercial and other technologies to essentially be able to recognize layouts and documents. And so I think the more people you kind of get involved and that share their work, the more likely there's a model out there that's just going to work for somebody who needs to use it off the shelf.

And then they won't have to create as many of their own examples to train the model, which will be very useful.

Oby: Great. So you're a professor of economics at Harvard. How do you see this research helping your field?

Melissa Dell: For a lot of questions that are, like, major questions in economics-- things like inequality, or how other resources are efficiently allocated across firms. Lots of the major questions, you just can't answer that with highly aggregated data. So you're not going to answer about inequality with state level data, or even county level data, because you've already aggregated away that variation that you care about.

And so for this reason, typically, researchers studying these questions will only use very, very recent, very modern data sets. But these are obviously, like, phenomenon that developed over time. And you can't really necessarily fully understand with a single kind of modern snapshot what leads to inequality, or what leads to the distribution of resources that we have across firms or individuals in society.

And actually, lots of times, the historical data that you need to kind understand things more deeply does exist, but it is trapped in hard copy. And if it's disaggregated, it's way too costly to digitize that manually. And so I'd really like to be able to make data like that available to researchers. I think another challenge is that people tend to-- they're going to study a context where there are data. And those are not necessarily representative of the context that we'd want to study.

So there tends to be data on, like, the US and other very high income countries, where people have had resources to create machine readable data in the first place, or to digitize it. And there's been much less work done on places kind of outside those contexts, which should be important to our understanding of economic phenomena. And so by having a technology that can be used to digitize information, to take raw document images and turn it into structured data at scale, it should make it easier for researchers to study understudied context.

And so you see that in the context of newspapers. Again, as I mentioned before, The New York Times and Wall Street Journal are interesting, but not necessarily representative. Now, you can digitize these small political newspapers from a tiny town in Nebraska, or a historical African-American newspaper. There's just like all kinds of publications out there that are important to understanding things, but they don't have the resources of The New York Times. And so you can't go and download it.

Oby: Definitely, thank you. That is-- I feel like my burning question at this point is what made you interested in this field? I mean, I think you've talked about so many of the different ways this can be used and the implications which I find really, really interesting when you think about-- you mentioned inequity, or equity. And I'm just curious, what brought you here?

Melissa Dell: Yeah, so my work in general focuses on economic growth, economic development, like, mostly outside the US in low income countries. And that's really one of those contexts, as I mentioned, where data might exist, but it probably-- somebody else hasn't already studied it and digitized it. And so just with a lot of questions that I wanted to answer in order to have the data, it requires being able to take documents and scan them and convert those raw images into a structured database that you can use to actually kind of understand your question.

And I think it's just, like, pretty unsatisfactory to try to get things hand entered. It really limits the scale of things. And then you end up spending a lot of time making sure everything's correct, and fixing lots of mistakes. And it can be kind of tedious. And so I really wanted to kind of develop a technology that would make these sorts of contexts easier for people to study, because at the end of the day, the most studied contexts are going to be the ones that are most straightforward to get data for, which really leads to overrepresentation of certain groups in our research.

Oby: Well, thank you so much for joining us. It has been a pleasure to have this conversation with you.

Melissa Dell: Yeah, thank you so much. This has been great.

Brendan: Thank you for listening. If you've enjoyed this podcast, please rate us on iTunes, and help us spread the word about the amazing research taking place across the Harvard community.

Oby: To learn more about the guests on this episode, visit our website, catalyst.harvard.edu/thinkresearch.

[music playing]