PSA: The Medill Map People Are Taking Corrections

By Alice Dreger

I’m happy to report we’ve broken through to real communication with the folks at the Medill Local News map project, and they’re now taking our requests for corrections seriously. 

Since we started raising concerns about the map back in January, the response of Zach Metzger, then the project’s Database Manager, has been “You just don’t understand our methodology.” While the project’s methodology does throw up yellow flags – more on that below – that hasn’t been the big problem.

The big problem has been the errors. We’ve previously reported on dozens of mapping mistakes here, here, here and here. 

This week, I got tired of trying to get action from Metzger, who was recently promoted to Director of the State of Local News Project. (Metzger is a Ph.D. candidate in English with no evident background in database management or journalism.) So, I wrote to his boss, Tim Franklin, pointing out specific errors and reminding him that we all share the same values and goals – accuracy being chief among them.

That appeal worked.

Let’s run through some of Metzger’s specific responses before we get to the methodology pickle.

I pointed out in my email to Franklin an error we have not earlier reported – that the Baltimore Sun and Baltimore Banner, two of the most prominent local news providers in that area, are both missing from the map for that county. (This error was noted to me by Amos Gelb, publisher of Baltimore Witness, which is also not on the map.) 

“The Baltimore Sun and Baltimore Banner are both in our database and the Sun appears in our Maryland state map,” Metzger answered. So, what gives? 

“Upon investigating the issue, it seems that during our last mapping update, there was a geographical coding error (in separating Baltimore county from Baltimore City) that caused the city to not display correctly. We are working with our development team to rectify this.”

That coding error also appears to explain problems discovered in northern Virginia by Scott Brodbeck of Arlington Now.

What about missing local public broadcasting stations (NPR affiliates) that deliver local news? I gave as an example 91.3 WLVR in Pennsylvania’s Lehigh Valley. (This error was shared with us by Yoni Greenbaum, Chief Operating Officer for Univest Public Media Center in Bethlehem.)

Says Metzger, “For public broadcasting stations, much of our data comes from CPB [the Corporation for Public Broadcasting]. We currently have WDIY that shares some infrastructure with WLVR. We will be adding in WLVR in our next update as part of our regular public radio research.”

And how about the problem of the map missing the Wayne & Garfield County Insider in Utah? Erica Walz, who runs that newspaper, told us in May that she had “contacted the Medill people FOUR times about the fact that we exist and that they have repeatedly missed/neglected our existence in their study.” The Medill map displayed a total news desert for Wayne County despite a real newspaper that's been surviving there for 30 years.

Metzger replies, “We have been in close contact with Erica, and the Wayne & Garfield County Insider is indeed in our database and on the map for Utah. My last email with Erica was on 5/9.” 

Perhaps not coincidentally, May 9 was the date we published Erica’s lament.

Why was the Daytona Times listed as being in Hillsborough County, on the wrong side of the state of Florida?

“This has been corrected,” Metzger says. “The mailing address for the Daytona Times caused the county coding error.”

How about the double counting caused by including The Toledo Journal once under that name and once under “Toledo Journal”?

“This has also been corrected.”

As to the problem of a number of cases of ethnic outlets either not being counted as ethnic or not being counted at all, Metzger had this to say:

“We have added several ethnic media outlets based off of your previous suggestions, including the Arab American News, the Detroit Jewish News, AfroLA, and others. As we have mentioned before, in last year's list of ethnic media, we tried to establish a baseline by reexamining our previous 2018 list, with the intention of building off of that going forwards. This is an area that we hope to continue to expand in the future.”

Mind you, this is all just a sampling of the errors we've discovered through crowdsourcing. But in his email from yesterday, Metzger assured us all that, despite my previous experience of being brushed off and told I just don’t understand the methodology, the door is now open to corrections.

“We would be happy to review any other suggested changes you have, and we will make edits as needed as part of our continuous updates,” he writes. “Since the beginning of 2024, we have made over 70 edits to our database, including reactions to real-time closures, mergers, or frequency changes, as well as corrections to source data. In your email, you mention that you have been contacted by ‘a few dozen’ people; if you would like to have any of those folks reach out to us, we can look into any issues and address their concerns directly.”

He also noted, “As part of our research, we are also always looking for ways to make our data visually clearer and more transparent, and have been working on refinements to the way that our maps are presented and used. It should be noted however, that we are using a third party development team for our maps, which means that changes to the maps as well as the underlying data may not be immediately reflected, and glitches can occur in how the maps are rendered.”

So, what about the methodology question?

If you look at the Medill mapping project page, you most definitely get the impression that the project is carefully mapping where there is and isn’t local news. Heck, the page is entitled “The State of Local News 2023: Vanishing Newspapers, Digital Divides, and Reaching Underserved Communities,” as if it really covers all of that nationwide.

But, in fact, if you click on the tiny “methodology” link (see red arrow below), you’ll discover the project has a lot of limitations that aren’t evident from the map and the big claims. 

There’s a fair bit of subjectivity going into the map, as unnamed individuals at the Local News project are deciding which news outlets fit the FCC’s definition from 2012 of serving “critical information needs.” If the Medill screeners think you don’t fit that 12-year-old definition, you’re probably not going to make the map.

Lots more organizations never make it to the Medill screeners because their data is being culled from other lists, like association memberships. Plenty of independent newspapers don't belong to many or even a single association. (People who run independent papers have a funny habit of staying independent.) Under Medill's methodology, they don't necessarily exist.

In his communication to me, Metzger tacitly acknowledged that the Medill mapping project isn’t really presenting what people think it is. He wrote, “Medill is helping to lead a national consortium exploring methods for a nation-wide ecosystem mapping project.” In other words, that’s not what the Medill project is really doing now.

I asked Dr. Alice Fleerackers to have a look at the website of the Medill project to tell me what she, as a data presentation expert, sees at the Medill map site.

Fleerackers is currently a Postdoctoral Fellow at the University of British Columbia’s School of Journalism, Writing, and Media and at Simon Fraser University’s Scholarly Communications Lab. 

Fleerackers noted that the Medill map “is public facing and the landing page makes bold claims that are framed as if they are certain and settled facts. This means there is a lot of potential to mislead audiences who don’t have the expertise, time, or resources to vet the science themselves.”

She suggested the project would be improved substantially if the researchers “note [the study’s] limitations on the landing page for the map, and not just somewhere deep in the report associated with it.”

I told Fleerackers that many of us wish the Medill team would also talk to those of us “on the ground” to improve their data. She liked the concept:

“Your idea of providing a way for journalists and, ideally, members of local communities to share input could significantly strengthen the research. After all, they are the people best positioned to validate the findings of this important and ambitious project.” 

Got a correction to the map? Mail it to the Medill team here.

Alice Dreger is a journalist, historian, and the publisher of Local News Blues. She founded East Lansing Info, a nonprofit digital investigative news service, and ran the operation for about ten years. Read more at the Local News Blues contributors page.