How The New York Times uses predictive analytics algorithms
by Ed Burns
Most print media companies have struggled to make money in the 21st century, but The New York Times is using predictive analytics tools to gain a competitive edge.
In the middle part of the last decade, when the Internet replaced print publications as the primary source of news for many people, revenue at most news organizations plummeted. Advertisers were less willing to pay high rates for space in print newspapers and online ads were less proven. This left news organizations scrambling.
Many still have not adjusted to the new business of news. But The New York Times, for one, is starting to make predictive analytics a major part of its business model in an effort to adjust to the modern realities. From trying to get more people to subscribe to promoting articles on social media, the news organization is letting predictive models guide many of its business decisions, and it's hoping this approach will make it as successful in the 21st century as it was in the last.
In a presentation at the Predictive Analytics World conference in Boston, the Times' chief data scientist, Chris Wiggins, talked about how he and his team use predictive analytics algorithms to do things such as funnel analysis to see how people become subscribers, and how to influence more to do so. They also use natural language processing to understand content topics that generate the most reader engagement, so marketing teams can know what types of articles to promote.
An outsider steps into the news
Wiggins may seem like a strange choice to lead the data operations at a newspaper. He has a Ph.D. in theoretical physics and has spent most of his career in academia doing biological research. But most of his research has taken advantage of machine learning and other advanced statistical methods. He said applying these types of predictive analytics models to the traditional business of newspapers is not so different than using them in biology, which historically has not been an exceptionally data-driven field of science.
For someone who has worked for years in higher education, Wiggins takes a decidedly unacademic approach to his work at the Times. He said he makes his team avoid projects that have only theoretical business value and instead focus on things that are clearly useful.
"It should be clear to everyone in the company why something we're doing is valuable to the company," Wiggins said. "You should only do things that are actionable."
To get to this point, Wiggins has built a team that leans more toward general computer science skills, rather than statistics. He said this tack is helpful in taking a model from development to production. Having people who know programming means they can build an app or Web portal more easily than some data scientists. They use Python for most projects, which is generally more programming-intensive, rather than a stats-centric predictive analytics tool like R.
"[Python] draws in more people that skew more computer science, but it also ensures when we're done with something, it doesn't die a cold death as a slide deck," Wiggins said.
Data does not make editorial decisions
Even as the Times scores some wins with predictive analytics algorithms, there is one area Wiggins said his team will never infiltrate: the editorial department. He acknowledged that there are lots of other news organizations out there that use analytics to drive editorial decisions, but he said it's important to know when to take a step back. Right now, the quality of the paper's editorial judgment is the primary thing that sets it apart from many competitors. It's hard to see how that aspect could improve by making it more data-driven.
But in most other areas, analytics is helping the Times become a 21st century news organization. Wiggins said the key is leading a change in mind-set that makes people look to data first to answer their questions.
"Data is more and more getting recognized at The New York Times as a first-class citizen," he said.
- Get the team involved when building predictive analytics models
- Big data adds a new wrinkle to predictive modeling
- Speed up predictive model building for best results
- Business focus is key when applying predictive analytics models
- Building predictive analytics models takes a team effort
- HP Vertica users put limits on using Hadoop for analytics
- Building predictive models more about value than glamour