On March 13, 2013, Argentine cardinal Jorge Mario Bergoglio was elected pope following the surprise resignation of Pope Benedict XVI.

One year into Francis’ papacy, an analysis by the Pew Research Center finds that the former Jesuit archbishop – who was named Time’s Person of the Year – ranked among the top global newsmakers in major U.S.-based digital news outlets.¹ Media interest in Francis was intense upon his election in March 2013, but it peaked four months later, in July 2013, around the time of his trip to Brazil for World Youth Day. On his flight back to the Vatican, Francis made headlines worldwide when he was quoted as responding to a reporter’s question about gay priests by saying, “Who am I to judge?”

On Twitter – where the tone of conversation often takes a more negative turn – the new pope, who has more than 3.7 million followers, enjoyed mostly positive treatment. An analysis of the Twitter conversation over the course of Francis’ first year finds that positive discussion about him outweighed negative discussion by a ratio of five-to-one. By contrast, Twitter conversation about Benedict during his final 12 months in office tended to have a negative tone. Indeed, there were more than twice as many negative comments as positive ones about Benedict on Twitter in the year before he resigned.

Francis Brings Renewed Visibility to Papacy

By any measure, Francis – in his first year as the bishop of Rome – brought renewed visibility to the papacy.

To assess how much, researchers used General Sentiment technology to measure the number of digital media references to Francis and compared his coverage to a selection of other well-known global figures during the same time period (March 13, 2013-Jan. 31, 2014).² In terms of media mentions, Francis ranked behind only such international figures as U.S. President Barack Obama, South African leader Nelson Mandela, who died in December 2013, and Syrian President Bashar al-Assad.

Francis received nearly 50,000 media mentions during the period studied, more than former U.S. Secretary of State Hillary Clinton, Russian President Vladimir Putin, British Prime Minister David Cameron, Israeli Prime Minister Benjamin Netanyahu and German Chancellor Angela Merkel.

Francis also received far more media mentions than other internationally recognized religious leaders, including Desmond Tutu – the former Anglican bishop who made headlines when fellow South African Nelson Mandela died – and the Dalai Lama.

How did Francis’ level of media attention compare with that of Benedict? Digital news archives going back to Benedict’s election as pope on April 19, 2005, are not readily available. But an analysis of the final two years of Benedict’s papacy shows that he was the subject of far less coverage at the end of his tenure than Francis received in his first year.

In his last two years as pope, Benedict was mentioned 18,317 times per year, on average, in the roughly two dozen digital news sources studied by Pew Research using General Sentiment technology.³ That amounts to just 38% of the 47,738 media mentions Francis received in his first year as pope.

There was a spike in interest in Benedict in February 2013, when he announced that he was resigning as pope. In addition, the former pope drew international media attention when he traveled to Germany in September 2011 and to Latin America in March 2012. And he made headlines for some public statements during the final two years of his papacy, including a prayer for diplomacy in Libya (April 2011) and an appeal for world peace (December 2011).

How did Francis’ year of intense media scrutiny compare with Benedict’s first year (2005-06)? Because the media environment has changed substantially in the past eight years, it is difficult to make exact comparisons.

But past Pew Research analyses of Benedict’s papacy suggests that the former pope generated significant amounts of coverage at times.⁴ For instance, Benedict’s six-day visit to the United States in April 2008 accounted for fully 37% of all religion-focused news coverage during the entire year.

In 2009, two of the top five religion-focused stories in the mainstream media centered on Benedict’s activities, including his visit to the Middle East that year and his pardon of a controversial bishop who had questioned the number of Jews killed in gas chambers during the Holocaust.

And in 2010, the clergy sexual abuse scandal in the Catholic Church reemerged internationally, largely centered on Benedict’s role in addressing the issue. It accounted for about 19% of all religion coverage that year, making it the year’s No. 2 religion story.

News Events that Drove Coverage of Pope Francis

Much of Francis’ first-year coverage came in bursts of attention, focused around key moments rather than in a steady stream of news stories.

Scheduled events drew high rates of coverage, including Francis’ first celebration of a Mass as pope (March 14, 2013), and his first foreign trip (to Brazil in July for World Youth Day). But at no other point did papal coverage spike as high as when Francis offered a comment about the role of gay and lesbian people in the church.

On July 29, the pope answered reporters’ questions for more than an hour on his flight from Brazil back to the Vatican, during which he posited, “If someone is gay and he searches for the Lord and has good will, who am I to judge?” In that month alone, Francis appeared more than 14,000 times in the news sources studied – nearly twice as often as in the later weeks of March, when he was introduced to the world as the new pope.

There was a similar, though smaller, spike of interest in Francis when he laid out his vision for the Catholic Church and his papacy in an interview with Antonio Spadaro, editor in chief of La Civilta Cattolica, published in September. In the interview, conducted on behalf of several major Jesuit publications from around the world, Francis suggested that not all moral teachings of the church are equivalent and that he wanted to “find a new balance.” Although he did not directly mention abortion, gay marriage or contraception in that context, he had referred specifically to those three issues slightly earlier in the interview: “We cannot insist only on issues related to abortion, gay marriage and the use of contraceptive methods. This is not possible.”

The new pope made headlines once again in December, when both Time and The Advocate (a publication aimed at the LGBT community) named him Person of the Year. Francis was mentioned more than 7,000 times that month, about the same as the month he was elected pope.

The Social Media Conversation about the Pope

In his first year, Francis has been received warmly by the public and Catholics alike. But how has Francis’ image played out on social media? An analysis of the Twitter conversation between March 2013 and January 2014 shows a largely positive tone, and one strikingly more positive than that of Benedict in his final year. The positive nature of the discussion about Francis also stands out given a past Pew Research analysis showing that Twitter conversations have a tendency to veer negative.

A Pew Research examination of Twitter using Crimson Hexagon technology yielded roughly 8 million relevant tweets involving Francis during the period studied.  The vast majority of the conversation (85%) about Francis was neutral in tone. Much of it can be attributed to users sharing pieces of news or information about the new pope, or simply referring to Francis in a generally neutral way. (Indeed, past Pew Research studies have found that the pass-along effect – which accounts for much of the neutral content in a Twitter conversation – often makes up the largest portion.) But of the remaining portion of the conversation that contained a discernible tone, 84% was positive and only 16% was negative.

Some users commented on Francis’ priorities: “I am Catholic and I like him for the same reason. #Francis cares for all, especially the poor,” tweeted one. Others were more generally effusive: “Yo shoutout to Catholics, they picked a good Pope this time. Good on you guys.”

Contrast that with the Twitter conversation involving Benedict in his final year, which comprised approximately 5 million relevant tweets. Here, a smaller portion of the Twitter conversation about Benedict was neutral (60% vs. 85% for Francis), and the portion that contained a discernible tone was more negative than positive. Indeed, fully 70% was negative, while 30% was positive.

Negative comments about Benedict on Twitter often criticized him for the church’s positions on the priesthood: “If the Pope thinks ordaining women is a ‘grave crime’ he should blame God for making 51% of ppl so unworthy of priesthood.” Others jabbed at his biography: “how fitting the current pope was a former Nazi youth.”

When Benedict began his own social media presence in December 2012, some tweets were more positive. For example: “Pope Benedict XVI joined Twitter! @Pontifex is his handle. (Welcome!) The way we communicate today is changing. Even the #Vatican gets it!”

The Pew Research Center used two different software platforms to determine the volume of attention paid to Pope Francis and Pope Benedict. To determine the amount of coverage in the 25 most visited news websites, Pew Research used a tool from General Sentiment. To determine the volume of attention on Twitter, Pew Research used software from Crimson Hexagon.

The majority of the data in this report come from the period between March 13, 2012, and Jan 31, 2014.

How Volume in the Media was Determined Using General Sentiment

The Pew Research Center is employing an automated coding platform created by General Sentiment (GS), a social analytics company, to conduct research into the volume of media coverage given to key topics. General Sentiment’s software tracks the volume of mentions for a person, place or subject online during any time period. The platform analyzes over 60 million sources of online content. GS saves all the public content from the RSS feeds of these sites, allowing users to track the volume of the content through natural language processing and text analytics.

This software has been used by a number of news organizations, businesses, and academics.

How General Sentiment Tracks Topics

The basis of General Sentiment’s software is a proprietary computer-learning method that discovers and tracks millions of topics contained in RSS feeds. The process is called “automated topic discovery.”

Each evening, GS’ computers scan all text contained in the millions of RSS feeds they follow and catalog the “topics” discussed each day. Topics can be names, places, hashtags or other common phrases. Most topics are proper nouns.

Once GS’ system catalogues a new topic, it stores information about that topic going forward. From then on, any time the topic appears on the RSS feeds, the system notes that occurrence. This allows for advanced tracking that identifies how often, and where, topics are mentioned online.

For example, imagine a story about a small-town mayor named Jane Rodgers that appeared on the RSS feed of a local newspaper. GS’ computer-learning system would identify the term “Jane Rodgers” as a new topic – if it were the first time that name appeared in public online content – and would then track every instance where “Jane Rodgers” appeared going forward. (In the cases where a name is common, there are advanced Boolean options that help distinguish between different people.)

There are two major benefits to this type of automated topic discovery process as compared to traditional keyword searches. The first is that GS can list the most frequently mentioned topics discussed on any given day, week or month. By definition, keyword searches are only effective when the researchers know what they are looking for. Some news stories, however, emerge even though researchers may be unaware of them. GS lists the most popular topics on a daily basis ensuring that we are not missing any major stories.

The second benefit is that the GS search platform allows for easy combinations of all relevant word variations of a topic. For example, when searching for mentions of President Barack Obama, the software also includes references that read “Pres. Obama,” “Barack,” “President Obama,” along with pronouns that are clearly referring to the same person. These types of detailed searches are difficult with traditional keyword searches.

The unit of measure for the GS volume statistics is the reference or mention as opposed to the story or sentence. That means that if a person is mentioned five times within the same post or article, it counts as five mentions.

Universe

An RSS feed is a web format that allows sites to syndicate frequently updated information automatically. General Sentiment tracks the RSS feeds of over 60 million online sources including national media, local media, blogs and social media. Users can sign up to view RSS feeds of sites, and see anytime new content appears. RSS feeds are also good tools for services like GS to collect large amounts of content in a systematic way.

For this particular report, Pew Research searched the 25 most-visited news sites according to audience data from Hitwise and comScore. The sites were as follows:

usnews.com abcnews.com nbcnews.com huffingtonpost.com newyorker.com theguardian.com examiner.com thedailybeast.com slate.com nytimes.com msn.com newser.com news.google.com msnbc.com usatoday.com cnn.com news.yahoo.com dailyfinance.com cbsnews.com washingtonpost.com theatlantic.com foxnews.com bbc.co.uk nydailynews.com topix.net

GS only tracks text available online through a site’s RSS feed. It does not include video or television transcripts. It also excludes duplicated stories that appear in more than one location. In other words, if two stories on different sites have the exact same text, only one of those stories will be included. If there are any differences at all between the texts of two stories – even one word change – then the two stories will both be included. If a specific wire story, for example, appears as a 1,000-word piece on one site, but as an edited version of 800 words on another, both versions of those stories are included in GS’ sample. This is done so that posts on aggregator sites that link to stories from other locations do not get counted repeatedly, Counting repeated stories appearing on different pages could have a disproportionate influence on results. This “deduplication” process is less of an issue when Pew Research examines a small group of major sites, since large news producers tend to include fewer stories produced by wire services and other outlets.

Unlike some other automated tools, GS has the capability to separate the text of reader comments from the text of news articles. This assures that the content measured is text produced by the news organizations and is not confused with responses by readers.

Testing

Pew Research spent four months conducting various tests of GS. In particular, researchers were interested in testing whether GS accurately measures the frequencies of topics. To answer this question, researchers compared results from GS with results based on the work of human coders reading the RSS feeds of several major news outlets.

For example, GS said that on USA Today’s RSS feed on Oct. 2, there were eight references to “Kenya” and four to “Christopher Cruz” (who was involved in a New York City biker attack). That was precisely the same numbers researchers found when looking through USA Today’s RSS feed.

Researchers repeated this same process for a number of other topics and websites including CNN.com, Washingtonpost.com and several local TV sites. In each instance, we were able to match up GS results and the RSS feeds.

A small caveat is that GS is dependent on what is freely available on RSS feeds. A few sites, such as the New York Times, put some of their content behind a paywall. In those instances, only free content is included in GS’ sample.

Pew Research will continually test GS, along with other automated measurement tools, in order to find the most accurate and valid methods for advancing the Center’s research agenda.

How Volume and Sentiment on Twitter was Determined Using Crimson Hexagon

The analysis of Twitter employed media research methods that combined Pew Research’s content analysis rules with computer coding software developed by Crimson Hexagon (CH). This report is based on examinations of more than 13 million tweets.

Crimson Hexagon is a software platform that identifies statistical patterns in words used in online texts. Researchers enter key terms using Boolean search logic so the software can identify relevant material to analyze. Pew Research draws its analysis sample from all public Twitter posts. Then a researcher trains the software to classify documents using examples from those collected posts. Finally, the software classifies the rest of the online content according to the patterns derived during the training.

Two different analyses were conducted for this project. The Boolean search used to identify tweets about Pope Benedict was (Pope OR Benedict OR Ratzinger). The Boolean search used to identify tweets about Pope Francis was (Pope OR Francis OR Bergoglio).

Tone of Twitter Response

Reaction on Twitter can often be at wide variance with public opinion. A Pew Research Center analysis last March compared the results of national polls to the tone of tweets about eight major news events and found that the Twitter conversation can be more liberal than survey responses, while at other times it is more conservative. During the 2012 presidential campaign, Twitter sentiment was much more critical of Republican candidate Mitt Romney than of President Obama.

Researchers classified more than 250 documents in order to “train” these specific Crimson Hexagon monitors. All documents were put into one of four categories: positive, neutral, negative or jokes. A tweet was considered positive if it clearly praised the pope in question. A tweet was considered negative if it was clearly critical of that pope.

CH monitors examine the entire discussion in the aggregate. To do that, the algorithm breaks up all relevant texts into subsections. Rather than the dividing each story, paragraph, sentence or word, CH treats the “assertion” as the unit of measurement. Thus, posts are divided up by the computer algorithm. Consequently, the results are not expressed in percent of newshole or percent of stories. Instead, the results are the percent of assertions out of the entire body of stories identified by the original Boolean search terms. We refer to the entire collection of assertions as the “conversation.”