Bending the Curve with Data: The Role of a Scientific Referee

When working with the public as a scientist, you often encounter conflict between parties that are asking scientific questions. The conflict can be divided into two activities, analysis and deliberation. One can assume these are sequential, but that is not the case. Generally, deliberation is paired with analysis as incoming information changes solutions for the problem. However, the role of a scientist in these disputes is problematic because each party will typically select the data that best fits its opinion, and a scientist needs to remain engaged during the deliberative process. Few scientists want to play advocate; they were trained to be unbiased evaluators. Many scientists are uncomfortable in an advocacy role, because serving in this role may require them to choose the data that supports one team over another. This has resulted in scientists stepping back from policy debates, as it has become more problematic for their careers. Those that are in a governmental role receive pressure from those in charge of the government at the time.

In the absence of a neutral, rational voice, science devolves into an “opinion” to be discerned from an internet poll. I think a scientist could provide a role in giving a voice to data: presenting what is known and how to deal with uncertainties in a neutral way as a referee ruling on what the data says (and doesn’t say) about the issue at hand. The “scientific referee” is a voice for the data. By indicating that is the role, it becomes difficult to degrade the data by degrading the scientist as biased. The scientific referee also allows a much higher level of data discussion to occur during deliberation.

There is a concept that scientists working at universities and government organizations already play this role, but the scientific referee has a distinct difference. By definition, they are not part of the game for scoring points. They can’t support the science being done as a cheerleader or player working on the science. They also can’t cheerlead or play for an agency who is responsible for policy. If there are signs that the referee is on the take, or more likely in this role, not being neutral by being pressured, a new referee is required.

In watching the COVID-19 virus issue play out across the United States in Spring 2020, I said a scientific referee would be required. My wife said that Anthony Fauci, NIAID Director, was playing that role for the United States. I explained that he can’t be the referee, as he needs to support the policy that the government is taking, whether that is supported by data or not. If he doesn’t support the administration, he will be diminished in his role, even if his approach is to accurately support the data in an unbiased way. One would hope all teams in a policy debate would not cherry pick their data or bury data that might offend their agency, but that generally isn’t possible. The referee can’t be an agency scientist or spokesperson. Likewise, an academic can’t be the referee if their research is in the discussion or if they are applying for grant dollars. The voice for the data must be explicitly outside of the deliberation in some regard. I explain the rules for this in more detail below.

In 2014, I was living in Oklahoma while it was in the middle of the greatest change in seismicity in history, from two earthquakes a year, to over 900 in 2016 (Fig 1). Over a hundred different petroleum companies were operating across a range of hundreds of square kilometers and seismicity was increasing rapidly. I was asked by my university library to talk about the science of the issue. That talk led to many others and my testing of an alternative role for a scientist, that of a scientific referee. In this role, I didn’t play advocate and was not responsible for the problem’s cause, management, or regulation. I presented the certainties and uncertainties in the research as neutrally as I could to allow the public debate between various stakeholders to be played out as fairly as possible. I don’t consider scientific conflicts as merely “games,” but the analogy of a referee is the best one I could find to describe the role as I experienced it and lived it. That process came by accepting talks or media interviews to provide a voice to the data. I answered questions via email and sent scientific papers and data to those that requested it, whether the recipients were scientific experts or not.

Over time, as I gave talks and interviews, I developed this set of rules, not for the game of debate between the advocates for different outcomes, but for the referee to follow. There was a lot of learning along the way, and the rules had to be tested, but I think the rules provide a framework to deal with scientific conflicts in a more effective way. I numbered the rules to keep track of them, but they do not go in any particular order of importance. If you find a scientific conflict or battle brewing in your own backyard or country that needs a good referee, I hope you find these “rules” helpful.

Rule 1: Tell them who you are.

I gave talks to citizen groups, Lions Clubs, oil companies, lawyers, and environmental activists. I was interviewed by newspapers, television stations, and documentary film makers. I started by telling them who I was, why I was there and what my role was. I was a full tenured professor at a university and thus wasn’t constrained the way the companies who had liability or the regulators who had responsibilities to the companies and the public. I only needed to be responsible to both groups about what was known, and importantly, what was not. I was the voice for the data, instead of for a position. I asked them to tell me if they felt I wasn’t being neutral, or to let me know what I wasn’t presenting in an impartial manner. I once was accused of being “overly neutral.” At various times along the way, each side thought I was working for the other, but after 6-12 months, they couldn’t find evidence that I was doing something other than the role of a referee as stated. In a game setting, people yell at the referee when a call doesn’t go their way, and people didn’t like if the data didn’t support their side, but the offer of incorporating new information if the data were updated in publications was powerful in fostering people to have faith in my neutrality.

Rule 2: Be two kinds of expert, not four.

I found two kinds of expertise came in handy. First, being a subject matter expert, in this case in geology, hydrogeology, and geophysics. Being able to talk about the peer reviewed information of a topic is essential as a scientific referee. I had an unusual background related to induced seismicity as a hydrogeologist who is also a certified professional geophysicist working at a geology department with a long history in the oil industry. As people would challenge my background, they would be confronted with a sufficiently broad background, proving that I could answer general as well as specific detailed questions. With this background, I was also a second kind of expert. I had sufficient background that I could be an expert witness in the legal side of the situation. As a graduate student, I had a course on Evidence in the Courtroom. By framing my talks as not a scientific proof, but as what the legal framework would be relative to the scientific uncertainty, lawyers and politicians had context for the scientific data. You will need some expertise in the legal and regulatory framework to be an effective referee. I had some in at the start and learned much more as I went.

The two experts I avoided being was the guy trying to fund his research and the guy who is presenting his scientific papers. The guy who is writing grants gets accused of “fundraising his research”. I heard that comment, and would publicly state that I was being sure not to write grants on the topics to maintain the referee status. I’m not sure if this is good academic business sense, but it is necessary to be a neutral referee. The work I presented was not my own. I didn’t have to defend that my work was valid, I only had to convey the certainty and uncertainty of the available peer reviewed literature. I could present the debates that existed in the literature, and indicate where the science was highly certain. I avoided making any graphs of my own, but I would colorize existing literature to make it clearer than an old black and white complex scientific diagram.

Rule 3: Stay to the end.

Most scientific controversies last several years, not a few days. Giving a single talk, no matter how well written or presented, will not provide what the stakeholders need during the controversy. Expect to be giving at least a talk a month during the referee period, as a minimum (Fig 1). There will be travel, with talks at all times of the day on any day of the week. This will take time. The talk slides need to be updated, and some slides will need continuous updates for each talk. While at a presentation, you may sit through a number of presentations before and after the referee, but you need to allocate time to stay to the end of the meeting. People learn the scientific referee concept and ask a lot of questions before and between public presentations. Lots of interesting information was provided when most of the audience had left.

Figure 1. Scientific referee activities compared to felt earthquakes in Oklahoma for one unpaid, untrained scientific referee, 2006-2019.

Rule 4: Social media

I’ve never had a Facebook, Twitter, or any other social media account, unless LinkedIn counts. I don’t post things, comment on things, or take pictures of my dinner to show my friends. I have traditionally said dumb things at random times. Memorializing those comments on social media seems like a bad idea to me. However, as a referee, social media had a big place in the debate. I trended twice on Facebook (my students say) as people were using videos of my presentations or interviews that I did with the media to “fight” about the science. Instead of throwing each other a scientific paper like trained scientist might do during a debate, people throw online videos of “experts” discussing the topic. As a guy who started out explaining the referee concept, my videos became tools people used in the debate. My biggest surprise was finding out that a little 17-minute interview with a web news agency had 50,000 views after only a couple of months. It was my first time presenting to 50,000 people about peer reviewed science.

Rule 5: What do you look like to them?

How you look and how you present yourself is important to how you are perceived by the public, the media and the stakeholders.  (This ties into social media because how you look not only affects your direct audience, but the public your audience chooses to share you and your information with.) I found that if I dressed too casually, I wasn’t a serious expert. If I dressed highly formal, I must be working for an oil company. I ended up in a set of corduroy sport coats with dress pants and a tie that made me into a scientist in people’s visual world. A referee in basketball has a uniform to let you know his role; same thing here. I’m also tall with a deep voice, so that probably helps with seeming authoritative.

Rule 6: Know what the media know

I became overly knowledgeable about where lighting and cameras could go in my office. I met US, Dutch, Korean, German, French, and other media interested in Oklahoma seismicity. The lesson I learned is that the majority were interested in presenting the story as accurately as possible. The ones that didn’t fit this pattern were telephone interviews which I found often were “shopping” for a quote. Most didn’t, but a couple wanted me to provide a colorful statement for their story, and I often unintendedly did. It would have been great if someone provided some media training, but I had to fly a bit blind in this realm. If I asked, the media folks were glad to provide advice. Learning about how to communicate on and off the record is valuable (though “off the record” is not necessarily off the record). Learning how FOIA (Freedom of Information Act) or similar rules function is incredibly valuable. Knowing what is accessible to the press allows the flow of information to be utilized to improve understanding for the situation.

Often the media wanted to know which team you were on. They didn’t have a context for a scientific referee. So, I over communicated the concept (which was almost enough to be useful). The fact that I wasn’t selling a book and didn’t have a social media presence actually helped (although it confused them). My impression is that most of the people they talked to want to have their own TV show or were busy shopping a book. The referee concept was confusing to them, but they knew they wanted a base set of facts, so I told them that I was a source for those things. This resulted in a lot of follow up calls as reporters learned the science that they were reporting on.

Rule 7: Don’t leave the mountain.

If you someday go to the Himalayas, climb a high peak, and go into an ornate temple, the person in funny robes inside may provide you with valuable life advice. If that same person approached you on the corner of your street, you might walk faster to get away from the nut. If asked to give a talk as a referee, you should take the opportunity. If you see people failing at resolving an issue, trying to fix the situation by approaching them with data to “fix” it; you will fail. It ends up with a feeling of advocacy for one side, or that you are just somewhat crazy. Your job as a referee is not to alter the speed of play during deliberation, even though you might think you could or should. The hurdles that slow deliberative processes usually clear within a month anyways.  Main lesson is to make sure people want to hear some facts when you provide them.

Rule 8: No pros, no cons; give fact-supported options

I learned this from my partners in the extension group at Oklahoma State. If you state available choices as pros or cons, you are implying that one choice is good and one is bad. You need to state what choices are available to deal with the situation, even choices you might consider “bad options.” So, list the options and explain what is the most likely expected outcome. I was often asked why certain options weren’t selected, but couldn’t provide an answer. Why simple things don’t happen sometimes is hard for a referee to answer. “Why did he miss an easy layup?” is something the referee doesn’t know. If the audience offers an option you didn’t think of, add it to the next talk.

Rule 9: No one knows the Referee

For your favorite sport, name for me a famous referee. If you can, I expect you are a true sports fanatic. In most cases, referees are anonymous beings that the crowd only likes if the call goes their way. Being a referee isn’t a popularity contest, because if it was, you’d lose.  People are not going to like you; they aren’t going to like what you have to say and they are going to really hate that you aren’t on their “side” of the debate. If you want to walk away from a talk being liked by everyone, being a referee is not for you. People will respect a referee who makes fair calls, but the same folks might call you names during the game.

Rule 10: These aren’t science slides.

If you have a slide with two graphs on it, you are not a referee. If you are defining a vector or an integral, you are not a referee. Have a sales or marketing person who knows nothing about your science look at the slides. Be prepared to have most things removed. Additionally, you may understand the concept through an equation or a graphic that explains the science, but that isn’t going to help with the public. Some will advise you to talk down to the public, don’t. The public is smart; they just haven’t spent 20 years studying your science; jargon and math that is for your discipline is simply a foreign language. An English-speaking audience would not learn much if you talked to them in Japanese. Your discipline’s terms are a foreign language to them.

One example I can give is explaining how an increase in fluid pressure can cause a fault to slip. It is a difficult concept for people to understand that a small change in pressure away from an injection well can move a giant piece of rock. There are a few diagrams available in the literature to illustrate this concept with a block diagram of a fault with the stress tensor described, as well as labels of various other components. You can simplify this diagram as much as you want, but it still requires the audience to think in 3D block diagrams (they don’t), to understand stress tensors (they don’t), and to understand your point (they won’t). I replaced the diagram with an Australian photo of a guy trying to sell a granite sphere that rotated when the fountain was turned on. I indicated that the rock weighed about 10,000 kilos and was moved by a ¾ hp pump moving a couple of liters of water a minute. The visual of a small fluid amount moving a rock was what was required and many people have seen these fountains in real life. The image was relatable to the public while still conveying a scientific concept.

Additionally, if you are the referee, you should only provide the data that have been peer reviewed. If there is a paper that you don’t think accurately reflects the situation, but one side is using it, you need to talk about why you don’t think it is accurate for the situation. In my case, people were hypothesizing earthquakes caused by drought, floods, droughts followed by floods, etc. They also indicated that no evidence was available for other mechanisms. If I only presented the best science for the situation and didn’t talk about other hypotheses out in the public, it would look like advocacy with one team being “favored.” You need to talk about the crazy options that people are suggesting on the news.

Rule 11: The regulator is not the referee.

Some people think it is the job of the regulator or a company to provide the information to the public about a situation. This does not work well, since the public typically sees the company as just covering their legal hide and the regulators as paid off by the company. Both could be trying to do as fair and impartial a job as possible, but public perception will rule. As a tenured university professor, I could say things that the companies or the regulators could not. Depending on what I needed to say, they were made happy or unhappy, but what was presented were the data and peer reviewed information that was available.

Rule 12: The referee is not the head patter.

The role of the referee is not to calm the public (although it is nice if you can). The role of the referee is to elevate the discussion of the science to the appropriate level. I sent peer reviewed publications to industrial people, government employees, and farmers. I would have a retiree ask me about the latest research in induced seismicity when their background was electronics or farming. My job as I saw it wasn’t to pat them on the head and tell them good people are working on the problem. If people aren’t confident in the decision makers, they want to know what the data really say. People didn’t know who to call about particular issues; I gave them the phone number or email. People didn’t know how to register their complaints or log their earthquakes; I told them. Sometimes they wanted a sense that good decisions were being made by those in power; having a referee in play made them think the game was fairer.

Rule 13: The referee has a community of support.

While giving talks and interviews, I utilized the scientific method in asking for advice and data from the scientific community. This ranged from what paper is best for saying this well-known fact in the simplest way, to looking for other ways people have explained complex things. The scientific community was very supportive of giving a voice to the data. They let me know when new science was published that might modify the answers or how to defend against random unsupported internet hypotheses.

Rule 14: Somebody should pay a referee.

Being a scientific referee is a huge time commitment. While valuable to all parties involved, there is no framework to fund this function. If people find this social structure useful over time, funding a referee for time and travel costs would be useful to encourage scientists to participate without going broke. Additionally, having university give tenured professors credit for referee activities would be useful to support the communities they are located near. The source of money needs to be evaluated to make sure that it is paying the referee, not paying off the referee, but is required to have someone dedicate the time required for the situation. (P.S. Thanks to the hosts for all the cookies and coffee I ate at my presentations). This is definitely not a spokesperson job at a government agency. There are lots of examples of that framework failing to be perceived as neutral or speaking freely.

Conclusion

Almost all modern societal decisions involve the intersection of scientific analysis of data with the deliberation of what impacts are acceptable for a community. The role of a scientific referee is valuable for making sure such a decision is made based on scientific data when possible. Additionally, the referee ensures that the data being used are uniform among the parties, and that they have been tested scientifically. The social model of a scientific referee would be a useful way for people trained in unbiased analysis of data to interact with groups trained in advocacy of a particular policy viewpoint; to effectively allow those groups to make evidence-based decisions that fairly represent both the known science and the desires of the stakeholders.

On a personal basis, I found being a referee pretty fulfilling work, though fairly expensive financially. I could try to make sure people knew the scientific facts about the problem, without trying to figure out how to be a full politician. As I scientist, I’m trained to be a lousy politician. It was valuable to have some commercial marketing experience to see science from a public perspective. I also found it very valuable to have a science in the courtroom class during my Ph.D. to figure out how lawmakers thought about some of the data. Oklahoma seismicity has heading back down with a relatively balanced approach between public and industry interests. The state bent the curve. The optimal solution might have been a better one, but I think a pragmatic one prevailed that was based in science. I didn’t build the science that was utilized, nor work in policy; but as a scientific referee, I gave the data and science a voice to allow the game to be played a little more fairly.

Acknowledgements

Thanks to Veronica Reed, Stacy Timmons, and Johnson Grimm-Bridgwater for reviewing the work. Thanks also to all the folks on all sides of the issue in Oklahoma that assisted with making a scientific referee possible.