Today’s post is the third of three in a series from Laura Saunders, a professor of Library and Information Science at Simmons University. The first part of her story, Just trying to get them to think about the nitty gritty of the process, unpacked her approach to teaching peer review in the classroom. In the second, Over time I just learned how to give better feedback in general, Laura shared her reflections on how she learned to perform the work of a referee. And in this third and final story part, Laura and I talk about the tension between her approach to teaching peer review and her practice of it.
Laura Saunders
Pronouns: She/her/hers
Position: Associate Professor, Simmons University School of Library and Information Science
Fun Fact: Laura likes to sing around the house, and sometimes her dog joins in.
Emily: Earlier you said something about how you tell students what peer review is for. I would like to hear more about what you think that actually is. Why should we be doing peer review and what is its function? What should it be for?
That’s a good question. What I usually tell the students it that it’s supposed to be a quality check so that it’s supposed to be that people with some expertise in the general field of the article or the book are able to look at the manuscript and say first of all overall this is good quality, but in particular to really look at things like what methods were used. Again, assuming this is a research article: What methods were used to study the research questions? What sort of analysis was used, were these appropriate to the questions that were being asked, were they implemented correctly or appropriately? And then whatever conclusions or inferences are drawn from that data, do they seem to line up with the data? So the idea is that someone who should know enough about all of these questions is able to look at that and make some call about whether all of those things were done well enough. But like I said I think that it doesn’t always work the way it’s supposed to.
Over time I think I have more and more tried to be more almost like a mentor…
Emily: Before, when you talked about how you approach refereeing, it doesn’t sound to me like you approach it as just a quality check. It sounds to me like you want to be helpful and collaborative. Does that ring true for you? So you’re telling your students that it’s about this quality check, but in practice you’re doing something different.
Actually that’s a really good point and I think you’re right that my understanding of the peer review, and so again how I relate that to my students is that it’s all about this quality check. Trying to make sure that things don’t get to print that shouldn’t be in print because there are major issues with them. But I think you’re absolutely right. That especially over time I think I have more and more tried to be more almost like a mentor and taking it on almost like a mentoring of trying to give some really helpful feedback with people with this sense of I really want you to get published. Here are the things that I think will help and I don’t know that I’ve really talked about that aspect with it to my students. And yet it’s not like I think that I’m alone in how I approach that either. Even though it wasn’t necessarily how it was ever presented to me either. I think that most of us do sort of try to take on that role and I think that’s important and probably something we should talk about more.
Emily: I’ve never heard someone say that they actually just do a quality check. At least in our field. Most people want to be supportive. They want to be collaborative. You said you had that one grant review and it was like two words. Maybe that person was just doing a quality check. Not to say that your work was of poor quality, but I’m just trying to think maybe their understanding of peer review is different than yours and that’s why. What are you actually expecting from this process?
Yeah, I guess that’s a good point and like I said, we’re never really taught how to do this. At least in my experiences we’re not taught how to do it and we don’t have a really lengthy conversation about it and so I think yeah, I’m sure there’s a lot of different perspectives on what’s the right way to do it or what are we supposed to be focusing on and that’s really problematic.
Let me put it this way. It’s not that I have extensively read the guidelines and aims and scopes and things like that of all these different journals, but I will say of the ones that I’ve read portal: Libraries and the Academy is the only one where I’ve actually seen them say that they take a mentoring approach to their feedback. So they actually have it written into their guidelines which I think is really pretty cool.
Emily: Does it make you feel differently about what you think peer review should do and what it should be for?
Yeah, like I said now that we’ve been talking about it I think that I will be talking about this a little bit differently in my classes, because I think it’s important like you said, at least in our field, to talk to people about this and to just understand the peer review a little bit more broadly and to think about the extent to which we do or do not approach it as mentoring and constructive feedback. I think it’s also important because there’s a push, especially for academic librarians who are on tenure-track, to publish and yet we don’t teach them how to do that usually in our master’s programs. And so if you happen to come from a social science background or a hard science background then you might be all set even without the MLIS preparing you for that. But for a lot of the other students—and something like 70 percent of our students at Simmons come from a humanities background—it can be really challenging in a lot of cases to go out and rate those articles not having been taught how to do it. And so if the peer reviewers take a little bit more of a mentoring approach you might actually really be doing a bigger service to the field even than just what the review itself does.
Emily: I don’t know if you’ve seen any of these peer reviewer courses that have been offered by some of these commercial entities like Publons. I guess I just wondered if you had any thoughts about what it means for a company or a publisher, a commercial publisher, to be filling the gap?
Yeah, it’s a great question because obviously there’s a part of me that there is a gap and somebody does need to address it so I think that’s a good thing. Without having taken any of the courses, I guess I would wonder if they’re going to be replicating certain systems. And if they’re going to be taking a very particular and narrow view of how peer review should be carried out, and again whether they would consider those mentoring aspects of it. I think maybe my sense is that many of us are adopting that perspective. I would just wonder. I think it could be a good thing and again, without having done it I can’t really say, I guess maybe I’m just skeptical of big publishers in general and maybe I would have to really check it out and see what I thought.
I’m pretty torn… because just learning what I’ve learned about implicit bias, I feel like it could work maybe either way.
Emily: Have you ever thought about or experienced open peer review? What does that phrase bring to mind?
Yeah. I mean my understanding is I guess it could probably be carried out in a couple of different ways, but my understanding is that open peer review would be when either the peer reviewers, the names of the peer reviewers are shared with the author and the authors’ name is shared with the peer reviewers, and/or a third option is that it’s all actually put out there openly so that it can be publicly viewed by the peer reviewer.
I know there was at least one journal that asked me at some point at the end of my review if I was willing to share my name. I’ve never been on the receiving end of that, and I’ve never reviewed someone knowing who had written the paper. In terms of my feelings about it. I’m pretty torn, especially in terms of the idea of sharing the author’s name because just learning what I’ve learned about implicit bias, I feel like it could work maybe either way. The happy thought would be that this would be a way for us to be more inclusive in publishing by being aware of: are we constantly approving these texts that are coming from certain perspectives? But the flip side would be that we’re going to actually consciously or unconsciously block more publications from people of color, knowing that’s how implicit bias works. So I’m torn. I would be happy to do it if I thought it was going to work the way it was supposed to.
Emily: Yes, but opaque peer review is also not working the way it’s supposed to.
Yeah, I mean it just raises that question of which would be the better approach or the lesser of two evils or whatever.
Emily: Well, if they’re both evil why are we doing it at all? Because of implicit bias I wonder, can we require implicit bias training? Can we, in a rubric for a referee, include some thinking reflective questions about bias as we’re reviewing something? In opaque review are replicating the voices that are familiar and so is there privilege inherent in that this person knows how to phrase a question in the performative academic way? And then is someone being excluded from that?
I think probably. Even though I wasn’t necessarily taught how to be a peer reviewer, I was taught how to write a research paper and there was a lot of things implicit in that and I wasn’t even taught that until I got to the PhD program. It’s not just the structure. It’s not just that the lit review comes before the findings. It’s really the phrasing, the style. Consciously or unconsciously when I’m reviewing papers, that’s what’s in the back of my mind. And I imagine that’s what’s in the back of the mind of many reviewers: this is what an academic paper sounds like. Even just as an example, I have one PhD student right now who is using grounded theory for her dissertation. And so she’s using first person and someone on the committee came back and was like “but you’re never supposed to use first person.” Just as one very simple example. So yeah, I think that the system probably does replicate those things. And then two, even in opaque reviews if someone’s first language isn’t English that is often obvious from the writing. I’ve reviewed for journals that have explicitly said we’re really looking to increase our global participation or whatever so yeah, I think there’s a lot there.
Even in LIS I think there’s some bias towards certain kinds of methodologies over others. There’s certain methodologies being more empiricist or more objective.
Emily: There’s a lot. Can you think of what the ideal model would be?
That’s a great question. I don’t know. Off the top of my head I would really have to give it more thought. I guess I do feel like some sort of review makes sense. You wouldn’t want to just say everybody publish everything and then we all have to try to wade through it. I do think that a more structured approach where people somewhere get training on, first of all, what’s the purpose of this approach, how do we do it, etc. But then really thinking more broadly about what qualifies as an academic paper, what questions are appropriate, what methodologies are sanctioned. All of that kind of stuff, and figuring out what biases we all are bringing to that when we do those reviews. Because even in LIS I think there’s some bias towards certain kinds of methodologies over others. There’s certain methodologies being more empiricist or more objective. And I think we need to have a broader conversation within the field. I think that goes beyond just peer review and it’s really kind of thinking within the field. What are we trying to do here?
Emily: Why do you think we have that?
I mean my sense is that it’s not just an LIS problem. It’s a problem across all different fields and I think that these are things that sadly I am just really beginning to become aware of. It wasn’t something that I necessarily thought a lot about as a student coming up through the ranks, but just the idea that for so long so much of the power structure was white and male and that within that structure, then, certain decisions were made about what is a good academic paper, what is formal academic writing, what are the sanctioned questions and the sanctioned methods? And then once that happened, many of us were just kind of replicating the system without ever questioning “why is this particular methodology superior to that methodology?” or “why are we not interested in these questions?” It’s something that I have been talking a bit about with some of my PhD students, because a number of my advisees have gotten push back in different areas where they’ve been told “oh, that’s not really a good question for your dissertation” or “you really can use that method but you have to pair it with this other method.” And my approach has really been, “this is your project and we want it to be the best it can be, but I want to work with you to make it your project,” if that makes sense. And that’s getting off on a little bit of a tangent, but I think if we could start thinking more broadly about just what kinds of things are worth pursuing or what methods are acceptable to pursue them with, I think it would be better off for it. We would have a richer field of literature.
Emily: People who are working towards tenure can’t bite off a project that is a longitudinal study and get promotion and tenure necessarily because you have a timeline. Maybe people also don’t understand qualitative methods. Quantitative methods are sometimes arguably quicker to write about. At what level do we reinvent the culture of promotion and tenure because it is so limiting (and in my view a performance of whiteness and a performance of maleness in a lot of places). On the other hand, being tenured I feel secure enough to be able to say no. I feel secure enough to be able to say “I think that is a very bad idea.” Are there ways that we can change the culture of promotion and tenure and keep the good things about it?
I don’t know but it is an excellent question. And in some ways I think I may have been in a little bit of a sweet spot at Simmons because we’re not a research intensive university. Of course I felt pressure to publish and things like that, absolutely. But I feel like I probably had a little bit more leeway in terms of what counted or how much was expected in comparison to maybe some of my colleagues who are at R1 universities. I guess I’m not exactly sure where I’m going with that, but just to say that yeah, I think the culture is pervasive for sure but even if there’s some gradations and I think that when so much emphasis is put on research and on quantity within a certain time period, it really does drive people to focus on things that they know they’re going to be able to push out and that are going to have enough traction to get accepted. Like you said there are some really good things about the tenure system in the sense that once you are on the other side you have a lot more leverage to push back on some of those things.
I think it’s really helped me to think a little bit about how I will share these things with my students going forward, certainly, and even as I’m peer reviewing continuing to think about how I do that.
Emily: Yes. I feel that replication too in terms of if you achieving tenure means that you are writing the article that the reviewers want because you know you need it published. And so maybe that is dictating your research methodology. Maybe that’s dictating the voice in which you’re writing your article. You’re not able to subvert the system, but still do really good work because the system wants this and you have to play with it in order to get to that place of power. I’m privileged because I have some power now because I am tenured. I think that’s the only power I have. These are the things I like to ruminate about and I appreciate you listening to me.
It’s fascinating. I’m really glad we got a chance to talk. I really appreciate having this time to talk about it because I think in a lot of ways, I feel like I’ve been approaching all of this and even teaching about all of this. Even though I felt like I was thinking about it a lot, this has really gotten me to think about it more and to articulate some ideas that maybe I was putting into practice without ever having articulated. So I think it’s great. I think it’s really helped me to think a little bit about how I will share these things with my students going forward, certainly, and even as I’m peer reviewing continuing to think about how I do that. I also just think, I mean you’ve raised some really great questions about the process – things that I know as a field we’ve been grappling with. But there’s such a shift in scholarly communication and publishing right now, that it does make me wonder if we might not be coming up on a good time to try and push for some kinds of changes. I’m not exactly sure how you do it, but I mean I published two textbooks. They are both coming out, and one is going to be coming out next month in open access.
Yeah, but that whole process, going through that process has made me think about just the whole idea of traditional publishing and again, the advantages that it brings and the disadvantages that it brings and I think peer review is, especially where it’s been attached in a lot of ways to traditional publishing, I think it’s at an inflection point. So it’s a good time for us to be having these conversations.
Stories of Open 