Stories of open began 3 years ago, when I sought to expand my research on open peer review. I was looking for a more involved qualitative research project that would gather and share personal experience narratives about peer review and open peer review in Library and Information Science.
As such I began to codify for myself a research project for an upcoming sabbatical. My aim with Stories of Open was always to gather and share personal experience stories. With an accepted book proposal, IRB approval, as well as a forthcoming sabbatical, I began to gather stories in the form of interviews. These stories, and their analysis using narrative inquiry methods, are forthcoming by ACRL Press.
But that is not enough. Everyone has a story to share. According to educational researcher Jeong-Hee Kim, narrative research’s purpose is to “… invite readers to a sphere of possible contact with a developing, incomplete and evolving situation, allowing them to re-think and re-evaluate their own views, prejudices, and experiences” (p. 235). Kim, Jeong-Hee. Understanding Narrative Inquiry. Thousand Oaks: SAGE Publications, 2016.
I am continuing to conduct interviews. My sincere hope is that I am able to share the transcripts of those interviews–our colleagues’ stories–on this website. The stories won’t be heavily analyzed, and the transcripts won’t be heavily edited. They’ll be offered in readable and digestible form.
My hope is that reading these stories inspires you to share your own, either here or with a colleague. My hope is that reading these stories makes you think about how our community can work to improve peer review. And my sincere hope is that the changes we collectively make to peer review will make it more just, more equitable, more inclusive, and our literature more diverse.
Please contact me if you would like to share your story or open or your experiences with peer review.
This is the second in a series of three presenting Debbie’s story. In the first part, Debbie shared her experiences submitting to a Canadian journal and one international in scope. Both of her experiences were positive, and she felt that the communication and transparency of the journal systems were a part of that.
In this second part of Debbie’s story, she shares a bit about her experiences as a peer reviewer herself and discusses the process she underwent submitting a peer-reviewed book chapter.
Fun Fact: Debbie enjoys going on horse riding holidays in different countries.
I did provide a review in the Canadian Journal of Information and Library Science (CJILS) and it was very helpful to be doing this after having had those most very recent, very different experiences. [The two article submissions discussed in the first part of her story.] It was interesting how the reviewer information was provided. It’s very clear about what they expect the reviewer to be reviewing and how to communicate it, and it was interesting you made that comments and how they can be inappropriate. Yeah, I’m sure they can be. And how they want to be really clear about not stating it’s not up to the reviewer to determine whether or not the article should be published and all of these other parameters. So that was really interesting. In fact, it would have been interesting to look at that even before submitting, because it helps you to think about, “okay, if I were looking at this as a reviewer what would I be looking for.” Even though you know you look at submission requirements– obviously it depends on the topic. It depends how you’ve written the paper, but those edited pieces are an excellent lens to be looking at even when you’re writing a paper I felt. So, that was a really interesting process, too.
Emily: Had you had any training aside from the guidelines that you saw from the journal to be offering a review?
Just subject expertise and really as a higher education teacher and masters and undergrad programs. I took it as those were the qualifications that were appropriate to that extent. Other than that it was the subject expertise.
Emily: But the guidelines were clear enough for you that you felt like you could take that and know what you were doing?
Yes, they were very specific, and I thought that was really helpful.
Emily: You mentioned something about having attended workshops about peer review while you were a doctoral student. Can you tell me more about that? I’m just going to reflect quickly: what I’ve heard from a lot of people that I’ve spoken to is that there’s never been really training or understanding about what the peer-review process looks like as an author, particularly for librarians only in master’s programs who don’t go through doctoral work.
In my program my supervisors did provide a certain amount of information about the peer-review process and specifically they were talking about the need to publish. They wanted us to be publishing and how they looked at our thesis and they said, “well, you can fillet it. I think I see four articles here.” Well, I didn’t have four articles in me. And then really within that context they were talking about that the process, the expectation of whether or not you will actually be accepted, taking the feedback that you receive and doing explicitly what is asked of you. Very similar to the process that you go through with your supervisors when you’re writing your thesis. So that was kind of the lens they put on it. But then a more fulsome workshop I did attend was not within the program but at the European Conference on Information Literacy (ECIL). They had a stream there which was for doctoral students and there was a very specific workshop talking about the process. They had people up there who – they had a journal editor and people who had done peer review to talk about the process and some of what I said earlier about take heart, they’re looking at it.
And the expectations. That was actually a really good session and the audience was all doctoral students so they were able to ask, the other people were a little more advanced with what they were publishing. That was really the extent of my experience. Really those two sets of conversations.
The ECIL was maybe two years ago now, so I don’t really remember all the details, but it certainly made it clear to me what the process was at the time, which I hadn’t necessarily been fully aware of or understanding how it happened. Obviously, my supervisors were advising from their personal experiences, which they had shared as far as submitting and undergoing peer review, and being peer reviewers themselves. So they did talk a little bit about that process. That was closer to when we were finishing up our program and they were encouraging us to think about publishing. But we were still trying to finish our theses so there’s only so much you can think about.
Emily: I think it’s interesting that you did have these opportunities. I’ve heard a lot of people had nothing. Even as a referee you mentioned that you had such clear guidelines. I’ve had so many people tell me “I have no idea what I was supposed to be doing. There were no guidelines” and personally I think that’s when you get into these issues of reviewers being inappropriate or saying things that are inappropriate. I haven’t heard very many stories of that in librarianship at all.
That makes no sense. We’re learning. We’re trying to learn and share in the scholarly conversation so yes; your opinion matters, but it has to be about the material and whether it’s furthering the conversation. And those were the conversations we had. Being recently in a program, you have to lose all of your ego about what you think, whether you write well, whether you’re good at researching, whether you’re communicating your topic appropriately, are you researching the right things. You really have to take the advice from the experts and learn to argue, too.
Emily: What do you think peer review should do or what should it be for? Why are we even doing it?
I do see the value in it because, even if we’re working with others, we do bring a particular lens and we can have blinders on. So I think those points about okay, this is really a North American or American-centric topic. That doesn’t apply to us. Well, that’s not what my research said, so I think the point is that it encourages us to ensure that we are researching effectively, we are communicating effectively, we are adding to the conversation, and that we are testing and evaluating throughout—testing and pushing forward. The whole point is to ensure that we are contributing, and if we’re not addressing the audiences effectively, if we have gaps, that’s really what the reviewers should be pointing out because they bring those expert lenses to this and say “well wait a minute, that’s all fine. What about this? Did you just not think to mention it? Did you miss that in your research?” I think those are the pieces that are more effective. Once published it’ll stand up on the research under its own. Because otherwise it’s back to that “this is my opinion, it’s an opinion piece.” Well, that’s fine but then it has to be taken as such. I find it to be really helpful for developing learning and that’s the learning from the author themselves. Learning from the experts.
Emily: So you see it as a conversation and as a learning opportunity. I wanted to go back a little bit to your experience where you reviewed and then you said you wish you had known what the review guidelines were when you were writing. While you were writing this book chapter, did your experience as a referee influence how you were writing at all or was there a relationship between those two now?
That happened because the book chapter is basically a paper I presented so that was all right when I was I the midst. I think I’d already received my feedback, so I was definitely thinking about that when I wrote the paper and I felt much better that I had created a stronger paper to present at this conference because it was information for professionals and academics from all over Latin America and Mexico and some from the States as well. It certainly improved what I wrote and how I presented it. So that is why I’m curious to know what the reviews will be like because I made sure to take the international perspective and to look for those examples from research very broadly, but that would also speak to the Latin experience. So was there anything – no, but I found something in Spain, so I used that to help me focus on the particular audience and to be really focused on what was the topic of that particular conference. And so I kept thinking about that. I remember even what one of my supervisors said—and this was what I should have remembered before I submitted that other article—was, “well, says who? Yes, you refer here but depending on the type of journal or publication you need to continually reference, even if you’ve referenced it a few lines before. You have to be very specific and back up anytime you’re making a statement.” So it’s exactly what I was doing in my thesis. I used that very much in my process for the writing for the paper that I presented. But I haven’t seen any feedback yet.
Emily: Is writing and publishing part of your job expectation or is it simply an outgrowth of your doctoral program that you’ve continued to do this?
It’s been an outgrowth. We’re not publishing a lot in this province. There are certain people who do but it’s not necessarily even part of the requirement. We’re a teaching university. So, it’s evolving and they have been. Many of them have done mostly conference presentations and they do have a fair bit of PD time like four weeks of PD time. But they’ll have the option of doing research in the future soon or finding grants and doing research. It has not been part of our university historically. Many of the librarians are faculty. I’m not faculty, I’m an administrator. But I think it’s really important to be leading and to encourage it—to lead this understanding within the university that we are all academics, and we are all contributing, and publishing and research are a form of my contribution. Many of my faculty present papers at conferences and do publish, but many do not. I just see it as a way of also modeling both for the people in my library, but at the university at large because my VP of Academics says “congratulations, I saw that you got those articles published and your expertise in your field.” This is a way of also signaling to the rest of the university the library is part of the academic community, that we’re part of our own scholarly conversations and that does overlap with what you are doing. So, those are some of the reasons why I’ve done that. And then it’s been opportunity. It’s not required.
Emily: Is it part of, for example, UBC culture for librarians to be researching and publishing?
I believe but I don’t think everyone. I think they can choose how they wish to do their PD. They may not be officially part of the faculty association. They might have a particular status. Do you know what I mean? Research and non-research. When doing my research there isn’t a lot – I notice that’s the problem. There isn’t a lot published in BC higher ed. No one is really required to do it.
I just don’t think that’s what inspires them. I find there are some librarians who are really engaged in particular topics and they are writing about student engagement or information literacy in particular. I think those are two biggies. Assessment. But there are a lot of librarians who are doing their collections work. They’re interested in teaching. They’ve never been interested in doing research or in publishing. Like I said our librarians actually do a fair bit, but not everyone. But they do other things like they’ll support conferences and they’ll do much more practical types of activities and definitely workshops and sharing information that way. But it is kind of like our teaching faculty at our university. Many of them do research. Most do not. Pedagogy is what they’re interested in.
There’s expectation for professional development activities, but really faculty can choose what it is they wish to do. So, like I said there are always the keen ones and I think that’s great because it’s great for our reputation. It’s great for their, whatever their career plans are as well. We’ll never be that. That is our mandate – teaching, regional. But maybe teaching and learning like the scholarship of teaching and learning. That seems to be of interest to people so I can see evolution specifically in that research area.
This is the first in a series of Debbie Schachter’s story. At the time of this interview, in August 2020, Debbie was the University Librarian at a small teaching university in North Vancouver, BC. Debbie has had a wide range of experience in libraries and beyond, including in the news media and social services. Her library background is in special libraries, public libraries, and academic libraries. Just recently she completed a Doctorate of Education degree from the University of Edinburgh, Scotland, which required a thesis (what they call it there), and has been able to publish some of the research she completed as part of that pursuit.
In addition to her current job, Debbie is an adjunct at the UBC Information School where she teaches Management of Information Organizations. She also teaches for a well-respected library technician program at Langara College.
Fun Fact: Debbie recently took up violin again after an almost 40-year hiatus.
In this first part of the series, Debbie discusses two parallel experiences she had undergoing peer review, one in a Canadian journal, and one in an international journal. Because Debbie had attended some workshops and presentations preparing her for what to expect in peer review processes, she felt prepared. Let’s hear more from Debbie.
Emily: Can you tell me a little bit about why you wanted to participate in this interview?
I thought the peer review process was really interesting because while I have written and published, many years ago I used to write a regular management column, information management through the Special Library Association (SLA). It could have been ten years. Anyway, it was a long time ago. And then I contributed to a book chapter on supervision and management for an ALA publication. But I hadn’t been involved in the peer review process, not since university obviously, and then I returned to get the doctorate. Most recently I had two divergent experiences in two very different peer reviewed journals dealing with feedback. I’d already attended a couple of sessions on what your experience might be like when you go to publish. But it can be sobering and it can be challenging, depending on what the reviewers say. I thought it would be helpful just to reiterate what I had been told about; yes, if they’re interested just because you feel strongly about your work, you may feel surprised at the feedback. An outright rejection is one thing, but “you can make this better” is something that you may look at as simply constructive feedback. I thought it would be helpful for my most recent experience doing this.
I identified a couple of journals that would be appropriate for my research topic. It was on critical information literacy teaching within the British Columbia higher ed context. I did a survey and then interviewed a large number of individuals representing the public institutions in this province, and a couple of the journals that were interesting to me were the Canadian Journal of Information and Library Science, because I was actually using a number of articles from that in my own literature review. And then the other because I’m a member of IFLA [International Federation of Library Associations and Institutions] and I presented on my research at an IFLA conference in Malaysia and also at the European Conference on Information Literacy in Europe, ECIL. So I had been invited through IFLA to submit to a special edition for the IFLA Journal. I knew that I was writing toward that and then for CJILS I was hoping to have a paper accepted. I submitted to the CJILS and there was quite a long period of time trying to get to the right person. There was a change in editors. While I was waiting I had submitted to the IFLA Journal. That one was a little bit more, should I say, it felt a little more rigorous. It might have been that the articles were different. The CJILS was a very positive and supportive type of process. I felt the feedback was gentle. It pointed out certain things that that would be areas of improvement. The interesting tone of the feedback was very much more supportive than what I did experience through the IFLA Journal, where there were two peer reviewers. There were many comments.
And for someone who doesn’t have experience writing for an international journal in particular—I’d heard this from someone else—there’s quite a bit of detail that is expected as far as the citation and referencing beyond what you might experience in a smaller journal. I was not entirely sure how to craft an appropriate article. Should it be like my thesis? Should it incorporate as much of the citation as I would in my thesis? Initially I hadn’t thought that it should, but clearly it should have because that was the response I received. They provide feedback because they want you to improve, but the experience can also give this sense of “oh, there’s all this work that needs to be done,” or “if I complete this work is it going to be accepted? Is it appropriate?” There wasn’t necessarily a good overall sense that if I completed all of this additional work that it would be accepted and that it was appropriate for the journal. I kind of got the sense that it would be because they were going through this effort, and they hadn’t rejected it outright, but to me it wasn’t entirely clear. So it was about the communication processes. The feedback was, I felt, I believe it was fair, but in some areas it was sort of—and perhaps it was just the tone of one of the reviewers—was “well, I don’t believe that this is true.” That kind of thing. And it was like, “okay, well maybe that’s just they want to see the other citations.”
Also, there can be a sense of what we’re writing about in North America might not be how they perceive certain topics in other countries or on other continents. So I took it with a grain of salt because I had already defended my thesis and to argue certain things, but I think it’s a strong reminder that we’re in a conversation and how people review is not necessarily going to be gentle. You might choose to receive it in a way that you feel is not the most constructive, but the reality is someone is putting effort into the review and asking these questions very honestly. Well, what is the evidence? Yes, I see that but maybe I don’t agree. Maybe you need to follow that up. That simply makes it a better final result. It is interesting that depending on the reviewer how you get such vastly different types of feedback, and that you cannot always predict what you might be anticipating when you submit an article. Again, having them accept it and then for review and edit, that is a very positive experience, but as I said if you’re not experienced it can feel like “oh, well I’m never going to be able to achieve what they’re looking for” or they simply don’t like it or whatever. I think for those who are new to this, they should take encouragement from it and not to feel bad if the first effort wasn’t perfect. And that’s exactly the feedback that I’d received – not personally, but when I went to some workshops at a conference recently, that was exactly what they were saying in those workshops for doctoral students. Don’t worry, you might not be accepted. No one is accepted initially. You’ll be fortunate if you’re accepted into the journal you wish to publish for. I felt for me it was an excellent learning experience. The CJILS was a positive experience, too. There were pieces that I needed to focus on. It was a very different article, so I think that’s why the feedback was quite different and it was more based specifically on my research. Again, I’m not surprised, but it is a comparison for two articles loosely associated how very different they can be perceived and the feedback provided.
Emily: Were both published? They were both ultimately accepted and published?
They both went through. This was that thing about really taking the feedback very literally and doing that; it’s just as when you do a doctoral thesis or master’s thesis. You’re getting feedback from your supervisor. I felt exactly the same way. This was stretching me and this was a learning experience; someone had taken the time to provide that feedback and if that’s what one reader thinks, what are other readers going to think? I was a little shocked by it frankly. You just have to set aside your ego or anything like that and again immerse yourself in the learning experience. I did a little more work in certain areas that they were looking for. For one of the journals I just added to the citations because I felt they wanted to see more and verified. I thought, “am I making this up?” I’m like no, I wasn’t. I verified some of those pieces cited as heavily as I did in my thesis. That kind of thing. Resubmitted and it was accepted with just some small grammatical editing and that kind of thing. And at the same with the other, which was a smaller article about my research, just a few changes, some additions. They were logical items and then it was published. So a very positive experience.
Emily: You mentioned that at CJILS that there was a changeover in the editor at the time. How much of a delay do you remember that introducing?
It could have been a couple of months because I think I probably emailed someone, didn’t get a response. Went back to the site, checked again. Because I’d relied on some Canadian studies in my paper, I thought, “I’m referencing them and I really should be publishing in that journal.” I received some information that there was a new format for the journal. They had a new submission process, so it was really a handoff between people at two different universities and so it wasn’t a huge amount of time, but it was just a coincidence of the timing when I was looking to submit. Other than that most journals have these systems that are really effective now, where you can actually see what’s happening with the process about your submission. I think that was the start of their new system so I really appreciated that. It was really helpful because even if you thought nothing’s happening you could go to the site and check. Oh, it’s under review. Great. Okay.
Emily: Did you notice a difference in the way the editors of the two journals handled your submission and feedback?
I think that they were similar. They were both very neutral, supportive. Clear about process. I think that’s the most important thing—and expectations. Perhaps if I’d been rejected I could have compared different information but they were both accepted so pretty much their processes were similar and they were encouraging me to both submit and then also to be monitoring the article’s the progression through the process in their system. And then the way they responded to the feedback—they provided the full feedback, but they also provided an encapsulation and then the specifics of it, which is really important.
Emily: So there was guidance given to you such as, “pay more attention to reviewer one about this and reviewer two about this” or, “I agree with this,” or that kind of thing?
I think both of them were basically saying that they concurred with the review. They were supportive of what they saw in the reviews and there was an overall message. Overall it looks like this is a valuable submission, blah, blah, blah. That kind of thing. However, there are a number of items so please review the specifics from the review comments. They were both consistent that way. I do recall with both of them that they talked about timelines and process, so they were very clearly managing the process and so I was appreciative of that. My experience was quite different.
For the IFLA Journal they decided to create a special issue on the topic and so that has happened and that’s the only other thing is don’t despair in the length of time. It seems like some of these things take forever and then you don’t know when it’s coming out even when it’s accepted. So that was part of “will we ever find out? What’s the deadline?” And the deadline is so much in advance of the actual publication. I’m part of the IFLA Library Theory and Research section and I did actually present at a conference at the beginning of March. That was my last holiday ever out of the country, due to COVID. And that was also the book chapter, so they’re going to do a peer-review process for that. I already submitted it (and I’m curious to know the result and unfortunately it didn’t happen in time for this conversation). I’m curious to know what that experience would be like. That’s very different. It’s through the Autonomous University in Mexico City, the information school there, and so that might have been interesting to be able to share, but I don’t know anything yet about that.
It’s been quiet around here for summer while I took two months off to drive around the country to visit friends and family I had not seen in years because of the pandemic. But I’m back at it and have plenty more stories to share with you. But before then, ACRL is hosting a webcast for #peerreviewweek21. The theme, Identity in Peer review, takes up a few chapters of Stories of Open, so it’s a perfect fit! (Need to read the OA version of the book? Find it at the bottom of the book’s description on the ALA Store.) While I, of course, think everyone should read the whole book, if you are most interested in identities, I recommend chapters: 4 – Roles of Peer Review; 5 – Dualities and Multiplicities in Peer Review; 9 – I Just Feel Like This Makes Sense to Me.
Opening Peer Review in LIS September 21, 2021 | 1:00-2:00pm CT
What does it mean to peer review in library and information science? What does it mean to be reviewed? How do our professional identities intersect with this vital research and publishing role? And what does it mean when peer review is opened to reveal these identities? In celebration of Peer Review Week 2021, this free webcast with Emily Ford will share insights into peer review in LIS as discussed in Stories of Open: Opening Peer Review through Narrative Inquiry, a newly published book from ACRL.
It is with excitement and a joyful anxiety that I break our regular programming to share this news: Stories of Open: Opening peer review through narrative inquiry is now available for purchase in the ALA Store! (It’s also an OA book, downloadable here.)
This book project is what prompted me to continue to gather and share stories on this website. From the introduction of the book:
I continue to work to edit and publish stories here, apologies for the lull and interruption, but things have been busy!
In this exercise of researching and writing—exploring ways of knowing through narrative of human experience—I have also come to be acquainted with our colleagues who bravely shared their thoughts. We theorized together, and together we explored their (and our collective) emotional experiences. Each and every conversation required folks to be vulnerable and trusting, and in return I have striven to offer you their experiences with loving kindness. This book is as much theirs as it is mine, and as it is yours. In the interest of protecting individual privacy, I have used pseudonyms for each individual mentioned below, with the exception of Stuart, who agreed to openly share their story. Stories from individuals with pseudonyms have been edited to omit personally identifiable information, and when needed, I worked with those individuals to edit their stories to include and exclude information as they felt comfortable. I would like to introduce you to ten incredibly generous and thoughtful individuals. The headings represent the title of each individual’s interpretive narrative.
This is the third in a series of three posts that comprise Hannah Gascho Rempel’s story. In her first story part, When you look at a body of literature…, Hannah discussed her recent ties to scholarly publishing, investigating its history, as well as her experiences as a journal editor. In her second story part, The system was meant for me, Hannah discusses her privilege and delves deeper into her experiences as an editor, hoping to make positive changes at the journal.
In this third and final story part, Hannah discusses her experiences withdrawing a journal article and publishing it elsewhere, as well her experiences with open peer review.
Hannah Gascho Rempel
Position: Professor and Science Librarian at Oregon State University Libraries
Fun Fact: Hannah teaches a Learning Through Play seminar class, in which students create games in small groups (and learn about research). This term the theme for their games is Space, and she’s really looking forward to seeing how they pull together learning opportunities for aliens and astronauts.
Emily: But what do you think peer review should be for? Like why are we doing this?
I was trying to think about that ahead of time, too. Ideally I would think of it as a conversation in a community that makes a work stronger. So, for example, there’s the open peer review style of reviewing at this one journal, it is a little bit of an apprenticing. I’m not sure if they’re still doing the Google Doc version but you see the comments along the way. I wouldn’t say the quality of the feedback I received via that particular venue was very good. I would say that because I don’t feel the reviewers I happened to be matched up with had either all that much experience or experience in my particular fields. And by fields I’m talking pretty broad like instruction librarianship. They just didn’t have that and so the feedback they gave was a little bit like, “okay, I will use my skills from my very first advisor and say the polite thank you for your helpful feedback. However, I know this and this and this too.”
So, the other side of that when it’s obviously closed and you get back the reviewer two because I have myself received the reviewer two-ish. Not a like horrible horrible version of that, but it’s this false community conversation then, [where you’re solely communicating via a written format, and there] are all these things about that written format that make it hard to [communicate] and they don’t quite get enough context and they don’t know you and they don’t know your history and that’s all by design. You’re not supposed to know all the things. So they give their feedback coming from up here and they’re like, “have you thought about this?” And you want say, “Totally. I’ve spent like half my career thinking about this. Like uh huh, yeah. But thanks.” And so the idea of that is of your hands motion in the air passing each other by and not having real conversation. The ideal of that [conversation in community] gets missed. Ideally you would be getting helpful feedback, because I do value much of the feedback that I have gotten. As with most people I, of course, don’t value it like two minutes after I’ve gotten it, but a day after I’ve gotten it then it’s situated a little more. You look back and you’re like, “Okay, it’s true.” How would they have known I have been thinking about that for half my career? I didn’t tell them that at all. How would anyone know? So yes, I should reframe that [idea I was writing about in my manuscript draft]. I should set it up better this way. Fair enough. And then you reread it, your own work later and you’re like, “Yeah, that was a better option. I’m glad I did that.”
Emily: It’s interesting to hear that you had not as robust of an experience with open review. In my experience open review has been more robust than closed review so that’s interesting that you had that experience. Peer review can – you never know. I’m wondering if that has colored what you think of open peer review now? Would you try again based on that experience?
I published twice in that open peer-reviewed journal, so I have tried it again although I wouldn’t say that the second time I tried it for the open peer review part of things. I’m still open to it. I’m open to a lot of things. I think in some – yeah, by having more experience in the profession just because I’ve been part of the profession longer, in theory I would then know people who I can get helpful feedback from. That said I don’t know all the people by any stretch. It’s a huge profession so I would always be missing out on some voices if I only ever sent my work to people that I know and value their feedback. So peer review can have that value when an editor is making some of those choices for you, finding new people and pulling in somebody who has a different background and maybe that would be helpful. I’m still open to having other approaches, but I think again if I could just give it to Janet Webster half the time I would probably get 90 percent of my helpful feedback from Janet Webster feedback and then 10 percent from the person that I don’t know behind, not even behind the wall of the open peer review but somebody that is totally new to me.
Emily: Yeah, interesting. Are you saying that because the journal asked you to say who are you going to ask to do this? I just want to connect back to that.
Right, because they do have the ask part of it. I did have one other more recent experience though of asking somebody. So the discipline that I, the science discipline that I grew up in used double-blind the whole way through, so recently I had a single-blind experience, -ish. I included a list of people to be included in the ask and then when I got the feedback I was like, “Oh, clearly they chose Greg.” Even though Greg’s name is nowhere on that. I was like, “Ha ha, wow. Greg, you can’t tone that part of you down at all.” But it was helpful so I guess having had a couple versions of that now, one at the openly reviewed journal and this other experience, asking someone – I guess what I’m getting at is there’s enough strategy in asking someone when it’s going to be for publication versus the informal Janet ask, when Janet is not going to be – I’m using Janet as shorthand for all the people, but when it’s not going to influence whether or not you get accepted or not.
Emily: What do you think given all these issues? You’ve seen issues of where on your end as author it’s supposedly opaque and you weren’t supposed to know it was Greg, but it was Greg. You have seen it as an editor where you have this reviewer two, where it’s not necessarily abusive, it’s just cursory and people just aren’t necessarily engaging deeply or robustly. How do we move past those issues of peer review—whether it’s an open peer review with transparent open identities or not—how do we fix the problems that we’re seeing in peer review?
That is a good question. I thought when I was an editor that a way that I would fix it was by including my feedback to the reviewers. So when a decision was made and the thank you for your service kind of email goes out to the reviewers and it included both of the reviewers’ feedback and then the content management system piped those two things in. It didn’t pipe in my feedback though, so I’d dutifully copy and paste my feedback as a “subtle” – Hannah is using air quotes – way of being like, “What you could have done was provide this kind of feedback.” I think for a few of the folks that did help, particularly people who I had invited to the editorial board—and not that it was about me inviting them, just maybe that they were newer and so were looking for inputs—so that did shape their feedback some. Just having an example of what was being looked for. But then the [peer reviewer feedback] form, I did change the form some over time. I changed the peer-review form to focus more on, to say, “Please look at…” It’s one of those things where we don’t always have a methods section in the kind of work we do, the kind of writing that we do, so it doesn’t follow the flow and so saying something on the form, like, “Do the results follow from the methods used?” Well that doesn’t even apply to a lot of the kinds of writing we do. So some of the checks that would be in a science writing just don’t follow. So, “Is it clear? Does it make logical sense?” sometimes is the best you can do and that feels so wide open. I tried to be a little more specific, but it was hard to do when there are all these forms of how people write in our scholarly conversation that are okay. I’m not saying that everything should be an empirical study by any stretch, but then it makes it harder to standardize and to know what you’re looking for. I will say the other thing I did try and get at in the form was that folks did not need to spend any time on copy editing and that grammatical decisions didn’t need to influence their feedback. And people did make that adjustment and I was glad. I wanted that for a couple of reasons that I’m guessing you can guess very easily. I didn’t want discrimination because of voice or tone across both, English as a first language, but also other kinds of writing that comes out. I didn’t want people to spend all their energy on this—as librarians we sometimes are wont to do that. “Is the Oxford comma here or not?” That’s on me [the editor] or that’s on actually Taylor & Francis [the publisher] folks. They [the publisher] did that level of things. So don’t spend all your energy there. What your energy is for is: does this fit in our conversation? Have the conversation.
Emily: Fit. It’s interesting that you use that word. Okay, so can you unpack that a little bit?
So, the particular journal has a scope. Not everything fits in that scope and so defining that scope is a changing conversation. And so the particular journal, the Journal of Web Librarianship, [they are involved in a changing conversation around:] what does web librarianship even mean? I mean what is that?
Emily: It’s everything we do now.
So then saying is it everything we do or is it a particular kind of thing we do? And what we do changes over time so where is that “fit” and where are we having that boundary? Because it’s shifting and here we are in 2020 and here it is still web librarianship? So, it’s a conversation to have about fit and so I did what editors typically do – they review it [an article submission] first and do a cursory check to see if the article matches the scope. So I can have some of those boundaries set for sure, and I can know this is like 100 miles from fit. But [it’s different if it is] 10 miles from fit and maybe our boundary should go out 10 miles. That I wanted to have other input on it.
Emily: Okay, that makes sense. I feel like there have been articles about they don’t “fit” in our workplace with a hiring things so it’s code for white supremacy. I just wanted to unpack that a little bit more.
Right. And it means there’s a community line and “there is a community line” is my understanding of the white supremacy version of fit too. To me fit still can have a purpose, but yes, if you don’t unpack it then you haven’t defined my meaning.
Emily: Did you have any particular story or anecdote that came to mind for you when you were thinking about your experiences?
I guess maybe two and I was going to say – I think they’re short and now you know that’s a lie. None of them – I don’t say anything short. So the first one I’ll [share] was on understanding that a rejection is not the end of the day. That’s something I would have learned from my plant sciences advisor. You get rejected by one journal, that’s nothing on you. You just move along to the next one. You take what feedback was helpful. Perhaps it wasn’t within the scope. Who knows. It could be any number of things but you move along and you’re going to get that thing published. It’s the outcome. I don’t know if it was my third or fourth article [in librarianship]. Anyway, fairly early on I got rejected and I’ve been rejected more than [that now], and that was my feeling about it. That’s how it works. It doesn’t always, you don’t always get all the things and you move along and I was able to get the article published then at the next place. That was not the experience that many of my colleagues had had, especially if they didn’t have a previous disciplinary experience in being part of a publishing community or an academic community. So having that as something that was more explicitly talked about and made okay that rejection wasn’t like close up shop, you’re never going to get tenure. Your life is over. Now you’re just moving along. That felt like another lack to me in a thing that was missing from maybe how librarians are, probably broader than that academics too – that it’s okay. You just move it along. So that was one [story/observation].
The other one is kind of related, but what I would consider my worst peer review feedback. It was for one of our more notable publications in librarianship, and what irritated me about that experience was the lack of editor interaction there. So I got a revise and resubmit on the first round after waiting like [at least] six months to get that feedback. The feedback was not fantastic, but okay. I took the feedback, made the changes, again waiting for forever, and it comes back with another revise and resubmit with even less substance to it and no mediation by the editor in either of those to say “it looks like you’re really just missing this one thing” or “it looks like you’ve failed to do that certain thing.” Nothing. And just kind of spinning it over and over again in this revise and resubmit cycle. And that was frustrating, especially because I was already an editor myself at that point and the quality of the feedback given on the second one would have been a conditional accept in my view, not a revise and resubmit. So it felt like there was just some laziness, over work, inattention. I don’t know what it was. And so I feel in our profession that if we’re not going to have a training system or this clear something – that it is on editors to take on some of that role and I felt that especially strongly at the time as an editor. That was puzzling to me.
Emily: Did you withdraw?
Totally. I withdrew.
Emily: Did you publish somewhere else?
Yes, and the editor at [the second place I submitted to] was like, “I have been an editor for five years. This is the first article that I’ve never had to give any feedback on. Neither reviewers had anything to say in terms of changes, blah, blah,” which isn’t so much to toot my own horn, because I had made changes based on that previous feedback so it had gotten feedback already. But it wasn’t very different at all from what I had resubmitted the second time there [at the previous journal]. I was like, “Well, exactly.”
Emily: What is your speculation of what happened at that journal where you did get this R&R twice?
I think the editor was kind of checked out.
Emily: Or swimming with submissions.
I’m sure that was true, yes. And that journal would have gotten many submissions.
Emily: Right. I think, too, the labor—again, I haven’t been an editor of a proprietary journal. I’ve only ever been on the editorial board and edited something that I invented with friends, so just take my comments with a grain of salt—but I would assume the labor that I’ve always put forth toward editing and things like that, that’s a labor of love. Sure, in a tenure-related position there is some recognition for that, but you just really need to put it on your CV and have a few examples of the quality and impact to point to. So where is the accountability in actually doing a really good job? And if something happens in your life like you’re splitting up with your partner or if you have a death in the family or if there’s a pandemic or you have a mental health crisis or anything like that, there’s no yeah – so there might be a flaw in the system. That’s very generous—maybe the editor had something happening and because the labor is donated a lot of times. What do you do?
Yeah, I know. That was my every evening for five years. That’s not true. That’s the other irritating thing about editing: it wasn’t consistent. I realized when I had interviewed for the role and asked the question of the previous editor “how many hours a week does this take?” And she sort of hemmed and hawed and I was like, “How can she not know?” And then as soon as I started I was like, “Ah, silly, silly girl. It just comes and goes and you can’t predict when everybody is going to submit five things at once.” Yes, it is a ton of work and nobody sees and nobody knows. This particular editor at the other journal – somebody sees and somebody knows, but yes, your general statements were all true.
Emily: If you were in charge of the peer review world in LIS what would be the first couple of things you would do to improve it?
Yeah, I’d have somebody shadow. I mean I’m pretty consistent with my apprenticing thing. I’d have somebody shadow, somebody else who had been identified as a strong peer reviewer, give them some examples of what was helpful over several ranges of article types, but I think practice is the only way that most things get better, so practice and then feedback and standardizing that as a thing, not as a “you’re in trouble because you’re getting feedback” or whatever, but just having a cycle there of practice and feedback, try it again and then you do the same for the next person in line and then demonstrate for them.
Emily: It’s really interesting that you said when we get feedback we feel like we’re in trouble. But feedback doesn’t mean you’re bad. But a lot of us have been taught or socialized that if we get feedback we’ve done bad and there’s shame associated with it.
And our ears close.
Emily: Yeah. All the feelings and like where is that coming from? Is that American? Is it whiteness? I guess I’m seeing a lot of this white supremacy and socialization. I mean I think that’s where we are culturally right now just with the grief of all of these murdered Black men mostly, some Black women. I think you hit it on the head, this, “I’m bad.”
Yeah, and the quick leap from my work is bad to I’m bad is amazingly fast. And it’s not anybody saying your work was bad. It’s just like, here are some other ideas. I was reading “White Fragility” earlier this winter. What I took away from her was that we need to do better with getting feedback about being racist. That’s kind of the core of what I took away. And where my brain stopped – and she gives multiple examples along the way. Where my brain stopped was like, “when have you seen anybody get feedback and then take productive action on it?” because it was just exactly what you’re saying. You give feedback about how you clean the sink, about how you did the whatever and people are up in arms about when you should have the door open or not.” And not to minimize the racism aspect, but I was like it’s not that it’s not about racism, but people aren’t taking any feedback well, so what will be the leap then that all of a sudden this will be the one thing they take feedback well on. Like, no. I’ve been wondering the same thing just culturally like what is the deal there. And so I was reading it when I was in Europe and then, of course, the world explodes after that. I’ve thought about collectivism and how that all plays out as well. And so you ask me whether it’s a U.S. thing. I think that it is somewhat of a U.S. thing. It’s not only U.S. but we’re such individuals both in taking everything on ourselves but also then everything is for ourselves. I don’t know.
Emily: So it’s like that kind of bootstraps mythology, individualism. This area, this fence around my yard, this is my property. And capitalism is part of that, too, I think. The U.S. capitalism. Lovely.
This post is the second in a series of three comprising Hannah Gascho Rempel’s story. The first post, When you look at a body of literature…, discussed some of Hannah’s current involvement and curiosities about scholarly publishing, stemming from the history of scholarly publishing, to her experiences as an editor.
In this second story part, Hannah unpacks her privilege and successes in scholarly publishing, and discusses limits of LIS education.
Hannah Gascho Rempel
Position: Professor and Science Librarian at Oregon State University Libraries
Fun Fact:Hannah was on sabbatical in the Czech Republic last year and enjoyed doing things like going to soccer matches and people watching while riding on trams and subways (and working).
Emily: You said 30 minutes ago that you were good at publishing. You have a book forthcoming that you’re working on. You have multiple articles. How do you think you got good at it?
I practiced it. I’m a white girl who school was meant for and designed for. I follow the directions. I know how to follow the genre rules and play by those rules and adjust myself accordingly. The system was meant for me.
Emily: How did you learn to serve as a referee?
So I was thinking about that ahead of time and ironically (or just whateverly) I actually have not refereed very many times. I have referred less than five times overall so more of my refereeing came during my [journal] editing days when I was actually the third referee on every single article—sometimes to a more or less degree—but every single article I reviewed. Which isn’t the way it looks on the outside, but at least at that journal that’s how it really happens on the inside. How I learned to do it: some through my first experience publishing back when I was still in horticulture; observing how my advisor—I can’t say that I ever observed her reviewing another article and how she gave that feedback, but I saw the feedback we got from the main article that came out of my thesis work and how she responded and taught me to respond to the feedback. And so [observing] what feedback was valuable and what could easily be dismissed. And so taking that logic then and trying to focus on the kinds of valuable feedback [in my refereeing]. Otherwise in librarianship all of the instances [journals] that I did review in had a [peer reviewer] form to fill out and so that was the guidance that was given there. But more of it, I would say, has come from the feedback I’ve gotten, not just in my plant sciences world but in librarianship, and what I’ve observed as being helpful feedback and then I tried to mimic that myself.
Emily: It sounds like there really wasn’t anything formal. It was just somebody who took an interest to be a good advisor and mentor you. Was that standard in horticulture and plant science?
No, I wouldn’t say so. I wouldn’t say that it was exceptional, that she was the only one, but there were certainly people who would not have had that experience. I would say it was common enough that other professors I’ve interacted with who would be a similar age to me, would have had that experience as well. Whether it was the advisor or a postdoc that you sat next to didn’t matter, but there was definitely some apprenticing sort of approach to it where you saw how somebody else did it, how they responded to feedback or gave feedback themselves, and then you learned that way.
Emily: If you didn’t have that background how would you have learned to do this if you just went to librarianship and not with the masters in plant science?
Yeah, that’s a good question because the closest I guess I can say that I would get is at Oregon State University. We have had a research and writing group that does meet regularly, although less so now, but we also have kind of informally [a network of] people that you know are [good at giving] certain kinds of feedback. So you would go to X person. You were coached early on that you go to X person if you want copy editing feedback. You go to X person if you want structural feedback or deep thoughts kind of feedback. And so I learned more obviously probably from the deep thought sort of feedback givers. There are a couple of people like Janet Webster would be one of those people so you get feedback from them a time or two and then you’re like, “I got it. Every time I should do the methods like this,” instead of whatever other way.
Emily: That’s interesting because it sounds like even though you have this group, whether it’s formal or informal, but it’s been an on-the-job experience. It’s not like in library school you’re learning about this process. As a journal editor do you have any stories of seeing reviews that were just totally out there where you had a reviewer two? How did you manage as the journal editor, the reviews you were seeing and then sending to authors?
Yes, I do know reviewer two. In my experience, I’ll start with – I saw my role as the editor to be a mentor to authors. I understood that in librarianship people haven’t had the training in this. So, when the question always comes up should you have learned X in library school. I have no expectations that anyone should have learned anything in library school. I don’t start with that premise at all. I’m amazed if we come out with anything. [laughs] Okay. I’m being flip; I know that’s not really true. What I mean is that library school represents a snapshot in time of some of the values of that discipline. And it isn’t possible that school will be able to predict all the twists and turns of an individual’s career, or all of the changes to a discipline or profession. It will always be a work in progress. So, I know that they haven’t come out with anything [training related to academic research writing that leads to published articles] so I saw my role to mentor the authors, which, for me, meant then that I would take the reviews and triangulate what was given, make sure to read the article myself (I would always read an article when it came in to see if it was appropriate and to see if it matched the scope). But then I would do a second read and particularly, depending on the feedback given by the reviewers, I’d take a deep read and give extensive feedback myself so that I knew in context what the reviewers’ feedback meant. I didn’t hide reviewers’ feedback ever. So, if there was a horrible reviewer two it wasn’t like I hid their feedback. But I would contextualize the feedback that was given and bullet out what I expected the actionable feedback to be for the author so that they were addressing that rather than two contradictory smorgasbord approaches of vanity items from the reviewers. So that all said – I am trying to think of an actual instance of a reviewer two. I am having a hard time coming up with reviewer two. What I had was so many cursory reviews that were so insubstantial as to be meaningless, and that’s what led me to need to do a really deep read. Because if there was nothing there I would have known based on my first read, that the content, the idea matched our scope. But the execution, whatever it might be, needed work and no info was given [from the referee]. So the more realistic situation was that, in my view, reviewers had no idea how to think like scholars in that way and provide feedback that could lead to action and improvement.
Emily: And why do you think that is?
Back to library school.
Emily: Okay, so it comes back to this lack of training?
Yeah, so lack of a research-driven approach to our profession. That leads to having that training whether that comes from library school or on the job or whatever it might be.
Emily: Who do you think should be responsible for that? You say keep going back to library school but obviously it’s not happening in library school.
Well, this kind of writing isn’t an end goal of many librarians. So for public librarians, for example, who have no clear need to publish, some might have a desire to publish and would work at it, but it’s definitely not an across the board [expectation], whereas the expectation in academia as we know it, is if you’re going to get your degree in physics, the jobs in physics will be professor. Here’s how that works and the apprenticing is very clear. Librarianship doesn’t necessarily end with this as the goal, so I think without having tracks in library school – so I went to the University of Washington’s iSchool [in the mid 2000s]. There wasn’t really an academic librarian track so it wasn’t a way to specialize along those lines. That’s the only way I guess I would see it happening is if you were required to do the work of apprenticing for that profession.
Emily: You keep using this word apprentice and I really like it. It’s not one that I’ve heard someone use before with librarianship as much. You hear internship but it’s not necessarily the same as apprentice. I had this idea while you were talking why don’t we have editorial apprentices? Why don’t we do that? It seems like it would be a really rich training ground to learn a lot of all of the other stuff.
This is the first of three parts of a conversation with with Hannah Gascho Rempel, Professor and Science Librarian at Oregon State University Libraries. Hannah came to librarianship as a second career, and has been a librarian for about 15 years. Her first was in the plant sciences, in horticulture. Her training in science afforded her the opportunity to have some training in scholarly publishing practices before coming to librarianship. She is the former editor of Journal of Web Librarianship. She left that role before she went on sabbatical to the Czech Republic as a Fulbright Scholar in 2019, and then made her way back to the U.S. as the 2020 COVID-19 pandemic hit. Hannah is also currently working on a co-edited book entitled “Teaching Critical Reading Skills: Strategies for Academic Librarians” that she expects to be published by ACRL Press in early 2022.
I should also note that I know Hannah through the small academic librarian community in Oregon.
Hannah Gascho Rempel
Position: Professor and Science Librarian at Oregon State University Libraries
Fun Fact: Hannah is looking forward to growing sunflowers this year rather than vegetables in her garden to give the deer something different to munch on for a change.
Throughout our conversation I notice the parallels Hannah is able to draw from her experience as a scientist and scholarly publishing in the sciences, and how she brings that knowledge to her career and endeavors as a librarian. What particularly strikes me is that Hannah is able to clearly articulate the different ways peer review is understood in some scientific fields versus the way she and I both see it as a mutable evolving process.
In this first part of Hannah’s story, we swing from the history to scholarly publishing to discussing present practices.
Let’s let Hannah introduce herself further.
I’m mid-career. I’m also midlife, mid-forties sort of person. I had another career before I was a librarian so that other start of a career was in the plant sciences in horticulture. I have a master’s degree from that field along with some of the publishing training that goes along with that and the interacting with other scientists that goes along with that. And then I switched gears to librarianship and have been in library land now for over a dozen years, 15 years. That’s a little bit of a shift, but for me it’s mattered for my publishing history and my thinking about academia because I started thinking about academia outside of librarianship and that definitely flavors how I continue to approach the research and writing process and how I train others whether it’s grad students at Oregon State University where I work or my fellow colleagues who are starting out in the profession so both of those things are influenced by my start.
Emily: Can you tell me a little bit about why you responded to the call for participation and why you wanted to participate in this particular conversation?
I, too, am interested in peer review. Not to your level. Mine has been in terms of being an editor of a journal as one aspect of it and then needing to think through what to do with the peer reviews that one gets. How to coach colleagues in the peer reviews that they’re giving without being too overbearing and wanting there to be this professional understanding, but there isn’t, I would say in our field, in the same way there isn’t an apprenticing that happens in the same way I would have experienced it in my previous sciences career. My advisor in that career was very clearly apprenticing me to something. She was demonstrating how she was responding to feedback, how one did that with the idea that I would continue in that way. But then my other part of the interest in peer review has been because I’ve been on a graduate student’s committee for the past year-and-a-half—she’s in forestry at Oregon State University—but her field would be a little bit more human dimensions of forestry kind of focus. So social science aspects of it, and part of her question has been looking at how people of color and underrepresented people are represented in the forestry literature of the past hundred years. And when you look at a body of literature as scholarly literature for the past hundred years it becomes really interesting. How to normalize that when you’re looking at how somebody was having a conversation in 1920; how is that different than the conversation in 1970 versus now? And to put things into categories like we want to do in academia, you start off by looking at the literature that was published by discipline. So what falls into fisheries versus forestry versus wildlife. And then what falls into peer-reviewed versus not peer-reviewed and opening that up and trying to work with her on this so she can ask questions. But the questions all start being like, well not nonsensical, but you’re just not having the same conversation anymore so looking at what peer review meant to someone pre-1970. And then on top of that having an understanding that she can communicate to her advisors and the rest of her committee who would tend to be more traditional scientists who think peer review does mean a very particular thing and that it is a stable conversation across time. And this isn’t to say they’re stupid intractable people at all. That’s not it. It’s just they haven’t thought about it as something that wasn’t always there. So I’ve done some more reading on the history. I don’t know all the things about history of peer review but it’s fascinating and I like it that you’re documenting how it is now because it won’t always be this way and it is in flux so that as we move along the way in the profession we can see that.
Emily: There is a scholar named Aileen Fyfe out of the UK who has done some interesting work on the history of scholarly publishing. Fyfe and their colleagues kind of go into this history of peer review noting that it was in the 1960s and 70s that brought about peer review to the proprietary publishing market. What we think of as traditional peer review today was actually market driven[see Untangling Academic Publishing].
The other version that I’ve heard, too, that I’m sure you know about as well is the government regulatory aspect of it. So they were getting input that congress was saying “oh, well you’re making this report about X. We think you should do this and they’re like if we want to be able to say as a community no, that idea is bonkers and you have nothing, we need to have something very official looking about it so it was manufactured to get that. It was sort of yes, depending on what function you think congress serves if you think they are representing a community it was sort of suppressing a community input which I don’t know that I’m going to stand up for that one very hard but it was to protect the scientific community was the way that I’ve read it too.
My grad student, whose disciplinary home is natural resources—I was trying to figure out okay, well Journal of Forestry, Journal of Wildlife Sciences, to figure out those journals’ histories and look back. We have their journals. I can look in our catalog and find them and so looking at their table of contents and how things are described over time, but it’s fuzzy. It’s not at all transparent and it’s not like there’s this narrative for each of the disciplines. It’s like there were some of these overarching scientific conversations, but then it must have filtered down somehow and so was that at a conference, was that just in somebody’s hallway? How did it actually happen? That I can’t figure out and how long it took to filter from this 1960s mark to all of these other areas. I can’t tell.
Yeah. But it’s interesting just like when the trail is not – and I think like genuinely it’s interesting. It wouldn’t be as interesting if you were like “oh, the answer is on page two.” Anyway, I’ve been thinking about it for a while both professionally, and I have a curiosity from the historical side, too.
Emily: Your interest is a whole other layer of intellectual labor that’s related to our job serving patrons, which is interesting. Can you tell me a little bit about your relationship with academic publishing?
Sure. At the institution where I work we are tenure-track so some of my relationship with publishing is driven by needing to pursue tenure and promotion in order to be employed. So I started publishing in the library land stuff as soon as I started in libraries and – do with what you want but I’m good at this. This is a thing I know how to do and I’m good at it. So I’ve published 15-20 peer-reviewed journal articles. I’ve written two books. I also have an ACRL publication in the pipeline [“Teaching Critical Reading Skills: Strategies for Academic Librarians”]. It will be an edited collection though.
I’ll back up and just say a little bit on the tenure track. I’m now a full professor so in each of my stages I’ve experimented with some different approaches. Pre-tenure I was probably what you would term a little more conservative. I focused more on getting my articles published and targeted venues that I viewed as higher impact. So that stage of focusing on College & Research Libraries or Journal of Academic Librarianship kinds of places. And then after I got tenure I focused more on publishing in open access venues with less concern for the prestige factor of them and playing around a little bit more with some different research methods and styles as well as trying out some books, which I had never done before. As a science person I’m very article focused so it was entertaining to try out books for a change. I got full professor a year ago and so in my mind that moved me more strongly into a mentoring role so we’ll see what that looks like in terms of publishing myself. I don’t see any need to push myself to the first author position so I’m not sure where that trajectory will go. I’ve written things and I see it as a way to communicate with a larger community for sure, but then the other aspect of publishing it was being a journal editor and so that was not with – so it was sort of a two-facedness about myself. I started that after I got tenure but it was a Taylor & Francis publication. I myself was publishing in open access venues but was not an editor for an open access publication. [sigh] All the choices. I got to see a little bit more of what publishing looked like and interacted with a publisher and what other people’s goals may or may not have been from that vantage point.
Emily: You have goals as an editor. The Taylor & Francis as a company maybe has goals and then the editorial board or the author have goals. I don’t know what your experience was but is there a story or two?
Yeah, or at least notes. Story makes it sound more interesting than it felt. I guess I would say to start that Taylor & Francis was pretty hands off during my whole five year term. The managing editor for the library sciences wing of publishing (if that was their title), the person who was initially in that role switched jobs within several months after me starting. Based on [conversations I had with] the previous editor, that the previous relationship [with the managing editor] would have been a little bit more in depth. And then throughout time there were multiple changes in staff of people I interacted with so– there was no thinking about or strategizing about direction or journal choices or anything. More of just getting the issue off to press. That kind of work. Those people changed many times so the consistency there was very low and it was a small – well it was just me. As the editor I didn’t have an associate editor for most of that time. Most of the work in terms of interacting with Taylor & Francis was just to keep the thing functioning in terms of getting another issue out. So they had no interest whatsoever in giving input.
Here’s the little note story part, I did have questions about open access options, because at Taylor & Francis you can, of course, you can pay a gazillion dollars and have a fully open access version of your article which, of course no one’s going to do. But then you can also have a preprint, [so I asked the publisher] is that true, what would they allow? Would they allow a post-print? You couldn’t put up the actual journal formatted post-print but you could put up your version after a certain amount of time and in an institutional repository. The whole not very fantastic but at least an option.
Emily: It was embargoed.
Not very clearly embargoed. A very short embargo, but yeah. That was an option and so I did interact with them some to make sure that I was clear on the rules, that the linking happened from the journal website which they managed. I couldn’t change anything on the journal website without interacting with a person. I always had to have somebody make all those changes—the this is not centered correctly, etc. Somebody had to do that for me. So I did put up the author instructions for how to post a pre-print.
And then in terms of author goals, I made sure in the article acceptance notices to put in info about here’s what next steps to take; to always include the directions very explicitly on how authors could at least put things in their own institutional repository. I started tracking them by doing Google Scholar searches of what had been published in the journal and seeing what showed up in Google Scholar as in repositories—almost nothing—which is just fascinating to me that folks in this profession are doing so little work themselves to put their own work in institutional repositories! And I know that it’s not as simple everywhere and that not all libraries do have their own institutional repositories so I had started including a link to a more general repository folks could use. Anyway, it just didn’t feel like that was a goal that folks who were choosing to publish there had – that it wasn’t that important to them.
You asked about the editorial board. Also, it was an international journal. There were folks on the board from all over the world and I didn’t recruit all of them because I was the second editor for this journal. Some of them were legacy. I did recruit some of them though, and we didn’t meet very much at all. They didn’t give a lot of input. They served as reviewers on a somewhat regular basis, but in terms of giving direction that was pretty much all on me and it was only if there was a particular need, like we need articles for a special issue. We need help on something. That’s when that would happen but it wasn’t a direction providing body.
Emily: How do you make sense of that authors weren’t really interested in posting their articles in an OA repository?
I can’t make sense of it very much. Sometimes I wonder, so while I’ve been a librarian for what starts feeling like quite a while, I’ve only worked at one institution and I start to feel some of the limitations of that in terms of understanding culture across multiple places. So at my own institution, very soon after I started working there, we passed an OA mandate in our library and then [an OA mandate was passed for] the whole institution—which doesn’t have all that much teeth to it but it was talked about very regularly. It was a high priority. It was something that we were clearly meant to value and I can only assume that doesn’t happen really at all that many places. And the people who were publishing during my tenure [as an editor] came from a mix of places so it wasn’t all R1s by any stretch and that’s great. I’m happy for that. Some comprehensive places, a couple public libraries, not very much, but if the message just isn’t held all that strongly across different parts of our community.
Emily: It seems a little ironic to me, especially given the journal that they were publishing in. [laughs] I just want to note that you rolled your eyes.
Rightfully so. Can I roll them back the other way?
Emily: Despite the OA mandate at OSU, you mentioned that you didn’t really start publishing OA articles until you had achieved tenure. What led you to not pursue that as much until after tenure? Because I would just assume again that your guidelines or your documents for promotion would have something about OA in them if there’s an OA mandate.
So, all of my works are in our repository and for our mandate that was what was requested. So pre-tenure, all that stuff is still in the repository. It was post-tenure that I made the choice to go with fully open access journals that wouldn’t require repository mediation.
Emily: Why is that?
In part to make [long pause] – it sounds overly snooty or something to say I’m helping out those journals, but I guess to make them more of an accessible choice if people are clearly trying to publish in them instead of everybody only trying to publish in Journal of Academic Librarianship, for example, or trying out other venues instead.
Emily: Pre-tenure were you worried about the impact of the journal? What was your concern? Or was it just something you didn’t consider?
I would have had some [articles] that were actually in open access journals but that would have linked more towards the community there. So, for example, a science librarian journal that I published in early on that is online that had open access. It is online. It’s only online. It’s clunky. It’s gotten better so it made sense for the community and that was my goal, so I would say that it was actually more of an outgrowth of me transitioning from my science career—where you have this topic so logically you match it up with this outlet that gets you the most bang for your buck and has the correct scope for what you’re talking about. So those two things being very valuable, that’s what you do and that’s how you move forward [in the sciences]. So that was the messaging I feel like I had most recently and strongly, and that’s what made logical sense to me then. And so it would have been more that it made sense to try and publish in C&RL, which is still open but not quite as easy. So getting in a bigger place was the value I brought from that past experience.
Position: Associate Professor, Simmons University School of Library and Information Science
Fun Fact: Laura likes to sing around the house, and sometimes her dog joins in.
Emily: Earlier you said something about how you tell students what peer review is for. I would like to hear more about what you think that actually is. Why should we be doing peer review and what is its function? What should it be for?
That’s a good question. What I usually tell the students it that it’s supposed to be a quality check so that it’s supposed to be that people with some expertise in the general field of the article or the book are able to look at the manuscript and say first of all overall this is good quality, but in particular to really look at things like what methods were used. Again, assuming this is a research article: What methods were used to study the research questions? What sort of analysis was used, were these appropriate to the questions that were being asked, were they implemented correctly or appropriately? And then whatever conclusions or inferences are drawn from that data, do they seem to line up with the data? So the idea is that someone who should know enough about all of these questions is able to look at that and make some call about whether all of those things were done well enough. But like I said I think that it doesn’t always work the way it’s supposed to.
Emily: Before, when you talked about how you approach refereeing, it doesn’t sound to me like you approach it as just a quality check. It sounds to me like you want to be helpful and collaborative. Does that ring true for you? So you’re telling your students that it’s about this quality check, but in practice you’re doing something different.
Actually that’s a really good point and I think you’re right that my understanding of the peer review, and so again how I relate that to my students is that it’s all about this quality check. Trying to make sure that things don’t get to print that shouldn’t be in print because there are major issues with them. But I think you’re absolutely right. That especially over time I think I have more and more tried to be more almost like a mentor and taking it on almost like a mentoring of trying to give some really helpful feedback with people with this sense of I really want you to get published. Here are the things that I think will help and I don’t know that I’ve really talked about that aspect with it to my students. And yet it’s not like I think that I’m alone in how I approach that either. Even though it wasn’t necessarily how it was ever presented to me either. I think that most of us do sort of try to take on that role and I think that’s important and probably something we should talk about more.
Emily: I’ve never heard someone say that they actually just do a quality check. At least in our field. Most people want to be supportive. They want to be collaborative. You said you had that one grant review and it was like two words. Maybe that person was just doing a quality check. Not to say that your work was of poor quality, but I’m just trying to think maybe their understanding of peer review is different than yours and that’s why. What are you actually expecting from this process?
Yeah, I guess that’s a good point and like I said, we’re never really taught how to do this. At least in my experiences we’re not taught how to do it and we don’t have a really lengthy conversation about it and so I think yeah, I’m sure there’s a lot of different perspectives on what’s the right way to do it or what are we supposed to be focusing on and that’s really problematic.
Let me put it this way. It’s not that I have extensively read the guidelines and aims and scopes and things like that of all these different journals, but I will say of the ones that I’ve read portal: Libraries and the Academy is the only one where I’ve actually seen them say that they take a mentoring approach to their feedback. So they actually have it written into their guidelines which I think is really pretty cool.
Emily: Does it make you feel differently about what you think peer review should do and what it should be for?
Yeah, like I said now that we’ve been talking about it I think that I will be talking about this a little bit differently in my classes, because I think it’s important like you said, at least in our field, to talk to people about this and to just understand the peer review a little bit more broadly and to think about the extent to which we do or do not approach it as mentoring and constructive feedback. I think it’s also important because there’s a push, especially for academic librarians who are on tenure-track, to publish and yet we don’t teach them how to do that usually in our master’s programs. And so if you happen to come from a social science background or a hard science background then you might be all set even without the MLIS preparing you for that. But for a lot of the other students—and something like 70 percent of our students at Simmons come from a humanities background—it can be really challenging in a lot of cases to go out and rate those articles not having been taught how to do it. And so if the peer reviewers take a little bit more of a mentoring approach you might actually really be doing a bigger service to the field even than just what the review itself does.
Emily: I don’t know if you’ve seen any of these peer reviewer courses that have been offered by some of these commercial entities like Publons. I guess I just wondered if you had any thoughts about what it means for a company or a publisher, a commercial publisher, to be filling the gap?
Yeah, it’s a great question because obviously there’s a part of me that there is a gap and somebody does need to address it so I think that’s a good thing. Without having taken any of the courses, I guess I would wonder if they’re going to be replicating certain systems. And if they’re going to be taking a very particular and narrow view of how peer review should be carried out, and again whether they would consider those mentoring aspects of it. I think maybe my sense is that many of us are adopting that perspective. I would just wonder. I think it could be a good thing and again, without having done it I can’t really say, I guess maybe I’m just skeptical of big publishers in general and maybe I would have to really check it out and see what I thought.
Emily: Have you ever thought about or experienced open peer review? What does that phrase bring to mind?
Yeah. I mean my understanding is I guess it could probably be carried out in a couple of different ways, but my understanding is that open peer review would be when either the peer reviewers, the names of the peer reviewers are shared with the author and the authors’ name is shared with the peer reviewers, and/or a third option is that it’s all actually put out there openly so that it can be publicly viewed by the peer reviewer.
I know there was at least one journal that asked me at some point at the end of my review if I was willing to share my name. I’ve never been on the receiving end of that, and I’ve never reviewed someone knowing who had written the paper. In terms of my feelings about it. I’m pretty torn, especially in terms of the idea of sharing the author’s name because just learning what I’ve learned about implicit bias, I feel like it could work maybe either way. The happy thought would be that this would be a way for us to be more inclusive in publishing by being aware of: are we constantly approving these texts that are coming from certain perspectives? But the flip side would be that we’re going to actually consciously or unconsciously block more publications from people of color, knowing that’s how implicit bias works. So I’m torn. I would be happy to do it if I thought it was going to work the way it was supposed to.
Emily: Yes, but opaque peer review is also not working the way it’s supposed to.
Yeah, I mean it just raises that question of which would be the better approach or the lesser of two evils or whatever.
Emily: Well, if they’re both evil why are we doing it at all? Because of implicit bias I wonder, can we require implicit bias training? Can we, in a rubric for a referee, include some thinking reflective questions about bias as we’re reviewing something? In opaque review are replicating the voices that are familiar and so is there privilege inherent in that this person knows how to phrase a question in the performative academic way? And then is someone being excluded from that?
I think probably. Even though I wasn’t necessarily taught how to be a peer reviewer, I was taught how to write a research paper and there was a lot of things implicit in that and I wasn’t even taught that until I got to the PhD program. It’s not just the structure. It’s not just that the lit review comes before the findings. It’s really the phrasing, the style. Consciously or unconsciously when I’m reviewing papers, that’s what’s in the back of my mind. And I imagine that’s what’s in the back of the mind of many reviewers: this is what an academic paper sounds like. Even just as an example, I have one PhD student right now who is using grounded theory for her dissertation. And so she’s using first person and someone on the committee came back and was like “but you’re never supposed to use first person.” Just as one very simple example. So yeah, I think that the system probably does replicate those things. And then two, even in opaque reviews if someone’s first language isn’t English that is often obvious from the writing. I’ve reviewed for journals that have explicitly said we’re really looking to increase our global participation or whatever so yeah, I think there’s a lot there.
Emily: There’s a lot. Can you think of what the ideal model would be?
That’s a great question. I don’t know. Off the top of my head I would really have to give it more thought. I guess I do feel like some sort of review makes sense. You wouldn’t want to just say everybody publish everything and then we all have to try to wade through it. I do think that a more structured approach where people somewhere get training on, first of all, what’s the purpose of this approach, how do we do it, etc. But then really thinking more broadly about what qualifies as an academic paper, what questions are appropriate, what methodologies are sanctioned. All of that kind of stuff, and figuring out what biases we all are bringing to that when we do those reviews. Because even in LIS I think there’s some bias towards certain kinds of methodologies over others. There’s certain methodologies being more empiricist or more objective. And I think we need to have a broader conversation within the field. I think that goes beyond just peer review and it’s really kind of thinking within the field. What are we trying to do here?
Emily: Why do you think we have that?
I mean my sense is that it’s not just an LIS problem. It’s a problem across all different fields and I think that these are things that sadly I am just really beginning to become aware of. It wasn’t something that I necessarily thought a lot about as a student coming up through the ranks, but just the idea that for so long so much of the power structure was white and male and that within that structure, then, certain decisions were made about what is a good academic paper, what is formal academic writing, what are the sanctioned questions and the sanctioned methods? And then once that happened, many of us were just kind of replicating the system without ever questioning “why is this particular methodology superior to that methodology?” or “why are we not interested in these questions?” It’s something that I have been talking a bit about with some of my PhD students, because a number of my advisees have gotten push back in different areas where they’ve been told “oh, that’s not really a good question for your dissertation” or “you really can use that method but you have to pair it with this other method.” And my approach has really been, “this is your project and we want it to be the best it can be, but I want to work with you to make it your project,” if that makes sense. And that’s getting off on a little bit of a tangent, but I think if we could start thinking more broadly about just what kinds of things are worth pursuing or what methods are acceptable to pursue them with, I think it would be better off for it. We would have a richer field of literature.
Emily: People who are working towards tenure can’t bite off a project that is a longitudinal study and get promotion and tenure necessarily because you have a timeline. Maybe people also don’t understand qualitative methods. Quantitative methods are sometimes arguably quicker to write about. At what level do we reinvent the culture of promotion and tenure because it is so limiting (and in my view a performance of whiteness and a performance of maleness in a lot of places). On the other hand, being tenured I feel secure enough to be able to say no. I feel secure enough to be able to say “I think that is a very bad idea.” Are there ways that we can change the culture of promotion and tenure and keep the good things about it?
I don’t know but it is an excellent question. And in some ways I think I may have been in a little bit of a sweet spot at Simmons because we’re not a research intensive university. Of course I felt pressure to publish and things like that, absolutely. But I feel like I probably had a little bit more leeway in terms of what counted or how much was expected in comparison to maybe some of my colleagues who are at R1 universities. I guess I’m not exactly sure where I’m going with that, but just to say that yeah, I think the culture is pervasive for sure but even if there’s some gradations and I think that when so much emphasis is put on research and on quantity within a certain time period, it really does drive people to focus on things that they know they’re going to be able to push out and that are going to have enough traction to get accepted. Like you said there are some really good things about the tenure system in the sense that once you are on the other side you have a lot more leverage to push back on some of those things.
Emily: Yes. I feel that replication too in terms of if you achieving tenure means that you are writing the article that the reviewers want because you know you need it published. And so maybe that is dictating your research methodology. Maybe that’s dictating the voice in which you’re writing your article. You’re not able to subvert the system, but still do really good work because the system wants this and you have to play with it in order to get to that place of power. I’m privileged because I have some power now because I am tenured. I think that’s the only power I have. These are the things I like to ruminate about and I appreciate you listening to me.
It’s fascinating. I’m really glad we got a chance to talk. I really appreciate having this time to talk about it because I think in a lot of ways, I feel like I’ve been approaching all of this and even teaching about all of this. Even though I felt like I was thinking about it a lot, this has really gotten me to think about it more and to articulate some ideas that maybe I was putting into practice without ever having articulated. So I think it’s great. I think it’s really helped me to think a little bit about how I will share these things with my students going forward, certainly, and even as I’m peer reviewing continuing to think about how I do that. I also just think, I mean you’ve raised some really great questions about the process – things that I know as a field we’ve been grappling with. But there’s such a shift in scholarly communication and publishing right now, that it does make me wonder if we might not be coming up on a good time to try and push for some kinds of changes. I’m not exactly sure how you do it, but I mean I published two textbooks. They are both coming out, and one is going to be coming out next month in open access.
Yeah, but that whole process, going through that process has made me think about just the whole idea of traditional publishing and again, the advantages that it brings and the disadvantages that it brings and I think peer review is, especially where it’s been attached in a lot of ways to traditional publishing, I think it’s at an inflection point. So it’s a good time for us to be having these conversations.
Today’s post is the second in a series of three parts that comprise Laura’s story. The first part,Just trying to get them to think about the nitty gritty of the process, unpacked her approach to teaching peer review in the classroom. The portion of our conversation shared in this post discusses Laura’s experiences as a referee, and her reflections on how she learned to do refereeing work.
Position: Associate Professor, Simmons University School of Library and Information Science
Fun Fact: Laura just got elected Secretary of the Board of Trustees for the Somerville Public Library.
Emily: So how did you learn to referee? Was it just practice over time? Did anyone mentor you?
I think it was a little bit of a few things. I wouldn’t say I was really mentored per se, except that I did work with two faculty members who were editors of a journal—even just by observing some of the things that they did. Both of them gave me feedback on some of my early drafts of articles. So kind of seeing how they did it I think actually was helpful. I guess there was some mentoring. I think the more that I was teaching—I don’t want to say that peer review is like grading papers because it’s not exactly—but there was some sort of give and take there where I also learned over time how to give better feedback on papers, just again by practicing and doing more of it. And so I feel like over time I just learned how to give better feedback in general. Things that were more constructive, more specific, etc. And then I think also some of it came from being reviewed myself and recognizing the things that were helpful and the things that were less helpful.
Emily: It sounds like you were self-directed in learning about it. It was just experience.
Yes, I think so. I mean I would say there wasn’t really anything in any of my formal training as a master’s student or PhD student where we were actually really taught how to do that. I mean there may be some places where this is happening, where it’s being done better now because I know a lot of my colleagues, for instance, are having their students peer review each other more, and they’re doing it with more guidance. Again, reviewing your peer student on a paper they’re writing for class is not exactly the same as peer reviewing a journal article, but at least you learn a little about giving feedback. I had been an English major and so how you write papers as an English major is pretty different than how you’re supposed to write research articles. And once I took the research methods class as a PhD student, essentially where they really broke down a research article, a few things clicked for me. It was like okay, now that I understand the structure of this article, I also know what I’m looking for. It was helpful for me in writing articles, but then over time as I reflected on it, it became helpful in terms of how do I analyze an article.
Emily: Do you have any experiences or anecdotes as a referee that are sticking out in your mind?
There have been one or two times where I’ve gotten some articles that started off strong, so they posed a question that I thought was really interesting, but then it just really seemed to fall apart. And those are hard because I want to be supportive in my feedback—and I’m always thinking about the fact that these are my colleagues even if they’re not people that I know directly. These are my colleagues who, like me, are trying to just do their work and do it well. So I think what’s sticking out is in some of these cases where I feel like there’s really some major issues, but trying to think about: do I just say reject this, forget about it? Or do I say, “here are all the things that you could do to try and fix this article.” Like I said I’ve been leaning more towards just giving the more detailed feedback and trying to encourage people. Again, we’re also working in a field where in most cases the issues that we’re looking at are either not executing the methodology as well as they should, or not doing a good job of analysis. It’s not like clinical trials or medical research or anything like that where anyone is going to be harmed necessarily by this. I feel like there’s more opportunity to say “look, just keep working on it. I think there’s something here that you really need to tease it out, give it more support.”
I do remember one time, and this was quite a while ago, I did suggest rejecting a particular article and it was mainly because it was incredibly, it seemed to me to be a very narrowly researched question with an incredible small sample size. And what’s interesting is that article did get published [and I don’t know that it went through a second round of review] or if it did it didn’t come back to me.
…. it really kind of made me think of was I using really looking at it from the perspective of this particular journal…. So basically I rejected something but it got published by that journal anyway, and it really got me thinking about whether I was not understanding what the perspective of the journal was. And it’s not as though I review always for the same journal either and so trying to remember and think about who is the readership of this journal and what is really appropriate for them I guess.
Emily: How do you think we could solve that problem? What would you as a reviewer need to be able to do that work better?
I think part of me worries that the information is all there and maybe I’m not just taking enough time to think about it. Like I said a lot of times they will send you some guidelines and things like that about the journal. In some cases I’m just thinking like “oh, I know this journal. I’ve written for this journal.” To an extent I think that the information in a lot of cases is there and we just need to take the time. I also think we probably could do a better job of packaging and sharing that information. And I think one of the problems – like I said we’re already volunteering our time to do the peer review. It takes a substantial amount of time to do it well, and if I have to add another – even if it’s another 20 minutes or half an hour to read through all of these guidelines and get this particular perspective—it is a challenge. It makes it harder for me to agree to do it, so I wonder if there’s some way without losing the quality and all the nuance. If there’s a way to just really, in a streamlined way, just say, “here are the things we want you to look at when you’re looking at this article. Here are the things you really need to know about the readership of our particular journal.”
This is a little bit tangential, but the other thing I have found really frustrating is that some journals give you a template. Often it’s set up like a survey where there’s a few closed-ended questions that you have to fill in the dots. Sometimes the questions do not line up with the article that you have just read. That’s really challenging and I think that’s something they need to think about. If you publish a range of types of articles, maybe research articles and more philosophical kinds of things, you might have to have a couple of different templates for people to respond to.
This story comes from Laura Saunders, a professor at the Simmons University School of Library and Information Science. We spoke in late July of 2020. This story will be published in three parts. This first chapter discusses Laura’s approach to teaching peer review in the classroom.
As you read through Laura’s story, you may find tension in her telling of how she teaches peer review in the classroom and how she herself approaches peer review when she practices it. (I know I noticed it.) We got to think about that tension together. Maybe her story will help you uncover the tensions in your own teaching practices as they relate to the practice of the thing itself.
Position: Associate Professor, Simmons University School of Library and Information Science
Fun Fact: Laura is currently studying Italian
In addition to her own research and publishing, Laura works to elucidate the peer-review process for her students. Among the classes that she regularly teaches, she teaches information Sources and Services, a user instruction and information literacy class, and an academic libraries class. Prior to becoming a professor, Laura was managing a small career resource library. It was affiliated with an academic library that also housed career counselors. Half of Laura’s time was spent providing reference and instruction services as well. She then pursued a PhD as a part-time student while working, and landed at Simmons. In our discussion Laura talked about learning to referee and refereeing, teaching peer review to students, and we also ruminated together on editorial power and open review.
Emily: Can you tell me a little bit about why you wanted to participate in this interview?
I think it’s a few reasons. One reason of course is just because I have been a peer reviewer and been the subject of peer review. So the process definitely interests me, but it also interests me in a little bit more of an academic sense for a couple of different reasons. One being because I teach librarians, emerging professional librarians and so part of what we talk about is peer review in the sense that some of them, especially the ones who want to be academic librarians, are going to have work with patrons who will have to look at peer-reviewed papers for their research, and so part of what they need to be able to do is not just locate those papers, but really understand what peer review is and what the process is so they can help their patrons understand that. Some of them are going to go on to publish themselves, and so they might be involved in that.
One of my PhD students who recently graduated, did her dissertation on financial conflicts of interest in scientific publications. And part of what she looked at—even though it was a little bit outside of the scope—she ended up including in her review some information about scientific things that had been retracted, and so failures of peer review. In that regard it’s kind of an interesting question.
Emily: How do you teach peer review in the classroom?
I actually it’s a little bit of a mix of trying to just give as good an overview as I can of what the process is and how it’s supposed to work. So just kind of explaining to my students why do we do peer review and what is it that we’re hoping to accomplish by this and then how does it work. It’s this idea that I focus a lot on peer-reviewed journal articles, so I’ll talk about that process of somebody writing an article, sending it out and that it will get reviewed by two experts in the field who will give feedback, and then you’ve got this range of possibilities from accept to reject and everything in between, with major revisions and minor revisions and so on. I’ll kind of go through that on a really basic level, and then from there I’ll usually just share with them some of my personal experiences because I feel like it helps them to understand, especially those who haven’t ever been through it.
I’ll talk to them a little bit about when peer review fails, and at times they’ll get really, I think rightfully, kind of annoyed with the idea that people aren’t doing their jobs. But then I’ll often ask them, “Well how much do you think peer reviewers get paid to do this?” And there’s this look. And people are looking around. And when I tell them, “Nothing. We do this out of professional courtesy,” it gets them thinking more about what is really involved, what the labor is that’s involved, and why it may not always be done as thoroughly as we would hope and expect. So just trying to get them to think about the nitty gritty of the process. Because on the surface level it sounds like something that should work really well, and it’s a nice setup and whatever, but thinking about all the little pieces and how they fit together and the fact that most of the time reviewers are doing this work without getting paid—and you do get some professional return on it in the sense that you can say you did this service to the profession, but since most of it is blind peer-reviewed no one is ever really going to know what you did and how well you did it. So just talking through all of that with them and sharing some of my experiences.
Emily: Do you have particular experiences that you find yourself sharing over and over with them?
I suppose so, yeah. There’s one, and it’s funny because I don’t really remember what the review itself was, but it was one of those things where the advice really stuck with me. It was while I was working on my PhD I was collaborating with my advisor on a paper. We sent it out and it came back with a request for revisions. I was reading through it and talking through it with him and he said something along the lines of, “you have to understand, sometimes people are reviewing the paper they wish you wrote instead of the one you actually wrote.” And it made something click with me that I have found to be helpful going forward. Sometimes when I get reviews now, instead of reading them as being overly critical or they just didn’t understand, I’ll start thinking about, “okay, what might have been in their mind? What were they looking for that was different from what I was actually giving them?” I share that story quite a bit.
I was recently collaborating with a master’s student who ran a survey for an independent study that I oversaw. I thought it was a really interesting research question, and I encouraged her to publish it, and I worked with her and we ended up going through two full rounds of revisions and at one point she asked me “is it normal for this to happen?” I could tell that she was feeling maybe a little bit like they were saying it wasn’t good work or something like that. As we talked through it and then as we got to the final round, she said to me “wow, I really see how much better this paper is now and how it was really worth it.” The paper is going to come out pretty soon.
I’ve seen that full range of the feedback that really isn’t particularly helpful. I submitted a grant proposal two years ago and it was reviewed by five people and they sent all five reviewers scores. It was interesting because the person who gave me the lowest score gave me two words of feedback. I have shared that story with my classes since then because I said, “what am I supposed to do with this?” I mean they told me nothing. If you’re going to volunteer to review you should at least put some effort into it.
Emily: At a journal you might have an editor that will tell you where to focus, what to work on. Does a grant officer play that same role?
Yes, at least for the one that I applied for they do. You’re able to meet with them and they can give you some high-level feedback. With regard to that particular grant proposal that did not get funded, she couldn’t go into a whole lot of detail, but she was able to give me a sense of what was the general consensus. What are some of the areas that I might work on to strengthen it the next time around and stuff like that.
Emily: You mentioned something about this whole idea of reviewers reviewing the article that they thought that they wanted to read versus actually reading the article that was submitted. Can you talk a little bit more about that?
I can remember in particular one study that I did where in two rounds of feedback one of the reviewers both times said something like, “well, you really should have asked this question as part of the survey.” And it was kind of like well, sure. To be honest I’m not sure I even agreed that that question necessarily should have been there, but whether it should have been or not, it wasn’t, so there was nothing we could do about it at this point. I think the first time around I kind of glossed over it, but then the second time around in my response to the reviewers I tried to say something like I appreciate this point and I mentioned in the area for further research that that could be something that someone could look at in the future. I responded, in reality it wasn’t there and here are some of the reasons why it may have been just as well not to have it there. I tried to give a response that I hoped would help them to understand what my thinking was.
Emily: Do you think the reviewer even got to see that comment?
I am not sure. It’s a really good question because I think it depends on the journal. When I have been a reviewer and I’ve gotten a second round, it has worked both ways. In some cases I just get a new revised article and in some cases I get the revised article along with my original comments and any response that the author wrote so I think it really depends on the journal.
Emily: Are you screaming into an abyss where you’re offering this response to feedback and the only person reading it is an editor? Where does that conversation go?
It’s a great question and, in fact, that made me think, too, just recently I submitted an article. I’ve been trying to look at how faculty members in LIS teach soft skills in their classrooms, so things like interpersonal skills, and I’m working with a collaborator, and we ended up deciding to survey people outside of LIS, too, so we could do some cross-disciplinary comparisons. So we sent the article out, the one that looks at these several fields that we surveyed. I sent it to one journal in higher education that I thought looked like a good match. We made a few points about active learning, that in order to learn these kinds of personal and interpersonal skills you really do need to practice, because interestingly and probably not surprisingly, a lot of the top methods the people said they used were readings and lectures. How well is someone going to learn interpersonal communication if all they’re doing is sitting and listening to a lecture? So it seemed like a good match, and the editors sent it back and said, “I’m sorry but I’m not going to forward this to the reviewers because your suggestions” – she said something along the lines of “you’re making suggestions about how to change classes that instructors could not use in their individual classes. Those kinds of changes would have to be made at a program level.” I was thinking, “no, they wouldn’t actually.” But again, it was kind of this awkward position where I definitely disagreed with what the editor was saying, and she wrote me this really long thoughtful response but I kind of felt like, “no, I really don’t think this is the case,” but I also felt like she had made her decision and the way it was written it just kind of seemed like it was pretty clearly a no and it didn’t seem like it was worth it to write back and argue. But there was a part of me that thought this could have actually been an interesting conversation and who knows, maybe even one that might have been of interest to the readers of that publication.
Emily: I was thinking about the power that this editor had in terms of selection bias of what’s getting published in that journal and what’s actually getting out in the literature and how it’s approached. So I feel like peer review is one of those where, without practicing it, how do you learn it?
Well I think you raise a really good point about the power that the editor wields. Definitely this was interesting so it was one of the very few times that I had a general editor reject something out of hand. And I did think that was interesting because like you said it does mean that there’s definitely a lot of power there in terms of just deciding what gets through the – it’s a gate-keeping function, right? And so what actually gets through the gate and when one person is doing all of that or most of that, there’s probably going to be some skew towards whatever their particular areas of interest might be, even unconsciously. I think that’s interesting and I think that could have been a case where maybe if it had gone out to a couple of different people, their responses or reactions may have been slightly different. Who knows? It is something that I’ve thought about a lot and again something that I’ve brought back to the classroom.
I remember quite a few years ago when I was first starting to teach in LIS, hearing a story on NPR that they had done a study of articles in health and medical journals. What they found is that, and I’m not sure if I’m remembering all the statistics correctly, but in clinical trials when the results were positive, the articles got published something like 60 percent of the time. When the results were negative, they only got published 12 percent of the time. I remember that really striking me because those negative results could be just as important to know about. But, of course, for lots of different reasons they’re not getting published and there’s a reason to look at those kinds of publishing trends in medicine, but we’ve got to figure it’s happening in other fields, too. I’m sure in LIS there aren’t a lot of people writing articles about how we failed or how it didn’t work for us. Again, you can think of all the reasons why people may not want to write those things, but you do wonder what are all those things that are out there that would be valuable knowledge that aren’t getting published. And then it raises questions, too, about not just the content necessarily, but the voices that are being stifled and the idea that often there’s going to be some bias towards different questions, different kind of methodology, different content, different authors, all of that kind of stuff. I think this idea of this objective blind peer review suggests that all of those biases get pulled out. But, of course, we know that’s not the case.
So as you said I think that does come back to what I try to teach in the classroom, and just really getting folks to think about, because so often and it’s been a long time since I’ve been in the library—teaching as a librarian as opposed to a library faculty member. I remember back when I was in classes where I would see a librarian say to the class, “okay, I know you need to look for peer-reviewed articles for this class, so in this database you can check this little box and get peer-reviewed articles.” And the implication seemed to be, or I was at least afraid that the message the students were getting in these cases were, “once I click that box, I can trust anything that comes back.” I felt like rarely did we have a discussion about how what you get might still be problematic. I feel like those discussions might be happening more now than they were 15 years ago or so. But trying to get my students to think about that and then finally I think it’s absolutely true: we’re not really taught how to peer review. Sometimes the journal will send you some guidelines when they ask you to peer review and they’ll go throw a set of guidelines at you or something. It is absolutely something that I had to learn over time and I think I probably do a much better job of it now. Do you know what I mean? I think about some of those people who got my first reviews who probably weren’t feeling too much like they were more useful than some of the ones that I’ve been complaining about because I didn’t really know what I was doing.