Evaluating Scientific Impact

Current practices for evaluating potential hires, tenure cases and grant applications relies almost entirely on research publications and previous grant funding. While this system doesn't seem all that great, I suppose its better than the old system of evaluating on "potential for brilliance." Dorothy Bishop wrote in a Timers Higher Education op-ed that this system of evaluation is not only hurting professors but our entire scientific enterprise. While the topic dovetails nicely with my previous post on improving the scientific journal, in particular not cherry picking what to publish, I want to focus on how hiring committees are evaluating potential colleagues and evaluations during training. In particular how do you separate the role of advisors from trainee ability. By focusing on number of publications and appearance of those publications in "glamour mags" like SCN what are we actually evaluating and what are we missing?

I'm particularly interested in the effects of undergraduate experiences setting up future scientists for different career trajectories. I chose to study at a small liberal arts college where I was able to gain broad training as well as play intercollegiate sports and participate in a number of other extracurricular activities like research. However, research activities at small liberal arts colleges that don't place as much stress on independent research for their faculty (in contrast to schools like Swarthmore and Williams which are small liberal arts colleges that expect productive research agendas) are different from R1 research institutes. At R1 institutes, the bigger the base, the higher the peak. That is usually a motto I use to describe my running preparation, but it works for research programs that rely on a lot of cheap labor in order to chase ever more research dollars and research investigations. In R1 labs, undergraduates are encouraged to volunteer in labs as early as their first year where the "research" they will take part in will involve such "research" tasks as washing glassware, entering data into spreadsheets, and calling participants. However, by biding their time, by the time they are juniors and seniors, they may have moved up the chain to the point that they are actually conducting research and providing substantive contributions worthy of research publications and if they are in a prestigious lab those publications may end up in a glamour mag. Research at small liberal arts colleges generally starts later but skips most of the janitorial aspects of research and throws you into the fire completing a research project from inception to presentation. In my own experience I didn't start research until the fall of my junior year, but I completed a literature review, developed an experiment, developed stimuli, ran participants, analyzed the data and presented it at local conferences. This led to an entirely new study my senior year that in the end helped me gain distinction in my major. However, while I had what was likely a better preparation to conduct research at a graduate level, I didn't have any publications. So could someone's entire career trajectory be decided at 17? By deciding on a career in research in high-school, what choice of major and school and lab could set you up for possibly reaching the highest heights?

Exploring the further downstream consequences, do publications in undergrad get you into either prestigious (CHYMPS) universities or high-profile labs (HHMI etc) which then lead to possibly more publications or a better chance of publications in glamour mags. Going back to the influence of the advisor, what happens if the culture of the lab is that everyone is just assigned a project and by doing the work they get the publication? What if the a lab is more liberal in putting people on papers for their contributions? Like most jobs, there seem to be certain amorphous/ambiguous attributes that are desired when looking for new hires. In my minimal experience on the market, I've come to understand that the training experience I and many of my classmates had at the University of Iowa Neuroscience program is not what is generally experienced in other programs. I get the impression that most people assume that graduate students just work on projects that are assigned to them and the logical follow-ups to those projects. However, most of the graduate students that I worked with worked on the "themes" of their graduate advisor but were independent and distinct projects that they led. In fact, what I've been told a post-doctoral position is supposed to be for (i.e., proving that you can direct and lead independent research) is what our graduate experience was.

Relating our experience to hiring practices and whether career trajectories are decided at 17 I like to contrast hiring in academia to sports recruiting. Academic hiring looks for the seasoned, but not too seasoned candidate, someone who has put in their dues at the various levels but didn't have any time off the track towards professorship. In sports this approach would be similar to tried and true process of overpaying veteran players who have shown they can produce but may not pan out based on the large investment you make in them. Another approach that is very popular in sports is looking for "high-ceiling" prospects, individuals that look to have the skills to be amazing but are young and unproven. In science two candidates a year apart, one just graduating a the other with one year of post-doctoral training but fairly similar CVs, the job would almost always go to the one with one more year of training.

Going back to Dorothy's article, I'm interested to hear what comes from the meeting in April, but I do not know how scientists will be judged in the future. As the granting and publishing fields change, there has to be a change in how we evaluate scientists, especially people who plan to pursue primarily research careers and people who pursue primarily teaching careers.