Bayesian Chemistry: How Chemical Experts Improve

One of my favorite books is Nate Silver’s The Signal and the Noise. Silver is well known as the founder of, a data-driven news site covering everything from the American economic outlook to the ethnic distribution of NBA fans. In his book, Silver describes his philosophy of prediction and champions Bayesian reasoning. He sensibly asserts that a firm understanding of statistics and probability is essential for making good predictions. Reading the book leaves me wondering about the intersection of statistics, probability, and chemistry.

Even today, I would argue that statistics and probability are underappreciated in organic chemistry.

Of course, chemical theory owes a great deal to statistics and probability. Quantum mechanics is an entirely probabilistic theory, although the concrete orbital shapes organic chemists tend to draw tempt us to think otherwise. Statistical mechanics is built on the idea that a collection of trillions upon trillions of molecules behaves like a massive sample. No social science experiment could ever hope to approach our sample size! In this context, the challenge is developing a theory that fits our clearly high-quality samples (and the theory of statistical mechanics is notoriously complex).

In other areas of chemistry, however, probability and statistics are unfortunately absent. Chemical reactions and synthesis come to mind: one can imagine a reaction system as governed by a set of probabilities—one for each reaction that might occur. The distribution of products formed before isolation depends on these probabilities. When it comes to organic chemistry, most compounds of appreciable size contain multiple functional groups, each of which is susceptible to reactions with sufficiently harsh reagents. Methods development and synthetic planning both involve minimizing the probability of undesirable processes—even if their likelihoods cannot be reduced to exactly zero. Using computer programs to aid synthesis has fallen out of fashion (unless SciFinder counts), but I can imagine a next-generation synthesis program as a Watson-esque guide that lays out several different routes with probabilities of success or “optimal-ness,” based on data from the literature.* Continue reading →


Prelim Exam Reflections

Well, here I am after last Friday’s prelim exam, having emerged victorious (?) against the forces of pseudoscientific bullshit. Apparently, what I’m doing here can adequately be judged as “science” according to a panel of four experts on the subject. Hooray! The preliminary exam (or whatever your institution of choice may call it) is the gateway to Ph.D. candidacy—a demonstration of graduate work to date and a plan for completing a thesis project. It’s more or less the grad school version of a job performance evaluation.

Here is, in a nutshell, the mental process I went through while preparing for prelim (in soliloquy form):

“Wow, prelim is coming up soon. I wonder if I’ve done enough work to satisfy my committee? I’d better work my ass off.”
“Right, wouldn’t want them to suggest something you haven’t done, but thought about.”
“Hmm, you know, I think I’m the only person in the world who does what I do.”
“O rly?”
“Yeah…so that means, all I have to do is present my work in a well supported, rational fashion, and convince the committee that my stuff is actually an advance over previous stuff.”
“Nice! So you don’t have to know everything about everything?”

Presenting work to a committee of professors in a well-supported, rational manner is much less frightening than the typical prelim image of being locked in a room with the academic equivalent of four ravenous dogs. Fortunately, it’s also much closer to reality. Are they going to make you sweat? Sure, because you’re the only person who does what you do and you have to convince the committee that what you do is worthwhile. Are you going to fail because your committee just plain doesn’t like you? Nooooo…that’s a little thing called illegal. What I learned from prelim is this: do things well supported by theoretical and/or literature precedent, and passing is assured. Those who screw around and half-ass stuff more or less know they won’t pass going in; somehow, it’s the impressions of these squeaky wheels that have determined the mainstream mythos. To adapt a saying hanging over one of my labmates’ desks, to pass prelim:

“Do epic shit (and epic shit isn’t that tough to do).”