Thoughts on doing research

Research is something I find really exciting. There are parts of the world that seem very magical to me, and I want to do my part in figuring out how they work.

Yet I find research projects hard, with high variance—sometimes I learn a lot and/or things work well, and sometimes it's a wash. Because research is fundamentally about figuring out stuff that isn't already figured out, much of this is unavoidable. But some approaches definitely have to be better than others. This is a rough note containing my thinking on what makes research hard, and how to approach it (as someone who doesn't have good research taste yet). Disclaimer: Maybe it's useful to see my thought process, but I wouldn't be surprised if I have very different opinions than what I'm writing here in a few years.

Research projects have several distinguishing properties that make them more difficult:

• They're ill-defined. You pick the problem and think of potential solutions.
• Progress is non-linear, and hard to estimate. When building a particular system in software, you can roughly estimate how long things will take. Granted, it'll probably take thrice as long, but with research it's often hard to make an estimate at all, or even be confident you'll succeed!
• Time Management. Research projects seem to have a never-ending appetite for your time. There are always more things you can try, and knowing how much time to invest in each idea is often quite subtle. Also, maintaining motivation and excitement is more difficult because of the nonlinear progress.

What are possible failure modes that can come from this? You might...

• ...pick a problem that isn't actually important.
• ...work on ideas that have a low probability of working or teaching you anything.
• ...spend too much time on a project/idea (quitting too late)
• ...spend too little time on a project/idea (quitting too early)
• ...execute on ideas too slowly
• ...not learn from "failed" experiments / be too passive
• ...not get better (at the process of doing research) over time

Some of these result from failures of tacit knowledge (research "taste"):

• picking good problems and ideas
• knowing when to double-down vs. give up on an idea

There are probably some things you can do to improve the rate at which you improve your taste, but I don't think it's reasonable to expect yourself to drastically improve your judgement in these cases without a significant amount of practice (probably years).

The other failure modes are failures of execution:

• not executing quickly enough
• not learning from "failed" experiments
• not getting better over time ¹

These seem to be more firmly in your control. There are things that you can do to manage yourself and plan projects such that you execute on ideas at a reasonable pace (whether or not they work out), and spend time thinking about takeaways (both about your ideas and your process).

So what should you do, concretely?

Without a developed sense of research taste, it's hard to avoid the possibility of over-committing time to bad ideas, which can also be demoralizing (and this will impact productivity, and is also annoying).

So here's what seems reasonable:

• Get supervision from mentors with better taste. This is probably self-obvious, but (1) being given promising ideas and (2) having a model that you can imitate over time to pick up on tacit knowledge are the main benefits of mentorship.
  • Meeting regularly (esp. with someone with better taste, who you respect and want to view you positively) is a great way to enhance motivation and guide your thinking.
• Run shorter projects. This serves double duty as it decreases the risk of each project while also tightening the feedback loop for improving taste and execution.
• Lower bound success. Can you pick / design projects such that even if stuff doesn't work, you'll still find out something interesting about the problem?
• Focus / measure your progress locally.
  • Aim for improved understanding (doing science instead of throwing spaghetti at the wall). Instead of judging your progress with "how close am I to having something novel enough to publish a paper?" reframe to "Has my understanding of the problem improved within the last week?"²
  • have deliverables - write your thoughts and have people read them.
    • articulating your understanding of the problem, your idea, and your results will force you to understand it well. It will also make it easier for others to give feedback (which is great, because other's blind spots are only weakly correlated with yours).
• Manage your time well. This is made easier with shorter projects. Estimate how long things will take, track how long they actually do, and put it on a graph.
• Reflect. Do a post-mortem after each project and think hard about what went well and what to update.