To Agile or To(o) Lean

Research is agile. Not only in its software development, but in its entirety. Simply because one rarely knows where it’s going next. Research is, by its very definition, explorative. You have a problem, come up with possible solutions, implement the best, evaluate it, and either continue or backtrack. Adapt to change (new insights). Improve iteratively.

Because you don’t know where it’s going and how to get there, it’s hard to predict research projects. Sure, you have a vision, a plan, and probably even a concrete approach in mind. But some aspect of what your doing has not been tried before, thus, your approach may not work, your plan may fail, and your vision falter. And since time is short and the search space vast, you want to be efficient. Avoid waste. Be lean.

There’s a tension in doing explorative work and minimizing waste at the same time. Exploring means that every step you take may fail. And though one usually learns from running into dead ends, doing so is not exactly efficient. If it happens a couple of times in a row, one quickly feels like making no progress at all.

When deadlines approach and no working research prototype is in sight, we get stressed. As working hours accumulate, we become increasingly short sighted and start to neglect practices we’re not fluent in yet. Work quality drops. We make mistakes. Get more stressed. Work even longer hours.

I found myself locked in this vicious circle only last summer. I was working on an ambitious project with a big deadline closing in, and my prototype just didn’t want to work the way I wanted it to. I threw hours at it. Long working days and heroic night shifts. But I’ll be honest: I didn’t get it to work in time.

My plan failed. This hurt.

Sine I cannot change what happened, the only thing I can do is step back, take a deep breath, and analyze what to do differently next time:

Looking back, I see that, as the deadline drew closer, I slowly went from systematically searching for a solution to randomly trying out things. “Random” in the sense that I jumped onto my first idea about what the problem might be and implemented whatever solution popped up in my head, hoping for the entire problem to go away. The things I tried became ever bigger, until I even completely rewrote a major component I deemed at fault over night. In this desperate attempt to make my changes have more impact and to not waste time going too slow, I gave up control. Regardless of what I did, the results stayed pretty much the same. And there was no way for me to know whether this was because all the changes I did were simply ineffective or because some evened out the effects of others.

Lesson Learned #1: In exploratory work you wanna take small steps. Try something, evaluate it, and then either continue or backtrack. Fail fast. And when you do, ensure you know why. Time spent on this is essential, not waste.

Moreover, in my struggle to reach my goal, I forgot that plans can change. I didn’t take the time to look at where I was and where I could go from there, but focused on where I had originally planned to get and how far away this still was. I entered a tunnel and did look neither left nor right.I failed to see how much I already achieved and to check what was required to get from what I got to a viable result.

Lesson Learned #2: You always learn as you go. Take your time to consider how what you learned effects your plans and goals. Doing so may save much more time than it costs and ensure you get viable results in time. This is the true goal. Whatever else you might have envisioned.

Much to learn I still have. But all is not lost. After reject is before submission.