Academics code

Too Many Things

2020-03-06T00:00:00+00:00

Back when I was a PhD student, I always had too many™️ projects going on in parallel. There usually were one or two things in my own paper pipeline, a handfull of other paper projects I was involved in, a couple of theses I supervised, some papers to read or review, some teaching I assisted with for my funding, some research projects I contributed to for my funding, some programming tasks waiting, a blog post to write, and some upcoming event to organize. At least. Needless to say that, strictly speaking, not all of these projects had to happen (though the things I could’ve dropped mostly happened to be the ones I most wanted to do). But somehow it seemed unavoidable to suffer this kind of multitasking. It was inherent in the job. It would get better once I finished and left for industry. Sure thing!

Well, now, here I am. And it ain’t no different…

The projects certainly changed, but it seems I have to stop kidding myself and admit that the cause for them going on in parallel isn’t the job, but me. Which means I’m in power to do something about it. Great. Let’s dig into the challenge of managing my priorities such that I get the most out of my available time, given there’s always more to do than I can possible hope to accomplish within the said time.

Once again, I found a gem on this topic at Manager Tools. They suggest a simple four-step process to better align my priorities and my doing. Here’s what I did following their advice. For the instructions, I recommend you listen to the podcast.

What have I been doing?

Before I can decide where my priorities are, I first need to collect what I am working on. I sat down with a pen and paper and tried to remember as many things as possible I’d been working on over the last three weeks. Here’s what I came up with:

Develop new Test-Gap-Analysis demo
Support Pilot with Customer A
Develop time-tracking app
Plan participation in IT Career Summit
Support Pilot with Customer B
Support Pilot with Customer C
Support Pilot with Customer D
Plan participation in SE conference
Plan Pilot with Customer E
O3s
Improving Pilots (retrospective)
Participate in orga council (HUB31)
Plan participation in Konaktiva
Develop on Teamscale’s Visual Studio extension
Prepare quote for Customer F
Follow up with leads
Check CfPs for upcoming conferences
Rate CQSE on kununu and Glassdoor
Office management
Write documentation for Teamscale Ephemeral Profiler
Self-development (podcasts and reading)
Check proof of OBJEKTspektrum article

That’s 22 things I had worked on in 21 days. Didn’t sound so bad, now that I saw the numbers, but of course my time ‘s not equally distributed among these activities. And there’s some activities in there that I could’ve easily spent three weeks on each. And how does this align with what I should be doing, anyways?

What should I be doing?

Next, I went for analyzing what I should’ve been doing. I’m working in roughly equal parts in three of our internal teams, namely Pilots, Development, and Marketing. Also I’m preparing to switch from Development to Recruiting, which already reflects in my activities. To match this assignment with my above activities, I categorized the activities, first, by team and, second, by orthogonal activities within the teams. The result looked like this:

Pilots (doing)
Pilots (improvements)
Pilots (sales)
Development
Marketing (conferences)
Marketing (material)
Recruiting
Other (self-development)
Other (office management)

Good news is that I have been working for all of the teams I’m supposed to be working for and I didn’t miss any of the activities within the teams that I signed up for. Moreover, there’s also some time I spent on activities outside the scope of any of the teams, which is to be expected. So far so good.

Now, nine activities are certainly to much to focus on. Manager Tools recommends choosing three to five key priorities (while the actual goal is getting down to one or two). Since my list wasn’t too long, I decided to go for three and chose those that I judged to carry the highest strategic value (marked in bold above).

How much did I work on what?

Now for the reality check: How much did I actually work on the things I’ve been working on and how does that align with my key priorities?

Luckily, I could go directly to my time tracking and obtain exact data for this step. Using the data from the last three weeks, I got the following list of activities, ordered by relative amount of time I spent on them:

24.0% Pilots (doing)
21.5% Pilots (sales)
9.7% Development
9.2% Recruiting
8.8% Other (self-development)
6.2% Other (processing e-mail)
5.3% Pilots (improvements)
4.3% Other (planning work)
3.8% Other (general meetings)
2.5% Other (office management)
2.0% Marketing (material)
1.3% Marketing (conferences)
1.3% Other (paperwork)

While this data reflected most of my expectations, there certainly were some surprises waiting.

First, I’d forgotten some activities outside the scope of the teams, which consume a significant amount of my time (15.3%). Interestingly, the data already showed the effects of my efforts to process mails and plan work faster, which I found encouraging. Created a todo for myself to check how this evolves in the next months.

Second, I was really surprised by how much sales activity I’d been involved in. While I’d been aware that I was active in that area, I didn’t realize that it consumes that much time! This insight alone made this whole experiment worthwhile. Planning for this explicitly, from now on.

Third, my shift from Development to Recruiting is already visible, as expected.

Fourth, I invested relatively little time in Pilot improvements and Marketing material. This is because I only started working on the topics during the three weeks I analyzed. Similarly, the small investment in Marketing conferences is because there simply were no conferences in this period. This suggest that it would be interesting to look at these numbers over a longer time. Doing this would require some more automation, however, so I decided to stick with what I had for now.

Focusing on Key Priorities

I decided that my number one priority are Pilots improvements. The goal here is to enable our customers to do more work on their own, which should increase the team’s long-term effectiveness. This seems like a good thing to focus on.

To put this decision to action, I scheduled time to work on the improvements on my calendar. I had to skip three weeks into the future, but from then on could schedule two 90-minute blocks per week. This commitment alone strengthened my resolution to focus on this activity.

As of this writing, I’m through the first such week. I had to sacrifice a little of the allotted time and move one block within the week. Nevertheless, I managed to work 2:45h on my number one priority. Not bad at all, right? And it’s on my calendar already for the weeks to come, so I’m confident that I can keep this up.

Finally, I scheduled a repeating todo to reevaluate my key priorities every three months. It’s unlikely that I will repeat the full process described here, but at least I want to consciously decide on my number one priority for the next period.

Overall, I think this experiment was well worth it and I can only encourage everyone who’s working on more than one thing to do it for himself. And, of course, I would be interested to learn whether you draw similarly interesting results from it.

Photoshop Requires Legacy Java 6 (OSX)

2018-04-17T00:00:00+00:00

It is a sad fact that my old version of Photoshop (CS4) requires a Java 6 installation to start. This is especially cumbersome, because OSX keeps removing Java 6 after OS updates (to eliminate the security threat, I assume)… Since it always takes me some time to find where I can download the installer again, I thought I’d share the link here:

https://support.apple.com/kb/dl1572

Hope it safes someone some time. Hi there, future me! 😉

Docker for Cross-platform Research Prototypes

2018-01-06T00:00:00+00:00

Something I like about my job in academia, is the freedom with respect to what I work with. When I first started, I chose and configured my own laptop. When I develop a research prototype, I use whatever programming languages, libraries, frameworks, or IDE I deem appropriate. This opens up a gigantic playground.

As a result, already in my current 4-person office, we have machines running OSX, Windows, and Debian and write our prototypes in at least seven different programming languages (Java, Scala, Python, PHP, Bash, Batch, and C#). This heterogeneity poses a serious challenge to collaboration, e.g., when additional people join projects. Take the following example:

When I started developing MUBench, a benchmarking platform, I quickly found myself with a quite complex setup. For example, in MUBench you run a benchmark experiment with something like the following command:

$> ./mubench.py run ex1 DemoDetector

This starts a Python application that retrieves the source code of benchmarking Java projects, which requires git and Subversion, and compiles them, which requires Apache Ant, Apache Maven, Gradle, and an Oracle JDK 8. Then it invokes the DemoDetector, a Java program, on these benchmarking projects and sends the results to a PHP web application for display, which requires a server and a database. And this is only the tip of the iceberg.

It soon showed that it was a hassle for even my closest collaborators to set up MUBench on their respective machines. In fact, it was already challenging for me to maintain a list of all the dependencies and determine a range of compatible versions for each of them. It became clear that, if I wanted others to use my platform, I desperately needed a more user-friendly setup.

It was around this time that Ben started a discussion about how Docker might help us to more easily produce paper artifacts. I quickly realized that Docker also presented a solution to the portability problem I was facing.

Docker in a Nutshell

Think of using Docker as of using very resource-efficient virtual machines. In the Docker universe, such a VM is called a container. Docker is available for OSX, Windows, and Linux. It makes use of the respective OSs native virtualization capabilities, to be as efficient as possible. In my experience, containers start up in at most a few seconds and do not incur a significant runtime or memory penalty.

The environment of a Docker container is defined by an image. To run a command (e.g., an application) in a specific environment, we instantiate a container from a respective image. The container essentially consists of a reference to the image and a writeable layer on top of it that captures the effect of the command we run. You may think of a layer like a change set in a version-control system. This has two important effects:

The container is very lightweight in terms of disk space, because it contains only the difference to the image and not the entire environment.
The image underlying the container remains unchanged by what happens inside the container, which makes it simple and efficient to instantiate further containers with the exact same environment.

In fact, Docker images are themselves composed from one or multiple layers. Each layer of an image captures the effect of a command applied to the state described by the layers below it. This allows Docker to efficiently reuse (parts of) images and to derive arbitrarily many complex images from the same base image, similar to branching in git version control.

A major difference between Docker containers and virtual machines is that containers are intended to serve for the execution of a single (though arbitrarily complex) command. Once the command terminates, so does the container. To execute another command, we instantiate a fresh container, to avoid stacking layers upon layers upon layers. Basically, this means that containers provide stateless environments. If you need to share state between multiple Docker containers, you may mount volumes into them and read/write data from/to them.

A Cross-platform, Ready-to-use Environment

Primarily, I wanted Docker to provide a platform-independent, ready-to-use runtime environment for running MUBench experiments. To achieve this, I created a Docker image that satisfies all the requirements of MUBench. This allows me to create containers from this image, mount the base directory of the MUBench project from my host filesystem into them, and run the same experiment commands as before, only within the container environment. The example command from above now looks like this:

$> docker run --rm \
    --mount source=/mubench/checkout/,target=/mubench \
    svamann/mubench \
    ./mubench.py run ex1 DemoDetector

It tells Docker to run the MUBench Python application in a fresh container created from the docker image svamann/mubench with the mentioned mount and to delete this container immediately after termination (--rm). With a simple shell script (and a respective batch script for Windows) to handle the creation of the container, I reduced this somewhat lengthy command to again look almost the same as originally:

$> ./mubench.sh run ex1 DemoDetector

As you can see, the use of Docker is almost completely transparent to users of MUBench. I published my Docker image on DockerHub, such that Docker automatically downloads it on first use. Hence, the previously difficult setup of MUBench—regardless of the user’s OS—now consists of only a single step: installing Docker.

In addition to the obvious benefit for other users of the benchmark, introducing Docker also had immediate benefits for myself. For example, I can update my host system without fear of breaking anything that MUBench depends on and I need not bother to maintain a list of MUBench’s requirements. Also, when I had to move my experiment environment to another machine, literally all I needed to do was install Docker on the new machine and copy over the data from previous experiments. Within half an hour, I was back up and running.

A CI Environment

Apart from a platform-independent environment for the use and development of MUBench, introducing Docker to the project had another benefit, which I did not anticipate.

I had long since used Shippable for the CI of MUBench. But since Shippable (and all other current CI service) provides no CI environment with Java, Python, and PHP preinstalled, I had problems with my multi-language project. My quick-fix solution was to use a Python environment and to install the other prerequisites of MUBench in the preparation step of every build. A quite inefficient thing to do.

While I had wondered about the lack of multi-language CI environments, things became clear when I realized that Shippable (and many other CI services) build on Docker. There is simply no need to provide more complex environments out-of-the-box, since you can use any Docker image from DockerHub as your environment. Hence, I derived my own CI image from my environment image (by additionally installing some test dependencies) and configured Shippable to use it for MUBench. E viola. My project-tailored, ready-to-use multi-language CI environment was born.

Just Use My Testing Infrastructure

2017-09-26T00:00:00+00:00

I use the Maven dependency ecosystem to provide code dependencies to my students via a Maven repository hosted on my group’s artifact page. This gives students access to my code, allows them to conveniently view the corresponding source, allows me to quickly distribute updates, and forces me to more cleanly separate my code.

Since I usually insist on automated tests, I repeatedly find myself with testing infrastructure code that I want to distribute alongsite the respective production code from some module, say foo. There’s several ways to achieve this:

I could simply include the testing code with the production code, such that it would be available from foo itself. However, this would force me to turn testing dependencies, such as hamcrest, into production dependencies, which I consider a no-go.
I could create a second Maven module foo-test alongside the production module foo and move all the testing code there. This would allow others to depend on foo-test in the test scope, which resolves the dependency issue. However, since foo-test usually depends on foo and I cannot (and don’t want to) declare cyclic dependencies, I cannot use the testing infrastructure in foo-test to write tests in foo. Therefore, I have to write the tests for foo in foo-test, which I find counter-intuitive. Moreover, it breaks code coverage measurement, because this happens module-wise in Maven, such that running the tests in foo-test does not add to the coverage of the code in foo.
I could just use Maven to solve the problem for me. Turns out this is actually quite simple, using the maven-jar-plugin. This gives me a clean separation of testing and production dependencies and allows the respective code to reside in the conventional locations, at the same time.

To provide all the test code from foo in addition to its production code when I deploy the module, I simply insert the following configuration in the module’s pom.xml:

<project>
  ...
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <!-- Change to the version that fits your environment! -->
        <version>3.0.2</version>
        <executions>
          <execution>
            <goals>
              <!-- Deploy test code as a separate artifact. -->
              <goal>test-jar</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      ...
    </plugins>
    ...
  </build>
</project>

Running maven package will now create an artifact called foo-0.0.1-SNAPSHOT-tests.jar, which contains all the test code and declares the respective production and test dependencies in its pom.xml. This artifact gets deployed alongside the main artifact.

To depend on foo and its testing code in another module or project, I insert the following dependency in the respective module’s pom.xml:

<project>
  ...
  <dependencies>
    ...
    <dependency>
      <groupId>foo</groupId>
      <artifactId>foo</artifactId>
      <version>0.0.1-SNAPSHOT</version>
    </dependency>
    <dependency>
      <groupId>foo</groupId>
      <artifactId>foo</artifactId>
      <version>0.0.1-SNAPSHOT</version>
      <type>test-jar</type>
      <scope>test</scope>
    </dependency>
    ...
  </dependencies>
  ...
</project>

Note that I declare two dependencies, one on the production code and one on the test code, each with the respective scope. By setting the dependency type to test-jar, Maven knows to look for an artifact generated by the test-jar goal, as configured above. The default value for type is jar, which corresponds to the respective module’s main artifact (production code).

And that’s it. Problem solved.

I try to make it easy for my students to use my code. This includes testing infrastructure. Why should they reinvent the wheel? Like all of us, they are more likely to write tests, the easier it is for them to actually do so. No excuses ;)

Back to Schedule

2017-09-09T00:00:00+00:00

As you might have noticed, if you’ve been following this blog through the first half of 2017, I stopped writing somewhen in the beginning of June. This had two reasons:

I had an important deadline coming up end of August, so I had plenty of other things to do.
I had written a burst of blog posts during the ICSE’17 conference, which left me a bit burned out.

While I generally enjoy blogging, it is not critical to me or anybody else, so I follow Matt’s advice on (academic) blogging and allow myself to not blog before important deadlines. However, having stopped my regular schedule, I need to make an effort to get back into it again. Plus, the exhaustion following the deadline kept me from returning to my schedule right after the deadline, which makes it more likely that I don’t return to it at all. This is a cold-start problem. In other words, a motivation problem.

Live blogging during ICSE’17 was an interesting experience, but it left quite exhausted with respect to writing.¹ Additionally, the deadline workload kept me from really recovering from this, because preparing a conference submission requires a lot of writing as well. This additionally reduced my motivation to return to my blogging schedule.

Now, I’m on the ESEC/FSE’17 conference, and just happened to chat with André Meyer about blogging. This made me realize that Academics Code runs the risk of suffering a similar fate as my somewhat-dead screencast project Let’s Developer. A cause of this is that, while I’m generally motivated to produce screencasts and blogposts (because I myself learn much about the respective topics), I find it more difficult to motivate myself to invest the additional effort required to publish the results (such as cutting and encoding video). To do this, I need external motivation.

For example, I ran Let’s Developer for about 1.5 years with a pretty regular publishing schedule (two to three episodes per week). During pretty much the entire time, my flatmate would reliably watch my videos within a day after me publishing them and start to rant about all my mistakes and the questions I had left unanswered the second he next saw me. I learned much from the discussions that usually followed and drew inspiration for new episodes from them. This kept me motivated to continue investing the publishing effort. About two months after I moved out of our shared appartement, Let’s Developer died a sudden death.

Admittedly, the publishing effort for a blog post is much smaller than for a screencast and I receive more feedback on my blog post than on most of my screencasts.² Therefore, this time, it feels like the discussions I had over the last days on ESEC/FSE were sufficient to motivate more posts. Nevertheless, if you like reading this blog (be it because you agree or disagree with my thoughts), I would appreciate you sharing your thoughts with me every now and then. If you do, I assure you, more is to come.

Thank you.

Maybe that’s why Matt advices not to blog too frequently. ↩
YouTube viewers are a surprisingly quiet lot; their feedback is mostly limited to a like or a channel subscription. However, another reason for getting less feedback might actually be that videos are less accessible than written posts, in the sense that many people would quickly read a post while waiting on the bus or in between two tasks, but are less likely to watch a video in the same situation. ↩

About Assumptions

2017-07-21T00:00:00+00:00

Some weeks ago, I attended a talk about IntelliJ IDEA. It was an interactive talk, to the most part, where attendees would ask questions about the IDE and then the speaker, Yann Cébron, would share the thoughts of IDEA developers and himself on the matter. At the same time, he would live demo what he was talking about. I like this format a lot, as a way to get to know new tools or to get to know tools better.

At some point during the talk we discussed IDEA’s “semantic” expansion of the editor selection. An awesome feature I wouldn’t wanna live without. Here, Yann stopped and proclaimed that we would now venture behind the scenes, to get an exclusive insight on how IDEA works it’s magic. That caught my attention. We were to be initiated into the arcane arts!

Then Yann told us that the selection is based on a mysterious thing called the AST…¹

“Well… thanks Captain Obvious,” I thought. “Why bother to stop and make a fuzz around such an obvious thing?!”

Truly, why?

I paused.

The most likely explanation was that, in Yann’s experience, this was not as obvious to everybody in the room as it appeared to me.

Why was it obvious to me, anyway?

I realized that, since my Computer Science 101 courses, I’ve been taught about how parsers, interpreters, and compilers work. You just can’t get through this without abstract syntax trees (ASTs), which is why they became a fundamental part of my understanding of programming languages and of how I think about code. Thinking about it, I have to admit that, though I believe that it’s beneficial for programmers to know about underlying concepts of programming languages, objectively, there’s probably no need to know about ASTs in order to program…

I had assumed that everybody else’s view on things was the same as mine. We all do this frequently, because it is by the assumption of shared knowledge that we are able to communicate. When I talk to another developer, I assume she knows what code and syntax are, what a compiler does (from a user’s perspective), and what an IDE is. After all, where would we end if we had to clarify all these terms before each and every conversation?

When I judged Yann’s explanation of IDEA’s “semantic” expansion of the editor selection, I did this based on my perspective and assumed that everybody else looked at it the same way. Under this assumption, the explanation seemed trivial and boring. Does this assumption hold? I don’t know. And since I don’t know, it’s probably safer to assume it doesn’t hold and verify it, instead of passing judgement as if it would hold.

Implicit assumptions may cause much confusion and miscommunication. Conversely, sometimes, it’s making them explicit what helps others the most when we try to explain something, be it in a talk, a lecture, a discussion, or in a scientific publication. For example, by asking our audience whether they are familiar with ASTs or by referring to supplementary material for those who may not be to read up on.

And also, for our own sake, it is important to, every now and then, consciously reflect on our assumptions and decide, for each single one, whether we can actually take them for granted. For example, understanding that ASTs are not necessarily common knowledge made me realize that things building on ASTs, such as the automated refactorings offered by modern IDEs (and how and why they sometimes fails) are probably not obvious either and, therefore, may be interesting topics for lectures, talks, or blog posts.

Challenge your assumptions.

Thank you, Yann, for this great event and for opening my eyes.

I’m exaggerating a little, for the sake of dramaturgy and delivering my message ;) ↩

Naming Tests

2017-06-18T00:00:00+00:00

Recently, I received an email from a former student assistant of mine, who observed that I name tests different today, compared to when we were working together some years ago. He was curious to learn what guidelines I’m following now and what led me to them. Let me try to recap.

Test Classes

I don’t remember quite distinctly, but I assume that when I first started to write tests, I named tests quite arbitrarily, in the sense that I didn’t follow any particular naming scheme. After I’d worked on my first larger test suites, however, I realized that this makes it difficult to find tests, which led to test duplication and confusion on my part. I started to look for more structure. Structuring of tests, which is closely related to naming, happens on multiple levels. Let’s start by looking at what I consider the most-obvious one: test classes.¹

It is a wide-spread convention to create one test class XTest (or TestX in some languages) per production code class X. This obviously makes it easy to discover the tests corresponding to any particular class. Also, this approach is supported by development tools, such as the Eclipse plugin MoreUnit or IntelliJ IDEA, which makes creating/accessing a test/production class from the respective other straightforward. Therefore, I follow this convention for the most part.

Test Methods

Sometimes, I find that a particular test class grows too large for my taste. I cannot name any specific threshold for this, but rather trust my intuition and my dislike for scrolling around. I find that this is often a sign that my production class is having multiple responsibilities, but there’s also cases where the responsibility just has many corner cases I want to test. In either case, my immediate reaction is to split up the test class into multiple classes and to let this process guide my decision whether to split up the production class, too, or not. To figure out where to split a large test class, I use naming on the next-lower level: test methods.

I think the first naming convention for test methods I encountered was naming them like shouldDoFoo[WhenBar].² I like the idea of this convention, as it nudges us to think about tests as expectations, such as “X should do foo [when bar],” and to formulate their names in a uniform way. Therefore, I followed this quite strictly for a while.

Eventually, I realized that the prefix should does not carry information other than saying that this is a test, which is usually redundant.² Therefore, I started to drop should and to name tests expressing my expectations in plain present tense, like in “X does foo [when bar].” This follows the same idea as before, but makes names a little shorter. Similarly, I no longer strictly use the when to signal the beginning of preconditions.

Splitting Test Classes

Coming back to splitting test classes, I first try to group test method by common name prefixes. This often leads me to groups of cohesive functionality, such as “X finds product by id” and “X finds product by name,” which are candidates for extraction into separate test classes.

I find that when I discover a group of tests that start with a different verb than the other tests, this often leads me to extract the corresponding functionality into a dedicated production class, too, because a different kind of action often corresponds to a different responsibility.

On the other hand, when I find that all my tests use the same verb (or merely one verb and its negation, e.g., “finds” and “misses”), I usually have a case of cohesive functionality with many corner cases, in which case I split the test class, but leave the production class as one. As a guide for splitting the tests in this case, I group the tests by setup/fixture, which is often reflected in the tests’ name suffixes, as in “X fails to find foo in empty database” vs. “X finds foo in database with multiple foos.” For each such group, I create a test class named like XLookupInEmptyDatabaseTest. This has worked for me so far, but I didn’t yet take it all too far, so there may be problems I’m missing here.

Higher-level Tests

So far, I had mostly low-level unit tests in mind. When it comes to higher-level tests,³ there’s some differences to consider. For example, instead of naming these tests after a particular production class, I try to name them after the use case they cover, such as a SellSingleItemTest for a point-of-sale system. This approach usually leads me to cohesive test bundles, where each individual test covers one scenario, such as successful sale, cancelled sale, or unknown item sale.

Since I generally have much fewer of these tests, I have less need for structure and are more liberal with my naming. I focus on expressing the business case in the names, so that future me (or somebody else) understands why the system needs to pass a particular test.

Further Considerations

There’s also test-specific naming going on inside test methods. For example, xUnit speaks of the three test stages: setup, exercising, and validation. Accordingly, the naming scheme given/when/then has been proposed as prefixes for helper methods used in the respective parts of tests. I generally like this idea for similar reasons as the should convention for test names. However, I found that it is difficult to follow, at least in some languages, because much of the vocabulary is mandated by the respective testing framework, such as assertX in many unit testing frameworks or verify in mocking libraries. Especially when using multiple libraries, consistency goes down the river. Nevertheless, I find this framework useful for thinking about test elements.

Finally, there’s also some naming to do on the level of namespaces/packages. Here I mostly follow the convention to use the same package names for test code as for production code and place component tests on the highest level containing production code for the component.

Conclusion

I use test names as a means to give structure to test code, such that it becomes easier to find tests and, thereby, to ensure that all cases are covered (exactly once). The most important factor here is that following a convention helps me to get my thinking straight.

What are your experiences with structuring tests? What, if any, conventions do you follow and why? Where do you see problems and benefits? Where do you agree or disagree with me? And, most importantly, where do you think I’m missing somethings? I’m curious to hear what you e got to say!

I’m mainly thinking about object-oriented languages here, because that’s were I’m most at home. Still, I apply much of my thinking here also when working with procedural or functional languages, and didn’t find this to be problematic, yet. ↩
Many testing frameworks mandate naming conventions, such as prefixing test methods with test_ or annotating them with @Test. I don’t consider this part of the name, as it’s simply a technical necessity. ↩ ↩²
Note that these might still be unit tests. I find that these additional thoughts often already apply when I start mocking other components that the unit under test relies on. For actual integration tests I apply the same principles, but name classes with a suffix, such as IntegrationTest, mostly because I sometimes want to separate such tests, e.g., by pattern matching on class names in a build configuration. ↩

Precise Condition Synthesis for Program Repair

2017-05-25T00:00:00+00:00

In test-based program repair, the problem to solve is to take a program with a test suite and at least one failing test, to generate a match that makes the program pass all tests. A big problem to this approach is that in real-world projects tests suites are often too weak to guarantee correctness. This leads to very low precision of repair tools, such that they are unlikely to be adopted in practice. Therefore, Yingfei Xiong’s work aims to increase the precision of repair tools.

Yingfei presents an approach called Accurate Condition Synthesis (ACS). ACS uses two sets of template for repairs:

Oracle Returning templates that conditionally return (or throw) the expected result (as defined by the test), where the condition is the synthesis task.
Condition Modification which either widens or narrows an existing condition by a synthesized predicate.

The challenge is that the space of possible conditions is too large for enumeration. Therefore, they synthesize the conditions together with a probability of it being correct and to stop synthesis when the probability is too low. Therefore, the authors apply divide-and-conquer: They first rank the involved variables and second rank the predicate per variable. This reduces the search space drastically, especially because the number of variables is usually small.

One strategy to rank variables is to order them by their data-dependency order. An alternative method is to prioritize variables that appear in the JavaDoc documentation. To rank predicates they use the variable type, the variable name, and the enclosing method name. To approximate respective probabilities, they query GitHub and consider only predicates that appear frequently enough.

The evaluation on four real-world projects shows that the new approach has a precision of 78.3% in average, which widely outperforms existing approaches (around 20% precision).

Read the paper preprint.

A Dissection of the Test-Driven Development Process: Does It really Matter to Test-First or to Test-Last?

2017-05-25T00:00:00+00:00

Test-driven development is a testing technique that consists of the three actions “writing a failing test”, “making the test pass with the minimum amount of code possible”, and “clean up the code” that are repeated in a perpetuous cycle.

For their work, Davide Fucci, Hakan Erdogmus, Burak Turhan, Markku Oivo, and Natalia Juristo assume that in reality, developers are likely to diverge from this ideal process and that they probably do not spend equal time in each phase. To investigate on this phenomenon and how it affects the number of mistakes developer make (quality) and the time they need to solve the tasks (perfromance), they conducted an experiment in which they tracked developers interactions with Eclipse to capture when developers are in which phase and for how long. A total of 39 professional developers from 2 companies participated in this study. Each participant solved 3 tasks (2 green-field tasks and one green-field tasks), which led to 82 tracked task-solving action sequences.

Based on the experiment data, they analyzed whether there is a correlation between the adherence to the TDD cycle and both quality and performances. They find that smaller granularity of the activity phases, uniformity of time spend on each action, and less time spend on refactoring correlates to higher quality, as well as better perfromance. Furthermore, it did not matter whether developers wrote the tests or the respective implementation first (i.e., the order of performing these actions did not matter). They conclude that this is most likely because developers work in baby steps in either case, which is what causes the positive effects. The idea of test-first programming is essentially a way to force developers to work in small steps.

Read the journal paper

Learning Syntactic Program Transformations from Examples

2017-05-25T00:00:00+00:00

Reudismam Rolim observes that many edits are performed repeatedly, for example, when applying corresponding fixes to multiple locations. Such edits can be generalized to AST transformation, much like many developer-assistance tools use them to provide automated refactorings. However, in these cases the transformations are predefined manually. Therefore, Reudismam wants to automatically learning such transformations from change examples. He presents his prototype Refazer.

Refazer builds on a new domain-specific language for the specification of transformations (someone from the audience noted that there are many existing languages to express model transformation, which could be reused instead). It uses a domain-specific deductive algorithm to synthesize tranformations in the DSL from change examples and ranks them, by favoring transformations that reuse more existing AST nodes, transformations that consider the context of the transformed nodes, and smaller transformations.

The authors evaluated the approach in two case studies:

In the first study, they applied Refrazer to a set of real-world projects to learn reoccurring changes. They submitted the proposed changes as change requests to developers and found that the tool proposed a high rate of valid transformation. On average, Refrazer needed 2.9 examples to learn a transformation. This shows that the tool can be used to ensure intra-project consistency.

In the second case study, they applied the approach to suggest fix transformations for buggy student-assignments submissions. They select assignment with several correct and at least one broken submission. The transformations learned from the correct submission helped to fix the incorrect assignments in 87% of the cases. Furthermore, Refrazer successfully fixes the student submissions significantly faster than the students themselves are able to, i.e., in fewer corrective iterations.

Read the paper preprint