10 October 2024

Attendance: 4 in person, 3 online

Workshop retrospective

We recently ran an Introduction to Debugging workshop at the SPECTRUM Annual Meeting 2024 (23-25 September). We began this meeting by reflecting on what went well, and what could be improved for future versions of this workshop.

Key message

Feedback on the workshop was generally positive, although we encountered some technical challenges on the day.

• Some workshop attendees reported an interest in running this workshop at a future WIMSIG event.

• Shortly after the workshop, Emma McBryde informed us that she encountered a difficult coding problem and solved it quickly thanks to the knowledge and skills that she learned from our workshop.

• While almost all of the attendees already had R and Python installed, in future iterations we should ensure that each attendee already has R and Python installed prior to the workshop.

• Limited internet connectivity in the workshop venue meant that we had to ask participants to download the Python and R scripts ahead of time, so we added links to each script to the Resources page.

We concluded that this workshop is a good foundation for planning the debugging and testing topical guides.

• The Debugging Book might be a useful reference.

Future training events

Given the success of the Introduction to Debugging workshop, we asked participants if there were other topics or skills that they would like to learn in a workshop or similar type of event.

Key message

There was broad interest in a testing workshop, and we discussed how good coding practices, testing, and debugging are highly interrelated skills.

Several attendees immediately nominated learning how to test their code as a highly desirable skill, and a logical follow-up to the debugging workshop.

• Saras noted that it's a skill that needs to actively maintained, and that it becomes harder and harder to do well when you haven't done it for a while.

• As Rob pointed out in his presentation at the SPECTRUM & SPARK Annual Meeting in 2023, most of the introductions and training guides only present trivial examples, and these are not sufficiently detailed or relevant for most researchers.

  • We need detailed examples that are relevant to our research domain(s). Obvious candidates include moderately complex compartmental models (both deterministic and stochastic), and agent-based models.

How do people use tests?

Nick asked how other people introduced testing into code bases that are not structured as software packages.

Key message

Perhaps the most difficult challenge is deciding what to test.

• Nick himself has implemented tests as a separate workflow in the Test, Trace, Isolation and Quarantine (TTIQ) repository.

• Rob mentioned that he uses Continuous Integration (on the University of Melbourne's GitLab instance) to automatically run tests for most of his research code. For example, this has allowed him to rapidly catch and identify errors in his pypfilt package.

  • But as Nick pointed out, this approach relies on your code already being reproducible, and perhaps this is too much of a hurdle for most people?

• Eamon explained that he typically begins by writing tests, and makes them a part of his (C++) build system. He sets up his workflow so that doesn't allow running simulations or experiments unless all of the tests are successful.

• Rob explained that he finds tests very useful for articulating his ideas about, and expectations of, his code.

A related question was deciding what to test? It isn't necessarily useful to aim for 100% test coverage (meaning that every line of code is executed by at least one case). So how do you write "maximum value" tests?

We all agreed that it can be very scary to think about having your code subject to scrutiny after publication.

• A high-profile example is Keisha Prem's 20211 social contact matrices paper. This paper received a lot of attention and scrutiny, and a correction was published in 2024 that provided a corrected version of the code.

• Saras mentioned that as a journal editor, whenever a reviewer remarked that a result or output in a paper "looked weird", the authors would investigate and respond that it was an error.

• Rob mentioned that he encountered this situation (as a reviewer) earlier in the year.

Writing testable code

Key message

Attendees reflected on their experiences of testing and debugging. A common finding was that there is an art to writing code that is easy to test and debug.

As we highlight at the start of the Writing code guide, code should be written for people to read!

This is not easy! Much like writing good research papers, this is a skill that takes time to learn, and it gets easier with practice.

• Nick has given a talk, practical functions, about writing code to make it easier to debug and test.

• Saras reflected on a project that began with a large amount of inherited code. The first step was to reorganise the code in order to understand it, and to make it easier to test and debug.

• Saras also shared a relevant quote from the UK Research Software Engineering conference:

  Typically we see of code as a snapshot of a paper, but we could instead consider the paper as a snapshot of the code.

The code is the living project, the manifestation of what you've done. It is the ultimate "methods section", and should be written in a way that explains and clarifies what you've done and why you've done it.

Refactoring

We can't expect to write "perfect" code, because our goals and expectations can change during the research process. So we may need to step back and restructure our code at times — this is known as "refactoring".

This is the coding equivalent of rewriting text (e.g., in a paper) to improve its logical flow and clarity.

A "show and tell" event

Key message

There was unanimous agreement that a "show and tell" event, where people present examples of well-written code, debugging, testing, etc, would be a great way to share knowledge and good practices.

This could also be an "on ramp" for peer code review?

Several attendees said they would love to see a talk about testing the sort of code that researchers write — as opposed to general software — and finding a good balance for research code, which is often only intended for reuse in a rather narrow sense.

• Nefel observed that (some) PhD students do discuss ideas about how to implement certain things in their code, and may even be doing some form of peer code review. She remarked that peer code review during development sounds really cool and fascinating, but also terrifying to do in retrospect — there's an understandable feeling that you don't want to know about errors in retrospect!

• Several attendees stress that making mistakes in your code is not rare at all, and we should normalise it to remove any sense of shame.

• Eamon explained that he likes to write tests while writing code, and develops his projects iteratively, creating test scripts as he goes and retaining them for regression testing.

• Saras responded that she wanted a "show and tell" where Eamon could demonstrate how his build process works.

We ended with a brief discussion about how a "show and tell" event might be structured, and came up with the following suggestions:

• A half-day or whole-day event would probably be more useful and effective than only running for 1-2 hours;

  • We might to obtain a modest amount of funding, so that we could provide basic catering for attendees.

• Give each attendee a short amount of time (perhaps 10-15 minutes?) to present some aspect of their work, coding practices, workflow, etc;

  • This could be something that they found particularly valuable;

  • Or it could be a request for suggestions/feedback about how to improve something — peer code review in miniature!

• Finish by summarising common themes, issues, solutions, etc, and potentially identify some general guidelines for people to incorporate into their research.