Improving Experimental Evaluation
|
Progress in computer systems research is driven by experimental
evaluation. Our community can only make advances when new systems can be
shown to improve over earlier ones.
Unfortunately, as our paper
ISMM'13
shows, computer science lags far behind other disciplines. Fortunately, it
also shows that we can perform rigorous experiments within a reasonable
budget of experimentation time.
|
|
|
|
We encourage researchers to help us improve the standard of experimental evaluation.
|
|
Read The Truth, the Whole Truth, and Nothing but the Truth: A Pragmatic Guide to Assessing Empirical Evaluations.
Stephen M Blackburn, Amer Diwan, Mattias Hauswirth, Peter F Sweeney, Jose Nelson Amaral, Tim Brecht, Lubomr Bulej, Cliff Click, Lieven Eeckhout, Sebastian Fischmeister, Daniel Frampton, Laurie J Hendren, Michael Hind, Antony L Hosking, Richard Jones, Tomas Kalibera, Nathan Keynes, Nathaniel Nystrom and Andreas Zeller.
ACM Transactions on Programming Languages and Systems (TOPLAS), 38 (4), 2016.
|
Read Repeatability, Reproducibility and Rigor in Systems Research.
Jan Vitek and Tomas Kalibera.
EMSOFT'11, International Conference on Embedded Software. IEEE
2011.
|
Visit the Evaluate Collaboratory, a resource and a hub for everybody
interested in understanding and improving the state of practice
in experimental evaluation.
|
Sign the Letter to PC chairs, arguing the importance of observation and
reproduction studies.
|
|
|
Novel Garbage Collection Algorithms
The
Beltway framework
of collectors, developed with colleagues at the Australian National University
and the Universities of Massachusetts and Texas, not only generalises
existing copying collectors but also makes possible the design of new
collectors that outperform state of the art generational copying collectors.
The Lifetime Aware GC (LACE)
project builds on Beltway to provide a novel GC framework that can
flexibly exploit object lifetime analysis yet tolerates imprecision.
With
Andy King, I developed
a new static analysis and a novel GC framework that allows truely
independent collection of thread-local heaps. In contrast to previous
work, our solution safely classifies objects even in the presence of dynamic
class loading, requires neither write-barriers that may do unbounded work,
nor synchronisation, nor locks during thread-local collections; our analysis
is sufficiently fast to permit its integration into a high-performance,
production-quality virtual machine.
|
Visualising the Heap
Deep understanding of program behaviour is essential to the design of the next
generation of garbage collectors and explicit allocators. Until now no
satisfactory tools have been available to assist the implementer in gaining
an understanding of heap behaviour. GCspy
is an architectural framework for the collection, transmission, storage
and replay of memory management behaviour. Its architecture allows easy
incorporation into any memory management system: it is not limited
to garbage-collected languages. It requires only small changes to the
system in which it is incorporated but provides a simple to use yet powerful
data-gathering API, that scales to allow very large heaps to be visualised
effectively and efficiently. GCspy allows already-running, local or remote,
systems to be visualised and those systems to run at full speed outside
the points at which data is gathered. Its visualisation tool presents
this information in a number of novel ways. GCspy is the product of an
Memory Management Network collaboration
between the universities of Glasgow (Tony Printezis) and Kent (Richard
Jones). Screen shots can be found here.
|
Distributed Reference Counting
Luc
Moreau, Peter Dickman and I have been investigated how to formalise
distributed reference counting, starting with the Java RMI algorithm.
We used our formalisation to derive an invariant-based proof of correctness
of the algorithm that avoids notoriously difficult temporal reasoning.
A novel graphical representation of the algorithm allows intuitive explanations,
and helped uncover a bug in the original. It also extends to describe
fault tolerance in a clear way..
|
Distributed Garbage Collection
My
PhD student, Dr. Helena Rodrigues, and I used a reference listing
scheme augmented by partial tracing in a garbage
collector for distributed systems that can reclaim distributed, cyclic
garbage data structures. The collector is designed to be flexible thereby
allowing efficiency, expediency and fault-tolerance to be traded against
completeness. The algorithm places no overhead on local collectors and
suspends local mutators only briefly. Partial tracing of the distributed
graph involves only objects thought to be part of a garbage cycle: no
collaboration with other processes is required.
|
A taxonomy for Distributed Garbage Collection. I have
also been thinking about a taxonomy for distributed garbage collection.
Sylvain Louboutin correctly observed that such distributed collectors
need to free themselves from the legacy of centralised collectors. A taxonomy
that avoids this legacy can illuminate new areas for distributed garbage
collection research. My (rather old)
Microsoft Research Lecture (7/8/00) presents an outline of such a
taxonomy (PowerPoint show).
|
|
Students & post-docs
Here's what became of some of my former students and post-docs:
- Carl Ritson went to work for a start-up, RSRCHXchange.
- Matthew Mole works for Kent Police.
- Laurence Hellyer works for Smith & Williamson, a financial services company.
- Tomas Kalibera is working with Oracle and Purdue University on the R statistics
language in Prague.
- Chris Ryder went to work in one of ARM's compiler groups in
Cambridge.
- Sebastien Marion founded
Comufy, providing
novel ways for businesses to communicate with their customers.
- Andy King won first place
in the graduate division of the 2003 ACM
Student Research Competition for his work on Removing
GC Synchronisation.
After joining Microsoft in Redmond, USA, Andy now works for VMWare in Palo Alto.
[thesis]
- Eric Bodden, an undergraduate
project student, won the undergraduate
division for 2004 for his work on a A
Lightweight LTL Runtime Verification Tool for Java (conducted at
the Aachen University of Technology).
Following a PhD at McGill, he now heads the Secure Software
Engineering Group at the European Centre for Security and Privacy by Design (EC SPRIDE).
- Helena Rodrigues
is an Assistant Professor at Departamento de Sistemas de Informaçao
at Universidade do Minho.
[thesis]
- Stephen Thomas joined the University of Nottingham as a
Lecturer before moving to Insignia Solutions, where he led the GC
work for Insignia's Jeode JVM.
He now works for Intel in Cambridge.
|