The 20th CREST Open Workshop

The Oracle Problem for Automated Software Testing

Date: 21 - 22 May 2012

Venue: Engineering Front Executive Suite, Roberts Building, UCL

Overview:
Testing involves examining the behaviour of a system in order to discover potential faults. The problem of determining the desired correct behaviour for a given input is called the Oracle Problem. Since manual testing is expensive and time consuming there has been a great deal of work on automation and part automation of Software Testing. However, the problem of automating the Oracle remains a bottleneck that inhibits progress in increased automation for test effectiveness and efficiency. There are several fruitful avenues through which this problem can be addressed including (but not limited to) specification (creation and mining), model and contract based development and testing, metamorphic testing, search based testing and human oracle cost reduction. This workshop will bring together researchers and practitioners working on these approaches to attacking the Oracle Problem to share ideas, recent result and to discuss and develop the research agenda for future work on controlling, managing and automating the Software Testing Oracle.

This workshop is held jointly by the CREST Open Workshop (COWs) programme and the EPSRC RE-COST project (RECOST), a partnership between UCL and the University of Sheffield.

Programme: (Downloadable programme in PDF)

+++++++++21^st May 2012 – DAY 1+++++++++

09:30 Arrival, Coffee and Pastries

10:15 Welcome and Introductions (Video, Introduction)

Mark Harman, CREST Centre, SSE Group, Department of Computer Science, UCL, UK

11:00 Overview of Oracles in Testing (Video, Slides)

Shin Yoo, CREST Centre, SSE Group, Department of Computer Science, UCL, UK and Muzammil Shahbaz, University of Sheffield, UK

11:30 Refreshments

12:00 Runtime monitoring and testing of cloud services (Video, Slides)

Wolfgang Grieskamp, Google Inc., USA

12:30 Supporting Test Oracle Construction (Video, Slides)

Matt Staats, Division of Web Science and Technology, KAIST, South Korea

13:00 Sandwich lunch at the venue

14:00 Testing is nothing but reckoning: An Empiricist Perspective on Testing

Neil Walkinshaw, University of Leicester, UK

14:30 Mining anomaly detectors (Video, Slides)

Paolo Tonella, Software Engineering (SE) Research Unit, Fondazione Bruno Kessler (FBK), Italy

15:00 Popular Delusions, Crowds, and the Coming Deluge: end of the Oracle? (Video, Slides)

Robert Binder, System Verification Associates LLC, USA

15:30 Refreshments

16:00 Automating Test Automation (Slides)

Saurabh Sinha, IBM Research, India

16:30 Oracles and testing at Microsoft

Wofram Schulte, Microsoft Research, USA

17:00 Wrap up and discussion

Mark Harman, CREST Centre, SSE Group, Department of Computer Science, UCL, UK

18:00 Close

18:15 Light Dinner at the Marlborough Arms

+++++++++22nd May 2012 – Day 2+++++++++

09:30 Arrival, Coffee and Pastries

10:00 Oracles in TTCN-3 and UTP (Video, Slides)

Ina Schieferdecker, Fraunhofer-Institut für Offene Kommunikationssysteme (FOKUS), Germany

10:30 When the earth is the oracle

Robert Nilsson, Google Inc., Switzerland

11:00 Metamorphic Testing (Video, Slides)

Tsong Chen, Centre for Computing and Engineering Software Systems, Swinburne University of Technology, Australia

11:30 Refreshments

12:00 Oracle Problem: An Industrial Perspective

Andrea Arcuri, Schlumberger and Simula Research Laboratory, Norway

12:30 Which Test Oracle Should I Use for Effective GUI Testing?

Atif Memon, Dept. of Computer Science, University of Maryland, USA

13:00 Sandwich lunch at the venue

14:00 Implicit oracles from software redundancy (Video)

Mauro Pezzè, University of Milano Bicocca, Italy and University of Lugano, Switzerland

14:30 Carving invariants (Video, Slides)

Andreas Zeller, Computer Science, Saarland University, Germany

15:00 Refreshments

15:30 Mining web pages for realistic test inputs to lower oracle cost (Video)

Phil McMinn, University of Sheffield, UK

16:00 Generating oracles with mutation testing and the crowd

Gordon Fraser, University of Sheffield, UK

16:30 Wrap up

17:30 Close

Photos:

This workshop is supported by the following sponsors:

Registered Attendees: THIS WORKSHOP IS FULL NOW AND REGISTRATION IS NOW CLOSED!

César Andrés, Dep. Sistemas Informáticos y Computación Facultad de Informática, Universidad Complutense de Madrid, Spain
Kelly Androutsopoulos, CREST Centre, SSE Group, Department of Computer Science, UCL, UK
Andrea Arcuri, Schlumberger and Simula Research Laboratory, Norway
Robert Binder, System Verification Associates LLC, USA
Emil Börjesson, Chalmers University of Technology, Sweden
Mustafa Bozkurt, CREST Centre, SSE Group, Department of Computer Science, UCL, UK
Tsong Chen, Centre for Computing and Engineering Software Systems, Swinburne University of Technology, Australia
David Clark, CREST Centre, SSE Group, Department of Computer Science, UCL, UK
Sigrid Eldh, Ericsson AM & Karlstad University, Sweden
Michael Felderer, Institute of Computer Science, University of Innsbruck, Austria
Robert Feldt, Chalmers University of Technology, Sweden
Gordon Fraser, University of Sheffield, UK
Ahmad Nauman Ghazi, Software Engineering Research Lab (SERL), Blekinge Institute of Technology, Karlskrona, Sweden
Wolfgang Grieskamp, Google Inc., USA
Mark Harman, CREST Centre, SSE Group, Department of Computer Science, UCL, UK
Robert M. Hierons, School of Information Systems, Computing, and Mathematics, Brunel University, UK
Yue Jia, CREST Centre, SSE Group, Department of Computer Science, UCL, UK
Jens Krinke, CREST Centre, SSE Group, Department of Computer Science, UCL, UK
Kiran Lakhotia, CREST Centre, SSE Group, Department of Computer Science, UCL, UK
Emmanuel Letier, SSE Group, Department of Computer Science, UCL
Carolyn Mair, Southampton Solent University, UK
Bogdan Marculescu, Software Engineering Research Lab (SERL), Blekinge Institute of Technology, Karlskrona, Sweden
Rafael Martinez Torres, Dep. Sistemas Informáticos y Computación Facultad de Informática Universidad Complutense de Madrid, Spain
Atif Memon, Dept. of Computer Science, University of Maryland, USA
Mark Micallef, Department of Computer Science, University of Malta, Malta
Phil McMinn, University of Sheffield, UK
Matthew Moroz, CREST Centre, SSE Group, Department of Computer Science, UCL, UK
Robert Nilsson, Google Inc., Switzerland
Alberto Núñez, Dep. Sistemas Informáticos y Computación Facultad de Informática Universidad Complutense de Madrid, Spain
Manuel Núñez, Dep. Sistemas Informáticos y Computación Facultad de Informática Universidad Complutense de Madrid, Spain
Mike Papadakis, University of Luxembourg, Luxembourg
Mauro Pezzè, University of Milano Bicocca, Italy and University of Lugano, Switzerland
Wishnu Prasetya, Utrecht University, The Netherlands
Jian Ren, CREST Centre, SSE Group, Department of Computer Science, UCL, UK
Ina Schieferdecker, Fraunhofer-Institut für Offene Kommunikationssysteme (FOKUS), Germany
Wofram Schulte, IBM Research, USA
Ali Shahrokni, Chalmers University of Technology, Sweden
Saurabh Sinha, IBM Research, India
Matt Staats, Division of Web Science and Technology, KAIST, South Korea
Michael Tautschnig, University of Oxford, UK
Paolo Tonella, Software Engineering (SE) Research Unit, Fondazione Bruno Kessler (FBK), Italy
Richard Torkar, Chalmers University of Technology, Sweden
Muzammil Shahbaz, University of Sheffield, UK
Mattia Vivanti, University of Lugano, Switzerland
Neil Walkinshaw, University of Leicester, UK
Shin Yoo, CREST Centre, SSE Group, Department of Computer Science, UCL, UK
Andreas Zeller, Computer Science, Saarland University, Germany

This page was last modified on 16 Aug 2017.