Elicitation and data collection

A lot of Gordon's research has involved finding the best choice of methods to get accurate, complete information out of human beings.

This was a notorious problem in the early days of Artificial Intelligence, when it was known as the knowledge elicitation bottleneck. Getting accurate, complete knowledge out of human experts for use in expert systems turned out to be much more difficult than anyone had anticipated.

The same issue is important in a wide variety of other fields, ranging from engineering requirements to market research.

One of Gordon's most significant pieces of research, with Neil Maiden from City University, involved finding an explanation for why this issue was so difficult. The explanation was that researchers had been asking the wrong question. Instead of asking why people don't give you a complete, correct set of information, we should have been asking why people can't give it.

The ACRE (ACquisition of REquirements) paper by Gordon and Neil distinguishes four main categories of knowledge, memory and skill, plus numerous sub-categories, each of which has different implications for choice of method. For example, Short Term Memory has a duration of just a few seconds, so the only way to access it is via real-time elicitation using methods such as think-aloud technique.

This framework provides a systematic set of guidance for choice of elicitation method, grounded in findings from a range of literatures.

The ACRE paper: Maiden, N.A.M. & Rugg, G. (1996). ACRE: a framework for acquisition of requirements. Software Engineering Journal, 11(3).

 

A schematic diagram of the ACRE framework

We've also done a fair amount of research into various elicitation methods, particularly card sorts and laddering. We've tested a range of methods in a range of disciplines, typically producing interesting new findings. Much of this work has been in software requirements, software development and software evaluation, but we've also collaborated with colleagues in podiatry, and in other disciplines.

 

What has come through strongly is that choosing a suitable elicitation method is a key issue in gathering valid, accurate and complete knowledge.