User quality and business benefit of new information services using semantic technology

Tom Bösser, kea-pro

In the SEKT project the foundations for new information services were developed which provide easier access to information and offer additional functionality. Three prototype applications (digital library tools for search and collaboration, knowledge management for IT consultants, and a legal information service for judges on duty) were developed to test and validate the concept in real applications. Users were involved throughout the development process to assure that the new services correspond to user needs and requirements, and to carry out conclusive tests which show to which extent the performance of the users in executing their tasks is improved, and if the acceptance by users can be obtained - the essential condition for creating business benefit. Finally, the expected business benefit and cost estimates are considered in relation.

User analyses were carried out in three phases

(1) Early analyses of user needs
(2) Usability testing of the application (and iterative improvements)
(3) Field tests of the applications focused on information quality, user acceptance and cost-benefit considerations for the new applications.
The aim is to provide conclusive evidence of the potential of semantic knowledge technology to create business benefit.

User needs analyses: Questionnaires and methods to quantify user preferences were employed, in total responses from 264 potential users were analysed. Users gave important information about their preferences for the functions and attributes of the 3 new applications. There is a strong preference for “precision” over “recall” of information. Users want the right information at the right moment, appropriate to the context of work, but not simply more information. They look for “comfort”: Easy access to precise, correct and abundant information. Users have requirements, needs and preferences specific to their application domain and context. The 3 applications were developed specific to the demands of their prospective users.

Information quality for users

Quality of information services for users in procedural terms is characterized by the efficiency of access to information, and quality of the obtained information: The share of information which is relevant (”precision”), and the amount of total information retrieved (”recall”). Information quality is assessed by professional users typically by their own personal quality criteria. Prior analyses were carried out by asking 284 users to assess information quality in a variety of information search processes according to a number of criteria. A factor analysis showed that most of the variance is due to a single factor of information quality, a second factor “presentation quality” has considerably less weight.

Quantitative measurement of information search processes by users: A method to assess search processes quantitatively was developed, using the concept of “information foraging”, which assumes that users follow a search-and-browse strategy leading them progressively to their goal. According to the tasks and evaluation objectives, two or more measures are used to quantify search processes, minimally indicators of information quality and of progress in the search process. The end of the search process is defined by the “cut”, the termination of search.

Results of the field tests

The field tests of the three SEKT applications include measurements of performance and information quality in the search process, and user assessment of the entire applications and the new functionality provided. Data collection was performed on-line, in the same browser environment where the users carry out their search, or by independent observers.

Performance: In the legal application, where users work under time pressure, task time decreased from an average of 21 to 7 minutes with the new SEKT application, while information quality and the quality of the resulting document did not change. In the digital library application, task time did not change significantly (mean 15.7 minutes), but information quality was rated higher when the SEKT search engine was used. A characteristic function of progress versus quality of information shows that information quality is significantly higher throughout the search process, with the highest advantage at the start of the search process. (Lines show the linear trend, higher order trends were statistically not significant.)

Assessment of the semantic functionality by the users in the field tests

The results of user evaluation are summarized in a table where “++” denotes overall very positive,”+” positive, and “0” neutral responses.

Assessment of the SEKT functionality is positive in all user groups, as is the evaluation of user satisfaction of the entire applications with the standardized SUMI instrument. Benefits for users created by semantic technology in knowledge management systems are
- Efficiency in task performance, and improved quality of the solution
- Satisfaction of users and improved quality of the working environment
- Reduced workload and stress

Business benefit

Business benefit derives from the gain in efficiency, and the indirect benefit due to user satisfaction. Parameters calculated for a number of completed implementations indicate that the total cost for implementing knowledge management solutions range from 35 to 100  Euro per user per year over a lifetime of five years, assuming 3000 to 30000 users. Additional cost of semantic technology range between 0 and 20 Euro, depending on the baseline system. The cost-benefit relations indicate that the expected business benefit is highly positive. According to these results, semantic technology should become a standard component of knowledge management systems in future.

The decisive and critical success factor which we derive from our observations and experience in the SEKT project is a high degree of acceptance and a high ratio of use of knowledge management systems in the intended user population.

The analysis methods which we have developed are applicable for the analysis of information search behaviour, independent of technology. They can be configured for a wide range of problems, application contexts and languages. Studies can be conducted on location or remotely and within the same web environment where the subjects carry out their work, and under realistic conditions. The approach is applicable in the system development phase, and can also be used to guide organizational development during the system lifetime in knowledge-intensive work environments.