Fusion 2018 Special Session on
Evaluation of Technologies for Uncertainty Reasoning

Table of Contents:

• Organizers
• Description of the Special Session
• Research Topics that will Develop
• List of Papers/Presentations

Organizers

Paulo Costa                      George Mason University, Fairfax, VA, USA
Kathryn Laskey                 George Mason University, Fairfax, VA, USA
Anne-Laure Jousselme     NATO CMRE, La Spezia, Italy
Pieter DeVilliers                University of Pretoria, Pretoria, South Africa

 

Description of the Special Session

The ETUR Session is intended to report the latest results of the ISIF’s ETURWG, which aims to bring together advances and developments in the area of evaluation of uncertainty representation. The ETURWG special sessions started in Fusion 2010 and has been held ever since, with an attendance consistently averaging between 40 and 48 attendees. While most attendees consist of ETURWG participants new researchers and practitioners interested in uncertainty evaluation have attended the sessions and some joined the ETURWG.

Research Topics that will Develop

One of the main goals of information fusion is uncertainty reduction, which is dependent on the representation chosen. Uncertainty representation differs across the various levels of Information Fusion (as defined by the JDL/DFIG models). Given the advances in information fusion systems, there is a need to determine how to represent and evaluate situational (Level 2 Fusion), impact(Level 3 Fusion) and process refinement (Level 5 Fusion), which is not well standardized for the information fusion community.
The session will focus three topics: (1) to summarize the state of the art in uncertainty analysis, representation, and evaluation, (2) discussion of metrics for uncertainty representation assessment, and (3) survey uncertainty at all levels of fusion.
The impact to the ISIF community would be an organized session with a series of methods in uncertainty representation as coordinated with evaluation.
The techniques discussed and questions/answers would be important for the researchers in the ISIF community; however, the bigger impact would be for the customers of information fusion systems guidance on how to measure, evaluate, and improve systems that assess the situation beyond Level 1 fusion, as far as uncertainty representation and reasoning is concerned.
The customers of information fusion products would have some guidelines to draft requirements documentation, the state of the art in terms of uncertainty evaluation in fusion systems, and exposure to uncertainty reasoning practices that are key to information fusion system designs.

List of Papers / Presentations

Application of URREF Criteria to Assess Knowledge Representation in Cyber Threat Models
Valentina Dragos, Juergen Ziegler and Johan P de Villiers
Abstract:
Systems for threat analysis enable users to understand the nature and behavior of threats and to undertake a deeper analysis for detailed exploration of threat profile and risk estimation. Models for threat analysis require significant resources to be developed and are often relevant to limited application tasks. This paper investigated the implicit and explicit uncertainty assessments to be taken into account for threat analysis systems to be effective for providing a relevant threat characterization. The intent of this paper is twofold. The first is to present and discuss an approach to define a model for cyber threats within a simplified expert model and to translate it into a Bayesian network as a tool for the development of practical scenarios for cyber threats analysis. The second is to address the question of assessing the Bayesian network build and its intrinsic knowledge representation model and to Show how modeling decisions impact the outcome of the system. The paper describes the construction of an expert model and the corresponding BN to analyze cyber threats, investigates various types of induced uncertainty with the URREF criteria simplicity and expressiveness and implements an assessment procedure to evaluate the overall approach.

Experimental Comparison of Ad Hoc Methods for Classification of Maritime Vessels Based on Real-life AIS Data
Max Krueger
Abstract:
Classification of Maritime Vessels is a recurrent task in maritime surveillance systems. Classification can be conducted by different methods, e.g., Naïve Bayes, k Nearest Neighbor, Decision Tree, Fuzzy Rule, or Neural Networks. The Automatic Identification System (AIS) is cooperative system of VHF-radio data exchange. By broadcasting of navigational data and ships' information it supports maritime safety, surveillance, and information. Based on measured AIS datasets of five maritime hotspots, easily implementable standard classification (i.e. ad hoc) methods from Data Science are compared to each other. These are evaluated in terms of accuracy. This experimental evaluation is motivated by the following question: Up to which degree properties and behavior, e.g., vessel's type, can be detected by using large quantities of ship's positional, motion, and dimensions' data as provided by AIS? Future applications might include detection of fraudulent self-declarations of types, e.g., during illegal fishing activities.

Towards the Rational Development and Evaluation of Complex Fusion Systems: a URREF-Driven Approach
Gregor Pavlin, Anne-Laure Jousselme, Johan P de Villiers, Paulo C.G. Costa and Patrick de Oude
Abstract: 
The choices of the uncertainty representations and reasoning methods have a critical impact on the development and deployment life cycles of modern fusion solutions. They influence the development effort, the quality of the resulting solutions as well as the deployment costs. This paper shows how the URREF concepts enable a systematic \modifAL{and rational} development and evaluation of complex fusion systems. In particular, the paper proposes a URREF-driven approach to the development and deployment of composite fusion systems. In this approach the URREF criteria play a critical role throughout the development life cycle as they facilitate informed design choices and enable systematic and tractable evaluation of the resulting systems. The paper establishes relations between the URREF criteria and evaluation subjects in the context of a development life cycle. The concepts are illustrated with the help of a high-level fusion approach supporting estimation of the whereabouts of wildlife poachers.

Latent Variable Bayesian Networks Constructed Using Structural Equation Modelling
Alta de Waal and Keunyoung Yoo
Abstract: Bayesian networks in fusion systems often contain latent variables. They play an important role in fusion systems as they provide context which lead to better choices of data sources to fuse. Latent variables in Bayesian networks are mostly constructed by means of expert knowledge modelling. We propose using theory-driven structural equation modelling (SEM) to identify and structure latent variables in a Bayesian network. The linking of SEM and Bayesian networks is motivated by the fact that both methods are can be shown to be causal models. We compare this approach to a data-driven approach where latent factors are induced by means of unsupervised learning. We identify appropriate metrics for URREF ontology criteria for both approaches.

Information and Source Quality Ontology in Support to Maritime Situational Awareness
Elena Camossi and Anne-Laure Jousselme
Abstract: 
To support situation awareness, the benefit of a variety of sources makes no doubt although it brings additional challenges related to heterogeneity in data format, semantics, uncertainty type, for instance, but also challenges related to possible conflicting information. Information and source quality are intertwined concepts which assessments connect with the evaluation of uncertainty handling in information fusion solutions.While the Uncertainty Representation Reasoning Evaluation Framework (URREF) ontology focuses on assessment criteria, peripheral concepts still play a critical role in the assessment. In this paper, we propose an Information and Source Quality (ISQ) ontology formalising the relationships between information-related concepts, and discuss information interpretation in support of Maritime Situation Awareness.Specifically, this paper links the concepts of Information Source, Dataset and Piece of Information, and connects them to the corresponding quality concepts. Such concepts link to the upper level concepts of the URREF ontology Source (of information) and data Quality. The ontology further expands to the uncertainty modelling and the algorithm design. We conclude on future work and identify avenues leveraging this work, especially the extension to the formalisation of the evaluation process.