Fusion 2012 Special Session on
Evaluation of Technologies for Uncertainty Reasoning

On this page:

• Objective
• Description
• Anticipated Impact
• Panel 1: Uncertainty Evaluation: Current Status and Major Challenges
• Panel 2: Issues of Uncertainty Analysis in High-Level Information Fusion
• Papers (submitted)

 

Objective

The ETUR Session is intended to leverage on the latest results of the ISIF’s ETURWG, which aims to bring together advances and developments in the area of evaluation of uncertainty representation. The session will be attended by ETURWG participants, as well as other researchers and practitioners interested in uncertainty evaluation. The session will summarize the state of the art in uncertainty analysis, representation, and evaluation. By having a special session, the community can collectively address a common need for the ISIF community, coordinate with researchers in the area, and jointly assess perspectives in various evaluation techniques of uncertainty assessment and key to fusion, reduction of uncertainty.

Description

One of the main goals of information fusion is uncertainty reduction. Quantification of uncertainty reduction depends on how uncertainty is represented. Uncertainty representation differs across the various Levels of Information Fusion (as defined by the JDL/DFIG models). For Level 1 Fusion, standard measures of uncertainty reduction are widely accepted in the community. For tracking, uncertainty reduction corresponds to reducing spatial (distance) and temporal (time) errors; for identification, the goal is to increase the probability of detection and reduce the probability of false alarms.
Information fusion of hard and soft information from diverse sensor types still depends heavily on human cognition. This results in a scalability conundrum that current technologies are incapable of solving. Although there is widespread acknowledgement that an Information Fusion Evaluation Framework must support automated knowledge representation and reasoning with uncertainty, there is no consensus on the requirements such framework must meet, on the most appropriate technologies to satisfy these requirements, and on how to evaluate how well they are being met. A clearly defined, scientifically rigorous evaluation framework and metrics are needed to help information fusion researchers assess the suitability of various approaches and tools to their applications. The ETUR special session will be devoted to foster discussions about this evaluation framework in the context of the most recent results obtained at the ETURWG proceedings.

Anticipated Impact

The impact to the ISIF community would be an organized session with a series of methods in uncertainty representation as coordinated with evaluation. The techniques discussed and questions/answers would be important for the researchers in the ISIF community; however, the bigger impact would be for the customers of information fusion systems to determine how measure, evaluate, and approve systems that assess the situation beyond Level 1 fusion. The customers of information fusion products would have some guidelines to draft requirements documentation, the gain of fusion systems over current techniques, as well as issues that important in information fusion systems designs.

Panel 1: Uncertainty Evaluation: Current Status and Major Challenges

Summary

One of the basic applications of information fusion is to reduce uncertainty. The notion of position accuracy from sensor covariance reduction, confidence improvement from false alarm rejection from multimodal collections, and data filtering to limit cognitive overload are key elements of information fusion techniques to reduce uncertainty. With the advent of the various applications of information fusion, there are many instances of uncertainty from source characterization (i.e. pedigree), limiting testing for robust operations, and association of data over wide gaps in spectral, temporal, or spatial collections. This panel discussion seeks to motivate and highlight the discussion of uncertainty evaluation challenges in an information age. We envision a discussion that utilizes and expands techniques from low- level information fusion to the higher levels of information fusion. The panel is part of the ETURWG and thus has its roots in the development and support of the ISIF ETURWG. To get a qualified and diverse viewpoint, we are inviting you to be a member of the panel.

Moderator

Paulo Costa - George Mason University

Panelists

• Kathryn Laskey (GMU) – Evaluating Bayesian methods
• Erik Blasch (AFRL) – Progress and future of evaluation techniques
• Sten F Andler (University of Skövde) - Evaluation of Uncertainty Management Methods: precision vs. Imprecision
• Jean Dezert (ONERA) - A Fundamental Contradiction in Dempster-Shafer Theory
• Anne-Laure Jousselme (DRDC, Valcartier) - What's in an uncertainty representation?
• Gavin Powell (EADS) - Uncertainty Evaluation: Current Status and Major Challenges

Questions

1. Can an unbiased, general evaluation framework be achieved?
2. What methods are appropriate for Evaluation of UR?
3. What are the main criteria for evaluating UR in fusion systems?
4. What are examples of successful ETUR?
5. Are generated Data Sets useful, available, necessary?
6. What is future of ETUR methods or top unsolved challenges?

Panel Paper

• Uncertainty Evaluation: Current Status and Major Challenges

 

Panel 2: Issues of Uncertainty Analysis in High-Level Information Fusion

Summary

High-Level Information Fusion (HLIF) utilizes techniques from Low-Level Information Fusion (LLIF) to support situation/impact assessment, user involvement, and mission and resource management (SUM). Given the unbounded analysis of situations, events, users, resources, and missions; it is obvious that uncertainty is manifested by the nature of application requirements. In this panel, we seek appropriate discussions on methods and techniques to bound the problem of HLIF uncertainty analysis without alluding to high- performance statistical computational solutions (i.e. particle filters), mathematical assumptions (i.e. optimal Bayesian approaches with maximum likelihood solutions), or rigorous modeling and problem scoping (i.e. expert systems) which lead to time delays, brittleness, and rigidity, respectively. [We can change this sentence for the publication]. Given the various methods of LLIF and the complexity of HLIF, an interest to the ISIF community is to utilize diverse methods (such as those from other communities) that bridge the LLIF-HLIF gap of uncertainty analysis. The panel is part of the ETURWG and thus has its roots in the development and support of the ISIF ETURWG. To get a qualified and diverse viewpoint, we are inviting you to be a member of the panel.

Moderator

Erik Blasch - Air Force Research Laboratory

Panelists

• Paulo Costa (GMU) - HLIF Analysis with Bayesian Approaches
• Johan Schubert (FOI) – Information fusion management
• Dafni Stampoulos, represented by Gavin Powel (EADS) – Evidential methods in HLIF
• Ng Gee Wah – Computational Cognitive System for High Level Information Fusion
• Pierre Valin (DRDC, Valcartier)  - Situation Awareness and Uncertainty
• Rakesh Nagi (University at Buffalo) - HLIF Uncertainty Issues: Hard+Soft Fusion and Uncertainty

Questions

1. Are methods for dealing with uncertainty (e.g. learning, reasoning, using parametric distributions, etc) equally applicable to all HLIF situations?
2. Are preferable techniques of UR for LLIF and HLIF? How to bridge the LLIF and HLIF uncertainty evaluation gap?
3. Is there a way of generalizing when one method is preferable? Is there a theoretical justification for such choice (i.e. based on axioms, assumptions, requirements)?
4. Can multiple methods coexist and act synergistically?
5. How to handle cases when the observations / sensors cannot directly provide propositions about the hypothesis the user are interested in?

Panel Paper

• Issues of Uncertainty Analysis in High-Level Information Fusion

 

Special Session Papers

 

Fusion 2013 Special Session on
Evaluation of Technologies for Uncertainty Reasoning

On this page:

• Objective
• Description
• Anticipated Impact
• Panel: Construction of a unified evaluation framework: lessons learned and future steps
• Papers

 

Objective

The ETUR Session is intended to leverage on the latest results of the ISIF’s ETURWG, which aims to bring together advances and developments in the area of evaluation of uncertainty representation. The session will be attended by ETURWG participants, as well as other researchers and practitioners interested in uncertainty evaluation. The session will summarize the state of the art in uncertainty analysis, representation, and evaluation. By having a special session, the community can collectively address a common need for the ISIF community, coordinate with researchers in the area, and jointly assess perspectives in various evaluation techniques of uncertainty assessment and key to fusion, reduction of uncertainty.

Description

One of the main goals of information fusion is uncertainty reduction, which is dependent on the representation chosen. Uncertainty representation differs across the various levels of Information Fusion (as defined by the JDL/DFIG models). Given the advances in information fusion systems, there is a need to determine how to represent and evaluate situational (Level 2 Fusion), impact (Level 3 Fusion) and process refinement (Level 5 Fusion), which is not well standardized for the information fusion community. The goal of the ETUR Session is bring together advances and developments in the area of evaluation of uncertainty representation. The session will leverage on the current work of the ISIF’s ETURWG, a working group devoted to the topic, and bring together researchers in the area to summarize the state of the art in uncertainty analysis, representation, and evaluation.

Anticipated Impact

The impact to the ISIF community would be an organized session with a series of methods in uncertainty representation as coordinated with evaluation. The techniques discussed and questions/answers would be important for the researchers in the ISIF community; however, the bigger impact would be for the customers of information fusion systems to determine how measure, evaluate, and approve systems that assess the situation beyond Level 1 fusion. The customers of information fusion products would have some guidelines to draft requirements documentation, the gain of fusion systems over current techniques, as well as issues that important in information fusion systems designs.

Panel: Construction of a unified evaluation framework: lessons learned and future steps

Summary

Current advances in operational information fusion systems (IFSs) require common semantic ontologies for collection, storage, and access to various data, sensor, and information. One of the major contributions of information fusion is to reduce uncertainty. With an enormous amount of sensors, measurements, and systems; it is not always easy to determine the uncertainty reduction.  Thus, the Evaluation of Technologies for Uncertainty Representation Working Group (ETURWG) has formed under ISIF. Over the first two years, some lessons learned are: need to (1) define different types of uncertainty, (2) use cases for discussion, (3) multiple perspectives on a topic, and (4) metrics.  The future requires (1) standard data sets, (2) metric standards, and (3) comprehensive terminology. A use case is presented from one of the use cases on the ETURWG for Wide-Area Motion Imagery (WAMI) simultaneous tracking and identification.

Moderators

Paulo Costa / Anne-Laure Jousselme

Panelists

• Valentina Dragos
• Gavin Powell
• Paulo Costa
• Anne-Laure Jousselme
• Juergen Ziegler

Reference Questions/Topics

• What are the main lessons learned of the effort?
• What has been accomplished and how difficult was it?
• Is the current URREF useful? What is missing?
• When one would use the URREF work?
• How to better explain the URREF for people outside the group that built it?
• Should the ontology and associate proceedings be "frozen"?
• What is next for ETUR?

Special Session Papers

A total of 9 papers were accepted for presentation at Fusion 2013 ETUR Special Session. The following presentation schedule was defined in the FUSION 2013 program:

Thursday, July 11th, Room Sakarya A
Session Chair: Paulo Costa

ETUR Special Session: Thursday Schedule

Time	Paper Title	Authors (presenter is highlighted)
15:20 - 15:40	An Ontological Analysis of Uncertainties in Soft Data	Valentina Dragos
15:40 - 16:00	Measures of Conflicting Evidence in Bayesian Networks for Classification	Max Kruger
16:00 - 16:20	Traceable Uncertainty	H. Joe Steinhauer, Alexander Karlsson, Sten F. Andler
16:20 - 16:40	Evaluating Complex Fusion Systems Based on Casual Probabilistic Models	Frank Mignet, Gregor Pavlin, Patrick de Oude, Paulo C. G. Costa
16:40 - 17:00	URREF Reliability Versus Credibility in Information Fusion (STANAG 2511)	Erik Blasch, Valentina Dragos, Kathryn Laskey, Paulo Costa, Anne-Laure Jousselme, Jean Dezert

 

Fusion 2014 Special Session on
Evaluation of Technologies for Uncertainty Reasoning

Table of Contents:

• Organizers
• Description of the Special Session
• Research Topics that will Develop
• List of Papers/Presentations

 

Organizers

Paulo Costa - George Mason University
Kathryn Laskey - George Mason University
Anne-Laure Jousselme - DRDC-Valcartier
Erik Blasch - AFRL

Description of the Special Session

The ETUR Session is intended to report the latest results of the ISIF’s ETURWG, which aims to bring together advances and developments in the area of evaluation of uncertainty representation. The ETURWG special sessions started in Fusion 2010 and has been held ever since, which an attendance consistently averaging between 40 and 48 attendees. While most attendees consist of ETURWG participants new researchers and practitioners interested in uncertainty evaluation have attended the sessions and some stayed with the ETURWG.

Research Topics that will Develop

The session will focus three topics:
(1) to summarize the state of the art in uncertainty analysis, representation, and evaluation,
(2) discussion of metrics for uncertainty representation, and
(3) survey uncertainty at all levels of fusion.

The impact to the ISIF community would be an organized session with a series of methods in uncertainty representation as coordinated with evaluation. The techniques discussed and questions/answers would be important for the researchers in the ISIF community; however, the bigger impact would be for the customers of information fusion systems to determine how measure, evaluate, and approve systems that assess the situation beyond Level 1 fusion.

The customers of information fusion products would have some guidelines to draft requirements documentation, the gain of fusion systems over current techniques, as well as issues that important in information fusion systems designs. One of the main goals of information fusion is uncertainty reduction, which is dependent on the representation chosen. Uncertainty representation differs across the various levels of Information Fusion (as defined by the JDL/DFIG models). Given the advances in information fusion systems, there is a need to determine how to represent and evaluate situational (Level 2 Fusion), impact (Level 3 Fusion) and process refinement (Level 5 Fusion), which is not well standardized for the information fusion community.

 

List of Papers / Presentations

 

A total of 5 papers were accepted to the Fusion 2014 conference and assigned for presentation at the ETUR Special Session:

1. Assessment of Uncertainty in Soft Data
   Valentina Dragos
2. URREF Self-Confidence in Information Fusion Trust
   Erik Blasch , Audun Josang, Jean Dezert, Paulo C. G. Costa, and Anne-Laure Jousselme
3. Detection of Failing Sensors by Conflicting Evidence in Bayesian Classification
   Max Kruger
4. A URREF interpretation of Bayesian network information fusion
   Johan Pieter De Villiers, Gregor Pavlin, Paulo Costa, Kathryn Laskey, and Anne-Laure Jousselme
5. Characterization of Hard and Soft Sources: A Practical Illustration
   Anne-Laure Jousselme, Anne-Claire Boury-Brisset, Benoit Debaque, Donald Prévost

 

Fusion 2015 Special Session on
Evaluation of Technologies for Uncertainty Reasoning

Table of Contents:

• Organizers
• Description of the Special Session
• Research Topics that will Develop
• List of Papers/Presentations

 

Organizers

Paulo Costa - George Mason University
Kathryn Laskey - George Mason University
Anne-Laure Jousselme - NATO CMRE
Erik Blasch - AFRL

Description of the Special Session

The ETUR Session is intended to report the latest results of the ISIF’s ETURWG, which aims to bring together advances and developments in the area of evaluation of uncertainty representation. The ETURWG special sessions started in Fusion 2010 and has been held ever since, which an attendance consistently averaging between 40 and 48 attendees. While most attendees consist of ETURWG participants new researchers and practitioners interested in uncertainty evaluation have attended the sessions and some stayed with the ETURWG.

Research Topics that will Develop

The session will focus three topics:
(1) to summarize the state of the art in uncertainty analysis, representation, and evaluation,
(2) discussion of metrics for uncertainty representation, and
(3) survey uncertainty at all levels of fusion.

The impact to the ISIF community would be an organized session with a series of methods in uncertainty representation as coordinated with evaluation. The techniques discussed and questions/answers would be important for the researchers in the ISIF community; however, the bigger impact would be for the customers of information fusion systems to determine how measure, evaluate, and approve systems that assess the situation beyond Level 1 fusion.

The customers of information fusion products would have some guidelines to draft requirements documentation, the gain of fusion systems over current techniques, as well as issues that important in information fusion systems designs. One of the main goals of information fusion is uncertainty reduction, which is dependent on the representation chosen. Uncertainty representation differs across the various levels of Information Fusion (as defined by the JDL/DFIG models). Given the advances in information fusion systems, there is a need to determine how to represent and evaluate situational (Level 2 Fusion), impact (Level 3 Fusion) and process refinement (Level 5 Fusion), which is not well standardized for the information fusion community.

 

List of Papers / Presentations

 

A total of 5 papers were accepted to the Fusion 2015 conference and assigned for presentation at the ETUR Special Session:

• ​Dissecting uncertainty-based fusion techniques for maritime anomaly detection
  Anne-Laure Jousselme and Giuliana Pallotta
• A critical assessment of two methods for heterogeneous information fusion
  Valentina Dragos, Xavier Lerouvreur, Sylvain Gatepaille
• URREF for Veracity Assessment in Query-Based Information Fusion Systems
  Erik Blasch
• Uncertainty representation and evaluation in mathematical modelling for information fusion
  Pieter et al.
• Gradual and Binary Conflicts in Bayesian Networks Applied to Failure Detection
  Max Krüger

Fusion 2016 Special Session on
Evaluation of Technologies for Uncertainty Reasoning

Table of Contents:

• Organizers
• Description of the Special Session
• Research Topics that will Develop
• List of Papers/Presentations

 

Organizers

Paulo Costa                      George Mason University, Fairfax, VA, USA
Kathryn Laskey                 George Mason University, Fairfax, VA, USA
Anne-Laure Jousselme     NATO CMRE, La Spezia, Italy
Erik Blasch                        AFRL, Rome, NY, USA
Jüergen Ziegler                 IABG, Ottobrunn, Germany
Valentina Dragos              ONERA, France
Pieter DeVilliers                University of Pretoria, Pretoria, South Africa

Gregor Pavlin                    D-CIS Lab, Thales R&T, Delft, The Netherlands 

Description of the Special Session

The ETUR Session is intended to report the latest results of the ISIF’s ETURWG, which aims to bring together advances and developments in the area of evaluation of uncertainty representation. The ETURWG special sessions started in Fusion 2010 and has been held ever since, which an attendance consistently averaging between 40 and 48 attendees. While most attendees consist of ETURWG participants new researchers and practitioners interested in uncertainty evaluation have attended the sessions and some stayed with the ETURWG.

Research Topics that will Develop

The session will focus three topics:
(1) to summarize the state of the art in uncertainty analysis, representation, and evaluation,
(2) discussion of metrics for uncertainty representation, and
(3) survey uncertainty at all levels of fusion.

The impact to the ISIF community would be an organized session with a series of methods in uncertainty representation as coordinated with evaluation. The techniques discussed and questions/answers would be important for the researchers in the ISIF community; however, the bigger impact would be for the customers of information fusion systems to determine how measure, evaluate, and approve systems that assess the situation beyond Level 1 fusion.

The customers of information fusion products would have some guidelines to draft requirements documentation, the gain of fusion systems over current techniques, as well as issues that important in information fusion systems designs. One of the main goals of information fusion is uncertainty reduction, which is dependent on the representation chosen. Uncertainty representation differs across the various levels of Information Fusion (as defined by the JDL/DFIG models). Given the advances in information fusion systems, there is a need to determine how to represent and evaluate situational (Level 2 Fusion), impact (Level 3 Fusion) and process refinement (Level 5 Fusion), which is not well standardized for the information fusion community.

 

List of Papers / Presentations

1. Pragmatic data fusion uncertainty concerns: Tribute to Dave L. Hall.
   Blasch, E., Costa, P. C. G., De Villiers, J. P., Laskey, K. B., Llinas, J., & Jousselme, A.-L.
   https://ieeexplore.ieee.org/document/7527926

2. de Villiers, J. P., Jousselme, A.-L., de Waal, A., Pavlin, G., Laskey, K., Blasch, E., & Costa, P. (2016). Uncertainty evaluation of data and information fusion within the context of the decision loop. 2016 19th International Conference on Information Fusion (FUSION), 766–773. https://ieeexplore.ieee.org/document/7527964

3. de Waal, A., Koen, H., de Villiers, P., Roodt, H., Moorosi, N., & Pavlin, G. (2016). Construction and evaluation of Bayesian networks with expert-defined latent variables. 2016 19th International Conference on Information Fusion (FUSION), 774–781. https://ieeexplore.ieee.org/document/7527965

4. Dezert, J., Han, D., & Yin, H. (2016, July). A New Belief Function Based Approach for Multi-Criteria Decision-Making Support. FUSION 2016.

5. Heendeni, J. N., Premaratne, K., Murthi, M. N., Uscinski, J., & Scheutz, M. (2016). A generalization of Bayesian inference in the Dempster-Shafer belief theoretic framework. 2016 19th International Conference on Information Fusion (FUSION), 798–804. https://ieeexplore.ieee.org/document/7527968

6. Josang, A. (2016, July). Decision Making Under Vagueness and Uncertainty. 2016 19th International Conference on Information Fusion (FUSION). 2016 19th International Conference on Information Fusion (FUSION).

7. Jousselme, A.-L. (2016). Semantic criteria for the assessment of Uncertainty handling fusion models. 2016 19th International Conference on Information Fusion (FUSION), 488–495. https://ieeexplore.ieee.org/document/7527928

8. Krüger, M., & Ziegler, J. (2016). A similarity measure in Bayesian classification based on characteristic attributes of objects. 2016 19th International Conference on Information Fusion (FUSION), 480–487. https://ieeexplore.ieee.org/abstract/document/7527927

9. Pavlin, G., Claessens, R., de Oude, P., & Costa, P. C. G. (2016). Evaluation of a canonical model approach to probabilistic data association in tracking with particle filters. 2016 19th International Conference on Information Fusion (FUSION), 464–471. https://ieeexplore.ieee.org/document/7527925

10. Rogova, G., & Yager, R. (2016). Belief-based argumentation and golden rule for decision making with soft and hard information. 2016 19th International Conference on Information Fusion (FUSION), 790–797. https://ieeexplore.ieee.org/document/7527967

Fusion 2018 Special Session on
Evaluation of Technologies for Uncertainty Reasoning

Table of Contents:

• Organizers
• Description of the Special Session
• Research Topics that will Develop
• List of Papers/Presentations

 

Organizers

Paulo Costa                      George Mason University, Fairfax, VA, USA
Kathryn Laskey                 George Mason University, Fairfax, VA, USA
Anne-Laure Jousselme     NATO CMRE, La Spezia, Italy
Pieter DeVilliers                University of Pretoria, Pretoria, South Africa

 

Description of the Special Session

The ETUR Session is intended to report the latest results of the ISIF’s ETURWG, which aims to bring together advances and developments in the area of evaluation of uncertainty representation. The ETURWG special sessions started in Fusion 2010 and has been held ever since, with an attendance consistently averaging between 40 and 48 attendees. While most attendees consist of ETURWG participants new researchers and practitioners interested in uncertainty evaluation have attended the sessions and some joined the ETURWG.

Research Topics that will Develop

One of the main goals of information fusion is uncertainty reduction, which is dependent on the representation chosen. Uncertainty representation differs across the various levels of Information Fusion (as defined by the JDL/DFIG models). Given the advances in information fusion systems, there is a need to determine how to represent and evaluate situational (Level 2 Fusion), impact(Level 3 Fusion) and process refinement (Level 5 Fusion), which is not well standardized for the information fusion community.
The session will focus three topics: (1) to summarize the state of the art in uncertainty analysis, representation, and evaluation, (2) discussion of metrics for uncertainty representation assessment, and (3) survey uncertainty at all levels of fusion.
The impact to the ISIF community would be an organized session with a series of methods in uncertainty representation as coordinated with evaluation.
The techniques discussed and questions/answers would be important for the researchers in the ISIF community; however, the bigger impact would be for the customers of information fusion systems guidance on how to measure, evaluate, and improve systems that assess the situation beyond Level 1 fusion, as far as uncertainty representation and reasoning is concerned.
The customers of information fusion products would have some guidelines to draft requirements documentation, the state of the art in terms of uncertainty evaluation in fusion systems, and exposure to uncertainty reasoning practices that are key to information fusion system designs.

 

List of Papers / Presentations

Application of URREF Criteria to Assess Knowledge Representation in Cyber Threat Models
Valentina Dragos, Juergen Ziegler and Johan P de Villiers
Abstract:
Systems for threat analysis enable users to understand the nature and behavior of threats and to undertake a deeper analysis for detailed exploration of threat profile and risk estimation. Models for threat analysis require significant resources to be developed and are often relevant to limited application tasks. This paper investigated the implicit and explicit uncertainty assessments to be taken into account for threat analysis systems to be effective for providing a relevant threat characterization. The intent of this paper is twofold. The first is to present and discuss an approach to define a model for cyber threats within a simplified expert model and to translate it into a Bayesian network as a tool for the development of practical scenarios for cyber threats analysis. The second is to address the question of assessing the Bayesian network build and its intrinsic knowledge representation model and to Show how modeling decisions impact the outcome of the system. The paper describes the construction of an expert model and the corresponding BN to analyze cyber threats, investigates various types of induced uncertainty with the URREF criteria simplicity and expressiveness and implements an assessment procedure to evaluate the overall approach.

Experimental Comparison of Ad Hoc Methods for Classification of Maritime Vessels Based on Real-life AIS Data
Max Krueger
Abstract:
Classification of Maritime Vessels is a recurrent task in maritime surveillance systems. Classification can be conducted by different methods, e.g., Naïve Bayes, k Nearest Neighbor, Decision Tree, Fuzzy Rule, or Neural Networks. The Automatic Identification System (AIS) is cooperative system of VHF-radio data exchange. By broadcasting of navigational data and ships' information it supports maritime safety, surveillance, and information. Based on measured AIS datasets of five maritime hotspots, easily implementable standard classification (i.e. ad hoc) methods from Data Science are compared to each other. These are evaluated in terms of accuracy. This experimental evaluation is motivated by the following question: Up to which degree properties and behavior, e.g., vessel's type, can be detected by using large quantities of ship's positional, motion, and dimensions' data as provided by AIS? Future applications might include detection of fraudulent self-declarations of types, e.g., during illegal fishing activities.

Towards the Rational Development and Evaluation of Complex Fusion Systems: a URREF-Driven Approach
Gregor Pavlin, Anne-Laure Jousselme, Johan P de Villiers, Paulo C.G. Costa and Patrick de Oude
Abstract: 
The choices of the uncertainty representations and reasoning methods have a critical impact on the development and deployment life cycles of modern fusion solutions. They influence the development effort, the quality of the resulting solutions as well as the deployment costs. This paper shows how the URREF concepts enable a systematic \modifAL{and rational} development and evaluation of complex fusion systems. In particular, the paper proposes a URREF-driven approach to the development and deployment of composite fusion systems. In this approach the URREF criteria play a critical role throughout the development life cycle as they facilitate informed design choices and enable systematic and tractable evaluation of the resulting systems. The paper establishes relations between the URREF criteria and evaluation subjects in the context of a development life cycle. The concepts are illustrated with the help of a high-level fusion approach supporting estimation of the whereabouts of wildlife poachers.

Latent Variable Bayesian Networks Constructed Using Structural Equation Modelling
Alta de Waal and Keunyoung Yoo
Abstract: Bayesian networks in fusion systems often contain latent variables. They play an important role in fusion systems as they provide context which lead to better choices of data sources to fuse. Latent variables in Bayesian networks are mostly constructed by means of expert knowledge modelling. We propose using theory-driven structural equation modelling (SEM) to identify and structure latent variables in a Bayesian network. The linking of SEM and Bayesian networks is motivated by the fact that both methods are can be shown to be causal models. We compare this approach to a data-driven approach where latent factors are induced by means of unsupervised learning. We identify appropriate metrics for URREF ontology criteria for both approaches.

Information and Source Quality Ontology in Support to Maritime Situational Awareness
Elena Camossi and Anne-Laure Jousselme
Abstract: 
To support situation awareness, the benefit of a variety of sources makes no doubt although it brings additional challenges related to heterogeneity in data format, semantics, uncertainty type, for instance, but also challenges related to possible conflicting information. Information and source quality are intertwined concepts which assessments connect with the evaluation of uncertainty handling in information fusion solutions.While the Uncertainty Representation Reasoning Evaluation Framework (URREF) ontology focuses on assessment criteria, peripheral concepts still play a critical role in the assessment. In this paper, we propose an Information and Source Quality (ISQ) ontology formalising the relationships between information-related concepts, and discuss information interpretation in support of Maritime Situation Awareness.Specifically, this paper links the concepts of Information Source, Dataset and Piece of Information, and connects them to the corresponding quality concepts. Such concepts link to the upper level concepts of the URREF ontology Source (of information) and data Quality. The ontology further expands to the uncertainty modelling and the algorithm design. We conclude on future work and identify avenues leveraging this work, especially the extension to the formalisation of the evaluation process.

 

Fusion 2019 Special Sessions on Evaluation of Technologies for Uncertainty Reasoning

Papers - Special Session 1

Assessment of Trust in Opportunistic Reporting Using Belief Functions
Valentina Dragos, Jean Dezert and Kellyn Rein

Houses Bombing in Ravixe: a Bench for High Level Fusion Evaluation
Nicolas Museux, Claire Laudy and Mihai C Florea

Uncertainty Ontology for Veracity and Relevance
Erik Blasch, Carlos C. Insaurralde, Paulo C.G. Costa, Alta de Waal and Johan P de Villiers

Online System Evaluation and Learning of Data Source Models: a Probabilistic Generative Approach
Gregor Pavlin, Anne-Laure Jousselme, Johan P de Villiers, Paulo C.G. Costa, Kathryn Laskey, Alta de Waal and Franck Mignet

Detection of AIS Spoofing in Fishery Scenarios
Max Krueger

Papers - Special Session 2

How to Evaluate High Level Fusion Algorithms?
Nicolas Museux and Claire Laudy

Simulating Null Games for Uncertainty Evaluation in Green Security Games
Lisa Kirkland, Alta de Waal and Johan P de Villiers

Uncertainty-based Decision Making Using Deep Reinforcement Learning
Xujiang Zhao, Shu Hu, Jin-Hee Cho and Feng Chen

Cooperative Semi-supervised Regression Algorithm Based on Belief Functions Theory
Hongshun He, Deqiang Han and Yi Yang

Entropy-Based Metrics for URREF Criteria to Assess Uncertainty in Bayesian Networks for Cyber Threat Detection
Valentina Dragos, Juergen Ziegler, Johan P de Villiers, Alta de Waal, Anne-Laure Jousselme and Erik Blasch