Evaluation Criteria

This is the Uncertainty Representation and Reasoning Evaluation Framework, which is also explained at the Fussion 2012 paper attached below.

File Attachments: 

I. URREF Scope

The evaluation of how uncertainty is dealt with within a given Information Fusion (IF) system is distinct from, although closely related to, the evaluation of the overall performance of the system. Metrics for evaluating the overall performance of IF systems are more encompassing in scope than those focused on the uncertainty handling within the system.   The metrics for the overall system include the effects of the uncertainty representation, but there are also effects of other aspects of the fusion system that can affect the performance of the system. 

For example, fusion-system-level metrics include time­liness (how quickly the system can come to a conclusion within a specified precision level), accuracy (where can an object be found for a specified localization level) and confidence (what level of a probability match for a defined recall level).Clearly, different choices in uncertainty representation approaches will affect the achievable timeliness, accuracy, and confidence of a system, and therefore must be considered when evaluating both the system’s performance as a whole and the specific impact of the uncertainty handling approach. Yet, when evaluating timeliness (or any other system-level metrics), one will likely find some factors not directly related to the handling of uncertainty itself[1], such as object tracking and identification report updates (i.e., Level 1 fusion), situation and threat assessment relative to scenario constraints (i.e., Level 2/3 fusion), overall system architecture (e.g. centralized, distributed, etc.), data management processes and feedback / input control processes (i.e., Level 4 fusion considerations), and user-machine coordination based on operating systems (i.e., Level 5 fusion), and others.

In an ideal situation, evaluating how the management of uncertainty affects the overall performance of a fusion system would be just a matter of isolating the directly related aspects. Unfortunately, real-life information fusion systems are usually too complex to allow for such a clear-cut separation, and most of the aspects considered in system-wide performance are entangled with or influenced by uncertainty representation considerations to some degree. Isolating the impact of uncertainty handling in an IF system is thus a matter of understanding how intertwined the choice of an uncertainty representation and reasoning approach is to the major performance metrics used to evaluate the IF system itself.

The basic premise of the ETURWG is that achieving a level of understanding that supports an unbiased evaluation of the impact of uncertainty in an IF system is a task by itself complex enough to warrant status as an open research area.  Understanding what the distinctions are between fusion system-level performance criteria and fusion uncertainty representation performance criteria is the focus of the group. This paper presents one of the initial results of the ETURWG, the URREF ontology, and addresses how it is being currently used to support the development of an unbiased uncertainty reasoning evaluation framework.

This page is an initial attempt at determining the distinctions between evaluating the performance of an IF system and evaluating the impact of uncertainty on it. This would then allow us to establish an evaluation framework capable of supporting unbiased assessment of how the choice of uncertainty representation and reasoning impacts the performance of an IF system.  The basic idea behind the framework is to analyze an abstract fusion system and define its input data and output products.  In a futuristic prototypical information fusion system, the uncertainty representation approach would be “plug-and-playable.”  That is, one can run it with a Bayesian approach, then switch out the Bayesian approach for a Dempster-Shafer approach, then for a Fuzzy Random Set approach or have a combination of uncertainty reasoning methods.  The input data are the same in each case, as are the output products (but not necessarily the values in the output products). Figure 1 below depicts the boundaries of the uncertainty representation and reasoning evaluation framework (URREF).

Figure 1 - Boundaries of the Uncertainty Representation and Reasoning Evaluation Framework

There are two elements in the picture that are exogenous to the evaluation framework, named in the picture as “World being sensed” and “World being reported.” Between these two external elements, the boundary of the evaluation framework encompasses the way uncertainty is handled when data is input to the system, during the processes that occur within it, as well as when the final product is delivered to the IF system’s users.

The first external element refers to the events of interest to the IF system that happen in the world and are perceived by the system sources. Note that the implicit definition of sources in this case encompasses anything that can capture information and send it to the system. That is, both hard sources (e.g. imaging, radar, video, etc.) and soft sources (HUMINT reports, software alerts, etc.) are considered external to the evaluation system with respect to their associated sensorial capabilities, while the way they convey their information is within the scope of the system [1, 2, 3].

This is an important distinction between the evaluation of an IF system, which usually encompasses the sensitivity of its sensors, and the evaluation of its handling of uncertainty, which focuses only on how the uncertainty embedded in the sensors’ information is captured. The latter comprises what is called in the picture as the Input step, which involves assessing the system’s ability to represent uncertainty as an intrinsic part of the information being captured.  As an example, information regarding trust on the input from a given sensor is an important item to evaluate how the overall system handles uncertainty, although it might not be as critical for its overall performance.  A key question for evaluating uncertainty representation is what the uncertainty characteristics of the input data are, and how they affect the use of different uncertainty schemes.

In the ideal system model, having the appropriate data characteristics is critical.  If the characteristics do not span the range of uncertainty techniques, then the model may not give meaningful results about the operationally significant differences between the techniques.  Correctly identifying the desired input data characteristics will shape the future development of use cases and modeling data sets for those use case.  

Once information is in the IF system, it will be processed to generate the system’s deliverable that requires uncertainty characterization and reporting in the Output step. This process involves fusion techniques and algorithms that are directly affected by the uncertainty handling technique being used, and its impact on the system’s inferential process. In this case, the URREF criteria focus on aspects that are specific to the way uncertainty is considered and handled within the fusion process.  This is not an evaluation of the system’s performance as a whole. We want to understand how the uncertainty representation affects system performance, and whether different uncertainty representation schemes are more or less robust to variations in the remaining parts of the IF system architecture. But we want to focus specifically on the uncertainty representation aspects, and attempt, as best as possible, to separate those aspects from overall system performance and architecture issues.

After the information is fused and properly treated, then it is conveyed to the system’s users.  In the figure, these are represented by an image depicting decision-makers who would likely be supported by the IF system in their daily tasks.  The URREF output step involves the assessment of how information on uncertainty is presented to the users and, therefore, how it impacts the quality of their decision-making process.


  1. P.C.G. Costa, KC Chang, K.B. Laskey, T. Levitt, and W. Sun, “High-Level Fusion: Issues in Developing a Formal Theory”, Int. Conf. on Information Fusion, 2010.
  2. P.C.G. Costa, R.N. Carvalho, K.B. Laskey, and C.Y.Park, “Evaluating Uncertainty Representation and Reasoning in HLF systems”, Int. Conference on Information Fusion,  2011.
  3. A-L Jousselme, P. Maupin, E. Bossé, “Quantitative Approaches” Ch. 8 in Concepts, Models and Tools for Information Fusion,  Artech House, 2006.

1 This holds at least at a higher level of abstraction.  There might be interactions between the uncertainty representation approach and these system factors, but we begin with a presumption that they are not significant.


II. URREF Ontology

URREF ontology

The OWL file of the current version of the URREF ontology is always posted at http://eturwg.c4i.gmu.edu/files/ontologies/URREF.owl. This page includes links to the associated documentation of each version of the ontology.

Currently, we are working on version 3 using github. The pages listed below provide an overview of each version.

Version 1

  • This is the current version of the URREF Ontology. We are currently working on version 2 and when it becomes ready this page will be listed as "deprecated" and all links to the URREF ontology will point to the new version.

URREF ontology

THE URREF ontology (http://eturwg.c4i.gmu.edu/files/ontologies/URREF.owl) depicted in Figure 2 below is intended to provide a high-level guidance for defining the actual criteria items that will comprise a comprehensive uncertainty evaluation framework.

Figure 2 – URREF Ontology: Main Classes

Only the main classes are depicted in the figure, which also shows the classes that expand to subclasses with a small triangle at the right side of the oval. All classes and subclasses are explained below.


Information in its most restricted technical sense is an ordered sequence of symbols that can be interpreted as a message. Information can be recorded as signs, or transmitted as signals. Information is any kind of event that affects the state of a dynamic system. Conceptually, information is the message (utterance or expression) being conveyed.


A source is the origin of the information.


An expression in some logical language that evaluates to a truth-value (formula, axiom, assertion). It is then assumed that information will be presented in the form of sentences. So the uncertainty will be associated with sentences.


This captures the information about the nature of the uncertainty, i.e., whether the uncertainty is inherent in the phenomenon expressed by the sentence or it is the result of lack of knowledge of the agent. Figure 3 depicts the Uncertainty Nature class and its subclasses.

Figure 3 – URREF Ontology: Uncertainty Nature Class and Subclasses

II.4.1 Epistemic

Uncertainty Nature is considered epistemic when it is caused by lack of complete knowledge. That is, the event itself might be completely deterministic, but there is uncertainty about it due to missing information.

II.4.2 Aleatory

Uncertainty Nature is considered aleatory when it comes from the world; that is, uncertainty is an inherent property of the world. In contrast with Epistemic Uncertainty, which is due to the lack of complete knowledge.


Uncertainty derivation refers to the way it can be assessed. That is, how the uncertainty metrics can be derived. Figure 4 depicts the Uncertainty Derivation class and its subclasses.

Figure 4 – URREF Ontology: Uncertainty Derivation Class and Subclasses

II.5.1 Objective

Uncertainty derivation is considered as objective when it can be assessed in a formal way, e.g., via a repeatable derivation process..

II.5.2 Subjective

Uncertainty Derivation is considered subjective when it is assessed via a subjective judgment, e.g., a subject matter expert's estimation, a gambler's guess, etc. Note that even though one might use formal methods for this assessment, it is the assessment itself that defines the Uncertainty Derivation as subjective. For example, a meteorologist may follow a rather sophisticated procedure to establish a weather forecast, but if the numbers ultimately are derived from his judgment then it is a subjective uncertainty derivation.


Uncertainty Type is a concept that focuses on underlying characteristics of the information that make it uncertain. Figure 5 depicts the Uncertainty Type class and its subclasses.

Figure 5 – URREF Ontology: Uncertainty Type Class and Subclasses

II.6.1 Ambiguity

An ambiguous statement allows for more than one interpretation. It is distinct from vagueness, which is a statement about the lack of precision contained or available in the information. As an example from Wikipedia, the meaning of blue in the song title "Don’t It Make My Brown Eyes Blue" may be interpreted as a color or as a state of mind. In either case, there is no vagueness about the meanings themselves, but the sentence accepts both.

II.6.2 Incompleteness

Incompleteness uncertainty type occurs when information is missing.

II.6.3 Vagueness

This determines how vague a piece of information is in its description/measurement/observation. This is related to the actual information itself, and not the originating source.


The man is 2 meters tall – less vague

The man is about 2 meters tall – more vague.

II.6.1 Randomness

The information describes a process whose outcomes do not follow a deterministic pattern.

II.6.1 Inconsistency

Inconsistency is a measure of the extent to which information is explicitly contradictory or conflicting.


The Uncertainty Model class contains information on the mathematical theories for the representing and reasoning with the uncertainty types. The specific types of theories include, but are not limited to, the subclasses listed in Figure 6.

Figure 6 – URREF Ontology: Uncertainty Model Class and Subclasses

II.7.1 Belief Functions

Belief functions are closely related to probabilities. Beliefs in a hypothesis is calculated as the sum of the masses of all sets it encloses. A belief function differs from a Bayesian probability model in that one does not condition on those parts of the evidence for which no probabilities are specified. This ability to explicitly model the degree of ignorance makes the theory very appealing and has been applied in areas such as inconsistency handling in OWL ontologies (Nikolov et al., 2007) and ontology mapping (e.g. Yaghlane and Laamari, 2007).

II.7.2 Fuzzy Sets

In contrast to probabilistic formalisms, which allow for representing and processing degrees of uncertainty about ambiguous pieces of information, fuzzy formalisms allow for representing and processing degrees of truth about vague (or imprecise) pieces of information. It is important to point out that vague statements are truth-functional, that is, the degree of truth of a vague complex statement (which is constructed from elementary vague statements via logical operators) can be calculated from the degrees of truth of its constituents, while uncertain complex statements are generally not a function of the degrees of uncertainty of their constituents (Dubois and Prade, 1994).

II.7.3 Probability

Probability theory provides a mathematically sound representation language and formal calculus for rational degrees of belief, which gives different agents the freedom to have different beliefs about a given hypothesis.

II.7.4 Random Sets


II.7.5 Rough Sets

A rough set, first described by a Polish computer scientist Zdzisław I. Pawlak, is a formal approximation of a crisp set (i.e., conventional set) in terms of a pair of sets which give the lower and the upper approximation of the original set. In the standard version of rough set theory (Pawlak 1982, 1991), the lower- and upper-approximation sets are crisp sets, but in other variations, the approximating sets may be fuzzy set.


This is the main class of the URREF ontology, and it is meant to emcompass all the different aspects that must be considered when evaluating uncertainty handling in multi-sensor fusion systems. Figure 7 depicts the Criteria class and its subclasses

Image of the ETURWG URREF Criteria Class
Figure 3 – URREF Ontology: Criteria Class and Subclasses

II.8.1 Input Criteria

This general concept encompasses the criteria that directly affect the way evidence is input to the system. It mostly focuses on the source of input data or evidence, which can be tangible (sensing or physical), testimonial (human), documentary, or known missing (Schum, 1994).

II.8.1.1 Relevance to Problem

Relevance to Problem assess how a given uncertainty representation is able to capture how a given input is relevant to the problem that was the source of the data request. This is a criterion specific to high-level fusion systems that work at levels 3 and above of the JDL model.

II.8.1.2 Weight or Force of Evidence

Weight or Force of Evidence assess how a given uncertainty representation is able to capture by how much a given input can affect the processing and output of the fusion system. Ideally, this should be an objective assessment and the representation approach must provide a means to measure the degree of impact of an evidence item with a numerical scale. This criterion is especially useful for determining the value of information in systems that must trade-off their ability to capture more evidence with active sensors with the need to avoid being observed. That is, this criterion is especially important to systems that rely on value of information.

II.8.1.3 Credibility

Also known as believability, it mainly comprises the aspects that directly affect a sensor (soft or hard) in its ability to capture evidence.

II. Veracity

This is a measure of the sensor’s ability to provide a “truthful” report. That is, a measure of whether the sensor reports what it believes is true. The concept originated with human testimony (deliberate intention to deceive), but can be applied to sensor errors (such as sensor faults) that cause the sensor data to deviate from what would have been reported had the error not existed.
- A justice of the peace states the person was aged 35 – higher Veracity
- A known criminal and liar states the person was aged 35 – lower Veracity
Rationale: the sensor “justice of the peace” is more likely to produce a veritable information than the sensor “known criminal and liar”

II. Objectivity

This is a measure of bias, which applies to all types of sensors.

II. Observational sensitivity

This is a measure of whether the sensor can sense what it claims to have sensed, also precision of measurement.
Example 1:
- The sober man said the car was yellow – higher Observational Sensitivity
- The drunk man said the car was yellow – lower Observational Sensitivity
Example 2:
- The ANPR camera stated the number plate was YR59 WXT – higher Observational Sensitivity
- The witness said the number plate was YR59 WXT – lower Observational Sensitivity

II. Self-Confidence

This is a measure of the information credibility as evaluated by the sensor itself. This is particularly relevant for soft sensors (HUMINT data) as often such sources provide appreciations of the information conveyed (such as it is possible, it is true, etc).
The idea behind this measure is that than HUMINT data can potentially convey two “types” of information: the information itself (tomorrow the sun will shine) but also some qualification of this information (it is *possible* because it is summer time and most of the time we have shinny weather).
The purpose of this measure is to take advantage of this particularity, and to have a first evaluation of the uncertainty as expressed by the author himself.
It is obvious that we are dealing with the credibility of the information: the author is providing us some information, but it is also telling us how much he believes it himself.
We are not dealing (at least not directly) with information veracity: even if the author considers the information as possible, and he trusts it, it could be false at the end (even in summer time we can have a cloudy day).
The self-confidence is telling us how much the author trust the information, but not necessarily that this information is false or true.

II.8.2 Representation Criteria

This general concept encompasses the criteria that directly affect the way information is captured by and transmitted through the system. It can also be called as interfacing or transport criteria, as it deals with how the representational model deals with transmitting information within the system.

II.8.2.1 Evidence Handling

These criteria apply particularly to the ability of a given representation of uncertainty to capture specific characteristics of incomplete evidence that are available to or produced by the system.  The main focus is on measuring the quality of the evidence by assessing how well this evidence is able to support the development of a conclusion.

II. Conclusiveness

This is a measure of how well the available evidence will support a definitive conclusion (strongly select a hypothesis).

II. Ambiguity

This is a measure of the extent to which the set of data can be interpreted to support different conclusions.

II. Completeness

This is a measure of the range of the available evidence, in terms of how much is available and how much is unknown.  It assesses the variety and eliminative characteristics of the data.

II. Reliability

This is a measure of the overall truthfulness (accordance with reality) of the evidence.

II. Dissonance

This is a measure of the extent to which the evidence is explicitly contradictory or inconsistent. That is, if a conflict exists in the information supporting a hypothesis (e.g. two descriptions of the same event or entity are not compatible).

II.8.2.2 Knowledge Handling

These criteria is intended to measure the ability of a given uncertainty representation technique to convey knowledge.

II. Expressiveness

This is a measure of the representational power of a given technique.

II. Assessment

It should be practicable for a user of the system to make (and feel comfortable with) all the uncertainty assessments that are needed as input. The system should give some guidance on how to make the assessments. It should be able to handle judgments of various types, including expressions of uncertainty in natural language such as “if A then probably B”, and to combine qualitative judgments with quantitative assessments of uncertainty (adapted from Welley 1996).
Typical assessment questions include: a) which input can complete a result (help for planning of recce assets)? b) What is the state of the data? Is there enough data? c) What is the reliability of the most probable results? And others.

II. Adaptability

Adaptability criteria encompass the ability of the representational model to allow for different configurations of the model. As an example, an adaptable representational framework would have most of its elements configurable by Subject matter Experts (SME).
Typical configuration elements might include: a) changes in basic facts (knowledge); b) adding new rules and classes to the model; c) adding and modeling new input sources; and d) configuration of the possible output of the model.

II. Simplicity

Simplicity criteria are meant to access the level of complexity involved in dealing with the representational framework. In general, a representational model that allows users to execute common operations (e.g. configure the system, enter evidence, proceed with analysis, etc.) without requiring deep knowledge about the inner details of the technique (e.g. the mathematical underpinnings of the inferential process) should meet the simplicity criteria.   

II. Compatibility

This is a measure of how compatible a given knowledge representation is to data standards, and should be related to the degree of flexibility it has in being coded with various standards.

II.8.3. Reasoning Criteria

This general concept encompasses the criteria that directly affect the way the system transforms its data into knowledge. It can also be called as process or inference criteria, as it deals with how the uncertainty model performs operations with information.

II.8.3.1 Correctness


There can be correctness of results and correctness of a reasoning model. In general the correctness of results produced by a reasoning model is the degree to which the results represent the true state of the situation modeled. For the purpose of this definition it is useful to distinguish between three types of truth: 1) ground truth, 2) consensus truth and 3) subjective truth, as described below, where ground truth is the strongest and subjective truth is the weakest form of truth.

  1. Ground truth about a situation are the statements that describe the objectively observable state of the situation.

  2. Consensus truth about a situation are the statements that reflect the commonly shared opinion about the state of the situation, or that that describes the state of the situation in line with commonly accepted norms or standards.

  3. Subjective truth about a situation are the statements that reflect the analyst’s own opinion about the state of the situation.

Three different types of result correctness emerge from the three different types of truth, where objective correctness is the strongest, and subjective correctness is the weakest.

  1. Objective results correctness is the degree to which the results represent ground truths of a situation.

  2. Consensus results correctness is the degree to which the results represent consensus truths of a situation.

  3. Subjective results correctness is the degree to which the results represent subjective truths of a situation.

Depending on whether ground truth, consensus truth or subjective truth is available, the strongest form of correctness should be required for practical analysis.


II.8.3.2 Consistency


With regard to the definition of results correctness it is necessary to examine the case when it has only been observed once or a small number of times that results from a model represent the true state of a situation. Although a model produces correct results in some instances there might be other instances where the results are clearly wrong, in which case the model can not be considered to be correct in general. For example, an analyst could misinterpret what it means to add apples from two baskets, and erroneously think that it should be modeled with the product rule. Assume that the analyst tries a specific example with two apples in each basket, and computes the sum with the product rule, which results in 4 apples. When observing a real example of two baskets of two apples each it turns out that that adding them together also produces 4 apples. This result could mistakenly be interpreted so that the information produced by the model is correct simply because it reflects objective observation in this particular instance. It is questionable whether a model for analysing a situation can be characterised as a correct model just because it produces results that by coincidence correspond to the truth of the situation. In order for a model to be correct tt is natural to require that results produced by it are generally correct and not just by coincidence in specific instances of a situation. In order to distinguish between coincidentally correct results and generally correct results, it is necessary to require that results are consistently correct, in which case the model that produces them can be described as correct.

A model is correct for a specific situation when it consistently produces correct results in all instances of the situation. On a high level of abstraction a correct reasoning model according to the definition above must faithfully reflect the (class of) situations that are being modeled. A precise way of expressing the same principle is that the model is correct for a class of situations. On this basis it is possible to articulate three types of model correctness, to reflect the three types of result correctness.

On this basis it is possible to articulate the definition of consistency as follows. There are three types of consistency, to reflect the three types of correctness.

  1. Objective model correctness for a specific class of situations is the model’s ability to consistently produce objectively correct results for all possible situations in the class.

  1. Consensus model correctness for a specific class of situations is the model’s ability to consistently produces consensus correct results for all possible situations in the class.

  1. Subjective model correctness for a specific class of situations is the model’s ability to consistently produces subjectively correct results for all possible situations in the class.

Consistent results are results that have been produced by a correct model. Depending on whether ground truth or consensus truth is available, the strongest form of model correctness should be required for practical analysis.

II.8.3.3 Scalability

This is a measure of a representational technique’s ability to be used in different magnitudes of data within the same problem. It might be broken down into sub-criteria such as modularity.

II.8.3.4 Computational Cost

This is a measure of how much of the system’s computational resources are required by a given representational technique to produce its results.

II.8.3.5 Performance

These include metrics to assess how suitable the representational model is to handle the functional requirements of an information fusion system. Other system architecture factors also affect these metrics.

II. Timeliness (from data input to product output)

This measures how long a given uncertainty representation technique takes to produce its results since data is input. Taken from another perspective, it measures whether the representation technique is capable of producing results within the timeframe required by the system’s performance goals.

II. Throughput (average / peak rates through the system)

This is a measure of the average and (possibly) peak rate through the system.   This differs from timeliness in that a system can have a long timeliness, but still produce a large number of answers in a given amount of time.

II.8.4 Output Criteria

These criteria are usually related to the system’s results and its ability to communicate it to its users in a clear fashion.

II.8.4.1 Quality

This is a group of criteria meant to assess the informational quality of the system’s output. It is common to see in the literature the same concepts with different names. For example, Accuracy sometimes is used as a synonym of precision; sometimes they are used with the exact opposite of their use below.

II. Accuracy

Criteria on accuracy are meant to assess the output of the system in terms of “how right” the answers are. Usual metrics include rate of correct identification/hit, false alarm rate, etc.

II. Precision

Criteria on precision are meant to assess the output of the system in terms of “how good” the answers are. It is a measure of the granularity of the system’s output.

Accuracy and precision can be inversely related.  As one makes the granularity coarser, one can expect that the system will have a better accuracy. Precision can also be used to put a boundary on the certainty of the reported result.

II.8.4.2 Interpretation

The output of the system in terms of uncertainty representation and reasoning should have a clear meaning that is sufficiently definite to be used to guide assessment, to understand the conclusions of the system and use them as a basis for action, and to support the rules for combining and updating measures (adapted from Walley 1996).

II.8.4.3 Traceability

Traceability criteria focus on establishing a correlation between the outcome of the reasoning process with the various input and events computed by the system, so for example one can easily explain why and how the system arrived to a specific answer.

Link to the URREF ontology



Blasch, Eric, Pierre Valin and Eloi Bosse, “Measures of Effectiveness for High-Level Fusion”, 13th Conference on Information Fusion (FUSION), 2010.

Costa, Paulo C. G.; Carvalho, Rommel N.; Laskey, Kathryn B.; and Park, Cheol Y. (2011) Evaluating Uncertainty Representation and Reasoning in HLF systems. In Proceedings of the Fourteenth International Conference on Information Fusion (Fusion 2011). July 5-8, 2011, Chicago, IL, USA.

Costa, Paulo C. G.; Chang, KuoChu; Laskey, Kathryn B.; Levitt, Tod; and Sun, Wei (2010) High-Level Fusion: Issues in Developing a Formal Theory. In Proceedings of the Thirteenth International Conference of the Society of Information Fusion (FUSION 2010). July 26-29, 2010, Edinburgh, Scotland, UK.

Dubois, D.; and Prade, H. (1994) Can We Enforce Full Compositionality in Uncertainty Calculi. Proceedings AAAI-1994, pp. 149-154. AAAI Press.

Jousselme, A.-L., Maupin, P., Bossé, É., Quantitative approaches, in Concepts, Models and Tools for Information Fusion,  Artech House, chap. 8, 2006, pp. 169-210.

Nikolov, A.; Uren, V.; Motta, E.; and de Roeck, A. (2007) Using the Dempster-Shafer Theory of Evidence to Resolve ABox Inconsistencies. In Proceedings of the 3rd Workshop on Uncertainty Represention for the Semantic Web (URSW 2007). November 12, 2007. Busan, Korea.

Pawlak, Z., 1982. Rough sets. International Journal of Computer & Information Sciences, 11(5), pp.341-356.

Pawlak, Z., 1991. Rough sets: theoretical aspects of reasoning about data, Springer.

Schum, David, The Evidence Foundations Of Probabilistic Reasoning, Northwestern University Press, 1994.

Walley, P., Measures of uncertainty in expert systems, Artificial Intelligence, 83(1), May 1996, pp. 1-58

Yaghlane, B. B.; and Laamari, N. (2007) OWL-CM : OWL Combining Matcher Based on Belief Functions Theory. In Proceedings of the 2nd International Workshop on Ontology Matching (OM-2007). November 11, 2007. Busan, Korea.

Version 2

The Version 2 of the URREF ontology is an ongoing work. When we "freeze" its configuration the final result will be posted here. In the meantime, we should include the updates as comments below, so in the end we will have this page with the final version and some sort of historical account listed in the comment area below.

Version 3

Version 3 of the URREF ontology is an ongoing work and its structure has been "frozen" until Fusion 2019. Until then, we will be making modifications only to the properties as a result of the group's discussions. The URREF ontology code for any version can be obtained from our GitHub page at:


The official distribution of the ontology (i.e. its OWL file) can be accessed via the URL:



Examples and Tutorial

This page is meant to support the discussions on the examples of how IF systems can be evaluated using URREF. Each example will have a separate page for it:

- Rhino Poaching