• This is the current version of the URREF Ontology. We are currently working on version 2 and when it becomes ready this page will be listed as "deprecated" and all links to the URREF ontology will point to the new version.

URREF ontology

THE URREF ontology (http://eturwg.c4i.gmu.edu/files/ontologies/URREF.owl) depicted in Figure 2 below is intended to provide a high-level guidance for defining the actual criteria items that will comprise a comprehensive uncertainty evaluation framework.

Figure 2 – URREF Ontology: Main Classes

Only the main classes are depicted in the figure, which also shows the classes that expand to subclasses with a small triangle at the right side of the oval. All classes and subclasses are explained below.

II.1 INFORMATION

Information in its most restricted technical sense is an ordered sequence of symbols that can be interpreted as a message. Information can be recorded as signs, or transmitted as signals. Information is any kind of event that affects the state of a dynamic system. Conceptually, information is the message (utterance or expression) being conveyed.

II.2 SOURCE

A source is the origin of the information.

II.3 SENTENCE

An expression in some logical language that evaluates to a truth-value (formula, axiom, assertion). It is then assumed that information will be presented in the form of sentences. So the uncertainty will be associated with sentences.

II.4 UNCERTAINTY NATURE

This captures the information about the nature of the uncertainty, i.e., whether the uncertainty is inherent in the phenomenon expressed by the sentence or it is the result of lack of knowledge of the agent. Figure 3 depicts the Uncertainty Nature class and its subclasses.

Figure 3 – URREF Ontology: Uncertainty Nature Class and Subclasses

II.4.1 Epistemic

Uncertainty Nature is considered epistemic when it is caused by lack of complete knowledge. That is, the event itself might be completely deterministic, but there is uncertainty about it due to missing information.

II.4.2 Aleatory

Uncertainty Nature is considered aleatory when it comes from the world; that is, uncertainty is an inherent property of the world. In contrast with Epistemic Uncertainty, which is due to the lack of complete knowledge.

II.5 UNCERTAINTY DERIVATION

Uncertainty derivation refers to the way it can be assessed. That is, how the uncertainty metrics can be derived. Figure 4 depicts the Uncertainty Derivation class and its subclasses.

Figure 4 – URREF Ontology: Uncertainty Derivation Class and Subclasses

II.5.1 Objective

Uncertainty derivation is considered as objective when it can be assessed in a formal way, e.g., via a repeatable derivation process..

II.5.2 Subjective

Uncertainty Derivation is considered subjective when it is assessed via a subjective judgment, e.g., a subject matter expert's estimation, a gambler's guess, etc. Note that even though one might use formal methods for this assessment, it is the assessment itself that defines the Uncertainty Derivation as subjective. For example, a meteorologist may follow a rather sophisticated procedure to establish a weather forecast, but if the numbers ultimately are derived from his judgment then it is a subjective uncertainty derivation.

II.6 UNCERTAINTY TYPE

Uncertainty Type is a concept that focuses on underlying characteristics of the information that make it uncertain. Figure 5 depicts the Uncertainty Type class and its subclasses.

Figure 5 – URREF Ontology: Uncertainty Type Class and Subclasses

II.6.1 Ambiguity

An ambiguous statement allows for more than one interpretation. It is distinct from vagueness, which is a statement about the lack of precision contained or available in the information. As an example from Wikipedia, the meaning of blue in the song title "Don’t It Make My Brown Eyes Blue" may be interpreted as a color or as a state of mind. In either case, there is no vagueness about the meanings themselves, but the sentence accepts both.

II.6.2 Incompleteness

Incompleteness uncertainty type occurs when information is missing.

II.6.3 Vagueness

This determines how vague a piece of information is in its description/measurement/observation. This is related to the actual information itself, and not the originating source.

Examples:

The man is 2 meters tall – less vague

The man is about 2 meters tall – more vague.

II.6.1 Randomness

The information describes a process whose outcomes do not follow a deterministic pattern.

II.6.1 Inconsistency

Inconsistency is a measure of the extent to which information is explicitly contradictory or conflicting.

II.7 UNCERTAINTY MODEL

The Uncertainty Model class contains information on the mathematical theories for the representing and reasoning with the uncertainty types. The specific types of theories include, but are not limited to, the subclasses listed in Figure 6.

Figure 6 – URREF Ontology: Uncertainty Model Class and Subclasses

II.7.1 Belief Functions

Belief functions are closely related to probabilities. Beliefs in a hypothesis is calculated as the sum of the masses of all sets it encloses. A belief function differs from a Bayesian probability model in that one does not condition on those parts of the evidence for which no probabilities are specified. This ability to explicitly model the degree of ignorance makes the theory very appealing and has been applied in areas such as inconsistency handling in OWL ontologies (Nikolov et al., 2007) and ontology mapping (e.g. Yaghlane and Laamari, 2007).

II.7.2 Fuzzy Sets

In contrast to probabilistic formalisms, which allow for representing and processing degrees of uncertainty about ambiguous pieces of information, fuzzy formalisms allow for representing and processing degrees of truth about vague (or imprecise) pieces of information. It is important to point out that vague statements are truth-functional, that is, the degree of truth of a vague complex statement (which is constructed from elementary vague statements via logical operators) can be calculated from the degrees of truth of its constituents, while uncertain complex statements are generally not a function of the degrees of uncertainty of their constituents (Dubois and Prade, 1994).

II.7.3 Probability

Probability theory provides a mathematically sound representation language and formal calculus for rational degrees of belief, which gives different agents the freedom to have different beliefs about a given hypothesis.

II.7.4 Random Sets

II.7.5 Rough Sets

A rough set, first described by a Polish computer scientist Zdzisław I. Pawlak, is a formal approximation of a crisp set (i.e., conventional set) in terms of a pair of sets which give the lower and the upper approximation of the original set. In the standard version of rough set theory (Pawlak 1982, 1991), the lower- and upper-approximation sets are crisp sets, but in other variations, the approximating sets may be fuzzy set.

II.8 CRITERIA

This is the main class of the URREF ontology, and it is meant to emcompass all the different aspects that must be considered when evaluating uncertainty handling in multi-sensor fusion systems. Figure 7 depicts the Criteria class and its subclasses

Figure 3 – URREF Ontology: Criteria Class and Subclasses

II.8.1 Input Criteria

This general concept encompasses the criteria that directly affect the way evidence is input to the system. It mostly focuses on the source of input data or evidence, which can be tangible (sensing or physical), testimonial (human), documentary, or known missing (Schum, 1994).

II.8.1.1 Relevance to Problem

Relevance to Problem assess how a given uncertainty representation is able to capture how a given input is relevant to the problem that was the source of the data request. This is a criterion specific to high-level fusion systems that work at levels 3 and above of the JDL model.

II.8.1.2 Weight or Force of Evidence

Weight or Force of Evidence assess how a given uncertainty representation is able to capture by how much a given input can affect the processing and output of the fusion system. Ideally, this should be an objective assessment and the representation approach must provide a means to measure the degree of impact of an evidence item with a numerical scale. This criterion is especially useful for determining the value of information in systems that must trade-off their ability to capture more evidence with active sensors with the need to avoid being observed. That is, this criterion is especially important to systems that rely on value of information.

II.8.1.3 Credibility

Also known as believability, it mainly comprises the aspects that directly affect a sensor (soft or hard) in its ability to capture evidence.

II.8.1.3.1 Veracity

This is a measure of the sensor’s ability to provide a “truthful” report. That is, a measure of whether the sensor reports what it believes is true. The concept originated with human testimony (deliberate intention to deceive), but can be applied to sensor errors (such as sensor faults) that cause the sensor data to deviate from what would have been reported had the error not existed.
Example:
- A justice of the peace states the person was aged 35 – higher Veracity
- A known criminal and liar states the person was aged 35 – lower Veracity
Rationale: the sensor “justice of the peace” is more likely to produce a veritable information than the sensor “known criminal and liar”

II.8.1.3.2 Objectivity

This is a measure of bias, which applies to all types of sensors.

II.8.1.3.3 Observational sensitivity

This is a measure of whether the sensor can sense what it claims to have sensed, also precision of measurement.
Example 1:
- The sober man said the car was yellow – higher Observational Sensitivity
- The drunk man said the car was yellow – lower Observational Sensitivity
Example 2:
- The ANPR camera stated the number plate was YR59 WXT – higher Observational Sensitivity
- The witness said the number plate was YR59 WXT – lower Observational Sensitivity

II.8.1.3.4 Self-Confidence

This is a measure of the information credibility as evaluated by the sensor itself. This is particularly relevant for soft sensors (HUMINT data) as often such sources provide appreciations of the information conveyed (such as it is possible, it is true, etc).
The idea behind this measure is that than HUMINT data can potentially convey two “types” of information: the information itself (tomorrow the sun will shine) but also some qualification of this information (it is *possible* because it is summer time and most of the time we have shinny weather).
The purpose of this measure is to take advantage of this particularity, and to have a first evaluation of the uncertainty as expressed by the author himself.
It is obvious that we are dealing with the credibility of the information: the author is providing us some information, but it is also telling us how much he believes it himself.
We are not dealing (at least not directly) with information veracity: even if the author considers the information as possible, and he trusts it, it could be false at the end (even in summer time we can have a cloudy day).
The self-confidence is telling us how much the author trust the information, but not necessarily that this information is false or true.

II.8.2 Representation Criteria

This general concept encompasses the criteria that directly affect the way information is captured by and transmitted through the system. It can also be called as interfacing or transport criteria, as it deals with how the representational model deals with transmitting information within the system.

II.8.2.1 Evidence Handling

These criteria apply particularly to the ability of a given representation of uncertainty to capture specific characteristics of incomplete evidence that are available to or produced by the system.  The main focus is on measuring the quality of the evidence by assessing how well this evidence is able to support the development of a conclusion.

II.8.2.1.1 Conclusiveness

This is a measure of how well the available evidence will support a definitive conclusion (strongly select a hypothesis).

II.8.2.1.2 Ambiguity

This is a measure of the extent to which the set of data can be interpreted to support different conclusions.

II.8.2.1.3 Completeness

This is a measure of the range of the available evidence, in terms of how much is available and how much is unknown.  It assesses the variety and eliminative characteristics of the data.

II.8.2.1.4 Reliability

This is a measure of the overall truthfulness (accordance with reality) of the evidence.

II.8.2.1.5 Dissonance

This is a measure of the extent to which the evidence is explicitly contradictory or inconsistent. That is, if a conflict exists in the information supporting a hypothesis (e.g. two descriptions of the same event or entity are not compatible).

II.8.2.2 Knowledge Handling

These criteria is intended to measure the ability of a given uncertainty representation technique to convey knowledge.

II.8.2.2.1 Expressiveness

This is a measure of the representational power of a given technique.

II.8.2.2.1.1 Assessment

It should be practicable for a user of the system to make (and feel comfortable with) all the uncertainty assessments that are needed as input. The system should give some guidance on how to make the assessments. It should be able to handle judgments of various types, including expressions of uncertainty in natural language such as “if A then probably B”, and to combine qualitative judgments with quantitative assessments of uncertainty (adapted from Welley 1996).
Typical assessment questions include: a) which input can complete a result (help for planning of recce assets)? b) What is the state of the data? Is there enough data? c) What is the reliability of the most probable results? And others.

II. 8.2.2.1.2 Adaptability

Adaptability criteria encompass the ability of the representational model to allow for different configurations of the model. As an example, an adaptable representational framework would have most of its elements configurable by Subject matter Experts (SME).
Typical configuration elements might include: a) changes in basic facts (knowledge); b) adding new rules and classes to the model; c) adding and modeling new input sources; and d) configuration of the possible output of the model.

II. 8.2.2.1.3 Simplicity

Simplicity criteria are meant to access the level of complexity involved in dealing with the representational framework. In general, a representational model that allows users to execute common operations (e.g. configure the system, enter evidence, proceed with analysis, etc.) without requiring deep knowledge about the inner details of the technique (e.g. the mathematical underpinnings of the inferential process) should meet the simplicity criteria.   

II.8.2.2.2 Compatibility

This is a measure of how compatible a given knowledge representation is to data standards, and should be related to the degree of flexibility it has in being coded with various standards.

II.8.3. Reasoning Criteria

This general concept encompasses the criteria that directly affect the way the system transforms its data into knowledge. It can also be called as process or inference criteria, as it deals with how the uncertainty model performs operations with information.

II.8.3.1 Correctness

There can be correctness of results and correctness of a reasoning model. In general the correctness of results produced by a reasoning model is the degree to which the results represent the true state of the situation modeled. For the purpose of this definition it is useful to distinguish between three types of truth: 1) ground truth, 2) consensus truth and 3) subjective truth, as described below, where ground truth is the strongest and subjective truth is the weakest form of truth.

1. Ground truth about a situation are the statements that describe the objectively observable state of the situation.

2. Consensus truth about a situation are the statements that reflect the commonly shared opinion about the state of the situation, or that that describes the state of the situation in line with commonly accepted norms or standards.

3. Subjective truth about a situation are the statements that reflect the analyst’s own opinion about the state of the situation.

Three different types of result correctness emerge from the three different types of truth, where objective correctness is the strongest, and subjective correctness is the weakest.

1. Objective results correctness is the degree to which the results represent ground truths of a situation.

2. Consensus results correctness is the degree to which the results represent consensus truths of a situation.

3. Subjective results correctness is the degree to which the results represent subjective truths of a situation.

Depending on whether ground truth, consensus truth or subjective truth is available, the strongest form of correctness should be required for practical analysis.

II.8.3.2 Consistency

With regard to the definition of results correctness it is necessary to examine the case when it has only been observed once or a small number of times that results from a model represent the true state of a situation. Although a model produces correct results in some instances there might be other instances where the results are clearly wrong, in which case the model can not be considered to be correct in general. For example, an analyst could misinterpret what it means to add apples from two baskets, and erroneously think that it should be modeled with the product rule. Assume that the analyst tries a specific example with two apples in each basket, and computes the sum with the product rule, which results in 4 apples. When observing a real example of two baskets of two apples each it turns out that that adding them together also produces 4 apples. This result could mistakenly be interpreted so that the information produced by the model is correct simply because it reflects objective observation in this particular instance. It is questionable whether a model for analysing a situation can be characterised as a correct model just because it produces results that by coincidence correspond to the truth of the situation. In order for a model to be correct tt is natural to require that results produced by it are generally correct and not just by coincidence in specific instances of a situation. In order to distinguish between coincidentally correct results and generally correct results, it is necessary to require that results are consistently correct, in which case the model that produces them can be described as correct.

A model is correct for a specific situation when it consistently produces correct results in all instances of the situation. On a high level of abstraction a correct reasoning model according to the definition above must faithfully reflect the (class of) situations that are being modeled. A precise way of expressing the same principle is that the model is correct for a class of situations. On this basis it is possible to articulate three types of model correctness, to reflect the three types of result correctness.

On this basis it is possible to articulate the definition of consistency as follows. There are three types of consistency, to reflect the three types of correctness.

1. Objective model correctness for a specific class of situations is the model’s ability to consistently produce objectively correct results for all possible situations in the class.

2. Consensus model correctness for a specific class of situations is the model’s ability to consistently produces consensus correct results for all possible situations in the class.

3. Subjective model correctness for a specific class of situations is the model’s ability to consistently produces subjectively correct results for all possible situations in the class.

Consistent results are results that have been produced by a correct model. Depending on whether ground truth or consensus truth is available, the strongest form of model correctness should be required for practical analysis.

II.8.3.3 Scalability

This is a measure of a representational technique’s ability to be used in different magnitudes of data within the same problem. It might be broken down into sub-criteria such as modularity.

II.8.3.4 Computational Cost

This is a measure of how much of the system’s computational resources are required by a given representational technique to produce its results.

II.8.3.5 Performance

These include metrics to assess how suitable the representational model is to handle the functional requirements of an information fusion system. Other system architecture factors also affect these metrics.

II.8.3.5.1 Timeliness (from data input to product output)

This measures how long a given uncertainty representation technique takes to produce its results since data is input. Taken from another perspective, it measures whether the representation technique is capable of producing results within the timeframe required by the system’s performance goals.

II.8.3.5.2 Throughput (average / peak rates through the system)

This is a measure of the average and (possibly) peak rate through the system.   This differs from timeliness in that a system can have a long timeliness, but still produce a large number of answers in a given amount of time.

II.8.4 Output Criteria

These criteria are usually related to the system’s results and its ability to communicate it to its users in a clear fashion.

II.8.4.1 Quality

This is a group of criteria meant to assess the informational quality of the system’s output. It is common to see in the literature the same concepts with different names. For example, Accuracy sometimes is used as a synonym of precision; sometimes they are used with the exact opposite of their use below.

II.8.4.1.1 Accuracy

Criteria on accuracy are meant to assess the output of the system in terms of “how right” the answers are. Usual metrics include rate of correct identification/hit, false alarm rate, etc.

II.8.4.1.2 Precision

Criteria on precision are meant to assess the output of the system in terms of “how good” the answers are. It is a measure of the granularity of the system’s output.

Accuracy and precision can be inversely related.  As one makes the granularity coarser, one can expect that the system will have a better accuracy. Precision can also be used to put a boundary on the certainty of the reported result.

II.8.4.2 Interpretation

The output of the system in terms of uncertainty representation and reasoning should have a clear meaning that is sufficiently definite to be used to guide assessment, to understand the conclusions of the system and use them as a basis for action, and to support the rules for combining and updating measures (adapted from Walley 1996).

II.8.4.3 Traceability

Traceability criteria focus on establishing a correlation between the outcome of the reasoning process with the various input and events computed by the system, so for example one can easily explain why and how the system arrived to a specific answer.

Link to the URREF ontology

http://eturwg.c4i.gmu.edu/files/ontologies/URREF.owl

References

Blasch, Eric, Pierre Valin and Eloi Bosse, “Measures of Effectiveness for High-Level Fusion”, 13th Conference on Information Fusion (FUSION), 2010.

Costa, Paulo C. G.; Carvalho, Rommel N.; Laskey, Kathryn B.; and Park, Cheol Y. (2011) Evaluating Uncertainty Representation and Reasoning in HLF systems. In Proceedings of the Fourteenth International Conference on Information Fusion (Fusion 2011). July 5-8, 2011, Chicago, IL, USA.

Costa, Paulo C. G.; Chang, KuoChu; Laskey, Kathryn B.; Levitt, Tod; and Sun, Wei (2010) High-Level Fusion: Issues in Developing a Formal Theory. In Proceedings of the Thirteenth International Conference of the Society of Information Fusion (FUSION 2010). July 26-29, 2010, Edinburgh, Scotland, UK.

Dubois, D.; and Prade, H. (1994) Can We Enforce Full Compositionality in Uncertainty Calculi. Proceedings AAAI-1994, pp. 149-154. AAAI Press.

Jousselme, A.-L., Maupin, P., Bossé, É., Quantitative approaches, in Concepts, Models and Tools for Information Fusion,  Artech House, chap. 8, 2006, pp. 169-210.

Nikolov, A.; Uren, V.; Motta, E.; and de Roeck, A. (2007) Using the Dempster-Shafer Theory of Evidence to Resolve ABox Inconsistencies. In Proceedings of the 3rd Workshop on Uncertainty Represention for the Semantic Web (URSW 2007). November 12, 2007. Busan, Korea.

Pawlak, Z., 1982. Rough sets. International Journal of Computer & Information Sciences, 11(5), pp.341-356.

Pawlak, Z., 1991. Rough sets: theoretical aspects of reasoning about data, Springer.

Schum, David, The Evidence Foundations Of Probabilistic Reasoning, Northwestern University Press, 1994.

Walley, P., Measures of uncertainty in expert systems, Artificial Intelligence, 83(1), May 1996, pp. 1-58

Yaghlane, B. B.; and Laamari, N. (2007) OWL-CM : OWL Combining Matcher Based on Belief Functions Theory. In Proceedings of the 2nd International Workshop on Ontology Matching (OM-2007). November 11, 2007. Busan, Korea.