In preparation for the group’s planned discussion on use cases on December 7, I suggest that the first purpose for the use cases is not to provide data that we can use to evaluate different uncertainty methodologies. Rather, we should use them first to help us understand the scope and issues associated with our task. Two of the working group’s three tasks are:
To establish features required for any quantitative uncertainty representation to support the exchange of soft and hard information in a net-centric environment;
to define evaluation criteria supporting unbiased comparisons among different approaches applied to the use cases.
Without understanding what these features and criteria are, having use cases with datasets will make us an interesting repository for university students needing data for class projects, but isn’t likely to bring this working group success.
One way to establish these features and criteria is to use the use case scenarios as a guide to walk through what the fusion system must do in that case, what kind of input data it needs and what kinds of processes it must do with that data. The use case would also set the context and overarching requirements for the system. For this initial effort, we don’t need datasets; rather, we need a set of use case scenarios that cover the range of expected issues we expect to address within this working group.
Below I sketch out some of the things I’ve thought about going through one example.
Consider walking through Use Case #1 (ship tracking). The original scenario was that US intelligence received reporting that a merchant ship had left a certain country carrying a low-level radiological bomb, to be smuggled into the United States. At least some of the crew is witting, and the ship is expected to engage in deceptive behavior to mask its intent. The requirement is to identify which ship is carrying the bomb. Elements of data available(not on all ships) include
Ship track information on ships at sea between country X and the United States
Reports (intelligence , commercial, open source, etc) on ship departures, manifests, crew
manning and ownership relationships
Ship characteristics data (dimensions, cruise / max speeds, pictures)
Ship behavior reporting (visual and electronic behavior
Walking through this scenario in my mind, I come up with the following general questions that I think affect the overall objectives of this group. Some of them are:
Are the evaluation criteria for the uncertainty representation methodology the same as the evaluation criteria for a high level fusion system? Can an evaluation of different uncertainty representation methodologies be separated from an evaluation of the fusion system as a whole, or is the uncertainty representation so fundamental to the fusion system that the two cannot be evaluated separately? To use an automobile metaphor - are we evaluating different engines for a sports car, or is one engine fit for a sports car while another is fit for an 18 wheel tractor trailer? I suggest that if the uncertainty methodology cannot be separated away from the overall fusion system, our evaluation criteria need to account for when we have the sports car, versus the tractor trailer.
A major claim for the different uncertainty representation methodologies is that they handle different types of uncertainty better than other methods. In the use cases that we have, what types / ranges of uncertainties have we captured? Do we have the right set to adequately evaluate the different alternatives? What range of use cases / data sets do we need to conduct a proper evaluation? (Note: I don’t expect that any particular use case is “fair”, in the sense that any of the methodologies could be appropriately applied. But the set of use cases should be “fair” in that they give a good assessment of the range of applicability as well as differences in performance in a particular case).
In evaluating a high level fusion system, what are we assuming about the low level fusion that supports it? Is the uncertainty methodology assessment dependent on the understanding the methodologies used at all levels, or can they be partitioned? At least two choices exists and hybrids are possible:
a) The low level fusion process is an integral component of the high level process. Our assessment includes the effects of different uncertainty methodologies on both low and high level fusion. If so, we’ll need datasets that represent raw input data (e.g. sensor outputs , text reports, etc)
b) We only need the output of the low level fusion process, along with the uncertainty measures (e.g. track files, outputs of social analysis network analyses, etc)
The answer to this question affects our evaluation approach / criteria, our definition of features, and the requirements we have for the datasets in those use cases that we take forward for use in evaluating different uncertainty representation methodologies.
Combining the information involves processes that occur at different levels, with outputs from one process feeding another. For example, this scenario requires a process that looks at possible interactions between ship tracks to determine if a meeting at sea could have occurred (say to transfer the bomb to a smaller ship to bring into port). The performance of the overall process depends on how well the subprocesses occur? In evaluating an uncertainty methodology, how do I separate poor subprocess implementation from limitations in the methodology?
Going through this scenario also raises the following question in my mind. Most examples of high-level fusion processes are based on some form of an operational picture, that is map based with multiple elements situated or moving on the map. This is the classical military application, but also applies to areas like managing natural disaster responses. But high level fusion also applies in cases that resemble police detective work, rather than operational picture type fusion (the US intelligence effort prior to 9/11 case being the poster child for where this was not done). Do we want one or more use cases that is focused on a “detective like” problem?
Thoughts on this approach?
In preparation for the group’s planned discussion on use cases on December 7, I suggest that the first purpose for the use cases is not to provide data that we can use to evaluate different uncertainty methodologies. Rather, we should use them first to help us understand the scope and issues associated with our task. Two of the working group’s three tasks are:
Without understanding what these features and criteria are, having use cases with datasets will make us an interesting repository for university students needing data for class projects, but isn’t likely to bring this working group success.
One way to establish these features and criteria is to use the use case scenarios as a guide to walk through what the fusion system must do in that case, what kind of input data it needs and what kinds of processes it must do with that data. The use case would also set the context and overarching requirements for the system. For this initial effort, we don’t need datasets; rather, we need a set of use case scenarios that cover the range of expected issues we expect to address within this working group.
Below I sketch out some of the things I’ve thought about going through one example.
Consider walking through Use Case #1 (ship tracking). The original scenario was that US intelligence received reporting that a merchant ship had left a certain country carrying a low-level radiological bomb, to be smuggled into the United States. At least some of the crew is witting, and the ship is expected to engage in deceptive behavior to mask its intent. The requirement is to identify which ship is carrying the bomb. Elements of data available(not on all ships) include
Ship track information on ships at sea between country X and the United States
Reports (intelligence , commercial, open source, etc) on ship departures, manifests, crew
manning and ownership relationships
Ship characteristics data (dimensions, cruise / max speeds, pictures)
Ship behavior reporting (visual and electronic behavior
Walking through this scenario in my mind, I come up with the following general questions that I think affect the overall objectives of this group. Some of them are:
a) The low level fusion process is an integral component of the high level process. Our assessment includes the effects of different uncertainty methodologies on both low and high level fusion. If so, we’ll need datasets that represent raw input data (e.g. sensor outputs , text reports, etc)
b) We only need the output of the low level fusion process, along with the uncertainty measures (e.g. track files, outputs of social analysis network analyses, etc)
The answer to this question affects our evaluation approach / criteria, our definition of features, and the requirements we have for the datasets in those use cases that we take forward for use in evaluating different uncertainty representation methodologies.
Going through this scenario also raises the following question in my mind. Most examples of high-level fusion processes are based on some form of an operational picture, that is map based with multiple elements situated or moving on the map. This is the classical military application, but also applies to areas like managing natural disaster responses. But high level fusion also applies in cases that resemble police detective work, rather than operational picture type fusion (the US intelligence effort prior to 9/11 case being the poster child for where this was not done). Do we want one or more use cases that is focused on a “detective like” problem?
Thoughts on this approach?