Behavioral Model Validation in Reuse Scenarios

Introduction and Motivation

As engineering designers, we are faced with several competing objectives.  A prime example of this happens during decision making: we are expected to make effective decisions, but to do so in minimum time and at minimum expense.  Engineering designers are not alone in experiencing such conflicts.  Just like everyone else, we must manage the situation as best we can and move on.

Modeling is one way in which we cope with the demands of engineering design.  Rather than construct and measure the behavior of a system (a time-consuming and expensive endeavor), we typically develop a model and use it to predict system behavior.  When the system in question is a design alternative, these behavioral predictions inform the decisions we make about it.  It usually is faster and cheaper to develop and use a model than to build and test a real system.  Furthermore, we can use models in many situations where constructing or using the real system is not possible.

One can further reduce time and expense by reusing previously developed models.  Model reuse is beneficial because model development typically is more demanding than model use.  Although individual designers often reuse models they have developed themselves, it currently is less common for them to reuse models created by others.  There are two primary reasons for this:

1. It is difficult for them to search for and access models created by others.
2. Of the models to which they do have access, they often lack knowledge sufficient to determine their validity in the new use situation.

Recent advances in information technology and knowledge management address the first of these problems. Technologies such as design repositories (Szykman, Sriram et al. 1998; Szykman, Sriram et al. 2000) and behavioral model repositories (Mocko, Malak Jr. et al. 2004) allow users to search for and retrieve behavioral models created by others and stored remotely. Although it is not quite so simple, one can imagine this as “Googling” to find a model for a phenomenon or system of interest.

I address the second of these problems in my MS research.  Please read on for a summary of my ideas.  See the related publications list for further details.

Limitations of Existing Model Validation Approaches

Existing approaches for model validation are based on assumptions about the model development and use process.  These assumptions do not hold in general model reuse scenarios.  The following sequence of figures depict the problem at a high level.

Core Issue: Validation-Relevant Knowledge

The core issue in model validation for reuse scenarios is that model developers and model users each have different portions of the knowledge requisite for performing model validation.  Specifically:

• Model developers know the assumptions that are embodied in a model and their implications on model uncertainty.
• Model users know the intended use scenarios for the model.

In order to perform model validation, one requires the knowledge originating with both developers and users.  In many cases, model users can consult the developers of a model they intend to use.  However, this is not the case in general.  A model stored in a repository might outlast its developers (e.g., developers retire or move to a different company) or consultation may be difficult for other reasons (e.g., developers and users work at different locations).

The limitations of existing approaches to model validation stem from their being based on a framework that does not account for such separations between validation-relevant knowledge and those who need it. 

Validity Descriptions of Models: Context + Inaccuracy

In this work, I propose that model developers should characterize formally their validation-relevant knowledge.  These characterizations -- termed validity descriptions -- are unambiguous representations of the validation-relevant knowledge that originates with model developers.  This knowledge consists of a statement about the total uncertainty -- or, inaccuracy -- in a model over a well-defined set of circumstances -- or, context.  This is precisely the validation-relevant knowledge that model users lack.

Context

Context is the limited domain over which a model applies.  In a validity description, context serves to specify when claims about inaccuracy are trustable.  For instance, the relationship F=-kx has minimal inaccuracy under a particular set of conditions but can have a large inaccuracy under other conditions (e.g., if the material in question is not elastic, if the deformation becomes large, etc.,.). 

Inaccuracy

Inaccuracy refers to the total uncertainty present in a model (within a particular context).  This must include all sources of uncertainty.  Two particular sources of uncertainty are important to condition: aleatory uncertainty and epistemic uncertainty.  Aleatory uncertainty is due to natural random behavior (e.g., annealing, machining erros, radioactive decay. etc.,.).  Epistemic uncertainty is due to a lack of knowledge (e.g., modeling assumptions embodied in a model, not knowing what decision someone else will make, etc.,.).  Essentially, one can eliminate epistemic uncertainty (at a cost) by gathering more information while aleatory uncertainty is intrinsic to the process or phenomena under consideration.  Both types of uncertainty are common in engineering design.  Model developers must therefore account for both.

Performing Model Validation: Characterization and Assessment Steps

With validity descriptions as the representation for validation-relevant knowledge, it is possible to define model validation steps for developers and users:


Validity Characterization:  Model developers create a validity description for a model they develop.

Compatibility Assessment:  Model users determine whether the context of a model they are interested in reusing is compatible with that of the simulation problem they have in mind.

Adequacy Assessment:  Model users determine whether a model (that has passed compatibility assessment) is sufficiently accurate for their needs.


Traditionally, model validation is defined as the "substantiation that a computerized model within its domain of applicability possesses a satisfactory range of accuracy consistent with the intended application of the model" (Schlesinger, Crosbie et al. 1979).  The steps outlined above are consistent with this definition, but are appropriate for general model reuse scenarios.

Why Not Rely on Documentation?

Traditional, text-based documentation is not a good mode of knowledge transfer in this case.  It is common for model developers to make assumptions such as "spring assumed massless" and "standard temperature and pressure conditions are assumed."  The documentation for a model typically contains such phrases.  However, assumptions of this form never are met in practice.  Users are left to interpret what developers mean by such phrases.  How massive must something be before a zero-mass assumption becomes poor?  How far from standard temperature and pressure can we get before the model becomes intolerably inaccurate?  Even assumptions of the form "drag assumed negligible" embody implicit assumptions about what constitutes negligible.

The solution is to formalize the implications of various assumptions in a well-defined fashion.  This frees model users from interpreting the work of model developers and allows model developers to control better the way in which their models are used.  Formalized representations have the added benefit of being computer-interpretable, which is useful when searching repositories and automating computational steps.

Related Publications and References

• R. J. Malak, Jr. "A Framework for Validating Reusable Behavioral Models in Engineering Design," Master's Thesis, G.W. Woodruff School of Mechanical Engineering, Georgia Institute of Technology, Atlanta, GA, 2005. [paper(pdf)]
  
• R. J. Malak, Jr., and C. J. J. Paredis, “Foundations of Validating Reusable Behavioral Models in Engineering Design Problems,” in Proceedings of the 2004 Winter Simulation Conference, R .G. Ingalls, M. D. Rossetti, J. S. Smith, and B. A. Peters, eds., Washington D.C., December 5-8, 2004. [paper(pdf)][slides (pdf)]
  
• R. J. Malak, Jr., and C. J. J. Paredis, “On Characterizing and Assessing the Validity of Behavioral Models and Their Predictions,” in Proceedings of DETC 2004, Design Theory and Methodology Conference, paper no. DETC2004/DTM-57452, Salt Lake City, UT, September 28 – October 3, 2004. [paper(pdf)][slides (pdf)]
  
• G. Mocko, R. Malak, C. Paredis, R. Peak, “A Knowledge Repository for Behavioral Models in Engineering Design,” in Proceedings of DETC 2004, Computers and Information  in Engineering Conference, paper no. DETC2004/CIE-57746, Salt Lake City, UT, September 28 – October 3, 2004. [paper(pdf)][slides (pdf)]

Back to Top
Back to Home