Validation of a simulation is a critical success factor to an effective modeling programme. Over the years of completing many, many models we have acquired some experience about the techniques for validating models in a clear and objective manner – combined with practices inherited from the “Design of Experiments” (DOE) body of knowledge. The following is a treatise on the principles of validation which we apply to our projects.
Two levels of validation
A model is an outcome of a discussion between Subject Matter Experts (SMEs) and model developers. The developers translate the language of the SME into a form that can be readily modeled – most commonly in a series of graphic depictions, like process diagrams or flowcharts. The model developer then creates the simulation software code that represents the “rules” in the depictions.
Therefore there are two distinct levels of validation:
that the model is a faithful recreation of the SMEs knowledge, and
that the model correctly applies the rules.
Level 1 is done by constantly checking the results of the dialogue with the SMEs, who are the ultimate judges of the completeness and accuracy of the rule set. We create the information in graphic form, and route the graphics to the SMEs for approval. As a governing device, the previously established Hypothesis speaks to the proper level of granularity to be represented in the graphic depiction.
Level 2 builds upon Level 1. We assume that the exchange between the modelers and the SMEs, at some point, yields a correct process/rules representation, therefore we move on to the next logical question: does the model compute the consequence of transactions that flow across the process and fire the rules correctly?
Whilst this question seems more straightforward than Level 1 – simply “getting the math right” – our experience suggests that Level 2 represents 90% of the validation effort.
Level 2 validation is conducted by producing a parallel computation to the simulation computation – one that can be generated offline. In years past this was quite literally done “by hand”. Today we use more modern and rapid methods for parallel computation, and our team favors the Mathematica technical computing environment for such work (we occasionally use Microsoft Excel as well).
Level 2 validation explained
Every simulation is a series of computations and actions. Having generated process diagrams that describe each of these computations, we now have a platform for testing each type of computation to ensure its accuracy. In our experience, it is the rare case that errors are small – they are either in line with the simulation or obviously wrong.
The simulation will produce a text log file in CSV format indicating every individual action that the simulation takes over the course of a given simulation run. (Note that this log file is not only for validation purposes but also for post-simulation analysis). This is how the simulation exposes its computations to be compared to the parallel computation. Here is an example from a recent validation:
In this example, taken from the upstream energy industry, the model was asked to follow a table, which represents a gas well drilling program. So one could imagine the rule as follows:
“For a given named lateral, here are the number of wells to be drilled in that calendar year.”
This drilling activity comes at the tail end of many different rules that are applied. Therefore by checking the downstream end of the rule set, we are effectively checking those rules as well. This is another important philosophy that we use in validation.
The Mathematica code above reads the simulation log to make sure that the model is complying to the rule. We spot checked 3 exemplars from the table. The very first entry in the table shows that 8 wells are to be drilled in the Miera 2 lateral in 2009. As illustrated, 8 entries are shown, so the parallel computation and the intended rule agree.
This same kind of process is repeated again and again for all of the major computations in the model, including scorecard metrics. The code to do the validation is preserved so that a new validation may be performed as the code is changed.
General practice is to apply validation from top to bottom – meaning – as the first versions of the model are completed, we conduct overall validations – “sense checking” on obvious outcomes. We look for order-of-magnitude differences here, which if exhibited, are likely the result of a logic error inside the model. These kinds of errors are very common as the initial drafts of the model begin to emerge.
Validating a stochastic model
The above description applies to a deterministic model – one which produces the same outputs for a given set of inputs every time it is run. Some models, however, are stochastic in nature – they have randomization built into certain inputs or logic. As the model is run many, many times, a probability distribution of outcomes more closely resembles the range of real world observations.
Yet stochastic models are challenging to validate given the random nature of an individual simulation run.
Our response has been thus: we always build the deterministic version of the stochastic model first and completely validate it at Level 1 and 2. Then adding the stochastic inputs allows the modeler to isolate the effect of the stochastic addition only. At that point the job of the modeler is restricted to the determination that the model is reproducing the stochastic input and computing that all the way through to the probability distribution outcome.
History Matching
Another element of validation is history matching – comparing the output of the model to that which is observed in its real world equivalent. It is important to point out that history matching is but one element of an overall scheme to validate a model and that the model’s deviation from history does not necessarily imply that the model is wrong.
Keep in mind that a model, by definition, is wrong. It is an abstraction of a real system – an abstraction that is necessary in order to make a model tractable. Therefore there could be legitimate reasons for variation between the real world observation and the outcome of a model. When we see such variations, we go back into the logic of the model to determine the source of the variation, then make a judgment as to whether that source is left alone or “tuned” as appropriate.
Relative Analysis
We often make use of a technique known as Relative Analysis (RA). In RA, we are looking at two styles of simulation runs – the “As Is” and “To Be”. As Is runs of the model work from data and observations of the real system as they actually occurred in history. The model is a “puppet” of the transactions in the data set – forced to execute as recorded by the real system. In the To Be case, we rewind history and imagine the process change over the same set of transactions. It is as if we could go backwards in time and change a policy or physical configuration and rerun it – for example, changing machine downtime from an actual of 85% to a hypothetical 90%. We then observe the implications of this change on our scorecard set of metrics.
Relative analysis tends to exhibit a stronger case, as it is rooted in actual observations as opposed to making an absolute case over a predicted set of future transactions. We do not use relative analysis exclusively, rather, it is one of a series of toosl designed to instill confidence (or provoke necessary tuning) in the model.
