Logistics concepts and applications

¶ … Coherence Between the Use of Simulation and Experimental Research Design

The work of Kleijnen, Sanchez, and Lucas (2005) entitled: A User's Guide to the Brave New World of Designing Simulation Experiments" states that DOE or 'Design of Experiments' has a history that is not only rich but that has many "…theoretical developments and practical applications in a variety of fields. Success stories abound in agriculture, clinical trials, industrial product design, and many other areas. Yet, despite the impact DOE has had on other fields and the wealth of experimental designs that appear in the literature, we feel DOE is not used as widely or effectively in the practice of simulation as it should be." (Kleijnen, Sanchez, and Lucas (2005)

It is additionally stated in the work of Kleijnen, Sanchez, and Lucas (2005) that simulation has with computer technology advances lost some of the "constraints that characterize real-world experiments." This results in both opportunities and challenges. The beginning point of the simulation is stated to be three primary goals in analyzing simulation in that the clients may have in terms of: (1) developing a basic understanding of a particular simulation model or system; (2) finding robust decisions or policies; and (3) comparing the merits of various decisions or policies. (Kleijnen, Sanchez, and Lucas, 2005) p.265

The work of Chen, Tsui, Barton, and Mechesheimer (2004) entitled: "A Review on Design, Modeling and Applications of Computer Experiments" states that in complex system design "computer experiments are frequently the only practical approach to obtaining a solution. Typically, a simulation model of system performance is constructed based on knowledge of how the system operates." (p.273) in the event that measure of performance is not a straightforward calculation then the sampling via computer experiments "may be employed to estimate the measure." (Chen, Tsui, Barton, and Mechesheimer, 2004) p. 273 in the event that the simulation model is "computationally expensive, then the optimization may instead rely on a metamodel." (Chen, Tsui, Barton, and Mechesheimer, 2004) p.273

Chen, Tsui, Barton, and Mechesheimer (2004) state that there are two primary tasks in meta-modeling: (1) select a set of sample points in the design parameter space (i.e., an experimental design); and (2) fit statistical model(s) to the sample points. (2004) Methods for the first task may be used to conduct sampling in general. (Chen, Tsui, Barton, and Mechesheimer, 2004) p.273 it is stated that in real world experiments, "only a small number of factors are typically varied" because it is not practical or possible to "attempt to control more than…10 factors; many published experiments deal with fewer than 5." (Chen, Tsui, Barton, and Mechesheimer, 2004) p.269 Choice of performance measures is addressed and it is stated that some problems makes a requirement of "only relative answers…conversely, some problems require absolute answers." (Chen, Tsui, Barton, and Mechesheimer, 2004) p.269

The DOE framework is stated to be restrictive to use in that it suggests that "the appropriate goal of the study should be examining the expected value of a single performance measures." (Chen, Tsui, Barton, and Mechesheimer, 2004) p.269 Terminating simulations run until a specific event has occurred; Steady state simulations are stated to have no natural point of termination" resulting in ongoing generation of data for analysis. The simulation type has a great bearing on the design and analysis and it is stated that for the terminating simulation that it might be necessary to "…censor results if we are simulating rare events…" (2004) p.270

Chen, Tsui, Barton and Mechesheimer (2004) additionally state that the initial conditions for 'steady-state simulations…are "often chosen for convenience rather than relevance," (2004) p.270 the example stated is that a simulation of a computer network may begin with "…all servers and relay nodes operational and no demands on the system. Here, the simulation output of the warm-up period biases the estimated response. The length of the warm-up period affects the total time required for experimentation." (Chen, Tsui, Barton and Mechesheimer, 2004) p. 270. Chen, Tsui, Barton, and Mechesheimer (2004) state the fact that analysts have control over many things during the course of a simulation study. There are many things which are under the control of analysts during the simulation study. Analysts have control over many things during the course of a simulation study. This is stated to include the "maximum run time for terminating simulations.

For steady-state simulations this control includes the specification of the warm-up period as well as the run lengths including the time-series output is averaged or aggregated into batches." (2004) p. 271 the choice of the number of batches and batch sizes important topic of research in itself, and an implicit assumption in many simulation-analysis techniques is that appropriate batch sizes and warm-up periods are used. Often simulation-analysis techniques is that appropriate batch sizes and warm-up periods are used." (Chen, Tsui, Barton and Mechesheimer, 2004) p.272

Stated to be other simulation-specific factors which can be controlled are "the use of CRNs to facilitate comparisons across alternatives." (Chen, Tsui, Barton and Mechesheimer, 2004) p. 272 All potential factory layouts can be subjected to the same patterns of customer orders. Other variance-reduction techniques (VRTs) such as control variates importance sampling, have been developed for simulation output." (Chen, Tsui, Barton and Mechesheimer, 2004) p.272

Next addressed is the criteria for evaluating designs. The design is stated to be that which determines the "standard errors for the estimated metamodel parameters." (Chen, Tsui, Barton and Mechesheimer, 2004) p.273 Stated as an example is the following:

"A-optimality means that the sum of these standard errors is minimal. D-optimality considers the whole covariance matrix of the estimated parameters (not only the main diagonal) and means that the determinant of this matrix is minimal. G-optimality considers the mean squared error of the output predicted through the metamodel." (Chen, Tsui, Barton and Mechesheimer, 2004) p.274 These criteria are stated to be and to have been "used to evaluate designs proposed for analyzing simulation experiments…" however, it is stated to be unfortunate that these criteria 'require strong priori assumptions on the metamodels to be fit to the data and the nature of the response. These assumptions are usually violated in simulation." (Chen, Tsui, Barton and Mechesheimer, 2004) p.274 the designs should be "easy to construct if they are to be used in practice." (Chen, Tsui, Barton and Mechesheimer, 2004) p.274 However, it is stated that even some difficult designs are useful therefore tabulated designs are not ruled out for use especially when incorporated into software packages. Analysis is much easier if software for many platforms is available. Since regression software is stated to be readily available and since this is the most common tool used for analysis this is stated to be resolved. There are newer "surface-fitting methods" available as well which include: