There are many things which are under the control of analysts during the simulation study. Analysts have control over many things during the course of a simulation study. This is stated to include the "maximum run time for terminating simulations. For steady-state simulations this control includes the specification of the warm-up period as well as the run lengths including the time-series output is averaged or aggregated into batches." (2004) p. 271 the choice of the number of batches and batch sizes important topic of research in itself, and an implicit assumption in many simulation-analysis techniques is that appropriate batch sizes and warm-up periods are used. Often simulation-analysis techniques is that appropriate batch sizes and warm-up periods are used." (Chen, Tsui, Barton and Mechesheimer, 2004) p.272

Stated to be other simulation-specific factors which can be controlled are "the use of CRNs to facilitate comparisons across alternatives." (Chen, Tsui, Barton and Mechesheimer, 2004) p. 272 All potential factory layouts can be subjected to the same patterns of customer orders. Other variance-reduction techniques (VRTs) such as control variates importance sampling, have been developed for simulation output." (Chen, Tsui, Barton and Mechesheimer, 2004) p.272

Next addressed is the criteria for evaluating designs. The design is stated to be that which determines the "standard errors for the estimated metamodel parameters." (Chen, Tsui, Barton and Mechesheimer, 2004) p.273 Stated as an example is the following:

"A-optimality means that the sum of these standard errors is minimal. D-optimality considers the whole covariance matrix of the estimated parameters (not only the main diagonal) and means that the determinant of this matrix is minimal. G-optimality considers the mean squared error of the output predicted through the metamodel." (Chen, Tsui, Barton and Mechesheimer, 2004) p.274 These criteria are stated to be and to have been "used to evaluate designs proposed for analyzing simulation experiments…" however, it is stated to be unfortunate that these criteria 'require strong priori assumptions on the metamodels to be fit to the data and the...

These assumptions are usually violated in simulation." (Chen, Tsui, Barton and Mechesheimer, 2004) p.274 the designs should be "easy to construct if they are to be used in practice." (Chen, Tsui, Barton and Mechesheimer, 2004) p.274 However, it is stated that even some difficult designs are useful therefore tabulated designs are not ruled out for use especially when incorporated into software packages. Analysis is much easier if software for many platforms is available. Since regression software is stated to be readily available and since this is the most common tool used for analysis this is stated to be resolved. There are newer "surface-fitting methods" available as well which include:
(1) Kriging;

(2) neural nets;

(3) radial basis functions;

(4) splines;

(5) support-vector regression; and (6) wavelets, all of which are "metamodel construction methods that can be applied to data collected using a variety of experimental designs and may do a better job fitting certain complex response surfaces." (Chen, Tsui, Barton and Mechesheimer, 2004) p.274

Chen, Tsui, Barton and Mechesheimer (2004) states that all simulation practitioners should view DOE "as an integral part of any simulation study, while researchers should move beyond viewing the setting merely as an application area for traditional DOE methods." (2004, p. 285) Advocated for by these authors is the thinking first about three potential goals of a simulation experiment including:

(1) understanding a system;

(2) finding robust solutions; and (3) comparing two or more systems. (Chen, Tsui, Barton and Mechesheimer, 2004)

The contention stated is that these goals are "…often more appropriate than those typically used, namely, testing hypotheses about factor effects, seeking an optimal policy, or making predictions about performance." (Chen, Tsui, Barton and Mechesheimer, 2004) p., 287

Bibliography

Kleijnen, Jack P.C., Sanchez, Susan M., Lucas, Thomas W. (2005) a User's Guide the Brave New World of Designing Simulation Experiments. State of the Art Review. INFORMS Journal on…