Experimental Design In Research

Importance of an Experimental Design and Statistical Analysis

Experimental Design In Research,The importance of careful experimental design, Overview of statistical analysis,EDA always precedes formal (confirmatory) data analysis.

    Scientific learning is always an iterative process, as represented in. If we start at Current State of Knowledge, the next step is choosing a current theory to test or explore (or proposing a new theory). This step is often called “Constructing a Testable Hypothesis”. Any hypothesis must be allowed for different possible conclusions or it is pointless.

    For an exploratory goal, the different possible conclusions may be only vaguely specified. In contrast, much of statistical theory focuses on a specific, so-called “null hypothesis” (eg, reaction time is not affected by background noise) which often represents “nothing interesting going on” usually in terms of some effect being exactly equal to zero, as opposed to a more general, “alternative hypothesis” (eg, reaction time changes as the level of background noise changes), which encompasses any amount of change other than zero.

     The next step in the cycle is to “Design an Experiment”, followed by “Perform the Experiment”, “Perform Informal and Formal Statistical Analyses”, and finally “Interpret and Report”, which leads to possible modification of the “Current State of knowledge”.

The importance of careful experimental design

    Experimental design is a careful balancing of several features including "power", generalizability, various forms of "validity", practicality and cost. A thoughtful balancing of these features in advance will result in an experiment with the best chance of providing useful evidence to modify the current state of knowledge in a particular scientific field. 

    On the other hand, it is unfortunate that many experiments are designed with avoidable flaws. It is only rarely in these circumstances that statistical analysis can rescue the experimenter. This is an example of the old maxim “an ounce of prevention is worth a pound of cure”.

    "Our goal is always to actively design an experiment that has the best chance to produce meaningful, defensible evidence, rather than hoping that good statistical analysis may be able to correct for defects after the fact."

 Overview of statistical analysis

    Statistical analysis of experiments starts with graphical and non-graphical exploratory data analysis (EDA). EDA is useful for.

·       1:detection of mistakes

·       2:checking assumptions

·      3:determining relationships among the explanatory variables

·      4:assessing the direction and rough size of relationships between explanatory and outcome variables, and

·   5:preliminary selection of appropriate models of the relationship between an outcome variable and one or more explanatory variables.

“EDA always precedes formal (confirmatory) data analysis.”

    Most formal (confirmatory) statistical analyzes are based on models. Statistical models are ideal, mathematical representations of observable characteristics. Models are best divided into two components. The structural component of the model (or structural model) specifies the relationships between explanatory variables and the mean (or other key feature) of the outcome variables. 

    The “random” or “error” component of the model (or error model) characterizes the deviations of the individual observations from the mean. (Here, “error” does not indicate “mistake”.) The two model components are also called “signal” and “noise” respectively. Statisticians realize that no mathematical models are perfect representations of the real world, but some are close enough to reality to be useful. 

    A full description of a model should include all assumptions being made because statistical inference is impossible without assumptions, and sufficient deviation of reality from the assumptions will invalidate any statistical inferences. A slightly different point of view says that models describe how the distribution of the outcome varies with changes in the explanatory variables.

    "Statistical models have both a structural component and a random component which describe means and the pattern of deviation from the mean, respectively."

    A statistical test is always based on certain model assumptions about the population from which our sample comes. For example, a t-test includes the assumptions that the individual measurements are independent of each other, that the two groups being compared each have a Gaussian distribution, and that the standard deviations of the groups are equal. 

    The farther the truth is from these assumptions, the more likely it is that the t-test will give a misleading result. We will need to learn methods for assessing the truth of the assumptions, and we need to learn how "robust" each test is to assumption violation, ie, how far the assumptions can be "bent" before misleading conclusions are likely.

"Understanding the assumptions behind every statistical analysis we learn is critical to judging whether or not the statistical conclusions are believable."

    Statistical analyzes can and should be framed and reported in different ways in different circumstances. But all statistical statements should at least include information about their level of uncertainty. The main reporting mechanisms you will learn about here are confidence intervals for unknown quantities and p-values and power estimates for specific hypotheses.

    "p-values are not the only way to express inferential conclusions, and they are insufficient or even misleading in some cases."

An oversimplified concept map.

What you should learn here

    THe expectation is that many of you, coming into the course, have a “ conceptmap ” similar. This is typical of what students remember from a first course in statistics. By the end of the book and of course you should learn many things. You should be able to speak and write clearly using the appropriate technical language of statistics and experimental design. 

    You should know the definitions of the key terms and understand the sometimes-subtle differences between the meanings of these terms in the context of experimental design and analysis as opposed to their meanings in ordinary speech. You should understand a host of concepts and their interrelationships. 

    These concepts form a “concept-map” such as the one in that shows the relationships between many of the main concepts stressed in this course. The concepts and their relationships are the key to the practical use of statistics in the social and other sciences. As a bonus to the creation of your own concept map, you will find that these maps will stick with you much longer than individual facts.

    By actively working with data, you will gain the experience that becomes “ datasense ”. This requires learning to use a specific statistical computer package. Many excellent packages exist and are suitable for this purpose.

A reasonably complete concept map for this topic.

    From SPSS, but this is in no way an endorsement of SPSS over other packages. You should be able to design an experiment and discuss the choices that can be made and their competing positive and negative effects on the quality and feasibility of the experiment. You should know some of the pitfalls of carrying out experiments. It is critical to learn how to perform exploratory data analysis, assess data quality, and consider data transformations. 

    You should also learn how to choose and perform the most common statistical analyses. And you should be able to assess whether the assumptions of the analysis are appropriate for the given data. You should know how to consider and compare alternative models. Finally, you should be able to interpret and report your results correctly so that you can assess how y our experimental results may have changed the state of knowledge in your field.