2

I have conducted an experiment but I am now unsure of how to say, from a statistics point of view, that the data supports or not that a certain phenomenon has occurred, meaning it could be mere measurement error.

This was the experiment: a sample of steel had its ferrite(one common constituent of steels) content measured by a certain device 10 times, resulting in 10 values(likely hovering around the true value), a mean value and the standard deviation(is this really what should be being computed here?).

Then the sample received a heat treatment and again had its ferrite content measured by the same device 10 times, resulting in 10 values, a mean value and the standard deviation.

Let´s assume the values for the mean and standard deviation for the untreated sample are, respectively, 25 and 1.2. And the values for the treated sample are 23 and 1.

How can I make the proper statistical treatment here? How to go about computing how certain one can be when ascertaining the phenomenon did/did not happen?

EDIT: Actually, in the experiment, one sample was used to measure the non-heat treated ferrite content. Then four sets of samples, of the very same material/same batch of course, were heat treated for different lengths of time at the same temperature, each of the four sets treated at a different temperature.

For each set of samples, all of the samples were heat treated together, then at say, 300 seconds, one sample was taken out of the oven. Then, at 600 seconds another, at 6000 seconds another, and so on.

The ferrite content of all samples was measured using the same device.

The samples that were heat treated for long times have numbers that show clearly that something happened, even without proper statistical analysis. The problem is dealing with the samples treated for short times, as they showed numbers that are similar to the untreated sample, hence the need to test them for statistical significance.

aristotle85
  • 115
  • 1
    This would all depend on what is considered a phenomenon. – RK01 Jul 31 '15 at 12:57
  • @RK01, The phenomenon I'm looking for here is ferrite turning into something else, so as you heat treat the sample, the ferrite content goes down. – aristotle85 Jul 31 '15 at 13:26
  • Just an observation - a single sample is being used for all of your measurements, a single heating cycle was used to affect the treatment, and one measurement tool was used throughout. Each of those could be considered sources of variability that could show up in your data as an "effect" due to the treatment. Have you thought about those? – SWilliams Jul 31 '15 at 14:49
  • @SWilliams No, not really. The experiment design was simply given to me by a professor for me to execute it. Now I have to deal with it. I just edited the main post, please take a look at it again. – aristotle85 Jul 31 '15 at 16:10
  • OK - a much more interesting problem! You might first want to plot your results i.e. ferrite content vs. treatment duration $t$, and showing the content for untreated samples (say, $t=0$) as a baseline. Doing so will give you a good handle on the degree of variability within and between all of your samples as a function of treatment duration. Beyond that its risky for me to say what should happen next because the shape of the data will say a lot about what approach might be useful. I would imagine a regression would be helpful with great care taken to specify the model appropriately. – SWilliams Jul 31 '15 at 16:42

1 Answers1

2

Of course, it would be better if you had repeated this experiment for several steel specimens. But it is almost always desirable to have more data. The question here is what you can do with the data you have.

It seems you had $n_1 = 10$ ferrite determinations before heat treatment with $\bar X_1 = 25$ and $s_1 = 1.2.$ Then after heat treatment, you made $n_2 = 10$ comparable determinations with $\bar X_2 = 23$ and $s_2 = 1.0.$

Then, assuming the data are not markedly far from normal, a two-sample t test is appropriate for assessing whether the decrease in ferrite content is statistically significant.

You can look at the formulas in Sect. 1.3.5.3 of the NIST Engineering Handbook (search two sample t NIST online). For your data, I did the computation using Minitab statistical software, as follows:

 Two-Sample T-Test and CI 

 Sample   N   Mean  StDev  SE Mean
 1       10  25.00   1.20     0.38
 2       10  23.00   1.00     0.32

 Difference = mu (1) - mu (2)
 Estimate for difference:  2.000
 95% CI for difference:  (0.962, 3.038)
 T-Test of difference = 0 (vs not =): 
   T-Value = 4.05  P-Value = 0.001  DF = 18
Both use Pooled StDev = 1.1045

I used the 'pooled' version of the test, in which one assumes that the population variances are equal.

The results show that the decrease in ferrite content is statistically significant: (1) The P-value 0.001 indicates that if there truly was no change, there would be only about one chance in a thousand seeing a decrease as large as 2. (2) A 95% confidence interval for the true decrease is $(0.962, 3.038)$, which does not include $0$ difference as believable value. (3) Also, not shown in the Minitab printout: a 'critical value' is 2.101; if the absolute value $|T| = 4.05 > 2.101$, that is evidence at the 5% significance level that the two population means differ. (Any one of these three interpretations, suffices to answer your question.)

If you had provided data for ferrite measurements at higher temperatures or for longer heating times, one could use a more complicated ANOVA (analysis of variance) or regression design to decide whether any significant differences exist, and (to an extent) also to say which heat treatments differ from which other ones. You seem to feel confident, on this occasion, that these effects are obviously real. But bear in mind for future reference that it is possible to deal with more than one such comparison at a time in a statistical model.

Notes (sort of like lawyers' fine print):

(1) One might have used the Welch (separate variances) version of the two-sample t test, also referenced in the NIST handbook. It would give the same value of $T,$ used degrees of freedom $df = 17,$ and given a P-value that is not noticeably different. Here, the CI is $(0.958, 3.042)$ also essentially the same as above, and the critical value is 2.110.

(2) If, before seeing the data, you expected the ferrite content to decrease after the heat treatment, you could have used a one-sided test, resulting in an even smaller P-value.

(3) Of course, one could quibble and say that it is the passage of time, or some other reason other than heat treatment that caused the decrease in ferrite content (drift in accuracy of ferrite determination? humidity? lunar phase?). But the time order of untreated preceding treated is inevitable, and the objection seems silly.

(4) I see no justifiable way to use a paired t test here because I see no natural pairing of individual before-after measurements. Perhaps you could run a paired experiment if it were possible to measure the same exact pieces of steel before and after, but I don't think that is what you did.

(5) It would have been preferable to see the 20 individual measurements. That would have made it possible to do certain diagnostic tests. If you have these individual measurements available, edit then into your question and I will have a look. But the effect is sufficiently large that I do not foresee any change in interpretation.

BruceET
  • 51,500
  • Thank you for correcting my mistake! One should definitely use a two sample t-test (not paired!). I should have read the question properly, so I deleted my previous comment. – Joseph Aug 01 '15 at 06:12
  • @Bruce Trumbo, sorry for taking so long to respond. I am trying to read some theory on this subject but it will a little while to digest it. I'will get back soon. – aristotle85 Aug 03 '15 at 02:33