I am not quite sure what Bayesian method want to infer from the data? Frequentists assume the parameters generated the data are fixed constants so they can actually estimate those constants.
Bayesians assume the parameters are random, with their own distribution. If parameters are random, how can we estimate it?? Estimating random quantities probably results in some distribution??? I am not quite understand the philosophy of Bayesisans!!
The way I see it, there are two main differences
In Frequentist statistics, the goal is to find a single model that fits most of the data.
The technical term, is Maximum likelihood estimation.
In Bayesian statistics, the goal is to assign belief to each combination of the parameters in light of the data.
The posterior predictive is in fact, a linear combination of models that are likely.
Bayesian statistics gives you tools to incorporate your prior knowledge into the model.
A coin is tossed 3 times and the results are H,H and H.
A frequentist would conclude that there is a 100% chance of getting a H the fourth time.
A smarter frequentist would tell you to get more data and to never bother him again with such small sample.
Where a Bayesian could model her prior knowledge that the coin is likely to be be fair, and get posterior and posterior predictive distributions that tend more in favor of H but do not eliminate the possibility of T.
Given enough data, the Bayesian prior is worn away, and the frequentist models yield the same results as the Bayesian models.
