Csdrhrt

Why does non-parametric bootstrap not return the same sample over and over again?

My notes write:

Assume data $X_1,...,X_n$.

Sample data with replacement to produce $X_1^{(p)},...,X_n^{(p)}$

Now since both are length $n$, then how does this not produce always the same sample? I'm missing something.

Each member of the bootstrap sample is selected randomly with replacement from the data set. If we were to sample without replacement, then every sample would simply be a re-ordering of the same data. But, as a consequence of replacement, the bootstrap samples differ in how many times they include each data point (which may be once, multiple times, or not at all). On average, ~63% of data points appear at least once in a given bootstrap sample.

@user20160's explanation is fine. Here's an example of 10 bootstrap samples of the sequence from 1 to 5, showing that some values will be represented more than once and other values will not be represented (x <- 1:5; t(replicate(10,sort(sample(x,replace=TRUE)))))

      [,1] [,2] [,3] [,4] [,5]

 [1,]    2    2    4    4    5

 [2,]    1    1    1    2    4

 [3,]    3    3    3    5    5

 [4,]    1    1    1    2    3

 [5,]    1    1    2    3    3

 [6,]    1    2    3    4    4

 [7,]    2    2    3    4    5

 [8,]    3    3    3    4    4

 [9,]    1    1    2    3    5

[10,]    1    1    2    4    4

Just to confirm the answers here, the key misunderstanding is the questioner believes there is no replacement in the sampling. Thus if there are 10 elements and 10 random sampling events and 2 replications, each replication is identical to the other without replacement. The number of random sampling events can never exceed the original sample size.

However, with replacement the number of sampling events in theory could exceed the number of elements, thus the original sample size could increased to any given number. In practice however this would be erroneous because you would artificially lower the variance (which is a no no), the mean however would remain the same.

Just to clarify, increasing the number of replications is the correct approach to stabilise both the mean and variance. I'll refrain from elaborating.

Just to waffle, bootstrapping (nonparametric) is cool when you've no idea how to derrive the 95% confidence interval of the mean (sort the bootstrap and remove the upper and lower 2.5%). The technique has its critiques however.