Csdrhrt

I have 5 bags, each with 20 balls. In each bag one of the balls is special - it's red, the others are white. The game is to pick balls without replacing until I choose the special ball.

My friend says he is psychic and can pick a special ball statistically significantly greater than chance.

He picks the chosen ball (for example):
2nd on bag 1,
3rd on bag 2,
1st on bag 3,
4th on bag 4, and
2nd on bag 5.

This average (2.2) is obviously quite a good result and much lower than the expected value (10.5) - but can I show that he is very unlikely to have done so well by chance?

I feel it's the same problem as calculating to the probability of say rolling a total of 11 or lower on 5, 20 sided dice.

For a person picking at random, chance of getting the special on first pick is $ frac 1 {20}$. On second try it is $frac {19} {20} * frac 1 {19}$ = $ frac 1 {20}$. Following the same logic, we get for a single bag probability to pick the special ball on $n^{th}$ pick is

$P(n) = frac 1 {20}$

Using this we can construct a discrete uniform distribution for the number of pick we get the special ball.

$E(n) = 10.5$

$Var(n) = frac {(b - a + 1 ) ^ 2 -1} {12} = frac {399} {12}$

Now that we have a distribution, we need to do a goodness of fit test. Using a one sided chi-squared test we can show whether the results are meaningful or not. Since you only have 5 observations your degree of freedom would be really low. You can test it with different significance levels (or look at p-value)

However, as you pointed out in the comment,Chi-squared tests only test whether it's from the same distribution, not whether on lower end. We can do a monte carlo simulation to get cdf and get the answer that way, however since we are working on the tail, it might be slow to converge (importance sampling could be used ).

We can also use brute force to calculate it.

Theoretically, the results of the 5 pick can come up in $20^5$ ways. Your psychic friend got a total of 12. Now lets get the distribution of the sum.

To get a score of 5 (getting all at 1st try) there is only 1 way. We have to choose all of them at the first pick.

To get a score of 6 or less (get one at second try all others on the first) there are 5 + 1 = 6 ways.

To get a score of 7, and more lets come up with a general solution

Imagine the total is 11. (eg. he got the right ball on picks 2/3/2/2/2)
How many total ways there can be to arrange that? Let 0 denote a wrong pick and x denote the right pick. the example could be described as:

0x00x0x0x0x

This could be arranged in $frac {11!} {6!5!} $ = $11 choose 5$ ways
Since there is always going to be exactly 5 x and there could be no more than 19 consecutive 0, we can use $n choose 5$ up to 20 safely. If the arrangement happens something like the following

0xxx0xx0000

It means that we got the right ball on picks (2,1,1,2,1) and ended with 7, which is an arrangement included in $11 choose 5$

Since your friend got 12 points, it would be $frac {12 choose 5} {20^5} = 0.0002475. $ Since this is smaller than 1%, we can say he is statistically better at picking.

Hope it helps