Sunday, January 30, 2011

2100+ Flips, Anger at Sqrt(N)



2102 samples now.

This is about the time where I start to hate the nature of random numbers. It turns out, the uncertainty of the estimate of expectation values (i.e., the mean) scales as 1/sqrt(N), where N is the number of samples one has observed. In other words, to reduce the uncertainty by a factor of two, one has to increase the number of observations by a factor of four.

Right now, we have a roughly 1% 1-sigma confidence on our sample. So, we know the odds of each prize to about 1% with a confidence of about 68%. To improve the uncertainty to 0.5%... it would take about 8000 flips total. That's about 6000 flips to go. Ouch.


Cumulative stats summary:

Total Samples: 2102
Current total prize money won: 4871 g
Current estimated fair price: 2 g 31 s 73 c


Prize (gold) CountPercentageBootstrap Error (1-sigma)
0.1 111052.81%1.09%
0.5 41819.89%0.87%
1 33115.75%0.80%
5 1929.13%0.63%
20 381.81%0.29%
50 60.29%0.12%
200 60.29%0.12%
1000 10.05%0.05%
5000 00.00%NA


Here's a plot of the distribution:




Raw data page is updated.

7 comments:

  1. Hmm... this is cool. Would I be right in thinking the error is only about plus or minus 2.5 silver for the 'Current estimated fair price' at sigma-1?

    ReplyDelete
  2. Are you still gathering all this data by hand, or are you using some kind of addon for it? If you're doing it by hand, I could create a simple addon to somewhat ease the pain of the data gathering for you, if you want.

    ReplyDelete
  3. Not quite KerPow. The uncertainty of the individual bins is around 1%, but that can still have large fluctuations in the fair value. Think about it this way: given the data so far, if the probability of 5000g card is 0.01%, the fair value is around 2.8g, but if it is 0.02%, the fair value would be 3.3g. This distribution is quite fat-tailed. Side note: fat tails are what killed some Wall Street firms...

    ReplyDelete
  4. MaienM, thanks for the offer! I've looked into this once and couldn't find a simple way to tweak the API to do what I want, but if you have fresh ideas, I would love to hear it.

    ReplyDelete
  5. So far I've got a working addon that stores the time and the worth of the received card. The only thing that's left is to get the itemID, which is more of a challenge.

    The reason for this is that there is an event that triggers when you create an item, which is the case here. That event tells you the texture of the created item, but not the itemID. The textures for the 10 silver cards are all identical. The same goes for all other values. So this way you can determine the value of the received card, but not the itemID.

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. I've got it to store itemIDs as well now. I've also included a script/program to convert the gathered data into something you should be able to import into excel.

    Sample result.txt file: http://mmfail.dyndns.org/result.txt

    This result.txt is a very simple example, if you need other fields (like a wowhead link) I could add those to the result.txt generator for you.

    Addon + instructions: http://mmfail.dyndns.org/DataCollector.zip

    ReplyDelete