Smyslov_Fan wrote on 09/20/10 at 04:41:33:
Actually, chess.com's methodology is rather interesting.
They ran stats on the best correspondence players of pre-computer generations, such as Estrin and company. Then they took a look at how the moves, after they left the database, matched up to engines. They focused only on games against other 2200 players. They found a fairly consistent threshhold.
Leaving aside the question of whether this is to be believed at all, and further that of whether it was done with any rigor, the devil is in the term
fairly consistent, which indicates that the entire process is subject to
error. It is, therefore, a
statistical question.
Smyslov_Fan wrote on 09/20/10 at 04:41:33:
It doesn't matter which +2900 engine they use, the results will be the same.
The
same? I posit on the contrary that choice of engine is a source of variation.
Smyslov_Fan wrote on 09/20/10 at 04:41:33:
This is what they found:
The best a human can do in terms of matching a computer's play, is:
Top move: 65%
Top 2 moves 75%
Top 3 moves 85%
Clearly this is not the maximal possible degree of matching, since you contradict this below. So what is it, an average, a median, something somebody pulled out of thin air? What is the population? What is the sample? What are the methods? We are simply in the dark.
Smyslov_Fan wrote on 09/20/10 at 04:41:33:
Dembo's stats were:
Top move: 73.3
Top 2 88.2
Top 3 93.5
Since the process is subject to error, the question is, given some reasonably specified statistical model and some estimate of the variability of human-machine matchup,
with what degree of confidence can it be said that Dembo's matchup was different from that expected? Somehow I doubt that sufficient rigor was applied to this question. If the entire analysis and all the data were reported on the internet, one could review it. In the mean time, scant credit can be given to its alleged results.
Smyslov_Fan wrote on 09/20/10 at 04:41:33:
But for those scoffing at correlations as proof, that's what statistics studies, correlations.
Don't make me laugh. Until I required quite recently, I made my living this way, you know?
Smyslov_Fan wrote on 09/20/10 at 04:41:33:
The likelihood of a human matching a computer 100% approaches 0, so their approach makes sense. Perhaps the tolerances are a bit low though.
What tolerances? What approach? We simply don't know enough to evaluate the method used.
Smyslov_Fan wrote on 09/20/10 at 04:41:33:
It's very likely that Dembo cheated.
Shame on you.
Smyslov_Fan wrote on 09/20/10 at 04:41:33:
But I don't know if it's a certainty. I'd have to run the numbers myself, which doesn't interest me much.
Run the numbers yourself???? With what population? With what data? With what model? If you had the expertise to "run the numbers," I expect you would have raised the questions yourself by now.
And
of course it's not certain, since at best, it's a statistical result!!
Finally let me say that only an idiot would want to match a computer all the time, since computers remain rather stupid about many positions. So the very notion that rate of matchup with some engine is likely to correlate with success is bogus.