I read the paper. The authors are quite forthright in outlining their logic:
Quote:The central methodological assumption of this paper is that the player who first departs from a theoretical opening line (player of interest, PI) knows the theory up to the point of rupture. For example, assume that in the position after 1.e4 e5 2.Nf3 Nc6, white plays 3.a4. Assuming that 3.a4 is a novelty, knowledge is 4 ply deep. [...] Note that the goal of the present paper is to estimate the amount of rote opening knowledge, rather than to evaluate the quality of the novelty and to assess problem-solving skills. Thus, there was no point in discarding part of the data (i.e., bad novelties).
Bad novelties, it seems to me, might indicate that the preceding moves
were memorized, as the player erred as soon as he was out of book. At any rate, the authors support their assumption in the following way:
Quote:The probability of playing a sequence of theoretical moves by each side randomly choosing a legal move or randomly choosing a master-game-like (mgl) move (see below) is negligible. Based on the estimates for legal moves (n = 32.3) and mgl moves (n = 1.76) given in [1], [14], the corresponding probabilities for Masters are (1/32.3)18.01 = 6.596×10−28, and (1/1.76)18.01 = 3.787×10−5, respectively.
The master-game-like move idea would seem to be the authors' best chance to defend themselves. But this estimate turns out to be a simple mathematical assumption, namely that there are three master-like alternatives at each ply. Thus at move 10 there must be 3^20 or about 3.4 billion "master-like" sequences. The theory in a 3.4-million game database would then be 50 times smaller than that (3.4 billion/3.4 million, divided by 20 ply) even if every game were different from every other at move 10. As it is, of course, many, many games share the same path to move 10, so the database is many more times smaller. The authors conclude that unless good players were following memorized theory they would be for the most part choosing "master-like" moves that were
not in the database.
So there you have it. The whole study is founded on assumptions that each of us knows from personal experience to be false. In almost every game we play, we have memorized x moves of theory, but on analyzing our game it turns out we have been following previous games until move x+a, where a is often quite large.
The authors do consider the possibility that their assumption is wrong:
Quote:it is possible that some players played theoretical moves without knowing it, just by applying general heuristics. However, as suggested by the estimates provided earlier about the likelihood of finding a theoretical sequence by chance (e.g., 3.787×10−5 by sampling from master-game like moves), we do not expect this effect to be large, even if we cannot rule it out completely. Regularly finding theoretical moves without prior knowledge would imply that players play near perfectly in complicated situations, but we know that players of similar skill levels commit multiple errors during a game.
Again, we all know from experience that this is false. The authors seem to picture every opening moment as being as sharp and unclear as a middlegame from the Botvinnik Semi-Slav. It is not necessary to play incredibly well to reproduce theory "from applying general heuristics." A lot of theory is just obvious. A lot more is natural. If a 4-move general liquidation of rooks on the only open file occurs in theory, the players must have memorized those 8 ply, because their play was near-perfect in a complicated situation! And the player who follows the Back side of the Milner-Barry gambit without knowing it after 3.e5 c5 and wins the d-pawn after Bd3 must be a genius.
The authors incorrectly employ a statistical test called an analysis of covariance (ANCOVA):
Quote:An ANCOVA was carried out on opening knowledge with skill as the independent variable and length, color and relative skill as covariates. As predicted, the PI's skill level significantly affected opening knowledge, F(1, 76555) = 822.11, p<.01, MSE = 43.01, showing that opening knowledge varies as a function of skill. This is a crucial result showing that chess players accumulate static knowledge that guides them in the first phase of the game.
But this test must not under any circumstances be used when the independent variable could affect the covariate! So not only the authors' interpretations, but also their mathematics, depend on the incorrect assumption that ELO strength cannot affect the rate of playing book moves.
Then the authors move from correlation to causation, as Stefan suggests, making another scientific howler without apparent embarrassment.
Quote:In general, our results add further support to [a] view emphasizing the role of declarative knowledge in high levels of expertise; indeed, they support the importance of monochrestic and rote knowledge.
Would
you be afraid of a tactically and strategically weaker player who is booked?
Pages: [1] 
Extreme case of opening obsession (Read 25180 times)
