@EricTheRed Well, Khmelnitsky himself lists all these considerations and more in his cautionary text. Even USCF ratings mean different things in different contexts--underrated kids, sandbagging World Open players, players in small rural pools improving or aging and declining as a whole, etc., etc. And of course, what we play are games and tournaments, and as Kramnik says, "in the end there is the result." Rating is just another kind of result, but it inspires a kind of fatalism because we understandably use it as a placeholder for strength.

One thing I like about this book is that, paradoxically, its effect opposes the general effect of ratings. It tells you more than one number could tell you and enlists the numbers it uses into the service of self-improvement and qualitative sensitivity.

@knowledge vs. performance--in math, everyone's understanding (=knowledge here, let's say) is above his performance, because everyone drops negative signs and so on by accident sometimes, and these occurrences are not attributed to lack of understanding or knowledge. What would the reverse mean? That you were to get problems right by accident? But that would just be a compounding of careless mistakes; no one would attribute accidental right answers to understanding...so everyone's understanding is superior to his performance, but to varying degrees. In chess, we say that players have not performed as well as they could have, but though someone may win a match or tournament unexpectedly, we never say that over the long term players perform beyond their understanding (of, let's say, tactics). This is not a statistical mistake or lame excuse, but an aspect of the meaning of the concepts of understanding and performance.

Maybe Khmelnitsky's test doesn't measure knowledge or understanding or potential, etc., but just different kinds of performance (in the study, without an opponent in front of you, etc.). But I think we can trust him to have  mathematically eliminated systematic overestimation of rating within his large sample of reader-respondents--doing that is how he got these numbers! Now maybe his sample of respondents wasn't representative, but ours is much less so--a mere four or five data points from chess-book junkies.

Remember too that some are comparing FIDE to USCF, and that people's testing conditions probably vary widely.  For instance Monocle seems to have zoomed through half of the test in just a few days, whereas I solved only two positions a day when I took it.  I'm sure that there are many other variables in play as well.

I don't think that it's all that useful in estimating absolute playing strength, but for relative areas of strength and weakness I think that it's very good.

The book overestimated my rating by a little over a hundred points, but since it said that Tactics were my weakest area, I think that's probably fair. I've lost many games over the years from positions that I should have won by missing a tactical shot.

I think that the relative breakdown of my strengths and weaknesses was dead on, and I gained about 200 rating points in the years after tailoring my study plan around the results, so I can't complain. No test like this is perfect, but I can't think of a better one (or even a very comparable one).

Only six questions on openings... Is this test rigged to make everyone de-emphasize opening study? What (in general) would you have to miss in order to do poorly on the opening test?

And how did the book break down your areas of strengths and weaknesses?

Monocle wrote on 05/12/18 at 18:13:41:


I just checked the book, and that is not the case at all. On the contrary: if you look at the whole book, Khmelnitsky, professional actuary that he is, seems to have properly rendered the responses unpredictable from context. In many problems, despite temptations, there is no tactic that really works, and the proper evaluation is merely "White is better" or "Black is slightly better," etc. There are also answers to the effect that even though such-and-such works, White cannot be winning here, etc.--i.e., real end-node evaluation skill is tested in addition to finding moves (where else in chess literature is this crucial skill tested?)

Ubiquitous calculation? Yes--Khmelnitsky's books strike me as comparable to Yusupov's, where even in the strategic chapters you have to blend calculation and finding moves with strategic sense and goal-setting. Chess is like that. But if one does call it all calculation, then one should distinguish between strategic calculation, endgame calculation, attacking calculation, defensive calculation, etc.; for the questions do not all vary together perfectly, and Khmelnitsky effectively categorizes their emphases.

The Fischer book is a general one, like the first test book, not like the tactics book. Some of the positions are from simuls, etc., while others are from relatively famous games (but not necessarily from the most famous moments of those games).

I didn't feel the occasional problems in the scoring guidelines to be very pervasive. As an examination of the comments on ChessTempo will show, it really is hard to anticipate what various readers will miss; considering the difficulty of the task, I think Khmelnitsky did pretty well.

I did the original Exam book maybe 13 years ago and the tactics book about a decade ago, and they were very helpful in confirming what to work on at the time. I studied the recommended areas and improved. These books are gems if you want to verify what to study!

So, once every few decades someone does some statistical analysis on this sort of thing?  I haven't looked rigorously at Khmelnitsky's methods, but in this online era it's much easier to compile data, and he's still the only modern author that I'm aware of to have made an attempt at this kind of thing.  And given the anecdotal accounts I've heard firsthand (and experienced myself), I'd give his work some benefit of the doubt before being too critical.