g-dog, why did you add 5%? From a mathematical standpoint, why did you choose that particular number?

The methods used by chess.com staff to determine engine-using cheaters isn't known. They may well have a Top 3 matching kind of analysis going on, and then something else too. They also have an IM who leads their detection efforts; he may eyeball games for non-human seeming moves.

The Dembo analysis results you see come from members of chess.com, not staff. 

I was one of those who ran her games using Batch Analyzer (BA), the same software RHP games mods use for their cheat detection. The results of her 20-game batch exceeded the human thresholds+5% that I have derived after analyzing in total >13000 moves from WC otb matches, a few modern super GM tournament performances, and importantly, the top finishers in the 5th-11th ICCF WC Finals (>7700 moves)

For all benchmark games and all games of any suspected player, the same hardware, software, and BA settings are used.

All benchmark games have their database moves screened out from analysis using MegaBase 2010 and MegaCorr 4, with the DBs being rolled back to the time of the benchmark games.

My human thresholds based on the benchmark games are 61/78/85%. Add 5% to account for error and 66/83/90%=blatant engine use.

Her results:

{ YelenaDembo (Games: 20) }
{ Top 1 Match: 493/694 ( 71.0% )   
{ Top 2 Match: 609/694 ( 87.8% )   
{ Top 3 Match: 644/694 ( 92.8% ) 
{ Top 4 Match: 660/694 ( 95.1% ) 

Database moves were determined using MegaBase 2010 and MegaCorr 4 and screened out from analysis.

3 other BA users had the same results with the same games using varying engines and hardware. She exceeded the heritage human thresholds +5% of 65/80/90% in all cases.

The games were the 18 most recent vs. 2200+ opposition, plus 2 more most recent vs near 2200 to reach the 20 game batch size.

Insofar as engine use can be determined by a Top 3 analysis, she has been caught.

She apparently used engines freely during her games. She's a cheater and I wouldn't dare bet against it.

Is she 100% guilty? I wouldn't say that, but we are pretty close in my opinion.

There are, after all, other ways the data can be held up to the light.

Smyslov_Fan wrote on 09/20/10 at 17:41:50:


If you look at chess.com's website, they say they keep these detection routines secret.  So there is no possibility of reviewing them.  If chess.com would ever post their detailed methods and data, I would have some interest in reviewing their results.

I didn't attack you or patronize you so far as I am aware, other than to say "Shame on you" for saying that Dembo was "probably guilty of cheating."  That struck me as a particularly egregious remark.  Chess.com's secret mumbo-jumbo has impressed you for some reason, I have no idea why.  I take exception when secret methods are employed to label people "known cheaters."  It's the labeling much more than the banning that bothers me.  Who gives a frig who is banned from some bullshit chess site?  But when they post a list of people and say these are "known cheaters," that goes too far.

How much you and I agree and how much we disagree on these various points is something that others will have to discern, though I fail to see how anyone can read what I read and take it as agreement.

Yes, you're indeed correct. We also had a database of strong modern day OTB players. It's unfortunate that there's not a lot of information on really strong players that play CC without a computer. We did have match up rates for the team, which was good enough. All clearly under the limits.

No, I can't do more than pretty much what they did. I think we had some fancier stats stuff, but it's pretty much all the same. I can tell you that those numbers would be incredibly high. 

The best control group to match her stats to is not just the best OTB players in the world (since it's OTB and not correspondence - you can expect strong OTB players to play stronger than the best in CC because of the use of books, databases, and much longer time controls), but the strongest CC tournaments the worlds seen before computers were prominent or useful, and they're doing that. Using a statistical comparison is useful there, but not totally damning, because as Mark said, is not totally conclusive. The only thing you can say is that you're 95 or 98 percent confident that the means of the match up populations are different. That's pretty damn good, but not totally cut and dry. 

The options that we had on RedHotPawn were some more interesting statistical ideas, and some ideas that won't work here for a certain reason. I bet chess.com has similar methods and is using those also. 

One that is fairly obvious is to find awkward, computer like moves that don't make any sense from a human perspective. Yes, it's subjective, but it can be helpful in obvious cases - where human logic can't pinpoint why this player would play this move by any stretch, yet the engines and this player pick the move. Once again, the weight here is very low, but I was confident pulling the trigger when it was damning stats, plus strong players looking at the games, plus the other ideas along with it.

Markovich attacks me for my pointing out that chess.com seems to be inconsistent. His entire tone is patronizing, and yet we agree on many points, especially the uses of statistics.

Markovich's point-by-point "refutation" of my earlier post is actually mostly agreeing with what I said. I gave chess.com's method and pointed out flaws. The main place we disagreed is whether, based on that scant evidence, it is likely that WGM Dembo cheated. I think there is some merit to chess.com's method, and that it is likely she cheated. How likely? I don't know.

I would like to find out more about how chess.com has come to the conclusion that any very strong engine could be used to replicate the data.

I would also like to see if by doing so, every engine and no humans are caught. This would mean testing their hypothesis and publishing their results, preferrably in a reviewed journal.

I do believe that it is possible by using statistics to separate humans from engines. I do find chess.com's attempts flawed but interesting.

I do suspect that WGM Dembo was guilty of cheating. 

Markovich, you yourself said that chess.com's method is subject to error. That error can be measured. Are you willing to do the work to find out what that level of error is and whether WGM Dembo's numbers match that?

In the U.S., to successfully sue for libel, you must prove both that the statements published about you are false and that you have suffered damages.  In some places, I understand you also have to prove malice.

My limited understanding is that U.K. libel laws are much more generous to plaintiffs.