brabo wrote on 06/07/19 at 12:52:16:


Scid also has the same function, but it can't be found in the menu. You have to open an analysis engine. At the bottom toolbar hover over the notepad icon and the tooltip says Annotate.... Click on it to open the Annotate window. There's all kinds of parameters you can change, the most important ones are

• Time per move in sec.
  - The smallest time you can set is 0.1 sec.
• Use book
  - Here you can select a polyglot bin book from the /Books folder.
• From actual game to game
  - That's the batch feature, "from actual game" is the one you have open, when done with that one it will go on to the next game in the current filter and stop at the "to game" number you enter here.

It's perfect if you want to analyze the games with an engine, but if you only want to check them against a book 0.1 sec is too long. 20 opening moves for each side x 0.1 second per move is 4 seconds per game and say 5000 games in Twic gives 5.6 hours for the batch.

Of course 0.1 sec per move is not serious analysis. As you say, this kind of tool  is for analyzing own games which are in the hundreds not thousands.

Jonathan003 wrote on 11/14/20 at 21:08:30:

Yes! 
For each move in the new pgn, it steps through your repertoire pgn and finds the place where it matches. Look for the far right column having a "0". If there is a long run of "0" on the right, that represents a "novelty" for your repertoire. A short run of "0" represents a transposition.

rnbqkbnr/pp3ppp/4p3/2ppP3/3P4/2P5/PP3PPP/RNBQKBNR b KQkq -;hmvc 0;fmvn 4;id 1;fk 2;fmvnmax 4;
rnb1kbnr/pp3ppp/1q2p3/2ppP3/3P4/2P5/PP3PPP/RNBQKBNR w KQkq -;hmvc 1;fmvn 5;id 1;fk 0;fmvnmax 0;
rnb1kbnr/pp3ppp/1q2p3/2ppP3/3P4/2P2N2/PP3PPP/RNBQKB1R b KQkq -;hmvc 2;fmvn 5;id 1;fk 0;fmvnmax 0;
r1b1kbnr/pp3ppp/1qn1p3/2ppP3/3P4/2P2N2/PP3PPP/RNBQKB1R w KQkq -;hmvc 3;fmvn 6;id 1;fk 2;fmvnmax 6; 

Opcode translations

• hmvc - half-move clock, number of ply since the last pawn move or capture in the new game
• fmvn - full move number, number of moves so far in the new game
• id - identity, game number in the new pgn
• fk - foreign key, game number(s) in the repertoire pgn
• fmvnmax - full move number max, (largest) number of moves so far in the repertoire game(s)

In the sample output above

• fmvn 4;id 1;fk 2;fmvnmax 4; - new game 1 move 4 matches repertoire game 2 move 4
• fmvn 5;id 1;fk 0;fmvnmax 0; - new game 1 move 5 does not match the repertoire 
• fmvn 5;id 1;fk 0;fmvnmax 0; - new game 1 move 5 does not match the repertoire 
• fmvn 6;id 1;fk 2;fmvnmax 6; - new game 1 move 6 matches repertoire game 2 move 6

So on move 4 black played a "new" move (reached a position not in the repertoire), then on move 5 white played a "new" move (in reply to black's "new" move), then on move 5 black played a "non-new" move. Therefore black's "new" move on move 4 was a simple transposition. Whether the transposition is important for your repertoire is up to you. The simple idea is you open the new pgn, scroll down to game number id, scroll across to move number fmvn (having fmvnmax = 0), then use your brain (or engine) to decide whether or not to import that game to your repertoire. Software can only automate things up to a certain point.

Jonathan003 wrote on 11/14/20 at 21:08:30:

You do realize that all the games in the new database will have a move you don't already have a reply for. Unless one of them is an exact duplicate of a repertoire game that will always be the case. There is no way around the fact that you will have to look at every game. But it's not hopeless. You can use the fmvn/fmvnmax to reject some of the "novelties". Most people do an arbitrary cut-off at move 15 or 20. Only in certain critical lines would you go much beyond that.

Jonathan003 wrote on 11/14/20 at 21:08:30:

I'm not sure about the polyglot book idea. Those are designed for engines. You might find some use for practicing against an engine, but otherwise I would look at different avenues. There are good reasons for tracking more than one move for your side in a repertoire, but there's no way to tell the polyglot book what that reason is. To the engine, every move in a polyglot book is the same.

Jonathan003 wrote on 11/14/20 at 21:08:30:

Exactly. Most repertoire tools have some facility for importing pgn, but when it comes to exporting pgn, the situation is, shall we say, lossy. As an example, I think Chess Openings Wizard (Bookup) is a great piece of software, as long as you only intend to use COW exclusively. But I use lots of other software, so any effort I put into COW is wasted when I switch to another tool.

I haven't used Lucas Chess, so I don't know how it handles these things. You say it can "import the best moves", but I rather doubt that. Isn't it going to import "all" the moves? I recommend you search on brabo's blog for "green moves" where he uses a ChessBase tree. The idea is that a repertoire should show best moves and second-best moves, and a way to know which is which. After all, theory changes and today's best move can become tomorrow's second-best. Or you may decide second-best is less work while still being good enough. You should also look at Scid's "tree mask" feature -- very cool. I don't like the ChessBase tree at all, and I do like the Chess Assistant tree very much. But all these tools can only give ideas for how to handle your own repertoire. 

I strongly believe that keeping the repertoire in pgn in a form that allows for easy import to your repertoire tool of choice is, in the long run, the only way to avoid significant pain. By all means import to the tool, but consider it to be read-only, and make any changes to the repertoire in the original pgn. Otherwise, at some point, you are going to end up with information you can't export. The farther in the future that point is, the more information you will have lost.

Jonathan003 wrote on 11/03/20 at 19:06:49:

Not happening.

Ok I have downloaded the arhive.txt file
I really don't understand what to do.
Can you please make some video capture. From beginning to end so I understand wat to do on a windows 10 64 bit system.
Maybe also for others ho have little experience with command line tools and want to try this.
I think a video capture with mouse capture and maybe audio is the easiest way to explain something.

Jonathan003 wrote on 10/29/20 at 18:02:46:

Well actually there is dedicated software, but the OP mentioned freeware. I haven't looked closely at Scid, but maybe it has this capability. If you don't mind paying then CPT was mentioned earlier, and ChessBase also has a repertoire and "find novelty" feature.

Jonathan003 wrote on 10/29/20 at 18:02:46:

I'm not a programmer, just a business analyst, but I also spend time trying to learn some programming skills. It's an attitude not just towards computers but towards everything. You don't need to be a chef in order to cook, an electrician to do household wiring, an auto mechanic to repair your vehicle, etc., etc. (Well that last one is changing as the automobile evolves into a computer on wheels, but the simplest repairs can still be do-it-yourself.)

Quote:

There is some argument for sticking to what you do best and leaving everything else to an expert -- provided what you do best pays well enough that you can afford the expert's pay scale. Personal chef, anyone? If you want to go that route, ChessBase is the program for you. It has a feature to do almost anything you can think of (although it won't do it quite the way you want!). Its feature set makes it the MS Excel of the chess world, complete with the lock-in to Windows (actually not true of Excel anymore), the upgrade pains, and being expensive. I'm not against it, having paid for ChessBase and competing products many, many times over the decades. I'm just listing the pros and cons.

an ordinary chessplayer wrote on 10/30/20 at 05:46:29:

Where can I find the files 'fencomments.awk'
and 'epdcompare.awk'?
Should I find them somewhere on my computer or do I have to make them myself?

I managed to update the scripts I provided before. In fact I did do the earlier work on Linux, and "forgot" to check whether they would work on Windows, which of course they did not. They didn't run on macOS either, but I only got the Mac recently so did not have an opportunity to test that before. It was quite a chore getting these to work on Windows, I won't bore you with the details.

Here is my command session on Windows 10.

$ dir /b
epdcompare.awk
fencomments.awk
new.pgn
rep.pgn

$ awk --version
GNU Awk 3.1.3

$ pgn-extract --version
pgn-extract v19-04

$ pgn-extract -w200 --fencomments -orep1.pgn rep.pgn
Processing rep.pgn
- - - Edited game W3Q689U4ZN 2019.06.08
- - - Edited game W3Q689U4ZN 2019.06.08
2 games matched out of 2.

$ awk --re-interval -f fencomments.awk rep1.pgn > rep1.epd

$ pgn-extract -w200 --fencomments -onew1.pgn new.pgn
Processing new.pgn
- - - Edited game W3Q689U4ZN 2019.06.08
1 game matched out of 1.

$ awk --re-interval -f fencomments.awk new1.pgn > new1.epd

$ awk --re-interval -f epdcompare.awk CMP=rep1.epd new1.epd > cmp.epd

$
 

And attached is a single text-file "archive" of all the files from that session. To run on Windows, use the scripts as-is and the commands above. To run on Linux or macOS, comment the Windows-specific grepcmd on line 23, uncomment the Unixish grepcmd on line 22, and leave off the `--re-interval` switch from the command line.