Jonathan003 wrote on 10/25/20 at 13:29:10:

I admire your sense of adventure.

Jonathan003 wrote on 10/25/20 at 13:29:10:

I do detailed steps with screenshots for users at work, but then I am paid for it. Since you installed pgn-extract on Windows already, I know for a fact you can do the rest of the steps.

Jonathan003 wrote on 10/25/20 at 13:29:10:

Gawk is GNU awk. Awk is a pattern-matching language for dealing with text files, similar to grep and sed but more suitable for scripting. To be honest most people these days would prefer python or php or even perl. I use awk at work (for getting work done!) because they can stop me from installing the p's, but they can't stop me from installing gawk.

Gawk and pgn-extract are both command-line programs which you use by entering commands in a command interpreter -- cmd.exe on Windows. The command-line is very practical for text processing, but it's not visual like a GUI. It's more like walking around your house with a blindfold on, and you only navigate successfully because you already know where everything is.

• http://gnuwin32.sourceforge.net/packages
• Click on the "Gawk" link.
• Click on the "General Installation Instructions" link, which also includes usage instructions, and read up.
• Go back to the Gawk page.
• Probably you want the "Complete package, except sources" Setup option.
• Click on that to download the setup.exe.
• Ideally you would verify the download with the provided md5sum, but I'm not going to describe how to do that.
• Finally, follow the previous General Installation Instructions to install.

There is a big caveat to all of the above gawk installation steps. I have gawk working on a couple of Windows 10 machines, but I didn't use the setup method (it's exactly this that is blocked at my work), so I'm not really sure if it will succeed. The setup.exe on sourceforge was created for Windows Vista, and might not work on Windows 10. Or it might partially succeed but require tweaks after.

Anyway, fingers crossed, after installation open cmd.exe and type `gawk --version` (without the quotes), to verify that you can run it. With pgn-extract the command is `pgn-extract --version`.

Jonathan003 wrote on 10/25/20 at 13:29:10:

Yes, that's another way, assuming Unix means a Linux distribution. But it's another can of worms. Your Linux will come with awk but not with pgn-extract. To install pgn-extract via the package manager means you need to connect the VM to the network, which is not a trivial step. The other option is to build pgn-extract from source, which is even more fun.

Edited:

Jonathan003 wrote on 02/01/20 at 16:27:49:


Hmmm, I will give it a try. Sounds interesting

I use polyglot_tolerant for this.
I use the book merging options. And In Lucas chess there is an option to import all best moves from a bin book to a repertoire. And the repertoire can be exported to pgn.
If you want to know how I make a repertoire send me a private message.

Easy on unix or mac.
Only slightly less easy on windows. http://gnuwin32.sourceforge.net/packages.html

1. convert your repertoire.pgn(s) to repertoire.epd (e.g. use pgn-extract)
2. convert your new.pgn to new.epd
3. for each line in new.epd
   $ grep -F "fen" repertoire.epd

I wrote a couple of helper awk scripts to somewhat automate this task, and combined them in one attachment. You would have to separate them again before using. WIth these scripts, the exact sequence of commands to reproduce the above steps would be:

• $ pgn-extract -w200 --fencomments -orepertoire1.pgn repertoire.pgn
• $ gawk -f fencomments.awk repertoire1.pgn > repertoire1.epd
• $ pgn-extract -w200 --fencomments -onew1.pgn new.pgn
• $ gawk -f fencomments.awk new1.pgn > new1.epd
• $ gawk -f epdcompare.awk -v CMP=repertoire1.epd new1.epd > cmp.epd

With the following repertoire.pgn

[Event "rep game 1"]
[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 g6
6. f4 Bg7 7. e5 dxe5 8. fxe5 Ng4 9. Bb5+ Kf8 10. Ne6+
*

[Event "rep game 2"]
[Result "*"]

1. e4 e6 2. d4 d5 3. e5 c5 4. c3 Nc6 5. Nf3 Qb6
6. Bd3 cxd4 7. cxd4 Nxd4 8. Nxd4 Qxd4 9. Bb5+ Bd7 10. Bxd7+ Kxd7
11. Qxd4
* 

And the following new.pgn to compare it to:

[Event "new game 1"]
[Result "*"]

1. e4 e6 2. d4 d5 3. e5 c5 4. c3 Qb6 {N} 5. Nf3
5... Nc6 {transposition}
6. Bd3 cxd4 7. cxd4 Bd7 {N} 8. O-O Nxd4 9. Nxd4 Qxd4 10. Nc3 a6
* 

The output cmp.epd is below.

rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq e3;hmvc 0;fmvn 1;id 1;fk 1 2;fmvnmax 1;
rnbqkbnr/pppp1ppp/4p3/8/4P3/8/PPPP1PPP/RNBQKBNR w KQkq -;hmvc 0;fmvn 2;id 1;fk 2;fmvnmax 2;
rnbqkbnr/pppp1ppp/4p3/8/3PP3/8/PPP2PPP/RNBQKBNR b KQkq d3;hmvc 0;fmvn 2;id 1;fk 2;fmvnmax 2;
rnbqkbnr/ppp2ppp/4p3/3p4/3PP3/8/PPP2PPP/RNBQKBNR w KQkq d6;hmvc 0;fmvn 3;id 1;fk 2;fmvnmax 3;
rnbqkbnr/ppp2ppp/4p3/3pP3/3P4/8/PPP2PPP/RNBQKBNR b KQkq -;hmvc 0;fmvn 3;id 1;fk 2;fmvnmax 3;
rnbqkbnr/pp3ppp/4p3/2ppP3/3P4/8/PPP2PPP/RNBQKBNR w KQkq c6;hmvc 0;fmvn 4;id 1;fk 2;fmvnmax 4;
rnbqkbnr/pp3ppp/4p3/2ppP3/3P4/2P5/PP3PPP/RNBQKBNR b KQkq -;hmvc 0;fmvn 4;id 1;fk 2;fmvnmax 4;
rnb1kbnr/pp3ppp/1q2p3/2ppP3/3P4/2P5/PP3PPP/RNBQKBNR w KQkq -;hmvc 1;fmvn 5;id 1;fk 0;fmvnmax 0;
rnb1kbnr/pp3ppp/1q2p3/2ppP3/3P4/2P2N2/PP3PPP/RNBQKB1R b KQkq -;hmvc 2;fmvn 5;id 1;fk 0;fmvnmax 0;
r1b1kbnr/pp3ppp/1qn1p3/2ppP3/3P4/2P2N2/PP3PPP/RNBQKB1R w KQkq -;hmvc 3;fmvn 6;id 1;fk 2;fmvnmax 6;
r1b1kbnr/pp3ppp/1qn1p3/2ppP3/3P4/2PB1N2/PP3PPP/RNBQK2R b KQkq -;hmvc 4;fmvn 6;id 1;fk 2;fmvnmax 6;
r1b1kbnr/pp3ppp/1qn1p3/3pP3/3p4/2PB1N2/PP3PPP/RNBQK2R w KQkq -;hmvc 0;fmvn 7;id 1;fk 2;fmvnmax 7;
r1b1kbnr/pp3ppp/1qn1p3/3pP3/3P4/3B1N2/PP3PPP/RNBQK2R b KQkq -;hmvc 0;fmvn 7;id 1;fk 2;fmvnmax 7;
r3kbnr/pp1b1ppp/1qn1p3/3pP3/3P4/3B1N2/PP3PPP/RNBQK2R w KQkq -;hmvc 1;fmvn 8;id 1;fk 0;fmvnmax 0;
r3kbnr/pp1b1ppp/1qn1p3/3pP3/3P4/3B1N2/PP3PPP/RNBQ1RK1 b kq -;hmvc 2;fmvn 8;id 1;fk 0;fmvnmax 0;
r3kbnr/pp1b1ppp/1q2p3/3pP3/3n4/3B1N2/PP3PPP/RNBQ1RK1 w kq -;hmvc 0;fmvn 9;id 1;fk 0;fmvnmax 0;
r3kbnr/pp1b1ppp/1q2p3/3pP3/3N4/3B4/PP3PPP/RNBQ1RK1 b kq -;hmvc 0;fmvn 9;id 1;fk 0;fmvnmax 0;
r3kbnr/pp1b1ppp/4p3/3pP3/3q4/3B4/PP3PPP/RNBQ1RK1 w kq -;hmvc 0;fmvn 10;id 1;fk 0;fmvnmax 0;
r3kbnr/pp1b1ppp/4p3/3pP3/3q4/2NB4/PP3PPP/R1BQ1RK1 b kq -;hmvc 1;fmvn 10;id 1;fk 0;fmvnmax 0;
r3kbnr/1p1b1ppp/p3p3/3pP3/3q4/2NB4/PP3PPP/R1BQ1RK1 w kq -;hmvc 0;fmvn 11;id 1;fk 0;fmvnmax 0; 

Note that it matches by position and not by move. The two "N" moves result in positions with "fk 0", and the transposition is also caught.

• id - game number in the new.pgn
• fk - array of matching game numbers in repertoire.pgn
  Note that the first position matched both games in the repertoire.
• fmvn - full move number in new.pgn.
  Note that for black moves this is "off by one" from the last move played.
• fmvnmax - the largest "repertoire" full move number where the id-game in new.pgn matched any fk-game in repertoire.pgn

I could have output the sm opcode as well (supplied move), and for openings work this would have made it a little easier to scan the final epd.  I didn't do that because I'm actually planning to use epdcompare with endgame epd-files (not pgn-files). As it stands, you will probably have to open both new.pgn and cmp.epd at the same time and scan two files together. But since the OP asked about ChessBase, I assume the plan is: once the interesting new games are identified, merge by dragging them onto the appropriate game in the repertoire.

brabo wrote on 06/07/19 at 12:52:16:

Thanks for this interesting answer. Fritz is cheaper than chessbase so i'll probably buy it. The functionality you describe seems very interesting and as you said close to my need but I will realize  and perfectly understand it as soon as I will experiment this functionality.
Thanks again !

Hello everyone,

Is there a tool - if possible a freeware - 
that from two repertoire databases for each color  (in Chessbase or PGN format)
filters all games of another database or PGN (the target base) that are in my repertoire.

Ideally I'd like to know from what move me or my opponent are "out" of the repertoire.
I have never managed to do that with Chessbase 8 (yes I know I date).
And I hesitate to buy a newer version of Chessbase.
Is there a version of Chessbase that manages to do it?

Thank you in advance for your help.