brabo wrote on 06/07/19 at 12:52:16:


Scid also has the same function, but it can't be found in the menu. You have to open an analysis engine. At the bottom toolbar hover over the notepad icon and the tooltip says Annotate.... Click on it to open the Annotate window. There's all kinds of parameters you can change, the most important ones are

• Time per move in sec.
  - The smallest time you can set is 0.1 sec.
• Use book
  - Here you can select a polyglot bin book from the /Books folder.
• From actual game to game
  - That's the batch feature, "from actual game" is the one you have open, when done with that one it will go on to the next game in the current filter and stop at the "to game" number you enter here.

It's perfect if you want to analyze the games with an engine, but if you only want to check them against a book 0.1 sec is too long. 20 opening moves for each side x 0.1 second per move is 4 seconds per game and say 5000 games in Twic gives 5.6 hours for the batch.

Of course 0.1 sec per move is not serious analysis. As you say, this kind of tool  is for analyzing own games which are in the hundreds not thousands.

Jonathan003 wrote on 11/14/20 at 21:08:30:

Yes! 
For each move in the new pgn, it steps through your repertoire pgn and finds the place where it matches. Look for the far right column having a "0". If there is a long run of "0" on the right, that represents a "novelty" for your repertoire. A short run of "0" represents a transposition.

rnbqkbnr/pp3ppp/4p3/2ppP3/3P4/2P5/PP3PPP/RNBQKBNR b KQkq -;hmvc 0;fmvn 4;id 1;fk 2;fmvnmax 4;
rnb1kbnr/pp3ppp/1q2p3/2ppP3/3P4/2P5/PP3PPP/RNBQKBNR w KQkq -;hmvc 1;fmvn 5;id 1;fk 0;fmvnmax 0;
rnb1kbnr/pp3ppp/1q2p3/2ppP3/3P4/2P2N2/PP3PPP/RNBQKB1R b KQkq -;hmvc 2;fmvn 5;id 1;fk 0;fmvnmax 0;
r1b1kbnr/pp3ppp/1qn1p3/2ppP3/3P4/2P2N2/PP3PPP/RNBQKB1R w KQkq -;hmvc 3;fmvn 6;id 1;fk 2;fmvnmax 6; 

Opcode translations

• hmvc - half-move clock, number of ply since the last pawn move or capture in the new game
• fmvn - full move number, number of moves so far in the new game
• id - identity, game number in the new pgn
• fk - foreign key, game number(s) in the repertoire pgn
• fmvnmax - full move number max, (largest) number of moves so far in the repertoire game(s)

In the sample output above

• fmvn 4;id 1;fk 2;fmvnmax 4; - new game 1 move 4 matches repertoire game 2 move 4
• fmvn 5;id 1;fk 0;fmvnmax 0; - new game 1 move 5 does not match the repertoire 
• fmvn 5;id 1;fk 0;fmvnmax 0; - new game 1 move 5 does not match the repertoire 
• fmvn 6;id 1;fk 2;fmvnmax 6; - new game 1 move 6 matches repertoire game 2 move 6

So on move 4 black played a "new" move (reached a position not in the repertoire), then on move 5 white played a "new" move (in reply to black's "new" move), then on move 5 black played a "non-new" move. Therefore black's "new" move on move 4 was a simple transposition. Whether the transposition is important for your repertoire is up to you. The simple idea is you open the new pgn, scroll down to game number id, scroll across to move number fmvn (having fmvnmax = 0), then use your brain (or engine) to decide whether or not to import that game to your repertoire. Software can only automate things up to a certain point.

Jonathan003 wrote on 11/14/20 at 21:08:30:

You do realize that all the games in the new database will have a move you don't already have a reply for. Unless one of them is an exact duplicate of a repertoire game that will always be the case. There is no way around the fact that you will have to look at every game. But it's not hopeless. You can use the fmvn/fmvnmax to reject some of the "novelties". Most people do an arbitrary cut-off at move 15 or 20. Only in certain critical lines would you go much beyond that.

Jonathan003 wrote on 11/14/20 at 21:08:30:

I'm not sure about the polyglot book idea. Those are designed for engines. You might find some use for practicing against an engine, but otherwise I would look at different avenues. There are good reasons for tracking more than one move for your side in a repertoire, but there's no way to tell the polyglot book what that reason is. To the engine, every move in a polyglot book is the same.

Jonathan003 wrote on 11/14/20 at 21:08:30:

Exactly. Most repertoire tools have some facility for importing pgn, but when it comes to exporting pgn, the situation is, shall we say, lossy. As an example, I think Chess Openings Wizard (Bookup) is a great piece of software, as long as you only intend to use COW exclusively. But I use lots of other software, so any effort I put into COW is wasted when I switch to another tool.

I haven't used Lucas Chess, so I don't know how it handles these things. You say it can "import the best moves", but I rather doubt that. Isn't it going to import "all" the moves? I recommend you search on brabo's blog for "green moves" where he uses a ChessBase tree. The idea is that a repertoire should show best moves and second-best moves, and a way to know which is which. After all, theory changes and today's best move can become tomorrow's second-best. Or you may decide second-best is less work while still being good enough. You should also look at Scid's "tree mask" feature -- very cool. I don't like the ChessBase tree at all, and I do like the Chess Assistant tree very much. But all these tools can only give ideas for how to handle your own repertoire. 

I strongly believe that keeping the repertoire in pgn in a form that allows for easy import to your repertoire tool of choice is, in the long run, the only way to avoid significant pain. By all means import to the tool, but consider it to be read-only, and make any changes to the repertoire in the original pgn. Otherwise, at some point, you are going to end up with information you can't export. The farther in the future that point is, the more information you will have lost.

Jonathan003 wrote on 11/03/20 at 19:06:49:

Not happening.

Ok I have downloaded the arhive.txt file
I really don't understand what to do.
Can you please make some video capture. From beginning to end so I understand wat to do on a windows 10 64 bit system.
Maybe also for others ho have little experience with command line tools and want to try this.
I think a video capture with mouse capture and maybe audio is the easiest way to explain something.

Jonathan003 wrote on 10/29/20 at 18:02:46:

Well actually there is dedicated software, but the OP mentioned freeware. I haven't looked closely at Scid, but maybe it has this capability. If you don't mind paying then CPT was mentioned earlier, and ChessBase also has a repertoire and "find novelty" feature.

Jonathan003 wrote on 10/29/20 at 18:02:46:

I'm not a programmer, just a business analyst, but I also spend time trying to learn some programming skills. It's an attitude not just towards computers but towards everything. You don't need to be a chef in order to cook, an electrician to do household wiring, an auto mechanic to repair your vehicle, etc., etc. (Well that last one is changing as the automobile evolves into a computer on wheels, but the simplest repairs can still be do-it-yourself.)

Quote:

There is some argument for sticking to what you do best and leaving everything else to an expert -- provided what you do best pays well enough that you can afford the expert's pay scale. Personal chef, anyone? If you want to go that route, ChessBase is the program for you. It has a feature to do almost anything you can think of (although it won't do it quite the way you want!). Its feature set makes it the MS Excel of the chess world, complete with the lock-in to Windows (actually not true of Excel anymore), the upgrade pains, and being expensive. I'm not against it, having paid for ChessBase and competing products many, many times over the decades. I'm just listing the pros and cons.

an ordinary chessplayer wrote on 10/30/20 at 05:46:29:

Where can I find the files 'fencomments.awk'
and 'epdcompare.awk'?
Should I find them somewhere on my computer or do I have to make them myself?

I managed to update the scripts I provided before. In fact I did do the earlier work on Linux, and "forgot" to check whether they would work on Windows, which of course they did not. They didn't run on macOS either, but I only got the Mac recently so did not have an opportunity to test that before. It was quite a chore getting these to work on Windows, I won't bore you with the details.

Here is my command session on Windows 10.

$ dir /b
epdcompare.awk
fencomments.awk
new.pgn
rep.pgn

$ awk --version
GNU Awk 3.1.3

$ pgn-extract --version
pgn-extract v19-04

$ pgn-extract -w200 --fencomments -orep1.pgn rep.pgn
Processing rep.pgn
- - - Edited game W3Q689U4ZN 2019.06.08
- - - Edited game W3Q689U4ZN 2019.06.08
2 games matched out of 2.

$ awk --re-interval -f fencomments.awk rep1.pgn > rep1.epd

$ pgn-extract -w200 --fencomments -onew1.pgn new.pgn
Processing new.pgn
- - - Edited game W3Q689U4ZN 2019.06.08
1 game matched out of 1.

$ awk --re-interval -f fencomments.awk new1.pgn > new1.epd

$ awk --re-interval -f epdcompare.awk CMP=rep1.epd new1.epd > cmp.epd

$
 

And attached is a single text-file "archive" of all the files from that session. To run on Windows, use the scripts as-is and the commands above. To run on Linux or macOS, comment the Windows-specific grepcmd on line 23, uncomment the Unixish grepcmd on line 22, and leave off the `--re-interval` switch from the command line.

I can't post a weblink here, but if you search in YouTube for: 'Outil gratuit de recherche dans une base pgn'
You find a video in French explaining how to build repertoire with pgn-extract.
I don't understand French and I think you need a Unix system to do what he discribes.

Jonathan003 wrote on 10/29/20 at 13:22:47:

I learned something today. 

You're almost there. You just don't have enough experience with the command line to help it find the files it needs.

We'll get it working, but not right now. I want to get back to you on this. I thought I should test my own steps, but here at work gawk is choking on the FEN regular expression in my scripts. It's a little strange, because I did use these exact scripts, but probably did it on linux with a different version of gawk. And while looking at it I noticed that I also used grep but didn't mention that.

So let me rewrite the script to not use grep, and get the regular expression working in all versions of gawk, and test that it all works on windows/linux/macos. I can't do all that right now, but should be able to do it this weekend.

Jonathan003 wrote on 10/25/20 at 13:29:10:

I admire your sense of adventure.

Jonathan003 wrote on 10/25/20 at 13:29:10:

I do detailed steps with screenshots for users at work, but then I am paid for it. Since you installed pgn-extract on Windows already, I know for a fact you can do the rest of the steps. 

Jonathan003 wrote on 10/25/20 at 13:29:10:

Gawk is GNU awk. Awk is a pattern-matching language for dealing with text files, similar to grep and sed but more suitable for scripting. To be honest most people these days would prefer python or php or even perl. I use awk at work (for getting work done!) because they can stop me from installing the p's, but they can't stop me from installing gawk.

Gawk and pgn-extract are both command-line programs which you use by entering commands in a command interpreter -- cmd.exe on Windows. The command-line is very practical for text processing, but it's not visual like a GUI. It's more like walking around your house with a blindfold on, and you only navigate successfully because you already know where everything is.

• http://gnuwin32.sourceforge.net/packages
• Click on the "Gawk" link.
• Click on the "General Installation Instructions" link, which also includes usage instructions, and read up.
• Go back to the Gawk page.
• Probably you want the "Complete package, except sources" Setup option. 
• Click on that to download the setup.exe.
• Ideally you would verify the download with the provided md5sum, but I'm not going to describe how to do that.
• Finally, follow the previous General Installation Instructions to install.

There is a big caveat to all of the above gawk installation steps. I have gawk working on a couple of Windows 10 machines, but I didn't use the setup method (it's exactly this that is blocked at my work), so I'm not really sure if it will succeed. The setup.exe on sourceforge was created for Windows Vista, and might not work on Windows 10. Or it might partially succeed but require tweaks after.

Anyway, fingers crossed, after installation open cmd.exe and type `gawk --version` (without the quotes), to verify that you can run it. With pgn-extract the command is `pgn-extract --version`.

Jonathan003 wrote on 10/25/20 at 13:29:10:

Yes, that's another way, assuming Unix means a Linux distribution. But it's another can of worms. Your Linux will come with awk but not with pgn-extract. To install pgn-extract via the package manager means you need to connect the VM to the network, which is not a trivial step. The other option is to build pgn-extract from source, which is even more fun.

Edited:

Jonathan003 wrote on 02/01/20 at 16:27:49:


Hmmm, I will give it a try. Sounds interesting

I use polyglot_tolerant for this.
I use the book merging options. And In Lucas chess there is an option to import all best moves from a bin book to a repertoire. And the repertoire can be exported to pgn.
If you want to know how I make a repertoire send me a private message.

Easy on unix or mac. 
Only slightly less easy on windows. http://gnuwin32.sourceforge.net/packages.html

1. convert your repertoire.pgn(s) to repertoire.epd (e.g. use pgn-extract)
2. convert your new.pgn to new.epd
3. for each line in new.epd 
   $ grep -F "fen" repertoire.epd

I wrote a couple of helper awk scripts to somewhat automate this task, and combined them in one attachment. You would have to separate them again before using. WIth these scripts, the exact sequence of commands to reproduce the above steps would be:

• $ pgn-extract -w200 --fencomments -orepertoire1.pgn repertoire.pgn
• $ gawk -f fencomments.awk repertoire1.pgn > repertoire1.epd
• $ pgn-extract -w200 --fencomments -onew1.pgn new.pgn
• $ gawk -f fencomments.awk new1.pgn > new1.epd
• $ gawk -f epdcompare.awk -v CMP=repertoire1.epd new1.epd > cmp.epd

With the following repertoire.pgn 

[Event "rep game 1"]
[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 g6
6. f4 Bg7 7. e5 dxe5 8. fxe5 Ng4 9. Bb5+ Kf8 10. Ne6+
*

[Event "rep game 2"]
[Result "*"]

1. e4 e6 2. d4 d5 3. e5 c5 4. c3 Nc6 5. Nf3 Qb6
6. Bd3 cxd4 7. cxd4 Nxd4 8. Nxd4 Qxd4 9. Bb5+ Bd7 10. Bxd7+ Kxd7
11. Qxd4
* 

And the following new.pgn to compare it to:

[Event "new game 1"]
[Result "*"]

1. e4 e6 2. d4 d5 3. e5 c5 4. c3 Qb6 {N} 5. Nf3
5... Nc6 {transposition}
6. Bd3 cxd4 7. cxd4 Bd7 {N} 8. O-O Nxd4 9. Nxd4 Qxd4 10. Nc3 a6
* 

The output cmp.epd is below.

rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq e3;hmvc 0;fmvn 1;id 1;fk 1 2;fmvnmax 1;
rnbqkbnr/pppp1ppp/4p3/8/4P3/8/PPPP1PPP/RNBQKBNR w KQkq -;hmvc 0;fmvn 2;id 1;fk 2;fmvnmax 2;
rnbqkbnr/pppp1ppp/4p3/8/3PP3/8/PPP2PPP/RNBQKBNR b KQkq d3;hmvc 0;fmvn 2;id 1;fk 2;fmvnmax 2;
rnbqkbnr/ppp2ppp/4p3/3p4/3PP3/8/PPP2PPP/RNBQKBNR w KQkq d6;hmvc 0;fmvn 3;id 1;fk 2;fmvnmax 3;
rnbqkbnr/ppp2ppp/4p3/3pP3/3P4/8/PPP2PPP/RNBQKBNR b KQkq -;hmvc 0;fmvn 3;id 1;fk 2;fmvnmax 3;
rnbqkbnr/pp3ppp/4p3/2ppP3/3P4/8/PPP2PPP/RNBQKBNR w KQkq c6;hmvc 0;fmvn 4;id 1;fk 2;fmvnmax 4;
rnbqkbnr/pp3ppp/4p3/2ppP3/3P4/2P5/PP3PPP/RNBQKBNR b KQkq -;hmvc 0;fmvn 4;id 1;fk 2;fmvnmax 4;
rnb1kbnr/pp3ppp/1q2p3/2ppP3/3P4/2P5/PP3PPP/RNBQKBNR w KQkq -;hmvc 1;fmvn 5;id 1;fk 0;fmvnmax 0;
rnb1kbnr/pp3ppp/1q2p3/2ppP3/3P4/2P2N2/PP3PPP/RNBQKB1R b KQkq -;hmvc 2;fmvn 5;id 1;fk 0;fmvnmax 0;
r1b1kbnr/pp3ppp/1qn1p3/2ppP3/3P4/2P2N2/PP3PPP/RNBQKB1R w KQkq -;hmvc 3;fmvn 6;id 1;fk 2;fmvnmax 6;
r1b1kbnr/pp3ppp/1qn1p3/2ppP3/3P4/2PB1N2/PP3PPP/RNBQK2R b KQkq -;hmvc 4;fmvn 6;id 1;fk 2;fmvnmax 6;
r1b1kbnr/pp3ppp/1qn1p3/3pP3/3p4/2PB1N2/PP3PPP/RNBQK2R w KQkq -;hmvc 0;fmvn 7;id 1;fk 2;fmvnmax 7;
r1b1kbnr/pp3ppp/1qn1p3/3pP3/3P4/3B1N2/PP3PPP/RNBQK2R b KQkq -;hmvc 0;fmvn 7;id 1;fk 2;fmvnmax 7;
r3kbnr/pp1b1ppp/1qn1p3/3pP3/3P4/3B1N2/PP3PPP/RNBQK2R w KQkq -;hmvc 1;fmvn 8;id 1;fk 0;fmvnmax 0;
r3kbnr/pp1b1ppp/1qn1p3/3pP3/3P4/3B1N2/PP3PPP/RNBQ1RK1 b kq -;hmvc 2;fmvn 8;id 1;fk 0;fmvnmax 0;
r3kbnr/pp1b1ppp/1q2p3/3pP3/3n4/3B1N2/PP3PPP/RNBQ1RK1 w kq -;hmvc 0;fmvn 9;id 1;fk 0;fmvnmax 0;
r3kbnr/pp1b1ppp/1q2p3/3pP3/3N4/3B4/PP3PPP/RNBQ1RK1 b kq -;hmvc 0;fmvn 9;id 1;fk 0;fmvnmax 0;
r3kbnr/pp1b1ppp/4p3/3pP3/3q4/3B4/PP3PPP/RNBQ1RK1 w kq -;hmvc 0;fmvn 10;id 1;fk 0;fmvnmax 0;
r3kbnr/pp1b1ppp/4p3/3pP3/3q4/2NB4/PP3PPP/R1BQ1RK1 b kq -;hmvc 1;fmvn 10;id 1;fk 0;fmvnmax 0;
r3kbnr/1p1b1ppp/p3p3/3pP3/3q4/2NB4/PP3PPP/R1BQ1RK1 w kq -;hmvc 0;fmvn 11;id 1;fk 0;fmvnmax 0; 

Note that it matches by position and not by move. The two "N" moves result in positions with "fk 0", and the transposition is also caught.

• id - game number in the new.pgn
• fk - array of matching game numbers in repertoire.pgn
  Note that the first position matched both games in the repertoire.
• fmvn - full move number in new.pgn.
  Note that for black moves this is "off by one" from the last move played.
• fmvnmax - the largest "repertoire" full move number where the id-game in new.pgn matched any fk-game in repertoire.pgn

I could have output the sm opcode as well (supplied move), and for openings work this would have made it a little easier to scan the final epd.  I didn't do that because I'm actually planning to use epdcompare with endgame epd-files (not pgn-files). As it stands, you will probably have to open both new.pgn and cmp.epd at the same time and scan two files together. But since the OP asked about ChessBase, I assume the plan is: once the interesting new games are identified, merge by dragging them onto the appropriate game in the repertoire.

brabo wrote on 06/07/19 at 12:52:16:

Thanks for this interesting answer. Fritz is cheaper than chessbase so i'll probably buy it. The functionality you describe seems very interesting and as you said close to my need but I will realize  and perfectly understand it as soon as I will experiment this functionality.
Thanks again !

Hello everyone,

Is there a tool - if possible a freeware -   
that from two repertoire databases for each color  (in Chessbase or PGN format) 
filters all games of another database or PGN (the target base) that are in my repertoire.

Ideally I'd like to know from what move me or my opponent are "out" of the repertoire.
I have never managed to do that with Chessbase 8 (yes I know I date).
And I hesitate to buy a newer version of Chessbase. 
Is there a version of Chessbase that manages to do it?

Thank you in advance for your help.