Latest Updates:
Page Index Toggle Pages: [1] 2 
Topic Tools
Very Hot Topic (More than 25 Replies) Filter games according to my repertoire (Read 3152 times)
an ordinary chessplayer
God Member
*****
Offline


I used to be not bad.

Posts: 936
Location: Columbus, OH (USA)
Joined: 01/02/15
Re: Filter games according to my repertoire
Reply #26 - 11/15/20 at 18:36:39
Post Tools
Jonathan003 wrote on 11/14/20 at 21:08:30:
I managed to do what you described.
How can I use the output to fill in the gaps of my repertoire?

Yes!
For each move in the new pgn, it steps through your repertoire pgn and finds the place where it matches. Look for the far right column having a "0". If there is a long run of "0" on the right, that represents a "novelty" for your repertoire. A short run of "0" represents a transposition.

Code
Select All
rnbqkbnr/pp3ppp/4p3/2ppP3/3P4/2P5/PP3PPP/RNBQKBNR b KQkq -;hmvc 0;fmvn 4;id 1;fk 2;fmvnmax 4;
rnb1kbnr/pp3ppp/1q2p3/2ppP3/3P4/2P5/PP3PPP/RNBQKBNR w KQkq -;hmvc 1;fmvn 5;id 1;fk 0;fmvnmax 0;
rnb1kbnr/pp3ppp/1q2p3/2ppP3/3P4/2P2N2/PP3PPP/RNBQKB1R b KQkq -;hmvc 2;fmvn 5;id 1;fk 0;fmvnmax 0;
r1b1kbnr/pp3ppp/1qn1p3/2ppP3/3P4/2P2N2/PP3PPP/RNBQKB1R w KQkq -;hmvc 3;fmvn 6;id 1;fk 2;fmvnmax 6; 


Opcode translations
  • hmvc - half-move clock, number of ply since the last pawn move or capture in the new game
  • fmvn - full move number, number of moves so far in the new game
  • id - identity, game number in the new pgn
  • fk - foreign key, game number(s) in the repertoire pgn
  • fmvnmax - full move number max, (largest) number of moves so far in the repertoire game(s)

In the sample output above
  • fmvn 4;id 1;fk 2;fmvnmax 4; - new game 1 move 4 matches repertoire game 2 move 4
  • fmvn 5;id 1;fk 0;fmvnmax 0; - new game 1 move 5 does not match the repertoire
  • fmvn 5;id 1;fk 0;fmvnmax 0; - new game 1 move 5 does not match the repertoire
  • fmvn 6;id 1;fk 2;fmvnmax 6; - new game 1 move 6 matches repertoire game 2 move 6

So on move 4 black played a "new" move (reached a position not in the repertoire), then on move 5 white played a "new" move (in reply to black's "new" move), then on move 5 black played a "non-new" move. Therefore black's "new" move on move 4 was a simple transposition. Whether the transposition is important for your repertoire is up to you. The simple idea is you open the new pgn, scroll down to game number id, scroll across to move number fmvn (having fmvnmax = 0), then use your brain (or engine) to decide whether or not to import that game to your repertoire. Software can only automate things up to a certain point.

Jonathan003 wrote on 11/14/20 at 21:08:30:
For my white repertoire, I only want to find the games in the new database, with moves for black, where I don't already have a reply for.
And I want to do the same for my black repertoire.
I think that was also the question of the original poster of this topic.

You do realize that all the games in the new database will have a move you don't already have a reply for. Unless one of them is an exact duplicate of a repertoire game that will always be the case. There is no way around the fact that you will have to look at every game. But it's not hopeless. You can use the fmvn/fmvnmax to reject some of the "novelties". Most people do an arbitrary cut-off at move 15 or 20. Only in certain critical lines would you go much beyond that.

Jonathan003 wrote on 11/14/20 at 21:08:30:
I basically want to do what's described here:

See attachment

Under the title Book Merging

I'm not sure about the polyglot book idea. Those are designed for engines. You might find some use for practicing against an engine, but otherwise I would look at different avenues. There are good reasons for tracking more than one move for your side in a repertoire, but there's no way to tell the polyglot book what that reason is. To the engine, every move in a polyglot book is the same.

Jonathan003 wrote on 11/14/20 at 21:08:30:
I can import the best moves from the merged bin books to Opening Lines in Lucas Chess, and then export to pgn.
But I want to be able to do this directly with pgn games, so the game data information gets preserved. There is also some weird explosion of number of games if I use Lucas Chess for this.

Exactly. Most repertoire tools have some facility for importing pgn, but when it comes to exporting pgn, the situation is, shall we say, lossy. As an example, I think Chess Openings Wizard (Bookup) is a great piece of software, as long as you only intend to use COW exclusively. But I use lots of other software, so any effort I put into COW is wasted when I switch to another tool.

I haven't used Lucas Chess, so I don't know how it handles these things. You say it can "import the best moves", but I rather doubt that. Isn't it going to import "all" the moves? I recommend you search on brabo's blog for "green moves" where he uses a ChessBase tree. The idea is that a repertoire should show best moves and second-best moves, and a way to know which is which. After all, theory changes and today's best move can become tomorrow's second-best. Or you may decide second-best is less work while still being good enough. You should also look at Scid's "tree mask" feature -- very cool. I don't like the ChessBase tree at all, and I do like the Chess Assistant tree very much. But all these tools can only give ideas for how to handle your own repertoire.

I strongly believe that keeping the repertoire in pgn in a form that allows for easy import to your repertoire tool of choice is, in the long run, the only way to avoid significant pain. By all means import to the tool, but consider it to be read-only, and make any changes to the repertoire in the original pgn. Otherwise, at some point, you are going to end up with information you can't export. The farther in the future that point is, the more information you will have lost.
  
Back to top
 
IP Logged
 
Jonathan003
YaBB Newbies
*
Offline


I Love ChessPublishing!

Posts: 11
Location: Belgium
Joined: 01/29/20
Gender: Male
Re: Filter games according to my repertoire
Reply #25 - 11/14/20 at 21:08:30
Post Tools
I managed to do what you described.
How can I use the output to fill in the gaps of my repertoire?
For my white repertoire, I only want to find the games in the new database, with moves for black, where I don't already have a reply for.
And I want to do the same for my black repertoire.
I think that was also the question of the original poster of this topic.
I basically want to do what's described here:

See attachment

Under the title Book Merging

I can import the best moves from the merged bin books to Opening Lines in Lucas Chess, and then export to pgn.
But I want to be able to do this directly with pgn games, so the game data information gets preserved. There is also some weird explosion of number of games if I use Lucas Chess for this.
  

Manual_Polyglot.txt ( 19 KB | 4 Downloads )
Back to top
 
IP Logged
 
an ordinary chessplayer
God Member
*****
Offline


I used to be not bad.

Posts: 936
Location: Columbus, OH (USA)
Joined: 01/02/15
Re: Filter games according to my repertoire
Reply #24 - 11/03/20 at 19:55:27
Post Tools
Jonathan003 wrote on 11/03/20 at 19:06:49:
Can you please make some video capture. From beginning to end so I understand wat to do on a windows 10 64 bit system.

Not happening.
  
Back to top
 
IP Logged
 
Jonathan003
YaBB Newbies
*
Offline


I Love ChessPublishing!

Posts: 11
Location: Belgium
Joined: 01/29/20
Gender: Male
Re: Filter games according to my repertoire
Reply #23 - 11/03/20 at 19:16:25
Post Tools
an ordinary chessplayer wrote on 11/02/20 at 18:05:08:
Well actually there is dedicated software, but the OP mentioned freeware. I haven't looked closely at Scid, but maybe it has this capability. If you don't mind paying then CPT was mentioned earlier, and ChessBase also has a repertoire and "find novelty" feature.

I have all count off chess software on my windows 10 pc you can think of, both commercial as free . I really don't know of a tool to do this efficient. Lucas Chess comes closed by importing best moves from a a merged bin book.
  
Back to top
 
IP Logged
 
Jonathan003
YaBB Newbies
*
Offline


I Love ChessPublishing!

Posts: 11
Location: Belgium
Joined: 01/29/20
Gender: Male
Re: Filter games according to my repertoire
Reply #22 - 11/03/20 at 19:06:49
Post Tools
Ok I have downloaded the arhive.txt file
I really don't understand what to do.
Can you please make some video capture. From beginning to end so I understand wat to do on a windows 10 64 bit system.
Maybe also for others ho have little experience with command line tools and want to try this.
I think a video capture with mouse capture and maybe audio is the easiest way to explain something.
  
Back to top
 
IP Logged
 
an ordinary chessplayer
God Member
*****
Offline


I used to be not bad.

Posts: 936
Location: Columbus, OH (USA)
Joined: 01/02/15
Re: Filter games according to my repertoire
Reply #21 - 11/02/20 at 20:32:52
Post Tools
Jonathan003 wrote on 11/02/20 at 17:07:28:
an ordinary chessplayer wrote on 10/30/20 at 05:46:29:
It's the PATH environment variable that allows cmd.exe to find pgn-extract and awk (or gawk). The easiest way to find the other files is to run cmd.exe from the same folder as the files (not the same folder as awk or pgn-extract). Put these four files all by themselves in their own folder ...
  • fencomments.awk
  • epdcompare.awk
  • rep.pgn
  • new.pgn

Where can I find the files 'fencomments.awk'
and 'epdcompare.awk'?
Should I find them somewhere on my computer or do I have to make them myself?

They are in the file archive.txt which I attached at Reply #16. Look for, e.g., [fencomments.awk] and then copy from below that and down to the next section. The "[fencomments.awk]" is not part of the file fencomments.awk, hope that makes sense.

Jonathan003 wrote on 11/02/20 at 17:16:36:
I have little experience with command line tools.
If I type cmd in the adress C:Program Files (x86)GnuWin32
in
where awk is installed, and I type 'awk --version' I get the message GNU Awk 3.1.6
You talk about GNU Awk 3.1.3
Will this make any difference?

No, that version is about the same. There were some changes but my script does not use any of the changed features. The --re-interval option requires 3.0 or greater so you are good. The --version command was less to verify the version number, than to verify that cmd.exe could find awk.exe. I often find myself making typos on the command line, leaving out options, forgetting the differences between linux / windows / and now mac, etc. Doing things step-by-step like that just makes it easier to find out exactly where things went wrong. Shell admins do "ls" first and then "rm". Database admins do "select" first and then "update". It's a good habit to take baby steps even when you are sure you know what you are doing.
  
Back to top
 
IP Logged
 
an ordinary chessplayer
God Member
*****
Offline


I used to be not bad.

Posts: 936
Location: Columbus, OH (USA)
Joined: 01/02/15
Re: Filter games according to my repertoire
Reply #20 - 11/02/20 at 18:05:08
Post Tools
Jonathan003 wrote on 10/29/20 at 18:02:46:
I don't understand why there exist no dedicated chess software to do this.
It's a big gap in the chess market I think.

Well actually there is dedicated software, but the OP mentioned freeware. I haven't looked closely at Scid, but maybe it has this capability. If you don't mind paying then CPT was mentioned earlier, and ChessBase also has a repertoire and "find novelty" feature.

Jonathan003 wrote on 10/29/20 at 18:02:46:
Chess is not only for programmers I hope Smiley

I'm not a programmer, just a business analyst, but I also spend time trying to learn some programming skills. It's an attitude not just towards computers but towards everything. You don't need to be a chef in order to cook, an electrician to do household wiring, an auto mechanic to repair your vehicle, etc., etc. (Well that last one is changing as the automobile evolves into a computer on wheels, but the simplest repairs can still be do-it-yourself.)

Quote:
recognize programming as the new literacy of the digital age
--Rushkoff (2011) Program or Be Programmed

There is some argument for sticking to what you do best and leaving everything else to an expert -- provided what you do best pays well enough that you can afford the expert's pay scale. Personal chef, anyone? If you want to go that route, ChessBase is the program for you. It has a feature to do almost anything you can think of (although it won't do it quite the way you want!). Its feature set makes it the MS Excel of the chess world, complete with the lock-in to Windows (actually not true of Excel anymore), the upgrade pains, and being expensive. I'm not against it, having paid for ChessBase and competing products many, many times over the decades. I'm just listing the pros and cons.
  
Back to top
 
IP Logged
 
Jonathan003
YaBB Newbies
*
Offline


I Love ChessPublishing!

Posts: 11
Location: Belgium
Joined: 01/29/20
Gender: Male
Re: Filter games according to my repertoire
Reply #19 - 11/02/20 at 17:16:36
Post Tools
I have little experience with command line tools.
If I type cmd in the adress C:\Program Files (x86)\GnuWin32\bin
where awk is installed, and I type 'awk --version' I get the message GNU Awk 3.1.6
You talk about GNU Awk 3.1.3
Will this make any differense?
  
Back to top
 
IP Logged
 
Jonathan003
YaBB Newbies
*
Offline


I Love ChessPublishing!

Posts: 11
Location: Belgium
Joined: 01/29/20
Gender: Male
Re: Filter games according to my repertoire
Reply #18 - 11/02/20 at 17:07:28
Post Tools
an ordinary chessplayer wrote on 10/30/20 at 05:46:29:
It's the PATH environment variable that allows cmd.exe to find pgn-extract and awk (or gawk). The easiest way to find the other files is to run cmd.exe from the same folder as the files (not the same folder as awk or pgn-extract). Put these four files all by themselves in their own folder ...
  • fencomments.awk
  • epdcompare.awk
  • rep.pgn
  • new.pgn

Where can I find the files 'fencomments.awk'
and 'epdcompare.awk'?
Should I find them somewhere on my computer or do I have to make them myself?

  
Back to top
 
IP Logged
 
an ordinary chessplayer
God Member
*****
Offline


I used to be not bad.

Posts: 936
Location: Columbus, OH (USA)
Joined: 01/02/15
Re: Filter games according to my repertoire
Reply #17 - 10/30/20 at 05:46:29
Post Tools
Jonathan003 wrote on 10/29/20 at 13:22:47:
For step two I have some problems. I downloaded and installed Gawk Complete package, except sources from here: gnuwin32.sourceforge.net/packages/gawk

I first tried to copy the pgn repertoire1.pgn create with Pgn-Extract to the folder where a installed Gawk C:\Program Files (x86)\GnuWin32\bin
Than if I click the adres bar and type cmd for the command prompt, and I enter this line: 'awk -f fencomments.awk repertoire1.pgn > repertoire1.epd'
I get the message 'Access is denied'
So I copied the folder  'C:\Program Files (x86)\GnuWin32' to my documents folder and tried to do the same.
But then I get this message:
awk: fatal: can't open source file `fencomments.awk' for reading (No such file or directory)

What do I do wrong?


You want to keep your programs and your data files separate. So undo the copy of "GnuWin32" to my documents. You can use the executable as installed in "C:\Program Files (x86)\GnuWin32\bin", and use your data files in Documents together in one command. But cmd.exe has its own ideas about where things are located, and needs some help finding things. cmd.exe looks for things in the following rough order:
  1. absolute or relative path on the command line
  2. current directory
  3. path specified in the registry
  4. PATH environment variable

It's the PATH environment variable that allows cmd.exe to find pgn-extract and awk (or gawk). The easiest way to find the other files is to run cmd.exe from the same folder as the files (not the same folder as awk or pgn-extract). Put these four files all by themselves in their own folder ...
  • fencomments.awk
  • epdcompare.awk
  • rep.pgn
  • new.pgn

... then start cmd.exe in that folder using your method from above. After that the commands I gave in the post just before this one should work.
  
Back to top
 
IP Logged
 
an ordinary chessplayer
God Member
*****
Offline


I used to be not bad.

Posts: 936
Location: Columbus, OH (USA)
Joined: 01/02/15
Re: Filter games according to my repertoire
Reply #16 - 10/30/20 at 05:26:30
Post Tools
I managed to update the scripts I provided before. In fact I did do the earlier work on Linux, and "forgot" to check whether they would work on Windows, which of course they did not. They didn't run on macOS either, but I only got the Mac recently so did not have an opportunity to test that before. It was quite a chore getting these to work on Windows, I won't bore you with the details.

Here is my command session on Windows 10.
Code
Select All
$ dir /b
epdcompare.awk
fencomments.awk
new.pgn
rep.pgn

$ awk --version
GNU Awk 3.1.3

$ pgn-extract --version
pgn-extract v19-04

$ pgn-extract -w200 --fencomments -orep1.pgn rep.pgn
Processing rep.pgn
- - - Edited game W3Q689U4ZN 2019.06.08
- - - Edited game W3Q689U4ZN 2019.06.08
2 games matched out of 2.

$ awk --re-interval -f fencomments.awk rep1.pgn > rep1.epd

$ pgn-extract -w200 --fencomments -onew1.pgn new.pgn
Processing new.pgn
- - - Edited game W3Q689U4ZN 2019.06.08
1 game matched out of 1.

$ awk --re-interval -f fencomments.awk new1.pgn > new1.epd

$ awk --re-interval -f epdcompare.awk CMP=rep1.epd new1.epd > cmp.epd

$
 



And attached is a single text-file "archive" of all the files from that session. To run on Windows, use the scripts as-is and the commands above. To run on Linux or macOS, comment the Windows-specific grepcmd on line 23, uncomment the Unixish grepcmd on line 22, and leave off the `--re-interval` switch from the command line.
  

archive.txt ( 16 KB | 8 Downloads )
Back to top
 
IP Logged
 
an ordinary chessplayer
God Member
*****
Offline


I used to be not bad.

Posts: 936
Location: Columbus, OH (USA)
Joined: 01/02/15
Re: Filter games according to my repertoire
Reply #15 - 10/30/20 at 04:28:09
Post Tools
Jonathan003 wrote on 10/29/20 at 18:23:45:
I think you need a Unix system to do what he discribes.

Too bad it's a video, I'm not going to watch it. But if he's just using pgn-extract and other command line utilities, it can no doubt be done on Windows as well.
  
Back to top
 
IP Logged
 
Jonathan003
YaBB Newbies
*
Offline


I Love ChessPublishing!

Posts: 11
Location: Belgium
Joined: 01/29/20
Gender: Male
Re: Filter games according to my repertoire
Reply #14 - 10/29/20 at 18:23:45
Post Tools
I can't post a weblink here, but if you search in YouTube for: 'Outil gratuit de recherche dans une base pgn'
You find a video in French explaining how to build repertoire with pgn-extract.
I don't understand French and I think you need a Unix system to do what he discribes.
  
Back to top
 
IP Logged
 
Jonathan003
YaBB Newbies
*
Offline


I Love ChessPublishing!

Posts: 11
Location: Belgium
Joined: 01/29/20
Gender: Male
Re: Filter games according to my repertoire
Reply #13 - 10/29/20 at 18:02:46
Post Tools
an ordinary chessplayer wrote on 10/29/20 at 17:44:08:
We'll get it working, but not right now. I want to get back to you on this. I thought I should test my own steps, but here at work gawk is choking on the FEN regular expression in my scripts. It's a little strange, because I did use these exact scripts, but probably did it on linux with a different version of gawk. And while looking at it I noticed that I also used grep but didn't mention that.

So let me rewrite the script to not use grep, and get the regular expression working in all versions of gawk, and test that it all works on windows/linux/macos. I can't do all that right now, but should be able to do it this weekend.

Thanks I'm looking forward to this.
I don't understand why there exist no dedicated chess software to do this.
It's a big gap in the chess marked I think. Chess is not only for programmers I hope Smiley
  
Back to top
 
IP Logged
 
an ordinary chessplayer
God Member
*****
Offline


I used to be not bad.

Posts: 936
Location: Columbus, OH (USA)
Joined: 01/02/15
Re: Filter games according to my repertoire
Reply #12 - 10/29/20 at 17:44:08
Post Tools
Jonathan003 wrote on 10/29/20 at 13:22:47:
... Than if I click the adres bar and type cmd for the command prompt, ...

I learned something today.  Smiley

You're almost there. You just don't have enough experience with the command line to help it find the files it needs.

We'll get it working, but not right now. I want to get back to you on this. I thought I should test my own steps, but here at work gawk is choking on the FEN regular expression in my scripts. It's a little strange, because I did use these exact scripts, but probably did it on linux with a different version of gawk. And while looking at it I noticed that I also used grep but didn't mention that.

So let me rewrite the script to not use grep, and get the regular expression working in all versions of gawk, and test that it all works on windows/linux/macos. I can't do all that right now, but should be able to do it this weekend.
  
Back to top
 
IP Logged
 
Page Index Toggle Pages: [1] 2 
Topic Tools
Bookmarks: del.icio.us Digg Facebook Google Google+ Linked in reddit StumbleUpon Twitter Yahoo