Latest Updates:
Hot Topic (More than 10 Replies) How to clean multiple pgn files in a folder all at (Read 565 times)
Jonathan003
YaBB Newbies
*
Offline


I Love ChessPublishing!

Posts: 22
Location: Belgium
Joined: 01/29/20
Gender: Male
Re: How to clean multiple pgn files in a folder all at
Reply #10 - 08/30/23 at 19:26:16
Post Tools
I changed the my approche inspired by your script.
I created 4 batch files with the help of ChatGPT 3,5
They are in the attachment if you want to try them for yourself.


Put pgn-extract, eco.pgn and the .bat files, in the same folder as the pgn files to clean.
Drag the to clean pgn files to the desired bat file.
The cleaned pgn files are in the output folder.

I'm prety happy with the result, but I have to do more testing. If I find errors I will try your script.
Thanks for the effort!

This is a link to my Google drive for the batch files and a readme:
https://drive.google.com/file/d/1AkZCMnRn8OoM7rzrJwkNoT1Ctz0rDqno/view?usp=shari...

  
Back to top
 
IP Logged
 
Jonathan003
YaBB Newbies
*
Offline


I Love ChessPublishing!

Posts: 22
Location: Belgium
Joined: 01/29/20
Gender: Male
Re: How to clean multiple pgn files in a folder all at
Reply #9 - 08/30/23 at 12:44:40
Post Tools
$inputDirectory = "C:\Users\creme\Downloads\Chess\to_clean_pgns\input"
$outputDirectory = "C:\Users\creme\Downloads\Chess\to_clean_pgns\output"
$pgnExtractPath = "C:\Users\creme\Downloads\Chess\to_clean_pgns\pgn-extract.exe"

Get-ChildItem -Path $inputDirectory -Filter *.pgn | ForEach-Object {
    $outputFile = Join-Path $outputDirectory $_.Name
    & $pgnExtractPath -C -N -V -7 -o $outputFile $_.FullName
}

This is the final scipt from ChatGPT
I could never write such a script without the help of ChatGPT
I wonder what to study and how long it takes someone can write scripts like this?
  
Back to top
 
IP Logged
 
an ordinary chessplayer
God Member
*****
Offline


I used to be not bad.

Posts: 1634
Location: Columbus, OH (USA)
Joined: 01/02/15
Re: How to clean multiple pgn files in a folder all at
Reply #8 - 08/29/23 at 20:47:05
Post Tools
Hah, I considered writing you a powershell script, but I chose a cmd script because powershell uses objects, which is unnecessary complexity for running a basic stream processor like pgn-extract. That chatgpt script is similar to the talkchess script -- 20% there because it runs pgn-extract successfully, but it doesn't actually meet any of your other requirements.

Jonathan003 wrote on 08/29/23 at 19:50:54:
-o $outputFile


According to pgn-extract help file https://www.cs.kent.ac.uk/people/staff/djb/pgn-extract/help.html :
Quote:
Note that there must be no space between either -o or -a and the output file name.

In my testing, pgn-extract on the Linux command line doesn't mind a space. Nevertheless, I follow the documentation whenever possible.
  
Back to top
 
IP Logged
 
an ordinary chessplayer
God Member
*****
Offline


I used to be not bad.

Posts: 1634
Location: Columbus, OH (USA)
Joined: 01/02/15
Re: How to clean multiple pgn files in a folder all at
Reply #7 - 08/29/23 at 20:20:19
Post Tools
For user-friendly usage of the batch file, put a shortcut to the batch file in the Windows SendTo folder.
  • Find a permanent home for the batch file, so SendTo always knows where to look for it. This can be anywhere that makes sense to you. One likely candidate is the same folder where pgn-extract is installed. So, move pgn-clean.bat into "C:\Users\creme\Documents\static\programs\bin".
  • Create one shortcut for each set of pgn-extract options you commonly wish to run. For illustration, create one shortcut to remove comments, and a second shortcut to remove variants. (You can create shortcuts with different pgn-extract options according to your needs.)
    First shortcut:
    1. right-click on pgn-clean.bat and choose Create Shortcut.
    2. right-click on the shortcut and choose Rename, change it from "pgn-clean.bat - Shortcut" to
      'pgn-clean-C".
    3. Right-click again on the shortcut and choose Properties. In the Target field after pgn-clean.bat add a space followed by -C.
      C:\Users\creme\Docments\static\programs\bin\pgn-clean.bat -C
      Click OK.
    Second shortcut: Do the same three steps but use "V" instead of "C". To reiterate, this is just for demonstration purposes. You can use any pgn-extract options you like.
  • Move the shortcuts you created into the folder "C:\Users\creme\AppData\Roaming\Microsoft\Windows\SendTo"
Now whenever you want to remove all the comments from a pgn folder, right-click on it, choose SendTo, choose pgn-clean-C. To remove variants, choose pgn-clean-V. Just keep in mind how pgn-clean.bat uses the clean sub-folder and skips any pgn files already in there. The way Windows shortcuts work is by constructing command lines (targets) and executing them. And "SendTo" puts your folder at the end of that command line, which is where the batch file expects to find it. pgn-clean.bat stops processing command line arguments after the first folder, so if you select multiple folders and use SendTo, only one of the folders will be cleaned.

Note that running the batch file this way, the command window closes at the end, because it has (and needs) an "exit" command. If you want the command window to hang around, there are different ways to accomplish that. Here are two:
  1. Edit the shortcut Target, adding at the beginning cmd /K and a space.
    cmd /K C:\Users\crewe\Docments\static\programs\bin\pgn-clean.bat -C
  2. Edit the script, adding a pause statement (just pause on a line by itself), just above the exit statement, like so:
    :PROC_EXIT
      echo %DATE% %TIME% INFO 0 exiting [%~f0]
      pause
      exit /b %errorlevel%
  
Back to top
 
IP Logged
 
Jonathan003
YaBB Newbies
*
Offline


I Love ChessPublishing!

Posts: 22
Location: Belgium
Joined: 01/29/20
Gender: Male
Re: How to clean multiple pgn files in a folder all at
Reply #6 - 08/29/23 at 19:50:54
Post Tools
Thanks I will try the script later later.
I also created a scipt with the help of ChatGPT

Quote:
$inputDirectory = "C:\Users\creme\Downloads\Chess\to_clean_pgns\input"
$outputDirectory = "C:\Users\creme\Downloads\Chess\to_clean_pgns\output"
$pgnExtractPath = "C:\Users\creme\Downloads\Chess\to_clean_pgns\pgn-extract.exe"

Get-ChildItem -Path $inputDirectory -Filter *.pgn | ForEach-Object {
    $outputFile = Join-Path $outputDirectory $_.Name
    & $pgnExtractPath -C -N -V -o $outputFile --nocomments $_.FullName
}


I saved the text file to "Clean-PGN.ps1"
And I run the script using PowerShell: .\Clean-PGN.ps1
I have to do some more testing to see how it works.
The pgn files are not overwritten but the cleaned pgn fiiles are saved to another folder. That's also ok for me.

  
Back to top
 
IP Logged
 
an ordinary chessplayer
God Member
*****
Offline


I used to be not bad.

Posts: 1634
Location: Columbus, OH (USA)
Joined: 01/02/15
Re: How to clean multiple pgn files in a folder all at
Reply #5 - 08/29/23 at 18:34:43
Post Tools
Updated the script for error 703 description.
  

pgn-clean-bat_001.txt ( 2 KB | 10 Downloads )
Back to top
 
IP Logged
 
an ordinary chessplayer
God Member
*****
Offline


I used to be not bad.

Posts: 1634
Location: Columbus, OH (USA)
Joined: 01/02/15
Re: How to clean multiple pgn files in a folder all at
Reply #4 - 08/29/23 at 17:31:41
Post Tools
You did great. I made an error when I wrote the script, in your output it says:
Quote:
ERROR 703 pgn-extract does not exist

That is the error description for 701, for 703 it should be something like "pgn folder not provided". And indeed simply double-clicking on the batch file does not tell it which folder to clean.

For now try this.
  • In Windows Explorer open your pgn folder and then open cmd.exe there.
  • Using the mouse drag pgn-clean.bat from Windows Explorer into the cmd.exe window.
  • At the end of the command line type a space followed by a single period ("." means "current folder").
  • Type enter to run it.

Those steps should work for any pgn folder, not relying on pgn-clean.bat being in the same folder.

Later I will post a corrected version of the script.

Also I will write some instructions how to make it easy to run just using Windows Explorer and the mouse, bypassing cmd.exe entirely. (Although cmd.exe is good because you can give any pgn-extract options you want.)

By the way, the talkchess script wasn't going to do what you wanted, it was putting all the cleaned pgns in a single output file.
  
Back to top
 
IP Logged
 
Jonathan003
YaBB Newbies
*
Offline


I Love ChessPublishing!

Posts: 22
Location: Belgium
Joined: 01/29/20
Gender: Male
Re: How to clean multiple pgn files in a folder all at
Reply #3 - 08/29/23 at 08:53:44
Post Tools
By the way I asked the same question on talkchess.com

Someone gived me this script as exemple:

SET PATH=D:\bin;%PATH%

FOR %%F IN (%*) DO (
     pgn-extract -amerged.pgn -D -s -C -N -V -Z -Tr1-0 -Tr0-1 -Tr1/2-1/2 -lmerge_errors.txt -tstartpos.txt --fixtagstrings -w80 --minmoves 8 --nobadresults "%%F"
)
pgn-extract -Wuci -opedantic-clean.pgn -D -s -C -N -V -Z -lmerge_errors.txt -tstartpos.txt -w128 --minmoves 8 --nobadresults merged.pgn
DEL merged.pgn

He says that I could edit it a bit so I could do what I want.
But dousn't help me much becouse I'm an amateur.

I think your script wil much more helpfull once I get it to work, so thanks for the effort.
  
Back to top
 
IP Logged
 
Jonathan003
YaBB Newbies
*
Offline


I Love ChessPublishing!

Posts: 22
Location: Belgium
Joined: 01/29/20
Gender: Male
Re: How to clean multiple pgn files in a folder all at
Reply #2 - 08/29/23 at 08:49:04
Post Tools
Thank you verry much!
I need some help running the script becouse I'm complete amateur in this regard.
I created a new folder to try the script C:\Users\creme\Documents\my pgns
I opened Notepad and saved the text in pgn-clean-bat.txt
I saved the text file in  C:\Users\creme\Documents\my pgns

I changed the name of the text file to pgn-clean.txt and afterwards I changed the extention to .bat
For simplicity I use the same folder stucture as in your example.
And I added 3 pgn files to test it in the folder C:\Users\creme\Documents\my pgns
game1.pgn game2.pgn and game3.pgn

I created a new folder: C:\Users\creme\Documents\static\programs\bin
And I put the latest version of pgn-extract in it.

I canged "C:\Users\myusername\Documents\my pgns" to "C:\Users\creme\Documents\my pgns"

And  C:\Users\myusername\Documents\static\programs\bin\pgn-extract.exe" to "C:\Users\creme\Documents\static\programs\bin\pgn-extract.exe"

In th pgn-clean.bat text file.

Now if I go to the folder C:\Users\creme\Documents\my pgns
Tick the adress bar and type cmd, (to open the command prompt in this window).
If I now type pgn-clean.bat to try the scipt, I get the error:

C:\Users\creme\Documents\my pgns>pgn-clean.bat
Tue 08/29/2023 10:47:42.23 INFO 0 starting [C:\Users\creme\Documents\my pgns\pgn-clean.bat]
pgn-extract v22-11
Tue 08/29/2023 10:47:42.35 ERROR 703 pgn-extract does not exist [C:\Users\creme\Documents\static\programs\bin\pgn-extract.exe]
Tue 08/29/2023 10:47:42.35 INFO 0 exiting [C:\Users\creme\Documents\my pgns\pgn-clean.bat]









  
Back to top
 
IP Logged
 
an ordinary chessplayer
God Member
*****
Offline


I used to be not bad.

Posts: 1634
Location: Columbus, OH (USA)
Joined: 01/02/15
Re: How to clean multiple pgn files in a folder all at
Reply #1 - 08/29/23 at 02:17:32
Post Tools
Jonathan003 wrote on 08/28/23 at 18:38:24:
I wonder if it is possible somehow to clean multiple pgn's in a folder all at once?

Yes, and pgn-extract in a script is a good way to do it.

Jonathan003 wrote on 08/28/23 at 18:38:24:
I want a script to clean multiple pgn files in a folder, and overwrite the original pgn files with the cleaned ones, in one go.

What you are asking for is called "in-place editing". pgn-extract cannot do it, but a script can do it using a temporary file. However, it is generally considered a bad idea:
https://backreference.org/2011/01/29/in-place-editing-of-files/index.html
In your scenario, in-place editing is definitely a bad idea because all "unclean" parts of the pgn(s) will be lost, and in a worst case you may end up with an entire folder of zero-byte pgns, without any easy way to recover the originals.

That said, I wrote you a script that does most of what you asked. For any one folder you give to the script, it will run pgn-extract against all the .pgn files in the folder and save them to a (new) subfolder "clean". Options on the command-line before the folder will be passed along to pgn-extract, allowing you to keep or exclude anything that pgn-extract can keep or exclude. It's up to you to delete the originals and move the clean versions over. Beware, though -- Scid and pgn-extract handle a bad game within a pgn file differently. Don't expect cleaning with pgn-extract to give identical results to cleaning with Scid. Proceed with care, and verify your cleaned file(s) before deleting any original(s).

Caveat emptor: I tested this under Windows 10 using the latest 22.11 version of pgn-extract, but afterwards I made some cosmetic edits to the script before posting here (adding comments, removing my username, etc.). And I did those edits under Linux, without retesting. It should work as-is, but no guarantees.
Edited:
Updated pgn-clean.bat has been posted on 2023-08-29.
« Last Edit: 08/29/23 at 18:47:06 by an ordinary chessplayer »  

pgn-clean-bat.txt ( 2 KB | 19 Downloads )
Back to top
 
IP Logged
 
Jonathan003
YaBB Newbies
*
Offline


I Love ChessPublishing!

Posts: 22
Location: Belgium
Joined: 01/29/20
Gender: Male
How to clean multiple pgn files in a folder all at
08/28/23 at 18:38:24
Post Tools
I wonder if it is possible somehow to clean multiple pgn's in a folder all at once?
Now I use Scid on my Windows 11 Pc to clean the pgn's.
I open the pgn's and export the pgn's again, (I overwrite the orgignal pgn's).
I can choose in the export window of SCID If I also want to export variations ore comments.
But that's alot of work when you want to do it with multiple pgn files in a folder.
I wonder if there would be some easyer way?
I want to keep the pgn's seperated. I only want to clean them, so it dousn't give any problems when I import the pgn's in some opening training apps, ore if I want to make a bin book of one of the pgn's.
I want a script to clean multiple pgn files in a folder, and overwrite the original pgn files with the cleaned ones, in one go. And some option to keep variations or not, to keep comments or not, to keep graphical information like arrows; highlighted squars or not...
I want the pgn's to be clean so they don't give problems, in other software.
And I want the scipt to work on Windows 11.
I'm pretty sure it can be done with pgn-extract, I only have to know how.
I spend an awfully lot of time draging pgn's to SCID and exporting them again, to clean the pgn's.
So such a script would make my work much easyer.
  
Back to top
 
IP Logged
 
Bookmarks: del.icio.us Digg Facebook Google Google+ Linked in reddit StumbleUpon Twitter Yahoo