Latest Updates:
Hot Topic (More than 10 Replies) World largest offline chess database. (Read 6881 times)
katar
Senior Member
****
Offline


look another year went
by

Posts: 455
Location: LA
Joined: 09/21/05
Gender: Male
Re: World largest offline chess database.
Reply #10 - 12/30/15 at 21:49:00
Post Tools
proustiskeen wrote on 12/30/15 at 14:22:16:
Tell you what. Why don't you try putting Big Database 2016 (no annotations, so no authorial problems) on a torrent and letting Chessbase know that you did so? Shouldn't be a problem, since they're only data compilers.

I'm not a lawyer. I don't know if or how copyright works with compilation. What I do know is that what the OP is doing is ethically murky and, imho, wrong. Crowther is struggling to get by while having offered the chess world a great service for 20 years now, and this guy is taking Crowther's work and repackaging it without permission? For shame.


Chessbase's Big Database may have thin copyright protection as a compilation assuming its selection criteria are minimally creative, even though the individual elements (games) are not protected.  Thin copyright protection basically protects only against verbatim or literal copying.  So uploading a verbatim copy of Chessbase's product would probably be both a copyright violation as well as a breach of Chessbase's terms of use (whcih is a matter of contract rather than copyright).

Facts (like raw chess game scores) are not copyrighted, though the particular expression may be copyrighted.  Volunteers who cull facts from dozens of sources to compile and repackage those facts in, say, a Wikipedia article contribute to the public storehouse of accessible information or knowledge to the benefit of a global audience.  Is that shameful, really?
  

2078 uscf
Back to top
WWW  
IP Logged
 
proustiskeen
YaBB Moderator
*****
Offline


Hello from Omaha!

Posts: 678
Joined: 08/11/08
Re: World largest offline chess database.
Reply #9 - 12/30/15 at 14:22:16
Post Tools
Tell you what. Why don't you try putting Big Database 2016 (no annotations, so no authorial problems) on a torrent and letting Chessbase know that you did so? Shouldn't be a problem, since they're only data compilers.

I'm not a lawyer. I don't know if or how copyright works with compilation. What I do know is that what the OP is doing is ethically murky and, imho, wrong. Crowther is struggling to get by while having offered the chess world a great service for 20 years now, and this guy is taking Crowther's work and repackaging it without permission? For shame.
  
Back to top
WWW  
IP Logged
 
katar
Senior Member
****
Offline


look another year went
by

Posts: 455
Location: LA
Joined: 09/21/05
Gender: Male
Re: World largest offline chess database.
Reply #8 - 12/30/15 at 06:13:24
Post Tools
proustiskeen wrote on 12/24/15 at 04:51:23:
I'm not sure of the legality of taking the work of others (TWIC and ChessOK, notably) and then releasing them under your own name, even if it's open-source at SourceForge. I'm even less sure of the ethics of it.

It's perfectly legal, provided the games contain no variations, comments, or symbols (like !?=+- etc.).  A massive database of all games in existence from any source does not raise compilation copyright issues where there is no minimally creative "selection" or "arrangement" of the individual games.  This topic has been exhausted in other threads.

The game scores are the "work" of others like TWIC or ChessOK only in the sense that data compilation constitutes "work," as in manual labor.  TWIC and ChessOK are not authors or artists but merely data compilers.  From what i can tell, the OP is also no artist or author, but just a data compiler.  There is no copyright for hard work.
  

2078 uscf
Back to top
WWW  
IP Logged
 
Hadron
Full Member
***
Offline


Doctor, Doctor, Doctor..When
will you ever learn?

Posts: 195
Location: Levin, New Zealand.
Joined: 03/24/05
Gender: Male
Re: World largest offline chess database.
Reply #7 - 12/29/15 at 21:41:53
Post Tools
proustiskeen wrote on 12/24/15 at 04:51:23:
I'm not sure of the legality of taking the work of others (TWIC and ChessOK, notably) and then releasing them under your own name, even if it's open-source at SourceForge. I'm even less sure of the ethics of it.

I think your point poses a very interesting conundrum, just who owns a played game of chess?
I can see it being very easy to exercise intellectual property rights over an chess article and especially if it has already been published before (Me personally I would not take such a risk to do so. I translated a very old Kiasserber article from German into Chessbase/Broken English format and I had republishing permission turned down even though said article was 10 plus years old. I have never used it to this day)
I can even see it being quite easy to exercise intellectual property rights over ones own name in relationship to a game of chess played. In fact I believe Evgeny Sveshnikov has done so in the past, denying a tournament the right to reprint his games from that tournament without due recompense.
But on a played game itself? Without any sort of causal link to a game played how do you show you played it??
Time for a lie down...
Cheesy
  

I'm reminded again of something Short wrote recently, approximately "The biggest fallacy in chess is the quasi-religious belief in the primacy of the opening."
Back to top
 
IP Logged
 
Stefan Buecker
God Member
*****
Offline



Posts: 1386
Location: Germany
Joined: 02/11/09
Gender: Male
Re: World largest offline chess database.
Reply #6 - 12/26/15 at 23:56:52
Post Tools
proustiskeen wrote on 12/24/15 at 04:51:23:
I'm not sure of the legality of taking the work of others (TWIC and ChessOK, notably) and then releasing them under your own name, even if it's open-source at SourceForge. I'm even less sure of the ethics of it.

The OP has listed his sources. The link to ChessOk.com was http://chessok.com/?page_id=694 to a site titled "Free Game Service for Download". Similarly, I think TWIC doesn't claim to own a copyright on their games.

True, this is a tricky issue. There have been rumours about available large game collections - who may or may not have taken games from ChessBase. On the other side, once I found games from my own collection (researched in the library at The Hague) in a ChessBase collection. I had never been asked for my permission. What I did remember was that I gave these games to a "friend" - who apparently shared it with others. Why not let the companies fight it out and sue each other, if they believe they have a case.

Have the forum members boycotted "Quality Chess" when I argued that John Shaw had not credited my articles properly in his "The King's Gambit" (2013)? No.

In 2015, Quality Chess reprinted the 2013 work. Without any change.
  
Back to top
 
IP Logged
 
proustiskeen
YaBB Moderator
*****
Offline


Hello from Omaha!

Posts: 678
Joined: 08/11/08
Re: World largest offline chess database.
Reply #5 - 12/24/15 at 04:51:23
Post Tools
I'm not sure of the legality of taking the work of others (TWIC and ChessOK, notably) and then releasing them under your own name, even if it's open-source at SourceForge. I'm even less sure of the ethics of it.
  
Back to top
WWW  
IP Logged
 
Stefan Buecker
God Member
*****
Offline



Posts: 1386
Location: Germany
Joined: 02/11/09
Gender: Male
Re: World largest offline chess database.
Reply #4 - 12/23/15 at 21:29:03
Post Tools
Thanks, codekiddy, your OTB database is excellent. Searching for games with 1.b3 e5 2.Bb2 Nc6 3.Nf3 e4 4.Ne5, my first steps were MegaBase 2008 and Twic, the second step a search in the internet (e.g. 365chess and chessgames). Result: 20 games. Then came your OTB base, and it had 25 games, wow. Seven of these were "new". In the future I plan to start with Scid and build from there. It isn't superior to the ChessBase "Live" Base (25 games, including one "new"), but the latter limits pgn download to Premium customers.

Your other databases (online, engine, corr) are probably less useful for my purposes, but it is good to have them, and I'll check.
  
Back to top
 
IP Logged
 
codekiddy
YaBB Newbies
*
Offline


I Love ChessPublishing!

Posts: 7
Joined: 11/08/15
Re: World largest offline chess database.
Reply #3 - 11/11/15 at 08:20:40
Post Tools
Jupp53 wrote on 11/11/15 at 00:04:06:
Running the SCID4.6.2 maintance delete double games reduces the number of games to 13.449.988 games.

Some of the games show simply the opening. It's more about quality of the games than the pure # if you want do some work over it.

If you like you can get my revision of your db. Send me a pm in this case.

Edit adds:

Now I filtered all games out with a rating of both players below 2400. 2.2 mio games with many doubles. Filtering all games not having the same day (which is no day date), not having the same month, not having the same years leads to 0.88 mio games. Then I checked the first hundred games and found some book positions and many engine games.

You should create another db without engine games and book positions. The number of games will be less impressive. But as the rule is "garbage in, garbage out" you will get a much more useful db.


Hi, you're right about all you said and I apologize to all who wasted their bandwidth.
last few uploads were bad because I've doing some pure testings, I've made an update which should give the expected results, that is 15 Million "unique" games (more or less)

Please see this: (scroll down to see what I mean)
https://sourceforge.net/projects/codekiddy-chess/files/Databases/Update4/

proustiskeen wrote on 11/11/15 at 04:23:56:
Just, you know, out of curiousity: what was your original source for all of these games?

Here is a list

Thank you!
  
Back to top
 
IP Logged
 
proustiskeen
YaBB Moderator
*****
Offline


Hello from Omaha!

Posts: 678
Joined: 08/11/08
Re: World largest offline chess database.
Reply #2 - 11/11/15 at 04:23:56
Post Tools
Just, you know, out of curiousity: what was your original source for all of these games?
  
Back to top
WWW  
IP Logged
 
Jupp53
God Member
*****
Offline


be

Posts: 988
Location: Frankfurt/Main
Joined: 01/04/09
Gender: Male
Re: World largest offline chess database.
Reply #1 - 11/11/15 at 00:04:06
Post Tools
Running the SCID4.6.2 maintance delete double games reduces the number of games to 13.449.988 games.

Some of the games show simply the opening. It's more about quality of the games than the pure # if you want do some work over it.

If you like you can get my revision of your db. Send me a pm in this case.

Edit adds:

Now I filtered all games out with a rating of both players below 2400. 2.2 mio games with many doubles. Filtering all games not having the same day (which is no day date), not having the same month, not having the same years leads to 0.88 mio games. Then I checked the first hundred games and found some book positions and many engine games.

You should create another db without engine games and book positions. The number of games will be less impressive. But as the rule is "garbage in, garbage out" you will get a much more useful db.
« Last Edit: 11/11/15 at 01:05:08 by Jupp53 »  

Medical textbooks say I should be dead since April 2002.
Dum spiro spero. Smiley
Narcissm is the humans primary disease.
Back to top
 
IP Logged
 
codekiddy
YaBB Newbies
*
Offline


I Love ChessPublishing!

Posts: 7
Joined: 11/08/15
World largest offline chess database.
11/09/15 at 12:16:07
Post Tools
Hello guys!
I've been looking for largest chess games database for some time all over the internet, and there are people who made such databases by collecting games.

Most of these free databases are not exceeding more than 3 Million games, which is of course a lot, but I wanted larger one, and looked up for commercial solutions, and found some interesting sites offering chess databases of various sizes, such as 5 Million games or even 10 Million or more games.
However these commercial solutions cost money, and that's a big problem for people who simply can't or are not willing to pay, not only that but some websites don't even sell their databases, but instead offer online query only, therefore you have no control.
So I come up with an idea to create my own FREE largest database of games that will beat all other databases found on internet!

So I started collecting OTB games played by humans (not engines, not correspondence etc.) starting from year cca 1500 up until now, that is OTB games for last 500+ years.
I managed to collect over 30 Million games so far by downloading other people databases and PGN files from various websites, merged them all into one big database and started removing duplicate games.

So far this number reduced to some 15 Million games...
I also created a project page on sourceforge to share my super-duper-jambo database with other chess players around the globe for FREE!
I've been using Scid vs PC program to manage database and therefore the database is in scid format, meaning you can download it and do with it what ever you want, such as importing more games.

Please note that this project is currently still in progress and I'm seeking for help from other chess players to collect more games and to help maintain this database!
The plan is to make the database available to everyone and more important the more people there are willing to contribute the larger database we will have, for FREE!
I'm still not sure how could all this work, so if you're interested please either give a reply here or join discussion forum on my project page to share your ideas on how to proceed with this project.


If you're a chess novice you might ask your self:
Q: Why on the earth would anyone need a chess games database?
A: You can use database to do following things:
1. Study games played by chess masters last 500 years.
2. Study openings, mid games and endgames.
3. Create high quality opening books for use by chess engines
4. Search for games played in chess history.
5. play chess against the database (ie. to learn opening variations)
6. Create opening books for specific opening or variation and use chess engine to play that specific opening.
7. Solve puzzles and much much more!

My project page:
https://sourceforge.net/projects/codekiddy-chess/
Discusion forums:
https://sourceforge.net/p/codekiddy-chess/discussion

Thank you so much for attention!
~codekiddy
  
Back to top
 
IP Logged
 
Bookmarks: del.icio.us Digg Facebook Google Google+ Linked in reddit StumbleUpon Twitter Yahoo