Table of contents⌗

1. chessinsights.xyz
2. The chess.com API
3. Bulk Export of pgn

Chessinsights.xyz⌗

I’m one of the millions of people who took up online chess over the pandemic. Everyone has their own reasons for enjoying the game but for me it’s the fact that it’s one of the few “digital pass times” that doesn’t completely wreak my attention span. Another reason is that it’s a data driven game where you can gradually see your progress over time via the Elo rating system. Every move of every game from the moves, openings, endgames etc.. can be recorded and analysed to gather insights and improve your game.

Right now, the most popular platform for online chess is chess.com. They allow you to play unlimited games for free and keep a browsable archive of every game you’ve played. They provide at a glance statistics of a users overall games such as ELO (rating) over time and how many games the user won/lost. They also provide a premium service called insights, which gives you access to a dashboard with more detailed information about a users play style, however it costs a steep $17 a month and still lacks features I would like to see for such a price tag. After exploring the chess.com API I found that there was enough data to recreate several premium features for free and to add functionality not available on the site. I’m currently building that functionality into chessinsights.xyz. This post is about some of the methods I used to get this data from the API and integrate it into my project

The Chess.com API⌗

Before I go over what these features are and how I implemented them I think it would be helpful to go over the chess.com API. If you’re already familiar with it feel free to skip this part. Chess.com recently released a their published-data API. It’s free, requires no authentication and contains data we can use to create features missing on chess.com. I used the three following endpoints to grab the data available on chessinsights.xyz.

https://api.chess.com/pub/player/{userName}/stats⌗

This endpoint provides a JSON object with sub objects containing an overview of a users games by game mode. Chess.com has several game modes but the only ones I was interested in were the ones played with normal chess rules against real users which are the following:

If we curl this endpoint and pretty print it with the user hikaru as the username:

curl -X GET "https://api.chess.com/pub/player/hikaru/stats"  | json_pp

We’ll get something like this back:

...
...
{
   "chess_rapid" : {
      // Highest rated game, unix timestamp and url of game 
      "best" : { 
         "date" : 1645902514,
         "game" : "https://www.chess.com/game/live/61144117987",
         "rating" : 2927
      },
      "last" : { 
         "date" : 1676579962,
         "rating" : 2816,
         "rd" : 53
      },
      // total rapid games won/lost/drawn 
      "record" : { 
         "draw" : 142, 
         "loss" : 30,
         "win" : 135
      }
   }
}
...
...

https://api.chess.com/pub/player/{userName}/games/archives⌗

This endpoint returns an object with a list of urls pointing to endpoints representing every month the user has played games.

If we curl this endpoint with hikaru:

curl -X GET "https://api.chess.com/pub/player/hikaru/games/archives"  | json_pp

It will return:

{
   "archives" : [
      "https://api.chess.com/pub/player/hikaru/games/2014/01",
      "https://api.chess.com/pub/player/hikaru/games/2014/02",
      "https://api.chess.com/pub/player/hikaru/games/2014/03",
   ...

https://api.chess.com/pub/player/{userName}/games/YYYY/MM⌗

This endpoint will return a list of objects representing games played that month. We can use the previously mentioned endpoint to get a full list of these or if we want to. If the player didn’t play any games that month it will return an empty list.

Depending on how active the user is this could return a massive object so I would recommend redirecting the output to a JSON file to explore it.

curl -X GET "https://api.chess.com/pub/player/hikaru/games/2014/01" >> hikaru_2014_01.json

I removed some data from this return object to make it more readable

    "url": "https://www.chess.com/game/live/692670646",
        "pgn": "[Event \"Live Chess\"]\n n1. d4 etc...",
        "end_time": 1389052832, 
        "accuracies": { 
            "white": 93.75848310550438, 
            "black": 92.83682165258698
        },
        "uuid": "eba837bc-91e2-11de-8000-000000010001",
        "fen": "8/8/7p/7P/1p6/pBkpp2b/8/4K3 w - -", 
        "time_class": "blitz", 
        "rules": "chess",
        "white": {
            "rating": 2163,
            "result": "timeout",
            "@id": "https://api.chess.com/pub/player/godswill",
            "username": "Godswill",
        },
        "black": {
            "rating": 2438,
            "result": "win",
            "@id": "https://api.chess.com/pub/player/hikaru",
            "username": "Hikaru",
        }

Here are some data descriptions for the points might be ambiguous.

• pgn
  • specialized pgn data of the game including a url to a tutorial on the opening played.
• accuracies
  • Game engine evalueation of the players moves. only available if user has premium account and analysed the game.
• fen
  • final position of board

Missing Features⌗

I purchased the premium membership to see what I could learn and I noticed a few missing features wished were available. This is by no way intended to be criticism of chess.com, just a list of features I set out to create myself using their public API.

1. Bulk export of games to PGN⌗

A Portable Game Notation (.pgn) file is a structured file for storing data about a particular chess game. It usually look something like this:

[Site "example.com"]
[Date "YYYY.MM.DD"]
[White "White Username"]
[Black "Black Username"]
[Result "0-1"]

1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 etc..

The chess.com UI limits your exports to about 50 games at a time, making large profile exports a tedious task.

using our knowledge of the /player/{userName}/games/archives endpoint we can get a list of all /player/{userName}/games/YYYY/MM endpoints which store a list of objects that contain pgn data.

To write all of Magnus Carlsen’s games to a single pgn with python:

import requests, json

user = "MagnusCarlsen"
foo = requests.get(f"https://api.chess.com/pub/player/{user}/games/archives")
month_urls = foo.json()

all_pgn = []

for url in month_urls["archives"]:
    bar = requests.get(url)
    game_list = bar.json()
    for game in game_list["games"]:
        all_pgn.append(game['pgn'])

with open(f"{user}.pgn","w") as f:
    for i in all_pgn:
        f.write(i)

To do this in javascript I’m going to assume it’s within the context of some sort of user interface so we will need an html file with an input field and button to trigger the action

index.html

<html>

   <script scr="main.js"></script>

   <input id="userName">

   <button onclick="getAllPgn()">
      Get All PGN data
   </button>

</html>

main.js

async function getAllPgn() {
        let foo = document.getElementById("userName")
        let userName = foo.value

        archiveUrl = `https://api.chess.com/pub/player/${userName}/games/archives`;
    
        let response = await fetch(archiveUrl)
        let archiveMonths = await response.json()
        let archiveUrls = archiveMonths.archives
    
        let pgnData = "";

        for (let i=0; i<archiveUrls.length; i++) {
    
            let archive = await fetch(archiveUrls[i]);
            let archiveJson = await archive.json();
            let archiveGameList = archiveJson.games;
          
            for (let j=0; j<archiveGameList.length; j++) {
                  pgnData += archiveGameList[j].pgn + "\n";
            }
        }

        pgnFile = new Blob([pgnData], {type: 'text'})

        let link = window.document.createElement('a');
        link.href = window.URL.createObjectURL(pgnFile);

        let fname = `${userName}.pgn`;
        link.download = fname;

        document.body.appendChild(link);
        link.click();
        document.body.removeChild(link);

 }

2. Building Interactive Charts⌗

The charts available, while good at giving you an overall perspective could be great by allowing you to explore the games the chart is visualizing. The functionality to custom sort games in your archive already exists in the url parameters. For example this will search for all games in your archive where you won by checkmate and used the kings pawn opening. (You need to be signed in)

I ran out of energy to finish this post but if I don’t put it out as is, I never will finish it

…to be continued