Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 53 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,56 @@
# GameDB
<div align="center">
<h1 align="center">GameDB</h1>
<h4 align="center">Jekyll static site with IGDB data</h4>
</div>

[![GitHub Workflow Status (DB)](https://img.shields.io/github/actions/workflow/status/lizardbyte/gamedb/update-db.yml.svg?branch=master&label=update%20db&logo=github&style=for-the-badge)](https://github.com/LizardByte/GameDB/actions/workflows/update-db.yml?query=branch%3Amaster)
[![Codecov](https://img.shields.io/codecov/c/gh/LizardByte/GameDB.svg?token=AG91ICECDX&style=for-the-badge&logo=codecov&label=codecov)](https://app.codecov.io/gh/LizardByte/GameDB)
<div align="center">
<a href="https://github.com/LizardByte/GameDB/actions/workflows/ci-tests.yml?query=branch%3Amaster"><img src="https://img.shields.io/github/actions/workflow/status/lizardbyte/gamedb/ci-tests.yml.svg?branch=master&label=CI&logo=github&style=for-the-badge" alt="CI"></a>
<a href="https://github.com/LizardByte/GameDB/actions/workflows/update-db.yml?query=branch%3Amaster"><img src="https://img.shields.io/github/actions/workflow/status/lizardbyte/gamedb/update-db.yml.svg?branch=master&label=update%20db&logo=github&style=for-the-badge" alt="Update DB"></a>
<a href="https://app.codecov.io/gh/LizardByte/GameDB"><img src="https://img.shields.io/codecov/c/gh/LizardByte/GameDB.svg?token=AG91ICECDX&style=for-the-badge&logo=codecov&label=codecov" alt="Codecov"></a>
</div>

This repository clones IGDB to gh-pages to be consumed by LizardByte projects, such as LizardByte/Sunshine.
---

Information from YouTube API is also added to the database for videos.
<div align="center">
<a href="https://app.lizardbyte.dev/GameDB/characters/all.json"><img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapp.lizardbyte.dev%2FGameDB%2Fstats.json&query=%24.characters&label=characters&style=for-the-badge&color=blue" alt="Characters"></a>
<a href="https://app.lizardbyte.dev/GameDB/collections/all.json"><img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapp.lizardbyte.dev%2FGameDB%2Fstats.json&query=%24.collections&label=collections&style=for-the-badge&color=blue" alt="Collections"></a>
<a href="https://app.lizardbyte.dev/GameDB/franchises/all.json"><img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapp.lizardbyte.dev%2FGameDB%2Fstats.json&query=%24.franchises&label=franchises&style=for-the-badge&color=blue" alt="Franchises"></a>
<a href="https://app.lizardbyte.dev/GameDB/games"><img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapp.lizardbyte.dev%2FGameDB%2Fstats.json&query=%24.games&label=games&style=for-the-badge&color=blue" alt="Games"></a>
<a href="https://app.lizardbyte.dev/GameDB/platforms/all.json"><img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapp.lizardbyte.dev%2FGameDB%2Fstats.json&query=%24.platforms&label=platforms&style=for-the-badge&color=blue" alt="Platforms"></a>
<a href="https://app.lizardbyte.dev/GameDB/videos"><img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapp.lizardbyte.dev%2FGameDB%2Fstats.json&query=%24.videos&label=videos&style=for-the-badge&color=blue" alt="Videos"></a>
</div>

## Overview

GameDB is a database of games, platforms, characters, collections, franchises, and videos sourced from
[IGDB](https://www.igdb.com) and published as a static JSON API via
[GitHub Pages](https://lizardbyte.github.io/GameDB). Video metadata is enriched with data from the YouTube API.

The data is intended to be consumed by [LizardByte](https://app.lizardbyte.dev) projects such as
[Sunshine](https://github.com/LizardByte/Sunshine).

## Data

The database is updated automatically on a schedule via the [update-db](.github/workflows/update-db.yml) workflow.
The generated JSON files are published to the `gh-pages` branch and served at
`https://app.lizardbyte.dev/GameDB/`.

| Endpoint | Description | URL |
|-------------|-------------------------------------------------------|-----------------------------------------------------------|
| Buckets | Game name search index, split by first two characters | `https://app.lizardbyte.dev/GameDB/buckets/<bucket>.json` |
| Characters | Individual character details and all characters | `https://app.lizardbyte.dev/GameDB/characters/<id>.json` |
| Collections | Individual collection details and all collections | `https://app.lizardbyte.dev/GameDB/collections/<id>.json` |
| Franchises | Individual franchise details and all franchises | `https://app.lizardbyte.dev/GameDB/franchises/<id>.json` |
| Games | Individual game details (no aggregate `all.json`) | `https://app.lizardbyte.dev/GameDB/games/<id>.json` |
| Platforms | Individual platform details and all platforms | `https://app.lizardbyte.dev/GameDB/platforms/<id>.json` |
| Videos | Individual YouTube video metadata | `https://app.lizardbyte.dev/GameDB/videos/<id>.json` |
| Stats | Total item counts per category | `https://app.lizardbyte.dev/GameDB/stats.json` |

`all.json` files (e.g. `characters/all.json`) contain a summary of every item in that category as a single
dictionary keyed by ID.

Buckets are used as a lightweight search index for game names. Each bucket file is named after the first two
alphanumeric characters of the game name (lowercased), e.g. `ha.json` for games starting with "Ha" such as
*Halo*. Games whose names contain a space as the second character are put into a bucket named after the first
character. Games whose names do not start with two alphanumeric characters are grouped into `@.json`. Each bucket
contains a dictionary of `{ id: { name } }` entries, keeping individual files small for fast lookups.
16 changes: 16 additions & 0 deletions src/update_db.py
Original file line number Diff line number Diff line change
Expand Up @@ -451,6 +451,20 @@ def _enrich_game_videos(full_dict: dict) -> None:
video['thumb'] = video_thumbs[0][1]['url']


def _write_stats(full_dict: dict) -> None:
"""
Write a ``stats.json`` file to the output directory containing item counts per category.

Parameters
----------
full_dict : dict
The combined data dictionary keyed by endpoint name.
"""
stats = {endpoint: len(items) for endpoint, items in full_dict.items()}
file_path = os.path.join(args.out_dir, 'stats')
write_json_files(file_path=file_path, data=stats)


def get_data():
"""
Get data from IGDB and YouTube.
Expand Down Expand Up @@ -614,6 +628,8 @@ def get_data():

_enrich_game_videos(full_dict=full_dict)

_write_stats(full_dict=full_dict)

for endpoint, endpoint_dict in full_dict.items():
print(f'writing individual files for {endpoint}')
for item_id, data in endpoint_dict.items():
Expand Down
19 changes: 19 additions & 0 deletions tests/unit/test_update_db.py
Original file line number Diff line number Diff line change
Expand Up @@ -445,6 +445,23 @@ def test_get_youtube(tmp_path):
assert 'yt_key_123' in called_url


def test_write_stats(tmp_path):
udb.args = _make_args(tmp_path)
full_dict = {
'characters': {1: {}, 2: {}, 3: {}},
'games': {10: {}, 20: {}},
'videos': {'abc': {}, 'def': {}},
}

with patch('src.update_db.write_json_files') as mock_write:
udb._write_stats(full_dict=full_dict)

mock_write.assert_called_once()
call_kwargs = mock_write.call_args[1]
assert 'stats' in call_kwargs['file_path']
assert call_kwargs['data'] == {'characters': 3, 'games': 2, 'videos': 2}


def test_get_platform_cross_reference(tmp_path):
udb.args = _make_args(tmp_path)

Expand Down Expand Up @@ -474,6 +491,7 @@ def test_get_data_orchestration(tmp_path):
patch('src.update_db._resolve_video_groups', return_value=[['vid1']]) as m_vgroups, \
patch('src.update_db._fetch_youtube_metadata') as m_yt, \
patch('src.update_db._enrich_game_videos') as m_enrich, \
patch('src.update_db._write_stats') as m_stats, \
patch('src.update_db.write_json_files') as m_write:
udb.get_data()

Expand All @@ -484,6 +502,7 @@ def test_get_data_orchestration(tmp_path):
m_vgroups.assert_called_once()
m_yt.assert_called_once()
m_enrich.assert_called_once()
m_stats.assert_called_once()
assert m_write.call_count >= 2 # bucket files + individual files


Expand Down