Register Ingester OC

Register Ingester OC is a data ingester for the OpenOwnership Register project. It processes bulk data published by OpenCorporates, and ingests records into Elasticsearch. It uses raw records only, and doesn't do any conversion into the Beneficial Ownership Data Standard (BODS) format.

Installation

Install and boot Register.

Configure your environment using the example file:

cp .env.example .env

Create the Elasticsearch indexes:

docker compose run ingester-oc create-indexes

Create the AWS Athena tables:

docker compose run ingester-oc create-tables add_ids
docker compose run ingester-oc create-tables alt_names
docker compose run ingester-oc create-tables companies

Testing

Run the tests:

docker compose run ingester-oc test

Usage

Find the directory relating to the data to download, e.g. 2023-10-01. This is then used in subsequent commands.

Decide on which bulk data file source to be ingested, e.g. companies. The options are:

• add_ids
• alt_names
• companies

There are now two options:

• ingest by using the helper script
• ingest by running the commands step-by-step

Helper script

To ingest the bulk data (published monthly):

docker compose run ingester-oc ingest-bulk 2023-10-01

Step-by-step

Download the bulk data file from OpenCorporates via SFTP (enter the password when prompted):

docker compose run ingester-oc download-from-oc companies /oc-sftp-prod/open_ownership/2023-10-01 storage/oc_file_companies

Split and upload file in Gzipped parts to AWS S3:

docker compose run ingester-oc upload-split-bulk-data companies 2023_10 storage/oc_file_companies

Convert OpenCorporates bulk data using AWS Athena:

docker compose run ingester-oc convert-oc-data companies 2023_10

Export OpenCorporates bulk data for ingestion into Elasticsearch:

docker compose run ingester-oc export-oc-data companies 2023_10

Ingest AWS S3 exported files into Elasticsearch:

docker compose run ingester-oc ingest-into-es companies 2023_10