Pull Request for NRNB issue 238: Using Foundation model for GRN Inference

[FaizalJnu]

Changes made and files added:

BLEval folder:

1. init.py file:

InputSettings class

• Added use_embeddings parameter to __init__ method.
• Implemented adjust_paths_for_embeddings method to modify file paths when using embeddings.
• Added get_true_edges_path method for retrieving the correct path to true edges file.

OutputSettings class

• Added datasets and use_embeddings parameters to __init__ method.
• Implemented adjust_paths_for_embeddings method to modify output paths when using embeddings.
• Added get_output_path method for constructing output file paths.

BLEval class

• Updated computeAUC method to support embeddings and use the new get_true_edges_path method.
• Updated computeEarlyPrec method to support the embeddings generation by passing self.input_settings into the EarlyPrec() function as well.

ConfigParser class

• Added use_embeddings parameter to parse method
• Updated __parse_input_settings and __parse_output_settings methods to support embeddings

Other improvements

• Updated import statements to use from pathlib import Path consistently.
• Replaced string concatenation with Path objects for better cross-platform compatibility.
• Added type hints to some method parameters.
• Removed unused imports and commented-out code.
• Improved code formatting and docstring consistency.

2. computeDGAUC file:

In PRROC function

• Introduced a new condition to handle cases where embeddings are used.
• Updated outDir path construction to use Path objects.
• Enhanced file existence checking.
• Updated file reading to use the new ranked_edges_path.
• Improved error message formatting using f-strings.
• Updated PRName and ROCName to remove leading slashes.

3. computeEarlyPrec file:

In EarlyPrec() function

• Added an if/else statement to use the correct paths in case of embeddings.

Unchanged elements

• The core computeScores and computeEarlyPrec function call remains the same.
• The structure of the if directed: and else: blocks is maintained.

Blevaluator.py file:

In config get_parser() function:

• Added parsing arguement for recognizing '--use_embeddings' flag.

In main() function:

• Post-Run Evaluation Initialization: Initialization of the evaluation summarizer (evalSummarizer) using the parsed input and output settings. This replaces any hardcoded settings with dynamic configuration, allowing for greater flexibility and adaptability.
  Output Directory Construction:

• The output directory is now dynamically constructed based on the evaluation settings (evalSummarizer.output_settings.base_dir, evalSummarizer.input_settings.datadir, and evalSummarizer.output_settings.output_prefix). This change ensures that the output files are stored in a well-organized directory structure that reflects the evaluation parameters.

Unchanged Logic:
The core functionality for computing and saving the AUPRC and AUROC values remains unchanged. The logic has been preserved but now exists within a more structured and configurable context.

BLRun Folder:

1. generate_embeds.py:

• This file is an entirely new addition that runs as the other algorithmRunner files.
• It calls upon a dockerised script and model to generate embeddings.

2. runner.py:

• The addition of the self.use_embeddings attribute in the class constructor enables conditional embedding generation based on user parameters (params.get('use_embeddings', False)). This makes the use of embeddings optional and configurable, providing flexibility depending on the dataset and the specific needs of the user.
• self.embeddings_file is introduced to store the path where the generated embeddings will be saved. This is initialized as None and later assigned a value during the embedding generation process.

In generate_embeds method:

• This method is the centerpiece of the update, responsible for generating embeddings by invoking an external script (generate_embeds.py). Key steps include:
• Path Construction: The script dynamically constructs paths for the embeddings file and the new input directory based on the provided expression data file (self.exprData).
  Subprocess Execution: The script is executed via a subprocess command, with robust error handling to ensure that any issues are clearly reported.
• File Management: Upon successful generation, the embeddings file is moved to the specified directory, and a reference network file (refNetwork.csv) is copied to the new directory if it exists. This ensures that all necessary files are in place for subsequent processing.
• Directory Update: The input directory (self.inputDir) and expression data file (self.exprData) are updated to reflect the newly generated embeddings, ensuring that the workflow continues with the correct data.

- The generateInputs() method has been updated to call generate_embeddings() when self.use_embeddings is True. This ensures that the embeddings are generated before any other input processing steps are performed.
- The run() method has been updated to ensure that the output directory is created based on the processed input directory, maintaining a clear separation of inputs and outputs.

BLRunner.py file:

The get_parser() function now includes a new command-line argument --use_embeddings, which allows users to specify whether embeddings should be used in the pipeline. This argument is a boolean flag (action='store_true'), meaning that it will default to False unless explicitly provided by the user.

The parse_arguments() function has been updated to return the parsed arguments, including the newly added use_embeddings flag. This function now serves as the entry point for all command-line configurations, ensuring that the user's preferences are respected throughout the pipeline.

Model folder:

generate_embeddings.py:

• This new file introduces a comprehensive script for generating gene embeddings using a pre-trained Transformer model, enhancing the pipeline's ability to process expression data with advanced machine learning techniques.
• This function reads gene expression data from a CSV file, tokenizes the gene names based on the loaded vocabulary, and generates embeddings using the Transformer model's encoder.
• The generated embeddings are then stored in a dictionary, converted into a DataFrame, and saved to a CSV file (EmbeddingsData.csv). This output can be used for downstream analysis in the pathway reconstruction pipeline.

Dockerfile and req.yaml added to support a containerised environment for the embeddings generation pipeline.

Initialize.sh

• A bit of script is added here to initialize the docker container and pull the latest image of the model that we will be using for the embeddings generation.
• No other changes are made here.

ReadMe file

• ReadMe has been updated to represent the flow mentioned below:

CORE FLOW:

The pipeline, if run with the command

python BLRunner.py --config config-files/config.yaml --use_embeddings

will

1. make a dir named processedExpressionData in the dataset directory defined in config.yaml file.
2. For example, if expression data path was: inputs/example/GSD/ExpressionData.csv , Embeddings data will be generated and saved at inputs/example/GSD/processedExpressionData/EmbeddingsData.csv and the refNetwork.csv file will automatically be copied and saved in the processedExpressionData directory.
3. Now the GRN Inference will take place on this EmbeddingsData.csv file instead of the ExpressionData.csv file.

AND, if the command used is:

python BLEvaluator.py --config config-files/config.yaml --auc --use_embeddings

Both auc and epr pipelines have been modified to be used along with the use_embeddings flag

The Evaluation pipeline will run on the GRNs we Inferred using EmbeddingsData.csv

• input and output dir will be set to inputs/example/GSD/processedExpressionData and outputs/example/GSD/processedExpressionData respectfully.
• Score files generated will be saved here.