Hi
In the ReadMe file, you have provided this command: python QA.py TASK DATASET_NAME NLI_MODEL DATASET_VERSION
But I can't see any file QA.py in the repo. can you please provide that file?
Instead, I ran the following command: python main.py "EntailmentBank" "full_validity" "LLaMa3" "neg"
But I got errors in every instance of the EntailmentBank-Neg version.
Eg:- The first instance in the file results/reasoning/EB_neg_NEW-LLaMa3-05_26.jsonl (provided by the authors) doesn't show any error but I got the following error (when I ran the aforementioned main.py command using LLama3) for the same instance:
"logic_premises": [[], []], "logic_conclusion": ["-1", "-1"], "correspondance": [{}, {}], "errors": ["Logic Transform Error - Step n°: 0 name 'find_biggest_NP_parent' is not defined", "Logic Transform Error - Step n°: 1 name 'find_biggest_NP_parent' is not defined", ["Entailments Error - Step n°: 0 'str' object has no attribute 'get_variables_implications'", "Entailments Error - Step n°: 1 'str' object has no attribute 'get_variables_implications'"]], "entailments_dict": [{}, {}], "predicted_steps": [false, false]
Also, can you please tell how to calculate precision, recall, F0.5, and Somer's D using the json file in the directory results/reasoning/?
Hi
In the ReadMe file, you have provided this command:
python QA.py TASK DATASET_NAME NLI_MODEL DATASET_VERSIONBut I can't see any file QA.py in the repo. can you please provide that file?
Instead, I ran the following command:
python main.py "EntailmentBank" "full_validity" "LLaMa3" "neg"But I got errors in every instance of the
EntailmentBank-Negversion.Eg:- The first instance in the file
results/reasoning/EB_neg_NEW-LLaMa3-05_26.jsonl(provided by the authors) doesn't show any error but I got the following error (when I ran the aforementionedmain.pycommand using LLama3) for the same instance:"logic_premises": [[], []], "logic_conclusion": ["-1", "-1"], "correspondance": [{}, {}], "errors": ["Logic Transform Error - Step n°: 0 name 'find_biggest_NP_parent' is not defined", "Logic Transform Error - Step n°: 1 name 'find_biggest_NP_parent' is not defined", ["Entailments Error - Step n°: 0 'str' object has no attribute 'get_variables_implications'", "Entailments Error - Step n°: 1 'str' object has no attribute 'get_variables_implications'"]], "entailments_dict": [{}, {}], "predicted_steps": [false, false]Also, can you please tell how to calculate precision, recall, F0.5, and Somer's D using the json file in the directory
results/reasoning/?