Skip to content

MIDRC-749 Add disk space configuration to AWS batch nodes#123

Open
jawadqur wants to merge 5 commits into
masterfrom
fix/batch-storage
Open

MIDRC-749 Add disk space configuration to AWS batch nodes#123
jawadqur wants to merge 5 commits into
masterfrom
fix/batch-storage

Conversation

@jawadqur
Copy link
Copy Markdown
Contributor

Link to JIRA ticket if there is one:

New Features

Breaking Changes

Bug Fixes

Improvements

  • Added 50 GB disk to nexflow AWS batch workers.

Dependency updates

Deployment changes

@github-actions
Copy link
Copy Markdown

filepath $$\textcolor{#23d18b}{\tt{passed}}$$ SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_discoverypage.py}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_workspace.py}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$ $$\textcolor{#23d18b}{\tt{2}}$$ $$\textcolor{#23d18b}{\tt{2}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here

@jawadqur jawadqur changed the title Add 50gb disk to batch nodes. Add disk space configuration to AWS batch nodes Apr 14, 2025
Comment thread hatchery/config.go
@github-actions
Copy link
Copy Markdown

filepath $$\textcolor{#23d18b}{\tt{passed}}$$ $$\textcolor{#f14c4c}{\tt{failed}}$$ SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_discoverypage.py}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#f14c4c}{\tt{tests/test\_workspace.py}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#f14c4c}{\tt{1}}$$ $$\textcolor{#f14c4c}{\tt{1}}$$
$$\textcolor{#f14c4c}{\tt{TOTAL}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#f14c4c}{\tt{1}}$$ $$\textcolor{#f14c4c}{\tt{2}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here

@paulineribeyre paulineribeyre changed the title Add disk space configuration to AWS batch nodes MIDRC-749 Add disk space configuration to AWS batch nodes Apr 14, 2025
Comment thread hatchery/nextflow.go
if len(launchTemplate.LaunchTemplates) == 1 {
// TODO: Make sure user data in the existing launch template matches the user data
// we want (MIDRC-749)
// Check if the launch template has block device mappings
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a more generic way to compare the existing launch template to the one in the code? instead of only checking BlockDeviceMappings?

Also, this code wouldn't catch a difference in the BlockDeviceMappings definition, for example if we change the VolumeType, since we only check if there are BlockDeviceMappings at all

Maybe it's fine since this should be deprecated soon, but if we leave it as-is we should write a comment

Comment thread hatchery/nextflow.go
Comment on lines +637 to +638
LaunchTemplateData: &ec2.RequestLaunchTemplateData{
UserData: ltVersion.LaunchTemplateData.UserData, // Preserve existing user data
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not update it since the function already generated userData anyway?

Comment thread hatchery/nextflow.go
LaunchTemplateName: aws.String(launchTemplateName),
SourceVersion: aws.String(ltVersionStr),
VersionDescription: aws.String("Added block device mappings"),
LaunchTemplateData: &ec2.RequestLaunchTemplateData{
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we reuse instead of duplicating the RequestLaunchTemplateData definition?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants