Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
fe31d3e
Update .gitignore to ignore SSH private key files (*.pem)
saurabh3460 Jan 24, 2025
d254062
Add Jenkins test infrastructure and Terraform configuration
saurabh3460 Jan 24, 2025
e0a2822
Add README for Jenkins health codebundle testing instructions
saurabh3460 Jan 24, 2025
dc64a51
Update Jenkins server provisioning with Ubuntu 22.04, automated plugi…
saurabh3460 Jan 27, 2025
3dbfdbc
Add Jenkins failed build log retrieval script
saurabh3460 Jan 27, 2025
c3efa10
Add long-running Jenkins builds detection script
saurabh3460 Jan 27, 2025
15897af
Add Jenkins health runbook with failed builds and long-running builds…
saurabh3460 Jan 27, 2025
765fee5
Add Jenkins health SLI Robot Framework task for monitoring build health
saurabh3460 Jan 27, 2025
dd990d4
update faild_build_logs json
saurabh3460 Feb 4, 2025
ec0ae66
rename name to job_name in logn_running_builds.sh
saurabh3460 Feb 4, 2025
39773a7
add utils python script
saurabh3460 Feb 4, 2025
dbc41ad
Install docker in ec2 while installing jenkins
saurabh3460 Feb 4, 2025
287c02a
added folwing tasks in sli.robot:
saurabh3460 Feb 4, 2025
84c188c
added folwing tasks in runbook.robot:
saurabh3460 Feb 4, 2025
3e53b0a
add root README.md
saurabh3460 Feb 4, 2025
1206720
Update README.md with Jenkins configuration and pipeline examples
saurabh3460 Feb 4, 2025
3ab5c8e
wip
saurabh3460 Feb 4, 2025
99138e3
Add function to parse Jenkins Atom feed and extract logs
saurabh3460 Feb 5, 2025
494cf33
Add keyword to list Jenkins logs from Atom feed
saurabh3460 Feb 5, 2025
5160704
jenkins modifications. auto-generate password. remove nc dependency. …
stewartshea Feb 13, 2025
b860482
update template
stewartshea Feb 17, 2025
315ad58
update template
stewartshea Feb 17, 2025
c66c4eb
test tag match
stewartshea Feb 17, 2025
79eb612
try just tags
stewartshea Feb 17, 2025
e770200
debug tag matching further
stewartshea Feb 17, 2025
2d4e5e6
debug tag match
stewartshea Feb 17, 2025
d648746
try tag-values
stewartshea Feb 17, 2025
712e3eb
dump custom
stewartshea Feb 17, 2025
9ea2824
remove additional context in slx
stewartshea Feb 17, 2025
3a7da5d
remove var from alias
stewartshea Feb 17, 2025
f9f3288
rename templates and basename
stewartshea Feb 17, 2025
bb55e32
fix file typo
stewartshea Feb 17, 2025
b2abed3
fix template name
stewartshea Feb 17, 2025
6e727cc
add sli
stewartshea Feb 17, 2025
20ca81c
change alias
stewartshea Feb 17, 2025
22af470
add secrets upload for jenkins to taskfile
stewartshea Feb 17, 2025
a5f463d
fix template
stewartshea Feb 17, 2025
916befa
fix secret key
stewartshea Feb 17, 2025
42cd16a
update templates
stewartshea Feb 17, 2025
85b327d
moves utils.py into proper jenkins keyword with platform secret handling
stewartshea Feb 17, 2025
d407551
wip, automating the job defs
stewartshea Feb 17, 2025
a5202bf
increase timeout
stewartshea Feb 17, 2025
2e4ef65
add new jobs, plugins, and test varifying that the ec2 instance is ru…
stewartshea Feb 17, 2025
1c10324
try new property
stewartshea Feb 17, 2025
b254f88
debug resource match
stewartshea Feb 17, 2025
4330651
x
stewartshea Feb 17, 2025
7f44b37
x
stewartshea Feb 17, 2025
c3f2901
debug
stewartshea Feb 17, 2025
600a592
debug
stewartshea Feb 17, 2025
143fa95
x
stewartshea Feb 17, 2025
20147c9
x
stewartshea Feb 17, 2025
1931395
fix plugin installation
stewartshea Feb 18, 2025
1ae118c
redact sensitive information in parse_atom_feed
saurabh3460 Feb 19, 2025
2bf9c53
update Jenkins health checks to include instance name in task descrip…
saurabh3460 Feb 19, 2025
3af8102
update Jenkins health check tasks to include instance name in descrip…
saurabh3460 Feb 19, 2025
b5b96a4
jenkins cb: update path to robot file in sli template
saurabh3460 Feb 19, 2025
4e434e9
Jenkins hlt cb: add error patterns and suggestions
saurabh3460 Feb 19, 2025
01b6da0
add log analysis feature to suggest next steps for Java/Maven errors
saurabh3460 Feb 19, 2025
7c4bd36
update Jenkins runbook to analyze logs and suggest next steps for fai…
saurabh3460 Feb 19, 2025
5a72a1c
update runbook to use relative paths for bash scripts and adjust patt…
saurabh3460 Feb 19, 2025
e3c1717
update Jenkins health checks to use backticks for instance name variable
saurabh3460 Feb 19, 2025
b34eed7
jenkins hlt cb: use relative paths for bash scripts in sli
saurabh3460 Feb 19, 2025
891626b
add error patterns and suggestions for common Python exceptions
saurabh3460 Feb 19, 2025
2ea4552
increase maximum attempts for Jenkins readiness check from 30 to 100
saurabh3460 Feb 28, 2025
30b592f
add error patterns and suggestions for common Jenkins and Docker issues
saurabh3460 Feb 28, 2025
67fb87c
refactor suggestion handling in Jenkins error patterns to use a set f…
saurabh3460 Feb 28, 2025
13a136c
add Jenkins pipeline configuration for Python Docker project
saurabh3460 Feb 28, 2025
b3e8b5f
add Docker and Git plugin installation to Jenkins setup
saurabh3460 Feb 28, 2025
a420297
update Docker service configuration to use TCP and systemd overrides
saurabh3460 Mar 3, 2025
e07ef6b
enhance log analysis by normalizing input and prioritizing error cont…
saurabh3460 Mar 4, 2025
d2c93f2
refactor error patterns in Jenkins health checks to remove outdated s…
saurabh3460 Mar 4, 2025
dac2d47
improve error reporting in Jenkins health checks by enhancing suggest…
saurabh3460 Mar 4, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ build
*egg-info
.kube
*.out
# Ignore SSH private key files
*.pem

# Ignore Terraform state files
**/terraform.tfstate
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
apiVersion: runwhen.com/v1
kind: GenerationRules
spec:
platform: aws
generationRules:
- resourceTypes:
- ec2_instance
matchRules:
- type: and
matches:
- type: pattern
pattern: "jenkins-server"
properties: [tag-values]
mode: substring
- type: pattern
pattern: "running"
properties: [state/Name]
mode: substring

slxs:
- baseName: jenkins-instance-health
levelOfDetail: detailed
qualifiers: ["resource"]
baseTemplateName: jenkins-instance-health
outputItems:
- type: slx
- type: sli
- type: runbook
templateName: jenkins-instance-health-taskset.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
apiVersion: runwhen.com/v1
kind: ServiceLevelIndicator
metadata:
name: {{slx_name}}
labels:
{% include "common-labels.yaml" %}
annotations:
{% include "common-annotations.yaml" %}
spec:
displayUnitsLong: OK
displayUnitsShort: ok
locations:
- {{default_location}}
description: The number of failed Jenkins builds.
codeBundle:
{% if repo_url %}
repoUrl: {{repo_url}}
{% else %}
repoUrl: https://github.com/runwhen-contrib/rw-cli-codecollection.git
{% endif %}
{% if ref %}
ref: {{ref}}
{% else %}
ref: main
{% endif %}
pathToRobot: codebundles/jenkins-health/sli.robot
intervalStrategy: intermezzo
intervalSeconds: 600
configProvided:
- name: JENKINS_URL
value: {{custom.jenkins_url}}
secretsProvided:
- name: JENKINS_USERNAME
workspaceKey: {{custom.jenkins_username}}
- name: JENKINS_TOKEN
workspaceKey: {{custom.jenkins_token}}
alerts:
warning:
operator: <
threshold: '1'
for: '20m'
ticket:
operator: <
threshold: '1'
for: '40m'
page:
operator: '=='
threshold: '0'
for: ''
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
apiVersion: runwhen.com/v1
kind: ServiceLevelX
metadata:
name: {{slx_name}}
labels:
{% include "common-labels.yaml" %}
annotations:
{% include "common-annotations.yaml" %}
spec:
imageURL: https://storage.googleapis.com/runwhen-nonprod-shared-images/icons/Jenkins_logo.svg
alias: {{match_resource.resource.tags.Name}} Instance Health
# alias: Jenkins Health
asMeasuredBy: The number of failed Jenkins builds.
configProvided:
- name: SLX_PLACEHOLDER
value: SLX_PLACEHOLDER
owners:
- {{workspace.owner_email}}
statement: The number of failed Jenkins builds should be zero.
additionalContext: []
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
apiVersion: runwhen.com/v1
kind: Runbook
metadata:
name: {{slx_name}}
labels:
{% include "common-labels.yaml" %}
annotations:
{% include "common-annotations.yaml" %}
spec:
location: {{default_location}}
codeBundle:
{% if repo_url %}
repoUrl: {{repo_url}}
{% else %}
repoUrl: https://github.com/runwhen-contrib/rw-cli-codecollection.git
{% endif %}
{% if ref %}
ref: {{ref}}
{% else %}
ref: main
{% endif %}
pathToRobot: codebundles/jenkins-health/runbook.robot
configProvided:
- name: JENKINS_URL
value: {{custom.jenkins_url}}
secretsProvided:
- name: JENKINS_USERNAME
workspaceKey: {{custom.jenkins_username}}
- name: JENKINS_TOKEN
workspaceKey: {{custom.jenkins_token}}

135 changes: 135 additions & 0 deletions codebundles/jenkins-health/.test/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
### How to test this codebundle?

## Prerequisites

The following credentials and configuration are required:

- Jenkins URL
- Jenkins username
- Jenkins API token

## Configuration

**Infrastructure Deployment**

Purpose: Cloud infrastructure provisioning and management using Terraform

#### Credential Setup

Navigate to the `.test/terraform` directory and configure two secret files for authentication:

`cb.secret` - CloudCustodian and RunWhen Credentials

Create this file with the following environment variables:

```sh
export RW_PAT=""
export RW_WORKSPACE=""
export RW_API_URL="papi.beta.runwhen.com"

export JENKINS_URL=""
export JENKINS_USERNAME=""
export JENKINS_TOKEN=""
```


`tf.secret` - Terraform Deployment Credentials

Create this file with the following environment variables:

```sh
export AWS_DEFAULT_REGION=""
export AWS_ACCESS_KEY_ID=""
export AWS_SECRET_ACCESS_KEY=""
export AWS_SESSION_TOKEN="" # Optional: Include if using temporary credentials
```

#### Testing Workflow

1. Build test infra:
```sh
task build-infra
```

2. Configure Jenkins and create pipelines:

- **Initial Setup**: Follow the Jenkins UI prompts to install suggested plugins.

- **Reproducing Scenarios**:

- **Failed Pipeline Logs**:
Create a `Freestyle project` and choose the `Execute shell` option under `Build Steps` with an arbitrary script that will fail, such as a syntax error.

- **Long Running Pipelines**:
Create a `Freestyle project` and choose the `Execute shell` option under `Build Steps`. Use the following script:
```sh
#!/bin/bash

# Print the start time
echo "Script started at: $(date)"

# Sleep for 30 minutes (1800 seconds)
sleep 1800

# Print the end time
echo "Script ended at: $(date)"
```

- **Queued Builds**:
Create three `Freestyle projects` using the above long-running script. With the default Jenkins setup having two executors, triggering all three projects will result in one being queued for a long time.

- **Failed Tests**:
Create a `Pipeline` project and under the Definition section, paste the following Groovy script:
```groovy
pipeline {
agent any

tools {
// Install the Maven version configured as "M3" and add it to the path.
maven "M3"
}

stages {
stage('Build') {
steps {
// Get some code from a GitHub repository
git 'https://github.com/saurabh3460/simple-maven-project-with-tests.git'

// Run Maven on a Unix agent.
sh "mvn -Dmaven.test.failure.ignore=true clean package"

// To run Maven on a Windows agent, use
// bat "mvn -Dmaven.test.failure.ignore=true clean package"
}

post {
// If Maven was able to run the tests, even if some of the test
// failed, record the test results and archive the jar file.
success {
junit '**/target/surefire-reports/TEST-*.xml'
archiveArtifacts 'target/*.jar'
}
}
}
}
}
```


3. Generate RunWhen Configurations
```sh
tasks
```

4. Upload generated SLx to RunWhen Platform

```sh
task upload-slxs
```

5. At last, after testing, clean up the test infrastructure.

```sh
task clean
```

Loading