Post

Let's try: Rundeck

Rundeck has capability for managing job orchestration.

Let's try: Rundeck

Hi Hi all guys~

Long time ago, we were playing around Talend (Data Integration (EP 3 end)) to make data integration. And we found we can set schedule on the jobs with crontab and Task scheduler but they might not be comfortable sometimes.

Therefore, here is one of automated job tools.


rundeck logo Ref: https://www.rundeck.com

Rundeck has capability for managing job orchestration. In other words, it is for automatic job scenarios.

Rundeck’s features are scheduling jobs, plus:

  • Job logs and history
  • Error handling
  • Webhook supports (apply to send notifications to Slack)
  • etc.

This time, we will use Docker (Try Docker) and Git (Try Git) together to build a simple job deployment.

All steps we need to do is:

  1. Build a Talend job and upload to Git repo
  2. Build a Rundeck container
  3. Create a job in Rundeck

Let’s go!


1. Prepare job

Build a Talend job and upload to Git repo

1.1 Build a job

For example, I gonna create a job displaying 10 names as below:

Talend gen row Use tRowGeneratator and apply function getFirstname() for 10 names

Talend test Test and get the correct results

talend build build it

1.2 Push to repo

After that, we push the source code to Git repo. Done for first step.

repo

1.3 App passwords

Because Rundeck operates job with user “Rundeck” not “Root” of the container, we need an app password to allow other user to access the repo.

github dev setting Click on profile picture then Setting to access this page and select Developer settings

github gen token Personal access token > Generate new token

github config token Add Note and select scope by ticking repo

github review password Got app password as this token

For more information, please visit https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line


2. Prepare Rundeck

At this step, we download Docker image from this link https://hub.docker.com/r/rundeck/rundeck. Then connect Git repo from this container.

2.1 Access Container

with Root user and install Git.

1
2
3
docker exec -it -u root [docker_container_name]
root@docker $ apt-get update
root@docker $ apt-get install git-core

prep container

2.2 Prepare folder

Create a local Git repo folder. Let’s say we are going to use folder “test”

1
2
mkdir test
cd test/

2.3 Clone

Clone remote repo to the folder. After run this command, the app password must be input here.

1
git clone [https://github.com/username/git.git]

git clone

2.4 Update privileges

We have to change the owner, or we will get the error about permission denial.

1
chown -R rundeck [git_folder]

list

change owner

2.5 Update remote origin

This is for connecting remote repo with the app password.

1
2
git remote set-url origin [https://username:[email protected]/username/git.git]
git remote show origin

git remote

Trying git fetch and get no errors means completed connection.

1
git fetch --all

git fetch


3. Create Rundeck job

3.1 Login to Rundeck

Rundeck provides default accounts defined in the file /home/rundeck/server/config/realm.properties as the folllowings:

1
2
3
4
5
6
- username = admin
  password = admin
  role = admin, user
- username = user
  password = user
  role = user

These roles are mapped with Access Control in another configuration file.

realm.properties

When the Rundeck container started, open the browser and access http://localhost:4440. If we find this page below, login with admin account.

rundeck login

3.2 create project

new project click New Project+

fill project name Name it and other descriptions

3.3 create job

menu job click Job

job details At page “Details”, fill in job name and description

job workflow At page “Workflow”, we are setting commands here

job steps Add steps here

3.4 Fetch into local

We are going to add steps as scripts.

We put the commands below to update local repo and the job will be fresh to run every time.

1
2
3
cd /home/rundeck/test/test_rundeck/
git fetch – all
git reset – hard origin/master

rundeck exec git

git fetch --all is for retrieving all source code and git reset --hard origin/master is for cancelling all changes in local repo and keep only source code from remote repo.

Finally, we add step to run Talend source code.

1
sh /home/rundeck/test/test_rundeck/sample_job01_0.1/sample_job01/sample_job01_run.sh

rundeck exec job

3.5 Config

Set schedules, notification, and save it.

rundeck add steps Done adding workflow steps

rundeck schedule set schedule

rundeck notification set notifications

rundeck test run Saved. Try Run Job Now

rundeck result Got 10 names as expected


Job definitions

Additionally, we can import job definition files as the following figures. The files can be obtained by export via action button at right-handed side of the specific jobs.

rundeck job def Job Actions > Upload Definitions to enter import page

rundeck upload job def Select a file and operations


This is an obvious example and I would say I am currently taking care of this sort of processes. I just build a job and place it on Git repo then the flow will automatically do the rest.

That’s all for this story. Stay tuned for the next.

Bye~

This post is licensed under CC BY 4.0 by the author.