Hi Hi all guys~

Long time ago, we were playing around Talend (Data Integration (EP 3 end)) to make data integration. And we found we can set schedule on the jobs with crontab and Task scheduler but they might not be comfortable sometimes. Therefore, here is one of automated job tools.

Rundeck has capability for managing job orchestration. In other words, it is for automatic job scenarios.

Rundeck’s features are scheduling jobs, plus:

• Job logs and history
• Error handling
• Webhook supports (apply to send notifications to Slack)
• Etc.

This time, we will use Docker (Try Docker) and Git (Try Git) together to build a simple job deployment.

All steps we need to do are:

1. Build a Talend job and upload to Git repo
2. Build a Rundeck container
3. Create a job in Rundeck

Let’s go!

1. Build a Talend job and upload to Git repo

1.1 Build a job

For example, I gonna create a job displaying 10 names as below:

1.2 Push to Git repo

After that, we push the source code to Git repo. Done for first step.

1.3 App passwords

Because Rundeck operates job with user “Rundeck” not “Root” of the container, we need an app password to allow other user to access the repo.

For more information, please visit https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line

2. Build a Rundeck container

At this step, we download Docker image from this link https://hub.docker.com/r/rundeck/rundeck. Then connect Git repo from this container.

2.1 Access Container with Root user and install Git

docker exec -it -u root [docker_container_name]
root@docker $ apt-get update
root@docker $ apt-get install git-core

2.2 create a local Git repo folder

Let’s say we are going to use folder “test”

mkdir test
cd test/

2.3 clone remote repo to the folder

After run this command, the app password must be input here.

git clone [https://github.com/username/git.git]

2.4 change owner from Root to Rundeck

We have to change the owner, or we will get the error about permission denial.

chown -R rundeck [git_folder]

2.5 update remote origin

This is for connecting remote repo with the app password.

git remote set-url origin [https://username:[email protected]/username/git.git]
git remote show origin

Trying git fetch and get no errors means completed connection.

git fetch --all

3. Create a job in Rundeck

3.1 Login to Rundeck

Rundeck provides default accounts defined in the file /home/rundeck/server/config/realm.properties as the folllowings:

• username = admin
  password = admin
  role = admin, user
• username = user
  password = user
  role = user

These roles are mapped with Access Control in another configuration file. I have no explanations in this time.

When the Rundeck container started, open the browser and access http://localhost:4440. If we find this page below, login with admin account.

3.2 create a project

3.3 create a job

3.4 Download Remote repo and update source code in local repo

We are going to add steps as scripts.

We put the commands below to update local repo and the job will be fresh to run every time.

cd /home/rundeck/test/test_rundeck/
git fetch – all
git reset – hard origin/master

git fetch --all is for retrieving all source code and git reset --hard origin/master is for cancelling all changes in local repo and keep only source code from remote repo.

Finally, we add step to run Talend source code.

sh /home/rundeck/test/test_rundeck/sample_job01_0.1/sample_job01/sample_job01_run.sh

3.5 More configurations and run

Set schedules, notification, and save it.

Job definitions

Additionally, we can import job definition files as the following figures. The files can be obtained by export via action button at right-handed side of the specific jobs.

This is an obvious example and I would say I am currently taking care of this sort of processes. I just build a job and place it on Git repo then the flow will automatically do the rest.

That’s all for this story. Stay tuned for the next.

Bye~