Let's try: Rundeck
Rundeck has capability for managing job orchestration.
Hi Hi all guys~
Long time ago, we were playing around Talend (Data Integration (EP 3 end)) to make data integration. And we found we can set schedule on the jobs with crontab and Task scheduler but they might not be comfortable sometimes.
Therefore, here is one of automated job tools.
Rundeck has capability for managing job orchestration. In other words, it is for automatic job scenarios.
Rundeck’s features are scheduling jobs, plus:
- Job logs and history
- Error handling
- Webhook supports (apply to send notifications to Slack)
- etc.
This time, we will use Docker (Try Docker) and Git (Try Git) together to build a simple job deployment.
All steps we need to do is:
- Build a Talend job and upload to Git repo
- Build a Rundeck container
- Create a job in Rundeck
Let’s go!
1. Prepare job
Build a Talend job and upload to Git repo
1.1 Build a job
For example, I gonna create a job displaying 10 names as below:
Use tRowGeneratator and apply function getFirstname() for 10 names
Test and get the correct results
build it
1.2 Push to repo
After that, we push the source code to Git repo. Done for first step.
1.3 App passwords
Because Rundeck operates job with user “Rundeck” not “Root” of the container, we need an app password to allow other user to access the repo.
Click on profile picture then Setting to access this page and select Developer settings
Personal access token > Generate new token
Add Note and select scope by ticking repo
Got app password as this token
For more information, please visit https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line
2. Prepare Rundeck
At this step, we download Docker image from this link https://hub.docker.com/r/rundeck/rundeck. Then connect Git repo from this container.
2.1 Access Container
with Root user and install Git.
1
2
3
docker exec -it -u root [docker_container_name]
root@docker $ apt-get update
root@docker $ apt-get install git-core
2.2 Prepare folder
Create a local Git repo folder. Let’s say we are going to use folder “test”
1
2
mkdir test
cd test/
2.3 Clone
Clone remote repo to the folder. After run this command, the app password must be input here.
1
git clone [https://github.com/username/git.git]
2.4 Update privileges
We have to change the owner, or we will get the error about permission denial.
1
chown -R rundeck [git_folder]
2.5 Update remote origin
This is for connecting remote repo with the app password.
1
2
git remote set-url origin [https://username:app_password@github.com/username/git.git]
git remote show origin
Trying git fetch and get no errors means completed connection.
1
git fetch --all
3. Create Rundeck job
3.1 Login to Rundeck
Rundeck provides default accounts defined in the file /home/rundeck/server/config/realm.properties as the folllowings:
1
2
3
4
5
6
- username = admin
password = admin
role = admin, user
- username = user
password = user
role = user
These roles are mapped with Access Control in another configuration file.
When the Rundeck container started, open the browser and access http://localhost:4440. If we find this page below, login with admin account.
3.2 create project
click New Project+
Name it and other descriptions
3.3 create job
click Job
At page “Details”, fill in job name and description
At page “Workflow”, we are setting commands here
Add steps here
3.4 Fetch into local
We are going to add steps as scripts.
We put the commands below to update local repo and the job will be fresh to run every time.
1
2
3
cd /home/rundeck/test/test_rundeck/
git fetch – all
git reset – hard origin/master
git fetch --all is for retrieving all source code and git reset --hard origin/master is for cancelling all changes in local repo and keep only source code from remote repo.
Finally, we add step to run Talend source code.
1
sh /home/rundeck/test/test_rundeck/sample_job01_0.1/sample_job01/sample_job01_run.sh
3.5 Config
Set schedules, notification, and save it.
Done adding workflow steps
set schedule
set notifications
Saved. Try Run Job Now
Got 10 names as expected
Job definitions
Additionally, we can import job definition files as the following figures. The files can be obtained by export via action button at right-handed side of the specific jobs.
Job Actions > Upload Definitions to enter import page
Select a file and operations
This is an obvious example and I would say I am currently taking care of this sort of processes. I just build a job and place it on Git repo then the flow will automatically do the rest.
That’s all for this story. Stay tuned for the next.
Bye~