When it comes to complex transformations, we would design our flows to be more organized and clean.

We can generate a DAG in visual figure using a few steps.

Apache Beam is a tool from Apache for processing data, both batch and real-time basis.

Yes we just add a code to command Terminal to display mapped colors from the code.

In every single software development, there is a chance the program runs into an error.

2 simple stuff we can improve our source code to become a good self-explained source code.

This standard helps us communicate between source and consumer in order to develop the system a lot easier.

When call jq and supply a valid JSON data with a JSON path we want, we will obtain the expected result at ease.

Most of necessary, popular, or essential packages (and programs) can be found here.

We're gonna wrap up all basic part 1 to 5 and make it automated by using our CI/CD tool, Google Cloud Build