Let's try: dbt part 10 - packages

In this series

1. Let's try: dbt part 1 - prologue
2. Let's try: dbt part 2 - first start
3. Let's try: dbt part 3 - seed and source
4. Let's try: dbt part 4 - model materialization
5. Let's try: dbt part 5 - from Jinja to macros and hooks
6. Let's try: dbt part 6 - snapshots and analyses
7. Let's try: dbt part 7 - tests
8. Let's try: dbt part 8 - DAGs and docs
9. Let's try: dbt part 9 - variables
10. Let's try: dbt part 10 - packages

dbt community is one of big factors that makes developers love dbt. Now we can find third-party packages to improve productivity and development velocity.

---

dbt packages

Packages are external dbt projects we can install into our own projects and reuse for various functionalities. Here is the dbt packages hub homepage.

dbt - Package hub

hub.getdbt.com

And we are going to use some package right now.

1. define packages

Start from creating packages.yml in the root folder as same as dbt_project.yml as the example below.

packages:
  - package: dbt-labs/dbt_utils
    version: 1.1.0

2. install packages

Execute the command dbt deps to install and update packages according to packages.yml

$ dbt deps

19:12:54  Running with dbt=1.9.2
19:12:55  Updating lock file in file path: /path/to/dbt-project/package-lock.yml
19:12:55  Installing dbt-labs/dbt_utils
19:12:55  Installed from version 1.1.0
19:12:55  Updated version available: 1.4.1
19:12:55  
19:12:55  Updates available for packages: ['dbt-labs/dbt_utils']                 
Update your versions in packages.yml, then run dbt deps

That’s a prompt saying new version of dbt packages is available and we can update in packages.yml.

3. make use of the packages

Right now we have dbt-labs/dbt_utils package. The package provides plenty of useful functions.

For example, dbt_utils.not_null_proportion is to verify if number of not-null in the field is in the given range. And we are checking if “not null” values are 95% or more as below.

version: 2

models:
  - name: calc_grades
    columns:
      - name: id
        tests:
          - dbt_utils.not_null_proportion:
              at_least: 0.95
      - name: name
      - name: subject
      - name: grade
      - name: updated_at

Run dbt test and found it failed because number of “null” values are more than 5% in the field id.

$ dbt test

19:52:48  Running with dbt=1.9.2
19:52:48  Registered adapter: bigquery=1.9.1
19:52:49  Found 7 models, 1 seed, 1 test, 2 sources, 606 macros
19:52:49  
19:52:49  Concurrency: 4 threads (target='dev')
19:52:49  
19:52:50  1 of 1 START test dbt_utils_not_null_proportion_calc_grades_0_95__id ........... [RUN]
19:52:52  1 of 1 FAIL 1 dbt_utils_not_null_proportion_calc_grades_0_95__id ............... [FAIL 1 in 1.75s]
19:52:52  
19:52:52  Finished running 1 test in 0 hours 0 minutes and 2.88 seconds (2.88s).
19:52:52  
19:52:52  Completed with 1 error, 0 partial successes, and 0 warnings:
19:52:52  
19:52:52  Failure in test dbt_utils_not_null_proportion_calc_grades_0_95__id (models/schema.yml)
19:52:52    Got 1 result, configured to fail if != 0
19:52:52  
19:52:52    compiled code at target/compiled/dbt-project/models/schema.yml/dbt_utils_not_null_proportion_calc_grades_0_95__id.sql
19:52:52  
19:52:52  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

---

Interesting packages

These are some dbt packages that we can consider to use in our projects. Also much more to discover in the hub link above.

• dbt-labs/codegen generate dbt code.
• brooklyn-data/dbt_artifacts modelling dbt metadata.
• dbt-labs/dbt_utils Utility functions for dbt projects.
• elementary-data/elementary detect anomalies, monitor data quality, and build metadata tables.
• metaplane/dbt_expectations deploy GE-like tests in their data warehouse directly from dbt.
• dbt-labs/dbt_external_tables dbt macros to stage external sources.
• kristeligt-dagblad/dbt_ml train, audit and use BigQuery ML models.

---

Wrap up

• define packages in packages.yml with the key packages.
• install and update packages by dbt deps.