Renovate with docker containers

date: Sep 13, 2024

Renovate is used to help manage code dependencies. Checkout a previous post for more details on what Renovate is.

Renovate is able to parse dockerfiles to look for newer container images. This only works with either a sha256 digest or a version tag. Tags such as latest or some other arbitrary string will not work.

I tested running renovate as a docker container. Credentials can be used with environment variables within the container.

1

docker run --rm -e RENOVATE_TOKEN=$GITHUB_TOKEN -e RENOVATE_PLATFORM=github -e RENOVATE_AUTODISCOVER=true docker.io/renovate/renovate

With autodiscover enabled, renovate checked out all of the repos that the token had access to. Pull requests were opened in repositories that renovate detected with dependencies.

Here is an example of log output for a repo with only docker detected:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


 INFO: Repository started (repository=acaylor/dockertest)
       "renovateVersion": "37.108.1"
 INFO: Branch created (repository=acaylor/dockertest, branch=renovate/configure)
       "commit": "8a094b8c7e9ece2ad27091f5cce6c3dd66b81672",
       "onboarding": true
 INFO: Dependency extraction complete (repository=acaylor/dockertest, baseBranch=master)
       "stats": {
         "managers": {"dockerfile": {"fileCount": 1, "depCount": 1}},
         "total": {"fileCount": 1, "depCount": 1}
       }
 INFO: Onboarding PR created (repository=acaylor/dockertest)
       "pr": "Pull Request #1"
 INFO: Repository finished (repository=acaylor/dockertest)
       "cloned": true,
       "durationMs": 5883

Here is an example of a PR that renovate will open to update a container image.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


Welcome to [Renovate](https://github.com/renovatebot/renovate)! This is an onboarding MR to help you understand and   configure settings before regular Merge Requests begin.

🚦 To activate Renovate, merge this Merge Request. To disable Renovate, simply close this Merge Request unmerged.



---
### Detected Package Files

* `docker-compose.yml` (docker-compose)
* `Dockerfile` (dockerfile)

### What to Expect

With your current configuration, Renovate will create 1 Merge Request:

<details>
<summary>Update k8s.gcr.io/echoserver Docker tag to v1.10</summary>

 - Schedule: ["at any time"]
 - Branch name: `renovate/k8s.gcr.io-echoserver-1.x`
 - Merge into: `main`
 - Upgrade k8s.gcr.io/echoserver to `1.10`


</details>


---

❓ Got questions? Check out Renovate's [Docs](https://docs.renovatebot.com/), particularly the Getting Started        section.
If you need any further assistance then you can also [request help here](https://github.com/renovatebot/renovate/     discussions).


---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

<!--renovate-config-hash:94693a990c975907e7f13da3309b9d56ba02b3983519b41786edf5cf031e457c-->

While renovate can generate pull requests to update dependencies, the logs can also be used to get more information.

Run renovate with debug logs and output to a file

Now with debug logs enabled, the data presented via pull requests to you is also available on the command line. I wrote this script to parse data from renovate debug logs.

I can run renovate as a container and redirect the debug output to a file.

1

docker run --rm -e LOG_LEVEL=debug -e LOG_FORMAT=json -e RENOVATE_TOKEN=$GITEA_TOKEN -e RENOVATE_PLATFORM=gitea -e RENOVATE_ENDPOINT="https://git.lan.ayjc.net/api/v1/" -e RENOVATE_AUTODISCOVER=true -e RENOVATE_AUTODISCOVER_FILTER="aj/containers_*" docker.io/renovate/renovate > renovate.log

This should create a file renovate.log

I have a solid foundation for getting more structured data from renovate.

Run renovate with logs set to DEBUG and json format.
- This produces a massive file over 10MB with 20k lines of json.
Parse the renovate logs
Store the dependency data in a database

Parsing log file full of json objects

There are a few approaches to parsing the debug logs. I used Python.

select log lines

The debug log file will have 20k lines of output. Before importing data into a database, we should clean the output. The logs will be JSON {} objects but not properly enclosed within an array.

Using Python, we can parse the initial log file into one JSON array.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


import json

# Open renovate log file
with open('renovate.log', 'r') as f_in, open('renovate-clean.json', 'w') as f_out:
    data = []

    # iterate over every line of the log fine
    for line in f_in:
        try:
            # parse the line as JSON
            obj = json.loads(line)
            # print("loading log file")

            data.append(obj)
            # print("appending line to new file")
        except ValueError:
            # ignore lines that cannot be parsed as json
            print('line is not JSON...ignoring...')
            pass

    # convert obj back into JSON and write to a new file
    print('Writing to file')
    json.dump(data, f_out)

1

Writing to file

This will create a file in the current directory that is a JSON array.

Optionally, jq can be used to make the file more human readable:

1

cat renovate-clean.json | jq > clean-jq.json

Parse the remaining data

The remaining data can now be parsed using Python.

This code will open a JSON file that needs to be valid JSON hence in the previous step I transformed the log file into a list of objects.

Then using the psycopg2 library, we connect to a PostgreSQL database and create a table to store dependency data. Install docs

Postgres supports JSON data type and using the jsonb column type will allow for querying the data within each row with JSON data.

Each row will contain:

The git repository
the file where the dependency is located
the data related to the dependency from Renovate logs

I have chosen to parse the data and write it to a postgresql database. There are other approaches that do not require a database server such as using a sqlite3 database which is contained in one file.

Run a postgres server with docker

1
2
3
4
5
6
7
8
9


docker run \
--env=POSTGRES_USER=renovate \
--env=POSTGRES_PASSWORD=renovate \
--env=POSTGRES_DB=renovate \
--volume=/var/lib/postgresql/data \
-p 5432:5432 \
--restart=no \
-d \
postgres

This will pull the image from the public docker hub and run it on your machine if you have docker installed. Eg. swap docker for podman if you use podman.

Install this python package to interface with a PostgreSQL database.

1

pip install psycopg2-binary

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11



    Collecting psycopg2-binary
      Downloading psycopg2_binary-2.9.9-cp312-cp312-macosx_11_0_arm64.whl.metadata (4.4 kB)
    Downloading psycopg2_binary-2.9.9-cp312-cp312-macosx_11_0_arm64.whl (2.6 MB)
    [2K   [38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m2.6/2.6 MB[0m [31m8.5 MB/s[0m eta [36m0:00:00[0m MB/s[0m eta [36m0:00:01[0m:02[0m
    [?25hInstalling collected packages: psycopg2-binary
    Successfully installed psycopg2-binary-2.9.9
    
    [1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m24.0[0m[39;49m -> [0m[32;49m24.2[0m
    [1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49m/opt/homebrew/Cellar/jupyterlab/4.2.3/libexec/bin/python -m pip install --upgrade pip[0m
    Note: you may need to restart the kernel to use updated packages.

This is the script described above

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51


import json
import psycopg2
from psycopg2.extras import Json


# Define the file to parse
file = 'renovate-clean.json'
depType = "docker-compose"
# Open renovate log file
with open(file) as f:
    data = json.load(f)

# Connect to the database
conn = psycopg2.connect(dbname='renovate', user='renovate', password='renovate', host='localhost')
cursor = conn.cursor()

# Create a table for renovate if it doesn't exist
cursor.execute('''CREATE TABLE IF NOT EXISTS renovate (
    id SERIAL PRIMARY KEY,
    repository text,
    packagefile text,
    data JSONB
)''')

# Assuming data is a list of dictionaries
for item in data:
    # if the item does not contain config, skip it
    if 'config' not in item or depType not in item['config']:
        print("No config found in item")
        continue

    # iterate through the items
    for config_item in item['config'][depType]:
        # if the config_item does not contain 'deps', skip it
        if 'deps' not in config_item:
            print("No deps found in config item")
            continue
        # iterate through the dependencies
        for dep_item in config_item['deps']:
            # convert the 'dep' object into JSON
            dep_json = json.dumps(dep_item)
            cursor.execute("INSERT INTO renovate (repository, packagefile, data) VALUES (%s, %s, %s)", (item['repository'], config_item['packageFile'], dep_json))
            print(f"Writing {config_item['packageFile']} to database")

# Commit the changes
conn.commit()
# Close the connection
cursor.close()
conn.close()

print('Done parsing renovate data from log file')

Here is example output not including many lines where no dependencies were found:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


    Writing adguard/docker-compose.yml to database
    Writing caddy/docker-compose.yml to database
    Writing cadvisor/docker-compose.yml to database
    Writing dokuwiki/docker-compose.yml to database
    Writing fluent-bit/docker-compose.yml to database
    Writing gitea/docker-compose.yml to database
    Writing ha/docker-compose.yml to database
    Writing kanboard/docker-compose.yml to database
    Writing node-exporter/docker-compose.yml to database
    Writing pihole/docker-compose.yml to database
    Writing portainer/docker-compose.yml to database
    Writing prom/docker-compose.yml to database
    Writing prom/docker-compose.yml to database
    Writing prom/docker-compose.yml to database
    Writing prometheus/docker-compose.yml to database
    Writing prometheus/docker-compose.yml to database
    Writing prometheus/docker-compose.yml to database
    Writing prometheus/docker-compose.yml to database
    Writing prometheus/docker-compose.yml to database
    Writing promtail/docker-compose.yml to database
    Writing proxy-manager/docker-compose.yml to database
    Writing proxy-manager/docker-compose.yml to database
    Writing unpoller/docker-compose.yml to database
    Writing wireguard/docker-compose.yml to database
...
    Writing adguard/docker-compose.yml to database
    Writing cadvisor/docker-compose.yml to database
    Writing promtail/docker-compose.yml to database
    No config found in item
...
    Writing adguard-sync/docker-compose.yml to database
    Writing adguard/docker-compose.yml to database
    Writing archived/adguard-exporter/docker-compose.yml to database
    Writing archived/nginx-exporter/docker-compose.yml to database
    Writing archived/unpoller/docker-compose.yml to database
    Writing caddy/docker-compose.yml to database
    Writing nut-exporter/docker-compose.yml to database
    Writing prometheus-stack/docker-compose.yml to database
    Writing prometheus-stack/docker-compose.yml to database
    Writing prometheus-stack/docker-compose.yml to database
    Writing prometheus-stack/docker-compose.yml to database
    Writing promtail/docker-compose.yml to database
    Writing scrypted/docker-compose.yml to database
...
    No config found in item
    No config found in item
    No config found in item
    Done parsing renovate data from log file

Assuming the program executes without errors, the specified database should have data in the renovate table.

1

SELECT * from renovate;

1

"id","repository","packagefile","data"

querying dependency data

For example, to query all packages that use a certain dependency:

1

select * from renovate where data->>'depName' = 'foo';

1

"id","repository","packagefile","data"

The dependency name is in a column data that is a jsonb type of data.

The data.updates[] array shows if there are updates available for the dependency. We can only show rows with an update proposed:

1

select * from renovate where jsonb_path_exists(data, '$.updates[*]."bucket"');

To summarize all of the dependencies in a file:

1

select packagefile,string_agg(data->>'packageName', ';') as dependency from renovate group by packagefile order by packagefile;

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


"packagefile","dependency"
adguard/docker-compose.yml,adguard/adguardhome;adguard/adguardhome;adguard/adguardhome
adguard-sync/docker-compose.yml,lscr.io/linuxserver/adguardhome-sync
archived/adguard-exporter/docker-compose.yml,ghcr.io/henrywhitaker3/adguard-exporter
archived/nginx-exporter/docker-compose.yml,docker.io/nginx/nginx-prometheus-exporter
archived/unpoller/docker-compose.yml,golift/unifi-poller
caddy/docker-compose.yml,caddy;caddy
cadvisor/docker-compose.yml,gcr.io/cadvisor/cadvisor-arm64;gcr.io/cadvisor/cadvisor
dokuwiki/docker-compose.yml,lscr.io/linuxserver/dokuwiki
fluent-bit/docker-compose.yml,grafana/fluent-bit-plugin-loki
gitea/docker-compose.yml,docker.io/gitea/gitea
ha/docker-compose.yml,ghcr.io/home-assistant/home-assistant
kanboard/docker-compose.yml,kanboard/kanboard
node-exporter/docker-compose.yml,quay.io/prometheus/node-exporter
nut-exporter/docker-compose.yml,hon95/prometheus-nut-exporter
pihole/docker-compose.yml,pihole/pihole
portainer/docker-compose.yml,portainer/portainer-ce
prom/docker-compose.yml,prom/prometheus;quay.io/prometheus/node-exporter;docker.io/grafana/grafana-oss
prometheus/docker-compose.yml,prom/prometheus;quay.io/prometheus/node-exporter;golift/unifi-poller;prom/blackbox-exporter;grafana/grafana-oss
prometheus-stack/docker-compose.yml,quay.io/prometheus/prometheus;quay.io/prometheus/node-exporter;docker.io/grafana/grafana;gcr.io/cadvisor/cadvisor
promtail/docker-compose.yml,grafana/promtail;grafana/promtail;grafana/promtail
proxy-manager/docker-compose.yml,jc21/nginx-proxy-manager;jc21/mariadb-aria
scrypted/docker-compose.yml,ghcr.io/koush/scrypted
unpoller/docker-compose.yml,golift/unifi-poller
wireguard/docker-compose.yml,lscr.io/linuxserver/wireguard

Clean up database

If you want to keep using the postgres database, you can clear out all rows from the renovate table either with SQL or python:

1

TRUNCATE TABLE renovate RESTART IDENTITY;

Add this to the python script. Hint: before the command to insert new values into the table.

1

cursor.execute(f"TRUNCATE TABLE {depType} RESTART IDENTITY")

Next steps

Now that data is in a database, there are many more queries that can be used by diving into SQL.