Templating config in version 2

[whysthatso]

I used to template out the config string, using variables for passes, etc.

with the change now using 'to_yaml', this seems to break. in plain yaml, the vars are not being interpolated, and the |- notation does not work, as it generates a string with all control characters present, which is not yaml compliant.

i'm sure i'm missing something, how is everybody else doing that? @markstos

[dbrennand]

Hey @whysthatso

Can you please provide an example of the usage you're describing?

[whysthatso]

this works up until version 1.4

autorestic_config: |-
  version: 2
  locations:
    "{{ inventory_hostname }}_{{ domain.replace('.','_') }}":
      hooks:
        before:
[...]

[markstos]

Did you try this?

autorestic_config:
  version: 2
  locations:
    "{{ inventory_hostname }}_{{ domain.replace('.','_') }}":

[whysthatso]

yes, and it keeps the quoted string as a literal in the result.

however, now i saw that it actually interpolated the same variables correctly when they are part of a list further down:

autorestic_config:
  version: 2
  locations:
    "{{ inventory_hostname }}_{{ wp_domain.replace('.','_') }}":
      hooks:
        before:
          - mkdir /tmp
          - mysqldump --databases prod_{{  domain.replace('.','_') }} | gzip > //tmp/prod_{{ domain.replace('.','_') }}.sql.gz
          - 'curl -m 10 --retry 5 -X POST -H "Content-Type: text/plain" --data "Starting backup for {{ inventory_hostname }}_{{ wp_domain }}" https://domain.com/ping/{{ healthcheck_backup_uuid }}/start'

so it seems to be related to this string being a key and in double quotes.

is there another syntax i could try?

[dbrennand]

Hey @whysthatso

Have you tried using the template lookup plugin?

So specify the contents of autorestic_config as you had it before in a Jinja template file located in a templates/autorestic_config.j2 and then:

autorestic_config: "{{ lookup('ansible.builtin.template', 'autorestic_config.j2') }}"

[markstos]

If #11 was problematic, it can be reverted.

[dbrennand]

Hey @whysthatso

Have you tried using the template lookup plugin?

So specify the contents of autorestic_config as you had it before in a Jinja template file located in a templates/autorestic_config.j2 and then:

autorestic_config: "{{ lookup('ansible.builtin.template', 'autorestic_config.j2') }}"

Hi @whysthatso - Did you manage to test this out yet?

[whysthatso]

Hey @whysthatso
Have you tried using the template lookup plugin?
So specify the contents of autorestic_config as you had it before in a Jinja template file located in a templates/autorestic_config.j2 and then:

autorestic_config: "{{ lookup('ansible.builtin.template', 'autorestic_config.j2') }}"

Hi @whysthatso - Did you manage to test this out yet?

i haven't tried this and i'm pretty sure this would work. i reverted to not using a variable as a key in that particular situation, since it did not add any value there. i'll close it and consider this edge case documentation. i think having an easily lintable config is more important than providing for any edge case. it was just an educating find after all :)

[whysthatso]

i'm running into another issue, regarding variables and backends:

i have a backblaze b2 backend defined like so:

backends:
  b2:
    type: b2
    path: 'bucket-name:/{{ domain }}'
    key: {{ vault_restic_pass }}
    env:
      B2_ACCOUNT_ID: '{{ vault_b2_restic_target_account_id }}'
      B2_ACCOUNT_KEY: {{ vault_b2_restic_target_account_key }}

i need this to output:

backends:
  b2:
    type: b2
    path: 'bucket-name:/example.com'
    key: key
    env:
      B2_ACCOUNT_ID: 'id'
      B2_ACCOUNT_KEY: key

however, in any variance i try to escape the quotes, they are always stripped out.

i've tried

"'bucket-name:/example.com'"

''''bucket-name:/example.com'''

"''bucket-name:/example.com""

error on templating

"'bucket-name:/example.com'"

'bucket-name:/example.com'

@dbrennand i've now also tried your version:

autorestic_config: "{{ lookup('ansible.builtin.template', 'autorestic_config.j2') }}"

contrent of my template looks like this:

version: 2
locations:
  wellpress:
    hooks:
      before:
...

but that just writes out a literal string with all control characters, not a proper yaml config.

i'm a bit out of options here, does anybody have working examples that they can add as documentation? @markstos

[dbrennand]

Hey @whysthatso - Do the values you're trying to pass through have special characters?

Ansible's to_yaml and to_nice_yaml use pyyaml under the hood: https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/filter/core.py#L45

Based on this comment:

YAML really doesn't need the quotes. Only if the string contains/starts with special characters, but in that case pyyaml will quote it automatically.

I have been looking into this, and I've been playing around with to_yaml and to_nice_yaml. I pushed 48ca916

Here is my example:

domain: example.com
b2_key: test!
b2_secret: testing12!*
autorestic_config:
  version: 2
  locations:
    home:
      from: /home/me
      to: remote
      # Every Monday
      cron: 0 0 * * MON
  backends:
    remote:
      type: b2
      path: "bucket-name:/{{ domain }}"
      env:
        B2_ACCOUNT_ID: "{{ b2_key }}"
        B2_ACCOUNT_KEY: "{{ b2_secret }}"

Which outputs to below, which has the double quotes around the values:

"backends":
    "remote":
        "env":
            "B2_ACCOUNT_ID": "test!"
            "B2_ACCOUNT_KEY": "testing12!*"
        "path": "bucket-name:/example.com"
        "type": "b2"
"locations":
    "home":
        "cron": "0 0 * * MON"
        "from": "/home/me"
        "to": "remote"
"version": ! "2" <--- This is messed up

Unfortunately, I think I may have accidentally stumbled across a bug, because for some reason, the output for version is missing the rest of the YAML tag to show it is an integer (!int), and autorestic complains:

root@instance:/# autorestic info
Using config: 	 /root/.autorestic.yml
version specified in config file is not an int
Using lock:	 /root/.autorestic.lock.yml

I tried in the autorestic_config casting to an int but this didn't work either:

...
autorestic_config:
  version: "{{ 2 | int }}"
...

When modifying the task to:

- name: Create autorestic configuration file
  ansible.builtin.copy:
    content: "{{ autorestic_config | to_nice_yaml(default_style='\"', canonical=True) }}"
...

It works but the output ~/.autorestic.yml is not ideal...

root@instance:/# autorestic info
Using config: 	 /root/.autorestic.yml


  Location: "home"  

From	← /home/me
To	→ remote
Cron	0 0 * * MON


  Backend: "remote"  

Type	b2
Path	bucket-name:/example.com
Env	✧ B2_ACCOUNT_KEY testing12!*
	✧ B2_ACCOUNT_ID test!

---
!!map {
    ? !!str "backends"
    : !!map {
        ? !!str "remote"
        : !!map {
            ? !!str "env"
            : !!map {
                ? !!str "B2_ACCOUNT_ID"
                : !!str "test!",
                ? !!str "B2_ACCOUNT_KEY"
                : !!str "testing12!*",
            },
            ? !!str "path"
            : !!str "bucket-name:/example.com",
            ? !!str "type"
            : !!str "b2",
        },
    },
    ? !!str "locations"
    : !!map {
        ? !!str "home"
        : !!map {
            ? !!str "cron"
            : !!str "0 0 * * MON",
            ? !!str "from"
            : !!str "/home/me",
            ? !!str "to"
            : !!str "remote",
        },
    },
    ? !!str "version"
    : !!int "2",
}

See how the version 2 shows the correct YAML tag !!int.

[dbrennand]

Hey @whysthatso

If time permits, could you please test changes I made in #14 (comment) and see if this works for you too?