Skip to content(if available)orjump to list(if available)

Dynamic YAML with Python computed properties for fusing API workflows and SQL

mdaniel

You seem to have stepped on the same landmine that Ansible did, by defaulting to the jinja2 [aka text/template silliness in golang] of using double mustaches in YAML. I hope you enjoy quoting things because you're going to be quoting everything for all time because "{" is a meaningful character in YAML. Contrast

      parameters:
        status: "{{ var('order_status') }}"
with

      parameters:
        # made famous by GitHub Actions
        status: ${{ var('order_status') }}

        # or the ASP.Net flavor:
        status2: <%= var('order_status2') %>

        # or the PHP flavor:
        status3: <?= var('order_status3') ?>
and, just like Ansible, it's going to get insaneo when your inner expression has a quote character, too, since you'll need to escape it from the YAML parser leading to leaning toothpick syndrome e.g.

      parameters:
        status: "{{ eval('echo \"hello\"') }}"
---

If you find my "but what about the DX?" compelling, also gravely consider why in the world `data_expression:` seems to get a pass, in that it is implicitly wrapped in the mustaches

---

edit: ah, that's why https://github.com/paloaltodatabases/sequor/blob/v1.2.0/src/... but https://github.com/paloaltodatabases/sequor/blob/v1.2.0/src/... is what I would suggest changing before you get a bunch of tech debt and have to introduce a breaking change. From

    str_rendered = Template(template_str, undefined=StrictUndefined).render(jinja_context)
to

      str_rendered = Template(template_str, undefined=StrictUndefined,
          variable_start_string="${{",
          variable_end_string="}}"
      ).render(jinja_context)
      # et al, if you want to fix the {# and {%, too

per https://jinja.palletsprojects.com/en/stable/api/#jinja2.Temp...

bz_bz_bz

Recalculating customer metrics like that in your main example seems like a massive waste of snowflake resources, no?

maxgrinev

Dynamic YAML with computed properties could have applications beyond API integrations. We use Python since it's familiar to data engineers, but our original prototype with JavaScript had even more compact syntax. Would love feedback on our approach and other use cases for dynamic YAML.

revskill

[flagged]