Python’s new t-strings

I did a safe OCaml implementation of this about 20 years ago, the latest version being here:

https://github.com/darioteixeira/pgocaml

Note that the variables are safely and correctly interpolated at compile time. And it's type checked across the boundary too, by checking (at compile time) the column types with the live database.

Aren't there other benefits to server-side parameter binding besides just SQL-injection safety? For instance, using PG's extended protocol (binary) instead of just raw SQL strings. Caching parameterized prepared statements, etc.

Also:

    db.execute(t"QUERY WHERE name = {name}")

    db.execute(f"QUERY WHERE name = {name}")

I don't think this new format specifier is in any way applicable to SQL queries.

Or you could use this in a library like sh with

    sh(t"stat {some_file}")

I'd have to take a look at the order things happen in shell, but you might even be able to increase security/foot-gun-potential a little bit here by turning this into something like `stat "$( base64 -d [base64 encoded content of some_file] )"`.

A potential concern is how close this looks to the pattern they're trying to override.

    db.execute(f"QUERY WHERE name = {name}")

    db.execute(t"QUERY WHERE name = {name}")

I don't see what it adds over f-string in that example?

3. It prevents the developer from trying

  db.execute(f"QUERY WHERE name = {name}")

  db.execute("QUERY WHERE name = %s" % name, ())

Python is not the first one to get this feature. It's been present in JS for some time now, and before that in C# (not sure if that's the origin or they also borrowed it from somewhere). Python adopted it based in part on successful experience in those other languages.

Assuming you also need to format non-values in the SQL (e.g. column names), how does the `execute` function is supposed to make the difference between stuff that should be formatted in the string vs a parametrized value?

> This is such a strange take on t-strings

I understand why this seems strange at first!

As Paul mentioned, we spent quite a lot of time considering these issues as PEP 750 came together. In the end, we concluded (a) the PEP leaves open quite a few potential approaches for tools to adopt (not just the one you suggest, as others here have pointed out), and (b) it's ultimately something that the broader tooling community needs to rally around and should probably be out of scope for the PEP itself.

So, with that background in mind, I am indeed hopeful we'll see the ecosystem adapt! :-)

I think you'll just see a pattern like:

    html(t"<h1>Hello</h1>")

JavaScript's tagged template literals are actually about as flexible as this, since you can dynamically choose the tag function, it's just very rare to do so, so tools assume a lot based on the name of the function. Python tools can basically do the same thing, and just not support t-strings that aren't nested inside a well-named processing function.

The original PEP and the original discussion had this in scope. We removed it to let this emerge later. There are different ways to signal the language -- some more friendly to tooling, some more robust.

PyCharm (as well as the other JetBrains IDEs) has for at least 10 years supported language injection in Python strings using a comment [0]. Worst case scenario there's no reason whatsoever that that couldn't be used by formatters to apply auto-formatting in a structured and deterministic way.

But that's far from the only option, either! IntelliJ for Java also supports annotations on arguments [1] that then mean that you get syntax highlighting everywhere you use a string literal with the given function:

    public void query(@Language("SQL") String sql)

> If a library or tool encounters an annotation Annotated[T, x] and has no special logic for the metadata, it should ignore the metadata and simply treat the annotation as T. As such, Annotated can be useful for code that wants to use annotations for purposes outside Python’s static typing system.

So something like this should be perfectly viable:

    SQL = Annotated[Template, "language", "SQL"]
    
    def query(sql_query: SQL):
        # do stuff with Template to sanitize
    
    query(t"SELECT * FROM foo WHERE bar={bar}")

[0] https://www.jetbrains.com/help/pycharm/using-language-inject...

[1] https://www.jetbrains.com/help/idea/using-language-injection...

[2] https://docs.python.org/3/library/typing.html#typing.Annotat...

Couldn’t you do this with a type annotation? e.g. SQLAlchemy could have a SQL type so tools like mypy could see a Template instance and confirm you’re using it safely but Black, Ruff, or SQLFluff could look for the more specialized Annotated[Template, SQL] to realize that the template could be formatted as SQL, and something like Django could even have Annotated[Template, Email], Annotated[Template, HTML], or Annotated[Template, JSX] to indicate what context the same templating syntax is targeting.

> The only way for anything to infer that the template string is supposed to turn into valid HTML or SQL is to base it of the apparent syntax in the string

Not the only thing. You can also look at how it is used. Your editor could know of how some popular libraries use t-strings, track which t-strings get passed into functions from those libraries, and use that to assume what grammar the t-string should follow.

Is that cheating? In some sense, yes, but it also is useful and likely will be worth it for quite a few programmers.

Python has had type annotations for a decade, and modern IDEs can interpret them.

Writing `query: SQL = t"SELECT ..."` is a small price to pay for such a DX boost.

This definitely seems like one of the motivations for implementing this feature in the first place - https://peps.python.org/pep-0750/#motivation.

Having more control over the interpolation of string values is a win IMO.

You would need triple-quotes to span multiple lines, but yes, that is exactly how it's intended to work. `db.get` will receive a Template instance, which stores string parts something like `('\n SELECT * FROM users\n WHERE city=', ' AND age>', '\n')` and interpolated parts like `(Interpolation('London'), Interpolation(21))`. It's then responsible for assembling and executing the query from that.

The SQLite extension for Tcl offers something similar:

    db1 eval {INSERT INTO t1 VALUES(5,$bigstring)} 

That’s the sort of thing people have built with the equivalent feature in JavaScript, so it should do. Eg https://github.com/andywer/squid is a nice example.

Yes, it's quite delightful.

Isn't the actually proper way to use prepared statements anyway? If we are doing that properly, then what does this t string business buy us for SQL usage from Python?

> This would be the nicest way to use SQL I have seen yet.

EF/EF Core has existed for years :)

https://learn.microsoft.com/en-us/ef/core/querying/sql-queri...

Thanks, I hate it. While it's nice syntactic sugar, the difference between an SQL injection vulnerability and a properly parametrized query is now a single letter that's easily missed

Python has always been a batteries-included language, so having a go at templated string interpolation —a feature other languages have had for decades— seems like a strange gripe.

It's far more essential than little utilities like textwrap or goliath packages like Python's bundled tkinter implementation.

This is a pretty simple and useful feature. I wouldn’t say that it bloats the language too much. Descriptors and metaclasses are much more complicated and have a lot more implications and have been in the language for a veeeeery long time. Is it decades already?

I'm truly glad that Go exists for people who like languages that are simple even to the point of frustration and I hope it never changes. But I'm glad that other languages exist for those of us for whom learning some syntax is not a barrier and having convenient ways to do common things is highly valued.

There’s an interesting trade off around maintenance. Python’s stdlib means there’s more consistency across projects and you can assume basic things like error handling which you had to check individually on each Go program, which is the kind of stuff which adds up when you have a lot of small programs.

This is especially noticeable with AWS Lambda where you can have a lot of useful stuff running for years without doing more than bumping the runtime version every year or two, but also that is one highly opinionated architecture so it’s not globally optimal for everyone.

Maybe it's good to come back to a discussion after a few days, more informed and with a cooler head ?

;-)

A t-string is a literal for a Template object which is a data holder, it doesn't actually do anything, so you would simply call

    bar(t"zoop")

This is exactly how Nim is. The f-string like equivalent uses a macro called "fmt" which has a short alias "&". So you can say:

    import std/strformat
    let world = "planet"
    echo &"hello {world}"

In general, with user-defined operators and templates and macros, Nim has all kinds of Lisp-like facilities to extend the language, but with a more static focus which helps for both correctness and performance.

This was the original proposed idea in the PEP (750), but it changed overtime. There is a section in the PEP to explain why it changed to t-strings if you are interested.

True. Then you could use

  sql"..."
  html"..."

Use a function?

    bar(“zoop”)

    print “foo” 

    print(“foo”)

In my own language design (nothing public yet - need to prioritize more practical things at the moment) this is definitely on the menu. Aside from a minimal set, keywords are implemented as compile-time macros; and it's intended that they can be extended, both "natively" and in the implementation language, by writing AST-manipulation code. But the handling of arithmetic expressions, as well as the broader line/block structure, is hard-coded. (I draw inspiration from both Python and the Lisp family.)

meh, the difference between bar"zoop" and bar("zoop") isn't really big enough to be worth it.

I like F strings a lot, but for the most part I think all of the various X-strings should just be classes that take a string as an argument.

> Is it just about not forgetting the sanitization/string manipulation part and forcing you to go through that?

This is a very big deal! It's also about centralizing that work. Now that sanitization can occur in the consumer of the t-string (for example, the API to your HTML renderer), rather than in every f-string.

Pretty much, yeah. The article highlights that people were using f-strings directly, and they wanted to provide an alternative for lightweight template/interpolation.

Another alternative:

  >>> a, b, c, d = (lambda a, b=45, c=None, **d: (a, b, c, d))(**{'a': 234, 'xyz': 32456})

just use this very sane pattern /s:

    >>> match {'b': 45, 'c': None}|{'a': 234, 'xyz': 32456}:
    >>>     case {'a': a, 'b': b, 'c': c, **d}: pass
    >>> print(a, b, c, d)
    234 45 None {'xyz': 32456}

It being a language feature gives you controlled access to the lexical scope, such that the template string can refer to variables by name rather than having to pass each value as a parameter. Doing it via parameters is repetitive and/or error-prone.

This feature actually can't be a library. It needs to be able to preserve the string before the variables are passed in, which a library would be unable to do because f-strings immediately replace all the values. The whole premise of t-strings is to know which values where "hard coded" and which ones are variables. A library can't possibly know that. And a function call wouldn't have access to the local variables. The only way to do this without language support is to pass `locals()` into every function call.

It's not as simple as "more features" == "closer to C++". Features are not equal to each other in terms of impact on language complexity.

t-strings don't interact with anything else in the language; they, as you yourself pointed out, could almost be an isolated library. That makes them low impact.

This is also true syntactically; they're just another type of string, denoted by "t" instead of "f". That's easy to fit into one's existing model of the language.

Moreover, even semantically, from the point of view of most language users, they are equivalent to f-strings in every way, so there's nothing to learn, really. It's only the library writers who need to learn about them.

Then we have to consider the upsides - the potential to eliminate SQL and HTML injection attacks. The value/cost is so high the feature a no-brainer.

If it's not a language feature, there's always a risk of fragmentation. Some people won't use it because it adds another dependency, that means fewer programmers will be familiar with it. Others will come up with their own, slightly incompatible implementation. See for example Perl and its various Object Orientation frameworks (Moose, Mouse, Moo, Mo, M, Mojolicious comes with its own...)

Other languages have a policy of prototyping such things out of core, and only adding it to the core language if it gains traction. Of course that works better if the language has a mechanism for extending the syntax out of core.

Counterpoint: It's good to add well designed and useful features to programming languages.

Because JS has it in the form of tagged template literals.

We have a handful of ways to create strings now.

Meanwhile, pytest is still not part of the standard library.

And $-strings (PEP 292) as well. :-)

I see each of these as distinct but overlapping; I'm (slowly) writing a guide to string formatting with all of these in mind, trying to emphasize when I'd choose one over the other. (fwiw I personally avoid % and + these days; $ is pretty uncommon in practice; f-, t-, and .format() all seem to have good unique uses.)

The last three generally shouldn't be used (`+` is sometimes fine, but not really for formatting), but I doubt we would ever get a py4 that removes them, given the stomachaches that py3 caused. It does feel odd that a t-string is just an f-string without args applied.

t-strings aren't really like these others.

It's definitely true that those four string formatting techniques violate the "one obvious way" advice.

I could scarcely believe this new t-string wasn't a joke. As an occasional, grudging Python programmer, I rue the accretion of string formatters over the years. My code bears the history of successive, imperfectly understood (by me)formatters.

"Situation: There are 14 competing standards...." https://xkcd.com/927/

That's the exact use case. Basically these are syntactic sugar for a very similar function signature.

Hi. I come from a JavaScript background.

I am wondering what is the reason behind not using a similar syntax to JavaScript? Seems simpler to me.

  # Compare this:
  template = t"<p>{evil}</p>"
  safe = html(template)

  # To this:
  safe = html"<p>{evil}</p>"

Would be interested in inclusion of PEP 292 [1] in your discussion here, which introduced `string.Template`. Is this Template going to be deprecated?

[1] https://peps.python.org/pep-0292/

This is super cool, thank you.