Instant SQL for results as you type in DuckDB UI

There’s an extension for that https://github.com/ywelsch/duckdb-psql

wohoo! glad you noticed that. Hamilton is amazing.

Is that talk available online?

The PRQL[1] syntax is built around pipelines and works pretty well.

I added a similar "get results as you type" feature to the SQLite integration in the Logfile Navigator (lnav)[2]. When entering PRQL queries, the preview will show the results for the current and previous stages of the pipeline. When you move the cursor around, the previews update accordingly. I was waiting years for something like PRQL to implement this since doing it with regular SQL requires more knowledge of the syntax and I didn't want to go down that path.

[1] - https://prql-lang.org [2] - https://lnav.org/2024/03/29/prql-support.html

Obviously one advantage of SQL is everyone knows it. But conceptually, I agree. I think [1]Malloy is also doing some really fantastic work in this area.

This is one of the reasons I'm excited about DuckDB's upcoming [2]PEG parser. If they can pull it off, we could have alternative dialects that run on DuckDB.

[1] https://www.malloydata.dev/ [2] https://duckdb.org/2024/11/22/runtime-extensible-parsers.htm...

Google SQL has it now:

https://cloud.google.com/blog/products/data-analytics/simpli...

It's pretty neat:

    FROM mydataset.Produce
    |> WHERE sales > 0
    |> AGGREGATE SUM(sales) AS total_sales, COUNT(\*) AS num_sales
       GROUP BY item;

I suspect you'll like PRQL: https://github.com/PRQL/prql

I haven’t tested but I believe there’s a prql extension for duckdb

Thanks for sharing this update with the world and including it on the local ui too.

Feature request: enable the tuning of when Instant SQL is run and displayed. The erroring out with flashing updates at nearly every keystoke while expanding on a query is distracting for me personally (my brain goes into troubleshooting vs thinking mode). I like the feature (so I will keep it on by default), but I’d like to have a few modes for it depending on my working context (specifically tuning of update frequency at separation characters [space, comma], end of statement [semicolon/newline], and injections [paste/autocomplete]).

Curious if there has been any thought given to open sourcing the UI? Of course there's no obligation to though!

Definitely something we want too! (I'm the author / lead for the UI)

Maybe in the next version they could also implement support for DROP, with autocorrect for the nearest (not yet dropped) table name.

for clarity: Instant SQL won't automatically run queries that write or delete data or metadata. It only runs queries that read data.

Can't it just run inside a transaction that isn't committed?

Young bobby tables at it again

ROFL

Me too (author of the post here). In fact, I was watching a seasoned data engineer at MotherDuck show me how they would attempt to debug a regex in a CTE. As a longtime SQL user, I felt the pain immediately; haven't we all been there before? Instant SQL followed from that.

Agree, definitely amazing feature. In the Python API you can get somewhere close with this kind of thing:

input_data = duckdb.sql("SELECT * FROM read_parquet('...')")

step_1 = duckdb.sql("SELECT ... FROM input_data JOIN ...")

step_2 = duckdb.sql("SELECT ... FROM step_1")

final = duckdb.sql("SELECT ... FROM step_2;")

Indeed, we are! We worked with DuckDB Labs to add the query_location information, which we're also enriching with the tokenizer to draw a path through the AST to the cursor location. I've been wanting to do this since forever, and now that we have it, there's actually a long tail of inspection / debugging / enrichment features we can add to our SQL editor.

This is a very cool feature. I don't know how useful it is or how I'd use it right now but I think I am going to get into some benchmarking and performance tweaking soon and this could be handy.

Can you go the other way? (E.g. edit the above and turn it back into SQL string)

I've used sqlglot to do this in the past, but doing it natively would be nice

The article says it does subqueries:

> Getting the AST is a big step forward, but we still need a way to take your cursor position in the editor and map it to a path through this AST. Otherwise, we can’t know which part of the query you're interested in previewing. So we built some simple tools that pair DuckDB’s parser with its tokenizer to enrich the parse tree, which we then use to pinpoint the start and end of all nodes, clauses, and select statements. This cursor-to-AST mapping enables us to show you a preview of exactly the SELECT statement you're working on, no matter where it appears in a complex query.

> What would be helpful would be to be able to visualize intermediate results -- if my cursor is inside of a subquery, show me the results of that subquery.

But that's exactly what they show in the blog post??

You should read the post! This is what the feature does.

You can use any variation of DuckDB valid syntax that you want! I prefer to put from first just because I think it's better, but Instant SQL works with traditional select __ from __ queries.

Yes it comes from a desire to impose intuition from other contexts onto something instead of building intuition with that thing.

SQL is a declarative language. The ordering of the statements was carefully thought through.

I will say it's harmless though, the clauses don't have any dependency in terms of meaning so it's fine to just allow them to be reordered in terms of the meaning of the query, but that's true of lots and lots of things in programming and just having a convention is usually better than allowing anything.

For example, you could totally allow this to be legal:

  def
      for x in whatever:
          print(x)
  print_whatever(whatever):

I think you see this all the time, people try to impose their own comfort onto things for no actual improvement.