Ask HN: Dense Tree Layout Algorithms

For this particular example, I feel like a combination of both human input + algo could work. Something like:

* Have graphviz or some other solver output the [x;y] pairs for each vertex.

* Use the initial positions to create, with a physics engine, a construct where vertices are masses connected by springs following the DAG.

* Add a resizable bounding box constraint around the whole thing.

* Push and pull the boundaries and masses until you get something nice.

What you say still applies: you may want to give the user the ability to resize the masses and re-run the layout algorithm, etc.

The choice and design of algorithm can itself be an aesthetic decision. Eg, OP generated some layouts, and spotted something that wasn't to their taste; the edges frequently overlap. So, we might approach the problem like this:

1. Choose an arbitrary algorithm to lay out trees where the edges don't overlap. There's many ways to do this, it doesn't need to be legible at this step.

2. Insert additional nodes along each edge. How many nodes is a tuning parameter we'll need to play with.

3. Use a traditional force-based graph layout algorithm, one that assigns charge to the nodes so that they repel each other. Because we've given the edges some nodes, they will have charge and won't bunch up together.

4. (Optional?) Detect edge overlaps. If they exist, randomize the initial state and try again.

(Note that the algorithm I've described is O(n^2). Patience will be necessary. You can probably refine it further, I only thought about it for a few minutes. Optimizations applied to physics engines (collision detection, gravity simulation) will apply.)

I've dabbled in generative art like this, and if you're willing to do the math and programming and iterate on the design, you can definitely get something of the aesthetic you are after. Bonus, the skills are transferrable to some other kinds of programming, like property testing and graphics.