Gradient propgation

[DimpyRed]

This is a demo introducing environmental gradients that are locally similar between nodes, and applies elevation to them. As a feature, it's complete enough to add to the game.

Code may need some cleaning, and there may need to be some optimization (worldgen is noticeably slower), but I'd like some other people's eyes and opinions on this for those activities.

[Ralith]

Unrelated change; please omit.

[Ralith]

It looks like we're expanding a small amount of information into a much larger amount before passing it all through a function call. Could we push all this down into EnviroFactors::varied_from? Maybe pass parent wholesale?

[Ralith]

What does this comment mean? Elaborate.

[Ralith]

style nit: include an empty line between top-level blocks

[Ralith]

@patowen can you comment on the significance of this pattern? I'm completely drawing a blank on why we don't e.g. just average everything together here, and on what exactly "ab" is and what this method for combining vectors looks like visually.

[patowen]

I'll have to take a close look at this. This seems similar to the previous "continue_from" logic, which was needed to handle the "one consistent offset per dividing plane" explained at

hypermine/docs/world_generation.md

Line 20 in ca2903c

To decide on a noise value for each node, we break the dodecahedral tiling up into these planes. We associate each plane with a randomly chosen noise delta, such that crossing a specific plane in one direction increases or decreases the noise value by a specific amount, and crossing the same plane from the other side has the opposite effect. Once we decide on a noise value for the root node, this definition fully determines the noise value of every other node.

. Effectively, if you have the value at three out of four nodes sharing an edge, the value for the fourth node is completely determined, and this logic figures out what it is.

[DimpyRed]

@Ralith imagine we have a grid of 4 nodes laid out as follows:

c   | b
-----------
a   | ab

We know a scalar value at a, b, and ab.

We can calculate that crossing the boundary from ab->a increases the value by (a.value - ab.value), and crossing the boundary from ab->b increases the value by (b.value - ab.value).

The way hypermine random-step noise functions work, we declare that the effect on the value of moving from ab->a is the same as that of b->c, in other words a plane is one big boundary that acts the same everywhere on it.

From that we calculate the value of c as (ab.value + (a.value - ab.value) + (b.value - ab.value)), which simplifies to the classic expression (a.value + (b.value - ab.value))

[Ralith]

style: this is kind of hard to visually parse and has serious rightward drift. Lift out some intermediate variables?

[Ralith]

Could this be simplified using checked_sub to combine the subtraction with the branching?

[Ralith]

Rather than panicking, could we pick an arbitrary direction here?

[Ralith]

Why not use a rotation here and avoid the hazard?

[Ralith]

Please squash together commits unless they're logically independent changes that should be reviewed separately.

@patowen, please review for isotropy. Seems fine AFAICT but you have a better understanding of the problem than I.

worldgen is noticeably slower

How much slower? Where is the main cost? If we don't have a simple headless benchmark yet, we should throw one together.

[patowen]

@patowen, please review for isotropy. Seems fine AFAICT but you have a better understanding of the problem than I.

I don't believe this is isotropic as it is currently written, mainly because the principles that make the current worldgen isotropic don't carry over to this PR. A lot of this depends on the continue_from logic.

In the current (outside of this PR) worldgen logic, the elevation increases or decreases when you cross any plane in the tiling. The continue_from logic helps ensure that whenever you cross a specific plane, the elevation change is always the same. However, this logic is highly sensitive, and it doesn't work as intended for gradient_magnitude because of the way it's capped. I believe continue_from correctly ensures that the change in gradient_magnitude is fixed for any given plane you cross, but the right setup can cause this logic to force gradient_magnitude to rise above ELEVATION_GRADIENT_MAGNITUDE_CEILING for child nodes with two parents, but not child nodes with one parent, making these two types of nodes distinguishable from each other.

I expect that gradient_direction has similar issues, as the root cause is related to the value not being a simple symmetrical addition or subtraction. If we just look at perturb alone, it seems fine, where going backwards or forwards is equivalent, but I don't see a good way to get it to play well with continue_from.

The effect of the gradient on the elevation is also a potential issue. For instance, it plays a role in varied_from, but it does not play a role in continued_from. While I haven't ruled out that the gradients are set correctly in continued_from for elevation and gradient to be consistent with each other, I find it unlikely.

If we believe that these issues are okay to ignore, I'll likely need to try to see if there's a way to systematically find a spot where the lack of isotropy becomes obvious. This may take a while for me to find and/or rule out as a possibility.

However, since I don't like just being a downer, I'll also try to include some ideas as well:

I think that a conceptually easier way to guarantee isotropy for any value you want to vary across hyperbolic space is to rely on unbounded scalar environmental factors and use them as input to a function (possibly a noise function in Euclidean space, or something else).

This doesn't immediately solve the "gradient" problem, though, since even if we assign a gradient to each node in an isotropic way, actually applying that gradient to the elevation is nontrivial, since the naive approach will cause the gradient to fail to apply when continue_from is called. However, as I write this, I do wonder if there's a good way to design another noise function with similar principles but whose domain is the dividing planes of the tiling instead of the nodes. This could achieve a similar purpose of the gradients by instead adding some consistency to the plane-by-plane elevation changes. (EDIT: I think that approach leads to a dead end, though. Unlike nodes, planes are always only a few steps away from each other, since, ignoring the tiling, for every pair of planes, there is a third plane that can get arbitrarily close to both.)

I also think a fair amount of terrain variety can be set up with some simpler approaches, such as having an elevation and elevation multiplier environmental factor, where a low elevation multiplier means that everything stays close to the ground plane.

[Ralith]

I expect that gradient_direction has similar issues, as the root cause is related to the value not being a simple symmetrical addition or subtraction.

Can we formalize the necessary properties here? Then we could document, abstract, and test in those terms, which should make it much easier to preserve isotropy.

Would using a random rotation to perturb the direction satisfy those properties? Rotation is not commutative, but it is invertible...

If we believe that these issues are okay to ignore

I'd much prefer to maintain isotropy formally, compromising at most on numerical issues (though maybe we could avoid even those by using intrinsically closed domains, like rotations). We should fully explore whether an effect along the lines proposed here, which is a compelling improvement in output, is feasible under that constraint.

I think that a conceptually easier way to guarantee isotropy for any value you want to vary across hyperbolic space is to rely on unbounded scalar environmental factors and use them as input to a function

Deferring the magnitude clamping to the last moment, rather than applying it as part of factor propagation, should be an easy fix. The effect would be much larger regions where the effective value is saturated, but that's probably fine?

This doesn't immediately solve the "gradient" problem, though, since even if we assign a gradient to each node in an isotropic way, actually applying that gradient to the elevation is nontrivial, since the naive approach will cause the gradient to fail to apply when continue_from is called.

Can you elaborate on "fail to apply"? In general, is it possible to have data dependencies between environment factors, or must they always evolve independently?

elevation and elevation multiplier environmental factor

We want to vary the spatial frequency of terrain height. Only varying the amplitude isn't a good substitute.

[DimpyRed]

Hi all, thank you for the feedback. Some flaws in the isotropy of the current method were highlighted.

I want to offer different of gradient propagation that should be a little easier to conceptualize and guarantee isotropy on.

varied_from steps:

1. The parent's elevation derivative in direction of traversal is calculated from the gradient and face normal.
2. A scalar-valued normally-distributed "gradient delta" is chosen.
3. The child node's elevation is set to parent elevation + (parent elevation derivative + (gradient delta) /2) * distance
4. The child node's gradient is set to parent gradient + gradient delta * displacement

continue_from steps:

1. Set the elevation of child node c to that of a + (b - c) as usual.
2. Calculate magnitude of the gradient of parent a in direction ab->a, and magnitude of the gradient of parent b in direction ab->b.
3. The gradient of c is constructed as follows:

• The component in direction b->c is equal to the gradient at node a in direction ab->a.
• The component in direction a->c is equal to the gradient at node b in direction ab->b.
• The component perpendicular to both the two components above is inherited from any of the parents (all three parents and the child will have the same perpendicular component)

The idea is all nodes touching a plane have the same "towards plane" gradient component, and same elevation change with it's neighbor on the other side of the plane.

[patowen]

I find this tricky to analyze, so I will get to this when I have time (as I also need to find some time to prioritize my megastructure work). I do have a few questions and comments, though, that I believe will help illustrate what is needed from isotropy.

• To make isotropy straightforward to understand, the relationship between gradient and elevation needs to be isotropic. I have some concerns about the bias towards the using the parent's gradient when determining the elevation difference between the two nodes. I'd also recommend that in continue_from, all three potential parent nodes are taken into account when you're conceptualizing the algorithm. Checking that gradient and elevation match properly can be done with some unit tests.
• The harder part of this problem is allowing the gradients themselves to still be isotropic when this constraint is in place. This is a very strong condition because not only does the probability distribution of gradient relationships between every pair of adjacent nodes need to be constant, but even the probability distribution of gradient relationships among larger groups of nodes (such as every quad of line-segment-adjacent nodes). Because varied_from and continued_from have different logic from each other, it's not easy to tell from your description that this is the case.

To answer one of @Ralith's question's, commutativity has some conceptual importance because you need to be able to get back to where you were by taking an ABAB path across node sides.

I'm really not sure what the best approach is for the algorithm here, though. One could definitely write tests to statistically analyze the various gradient distributions across pairs of nodes or quads of nodes, but this should ideally be derivable from first principles in a way that future readers of the code can understand. (But the tests would at least catch mistakes). I would recommend proving the correctness of the algorithm with Rocq, but that's an extremely tall order.

As for how to justify whether an algorithm is isotropic, my recommendation would be to approach it from a different angle. In other words, don't start by describing what the algorithm actually does with varied_from and continued_from. Instead, start by describing what the algorithm will actually produce, using language that is invariant to the isometries we care about, such as talking about adjancent nodes instead of parent/child nodes. Once that fully describes the function being created, the next step becomes finding an algorithm to produce that function and justifying why that algorithm achieves the stated goal.

For example, the elevation function in the master branch can be described as follows: "For each dividing plane in the dodecahedral honeycomb, choose a random value to use as an elevation offset. Then, the difference in elevation between any two adjacent nodes is decided by the plane dividing them. Choose an arbitrary node and decide on an arbitrary elevation for it to fully determine the elevations of the other nodes."

[patowen]

I still believe it's possible to make terrain that looks this good while having it fully controlled by independently varying environmental factors. For instance, we could have two environmental factors that together determine elevation. These environmental factors would then choose an elevation from a function of our choosing with domain R^2, a kind of reference heightmap. This reference heightmap could have areas of more frequent and less frequent elevation change, which would then translate to rougher and smoother terrain in the game.

My understanding is that one of the main goals of using gradient here as a proxy to control elevation is to add an extra degree of smoothness, since right now, any progress made by crossing one node boundary can be undone by crossing another node boundary. I'm not sure how much this matters in practice, as there are enough directions to choose from that you should be able to continue to make progress by going in a different direction, but it's possible that there are other tangible benefits I'm missing.

If we do insist on needing some smoothness like this, it's possible that there's a different algorithm to accomplish this whose isotropy is easier to justify. For example, one that seems promising relates a bit to megastructures: "Every node has some small probability of being an 'influencer'. Each influencer is associated with an elevation delta. To find the elevation of a specific node, use the weighted sum of the elevation deltas of all influencers within a given radius, where the farther away the influencer has, the smaller the weight. This algorithm is made feasible by choosing a probability low enough that any given node only has on the order of 5 or so influencers in range."

[Ralith]

choose an elevation from a function of our choosing with domain R^2

I agree this could work, but (per my outstanding questions) I still want to fully explore whether higher-order environment factors can work. Intuitively it's not at all obvious to me why "the elevation is the second integral of values assigned to planes" should be any less isotropic than the first integral.

I'm not sure how much this matters in practice, as there are enough directions to choose from that you should be able to continue to make progress by going in a different direction, but it's possible that there are other tangible benefits I'm missing.

The tangible benefit is that the terrain is more varied and interesting. Our criteria for terrain isn't "you can make progress", it's "exploring is fun".

use the weighted sum of the elevation deltas of all influencers within a given radius

I think we need to be extremely wary of anything that queries a neighborhood larger than 1 due to exponential blowup.

[patowen]

choose an elevation from a function of our choosing with domain R^2

I agree this could work, but (per my outstanding questions) I still want to fully explore whether higher-order environment factors can work. Intuitively it's not at all obvious to me why "the elevation is the second integral of values assigned to planes" should be any less isotropic than the first integral.

The main problem is that the concept of a "second integral" becomes difficult when working with a domain other than the real numbers (or possibly complex numbers). When the domain is H^3, the derivative of a scalar field is essentially a vector field (possibly a "tangent vector" field, since H^3 is curved), and the curl of this vector field has to be 0. The derivative of that vector field would probably be some kind of tensor field (I'm not that familiar with this math, so I can't speak in absolutes anymore), and there would be even more constraints.

Therefore, if you want to integrate something twice, you have to integrate this tensor field that has a bunch of constraints. Trying to make such a field isotropic while also obeying all the constraints is not something I know how to do, nor do I know how hard it would be.

The tangible benefit is that the terrain is more varied and interesting. Our criteria for terrain isn't "you can make progress", it's "exploring is fun".

I meant more in the sense of whether the same effect can be achieved without trying to make gradients correlate. For example, because there are so many directions you can traverse, it seems plausible that the terrain could look like the gradients correlate, but it's really just because you are moving in the direction where they correlate by chance. Again, I don't know if this is actually the case, as that would require some experimentation.

use the weighted sum of the elevation deltas of all influencers within a given radius

I think we need to be extremely wary of anything that queries a neighborhood larger than 1 due to exponential blowup.

I'm not describing the algorithm here. I'm just describing the function. I believe there is an efficient way to compute this function that does not require actually expanding the graph up to these influencer nodes. It's similar logic to megastructures. In fact, if we keep the algorithm discrete and base distance to these influencer nodes on number of steps, there might even be a clever way to have huge numbers of influencer nodes in O(n), where n is the maximum influence radius, although I might be missing something that makes this infeasible. I could make a proof of concept at some point if this is interesting at all.

[DimpyRed]

I have a correction to make regarding how node c's gradient should be calculated in continue_from in proposed method 2.

False assumptions:
δy/δ(A->C) at A = δy/δ(AB->B) at AB
δy/δ(B->C) at B = δy/δ(AB->A) at AB
False conclusion
δy/δ(A->C) at C = δy/d(A->C) at A + Δ(δy/δ(AB->B)) from AB->B = δy/δ(AB->B) at B
δy/δ(B->C) at C = δy/d(B->C) at B + Δ(δy/δ(AB->A)) from AB->A = δy/δ(AB->A) at A

Due to holonomy, it's actually
δy/δ(A->C) at A = δy/δ(AB->B) at AB + Δ(δy/δ(AB->A)) from AB->A * cos(2π/5)
δy/δ(B->C) at A = δy/δ(AB->A) at AB + Δ(δy/δ(AB->B)) from AB->B * cos(2π/5)
Therefore
δy/δ(A->C) at C = δy/δ(A->C) at A + Δ(δy/δ(AB->B)) from AB->B + Δ(δy/δ(AB->A)) from AB->A * 2cos(2π/5)
δy/δ(B->C) at C = δy/δ(B->C) at B + Δ(δy/δ(AB->C)) from AB->A + Δ(δy/δ(AB->B)) from AB->B * 2cos(2π/5)