Use dedicated style for landcover polygons over water

[tpetillon]

Fixes #3854
Fixes #3707
Fixes #3840

This PR allows rendering on-land and over-water landcover polygons separately, in order to have different styles applied to each case. Compositing operations are used to achieve this: first the landcover polygons are drawn in their on-land style, then water areas are cut out, landcover polygons are drawn in their over-water style (when they have one) from below, and finally water areas are drawn, from below as well.

Before-afters:

Overview of affected styles:

Notable points:

• This is mostly for the intertidal zone, but all water areas are concerned here.
• Transparency is not used, colours for both styles are wholly separate. This allows for a greater control over the style. The proposed colours have been mostly obtained through transparency though.
• These colours can most probably be improved. To minimise changes the colours of polygons that were already rendered over water (e.g. tidal flats) are unchanged.
• Polygon patterns are not affected. Rendering them together with fill colours would be nice, as it would allow for different patterns over land or water if desired (could be useful for rocky areas), and it would allow drawing them below water lines (streams). However in the current style these patterns are sometimes used to make nonsensical overlaps (like forest over lake) visible to the users. So the effects of changing this should be considered.
• This PR is compatible with rendering tidal channels (Support waterway=tidal_channel & retroactively waterway=stream over tidalflat #3865), differentiating water colours (Render ocean and seas differently than fresh water #3895), and adding a border to the ocean polygons (Restore linear coastline rendering #3926). But I want to keep this PR as small as possible.
• One notable unknown is how performance is affected. Using composite operations and drawing some features multiple times must increase render times, but I don’t know of a good way to test this.

[dch0ph]

Very nice!

Is this using Kosmtik for display / rendering?

I've tried testing the PR with my renderd set-up (mapnik 3.1.0 claimed), but the ocean dst-over fails:

I haven't had much success with the mapnik compositing operations, but this may reflect an ageing mapnik version. There is little chance of working out from the horrific mapnik documentation what was working correctly in what versions.

P.S. I haven't looked over the code in any detail, but did notice that you have a repeat SQL query for the ocean cutouts and ocean fills (these can be cached e.g. as the amenity-points query).

[imagico]

Thanks for working on this.

A number of remarks:

• The approach used is sound and IMO suitable for this style. As is to not differentiate landcover-area-symbols over water and over land - if pattern differentiation is desired for landcover that pattern should be rendered in the main landcover layer and not the overlay. See also Render bare_rock pattern in landcover-area-symbols #3851 (comment)
• Currently, as we render all water features in uniform color, the different layers for the backdrop are unnecessary, a single backdrop with a generated mercator square polygon is sufficient. See https://github.com/imagico/osm-carto-alternative-colors/blob/21d81b82ac14f6abfe08a9b96d88c1437fc5ef7b/project.mml#L536
• Rendering the glaciers in the landcover layer won't work because of the outlining of the glaciers.
• The idea to newly introduce a large number of new polygon fill colors is not going to work i think. I don't see a need to introduce any new colors at all with this change at all since the landcovers in question are well differentiated with patterns already.
• It is important that the landcovers over water in appearance are similar to one another and at the same time substantially different from the land rendering so that the distinction between land and water is universally well visible. This is not the case with the suggested colors and as a result the land-water distinction is often unclear.
• natural=sand should not be included in the landcovers rendered over water IMO. There is no consensus among mappers that natural=sand is equally usable for wet, waterlogged sand and for wind blown dunes. Mapping wet sand with natural=sand is globally a rare practice and us communicating to mappers that we do not specifically endorse overlaps between natural=sand and water seems a prudent choice. See Remove overlay pattern for natural=sand #3855 and sand vs shoal in tidal range #4513 for some further background on this.

[dch0ph]

• natural=sand should not be included in the landcovers rendered over water IMO.

Yes, natural=shoal with different surfaces seems the accepted tagging. That said, I'm struggling to differentiate natural=shoal from wetland=tidalflat, except that the later seems to imply a default surface=mud. Do we want tidal natural=beach + surface=X and natural=shoal + surface=X to render in the same way?

• The idea to newly introduce a large number of new polygon fill colors is not going to work i think. I don't see a need to introduce any new colors at all with this change at all since the landcovers in question are well differentiated with patterns already.

I agree that the landcover patterns should be doing most of the work. But I do think that some surfaces will be hard to differentiate on pattern alone against a blue background, and so limited use of new colours may be helpful e.g. surface=sand.

My main concern is whether to restrict the PR to ocean and optionally tidal natural=water (which we probably do want to render in a lighter shade as a next step). Effectively rendering landcover underneath permanent water, e.g. rivers, will firstly lead to a lot of changes; many rivers will be mapped directly over, say, natural=bare_rock and a lot of visual noise will be introduced if patterns are now rendered. Then it might encourage micromapping where mappers start to map the surfaces under water areas. The main benefit of this PR is for tidal areas, as shown above, where currently the tidal zone is obscured because the coastline is at the high tide limit.

[imagico]

I'm struggling to differentiate natural=shoal from wetland=tidalflat, except that the later seems to imply a default surface=mud

World wide most wetland=tidalflat mapped without surface are sand or mixed sand and finer material.

Do we want tidal natural=beach + surface=X and natural=shoal + surface=X to render in the same way?

Currently beach and shoal are rendered identically - changing that would be a separate design decision. See also #3864.

[tpetillon]

Currently, as we render all water features in uniform color, the different layers for the backdrop are unnecessary, a single backdrop with a generated mercator square polygon is sufficient.

I included this, thanks.

P.S. I haven't looked over the code in any detail, but did notice that you have a repeat SQL query for the ocean cutouts and ocean fills (these can be cached e.g. as the amenity-points query).

No longer needed thanks to the point above.

natural=sand should not be included in the landcovers rendered over water IMO.

Removed it.

The idea to newly introduce a large number of new polygon fill colors is not going to work i think.

That was my first attempt, but it didn’t look very nice. I retried it, and it looks like we can find a single colour that works well after all.

Colours and patterns still need some fine tuning. I haven’t fixed glaciers yet.

My main concern is whether to restrict the PR to ocean and optionally tidal natural=water

My first draft only rendered landcover polygons over the ocean, and I’m very unsure about including regular water polygons in this PR, so I can remove them. Filtering on tidal=yes might be a good compromise.

Current state:

[tpetillon]

Is this using Kosmtik for display / rendering?

Yes, with Mapnik 3.1.0.

[imagico]

As a reference for discussing design:

current master:

this PR:

AC-Style:

IMO using the tidalflat over water fill color also on the vegetated tidal wetlands has some appeal. It would avoid weird looking cases like here:

https://www.openstreetmap.org/#map=16/53.67236/6.97125

It will work less on non-tidal wetlands that are practically mapped over water, primarily reedbed and swamp (representing cases of vegetation growth in permanent standing water)

[dch0ph]

Is this using Kosmtik for display / rendering?

Yes, with Mapnik 3.1.0.

Thanks for clarifying. It seems that everything works fine with the raster output of mapnik 3.1.0, but there seem to be issues with the Cairo rendering (used for SVG/PDF). It would be a shame to lose vector output - high-quality print maps being an important personal use case. I suspect that these issues may be resolved in Mapnik 4.0.x (there do seem to be a lot of interesting fixes/editions in the 4.0 release), but I think that switching to Mapnik 4 is a discussion for another day, and breaking vector output is probably an acceptable trade-off for improved functionality.

My main concern is whether to restrict the PR to ocean and optionally tidal natural=water

My first draft only rendered landcover polygons over the ocean, and I’m very unsure about including regular water polygons in this PR, so I can remove them. Filtering on tidal=yes might be a good compromise.

I had misunderstood the significance of the river test (it was a river in a tidal/ocean area). But I do think including natural=water + tidal=yes would be useful. There are some interesting examples in the Humber:

The "sands" have been mapped as natural=mud even though at least Redcliff Sand looks like a sand bar. This is presumably because natural=mud is treated as an exception and is rendered "over" water in the landcover symbols layer. I guess this special case will be redundant in the current PR?

Currently beach and shoal are rendered identically - changing that would be a separate design decision. See also #3864.

That works. One design decision would be whether to only render natural=shoal in the tidal rendering, i.e. only select in "tidal landcover"?

[tpetillon]

Tidal landcover is now only rendered over oceans and water polygon tagged tidal=yes.

IMO using the tidalflat over water fill color also on the vegetated tidal wetlands has some appeal. It would avoid weird looking cases like here:

https://www.openstreetmap.org/#map=16/53.67236/6.97125

That’s one of the main motivations for having a fill colour for these polygons. Currently in many places the areas closer to the coastline appear more water-y than the tidal flats further at sea, when they spend less time under water. With a fill colour for all tidal landcover we can have a more consistent land to intertidal zone to sea transition.

The "sands" have been mapped as natural=mud even though at least Redcliff Sand looks like a sand bar. This is presumably because natural=mud is treated as an exception and is rendered "over" water in the landcover symbols layer. I guess this special case will be redundant in the current PR?

Yes, tidal flats become just one among the list of landcover polygons rendered over tidal areas. This way the incentive to tag some areas incorrectly (or in a less precise way) in order to make them show on the map should disappear.

Is this using Kosmtik for display / rendering?

Yes, with Mapnik 3.1.0.

Thanks for clarifying. It seems that everything works fine with the raster output of mapnik 3.1.0, but there seem to be issues with the Cairo rendering

That’s unfortunate. It would be interesting to test with Mapnik 4. If these issues are fixed, it would at least give a way forward where no functionality is lost. But I don’t know if Kosmtik is even compatible with it.

[imagico]

Tidal landcover is now only rendered over oceans and water polygon tagged tidal=yes.

We need to be careful with that for two reasons:

• we don't want to nudge mappers to broadly add tidal=yes to water polygons to ensure overlapping landcover is rendered correctly. Often riverbank polygons for example are not split at a well defined limit of the tidal influence and forcing mappers to do so (without the necesssary local knowledge) or to broadly add tidal=yes to the whole thing to get correct rendering would be counterproductive.
• not all landcovers that are rendered in a dedicated fashion over water with this PR are only mapped overlapping water in tidal settings. In particular beaches as a geomorphologically defined tag extend below the water level even in a non-tidal setting and can correctly be mapped as such. There is probably no consensus among mappers if natural=bare_rock is valid tagging for characterizing a permanently water covered lakebed/riverbed - but such mapping can be considered to make sense.

By the way: While, so far, the only landcovers suggested for special rendering over water are physical geography characterizations, this does not have to stay this way. It would, for example, be potentially worth considering rendering certain industrial landcovers (like wind parks, solar farms) in such fashion as well - and that would not be limited to a tidal setting.

It seems tidal=yes is used at least as much on landcover polygons as it is used on water polygons. Therefore interpreting it on landcovers to distinguish between tidal cases and non-tidal cases could be worth considering in some cases.

[dch0ph]

• we don't want to nudge mappers to broadly add tidal=yes to water polygons to ensure overlapping landcover is rendered correctly. Often riverbank polygons for example are not split at a well defined limit of the tidal influence and forcing mappers to do so (without the necessary local knowledge) or to broadly add tidal=yes to the whole thing to get correct rendering

As I understand it, the PR would be introducing rendering that is especially useful for tidal cover - where the normal landcover is periodically covered by water. So I don't think there would be an incentive to add tidal=yes to "static" water areas - it would be seen as tagging for the renderer.

I agree that "riverbank polygons for example are not split at a well defined limit of the tidal influence", but the split between river and ocean is particularly variable - in some cases, the coastline is fixed at the river mouth (leaving large areas as tidal river, as the Humber above), while in others the coastline is pulled far inland. Rendering tidal patterns over ocean but not tidal river would drive a lot of coastline changes.

• In particular beaches as a geomorphologically defined tag extend below the water level even in a non-tidal setting and can correctly be mapped as such. There is probably no consensus among mappers if natural=bare_rock is valid tagging for characterizing a permanently water covered lakebed/riverbed - but such mapping can be considered to make sense.

Yes, I can see this point. You can have a "river beach" or area of shingle that extends into the water, and I imagine that a lot of river / river areas are directly mapped over say natural=bare_rock. But it would add a lot of noise to start showing a pattern on thin areas. I think most mappers are content that single/sand "under" a river is not indicated - the "complaints" are focussed on tidal / ocean areas.

That’s unfortunate. It would be interesting to test with Mapnik 4

Let's ignore this. Using Carto for SVG/PDF is probably a fairly niche application. Mapnik 4.0.1 is available with the current (short life-time) Ubuntu release, so could in theory be tested. It will probably be next year before the Long Term Support version of Ubuntu defaults to Mapnik 4.

[tpetillon]

It seems tidal=yes is used at least as much on landcover polygons as it is used on water polygons. Therefore interpreting it on landcovers to distinguish between tidal cases and non-tidal cases could be worth considering in some cases.

Indeed, tagging tidal landcover features with tidal=yes is fairly widespread and well documented on the wiki.

What I think we want to avoid, in order to conform to current mapping practices:

• Drawing landcover over non-tidal but intermittent water as tidal landcover. When relevant, this is usually expressed through intermittent=yes.
• Drawing permanently underwater landcover as tidal landcover. In practice the only well-accepted tag for this kind of feature is natural=reef, and it already has its own rendering. But as you said, it could be extended to other types of surfaces, so we don’t want an automatic catch-all of every landcover that overlaps water.
• If possible, drawing landcover mapping errors (like regular forest over water) in a way that makes them look intended. The current scheme of drawing only symbols for these polygons is convenient to spot errors.
• Relying on the distinction between ocean and rivers. Placing the limit between the two is a complex problem, and mapping practices are not always in agreement from one area to the next. Sometimes tidal estuaries are included in the ocean, other times in water=river polygons.

What we can do is:

• Consider some tags as implicitly representing tidal areas when over water, like tidal flats.
• Allow some other landcover tags to represent tidal areas when over water, but only if they have the tidal=yes tag.

This way the case of a forest drawn over a river isn’t affected, and the rendering lets mappers decide what should be considered tidal through tags.

We could even opt for not rendering tidal landcover polygons over land. The idea here being that by forcing polygons to be either tidal or not tidal, we can avoid pushing mappers towards adding tidal=yes to large polygons just so that any potential part overlapping water gets a tidal style. Similarly to how encouraging people to add tidal=yes to wide water bodies isn’t desirable.

(In fact, it would even be possible to ignore ocean and water polygons altogether, do no cutouts, and simply rely on tidal=yes to select between styles.)

Of course some polygons that are intended to represent tidal areas are missing the tidal=yes. And beaches are a problem. There is no strong consensus for any mapping strategy, including whether or not they should be split at the high-water mark. They cannot be considered intrinsicallay tidal, because they are also used on lake shores.

It would, for example, be potentially worth considering rendering certain industrial landcovers (like wind parks, solar farms) in such fashion as well

That’s a good point, and a thing to consider for further improvements.

Mapnik 4.0.1 is available with the current (short life-time) Ubuntu release, so could in theory be tested.

I tried generating SVGs with Mapnik 4.0.1 and it works. The files are quite large though (multiple megabytes), due to patterns. I haven’t tried to do the same on master nor with an older version of Mapnik, so I don’t know if it could be a regression.

[imagico]

To avoid different people working for and arguing about different goals: #3854 and #3840 are about differentiating better landcover and water overlapping in general, not just tidal area depiction. That is a useful aim under the goal of the project independent of what the aim of the overlap in mapping is and even if the overlap is not intentional.

Overall we have essentially the following classes of landcover:

• things that - by mapper consensus - are only valid to be mapped overlapping water. That is essentially natural=reef, in addition things like landuse=aquaculture. For those it is important to not simply hide them over land so feedback is provided when mappers incorrectly map them overlapping land. The current rendering of natural=reef with an overlay pattern accomplishes that well.

• land only landcovers. This includes, for example, most physical vegetation related landcovers. Here it is an often accepted mapping practice that small waterbodies can be drawn over the landcover (like a small pond in a forest). Rendering should reflect that in the way that the the water area is not hidden in the overlap but also not drawn in a way that indicates an error. The current forest rendering accomplishes that well. Other landcovers are drawn fully below the water layer without any feedback on the overlap - which is fine as well.

• landcovers that are only valid in the tidal zone. That is essentially only wetland=tidalflat - though we currently render that styling wise identially with natural=mud - which is not limited in use to the tidal range. Here it is essential not to hide when wetland=tidalflat is mapped outside the tidal range but also not to obscure the coastline in those cases. Hence different styling over water and over land is needed. Over land this could be drawn in a dedicated 'error indicator' styling (see Refocus project goals to show where data is weak / incomplete #4723). Since wetland=tidalflat implies tidal range a dedicated tagging as tidal (via tidal=yes either on landcover or water area) is not considered necessary by mappers so it would be questionably if we require that.

• dedicated water edge landcovers that can exist in the tidal range but also outside of it:

  • natural=beach
  • coastal vegetated wetlands (saltmarsh, mangrove, swamp)
  • non-coastal water edge vegetated wetlands that can exist overlapping permanent water (reedbed, marsh, swamp)

  Here it is important not to obscure the water edge, to not fully hide the landcover when overlapping water and not to limit display fully to cases explicitly tagged as tidal. A differentiation between tidal and non-tidal cases (like having a plain color fill only in the tidal case while always showing the symbol overlay) can be considered - though it might make things too complicated for mappers to understand.

• not explicitly water related landcovers that are widely used for tidal range mapping. That is - in particular - natural=bare_rock, to a lesser extent natural=shingle. Here the question is if it is mapper consensus that an explicit mapping as tidal=yes is required (either on the landcover or on the water area). If that is the case that could be used in rendering.

• abstract landcovers (like landuse=industrial, leisure=pitch) that are also used over water or in the tidal range. These are currently not shown over water. If they are shown then that should - ideally - be in a dedicated styling to avoid hiding the water edge. It does not have to be a different fill color - an outline only rendering would, for example, be an option here as well.

Regarding intermittent water: We currently render that with an overlay pattern. The specific pattern is poorly readable and should be changed - but the principal idea of using a - partially opaque, partially fully transparent - pattern is sound. I have not seen any viable alternative suggestion so far. However, with such a pattern in use the fully transparent parts of course need to have the land version of the landcover shine through and not the water version. Otherwise it would be intuitively misleading.

These are my ideas on how to approach this matter - but feel welcome to develop ideas and argue for them beyond that. You should stay, however, close to current consensus mapping practice in OSM. A rendering concept that is based on an ideal mapping and tagging model that does not have broad support among mappers is not going to work for us.

[dch0ph]

I mostly concur with this, but with a few reservations

For those it is important to not simply hide them over land so feedback is provided when mappers incorrectly map them overlapping land. The current rendering of natural=reef with an overlay pattern accomplishes that well.

I'm not sure that the natural=reef pattern is sufficiently distinctive for most people to realise that, say, a reef in the middle of a desert was bad mapping.

Here it is essential not to hide when wetland=tidalflat is mapped outside the tidal range but also not to obscure the coastline in those cases. Hence different styling over water and over land is needed.

I'm wary of introducing "error stylings". The existing tools of either a pattern that is sufficiently distinctive so that errors are obvious (a lake in the middle of a wood needs a multi-polygon) or not rendering (leisure=pitch in an ocean will be hidden by the ocean layer) seem sufficient. There is the risk that mappers don't understand the error styling and adding SQL / designs for bad mapping seems inefficient. Adding a feature which is then not rendered is very direct mapper feedback!

That is - in particular - natural=bare_rock, to a lesser extent natural=shingle. Here the question is if it is mapper consensus that an explicit mapping as tidal=yes is required (either on the landcover or on the water area).

There's extremely low usage of tidal=yes on shingle (too low to be countered by taginfo) or bare_rock (12k out of 1.1m). Given that tidal zones should be covered by tidal water (mapping high water mark), I think we should ignore tidal on landcovers.

In terms of the current PR, am I right in thinking that the outstanding issues are:

• The queries above (which might or might not require changes).
• Deciding whether non-ocean water with tidal=yes should be included with ocean areas
  ?

[imagico]

For those it is important to not simply hide them over land so feedback is provided when mappers incorrectly map them overlapping land. The current rendering of natural=reef with an overlay pattern accomplishes that well.

I'm not sure that the natural=reef pattern is sufficiently distinctive for most people to realise that, say, a reef in the middle of a desert was bad mapping.

No, but that is not a practically relevant problem. The common type of larger error in natural=reef mapping is to not cut out the island from ring reefs around islands. And there the overlay pattern provides suitable feedback to make mappers aware of the problem. Complete misuse to map something fundamentally different is not something we need to worry about here.

Here it is essential not to hide when wetland=tidalflat is mapped outside the tidal range but also not to obscure the coastline in those cases. Hence different styling over water and over land is needed.

I'm wary of introducing "error stylings". [...]

This is not my main point here. The important thing is not to hide mapping in situation where it cannot be correct. And since different styling over water and over land is therefore going to be needed, making the styling over land (which - in case of tidalflat - is universally incorrect mapping) decidedly not implying something is correctly mapped is worth considering. It does not have to loudly scream error or even appear distinctly wrong. It should just not positively communicate to the mapper that this is correct mapping.

That is - in particular - natural=bare_rock, to a lesser extent natural=shingle. Here the question is if it is mapper consensus that an explicit mapping as tidal=yes is required (either on the landcover or on the water area).

There's extremely low usage of tidal=yes on shingle (too low to be countered by taginfo) or bare_rock (12k out of 1.1m). Given that tidal zones should be covered by tidal water (mapping high water mark), I think we should ignore tidal on landcovers.

The absolute numbers say relatively little here since both bare_rock and shingle are rarely used in tidal settings. The important thing to know would be what percentage of those polygons in the tidal range have a tidal=yes. That is not easy to find out since you need to analyze the coastline polygons and the landcover polygons together.

natural=shingle is not generally used in coastal settings since most coastal shingle is beaches and natural=beach then is the more descriptive tagging. Hence the focus here is on natural=bare_rock.

[dch0ph]

The common type of larger error in natural=reef mapping is to not cut out the island from ring reefs around islands. And there the overlay pattern provides suitable feedback to make mappers aware of the problem.

Fine - I hadn't visualised the scenario you were thinking of.

The absolute numbers say relatively little here since both bare_rock and shingle are rarely used in tidal settings. The important thing to know would be what percentage of those polygons in the tidal range have a tidal=yes.

True, but 12k uses is a pretty small number globally given that the majority are likely to be covered by tidal water. Either way, I don't think adding specific rendering for tidal=yes on natural=bare_rock is essential.

Ultimately I was keen to narrow down if any further changes are needed in this PR.