51 Commits

Author SHA1 Message Date
Jorge Sanz
ace759590e
Parallel capability to layer functions (#728)
This PR allows queries to be parallelized on recent versions of Postgres. The `PARALLEL SAFE` modifier has been added to the layer functions and a PLPGSQL function to convert strings into number has been replaced.

`PARALLEL SAFE` is a modifier for `CREATE FUNCTION` available since Postgres 9.6, so this change does not break current OpenMapTiles supported database version. More details about this topic [here](https://www.postgresql.org/docs/current/parallel-safety.html) and at the reference documentation for [`CREATE FUNCTION`](https://www.postgresql.org/docs/current/sql-createfunction.html).

### Testing procedure

The procedure to test this was:

* Imported `spain.pbf` in a clean environment
* Dumped the OpenMapTiles database from the Postgres Docker image
* Created a clean Postgres 12 database using the default Docker image
* Installed `postgis` 3 from the default Debian package and `osml10n` 2.5.8 from the repository (`make`, etc.)
* Restored the dump
* Lowered the postgres planner parameters for triggering parallel plans:
```sql
set parallel_setup_cost = 5;
set parallel_tuple_cost = 0.005;
```
* Manually added the `PARALLEL SAFE` modifier to each function involved in layer queries (not on updates or inserting functions).
* For each layer, run a testing query to confirm parallel workers were created, something like this:
```sql
explain analyze 
select * from layer_aerodrome_label(tilebbox(8,128,95),10,null)
union all
select * from layer_aerodrome_label(tilebbox(8,128,97),10,null);
```
* After all the layers were processed and confirmed to start parallel executions, a more complete example was run. This example just retrieves the geometries for all the layers from the same tile but without using any MVT related function.

<details><summary>Testing query</summary>

```sql
-- Using the function layer_landuse
explain analyze 
select geometry from layer_water(tilebbox(14,8020,6178),14)
union all
select geometry from layer_waterway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landcover(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landuse(tilebbox(14,8020,6178),14)
union all
select geometry from layer_mountain_peak(tilebbox(14,8020,6178),14)
union all
select geometry from layer_park(tilebbox(14,8020,6178),14)
union all
select geometry from layer_boundary(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aeroway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation(tilebbox(14,8020,6178),14)
union all
select geometry from layer_building(tilebbox(14,8020,6178),14)
union all
select geometry from layer_water_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_place(tilebbox(14,8020,6178),14)
union all
select geometry from layer_housenumber(tilebbox(14,8020,6178),14)
union all
select geometry from layer_poi(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aerodrome_label(tilebbox(14,8020,6178),14);
```
</details>

You can inspect the execution plan and results on [this page](https://explain.dalibo.com/plan/3z). Also [attaching](https://github.com/openmaptiles/openmaptiles/files/3951822/explain-tile-simple.tar.gz) the query and JSON output for future reference. The website gives a ton of details, but you may want to search for nodes mentioning `workers` or `parallel` like in this area referring to `osm_border` or `osm_aeroway_linestring` entities

![image](https://user-images.githubusercontent.com/188264/70647153-9cac9300-1c48-11ea-96ea-ac7a1e2f4a79.png)

### Next steps

Since the execution plan is not showing a parallel append at the top level, meaning it's not running each layer individually, I want to continue experimenting with parameters and queries to see if it's possible to even parallelize more the request.

I will post my finding here, even no change in the code should happen.


cc. @nyurik

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2020-01-31 19:36:02 -05:00
Yuri Astrakhan
c9e7ad90c6
Remove unneeded "else null" in conditions (#732)
Minor code cleanup:
SQL already returns NULL in the "WHEN" condition
if it is not matched by any of the cases.

Co-authored-by: Eva Jelinkova <evka.jelinkova@gmail.com>
2020-01-22 17:24:28 -05:00
Eva Jelinkova
02c202e2f7
Revert "Remove the area condition on highway polygon." 2019-12-11 22:22:59 +01:00
François de Metz
cf219c1e09
Include only polygons and multipolygons highways on zoom >= 13. 2019-11-05 12:31:54 +01:00
Eva Jelinkova
bdeb6c8b2e
Merge branch 'master' into bicycle-foot-horse-mtb_scale 2019-10-31 18:52:12 +01:00
zstadler
2738c649ca Fix typos 2019-10-26 22:52:14 +03:00
zstadler
556bfb889d
Merge branch 'master' into bicycle-foot-horse-mtb_scale 2019-10-25 16:09:44 +03:00
golubev
81c90be19b layers/transportation/: fix SQL mistakes, modify indices (#321) 2019-10-04 14:00:14 +03:00
golubev
0636466cfd layers/transportation/: differentiate roads under construction (#321) 2019-10-04 13:08:49 +03:00
zstadler
2357bde1b9 Add bicycle, foot, horse, and mtb_scale to the transportation layer
Resolve https://github.com/openmaptiles/openmaptiles/issues/422
Resolve https://github.com/openmaptiles/openmaptiles/issues/512
Resolve https://github.com/openmaptiles/openmaptiles/issues/602

An extended and refined alternative to https://github.com/openmaptiles/openmaptiles/pull/573
2019-04-08 00:08:21 +03:00
Phyks (Lucas Verney)
da00063e0d Add surface field for highways
Keep surface field from OSM on highways, generalize it to two values:
"paved" and "unpaved".

This is a fix for #389 and a partial fix for #422.
2019-01-09 15:59:32 +01:00
jirik
765d6fbc76 Add line piers 2018-11-07 16:19:19 +01:00
jirik
4304c756ad Add polygon piers 2018-11-07 16:19:19 +01:00
Jiri Kozel
fce0ba1ce1
Add layer attribute to transportation (#441)
Fix #280
2018-04-11 15:28:23 +02:00
Jiri Kozel
e6efe363ed
Add polygon bridges to transportation (#437) 2018-04-04 10:39:46 +02:00
Frédéric Rodrigo
b211beafe2 Use direction as type for oneway, support oneway-opposite #376 2018-01-16 10:29:07 +01:00
jirik
e95f2d476b Add layer, level, and indoor tags for footways and steps 2017-11-09 10:13:53 +01:00
jirik
1f129c4184 Add subclass of paths in transportation 2017-11-09 08:51:14 +01:00
jirik
1cdf726384 Add platforms to transportation as paths 2017-11-09 08:51:14 +01:00
jirik
68785c1a21 Add shipping ways (ferries) to transportation 2017-11-08 16:28:30 +01:00
jirik
634de5e474 Add aerialways (cable cars) to transportation 2017-11-08 16:28:30 +01:00
jirik
137667b235 Create subclass attribute of railways 2017-11-08 16:00:13 +01:00
jirik
98aba61b05 Show narrow gauges sooner 2017-11-08 16:00:13 +01:00
jirik
ed90400fef Show rails and light rails sooner 2017-11-08 16:00:13 +01:00
jirik
3bb8a4bdde Fix #269 ETL diagram 2017-07-20 19:41:56 +02:00
Jiri Kozel
881e1c0881 Show railways one level sooner (#270)
* Adjust st_length filter based on zres in transportation layer

Probably related to #258

* Adjust building tolerance

* Show railways one level sooner
2017-05-31 15:28:06 +02:00
Jiri Kozel
8a44e72328 Adjust st_length filter based on zres in transportation layer (#259)
Probably related to #258
2017-05-23 16:54:59 +02:00
Jiri Kozel
a9b9c79a4b Fix #250 for zoom 9+ (#256) 2017-05-23 11:11:19 +02:00
jirik
f829f6cd19 Update diagrams of transportation and transportation_name 2017-03-31 13:48:57 +02:00
MartinMikita
782c9e0ead Fixed index and typo in doc. 2017-03-31 13:48:57 +02:00
MartinMikita
6bff55ae03 Added merged simplified transportation for zoom 8 2017-03-31 13:48:57 +02:00
MartinMikita
9781a0e671 Added merged simplified transportation for zoom 7 2017-03-31 13:48:57 +02:00
MartinMikita
52933f5cfc Separated zoom level 5 and 6, with different simplify tolerance and length of geometry. 2017-03-31 13:48:57 +02:00
jirik
cfc65c6884 Generalize roads more at zoom levels 4 -- 6 2017-03-31 13:48:57 +02:00
jirik
8ae45dd777 Update diagrams of transportation 2017-03-15 13:55:18 +01:00
jirik
ca1b9a95a8 Adjust zoom levels of transportation layer, stop using ne_10m_roads source 2017-03-15 11:05:16 +01:00
Lukas Martinelli
23c276f4dd Show generalized railways after z11 #146 2017-01-20 17:47:02 +01:00
Lukas Martinelli
95bcf9c1c1 Pull major railway up to z12 #77 2017-01-18 13:58:55 +01:00
stirringhalo
bab5c3d4b6 Switch from geom to geometry 2016-12-20 22:32:34 -05:00
Lukas Martinelli
a109ffd72e Remove subclass from transport 2016-12-02 12:36:10 +00:00
Lukas Martinelli
3e229a7b3c Map highway=step. Fixes #84 2016-12-02 11:15:47 +00:00
Lukas Martinelli
79e6f57ad7 Use only ne_10m_roads from Natural Earth #81 2016-12-01 14:23:14 +00:00
Lukas Martinelli
be349acd58 Fix etl doc issues with z14 2016-12-01 10:11:03 +00:00
Lukas Martinelli
4df25f25ef Fix wrong etldoc slots in transportation 2016-11-30 19:08:01 +00:00
Lukas Martinelli
d932fb96b3 Limit service values 2016-11-30 13:19:17 +00:00
Lukas Martinelli
4da3e3cc02 Leave attributes as NULL on low zoom levels 2016-11-28 14:19:16 +00:00
Lukas Martinelli
1d3171185b Split railway and highway into different tables 2016-11-28 13:18:02 +00:00
Lukas Martinelli
a04d5776e6 Use ClearTables scheme for classes 2016-11-28 11:01:43 +00:00
Lukas Martinelli
65419bdb84 Use service field on different zoom levels 2016-11-26 21:11:43 +00:00
Lukas Martinelli
e9f4c31e0a Return railways in transportation 2016-11-26 21:08:30 +00:00