26 Commits

Author SHA1 Message Date
Ludovic Delauné
924ccd6015
remove useless sql selection (#752) 2020-03-13 15:14:30 -04:00
Jorge Sanz
ace759590e
Parallel capability to layer functions (#728)
This PR allows queries to be parallelized on recent versions of Postgres. The `PARALLEL SAFE` modifier has been added to the layer functions and a PLPGSQL function to convert strings into number has been replaced.

`PARALLEL SAFE` is a modifier for `CREATE FUNCTION` available since Postgres 9.6, so this change does not break current OpenMapTiles supported database version. More details about this topic [here](https://www.postgresql.org/docs/current/parallel-safety.html) and at the reference documentation for [`CREATE FUNCTION`](https://www.postgresql.org/docs/current/sql-createfunction.html).

### Testing procedure

The procedure to test this was:

* Imported `spain.pbf` in a clean environment
* Dumped the OpenMapTiles database from the Postgres Docker image
* Created a clean Postgres 12 database using the default Docker image
* Installed `postgis` 3 from the default Debian package and `osml10n` 2.5.8 from the repository (`make`, etc.)
* Restored the dump
* Lowered the postgres planner parameters for triggering parallel plans:
```sql
set parallel_setup_cost = 5;
set parallel_tuple_cost = 0.005;
```
* Manually added the `PARALLEL SAFE` modifier to each function involved in layer queries (not on updates or inserting functions).
* For each layer, run a testing query to confirm parallel workers were created, something like this:
```sql
explain analyze 
select * from layer_aerodrome_label(tilebbox(8,128,95),10,null)
union all
select * from layer_aerodrome_label(tilebbox(8,128,97),10,null);
```
* After all the layers were processed and confirmed to start parallel executions, a more complete example was run. This example just retrieves the geometries for all the layers from the same tile but without using any MVT related function.

<details><summary>Testing query</summary>

```sql
-- Using the function layer_landuse
explain analyze 
select geometry from layer_water(tilebbox(14,8020,6178),14)
union all
select geometry from layer_waterway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landcover(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landuse(tilebbox(14,8020,6178),14)
union all
select geometry from layer_mountain_peak(tilebbox(14,8020,6178),14)
union all
select geometry from layer_park(tilebbox(14,8020,6178),14)
union all
select geometry from layer_boundary(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aeroway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation(tilebbox(14,8020,6178),14)
union all
select geometry from layer_building(tilebbox(14,8020,6178),14)
union all
select geometry from layer_water_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_place(tilebbox(14,8020,6178),14)
union all
select geometry from layer_housenumber(tilebbox(14,8020,6178),14)
union all
select geometry from layer_poi(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aerodrome_label(tilebbox(14,8020,6178),14);
```
</details>

You can inspect the execution plan and results on [this page](https://explain.dalibo.com/plan/3z). Also [attaching](https://github.com/openmaptiles/openmaptiles/files/3951822/explain-tile-simple.tar.gz) the query and JSON output for future reference. The website gives a ton of details, but you may want to search for nodes mentioning `workers` or `parallel` like in this area referring to `osm_border` or `osm_aeroway_linestring` entities

![image](https://user-images.githubusercontent.com/188264/70647153-9cac9300-1c48-11ea-96ea-ac7a1e2f4a79.png)

### Next steps

Since the execution plan is not showing a parallel append at the top level, meaning it's not running each layer individually, I want to continue experimenting with parameters and queries to see if it's possible to even parallelize more the request.

I will post my finding here, even no change in the code should happen.


cc. @nyurik

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2020-01-31 19:36:02 -05:00
Eva J
b62619c00c etl_diagram update 2019-11-27 12:50:24 +01:00
Eva J
60b4a4cebc removing ne_10m source 2019-11-27 11:51:15 +01:00
Eva J
fcee8571fb adding waterway=dam 2019-11-20 18:05:07 +01:00
Eva J
c320b92a4b place=suburb and place=neighbourhood added 2019-04-11 11:57:07 +02:00
nlehuby
940f70a415 Handle theme_park as a landuse 2018-01-12 15:40:54 +01:00
stirringhalo
90690d2a39 Switch to ZRes (#214) 2017-04-25 18:03:29 -04:00
stirringhalo
bab5c3d4b6 Switch from geom to geometry 2016-12-20 22:32:34 -05:00
ImreSamu
f5ac3def5e etldoc fixes 2016-12-04 02:52:52 +01:00
Lukas Martinelli
5ec4f0766b Add zoom level range slots to landcover diagram 2016-12-01 08:29:34 +00:00
Lukas Martinelli
c1b1913c7c Fix wrong etl docs 2016-11-30 18:38:28 +00:00
Lukas Martinelli
1ef69a9fd8 Show residential from OSM on z9 2016-11-30 14:05:15 +00:00
Lukas Martinelli
7ca0fc6646 Remove subclass for landuse 2016-11-30 09:32:14 +00:00
Lukas Martinelli
9ba6a0bb0f Fix missing column issues with landcover and landuse 2016-11-30 09:06:40 +00:00
Lukas Martinelli
6c4ac01dfb More intermediate zoom levels for residential 2016-11-29 14:00:32 +00:00
Lukas Martinelli
327bdc7313 Remove mappings from landcover and landuse 2016-11-28 19:16:00 +00:00
stirringhalo
4f795b6287 Quiet down 'has type unknown' 2016-11-15 23:37:41 -05:00
ImreSamu
5580f42458 add etldoc to layer_landuse 2016-11-10 02:12:39 +01:00
lukasmartinelli
2d55f4fcf0 Map leisure=park into landuse 2016-10-30 18:41:33 +01:00
lukasmartinelli
81876adcf4 Remove strictness from class function in landuse 2016-10-28 12:01:22 +02:00
lukasmartinelli
9c50026f16 Remove osm_id in vector tiles coz of huge size savings 2016-10-25 19:15:51 +02:00
lukasmartinelli
7ec0db640c Forgot bbox constraint 2016-10-25 12:03:24 +02:00
lukasmartinelli
030e1e3204 Add national parks 2016-10-24 21:49:03 +02:00
lukasmartinelli
bdea298990 Introduce landuse classes 2016-10-24 14:28:09 +02:00
lukasmartinelli
79d04f3792 Integrate urban into landuse 2016-10-24 12:02:33 +02:00