21 Commits

Author SHA1 Message Date
Jorge Sanz
ace759590e
Parallel capability to layer functions (#728)
This PR allows queries to be parallelized on recent versions of Postgres. The `PARALLEL SAFE` modifier has been added to the layer functions and a PLPGSQL function to convert strings into number has been replaced.

`PARALLEL SAFE` is a modifier for `CREATE FUNCTION` available since Postgres 9.6, so this change does not break current OpenMapTiles supported database version. More details about this topic [here](https://www.postgresql.org/docs/current/parallel-safety.html) and at the reference documentation for [`CREATE FUNCTION`](https://www.postgresql.org/docs/current/sql-createfunction.html).

### Testing procedure

The procedure to test this was:

* Imported `spain.pbf` in a clean environment
* Dumped the OpenMapTiles database from the Postgres Docker image
* Created a clean Postgres 12 database using the default Docker image
* Installed `postgis` 3 from the default Debian package and `osml10n` 2.5.8 from the repository (`make`, etc.)
* Restored the dump
* Lowered the postgres planner parameters for triggering parallel plans:
```sql
set parallel_setup_cost = 5;
set parallel_tuple_cost = 0.005;
```
* Manually added the `PARALLEL SAFE` modifier to each function involved in layer queries (not on updates or inserting functions).
* For each layer, run a testing query to confirm parallel workers were created, something like this:
```sql
explain analyze 
select * from layer_aerodrome_label(tilebbox(8,128,95),10,null)
union all
select * from layer_aerodrome_label(tilebbox(8,128,97),10,null);
```
* After all the layers were processed and confirmed to start parallel executions, a more complete example was run. This example just retrieves the geometries for all the layers from the same tile but without using any MVT related function.

<details><summary>Testing query</summary>

```sql
-- Using the function layer_landuse
explain analyze 
select geometry from layer_water(tilebbox(14,8020,6178),14)
union all
select geometry from layer_waterway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landcover(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landuse(tilebbox(14,8020,6178),14)
union all
select geometry from layer_mountain_peak(tilebbox(14,8020,6178),14)
union all
select geometry from layer_park(tilebbox(14,8020,6178),14)
union all
select geometry from layer_boundary(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aeroway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation(tilebbox(14,8020,6178),14)
union all
select geometry from layer_building(tilebbox(14,8020,6178),14)
union all
select geometry from layer_water_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_place(tilebbox(14,8020,6178),14)
union all
select geometry from layer_housenumber(tilebbox(14,8020,6178),14)
union all
select geometry from layer_poi(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aerodrome_label(tilebbox(14,8020,6178),14);
```
</details>

You can inspect the execution plan and results on [this page](https://explain.dalibo.com/plan/3z). Also [attaching](https://github.com/openmaptiles/openmaptiles/files/3951822/explain-tile-simple.tar.gz) the query and JSON output for future reference. The website gives a ton of details, but you may want to search for nodes mentioning `workers` or `parallel` like in this area referring to `osm_border` or `osm_aeroway_linestring` entities

![image](https://user-images.githubusercontent.com/188264/70647153-9cac9300-1c48-11ea-96ea-ac7a1e2f4a79.png)

### Next steps

Since the execution plan is not showing a parallel append at the top level, meaning it's not running each layer individually, I want to continue experimenting with parameters and queries to see if it's possible to even parallelize more the request.

I will post my finding here, even no change in the code should happen.


cc. @nyurik

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2020-01-31 19:36:02 -05:00
Yuri Astrakhan
c9e7ad90c6
Remove unneeded "else null" in conditions (#732)
Minor code cleanup:
SQL already returns NULL in the "WHEN" condition
if it is not matched by any of the cases.

Co-authored-by: Eva Jelinkova <evka.jelinkova@gmail.com>
2020-01-22 17:24:28 -05:00
golubev
73307a610b layers/transportation_name/: add roads under construction, fix SQL error (#321) 2019-10-04 14:50:05 +03:00
jirik
63f6915fab Add missing transportation_name features
Fix #372
2017-11-25 11:10:01 +01:00
jirik
e95f2d476b Add layer, level, and indoor tags for footways and steps 2017-11-09 10:13:53 +01:00
jirik
0b94704436 Add subclass of paths in transportation_name 2017-11-09 10:13:53 +01:00
jirik
1cdf726384 Add platforms to transportation as paths 2017-11-09 08:51:14 +01:00
Jiri Kozel
51bc8fad35 Multilinguality (#279)
Improve multilinguality: names in 57 languages, name:latin, name:nonlatin, name_int. Fixes #211 #252 #80.

See #279 for more info.
2017-06-12 17:53:47 +02:00
jirik
4c6d30066d Add German names (name_de), unify English names (name_en) 2017-03-17 12:56:54 +01:00
jirik
fad3cab2cd Adjust zoom levels of transportation_name, update diagrams 2017-03-15 13:53:09 +01:00
jirik
c026da1324 Change default value of transportation_name's network attribute 2017-03-15 11:10:06 +01:00
jirik
6b523a5e0a Adjust zoom levels of transportation_name layer 2017-03-15 11:08:26 +01:00
jirik
5568d3ba44 Add default value to network attribute of layer_transportation_name 2017-03-01 15:59:10 +01:00
jirik
5db1b0e80e Add network type to transportation_name layer, adjust 'ref' attribute 2017-03-01 14:55:02 +01:00
Lukas Martinelli
0fb7981bb9 Ensure empty name is NULL in name_en 2017-01-18 16:26:11 +01:00
Lukas Martinelli
25fea38fe2 Add name_en to transportation_name #88 2017-01-18 16:13:37 +01:00
ImreSamu
f5ac3def5e etldoc fixes 2016-12-04 02:52:52 +01:00
Lukas Martinelli
681b50ff1c Remove subclass from transportation_name 2016-12-02 12:39:10 +00:00
Lukas Martinelli
de2f538ef5 Use new classes in transport_name and add network 2016-11-28 11:09:57 +00:00
Lukas Martinelli
e9f4c31e0a Return railways in transportation 2016-11-26 21:08:30 +00:00
Lukas Martinelli
35b6951c81 Move highway and railway into transportation 2016-11-26 20:22:18 +00:00