Support offline vector maps (MBTiles, MVT, gzipped pbf)

[ktzouno]

Supporting OpenMapTiles offline Vector maps (MBtiles, MVT, gzipped-pbf format) with overzoom functionality.
(Related to #708 )

Test mbtiles map:
vector.mbtiles.zip

[devemux86]

Cannot load vtm-android module with extra dependencies, which may be not useful by all users.

We could handle it in separate vtm-android-mvt module, so that everyone can pick only the needed dependencies.

[devemux86]

Thanks for the work!

Current approach with the workers seems complicated and finding them in same class.
We could follow the former separated structure retaining library's naming:

• MBTilesBitmapTileSource + MBTilesBitmapTileDataSource (like now)
• MBTilesMvtTileSource + MBTilesMvtTileDataSource (in new vtm-android-mvt module)
• They can extend MBTilesTileSource + MBTilesTileDataSource (for common fields / methods)

That way everyone can select the wanted tile source and the mvt dependencies separately if needed.

[ktzouno]

@devemux86 thanks for your fast review. I created the new vtm-android-mvt module, so the automation I introduced with the workers is not available anymore and I converted it to the architecture you suggested above.

[devemux86]

Thanks for the nice clean work!

Squashed and merged via a9e18a2 (with minor improvements).

[ktzouno]

Seems that GeometryBuffer is not thread safe, so the setOrCheck() method throws: https://github.com/mapsforge/vtm/blob/master/vtm/src/org/oscim/core/GeometryBuffer.java#L418 Maybe it can be resolved in a better way that creating 1 MvtDecoder per thread with the ThreadLocal . On 09.09.19, 11:50, "Emux" <[email protected]> wrote: @devemux86 commented on this pull request. In vtm-android-mvt/src/org/oscim/android/mvt/tiling/source/mbtiles/MBTilesMvtTileDataSource.java:

+

+import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.util.Arrays; +import java.util.List; +import java.util.Locale; +import java.util.zip.GZIPInputStream; + +public class MBTilesMvtTileDataSource extends MBTilesTileDataSource { +    private final static List<String> SUPPORTED_FORMATS = Arrays.asList("pbf"); + +    public static final String WHERE_FORMAT = "zoom_level=%d AND tile_column=%d AND tile_row=%d"; + +    private final String mLocale; + +    private final ThreadLocal<MvtTileDecoder> mThreadLocalMvtTileDecoders = new ThreadLocal<MvtTileDecoder>() { Why the need of ThreadLocal for the decoder? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[ktzouno]

There are 2 drawbacks in the Overzoom for the Online Tiles: # Requires to pre-define on which zoomLevel Overzooming should be applied – but zoomLevels closer to 0 will contain tiles for most of the planet, while bigger zoomLevels will be optimized for the area that the App cares about. # Will try once per zoomLevel to get a tile. So theoretically if overzooming from zoomLevel 14 could trigger 14 DB queries. With my implementation when the App tries to load a tile that doesn’t exist, no matter what the zoomLevel is, it will try to Overzoom with a query that loads all the tiles for that spot on all zoomLevels. Example: if a tile in zoomLevel 14 is not found, the Overzoom will request the equivalent tiles on zoomlevels 13, 12, 11,…, 0 all in the same query, and use “ORDER BY zoomLevel DESC LIMIT 1” to get the “best” tile. On 09.09.19, 12:15, "Emux" <[email protected]> wrote: I wonder if overzoom could be handled like we do already in online tile sources? e.g. see OpenMapTilesMvtTileSource — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[devemux86]

creating 1 MvtDecoder per thread with the ThreadLocal

Indeed that's not nice.

MvtTileDecoder uses a cached MapElement for performance reasons.
But we had not any synchronization issues with all other mvt tile sources.

[devemux86]

Actually that is how overzoom works and specified for online tile sources, where all above tiles do exist.

Requires to pre-define on which zoomLevel Overzooming should be applied

That is expected and known for all online tile sources according to their documentation.

a query that loads all the tiles for that spot on all zoomLevels

Some providers have small overzoom levels and heavy tiles, so that could be bad for performance.

Databases are certainly different from online requests.
VTM overzoom was implemented with the 2nd concept.

[ktzouno]

Requires to pre-define on which zoomLevel Overzooming should be applied

That is expected and known for all online tile sources according to their documentation.

Lets assume we have an mbtiles DB for my city:

Area	zoomLevels
My City	0-14 (guaranteed)
My Country	0-10 (maybe)
My Continent	0-4 (maybe)

So assuming that the app allows the user to zoom anywhere on the planet, the available tiles' zoomLevel will be much less than 14.
14 is the guaranteed zoomLevel assuming that the user Overzooms inside the My City area.

[ktzouno]

a query that loads all the tiles for that spot on all zoomLevels

Some providers have small overzoom levels and heavy tiles, so that could be bad for performance.

I didn't spend much time on the performance. In case it wasn't clear, my overzooming-query won't "process" all the tiles it gets back from all zoomLevels, only the max-zoomLevel tile.

But I'd check 3 Overzooming modes to assess their performance. e.g. on max-zoomLevel 14 DB:

1. One query for the current zoomLevel, one per zoomLevel till the 0 one => can lead to 14 queries per tile.
2. One query for the current zoomLevel, one more query to Overzoom if needed => 2 queries per tile (I implemented this)
3. One query for all zoomLevels => 1 query per tile.

[devemux86]

I was talking about online tile sources, not offline MBTiles, they're different concepts.

If test the various online mvt tile sources, can zoom / scroll anywhere in the planet beyond OMT overzoom 14 and the map should fill with content from above levels step by step.
Though because the overzoom level is small, their tiles are usually heavy and not have much performance compared to e.g. OSciM tiles.