GeoPackage Executable Test Suite

I have an update to my previous post regarding conformance testing. OGC has recently approved and released the Executable Test Suite (ETS) for GeoPackage 1.2. This tool allows for the testing of individual GeoPackages to verify conformance and identify any non-compliant elements. Organizations can get OGC certified if they pass the test. The suite works on all GeoPackage versions from 1.0 to 1.2.1 - it will detect the GeoPackage version and alter the test requirements where appropriate. The test supports the following components:

• Core
• Features
• Tiles
• Attributes
• Extension Mechanism
• Non-Linear Geometry Types
• RTree Spatial Indexes
• Tiles Encoding WebP
• Metadata
• Schema
• WKT for Coordinate Reference Systems
• Tiled Gridded Coverage Data

There are two ways to run the ETS:

• If you have a GeoPackage that you want to test, go to the OGC Validation Website. From there, you can sign in (creating an account first if needed) and create a new session. Then you can select GeoPackage 1.2 from the standard list, provide your GeoPackage (by URL or as a file upload), and run the test. The tool will provide a report indicating tests that passed, failed, or were skipped.
• If you want to run the test suite locally or contribute to its development, see the GitHub Repository. There are instructions for compiling and running the tool locally.

I recommend the ETS for anyone who plans to use GeoPackage in an operational setting, particularly when interoperability with other GeoPackage implementations is desired. Produce a GeoPackage that is representative of your operational use, run it through the ETS, and make verify that it is compliant. That's all there is to it!

Versioning of Extensions?

The GeoPackage Extension Mechanism gives stakeholders a way to add GeoPackage capabilities in a way that minimizes interoperability risks to systems that do not understand the extension. From my perspective, if different implementers are going to implement similar capabilities, then I want them to implement in the same way. That is the whole point of interoperability. The theory is that a client that does not understand an extension will ignore the data. I believe that the extension mechanism is a GeoPackage strength.

The GeoPackage Standards Working Group has two questions for the community:

1. Is the extension mechanism working the way we want it to? Are we getting new capabilities? Are we getting the interoperability we are looking for?
2. What should we do if an OGC-adopted extension has to change? 
• We are leery of adding a "version" column to gpkg_extensions because GeoPackage clients that only understand version 1.2 or prior wouldn't even know about it. It is borderline whether this is a breaking change.
• An alternative that has some backing is to allow extensions to evolve as long as the changes are non-breaking but to force an extension to take a new name if the changes are breaking. 

We would like to get positive answers to these questions because there are a number of initiatives going on that have the potential to add a number of new adopted extensions. Do we need to pivot? I believe the answer is "no" but it is possible I am too close to the situation to make a fair assessment.

What About Vector Tiles in GeoPackage?

I periodically get questions about whether GeoPackage can/will/should support vector tiles. Without getting into the benefits or drawbacks of vector tiles, let's just say it is an emerging capability that is getting a lot of community interest. As it turns out, supporting vector tiles in GeoPackage is the easy part. We already have a precedent for handling types other than features and raster tiles - the tiled gridded coverage extension.

The real issue is that OGC has not yet adopted an approach for vector tiling. What is the content encoding for a vector tile? Is it Protocol Buffers (as per Mapbox Vector Tiles - MVT)? GeoJSON? Something else? OGC members are reluctant to support a vector tiling approach that is only used with GeoPackage. (Supporting one-off solutions often has a poor cost-benefit story.) OGC simply needs to make a decision and a community standard would suffice.

Once OGC makes a decision, it is a straightforward matter to support it in GeoPackage. We would create a new extension that would define a new content type that depends on the tiles option. I don't think there would be any other ancillary tables like were needed for tiled gridded coverages, but if there are we can add those. This would get us to a community extension which will satisfy some people. If we want an official (OGC adopted) extension then we have to go through the normal process of OAB approval, open comment period, TC vote, etc. This process will probably go smoothly as long as we have proof of implementation.

UPDATE: After talking with a few people, I went ahead and drafted a community extension based on the MVT approach.

UPDATE #2: In response to this issue, OGC initiated the Vector Tiles Pilot.

We Want Your (Technical) Feedback

Earlier this month, GeoPackage 1.2.1 (a corrigendum release) was approved by the OGC Technical Committee. In this release, we focused on clarifications and consistency. Look for publication and formal announcement soon. Until this occurs, you can review the release notes. 

I thought things would slow down after this but I was wrong. There has been a bunch of testing related to the Geospatial to the Edge Plugfest and this testing has revealed some areas of the standard that need to be refined. Some of these things are quite technical and I would like to get community feedback before we do anything rash. The following items have GitHub issues and corresponding pull requests. Please let us know if you think we are doing something wrong or if we can do better.

1. Support for views. We've said for the beginning that GeoPackage is supposed to support views where appropriate. We clarified this as part of version 1.2.0 and 1.2.1 but we need to do more because of our requirement for primary keys on user defined tables. The concept of primary keys does not exist for views in SQLite so we are now proposing that if there is no primary key column, then the first column must be primary-key-like, containing unique integers.
   Update: a competing approach would add a column to "gpkg_contents" to name the primary key column for a particular set of contents. I like this idea (more easily discoverable) but it may be too late to add even an optional column to that table.
2. Enforcing alignment between the CRSs in gpkg_contents, gpkg_geometry_columns, and gpkg_tile_matrix_set. I do not see how a GeoPackage can possibly be interoperable when the CRSs are not in alignment (how would a client decide which one to pick?) and I don't think anyone is doing this now, but since the question has come up more than once, we are taking the opportunity to tighten things up.
   Update: this issue was resolved.
3. Ensuring that feature geometries fall within the extents in the Well Known Binary (WKB) values. These extents are used as an optimization for spatial querying and indexing. If they are not valid (which is apparently happening on occasion), incorrect results will be returned.
   Update: This is still being discussed.

Since we are introducing new requirements as part of these changes, they are penciled in as part of a possible 1.3.0 release. We have no timeline for when this might occur but figure some time in 2019. As always, remain committed to maintaining reverse compatibility as much as possible. We will continue to avoid introducing requirements that have the potential to break existing GeoPackages unless those GeoPackages couldn't possibly work as designed.

Where We Are with Feature Styling and Portrayal

The GeoPackage SWG is very interested in solving the portrayal issue - how to encode style/symbology for GeoPackage feature data so that it can be portrayed or rendered in a client application. It is the #1 thing I hear about and I am committed to getting there. While proprietary solutions exist, we want something that will be interoperable. It is just taking a while and we need to be patient because there are a lot of moving parts. 

SWG members have asked for three things in a candidate solution:

1. SQL table-based (since GeoPackage is an SQLite encoding)
2. Consistent across the OGC baseline (so that implementers can build a single basic approach for the baseline, as opposed to a GeoPackage-specific implementation)
3. Loose coupling of the features and their styles (unlike KML, for example)

OGC hasn't nailed down a solution for #1 and #2 yet, but it is in progress. The Styled Layer Descriptor SWG has awoken from its decade-long slumber and is in the process of producing a v2 which is basically a total rewrite. In this version, they will implement a core and extensions model and support multiple encodings, resolving two major gaps in the original version. I was one of the contributors to a document called the Portrayal Concept Development Study Engineering Report which discusses these concepts in great detail. I hope for this document to be published in the next month or so. This will also be a hot topic at the OGC Technical Committee meetings in Ft. Collins next week.

To address #3, I believe the way to go is to harmonize GeoPackage and OWS Context so that the two work seamlessly with each other. I have written a discussion paper on the topic. I will present this paper to the GeoPackage SWG next week and if all goes well, it will be published by OGC later this month. I believe the concept is ready to go except for the style encoding. Once that is resolved, we should be able to standardize this pretty quickly. By the way, as far as I know, this would be the first time OGC adopts a standard that is an extension for two different standards simultaneously.

Where We Are with GeoPackage Extensions

A few weeks ago, GeoPackage achieved a new milestone when OGC adopted "OGC GeoPackage Extension for Tiled Gridded Coverage Data" (http://docs.opengeospatial.org/is/17-066r1/17-066r1.html). This is the first time OGC has adopted a GeoPackage extension outside of the encoding standard itself. We did this because we wanted the flexibility to create and revise the standard at our own pace without being constrained with the release schedule of the core standard (or vice versa).

What does this mean with regards to OGC compliance? By definition, a GeoPackage may contain extensions along with tiles, features, and attributes. An OGC-compliant GeoPackage is one that contains OGC-adopted extensions. The encoding standard itself currently has seven extensions (three of the original ten were later deprecated) and this new one is the eighth. They all have equal footing and the upcoming minor revision of the GeoPackage Executable Test Suite will include tests for all of them.

If OGC compliance isn't important to you, that's fine too. The extension mechanism is your friend. Use the gpkg_extensions table to advertise what extensions you are using (OGC-adopted or otherwise) and let client applications inspect the table and adjust their capabilities accordingly. You are always free to include non-adopted extensions in your GeoPackages too (no one is going to stop you) but OGC is unable to verify foreign extensions so there is no guarantee that they are fit for any particular purpose or that they will not cause compatibility issues with other software.

My Halloween Homage to Shapefile and Johnny Rotten

It is part of punk rock lore that in 1975 a young John Lydon turned some heads with an "I Hate Pink Floyd" shirt. He took a normal band t-shirt, scratched out the members' eyes, and scrawled his message on top with a marker. Soon he was invited to join a punk band (which then named itself the Sex Pistols) and took the name Johnny Rotten. And the rest is history.

  

Meanwhile back in the geo world, there has been a lot of back and forth about Shapefiles. The industry definitely has a love-hate relationship with them. On one hand they are indispensable but on the other hand there are a lot of people who want to banish them. Then there is the occasionally funny, occasionally annoying @shapefile on Twitter. When I saw two opposing items on Redbubble, my idea for a Halloween costume was born. All I needed was some markers, safety pins, hair spray, and blue hair dye.

The dirty little secret is that Lydon was actually a Pink Floyd fan—his shirt was more of a publicity stunt than a statement of opinion. Likewise, I can't help but have respect for the humble data format. Shapefiles are an important part of geo history and are for all intents and purposes the first interoperability the industry has achieved. While they have their limitations and probably are not the ideal format for many modern applications, they have enabled a lot of successful activities over the years and will continue to do so for many more.