Wednesday, June 20, 2018

We Want Your (Technical) Feedback

Earlier this month, GeoPackage 1.2.1 (a corrigendum release) was approved by the OGC Technical Committee. In this release, we focused on clarifications and consistency. Look for publication and formal announcement soon. Until this occurs, you can review the release notes. 

I thought things would slow down after this but I was wrong. There has been a bunch of testing related to the Geospatial to the Edge Plugfest and this testing has revealed some areas of the standard that need to be refined. Some of these things are quite technical and I would like to get community feedback before we do anything rash. The following items have GitHub issues and corresponding pull requests. Please let us know if you think we are doing something wrong or if we can do better.

  1. Support for views. We've said for the beginning that GeoPackage is supposed to support views where appropriate. We clarified this as part of version 1.2.0 and 1.2.1 but we need to do more because of our requirement for primary keys on user defined tables. The concept of primary keys does not exist for views in SQLite so we are now proposing that if there is no primary key column, then the first column must be primary-key-like, containing unique integers.
  2. Enforcing alignment between the CRSs in gpkg_contents, gpkg_geometry_columns, and gpkg_tile_matrix_set. I do not see how a GeoPackage can possibly be interoperable when the CRSs are not in alignment (how would a client decide which one to pick?) and I don't think anyone is doing this now, but since the question has come up more than once, we are taking the opportunity to tighten things up.
  3. Ensuring that feature geometries fall within the extents in the Well Known Binary (WKB) values. These extents are used as an optimization for spatial querying and indexing. If they are not valid (which is apparently happening on occasion), incorrect results will be returned.
Since we are introducing new requirements as part of these changes, they are penciled in as part of a possible 1.3.0 release. We have no timeline for when this might occur but figure some time in 2019. As always, remain committed to maintaining reverse compatibility as much as possible. We will continue to avoid introducing requirements that have the potential to break existing GeoPackages unless those GeoPackages couldn't possibly work as designed.

Friday, June 1, 2018

Where We Are with Feature Styling and Portrayal

The GeoPackage SWG is very interested in solving the portrayal issue - how to encode style/symbology for GeoPackage feature data so that it can be portrayed or rendered in a client application. It is the #1 thing I hear about and I am committed to getting there. While proprietary solutions exist, we want something that will be interoperable. It is just taking a while and we need to be patient because there are a lot of moving parts. 

SWG members have asked for three things in a candidate solution:
  1. SQL table-based (since GeoPackage is an SQLite encoding)
  2. Consistent across the OGC baseline (so that implementers can build a single basic approach for the baseline, as opposed to a GeoPackage-specific implementation)
  3. Loose coupling of the features and their styles (unlike KML, for example)
OGC hasn't nailed down a solution for #1 and #2 yet, but it is in progress. The Styled Layer Descriptor SWG has awoken from its decade-long slumber and is in the process of producing a v2 which is basically a total rewrite. In this version, they will implement a core and extensions model and support multiple encodings, resolving two major gaps in the original version. I was one of the contributors to a document called the Portrayal Concept Development Study Engineering Report which discusses these concepts in great detail. I hope for this document to be published in the next month or so. This will also be a hot topic at the OGC Technical Committee meetings in Ft. Collins next week.

To address #3, I believe the way to go is to harmonize GeoPackage and OWS Context so that the two work seamlessly with each other. I have written a discussion paper on the topic. I will present this paper to the GeoPackage SWG next week and if all goes well, it will be published by OGC later this month. I believe the concept is ready to go except for the style encoding. Once that is resolved, we should be able to standardize this pretty quickly. By the way, as far as I know, this would be the first time OGC adopts a standard that is an extension for two different standards simultaneously.

Tuesday, March 27, 2018

Where We Are with GeoPackage Extensions

A few weeks ago, GeoPackage achieved a new milestone when OGC adopted "OGC GeoPackage Extension for Tiled Gridded Coverage Data" (http://docs.opengeospatial.org/is/17-066r1/17-066r1.html). This is the first time OGC has adopted a GeoPackage extension outside of the encoding standard itself. We did this because we wanted the flexibility to create and revise the standard at our own pace without being constrained with the release schedule of the core standard (or vice versa).

What does this mean with regards to OGC compliance? By definition, a GeoPackage may contain extensions along with tiles, features, and attributes. An OGC-compliant GeoPackage is one that contains OGC-adopted extensions. The encoding standard itself currently has seven extensions (three of the original ten were later deprecated) and this new one is the eighth. They all have equal footing and the upcoming minor revision of the GeoPackage Executable Test Suite will include tests for all of them.

If OGC compliance isn't important to you, that's fine too. The extension mechanism is your friend. Use the gpkg_extensions table to advertise what extensions you are using (OGC-adopted or otherwise) and let client applications inspect the table and adjust their capabilities accordingly. You are always free to include non-adopted extensions in your GeoPackages too (no one is going to stop you) but OGC is unable to verify foreign extensions so there is no guarantee that they are fit for any particular purpose or that they will not cause compatibility issues with other software.

Wednesday, November 22, 2017

My Halloween Homage to Shapefile and Johnny Rotten

It is part of punk rock lore that in 1975 a young John Lydon turned some heads with an "I Hate Pink Floyd" shirt. He took a normal band t-shirt, scratched out the members' eyes, and scrawled his message on top with a marker. Soon he was invited to join a punk band (which then named itself the Sex Pistols) and took the name Johnny Rotten. And the rest is history.


Meanwhile back in the geo world, there has been a lot of back and forth about Shapefiles. The industry definitely has a love-hate relationship with them. On one hand they are indispensable but on the other hand there are a lot of people who want to banish them. Then there is the occasionally funny, occasionally annoying @shapefile on Twitter. When I saw two opposing items on Redbubble, my idea for a Halloween costume was born. All I needed was some markers, safety pins, hair spray, and blue hair dye.

The dirty little secret is that Lydon was actually a Pink Floyd fan—his shirt was more of a publicity stunt than a statement of opinion. Likewise, I can't help but have respect for the humble data format. Shapefiles are an important part of geo history and are for all intents and purposes the first interoperability the industry has achieved. While they have their limitations and probably are not the ideal format for many modern applications, they have enabled a lot of successful activities over the years and will continue to do so for many more.

Three November Updates

I have three GeoPackage-related updates for you. First of all, the GeoPackage SWG has wrapped up its work on the Tiled, Gridded Coverage Extension (formerly the elevation extension[1]). Look for OGC to make a press release soon. There will be an open comment period followed by an adoption vote (probably electronic) by the OGC Technical Committee. Of course if you have any comments, you do not have to wait until the press release. You can go directly to the GitHub repository and open an issue from there.

Second, you may have noticed a slight change to the geopackage.org page. We now have separate links to the adopted version of the standard and the working version on the top right. We have made a few administrative changes[2] that we want to get into the public eye even though they are not yet part of an approved document. These are now incorporated to the working version which is tentatively titled 1.3_SNAPSHOT.

There is a third development that will affect the standard, though it should not directly affect implementers. OGC is developing a Tile Matrix Set Encoding Standard[3]. This document extracts the TileMatrixSet definition from Web Map Tile Service (WMTS) and makes it a independent document designed to be referenced by other standards such as GeoPackage. If OGC adopts this standard, we will update the GeoPackage standard to bring them into alignment. I don't have a timeline for any of this, but whenever this work is complete we will probably release it as GeoPackage 1.3.

Happy Thanksgiving!

[1] For a recap of this topic, see paragraphs 2 and 3 of http://geopackage.blogspot.com/2017/05/good-news-bad-news.html
[2] See https://github.com/opengeospatial/geopackage/pull/387, https://github.com/opengeospatial/geopackage/pull/392, and https://github.com/opengeospatial/geopackage/pull/394 
[3] The working version of this draft standard, available to OGC members, is here: https://portal.opengeospatial.org/files/?artifact_id=76617.

Wednesday, September 20, 2017

More on SLD and SE

Styled Layer Descriptor (SLD) and Symbology Encoding (SE) are two related OGC standards. They have been around for over a decade but I think we can safely say that they have not definitively met their primary purpose - to enable the portrayal of feature data in an interoperable way. The most common complaint with these standards boils down to one thing - a high barrier to entry. This manifests itself in a few ways:
  1. SLD and SE are XML encodings. While many server-side developers are comfortable dealing with XML, this only represents a small fraction of the market these days. Most developers want nothing to do with XML if they can help it. JSON is a more natural choice for web applications, mobile applications, and many server-side frameworks. CSS is comfortable for many front-end developers. SQL developers (think GeoPackage) want to be working with SQL predominantly. A way past this issue would be to produce a conceptual model that could then be implemented in formats more appropriate for the target environment. The conceptual model could then be implemented a fairly small number of times and then abstracted for the various encodings.
  2. SLD and SE are fairly complicated. The documents are long and intimidating and compliance is an all-or-nothing affair. It would be helpful to have something akin to "Simple Features" for SLD and SE - a core that only covers the most important and common options. Like most things in software development, the Pareto Principle applies.
  3. Tools are lacking. There is currently no public executable test suite for the two standards. While GeoServer's SLD and SE support are pretty robust, alternatives are pretty hit-or-miss. If we want people to use these standards, we need the tools that make them easy to use.
On the flip side, SLD and SE also suffer from a lack of extensibility. There are use cases that cannot be handled by SLD and SE out of the box[1] and without an extension mechanism, there is no clear way for a developer to introduce interoperable solutions. This means that complex portrayals (think MIL-STD-2525) are a non-starter.

When a solution is not ideal for either the low end (lots of potential users) or the high end (lots of potential funding), all that is left is the skinny middle and it isn't enough. This is solvable if we have the will. That phrase is key - if we have the will[2]. The solution is not going to fall from the sky. It is going to take effort and leadership from a number of organizations over the course of more than a year. OGC appears to be willing to take the lead here but the effort requires community support to be successful. There will be opportunities to participate and I hope that you will find a way to do so. 

[1] The "TENET report" enumerates several limitations of the OGC SE, including hierarchical symbols, multiple delineations, and pivot points.
 
[2] Total tangent here. I first drafted this post on 9/13/17 and happened to include this phrase which comes from a Grant Hart song of the same name. That very night he passed away (unexpectedly, at least to me). RIP Grant.

Tuesday, September 12, 2017

Improving geopackage.org

Part of my job as GeoPackage SWG Chair is to improve outreach. This includes disseminating information (like this blog post) but it also means making information easier to discover. I feel good about the state of the specification / encoding standard but our outreach leaves room for improvement. One potential area of improvement is our website, geopackage.org. It is okay but it could be much better.

Over the next few months, I intend to revamp this site to make it more useful for its target audiences. When a visitor reaches the site, it should be self-evident where to go for relevant information. We're not there today. In response, I plan to reorganize the content to align it by reader type: general, developer, administrator, data publisher, data consumer, and SWG participant.

For this to work, I need more input from the user community. For example, a geopackage.org reader may wish to learn not only what GPKG implementations are out there but what they are useful for. Therefor I will be reaching out to vendors to clarify the role of the software that is currently listed under http://www.geopackage.org/implementations.html. I am also going to solicit user guides and demonstrations that can be incorporated into the site. This content can describe FOSS or proprietary software; my goal is to inform and to present options to the community.

If you can help me out here, I want to know about it.