Preparing for GeoPackage 1.0.2

The GeoPackage Standards Working Group (SWG) has voted to recommend the working version of the GeoPackage specification to the OGC Technical Committee for adoption as version 1.0.2. Because of some substantive changes that affect conformance, we decided to make version 1.0.2 a technical amendment.

I have been asked by OGC to provide release notes for this version. This is in progress and a draft version is available here [1]. For those of you who are not inclined to wade through the boilerplate text and other stuff, here is a summary of the substantive changes. (The hyperlinks at the beginning of each line point to the relevant GitHub Issue).

• 102. Requirement 45 is new and therefore there is no risk to existing implementations. Conforming to the new requirement would improve interoperability between systems.
• 130. Two columns in the Schema Extension were renamed to conform to the rule that all column names are lower case. There are no known uses of this table in the field so the risk of this change is low.
• 132. In the past, information regarding a single extension was scattered in at least four parts of the document. It was likely that a reader would not be able to find all of the relevant information about a particular extension. In addition, if a new extension was subsequently added to the document, it would change the numbering of the other annexes. In response, we decided to move all information regarding extensions into a single annex (Annex F), with one extension per sub-annex.
  While technically an administrative edit because no requirements were actually changed, a number of internal references were updated. References to numbered elements may break. (References to linked elements are unchanged.) Because of the potential impact to users of the standard (in documents, code samples, etc.), this is listed as substantive.
• 137. OGC has adopted 12-063r5, a new standard for describing coordinate reference systems as well-known text. The SWG determined that it would be too disruptive to update the references in the document to the new standard. In order to encourage adoption of the new standard, a new extension was formed. Implementations are new free to adopt the new standard at their own pace by conforming to the extension. Since this is a new extension, there is no risk to existing implementations.
• 147. Reports from the community indicated that the schema and metadata sections (formerly Sections 2.3 and 2.4 respectively) were not being used. Mandating those tables was creating work for implemententers with no tangible benefit. In response, the SWG determined that the best course of action would be to demote them to extensions. This reduces the length of the core of the standard, improving readability. Because of the lack of use, the risk of this change is low. 

The next steps are to create a PDF version of the document and to initiate a vote in the OGC Technical Committee. If I understand correctly, technical amendments can be approved via an electronic vote after an appropriate review period.

[1] Note: this file was updated on January 13. Feedback is welcome!

The One That Got Away

As a SWG chairman, I take pride in building consensus. We have tackled some tricky challenges this past year and I am pleased to say that we have gotten there with only one roll call vote.  What I found most of the time is that when there was objection to unanimous consent, there was often further work to be done. In a few situations we were able to come up with creative solutions that ended up satisfying everyone.

What was that one roll call vote? I was actually on the losing end of it. I wanted the elevation extension to encode the data in a purely binary encoding. I felt that introducing another format to the mix was adding an unnecessary layer of redirection that some developer was going to have to deal with down the line. The fact that libraries existed for many formats was not important to me because a) these libraries tended to be overkill for what we needed and b) reading a simple binary format is not particularly hard.

I lost. In the end I had to resign myself to the fact that people wanted a self-describing format (even if that made no sense to me). However, the story didn't end there. When all was said and done, we ended up building two extensions. The first laid out the format and specified PNG files for storing integer data. The second allowed floating point data to be stored in TIFF files. It is a reasonable compromise and I think it will work.

What I Want for Christmas (or 2016)

In my last blog post I talked about what the GeoPackage SWG accomplished in 2015. Now I want to look ahead to 2016. 

This is my wish list:

• From OGC:
• A way to publish multiple versions of the GeoPackage specification on geopackage.org (right now it just shows the working version)
• Speedy adoption of GeoPackage 1.0.2
• A repeatable process for producing HTML (and possibly other document types) from AsciiDoc so that we can easily publish other draft specifications in an easily readable form
• Commitment to encourage other SWGs to use the GitHub-based specification process that we pioneered here
• For users to automatically get added as observers in the project portal when they accept an observer agreement (you have no idea how big a nuisance this is!)
• From developers and/or software vendors:
• More content for geopackage.org, particularly FAQs and guidance on how to use the trickier parts of the standard (e.g., some of the extensions)
• Accurate, timely information on what products support what versions and/or portions of the standard
• From data providers:
• A commitment to publish their geospatial data as GeoPackage and/or GeoJSON instead of or in addition to proprietary formats or arcane government formats
• Feedback on what works well (or doesn't work well)
• From Image Matters (my employer):
• Support to continue in my role as SWG chair regardless of fickle funding situations
• A proper retirement sendoff for Paul Daisey, without whom GeoPackage never would have gotten this far

See you next year!

2015 Recap

As I type this, OGC is preparing for its Technical Committee meeting in Sydney, Australia. Unfortunately I cannot be there in person. Nevertheless, I want to take this opportunity to recap all of the work the GeoPackage Standards Working Group (SWG) has done in 2015. We have:

• Released Version 1.0.1 of the standard (OGC Document 12-128r11). This was a corrigendum release that corrected a bunch of errors that were identified, particularly in the features section of the document.
• Revamped geopackage.org to provide more precise information, including better FAQs, sample data, implementations, and guidance
• Completed the migration of almost all SWG activities to GitHub (everything but internal SWG communication and meeting information).
• Completed the Tiles Plugfest (as documented in this Engineering Report) which gave us a good idea of what interoperability problems we had with the tiles portion of the standard.
• Made a number of specification changes towards getting Version 1.0.2 ready for release. The most important changes are:
• Clarifying the tiles portion of the standard
• Organizing all extensions under a single annex
• Demoting the Schema and Metadata sections to an extension
• Creating a new extension to support OGC's new standard for describing Coordinate Reference Systems
• Produced a draft specification for an Elevation Extension and initiated an Interoperability Experiment (GPKG-EE IE) to evaluate it
• Conducted a survey to gauge interest in potential extensions.
• Started the discussion regarding a potential extension for symbology and styling.

It has been a busy year and I doubt we will get this much done next year. We do have a few things on our radar for the first half of 2016:

• Releasing Version 1.0.2
• Completing the GPKG-EE IE and hopefully releasing the Elevation Extension as an adopted extension
• Releasing at least a Discussion Paper regarding a potential extension for symbology and styling

However, the main measure of success next year will be utilization. We believe the standard is in a good state and that it solves the problems it was intended to solve. We ask data providers to consider using GeoPackage as an alternative (if not a complete replacement) to other formats. We ask data consumers to consider using GeoPackages when they are available and to let us know when using them proves beneficial. The SWG is here to help but the rest is up to you.

Clean-up in Section 2

We made a mistake and it is time to clean it up. Two sub-sections in the GeoPackage Specification, Schema and Metadata, have failed to provide any measurable benefit to users. I believe they were added to the specification with the best of intentions. Alas, as far as I can tell GeoPackage providers have been leaving the tables blank and even when they have been populated, they were not being read by clients. In the end all we are doing is making more work for people by making the presence of these tables mandatory.

In response, the Standards Working Group (SWG) voted to relegate the two sections into extensions. We will not remove these sections entirely but they will no longer appear in Section 2. As described separately, we have already agreed to compile all OGC-approved extensions into a single Annex. These changes will shorten the core of the standard and make the whole document more organized. We hope that this will make it more palatable to implementers. I plan to do the reorganization over the course of the next two weeks.

Meanwhile, the SWG's work is not done on this topic. I would like to reach the point that implementers look for opportunities to support extensions like these that do exist. Emerging profiles such as the National System for GeoINT Profile will use them and that is a good start.
To get beyond that, we will need to make a stronger case for the community benefits of providing this information.

Powers-of-Two Scale Sets

When we built the GeoPackage standard, there was a big push to limit tile sets to powers-of-two zoom levels. This is how most tile-based web maps (Google Maps, OpenStreetMap, etc.) work and there is a certain wisdom to it from a simplicity standpoint. However, I pushed back on this limitation for three reasons:

1. Existing cartographic products are based on a fixed scale which is some round number. When a cartographer designs a map to look right at 1:100,000 or 1:25,000 or whatever, it does not make sense to rescale it to 1:136,495 or 1:34,124[1] or some other arbitrary rational number.
2. Image products require a lossy transformation process to convert the raw (or base) image to a scale that matches the tile set. This transformation reduces image quality. When someone goes through the effort of collecting imagery at 0.5m (or better), it is a shame for that content to be degraded unnecessarily before reaching the end user.
3. When people look at imagery, they are not interested in looking at the second-highest zoom level. It just doesn't happen - they either zoom out enough to get their bearings or they zoom in as far as they can. If your tile set is based on powers-of-two, your second-highest zoom level adds about 25% of waste.

It doesn't have to be this way. "Because it is easier for the developer" is not a good justification for reducing quality or wasting space. Besides, It is not that hard for an application to look up a tile set and provide an appropriate set of zoom levels. 

Alas, we compromised by relegating support for non-powers-of-two zoom levels to an extension. I will be interested to see what happens as organizations commit to distributing raster products (especially imagery products) via GeoPackage and the ramifications of the powers-of-two approach become more apparent.

---

[1] These values are for the equator; the values at ±30° latitude are about 3.5% lower.

Extensions Survey

Over the summer, we conducted an online survey to get a feel for what potential extensions to GeoPackage people were most interested in. The candidate extensions came from a wide range of discussions with people inside and outside of OGC. The survey was anonymous and open to the public. The timestamps suggest that we got participation from across the globe.

The survey had three parts[1]. 

1. Whether the participant was more on the builder side (developer/integrator) or user side (administrator/user) of things
2. The attitude (strongly support, mildly support, neutral, or oppose) for each extension
3. Top three favorites.

I produced the chart below in mid-August[2] and the results speak for themselves. 

We plan to address symbology and styling next[3]. Our next step is to gather input on the desired use cases for this extension. This will be captured in a discussion paper that will be presented to OGC at an upcoming Technical Committee meeting. If you have any thoughts on this topic, let us know! If you comment here, over email, or on Twitter, I will be sure to see it.

---

[1] If I had a chance to do it over again, I would allow participants to provide free-form comments.
[2] The poll is still out there and responses continue to trickle in but it doesn't look like they will change the overall results significantly.
[3] Meanwhile, we are going to evaluate whether our recently published guidance on feature data is sufficient to cover the multi-resolution feature data case.

One Geometry per Table

Note: This post was mostly written by another GeoPackage SWG member, Thomas Neirynck of Luciad. Any editing mistakes are mine.

The GeoPackage encoding standard mandates in Requirements 22 and 30 that vector feature user tables shall only have a single geometry. This is advantageous for two reasons:

• It keeps the semantics of the data model concise: a feature has one shape. This is in line with other vector data formats with which GeoPackage is often compared, such as Shapefile or GeoJSON. By sharing this restriction, converting data from one format to another is more straightforward. This is particularly beneficial in enterprise workflows. For example, mobile components use a GeoPackage to view the business data, while the web-clients use a GeoJSON version of the same.
• Allowing multiple geometries per feature table would compromise GeoPackage's position in the GIS application ecosystem. Most ready-made GIS data viewers handle data formats in a uniform manner. Opening a file creates a layer on map and the layer shows the features contained in that file. There are usually some UI-capabilities to view properties of a feature or change the styling of a feature. If feature tables were to contain multiple geometries, the visualization of a GeoPackage in such a COTS-viewer would be much more complicated. What geometry should be shown on the map? All of them? Only the first? Editing tools would create a similar set of problems.

This restriction should not be seen as a limitation. Since a GeoPackage is a relational database, users can create rich data models to describe the required scenarios. Instead of adding more geometry columns to a single table, the general approach is to extract these geometries into a separate table, creating a one-to-many relationship between the geometries and the feature.

Because of the feedback we received on this topic, we recognized that we needed to produce some guidance to aid developers in producing compliant GeoPackages that handle non-trivial scenarios. This guidance has been published on geopackage.org. As always, we encourage public input on the specification itself and anything else that we can do to make things easier for the community.

Elevation Extension

Currently the GeoPackage Standards Working Group (SWG) is developing an extension to support the storing of elevation data in support of visualization and analysis operations[1]. We see this as a good thing because without it, there is no interoperable way to exchange elevation data in SQLite[2]. In the interest of simplicity, we are basing the design on the existing tile model. We use PNG files to store the elevation values as unsigned integers and there is an additional extension that supports the use of TIFF files to store floating point values.

So how does this extension stack up in my list of things that make a good extension?

• Clear use case? Check.
• Sound technical approach? We believe so.
• A commitment to implementation by multiple implementers? In progress...

Our next step is to conduct an interoperability experiment[3] to investigate the degree to which implementers are able to produce GeoPackages that conform to the extension and their software is able to use compliant GeoPackages effectively. We are in the planning stages now and are looking for participants. We plan to do the bulk of the work in January and February and present the results at the OGC Technical Committee Meeting in March 2016. 

If this experiment is successful, it proves both the technical approach and the commitment to implementation. At that point, the SWG will be ready to adopt the new extension.

---

[1] The ongoing work is available at: https://github.com/opengeospatial/geopackage-elevation
[2] As with any other extension, if a client does not conform with it, it will not see the elevation data but it will be able to see the features and tiles as it always has.
[3] See the Activity Plan

Coordinate Reference Systems

It is no secret. Coordinate reference systems (CRSs) are a constant issue in the geospatial community. Unfortunately they are a fact of life for many of us[1]. The GeoPackage standard requires that the CRSs in use be defined[2]. The standard encoding for CRS definitions that was available at the time was included in OGC 01-009 but that one is pretty dated and it has been causing problems in the field. In response, OGC recently adopted 12-063r5 to bring things up to date. The problem is getting people to start using it.

It is a classic chicken and egg problem. Consumers do not want to build support for data that isn’t being produced yet. Providers don’t want to produce data that consumers can’t read. Naturally, we’ve been asked to break the cycle. The traditional approach is to issue a new standard version that cites the new document but as mentioned previously, this is something we’re trying to avoid.
 
Our solution is to create another extension and to let a specific domain try it out. In this case it is the US National System for GeoINT (NSG) which represents the US defense and intelligence community. NSG has been working on a profile for GeoPackage which constrains and extends the standard. Among other things, the profile will contain an extension that updates the reference for CRS definitions from 01-009 to 12-063r5. Systems designed to work in this domain will be mandated to support the extension but others can adopt it at their own pace.

---

[1] For example, NGA does not support the use of Web Mercator and has developed other CRSs for use in tiles.
[2] There are three defaults, one for EPSG::4326, one for undefined geographic systems, and one for undefined cartesian systems.

Core and Extensions Model

The GeoPackage specification is organized using a core and extensions model. The core represents the “least common denominator” that all compliant systems must conform to. It is fairly compact; there are two required metadata tables (contents and coordinate reference systems), three optional metadata tables (one for extensions and two for user-defined metadata), and two core content types (features and tiles, at least one of which must be present).

When existing GeoPackage capabilities are insufficient, new ones can be added through extensions. This allows systems to satisfy additional requirements in an interoperable way without adversely affecting the other systems which do not. If a system reading the GeoPackage recognizes an extension then it can use the associated data as intended. If not, the associated data is ignored. When in doubt, system developers should stick to the core because arbitrary systems are only expected to conform to it.

Extensions may be developed inside or outside the SWG and the standard describes how this is done. When considering whether the SWG should approve a new extension, I look for three things:

• Clear use case
• Sound technical approach that does not adversely affect the core
• A commitment to implementation by multiple implementers 

These things keep us focused on producing extensions that practical, relevant, and interoperable.

---

[1] Extensions that are approved by OGC get a name with the “gpkg_” prefix.

Specification Stability vs. Maintenance

Changing a specification is disruptive because software developers have to make changes and systems need to be retested to ensure that everything still works as intended. Some organizations do not have the budget or schedule flexibility to do this sort of thing properly and in a timely manner. With this in mind, substantive changes to the core are to be avoided whenever possible. If it is up to me, there will be no 2.0 release of GeoPackage and I hope to avoid a 1.1 release (and certainly a 1.3 release) as well.

Like it or not, standards do need to be maintained. The GeoPackage Standards Working Group (SWG) uses the corrigenda[1] process to make corrections to the standard. We continue to review and inspect it to ensure that it is as clear as possible. We do as much as possible in public, using GitHub Issue Tracker to manage issues and relying on public feedback. For each issue, we review the passages in question, come to consensus on their intent, and rewrite them as needed.

A few months ago, we released a corrigendum that focused on the features portion of the standard. We are in the process of producing a second corrigendum that will focus on tiles[2]. We believe this second corrigendum will further improve the readability of the standard and reduce the risk of misinterpretation. While the formal PDF document is not ready yet, the changes that we have already agreed upon are reflected in the on-line specification.

---

[1] In printed works, a corrigendum is an insert listing errors and corrections. In the web-based world, that is a bit of an anachronism but at a minimum we need a way to fix errors, make clarifications, and show what has been updated.
[2] The GeoPackage Plugfest was an important part of this process because it identified parts of the standard that were not interpreted consistently.

GeoPackage.org and GitHub

Start at geopackage.org. That is my message every time I present about GeoPackage. GeoPackage is the first OGC project with its own home page. (The old structure was really only good for finding downloadable versions of the specification documents.) The new site provides access to most project-related resources including:

• A web-based version of the standard
• Implementations
• Sample data
• Relevant hyperlinks for partipants
• Frequently Asked Questions

Meanwhile, we have migrated most of our work to GitHub:

• The AsciiDoc of the specification itself [1]
• Issue Tracker
• GitHub Pages (for maintaining geopackage.org itself)[2]

We have had good results using the common GitHub workflows. The issue tracker has allowed much more public input than was ever possible before. It is much easier to manage changes to  geopackage.org through pull requests. The processes for managing the specification itself are still being improved but at least there is a short turnaround between a SWG decision and the corresponding update to the online specification. 

There are just a few things that remain on the old OGC site:

• Email lists (one public, one for SWG members only)
• Portal (for hosting internal files like presentations and for files that are too big to host on GH Pages)
• Wiki (now used primarily for meeting agenda, minutes, and a detailed change log)

---

[1] A back end process produces HTML from the specification AsciiDoc so that changes to the AsciiDoc are immediately reflected in the on-line version of specification.
[2] Via the gh-pages branch (particularly index.html) in the https://github.com/opengeospatial/geopackage repository

GeoPackage Vision

I am regularly asked about the vision for GeoPackage. For example, is it a simple container or does it have any higher level semantics? While the GeoPackage Standards Working Group (SWG) has not produced a formal vision statement, I believe something like the following would be appropriate:

The GeoPackage SWG develops and maintains the GeoPackage Encoding Standard as the standard for interoperable exchange of geospatial data and information using SQLite.

GeoPackage is simply a schema for data stored in an SQLite database. Note that I made no explicit mention of mobile here. GeoPackage is not just a standard for mobile! While mobile is clearly a major thrust of GeoPackage, it is not the only one. The great thing about SQLite is that it can be used in nearly every modern computing environment from a mobile device to the desktop up to the cloud. 

Because GeoPackage is so flexible, we envision it being used in a wide variety of ways. We foresee GeoPackage being a partial replacement[1] for the Shapefile format[2]. Organizations within the US government are exploring ways to distribute geospatial data assets in the format. As organizations continue to adopt GeoPackage, they will continue to find new and innovative uses for it. Some of them will be mobile-based and some will not.

---

[1] I don’t want to overstate this point. I use the term “partial” because GeoJSON is a perfectly suitable replacement for Shapefile for many scenarios. Both formats overcome many of Shapefile’s limitations. GeoPackage is more complex than GeoJSON but it also provides a number of additional capabilities that some scenarios demand.
[2] See the Shapefile Challenge at http://geometalab.tumblr.com/post/119804340842/the-shapefile-challenge

Introduction

GeoPackage is an Open Geospatial Consortium (OGC) standard for storing geospatial data and information in an SQLite database. The GeoPackage specification describes a set of conventions for storing the following content types:

• vector features
• tile matrix sets (pyramids) of imagery and raster maps at various scales
• schema
• metadata
• extensions

I was named the chairman of the GeoPackage Standards Working Group (SWG) about six months ago. The more I get acclimated to this role, the more I realize that this initiative will only be successful if it is embraced by the geospatial community at large (not just OGC, its members, or their customers). For that to happen, I need to be more active in reaching out. This blog is part of that effort.

In this blog, I will share my thoughts about where I think GeoPackage needs to go and what the SWG is doing to make that happen. While this content is my own, I do intend for it to reflect the consensus positions the SWG has reached. My style is to keep things brief, direct, and to the point so posts will be roughly the length of this one. I plan to post something about once a week for the next couple of months and then taper off to about once a month.