Tuesday, November 1, 2016

Internal Versioning for GeoPackage Files

We have a tricky situation and I need some feedback from the community to make sure we do it right. What we need is a way to identify the exact version of the GeoPackage in use in a particular file. We have tried to maintain compatibility between versions as much as possible, but there are subtle differences between one version to the next. Most applications handle these differences through duck-typing but conformance tests (like the emerging GeoPackage 1.0 ETS) do not have that option.

A recently opened ticket presents a possible way forward here. The recommendation is to for GeoPackage 1.2 to go back to using the SQLite application_id "GPKG" and to use the SQLite user_version to indicate the GeoPackage version. This would be an improvement over where we were heading - registering a series of application_id values (GP10, GP11...), suggesting completely different and incompatible versions.

There are two possible problems with this approach. The first is that GeoPackage providers are already using the user_version for their own uses. The other possible issue is that early versions of SQLite software don't support that field. We don't know of anyone that would be troubled by either. If you do, please let us know!

Tuesday, June 14, 2016

Maintaining Reverse Compatibility

Right now, OGC's Architecture Board (OAB) is trying to create guidance for describing and maintaining reverse compatibility. The following are my recommendations based on my experience with GeoPackage.

0. (Prime Directive) Do no harm. Practicality should be the most important consideration. Understand what existing implementations are doing so that you do not cause them new problems with the changes. However, do not shackle yourself unnecessarily. With new standards it is not unusual for obscure parts of the standard to be unused operationally. If they are wrong, fix them while you still have a chance.

1. The semantics of existing elements shall not be changed if they are in use operationally. In https://github.com/opengeospatial/geopackage/issues/137 we did not permit ourselves to change the way the WKT was encoded. Instead we created a new extension which adds a column.

2. New requirements may be added for any reason as long as doing so does not adversely impact interoperability. In https://github.com/opengeospatial/geopackage/issues/102 we knew we had an interoperability problem. The new requirement corrects that problem but there is nothing we can do about implementers that misinterpreted the (admittedly ambiguous) original text.

3. Existing requirements shall not be changed, relaxed, or removed if existing implementations rely on them being met. In https://github.com/opengeospatial/geopackage/issues/130 we had a consistency problem that we were able to correct because while existing implemetations were creating the rogue columns, they were not reading or writing to them. In https://github.com/opengeospatial/geopackage/issues/147 we moved a requirement from core to an extension because it wasn't being used operationally and it was just an annoyance for implementers.

Wednesday, March 23, 2016

Congratulating Paul Daisey

Today we officially celebrate Paul Daisey's retirement. I met Paul in 2009 and we worked together for several years, primarily supporting Army Geospatial Center. I want to share three things that I take away from my time working with him over the last 6-7 years.

1. I have great respect for Paul's efforts as the GeoPackage SWG chair. This role was challenging because of the large number of disparate groups who had strong opinions on how things should be done, particularly on the vector side. It is rare to find someone with the technical acumen, community respect, attention to detail, and diplomacy to get the job done. It is through his efforts that GeoPackage exists today.

2. Sometimes it is the little things. Paul was great at transcribing meetings. You always knew that if Paul was there, he'd make a clear record of who said what. Now whenever I go to a meeting and have to take notes, I always chuckle to myself that I wish Paul were there so that I didn't have to do it and because he could do it better than I could.

3. Paul showed me how I want to go out. Over time, he gradually reduced his hours so that he could spend more time with his family, his home, and his hobbies. If something important and interesting came up, he was available to step up. If there wasn't much going on, it was no big deal. That's how I want to retire. 

Congratulations, Paul.

Tuesday, February 2, 2016

A Clarification - 1.1.0

"I have never been wrong BUT I have been known to clarify my position from time to time." - me
A few months ago I mentioned that I want to avoid substantive changes and a version 1.1 of the GeoPackage standard. It didn't quite work out that way so I would like to clarify what administrative, substantive, and critical changes mean in the context of standards development.
  • Administrative changes are changes to wording, grammar, and usage that do not affect requirements
  • Substantive changes alter requirements
  • Critical changes modify requirements in a way that make existing files invalid or would make valid files unusable on systems designed to work with the original version
As mentioned previously, we made a number of substantive changes in the new version. In general, the goal of these changes is to make the standard better - easier to read, less ambiguous, and more consistent. The OGC Architecture Board (OAB) determined that these substantive changes were sufficient to call this a minor release (1.1) as opposed to a point release (1.0.2). In response, I have updated all relevant text to identify the upcoming release as GeoPackage 1.1.

I am fine with this. What I really should have said in September is that critical changes are to be avoided. I still want to avoid a "GeoPackage 2" or even a "GeoPackage 1.3". The GeoPackage developer community is as adamant about this as I am, if not more so. This was never more apparent than in the discussion about the CRS WKT extension. It took more work to get this extension to the point that it was satisfactory. Maybe now we are where we need to be. If you don't think it is there, let us know.  I want to get this right.

Tuesday, January 12, 2016

NSG Profile

NGA has a difficult job when it comes to GeoPackage. They need to figure out the right way to distribute large amounts of geospatial data to users all over the globe. I know I am biased but I think GeoPackage is the right format for much of that distribution. Arcane formats like VPF, RPF, and NITF are a pain to work with. GeoTIFF isn't so bad, but it is a bit of a dated format and its compression capabilities leave a lot to be desired. (Because of this we see a lot of proprietary compressed formats in the field and that is even worse.)

The process is complicated by NGA's domain-specific rules. The approaches that most people use for their GeoPackages won't work for them. Among other things, Web Mercator doesn't work for them. The key step for them is to produce a profile that describes the rules for a compliant National System for GEOINT (NSG) GeoPackage. This process is underway and the goal is to produce the NSG Profile. They hope to be finished in the next few months.

From the GeoPackage side of things, I am excited about the NSG Profile. If NGA is able to distribute data as GeoPackage, I think it will make a big difference in long-term adoption. In addition, the NSG Profile touches parts of the specification (particularly extensions) that have not been examined closely to this point. It would be good to get some guidance on best practices for using them. Bring it on!

Tuesday, January 5, 2016

Introducing Thomas Neirynck

We have had a small shake-up in the GeoPackage Standards Working Group (SWG). Roy Rathbun (NGA) has recently accepted another position within NGA and will not be as available to support OGC initiatives. Roy has therefore resigned as SWG vice-chair effective the selection of a replacement.

Thomas Neirynck (Luciad) was recently selected as the new SWG vice-chair. As an active developer of software that uses GeoPackage and similar technologies, he brings a perspective that neither Roy nor I could provide. He also represents the European geospatial community, a community that is vital to GeoPackage's long-term success. In the short term, I would like for him to oversee any changes to the emerging elevation extension that arise as part of the Interoperability Experiment. This will allow me to focus on getting GeoPackage 1.0.2 through the adoption process.

Thanks to Thomas for accepting the position and to Roy for his service to the SWG.

Saturday, December 26, 2015

Preparing for GeoPackage 1.0.2

The GeoPackage Standards Working Group (SWG) has voted to recommend the working version of the GeoPackage specification to the OGC Technical Committee for adoption as version 1.0.2. Because of some substantive changes that affect conformance, we decided to make version 1.0.2 a technical amendment.

I have been asked by OGC to provide release notes for this version. This is in progress and a draft version is available here [1]. For those of you who are not inclined to wade through the boilerplate text and other stuff, here is a summary of the substantive changes. (The hyperlinks at the beginning of each line point to the relevant GitHub Issue).

  • 102. Requirement 45 is new and therefore there is no risk to existing implementations. Conforming to the new requirement would improve interoperability between systems.
  • 130. Two columns in the Schema Extension were renamed to conform to the rule that all column names are lower case. There are no known uses of this table in the field so the risk of this change is low.
  • 132. In the past, information regarding a single extension was scattered in at least four parts of the document. It was likely that a reader would not be able to find all of the relevant information about a particular extension. In addition, if a new extension was subsequently added to the document, it would change the numbering of the other annexes. In response, we decided to move all information regarding extensions into a single annex (Annex F), with one extension per sub-annex.
    While technically an administrative edit because no requirements were actually changed, a number of internal references were updated. References to numbered elements may break. (References to linked elements are unchanged.) Because of the potential impact to users of the standard (in documents, code samples, etc.), this is listed as substantive.
  • 137. OGC has adopted 12-063r5, a new standard for describing coordinate reference systems as well-known text. The SWG determined that it would be too disruptive to update the references in the document to the new standard. In order to encourage adoption of the new standard, a new extension was formed. Implementations are new free to adopt the new standard at their own pace by conforming to the extension. Since this is a new extension, there is no risk to existing implementations.
  • 147. Reports from the community indicated that the schema and metadata sections (formerly Sections 2.3 and 2.4 respectively) were not being used. Mandating those tables was creating work for implemententers with no tangible benefit. In response, the SWG determined that the best course of action would be to demote them to extensions. This reduces the length of the core of the standard, improving readability. Because of the lack of use, the risk of this change is low. 
The next steps are to create a PDF version of the document and to initiate a vote in the OGC Technical Committee. If I understand correctly, technical amendments can be approved via an electronic vote after an appropriate review period.

[1] Note: this file was updated on January 13. Feedback is welcome!