Tuesday, February 28, 2017

Filling a Gap: Conformance Tests

As mentioned previously, the GeoPackage 1.2 open comment period is under way. While we were preparing for this, we were briefed on testing performed by a consultant to a high-profile US Government organization. This testing called GeoPackage interoperability into question. Our analysis indicated that these interoperability concerns were due to either a lack of understanding of GeoPackage scope or the non-compliance of the data used in the tests. This is partially our fault - we had not completed the executable conformance tests needed to evaluate GeoPackage compliance. We have since found that some of the sample data posted to geopackage.org (and used in these tests) is not even fully compliant.
We are trying to fix this. As of last month, the OGC-sponsored conformance tests for GeoPackage (available here) only contained tests for the core and tiles portion of version 1.0 of the standard. This month we have built tests for the features portion and the elevation extension and brought the whole thing up-to-date to version 1.2. In the process of building these tests, we have identified a number of (hopefully) minor issues that we will resolve during the required comment resolution period.

Following are our next steps:
  • Deploy the executable tests to the OGC testing site so that they are accessible
  • Update the structure of geopackage.org so that it displays multiple versions of the standard (1.0.2, 1.1.0, and the proposed 1.2.0)
  • Resolve the open issues generated during the comment period
  • Refresh all of the sample data posted on geopackage.org, ensuring that all data passes the conformance tests 
I ask for your patience as we work through these tasks. This is taking a long time, but we are committed to getting it right.

Tuesday, February 7, 2017

Preparing for GeoPackage 1.2

The GeoPackage SWG continues to make changes to the GeoPackage Encoding Standard. Our goal is to make the standard clear, concise, and self-consistent. Following on 1.0.1 (which focused on features) and 1.1.0 (which focused on tiles), we have made a number of changes focusing on extensions. As always, we insist on maintaining reverse compatibility. In fact there are no substantive changes to existing parts of the core this time around.

We plan to release this version as GeoPackage 1.2. For detailed information on this release, please review the release notes. The changes range from typographical fixes to substantive changes that alter requirements. Following is a summary of the substantive changes:
  • Adding an "Attributes" section to describe the use of non-spatial data
  • Deprecating Requirement 69 (a mandate for extension F.1 that was nearly impossible to achieve or verify)
  • Deprecating Annexes F.2, F.4, and F.5, three extensions that were determined to be non-interoperable and non-usable
  • Updating the column name for WKT for Coordinate Reference Systems (Annex F.10)
  • Adding the Elevation Extension as Annex F.11
  • Changing the way GeoPackage versions are declared based on community feedback 
We initiated a 30-day comment period on Friday. After the comment period concludes, we will address all of the feedback. After that we will request a vote by the OGC Technical Committee to adopt 1.2 as an official encoding standard. We hope that you will take this opportunity to check out the draft and let us know if there is anything that can be improved.

Wednesday, January 4, 2017

Fun with Interfaces and Pointers in Go

Now that I am doing server-side programming again, I have fallen in love with the Go language. Long story short, it provides just about everything I need to produce quality server-side code and deliberately withholds language features that have unintended consequences in code quality and maintenance. I believe this has enabled me to produce higher quality code faster than with other languages like Java.

However, it has not always been smooth sailing. I recently hit an unexpected bump in what I thought was fairly simple code. As it turned out, I had not fully grasped how Go handles interfaces and pointers. Allow me to explain the scenario, and in so doing, illustrate how Go interfaces and pointers work. (I only present the minimal code needed to get the point across. Anything extraneous to the point is omitted.)
  1. I wanted to have a custom error struct that had some state information in it. This was done by creating a struct called Error that implements the error interface's one required function, Error().
    type Error struct {
    Message string

    func (e Error) Error() string {
    return e.Message
  2. I wanted Error to lazily initialize its state. For this to work properly (i.e., retain that state after multiple calls to Error()), I needed to use a pointer receiver in my Error() function. Note the difference between the code below and the code above.
    type Error struct {
    Message string

    func (e *Error) Error() string {
    if e.Message == "" {
    e.Message = time.Now().String()
    return e.Message
  3. For functions like foo() that are only ever going to return my own struct, I figured that I could return *Error so that I would have full access to its members without any type checking. (As you will see later, this turned out to be a bad idea!)
    func foo() *Error {
    return nil
  4. Then I could subsequently return the results (an error or nil) to a higher function.
    func bar() error {
    return foo()
  5. Finally I would be able to call my bar function from high level code and check the return value.
    e := bar()
    if e == nil {
    fmt.Println("Hello, playground")
    } else {
    fmt.Println("Message:" + e.Error())

There is just one problem – this code segfaults. Try it out here.

Why this happens is probably not obvious to most inexperienced Go programmers. What is it that implements the error interface? Not Error, but rather *Error. The problem is that bar() must return a) something that implements the error interface or b) nil. Well, *Error implements that interface. As written here (without all of the other logic that makes the function meaningful) foo() returns a pointer to nil. A pointer to nil is not the same as nil. Repeat that to yourself, more than once if you have to.

So what have I done? I created a struct, used a pointer receiver to implement an interface, and called a function that returns a pointer to my struct, and returned that pointer (that just so happens to point to nil). Well crap, since nil is not the same as a pointer to nil the equality expression fails and we try to dereference a nil pointer. Kaboom! The fix is simple – change foo() to return error instead of *Error. Done! (Oh well, my assertion in #3 is wrong and so I will just have to manually check to see if my error happens to be a *Error.)

The moral of the story is that if your struct returns an error, return an error and not something else that implements the error interface. However, sometimes the journey is more important than the destination. You can't fully understand the language if you don't truly understand how the language handles interfaces and pointers.

Tuesday, January 3, 2017

Non-spatial Tables

About a year ago, my Twitter timeline blew up. It turns out the Esri User Conference was going on and this got a lot of people chatting together. The root of the matter was a perception that Esri was not providing reasonable support for non-spatial data within GeoPackages. This was leading people towards flawed, clumsy workarounds like inventing spatial columns to tack onto attribute tables(!). Uh, folks, this is not what we had in mind...

I am not here to point fingers and the fact is that both sides had a point. A strict read of GeoPackage v1.1 did not allow non-spatial attribute values. However, in practice data providers routinely need to deliver data that does not contain geometry properties. We agreed that it was not reasonable to require any GeoPackage that contained non-spatial tables to be declared and documented as an "Extended GeoPackage". This does not promote interoperability. 

In response, we modified the standard by adding a new Attributes section that describes how to store non-spatial attribute tables in a GeoPackage. The new section is present in the on-line (working) copy of the specification and it will be incorporated in the next release (tentatively numbered GeoPackage 1.2). We hope that this addition will clarify things and encourage people to use GeoPackages as intended. We consider the change to be low-risk because it creates a new encoding option that would be ignored by previous versions of the standard.

Wednesday, December 14, 2016

Elevation Extension Update

As previously mentioned, the GeoPackage SWG has been investigating an extension to support tiled, gridded elevation data. A year later, we are pleased to announced that we have completed our work on this extension and that we intend to release it as part of GeoPackage 1.2. This extension will share the mechanism used in the tiles portion of the standard.

The extension description describes the metadata tables needed to manage elevation tiles and encodings for the elevation values themselves. The extension supports two encodings. Most users will use the PNG encoding which uses 16-bit integers. An optional scale and offset allow for more efficient use of the 16-bit space. For those users who require greater resolution, there is a TIFF encoding which uses 32-bit floating point values. For convenience, we also provide abstract tests and table definition SQL.

To ensure that this approach will work, we conducted an interoperability experiment. In this experiment, documented in an Engineering Report[1], a number of participants produced GeoPackages containing elevation data and ran visualization and/or analytics on them. The experiment exposed a number of issues that the SWG subsequently resolved. At the OGC Technical Committee meeting in February, the SWG voted to add the extension to the standard. Do you think it is ready?

[1] At this time this link may only be available to OGC members. When it published on the main site I will update this post.

Tuesday, November 1, 2016

Internal Versioning for GeoPackage Files

We have a tricky situation and I need some feedback from the community to make sure we do it right. What we need is a way to identify the exact version of the GeoPackage in use in a particular file. We have tried to maintain compatibility between versions as much as possible, but there are subtle differences between one version to the next. Most applications handle these differences through duck-typing but conformance tests (like the emerging GeoPackage 1.0 ETS) do not have that option.

A recently opened ticket presents a possible way forward here. The recommendation is to for GeoPackage 1.2 to go back to using the SQLite application_id "GPKG" and to use the SQLite user_version to indicate the GeoPackage version. This would be an improvement over where we were heading - registering a series of application_id values (GP10, GP11...), suggesting completely different and incompatible versions.

There are two possible problems with this approach. The first is that GeoPackage providers are already using the user_version for their own uses. The other possible issue is that early versions of SQLite software don't support that field. We don't know of anyone that would be troubled by either. If you do, please let us know!

Tuesday, June 14, 2016

Maintaining Reverse Compatibility

Right now, OGC's Architecture Board (OAB) is trying to create guidance for describing and maintaining reverse compatibility. The following are my recommendations based on my experience with GeoPackage.

0. (Prime Directive) Do no harm. Practicality should be the most important consideration. Understand what existing implementations are doing so that you do not cause them new problems with the changes. However, do not shackle yourself unnecessarily. With new standards it is not unusual for obscure parts of the standard to be unused operationally. If they are wrong, fix them while you still have a chance.

1. The semantics of existing elements shall not be changed if they are in use operationally. In https://github.com/opengeospatial/geopackage/issues/137 we did not permit ourselves to change the way the WKT was encoded. Instead we created a new extension which adds a column.

2. New requirements may be added for any reason as long as doing so does not adversely impact interoperability. In https://github.com/opengeospatial/geopackage/issues/102 we knew we had an interoperability problem. The new requirement corrects that problem but there is nothing we can do about implementers that misinterpreted the (admittedly ambiguous) original text.

3. Existing requirements shall not be changed, relaxed, or removed if existing implementations rely on them being met. In https://github.com/opengeospatial/geopackage/issues/130 we had a consistency problem that we were able to correct because while existing implemetations were creating the rogue columns, they were not reading or writing to them. In https://github.com/opengeospatial/geopackage/issues/147 we moved a requirement from core to an extension because it wasn't being used operationally and it was just an annoyance for implementers.