Wednesday, November 22, 2017

My Halloween Homage to Shapefile and Johnny Rotten

It is part of punk rock lore that in 1975 a young John Lydon turned some heads with an "I Hate Pink Floyd" shirt. He took a normal band t-shirt, scratched out the members' eyes, and scrawled his message on top with a marker. Soon he was invited to join a punk band (which then named itself the Sex Pistols) and took the name Johnny Rotten. And the rest is history.


Meanwhile back in the geo world, there has been a lot of back and forth about Shapefiles. The industry definitely has a love-hate relationship with them. On one hand they are indispensable but on the other hand there are a lot of people who want to banish them. Then there is the occasionally funny, occasionally annoying @shapefile on Twitter. When I saw two opposing items on Redbubble, my idea for a Halloween costume was born. All I needed was some markers, safety pins, hair spray, and blue hair dye.

The dirty little secret is that Lydon was actually a Pink Floyd fan—his shirt was more of a publicity stunt than a statement of opinion. Likewise, I can't help but have respect for the humble data format. Shapefiles are an important part of geo history and are for all intents and purposes the first interoperability the industry has achieved. While they have their limitations and probably are not the ideal format for many modern applications, they have enabled a lot of successful activities over the years and will continue to do so for many more.

Three November Updates

I have three GeoPackage-related updates for you. First of all, the GeoPackage SWG has wrapped up its work on the Tiled, Gridded Coverage Extension (formerly the elevation extension[1]). Look for OGC to make a press release soon. There will be an open comment period followed by an adoption vote (probably electronic) by the OGC Technical Committee. Of course if you have any comments, you do not have to wait until the press release. You can go directly to the GitHub repository and open an issue from there.

Second, you may have noticed a slight change to the geopackage.org page. We now have separate links to the adopted version of the standard and the working version on the top right. We have made a few administrative changes[2] that we want to get into the public eye even though they are not yet part of an approved document. These are now incorporated to the working version which is tentatively titled 1.3_SNAPSHOT.

There is a third development that will affect the standard, though it should not directly affect implementers. OGC is developing a Tile Matrix Set Encoding Standard[3]. This document extracts the TileMatrixSet definition from Web Map Tile Service (WMTS) and makes it a independent document designed to be referenced by other standards such as GeoPackage. If OGC adopts this standard, we will update the GeoPackage standard to bring them into alignment. I don't have a timeline for any of this, but whenever this work is complete we will probably release it as GeoPackage 1.3.

Happy Thanksgiving!

[1] For a recap of this topic, see paragraphs 2 and 3 of http://geopackage.blogspot.com/2017/05/good-news-bad-news.html
[2] See https://github.com/opengeospatial/geopackage/pull/387, https://github.com/opengeospatial/geopackage/pull/392, and https://github.com/opengeospatial/geopackage/pull/394 
[3] The working version of this draft standard, available to OGC members, is here: https://portal.opengeospatial.org/files/?artifact_id=76617.

Wednesday, September 20, 2017

More on SLD and SE

Styled Layer Descriptor (SLD) and Symbology Encoding (SE) are two related OGC standards. They have been around for over a decade but I think we can safely say that they have not definitively met their primary purpose - to enable the portrayal of feature data in an interoperable way. The most common complaint with these standards boils down to one thing - a high barrier to entry. This manifests itself in a few ways:
  1. SLD and SE are XML encodings. While many server-side developers are comfortable dealing with XML, this only represents a small fraction of the market these days. Most developers want nothing to do with XML if they can help it. JSON is a more natural choice for web applications, mobile applications, and many server-side frameworks. CSS is comfortable for many front-end developers. SQL developers (think GeoPackage) want to be working with SQL predominantly. A way past this issue would be to produce a conceptual model that could then be implemented in formats more appropriate for the target environment. The conceptual model could then be implemented a fairly small number of times and then abstracted for the various encodings.
  2. SLD and SE are fairly complicated. The documents are long and intimidating and compliance is an all-or-nothing affair. It would be helpful to have something akin to "Simple Features" for SLD and SE - a core that only covers the most important and common options. Like most things in software development, the Pareto Principle applies.
  3. Tools are lacking. There is currently no public executable test suite for the two standards. While GeoServer's SLD and SE support are pretty robust, alternatives are pretty hit-or-miss. If we want people to use these standards, we need the tools that make them easy to use.
On the flip side, SLD and SE also suffer from a lack of extensibility. There are use cases that cannot be handled by SLD and SE out of the box[1] and without an extension mechanism, there is no clear way for a developer to introduce interoperable solutions. This means that complex portrayals (think MIL-STD-2525) are a non-starter.

When a solution is not ideal for either the low end (lots of potential users) or the high end (lots of potential funding), all that is left is the skinny middle and it isn't enough. This is solvable if we have the will. That phrase is key - if we have the will[2]. The solution is not going to fall from the sky. It is going to take effort and leadership from a number of organizations over the course of more than a year. OGC appears to be willing to take the lead here but the effort requires community support to be successful. There will be opportunities to participate and I hope that you will find a way to do so. 

[1] The "TENET report" enumerates several limitations of the OGC SE, including hierarchical symbols, multiple delineations, and pivot points.
 
[2] Total tangent here. I first drafted this post on 9/13/17 and happened to include this phrase which comes from a Grant Hart song of the same name. That very night he passed away (unexpectedly, at least to me). RIP Grant.

Tuesday, September 12, 2017

Improving geopackage.org

Part of my job as GeoPackage SWG Chair is to improve outreach. This includes disseminating information (like this blog post) but it also means making information easier to discover. I feel good about the state of the specification / encoding standard but our outreach leaves room for improvement. One potential area of improvement is our website, geopackage.org. It is okay but it could be much better.

Over the next few months, I intend to revamp this site to make it more useful for its target audiences. When a visitor reaches the site, it should be self-evident where to go for relevant information. We're not there today. In response, I plan to reorganize the content to align it by reader type: general, developer, administrator, data publisher, data consumer, and SWG participant.

For this to work, I need more input from the user community. For example, a geopackage.org reader may wish to learn not only what GPKG implementations are out there but what they are useful for. Therefor I will be reaching out to vendors to clarify the role of the software that is currently listed under http://www.geopackage.org/implementations.html. I am also going to solicit user guides and demonstrations that can be incorporated into the site. This content can describe FOSS or proprietary software; my goal is to inform and to present options to the community.

If you can help me out here, I want to know about it.

Friday, September 1, 2017

A Busy Time at GeoPackage

Things have been busy on the GeoPackage front. I sense an uptick in both interest and adoption of GeoPackage. FOSS4G was the most public example but there have been others. With this in mind, I would like to share three recent developments. It is a busy time at GeoPackage!

1. GeoPackage 1.2 has been formally adopted by OGC. The OGC standards page has the "official" PDF version of the document (edit: and now the release notes) and the HTML version is what is currently at geopackage.org/spec. Thanks go to the large number of people who made this release possible.

2. OGC has released "OGC GeoPackage Extension for Tiled Gridded Coverage Data" for public comment. This extension is a reworking of the elevation extension. The primary change was expanding the scope of the extension from just elevation data to any regular gridded coverage data. The normative changes from the original extension are minimal - just a few additional columns that allow us to define what content is held in the tiles. We expect that implementers will find it easy to transition from the elevation extension to this one.

 
3. OGC has issued a call for participants for the "GeoPackage Related Tables Extension Interoperability Experiment". Yes, that is a mouthful. The idea is that our friends at Compusult have produced an extension for associating tables with existing feature or attribute tables in a GeoPackage. Among other things, it can be used to establish a many-to-many relationship between features and multimedia files. We are looking for willing participants (OGC membership optional) to help us determine if the extension accomplishes what it is designed to do. If you are interested, please accept the Observer Agreement and plan to dial into the kickoff tentatively scheduled for September 25.

Saturday, July 8, 2017

Let's Make a GeoPackage Extensions Registry

One of the best ideas I have heard recently is that there needs to be a registry for proprietary GeoPackage extensions. While OGC has a number of registered extensions as part of the encoding standard, is it only natural that there are going to be proprietary ones out there. What we don't have today is a way for developers to see what exists as prior art. We then run the risk of developers reinventing the wheel and the longer this persists, the harder it is to deal with. The last thing we want is multiple competing extensions out there and no interoperability between them.

I believe an extensions registry will address this problem effectively. I will do the legwork for creating it. In the short term it is going to be pretty simple - just a web page off of geopackage.org with a bunch of hyperlinks. I will use the Leaflet Plugins page as inspiration; hopefully one day we will end up more like QGIS Plugins but it won't happen overnight.
Update: this is now live at http://www.geopackage.org/extensions.html.

To be added to the registry, I will need from the developer a completed Extension Template [1]. This template includes all of the information developers will need to understand what the extension does and how it works. I will post the template to geopackage.org (or more likely, link to it) and help publicize it. If the extension suits other people's needs then great! If a number of implementers start using a particular extension, let the GeoPackage SWG know and we will discuss whether it is ready to be adopted by OGC.

[1] The raw AsciiDoc of the template is on GitHub. If filling out a template is going to be a major problem, let me know and I'll see what can be done to help you out.

Saturday, July 1, 2017

The Styling / Portrayal Ad Hoc Meeting



On Thursday June 29, I facilitated a Styling / Portrayal Ad Hoc at the OGC Technical Committee Meeting. This ad hoc meeting was called because there is broad agreement that GeoPackage would benefit from standardized feature styling and portrayal capabilities. (Anecdotal evidence suggests that the lack of common/standardized solutions here is already inhibiting GeoPackage adoption.) However, there is also agreement in OGC circles that whatever capabilities used by GeoPackage should apply across all of OGC Simple Features. The meeting was attended by over 40 people (combined in-person and on-line[1]).

Much of the meeting centered around Styled Layer Descriptor (SLD) and Symbology Encoding (SE), two OGC encoding standards[2]. The consensus was that while these are annoying (XML!) formats to work with and that they have a number of issues, they are not that far off for meeting most needs. (The geospatial community has created a number of specifications over the years but none of the alternatives have been standardized and many of them have been abandoned.) The attendees tentatively agreed that attempting to move SLD/SE forward was the best available option for supporting the desired capabilities in a timely manner.

The attendees agreed on the following action items:
  Rechartering the SLD/SE Standards Working Group (SWG) with the goal of producing updated standards that feature a core and extensions model and/or multiple conformance levels so that there is some separation between essential and non-essential elements
  Finding someone to create an executable test suite, developer’s guide, and other materials to lower the bar for developer use
  Initiating a pilot to experiment with the SLD/SE changes
My own next step is to take these findings back to OGC’s Architecture Board and try to turn them into action.

[1] Don't get me started on the pathetic state of hybrid in-person / on-line meetings in 2017.
[2] There was also discussion of portrayal registries, but that is a topic for another day.


Wednesday, May 3, 2017

Good News / Bad News

First the good news. OGC's Architecture Board (OAB) has approved our request to send GeoPackage 1.2 to the Technical Committee (TC) for an electronic adoption vote. This vote will start in a couple of weeks and there is a high likelihood that this version will be fully adopted by OGC by the end of June. 

The bad news is that the Elevation Extension has been delayed. The OAB requested that the extension be removed from version 1.2 so that it can be worked on separately. The feeling was that elevation data was too important to be rushed and that more time was needed to align it to other parts of the OGC baseline including things called "Geographic information — Schema for coverage geometry and functions"[1] and "Coverage Implementation Schema"[2].

I know this decision will cause uncertainty in the GeoPackage community. The extension does what it does and there is nothing keeping people from using it as is right now. Your mileage may vary. We should probably give it an alias (something without a gpkg_ prefix) to distinguish it from other adopted extensions. Other than that, we will have to wait for the process to play out. I made sure that someone was committed to carrying the extension to adoption.

[1] This document is dual published as OGC Abstract Specification Topic 6 (Schema for coverage geometry and functions) and ISO 19123 (Geographic information — Schema for coverage geometry and functions).
[2] This document used to be called GMLCOV but it was renamed when version 1.0.1 was adopted.

Wednesday, March 29, 2017

GeoPackage vs. Extended GeoPackage

One constant area of confusion is the notion of "GeoPackage" vs. "Extended GeoPackage". The idea is that a file using just core elements would be a GeoPackage and that one including extensions would be an Extended GeoPackage. However, this has caused problems in practice and we have had to adapt. A couple of months ago I wrote about non-spatial tables. Recently we got rid of the clause that allowed Extended GeoPackages to use the .gpkx extension (no one was). It turns out that this wasn't enough. 

We have been getting some push-back on the WKT for Coordinate Reference Systems extension because it adds a column to the gpkg_spatial_ref_sys table. While we feel that we were justified in this decision, it does affect some design decisions. For one thing, Object Relational Mappings are more complicated when a column might or might not exist. However, we have gotten feedback from other developers that adding columns to tables as part of extensions is a reasonable and necessary technique. What to do?

I propose to tweak a few requirements to get past this. To summarize, the GeoPackage designation acts as a sort of compatibility mode. At the expense of extensions, you get maximum interoperability. If you produce a GeoPackage with extensions (by definition an Extended GeoPackage) then your risk of interoperability issues increases (though hopefully is still small). We hope that applications and libraries will grow to deal with extensions gracefully but in the meantime, avoiding unnecessary extensions does provide the greatest interoperability. 

Does this work for you? Let me know here, on the mailing list, or on Twitter.

Tuesday, February 28, 2017

Filling a Gap: Conformance Tests

As mentioned previously, the GeoPackage 1.2 open comment period is under way. While we were preparing for this, we were briefed on testing performed by a consultant to a high-profile US Government organization. This testing called GeoPackage interoperability into question. Our analysis indicated that these interoperability concerns were due to either a lack of understanding of GeoPackage scope or the non-compliance of the data used in the tests. This is partially our fault - we had not completed the executable conformance tests needed to evaluate GeoPackage compliance. We have since found that some of the sample data posted to geopackage.org (and used in these tests) is not even fully compliant.
 
We are trying to fix this. As of last month, the OGC-sponsored conformance tests for GeoPackage (available here) only contained tests for the core and tiles portion of version 1.0 of the standard. This month we have built tests for the features portion and the elevation extension and brought the whole thing up-to-date to version 1.2. In the process of building these tests, we have identified a number of (hopefully) minor issues that we will resolve during the required comment resolution period.

Following are our next steps:
  • Deploy the executable tests to the OGC testing site so that they are accessible
  • Update the structure of geopackage.org so that it displays multiple versions of the standard (1.0.2, 1.1.0, and the proposed 1.2.0)
  • Resolve the open issues generated during the comment period
  • Refresh all of the sample data posted on geopackage.org, ensuring that all data passes the conformance tests 
I ask for your patience as we work through these tasks. This is taking a long time, but we are committed to getting it right.

Tuesday, February 7, 2017

Preparing for GeoPackage 1.2

The GeoPackage SWG continues to make changes to the GeoPackage Encoding Standard. Our goal is to make the standard clear, concise, and self-consistent. Following on 1.0.1 (which focused on features) and 1.1.0 (which focused on tiles), we have made a number of changes focusing on extensions. As always, we insist on maintaining reverse compatibility. In fact there are no substantive changes to existing parts of the core this time around.

We plan to release this version as GeoPackage 1.2. For detailed information on this release, please review the release notes. The changes range from typographical fixes to substantive changes that alter requirements. Following is a summary of the substantive changes:
  • Adding an "Attributes" section to describe the use of non-spatial data
  • Deprecating Requirement 69 (a mandate for extension F.1 that was nearly impossible to achieve or verify)
  • Deprecating Annexes F.2, F.4, and F.5, three extensions that were determined to be non-interoperable and non-usable
  • Updating the column name for WKT for Coordinate Reference Systems (Annex F.10)
  • Adding the Elevation Extension as Annex F.11
  • Changing the way GeoPackage versions are declared based on community feedback 
We initiated a 30-day comment period on Friday. After the comment period concludes, we will address all of the feedback. After that we will request a vote by the OGC Technical Committee to adopt 1.2 as an official encoding standard. We hope that you will take this opportunity to check out the draft and let us know if there is anything that can be improved.

Wednesday, January 4, 2017

Fun with Interfaces and Pointers in Go

Now that I am doing server-side programming again, I have fallen in love with the Go language. Long story short, it provides just about everything I need to produce quality server-side code and deliberately withholds language features that have unintended consequences in code quality and maintenance. I believe this has enabled me to produce higher quality code faster than with other languages like Java.

However, it has not always been smooth sailing. I recently hit an unexpected bump in what I thought was fairly simple code. As it turned out, I had not fully grasped how Go handles interfaces and pointers. Allow me to explain the scenario, and in so doing, illustrate how Go interfaces and pointers work. (I only present the minimal code needed to get the point across. Anything extraneous to the point is omitted.)
  1. I wanted to have a custom error struct that had some state information in it. This was done by creating a struct called Error that implements the error interface's one required function, Error().
    type Error struct {
    Message string
    }

    func (e Error) Error() string {
    return e.Message
    }
  2. I wanted Error to lazily initialize its state. For this to work properly (i.e., retain that state after multiple calls to Error()), I needed to use a pointer receiver in my Error() function. Note the difference between the code below and the code above.
    type Error struct {
    Message string
    }

    func (e *Error) Error() string {
    if e.Message == "" {
    e.Message = time.Now().String()
    }
    return e.Message
    }
  3. For functions like foo() that are only ever going to return my own struct, I figured that I could return *Error so that I would have full access to its members without any type checking. (As you will see later, this turned out to be a bad idea!)
    func foo() *Error {
    return nil
    }
  4. Then I could subsequently return the results (an error or nil) to a higher function.
    func bar() error {
    return foo()
    }
  5. Finally I would be able to call my bar function from high level code and check the return value.
    e := bar()
    if e == nil {
    fmt.Println("Hello, playground")
    } else {
    fmt.Println("Message:" + e.Error())
    }

There is just one problem – this code segfaults. Try it out here.

Why this happens is probably not obvious to most inexperienced Go programmers. What is it that implements the error interface? Not Error, but rather *Error. The problem is that bar() must return a) something that implements the error interface or b) nil. Well, *Error implements that interface. As written here (without all of the other logic that makes the function meaningful) foo() returns a pointer to nil. A pointer to nil is not the same as nil. Repeat that to yourself, more than once if you have to.

So what have I done? I created a struct, used a pointer receiver to implement an interface, and called a function that returns a pointer to my struct, and returned that pointer (that just so happens to point to nil). Well crap, since nil is not the same as a pointer to nil the equality expression fails and we try to dereference a nil pointer. Kaboom! The fix is simple – change foo() to return error instead of *Error. Done! (Oh well, my assertion in #3 is wrong and so I will just have to manually check to see if my error happens to be a *Error.)

The moral of the story is that if your struct returns an error, return an error and not something else that implements the error interface. However, sometimes the journey is more important than the destination. You can't fully understand the language if you don't truly understand how the language handles interfaces and pointers.

Tuesday, January 3, 2017

Non-spatial Tables

About a year ago, my Twitter timeline blew up. It turns out the Esri User Conference was going on and this got a lot of people chatting together. The root of the matter was a perception that Esri was not providing reasonable support for non-spatial data within GeoPackages. This was leading people towards flawed, clumsy workarounds like inventing spatial columns to tack onto attribute tables(!). Uh, folks, this is not what we had in mind...

I am not here to point fingers and the fact is that both sides had a point. A strict read of GeoPackage v1.1 did not allow non-spatial attribute values. However, in practice data providers routinely need to deliver data that does not contain geometry properties. We agreed that it was not reasonable to require any GeoPackage that contained non-spatial tables to be declared and documented as an "Extended GeoPackage". This does not promote interoperability. 

In response, we modified the standard by adding a new Attributes section that describes how to store non-spatial attribute tables in a GeoPackage. The new section is present in the on-line (working) copy of the specification and it will be incorporated in the next release (tentatively numbered GeoPackage 1.2). We hope that this addition will clarify things and encourage people to use GeoPackages as intended. We consider the change to be low-risk because it creates a new encoding option that would be ignored by previous versions of the standard.