Publishing Open Government Data: Open Data Gives Real Power to the People

Identify

The ability to identify things by a URI/URL is the foundation of the Internet. Best practices for “open government data” depend on applying the architecture described in Architecture of the World Wide Web, Volume 1 [WEBARCH]. If you give permanent, patterned and/or discoverable URI/URLs to your data, processes and people will be able to find and use it more easily. URI/URLs can be used in databases and metadata as universal, unique identifiers (e.g. by appending a serial number or other internal naming system to a domain: http://www.example.gov/objects/optional-hierarchy/serial12345678.html).

Use internal IDs to identify specific data for machine re-use (e.g., http://www.census.gov/main/www/popclock.html). By using URIs, and incorporating internal links inside large files or documents, people can authoritatively cite the information on their own websites.

Document

Without documentation, data is not very useful. When possible, you should use industry standards such as those based on XML/RDF, as they tend to be self-documenting. Create a web page with a plain language description of the dataset to help search engines find the data, so people can use it. The description should explain how to find the pages and/or files, and describe the contents of the files. At a minimum, document the title, description, publication date and authoritative source for the data. Use clear language, and link to related information to help put the data into context. When possible, also fully document the description of each datum and the data standard and/or data validation (e.g., if datum is a temperature, also explain the related standard, such as Celsius, Fahrenheit or Kelvin). As standards emerge for describing the datasets, take advantage of them. Document search tools and ReSTful methods of obtaining the data when applicable.

Link

The next step beyond raw data is fully linked data. In other words, raw data can only be linked to, there are no links back out – but linked data contains links out to other data and documentation. There is a spectrum from raw to fully linked data, and standards such as RDF that can help you link your data.

Here are four expectations for the linked data web:

1. URIs are names for things;
2. HTTP URIs help people find those things;
3. When someone looks up a URI, they should find useful information; and
4. Links within your data to other URIs help people discover more, related things.

Using URIs as described provides for a rich and easily connected web of data that is easier for the public to search and use.  However, many databases do not build URIs, but instead use internally derived unique identifiers. To link this data into the larger web, we need to use systems that can anticipate or build the URIs prior to publishing the data.

Preserve

Preserving the data we publish is a top concern for governments. How can we ensure that data is findable and can be referenced for as long as people need it?  People may continue to cite older data at the original URI decades after it was initially published.

To maintain the history of a dataset, think about how to integrate new URIs for upgraded and new datasets, and structure your URIs accordingly.  Use versioning of datasets so people can cite and link to present and past versions. New and upgraded datasets can refer back to original datasets.  If needed, provide tools to translate older data.  Carefully document the changes between versions, and embed the version number/indicator within the data if possible. Consider the file format carefully, and opt for open standards that are not software specific.

See the References section for more information on standards and best practices for data preservation.

Expose Interfaces

If you really want to help people discover and explore the data you are publishing, there are some useful W3C standards which can help.  It is possible to make data human-readable by using XSLT for XML- and RDF-based formats. Make web pages machine-readable by embedding semantic information in the HTML, using RDFa and/or Microformats. (Note that search engines can take advantage of embedded semantic information, helping more people to find your data.) If it is too difficult to append XSL or CSS styles to old data, it is still possible to document how to transform the data with a specific or generic XSLT engines (for example, see http://www.xmldatasets.net).

External parties can create new and exciting interfaces that may not be obvious to the data publishers. For that reason, do not compromise the integrity of the data to create flashy interfaces. If you must create an interface, then publish the data separate from the interface, and ensure external parties have direct access to the raw data, so they can build their own interfaces if they wish.

Additionally, since all web documents using (X)HTML, XML and RDF can be used as an object database or ReSTful API, the public can create software, Web applications and mash-ups that use the datasets, and link back to the authoritative version on the government web site. Two W3C standard languages that make that possible are XQuery and SPARQL.  Once your data is published, governments themselves can also create a user-friendly, transparent interface that allows users to play with and refine the information in which they are interested.

Create Standard Names/URIs for All Government Objects

Having a unique identifier for every registered company or school (object) is as important as having information about those companies or schools. In addition to using open industry standards, an effort should be made to catalog all reference-able government entities, officers and objects.  The URIs can then be used within all of the data. This aids in discoverability, improves metadata, and ensures authenticity.