Every day, governments and government agencies publish more data on the Internet. Sharing this data enables greater transparency; delivers more efficient public services; and encourages greater public and commercial use and re-use of government information. Some governments have even created catalogs or portals (such as data.gov) to make it easy for the public to find and use this data.
Although the reasons may vary, the logistics and practicalities of opening government data are the same. To help governments open and share their data, the W3C eGov Interest Group has developed the following guidelines. These straightforward steps emphasize standards and methodologies to encourage publication of government data, allowing the public to use this data in new and innovative ways.
Open Data Gives Real Power to the People
The ability to identify things by a URI/URL is the foundation of the Internet. Best practices for “open government data” depend on applying the architecture described in Architecture of the World Wide Web, Volume 1 [WEBARCH]. If you give permanent, patterned and/or discoverable URI/URLs to your data, processes and people will be able to find and use it more easily. URI/URLs can be used in databases and metadata as universal, unique identifiers (e.g. by appending a serial number or other internal naming system to a domain: http://www.example.gov/objects/optional-hierarchy/serial12345678.html).
Use internal IDs to identify specific data for machine re-use (e.g., http://www.census.gov/main/www/popclock.html). By using URIs, and incorporating internal links inside large files or documents, people can authoritatively cite the information on their own websites.
Without documentation, data is not very useful. When possible, you should use industry standards such as those based on XML/RDF, as they tend to be self-documenting. Create a web page with a plain language description of the dataset to help search engines find the data, so people can use it. The description should explain how to find the pages and/or files, and describe the contents of the files. At a minimum, document the title, description, publication date and authoritative source for the data. Use clear language, and link to related information to help put the data into context. When possible, also fully document the description of each datum and the data standard and/or data validation (e.g., if datum is a temperature, also explain the related standard, such as Celsius, Fahrenheit or Kelvin). As standards emerge for describing the datasets, take advantage of them. Document search tools and ReSTful methods of obtaining the data when applicable.
Preserving the data we publish is a top concern for governments. How can we ensure that data is findable and can be referenced for as long as people need it? People may continue to cite older data at the original URI decades after it was initially published.
To maintain the history of a dataset, think about how to integrate new URIs for upgraded and new datasets, and structure your URIs accordingly. Use versioning of datasets so people can cite and link to present and past versions. New and upgraded datasets can refer back to original datasets. If needed, provide tools to translate older data. Carefully document the changes between versions, and embed the version number/indicator within the data if possible. Consider the file format carefully, and opt for open standards that are not software specific.
See the References section for more information on standards and best practices for data preservation.
If you really want to help people discover and explore the data you are publishing, there are some useful W3C standards which can help. It is possible to make data human-readable by using XSLT for XML- and RDF-based formats. Make web pages machine-readable by embedding semantic information in the HTML, using RDFa and/or Microformats. (Note that search engines can take advantage of embedded semantic information, helping more people to find your data.) If it is too difficult to append XSL or CSS styles to old data, it is still possible to document how to transform the data with a specific or generic XSLT engines (for example, see http://www.xmldatasets.net).
External parties can create new and exciting interfaces that may not be obvious to the data publishers. For that reason, do not compromise the integrity of the data to create flashy interfaces. If you must create an interface, then publish the data separate from the interface, and ensure external parties have direct access to the raw data, so they can build their own interfaces if they wish.
Additionally, since all web documents using (X)HTML, XML and RDF can be used as an object database or ReSTful API, the public can create software, Web applications and mash-ups that use the datasets, and link back to the authoritative version on the government web site. Two W3C standard languages that make that possible are XQuery and SPARQL. Once your data is published, governments themselves can also create a user-friendly, transparent interface that allows users to play with and refine the information in which they are interested.
Create Standard Names/URIs for All Government Objects
Having a unique identifier for every registered company or school (object) is as important as having information about those companies or schools. In addition to using open industry standards, an effort should be made to catalog all reference-able government entities, officers and objects. The URIs can then be used within all of the data. This aids in discoverability, improves metadata, and ensures authenticity.
Choosing What Data to Publish
All data that can be shared with the public should be opened for public dissemination. Data should be published in compliance with applicable laws and regulations, and only after addressing issues of security and privacy.
First, publish all data that is already available in other formats (such as print), or already being collected and shared with the public in other ways. Publish both highly structured data and text documents. Publish laws, regulations and codes, and public documents related to government agency mission, vision, values and goals.
There are many standards for such documents (such as AIIM’s emerging Strategy Markup Language (StratML), XBRL and others). Use of standards enables stakeholders to easily discover and provide feedback on items of interest to them. Creating and publishing new documents in various standard formats is easier than fixing after the fact.
As you create your documentation for the data, include and reference XML Schemas, DTDs and taxonomies. Explain how to discover all of the files/documents in a collection. Use syndication standards and tools to announce additions or changes. Provide complete documentation to enable automatically-generated data dictionaries and provide query services, which will make it easier for the public to find your documents and datasets.
Choosing the Right Format for the Data
There are many different data formats, but which will work best with your data? The primary format for human-readable data is (X)HTML.
Restrictions on the Use of the Data
Once you’ve made your data available, be sure to document clearly any legal or regulatory restrictions on the use of the data. In many cases, there are defined standards to insert copyright and/or licensing information right into the data (see metadata standards, such as Dublin Core). It is very important to be clear regarding the expected uses, credits, limitations and responsibilities that particular parties have in regard to working with and providing the data.