A JSON compatible serialization of RDF

JSON is a lightweight text-based open standard designed for human-readable data interchange.

That's not what we need.

We need a lightweight text-based JSON-compatible open standard designed for machine optimized RDF data interchange.

We need it to be JSON compatible because JSON tooling is widely implemented, JSON is well supported, embraced by most web development communities, and of course it's fast and simple.

Stress: It won't be JSON, it'll be JSON-compatible, and to avoid any confusion or indicating that it may be human friendly, we probably would be wise not to put the term "json" in it's name, although we will need to put it in the media type identifier "+json".

We need to standardize because there are numerous competing and differing JSON serializations at the minute, so when somebody asks for a JSON serialization on the web, they get back different results in different places - that's unexpected behaviour, and it's not interoperability - so we need to standardize a JSON-compatible serialization of RDF, with an IANA registered mediatype.

The serialization needs to be machine optimized and usable in multiple different scenarios, else there's no point standardizing it, because other JSON compatible serializations will still be needed and therefore created. Thus we need to define the various use-cases and create a JSON-compatible serialization which caters for them.

An RDF API

There's been a lot of call for cool RDF tooling, specifically in javascript, mentions of a jquery extension and a jquery-like library - I agree that would be nice, but also say why just one? I like and want choice, freedom and to see innovation in the marketplace, I don't just use jquery, I use more libraries and modules than I could name - however..

A library is not an API, and that's not what we need to standardize.

We need to enable interoperability between libraries, and where multiple libraries may or do have overlapping and conflicting functionality, we need to align and standardize it, but only when it produces unexpected behaviour or limits interoperability.

Primarily, this means standardizing interfaces for the RDF Concepts (literals, references, triple and graph) - as soon as we do that, all libraries which implement these interfaces will immediately become interoperable. Web Developers will be able to throw data between libraries, cherry picking the features they want or the coding styles they prefer, library implementers will be able to focus on areas they specialize in, others will be able to stun us with innovative new approaches and tools we'd never have thought of ourselves, and we'll have an open and ever growing truly cool suite of interoperable classes, libraries and tools.

Next up we can standardize interfaces for things like RDF Parsers, because as soon as we do that, all libraries on each specific platform can share parsers, and these can be developed, maintained and improved independent of any specific library - it will also allow the same libraries to be used with specialist low memory parsers optimized for mobile devices, thus broadening their potential user base and giving the end users + developers the best experience possible.

Another set of interfaces which need standardized, are those which are to be implemented by multiple different vendors on the same platform, i.e. the web browsers, i.e. anything near the DOM or HTML, i.e. the RDFa API - which is already being worked on.

Finally, an interface which will make linked data and RDF easier to work with by developers (carrying on from my previous post) is a "Data Object" interface, that is a simple object which holds the description of a single subject, where the properties are keys and the values are native values. I was wrong in my previous post, serving up simple JSON objects will help, but isn't perfect, because we don't know and can't determine whether people will want to use URIs, CURIEs or terms to access properties, thus an interface will be needed to handle this.

This isn't an exhaustive list, and each paragraph above can be considered to handle a different layer in the stack, but standardizing anything substantially more than that will stifle innovation and be counter productive, people need and want different libraries with different coding styles, features and different ways to access and query data. Different tools for different jobs, and for different people doing the same job.

If you disagree, please do share and let me know!

Clarification

Earlier this week I was quoted as follows:

Nathan’s recent post on Opening Linked Data, which is worth reading in its entirety, captures the essence of the issue:

You can’t shoe horn RDF in to JSON, no matter how hard you try - well, you can, but you loose all the benefits of JSON in the first place, because the data is RDF, triples and not objects, rdf nodes and not simple values

In other words, using JSON as the basis for an RDF syntax doesn’t actually win you anything in terms of the ease of processing of that RDF. In fact, I’ll go further and say it has exactly the same bad qualities as RDF/XML.

However I have to point out that I probably didn't make the context clear, I was referring specifically to making a human friendly/optimized JSON serialization of RDF, you can however, very easily, create a machine optimized JSON-compatible serialization of RDF, without the drawbacks of XML (you just don't pin it as being human friendly JSON as outlined above) because unlike XML which requires a full XML stack, and RDF/XML which can be serialized in any one of a billion ways, we have a chance here to make a clean lightweight unambiguous serialization of RDF, one which is based on a lightweight data interchange format and not on a heavy extensible document interchange format.

Hope that clarifies :)

Why? well the sem web community is packed full of the most technically skilled and decent people I've come across so far, so it can't be that, the tooling is pretty damn good, there's loads of data, most of the data's of a high quality, wanted by developers, and certainly more than usable. The concepts, theory and technical aspects are all solid as a rock. In short it's all good, apart from one rather important detail.

Our Linked Open Data isn't really open data, not in the eyes of the common web developer at least. To most web developers on the planet, open data is something they can get access to and use easily.

A good example of 'open data' for most developers, is the Twitter API.

Here's how a developer accesses it in PHP:

  $uri = 'http://api.twitter.com/1/statuses/show/7907258268647424.json';
  $tweet = json_decode(file_get_contents($uri));
  echo $tweet->user->description;

Here's how they access it in Javascript:

  uri = 'http://api.twitter.com/1/statuses/show/7907258268647424.json';
  $.getJSON(uri, function(tweet) {
    write( tweet.user.description );
  });

The reality of the matter is that you can't do this with Linked Open Data, and that's because you can't do it with RDF - and really, honestly, if it's not that simple, the masses won't use it, because, if it's not that simple, they can't use it.

The problems with the RDF formats

They're not perfect, and they are a very mixed bunch, for a change, let's look at the negatives.

RDF/XML
- Requires full XML tooling
- Can't read or write by hand
- butt ugly

Let's be honest, unless you have a full XML and RDF stack and you know what you're doing, RDF/XML is simply a no go zone.

RDFa
- Requires HTML/DOM/XML tooling
- An extension to a markup language, designed to augment annotated documents.

RDFa is great, but not for the general data use case, it's not a simple data interchange format like JSON, and you can't publish or consume it without specialist tooling, in fact it requires an even bigger, more complicated, stack than RDF/XML.

Turtle
- Requires a custom parser
- It's not yet seen as a data format by the masses.
- Doesn't have a registered media type.

If we're all honest, Turtle is the king of RDF serializations, it's small, powerful, easy to write and read, requires minimal tooling to parse. In fact, I'd quite happily say that Turtle is the best all round data format, period.

RDF/JSON and JSON-LD
- Square peg, round hole

You can't shoe horn RDF in to JSON, no matter how hard you try - well, you can, but you loose all the benefits of JSON in the first place, because the data is RDF, triples and not objects, rdf nodes and not simple values - I fear I'd better explain, quickly:

The benefit of JSON is that you can do the following:

  var u = tweet.user;                                // nested, simple objects
  write(u.message);                                  // simply "a string of text"
  if(u.geo_enabled) {                                // a boolean true
    var d = u.statuses_count * u.favourites_count;   // numbers..
  }

Anything more complicated than that and you've lost 95% of the benefits of JSON.

Here's the code to do that if(u.geo_enabled) with JSON/RDF:

  var tweet = rdf['http://example.org/tweet/12343'];
  var user = rdf[tweet['http://example.org/property/userid']];
  var geoenabled = user['http://example.org/property/geo_enabled'];
  if( Boolean(geoenabled.value) ) {

and JSON-LD:

  var tweet, user;
  for(o in jsonld) {
    if(jsonld[o]["@"] == 'http://example.org/tweet/12343') tweet = jsonld[o];
  }
  for(o in jsonld) {
    if(jsonld[o]["@"] == tweet['twit:userid']) user = jsonld[o];
  }
  if(user['twit:geo_enabled']) {

Remember, those two examples are only for the simple if line from the benefits of JSON example, can you even imagine all four lines?

Clearly, RDF in JSON is of little to no use to anybody, you can see plainly yourself, 95% of the benefits are lost and it's just another RDF serialization that's pretty much unusable without tooling. The only benefit JSON serializations of RDF have, are that you don't require an XML stack, which is quite a large benefit tbh.

The problem with RDF

I've been a little unfair there, you see the problem isn't with the serializations, we can't make a "better" serialization of RDF for general web developers, because the real problem is that the data's RDF, it's triples not simple objects, URIs rather than simple terms, RDF Nodes with a language or type, and not just simple values.

An array of RDF triples, or a structure of RDF, just simply isn't usable in most (all?) programming languages, by a typical web developer, without specialist tooling and libraries or APIs.

Am I saying RDF is bad? no of course not, it's awesomely brilliant in every way, it powers a paradigm shift and will have huge positive effects on the web and the human race. You know the score :)

What I am saying, is that we're not backwards compatible, we're not making our data open in formats which are usable by normal developers, developers who need and want the data, want the links, but not the semwebbery. Hell even most of us who are heavy sem web users only consider the ontology+reasoning side enabled by properties-with-uris some of the time, most of the time <http://xmlns.com/foaf/0.1/name> might as well just be "name".

Opening Linked Data

So, here's what we need to do, we need to just accept that although we publish linked data as RDF, we also need to publish the data as simple objects so the world can use the data.

Given the linked data:

  @prefix : <http://webr3.org/nathan#> .
  @prefix foaf: <http://xmlns.com/foaf/0.1/> .
  @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .

  :me a foaf:Person;
    foaf:age 29;
    foaf:holdsAccount [ foaf:accountName "webr3";
        foaf:homepage <http://twitter.com/webr3>;
        rdfs:label "Nathan's twitter account"@en ];
    foaf:homepage <http://webr3.org>;
    foaf:knows <http://example.com/bob#me>;
    foaf:name "Nathan";
    foaf:nick "webr3", "nath" .

  <http://example.com/bob#me> a foaf:Person;
    foaf:name "Bob" .

We also need to publish it like this:

{
  "http://webr3.org/nathan#me": {
    "a": "http://xmlns.com/foaf/0.1/Person",
    "age": 29,
    "holdsAccount": {
      "accountName": "webr3",
      "homepage": "http://twitter.com/webr3",
      "label": "Nathan's twitter account"
    },
    "homepage": "http://webr3.org",
    "knows": "http://example.com/bob#me",
    "name": "Nathan",
    "nick": [ "webr3", "nath" ]
  },
  "http://example.com/bob#me": {
    "a": "http://xmlns.com/foaf/0.1/Person",
    "name": "Bob"
  }
}

and now one can do: me.holdsAccount.label and get back a string, in any language.

What have we lost?
Well nothing in the grand scheme of things because we're still publishing the RDF in other formats via conneg, however in this specific serialization of the data: we've lost the properties, the .language and the .type, although basic types are still there, dates are detectable, numbers are supported, booleans are supported, everything else is just a PlainLiteral, a string.

What's still there?
The data, http names for things, follow your nose, rdf types ++usable-accessible-data in a web standard format. It's 3.5 if not 4.5 star data!

Other considerations
Probably be wise to allow direct access to a .json URI too, so people can simple-GET the data (in addition to exposing via conneg).

Should be an easy hit, any good reasons why not?

The other RDF Serializations

While we're here, there are a few other things that need tidied up, properly.

Turtle
Let's make it the standard RDF serialization, with a proper registered media types of application/turtle and text/turtle, fix any bugs in the spec (if any), and possibly allow an optional comma in those lists (1,2,3).

Let's just accept that RDFa is great, but sometimes you just want to embed a chunk of RDF in an HTML document, you know we all want to on occasion, people are shouting for it, it's easy to do, to deploy, doesn't break anything, and it's a really easy hit - so, in the Turtle standard spec pop a note that shows how to include it in an HTML document.

<script type="text/turtle">
  // turtle in here, as-is, no special encoding
</script>

RDF/XML
Leave it as is, let it run it's course naturally, and in many respects just forget it moving forwards, everybody supports it already, that won't change, but there's approximately zero need to keep pushing it.

RDFa
As awesome as it is, just see it for what it is, RDF in attributes, it's like the missing markup features of HTML, for annotating documents and describing the things described in the documents with annotations. It's not a simple data or RDF format, and it's not really suited to just dropping chunks of machine readable RDF in to an HTML document, Turtle in HTML will do that far better.

RDF/JSON and JSON-LD
Standardize a JSON serialization as a replacement / alternative to XML - but, admit and stipulate before hand that it's not usable as-is, or really writable and is arguably human readable. It simply needs to be a fast, unambiguous, optimized for the machine, RDF serialization - no bells and whistles for humans, no "12^^xsd:type" in a string - just something that you can JSON.parse and run circa 10-20 standardized lines of code over to get back an RDFGraph of RDFTriples.

Fin & end.

As we all know, one of the biggest hindrances to innovation in the various areas of computer science is the human brain, and specifically the tendency to think inside, rather than outside the box. There are things we never even consider or imagine because we just can't think outside of what we already know/presume - a small percentage of gifted individuals can, but on the whole, we as humans/geeks/programmers cannot.

To keep this short and simple, let's return to the aforementioned point, with javascript, any libraries you write can be used on both client and server. Great eh, write an AMF parser and it works on both client and server -wonderful.

Now, let's flip the example to a 'shopping basket', you see what we did there? now you've got an ecommerce application on the client side (think about that for a minute, you could put items from any 'website' in your basket, not one, any/all - oh and keep http stateless, and, and..) - here's another one, an authorization protocol like openid or webid - so a server could 'login' to a browser.. - how about an HTTP server in js, yes now it's on the client side and the server side - and when each side is both client and server isn't that just p2p?.

This universality breaks down the barriers of convention, it gets rid of that wall we have between server and client - can you imagine how many 'server side' technologies, libs, patterns and architectures nobody has ever even considered working with on the client side, can you imagine the implications and the period of innovation this will bring on?

Conversely, can you imagine how many client side technologies and libs nobody has thought about putting on a server? fancy using HTML as a data storage serialization and editing it via jquery on the server perhaps?

Maybe you can't imagine, maybe I can't - but fact is it will be possible, and people will start doing it, often just by lack of these assumptions and notions the current generation(s) of programmers have, they won't assume that something we've always used on the server side will be on the client, they'll just do it because it makes sense, and we, well we'll all be astounded.

Make sense?

Generations One and Two

Historically we have been through two generations of the web so far, not web 1 and web 2.0, but rather the first round of mounting the presentation tier on the web (static publishing documents and media), then the second round of mounting the application tier on the web (from forms through to the current api's we see everywhere).

The next round, and what many are currently hacking at, is mounting the data tier on the web, mainly via the "Linked Data" movement which you may have noted.

This third round is critical for two primary reasons:

1. By mounting the data tier on the web, it allows us to remount the application tier again, but this time properly, where HTTP is the API between the Application tier and the Data tier. (true REST)
2. The "Internet of Things" is very much dependant on the "Things" speaking the same language and sharing a single Data Model, where the model remains the same but the vocabulary can change.

Generation Three

"Linked Data" is in many ways the solution to the above, because it has a Universal Data Model (EAV), a Universal API (HTTP), and Universal Addressing & Identification (the duality of http URIs) - more importantly though, "Linked Data" provides the means to mount and expose both ABox and TBox statements on the web - this is the key detail where RDF wins over other EAV based data models; the ability to store both the data and the vocabularies on the web, and both in the same manner.

Linked Data is currently tied (but not bound) to RDF. However, RDF still has very critical limitations, primarily it doesn't have any notion of Named Graphs or Quads. Trust and provenance are essential moving forwards (with quads you can select which data your application trusts, and which it doesn't, based on the source, the Named Graph - without it you can't), as is the ability to transfer data from multiple sources at the same time (update streams, batch operations, merged data with provenance - again needing quads).

The above means we need an RDF2 (or standardisation & adoption of N3+rules), and moreover we need standardized non-xml serializations of said RDF(+2), the most important being rdf/json.

This brings me to HTML5, the key here isn't the new HTML5 document format, the semantics, the ability to embed microdata or anything like that - it's the introduced (or implied) dependency on JS /ECMAScript via the JS APIs. This leads to JS being rolled out on to the majority of devices - it's also important to note the server side JS movement in this too.

Generation Four

We've had a "Web of Documents", we're building a "Web of Data", the next logical step is to have a "Web of Applications", for this we need two things: a Universal Programming Language (JS...) and an "Internet of Things" which support the universal data model and the universal programming language.

As far as I'm concerned HTML5 plays a critical part in the big picture:

- Short term, it at least unites the browser vendors to better support the techs which developers use.
- Presently (and from here on) it allows us to start hacking at the web of applications.
- In the future it's legacy will be it's introduction of JS as a Universal Programming Language.

Sides:
It stands to reasons that RDF/JSON (or whatever supersedes RDF serialized in JSON) will also play a major part in this round.

We may well see a shift from client-server, server-server to application-application; where each machine on both sides comprises of HTTP Client, Server and Cache.

Generation Five

Leading on from here we get to the fifth round of the web, remounting the Presentation Tier, but this time where the presentation tier speaks to the application tier through HTTP, the Universal Interface.

HTML (together with JS and CSS) again plays a critical role in round five, because up until this point it remains the only way for the hackers to create this fifth generation, notably it will be a lot more than we are have now, after years of maturity, hacking and support universally. Naturally then we can assert that HTML will remain the core of the Web's presentation tier so long as the Web exists.

Additional Shifts

Widespread adoption and understanding of REST, truly without this we'll never get past generation three.

Data transformation, for a very long time data transformation will be an everyday service we all require, for legacy systems, legacy data, proprietry systems, data model conversion and so on - the key isn't to be able to understand all kinds of data / serializations / models, but to transform it in to what you do understand, easily.

The enabling of the Semantic Web, Linked Data provides the means to do this, it is the enabler - each EAV triple we put out is also a triple that can be reasoned over, full machine understanding of all our data is an ultimate goal that will take many years. Generation three very much pushes the semantic web in to the realms of the real world, no longer for the strictly academic.

The great cleanse, all the data from generations one through three (and possibly beyond) will need cleansed - as primary universal vocabularies emerge the task will be to clean what's on the web and migrate it in to the new models, reasoning and cleaning it as we go.

Cross compilation and seperation of syntax from machine code - another movement that is gaining momentum, no longer are we tied to a specific syntax for a specific JVM - not far off is the day where we can program in our preffered language, and compile it to whatever target we need. This is all ready happening in many quarters (like haxe) and soon will be the norm.

Seperation of human readable data syntax and data storage/transfer serialization. This has long been understood, however with EXI comes a light serialization of XML for the machine & transfer; logically from there we can expect the same in the future for RDF(2) and most data moving forwards.

Much of the code needed for the Application Tier will disappear over time, typically most application code involves taking some data and turning it in to some new data, processing it - when you can do this on the fly via data transformations, rules, reasoning, inference, querying and similar, you remove the need for most f the code in between. It is entirely possible that most server side web applications could disappear.

End

It is testament to the strength and design of the web that it can support every incarnation of the web thus far and in the future all at the same time - All credit to Tim Berners-Lee, Roy T Fielding and the many, many others who contributed, evolved and continue to forge this thing we call the web.

Disclaimer
Obviously, this is all just my humble opinion, none of it is fact - but from stepping back a bit this is what I conclude, for now.

I've skipped loads of things, most of the techs I work on and think about daily, FOAF+SSL, web access control, read write web of structured linked data, loads more - the above is just a summary of where I think we're headed, primarily for my own reference :)

Universal doesn't necessarily mean universal - but it's a good enough word to convey what I'm thinking. Any other use of terminology you don't quite agree with, likewise, terminology is not the point of the post ;)

Best & happy to hear your thoughts and additions,

Nathan

I'm assuming that if you are reading this, you know what linked data is, and REST as per the dissertation of Roy T. Fielding. If not go do some reading :)

Interface Constraints

REST is defined by four interface constraints:

1. identification of resources
2. manipulation of resources through representations
3. self-descriptive messages
4. hypermedia as the engine of application state.

From here I'll look at each of these four constraints and build up the approach as I go.

What a resource is

Quoting extensively from REST 5.2.1.1 Resources and Resource Identifiers:

The key abstraction of information in REST is a resource. Any information that can be named can be a resource: a document or image, a temporal service (e.g. "today's weather in Los Angeles"), a collection of other resources, a non-virtual object (e.g. a person), and so on. In other words, any concept that might be the target of an author's hypertext reference must fit within the definition of a resource...

A resource is a conceptual mapping to a set of entities, not the entity that corresponds to the mapping at any particular point in time...

The values in the set may be resource representations and/or resource identifiers...

A resource can map to the empty set, which allows references to be made to a concept before any realization of that concept exists...

The only thing that is required to be static for a resource is the semantics of the mapping, since the semantics is what distinguishes one resource from another...

What a representation is

Again, quoting extensively from REST 5.2.1.2 Representations:

... A representation is a sequence of bytes, plus representation metadata to describe those bytes. Other commonly used but less precise names for a representation include: document, file, and HTTP message entity, instance, or variant...

If the value set of a resource at a given time consists of multiple representations, content negotiation may be used to select the best representation for inclusion in a given message...

The data format of a representation is known as a media type

Identification of resources

To do this properly I need to identify some resources, so for this I'm going to work with "Something" :)

• "Something" - a resource, a non-virtual object

At any point in time I have a description of Something which has multiple representations in different mediatypes, all semantically matching or equivalent:

• "something.rdf" - representation of Something with mediatype RDF+XML
• "something.n3" - representation of Something with mediatype RDF+N3
• "something.en.html" - representation of Something, in english, with mediatype text/html
• "something.de.html" representation of Something, in german, with mediatype text/html

Each one of those representations is also a resource because they can be the target of a hyperlink. Of course by resource I mean a conceptual mapping to each of the things listed, and I haven't assigned URIs but will..

To be able to make this set of representations manageable and to indicate they are in a set, I'm going to add in another resource which is a collection of resources, which can be considered a set of these equivalent representations of Something at a fixed point in time. For the purpose of this exercise, that point in time is today.

• "Something-20100311" - a resource which is a collection of equivalent representations of Something on the 11th March 2010.

Additionally, for the sake of argument, I'm going to say that a new set of representations (or version) is added every day - to handle this I then need one more resource, a collection of resources, where each resource in the collection is itself a collection of resources (one of the aforementioned and including the example "Something-20100311"). This will give me a conceptual mapping which covers time, and therefore everything I could need.

• "Somethings" - a resource which is a collection of resources, see above for full description!

Finally, I'm going to add in two shortcut resources which have no representation and are simply conceptual maps to the first and most current sets of representations.

• "first" - a resource which always maps to the first collection of representations of Something.
• "latest" - a resource which maps to the most recent collection of representations of Something.

Giving the resources URIs

Now to assign some URIs for this use case, there is no set structure and I'm not going to define one because it is up to each server (or manager of) to control it's own URI space, but for the sake of this exercise I'll define mine as follows:


base: http://data.webr3.org
...
/d/Something
/rg/Somethings
/rg/Somethings/first
/rg/Somethings/latest
/rg/Somethings/Something-20100311
/rg/Somethings/Something-20100311/something.rdf
/rg/Somethings/Something-20100311/something.n3
/rg/Somethings/Something-20100311/something.en.html
/rg/Somethings/Something-20100311/something.de.html
...
/rg/Somethings/Something-20100305
/rg/Somethings/Something-20100305/something.rdf
/rg/Somethings/Something-20100305/something.n3
/rg/Somethings/Something-20100305/something.en.html
/rg/Somethings/Something-20100305/something.de.html
...


From the above you can see that every possible representation has its own URI, in addition every collection of equivalent representations has its own URI, as does the collection of all those collections; and so does "Something" our non virtual object.

Also we've exposed multiple resources which could also be RESTful CRUD access points operating on an atompub style protocol. Small sentence, big potential, will cover approaches and protocols in later posts.

The Key resource

The most important thing, which I haven't yet covered, is that we've exposed a key resource, namely /rg/Somethings. This is a resource at the top of the representation chain which can be used to expose content negotiation, be it server or agent driven (or a mix of both), and regardless of the mappings and levels of collection further down the line this can always be a single point of entry to get representations.

I'll cover just how in a moment, but for now something important.

Important

I've had to give a fixed example just to make some progress, but we have to remember that every system has different needs, in some cases it may be that there is only a single fixed representation for a resource, whilst in others each strand of representation (like something.de.html) may take it's own versioning / temporal path. This could indicate that a structure such as the following may be in order:

...
/d/Something
/rg/Somethings
/rg/Somethings/first
/rg/Somethings/latest
/rg/Somethings/Something-20100311
/rg/Somethings/Something-20100305
/rg/Somethings/Something-rdf
/rg/Somethings/Something-rdf/20100311.rdf
/rg/Somethings/Something-rdf/20100305.rdf
/rg/Somethings/Something-html-en
/rg/Somethings/Something-html-en/20100311.html
/rg/Somethings/Something-html-en/20100305.html
/rg/Somethings/Something-html-de
/rg/Somethings/Something-html-de/20100308.html
/rg/Somethings/Something-html-de/20100303.html


The above highlights that whilst we may have added more resources, the core resources are still the same; remember that they are "conceptual maps", meaning that Something-20100311 may "map" to the version of en-html on the 11th and de-html on the 8th, because the de version was written first, then translated to english and from there rdf and so forth, but they are all semantically equivalent, containing the same information even though they were created at different times.

The Conceptual Maps are as follows, from what I can tell this should always cover any use-case, no matter how complex.


Thing 1-1 CollectionOfCollections
CollectionOfCollections 1-* CollectionOfEquivalentRepresentations
CollectionOfEquivalentRepresentations 1-* Representation


aside:At times like this I wish I'd had a chance to study computer science so that I could express these things formally, so you'll have to make sense of it as best you can :( sorry.

Exposing via Content Negotiation

In my research so far, I've been able to figure out how to expose all of the aforementioned via HTTP, RESTfully using content negotiation in a manner which seems to be transparent to existing web browsers, but exposes all the information needed in a manner that is visible to machines; without using any additional extensions headers. As follows:

1 The client does a normal GET request on our "Something", notice that no content negotiation is happening yet, we are simply asserting via a 303 "that the requested resource does not have a representation of its own that can be transferred by the server over HTTP."

#Request
GET /d/Something HTTP/1.1
Host: data.webr3.org
Accept: text/html;q=0.5, application/rdf+xml


#Response
HTTP/1.1 303 See Other
Location: http://data.webr3.org/rg/Somethings


2The client does a GET on the URI we specified in the Location field, namely to our key resource that can be used for content negotiation over all the representations.

#Request
GET /rg/Somethings HTTP/1.1
Host: data.webr3.org
Accept: text/html;q=0.5, application/rdf+xml


#Response
HTTP/1.1 300 Multiple Choices
Location: http://data.webr3.org/rg/Somethings/latest
Content-Type: application/xhtml+xml
Content-Length: 17400


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML+RDFa 1.0//EN"
"http://www.w3.org/MarkUp/DTD/xhtml-rdfa-1.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
xmlns:foaf="http://xmlns.com/foaf/0.1/"
xmlns:dc="http://purl.org/dc/elements/1.1/"
version="XHTML+RDFa 1.0" xml:lang="en">
...


Here's where it gets interesting and clients can take different routes; first the route of the typical user agent:

User Agent Route


#Request
GET /rg/Somethings/latest HTTP/1.1
Host: data.webr3.org
Accept: text/html;q=0.5, application/rdf+xml


#Response
HTTP/1.1 307 Temporary Redirect
Location: http://data.webr3.org/rg/Somethings/Something-20100311


#Request
GET /rg/Somethings/Something-20100311 HTTP/1.1
Host: data.webr3.org
Accept: text/html;q=0.5, application/rdf+xml


#Response
HTTP/1.1 302 Found
Vary: Accept
ETag: W/"xyzzy"
Last-Modified: Wed, 11 Mar 2010 12:45:26 GMT
Content-Type: application/xhtml+xml
Content-Length: 17400
Content-Language: en
Content-Location: http://data.webr3.org/rg/Somethings/Something-20100311/something.en.html


<!DOCTYPE html...


First you can see that the user agent simply goes straight through to the most recent content and what they expect to see; which is nice, with additional Server driven content negotiation.

Further, we can see that full cache control is in there which as we know speeds up the net, and further still we have a rather nifty "weak" entity tag; this entity tag is shared by all representations which are semantically equal, and asserts they are equal via the entity tag. It's also worth noting that you could add this entity tag to your RDF graphs and further assert provenance which could come in very handy down the line for POST and PUT implementations.

To recap, common user agents just go straight through to the expected resource via server driven content negotiation and can take full advantage of cache / control data.

The Machine Route

Back at 2 the server returned a 300 Multiple Choices as soon as /rg/Somethings was requested. All important was that the entity returned was XHTML+RDFa (although this could have been Atom or similar..), which means we can give both a human and machine readable list of all our various representations, the "machine" can then select which one it finds most fitting. The choices could be expressed using any suitable ontology; and further both Alternative and Link headers could be added if publishers wished.

I think that covers it all, if there are any errors or things I've missed please do let me know asap; but for now that'll do me - it's verbose, but I like verbose - prove it works then optimise it later :)

Very brief intro: SPARQL is a fantastic query language which works over RDF and thus Linked Data, Virtuoso amongst other things has a powerful QuadStore which can be queried via SPARQL, and Virtuoso's implementation of SPARQL + the extensive suite of extensions they have implemented makes it the most usable and powerful query langauge available (again, in my honest opinion). In short this combination was enough to make me drop normal RDBMS systems and never look back.

Rather than rambling on about how fantastic it is though; here are some Virtuoso specific sample SPARQL (+GEO) queries, which should hopefully wet your appetite and give you some inclination of what can be done.

Basic Geo Lookups

Things within 20km of New York City : RESULTS
SELECT DISTINCT ?resource ?label ?location
WHERE
{
<http://dbpedia.org/resource/New_York_City> geo:geometry ?sourceGEO .
?resource geo:geometry ?location ; rdfs:label ?label .
FILTER( bif:st_intersects( ?location, ?sourceGEO, 20 ) ) .
FILTER( lang(?label) = "en" )
}

Distance between New York City and London, England : RESULTS
SELECT (bif:st_distance(?nyl,?ll)) as ?distanceBetweenNewYorkCityAndLondon
WHERE
{
<http://dbpedia.org/resource/New_York_City> geo:geometry ?nyl .
<http://dbpedia.org/resource/London> geo:geometry ?ll .
}


Querying Time and Space

All Educational Institutions within 10km of Oxford, UK; ordered by date of establishment : RESULTS
SELECT DISTINCT ?thing as ?uri ?thingLabel as ?name ?date as ?established ?matchGEO as ?location
WHERE
{
<http://dbpedia.org/resource/Oxford> geo:geometry ?sourceGEO .
?resource geo:geometry ?matchGEO .
FILTER( bif:st_intersects( ?matchGEO, ?sourceGEO, 5 ) ) .
?thing ?somelink ?resource ; <http://dbpedia.org/ontology/established> ?date ; rdfs:label ?thingLabel . FILTER( lang(?thingLabel) = "en" )
} ORDER BY asc( ?date )

Historical cross section of events related to Edinburgh and the surrounding area (within 30km) during the 19th century : RESULTS
SELECT DISTINCT ?thing ?thingLabel ?dateMeaningLabel ?date ?matchGEO WHERE {
{
SELECT DISTINCT ?thing ?matchGEO
WHERE
{
<http://dbpedia.org/resource/Edinburgh> geo:geometry ?sourceGEO .
?resource geo:geometry ?matchGEO .
FILTER( bif:st_intersects( ?matchGEO, ?sourceGEO, 30 ) ) .
?thing ?somelink ?resource
}
}
{?property rdf:type owl:DatatypeProperty ; rdfs:range xsd:date } .
?thing ?dateMeaning ?date . FILTER( ?dateMeaning in( ?property ) ) . FILTER( ?date >= xsd:gYear("1800") && ?date <= xsd:gYear("1900") )
?dateMeaning rdfs:label ?dateMeaningLabel . FILTER( lang(?dateMeaningLabel) = "en" ) .
?thing rdfs:label ?thingLabel . FILTER( lang(?thingLabel) = "en" )
} ORDER BY asc( ?date )

Transitivity and Inference (v5 compatible)

Finding the shortest route between two "things" (HTML and XML in the example) : RESULTS
SELECT ?route ?jump WHERE
{
{ SELECT ?x ?y WHERE { ?x foaf:page ?xpage ; ?predicate ?y . filter( isURI(?y) ) } }
OPTION ( TRANSITIVE, T_DISTINCT, T_SHORTEST_ONLY, t_in(?x), t_out(?y), t_max(10), t_step('path_id') as ?path, t_step(?x) as ?route, t_step('step_no') AS ?jump )
. FILTER ( ?y = <http://dbpedia.org/resource/HTML> && ?x = <http://dbpedia.org/resource/XML> )
}


..and all routes between the two "things" : RESULTS
SELECT ?route ?path ?jump WHERE
{
{ SELECT ?x ?y WHERE { ?x foaf:page ?xpage ; ?predicate ?y . filter( isURI(?y) ) } }
OPTION ( TRANSITIVE, T_NO_CYCLES, t_in(?x), t_out(?y), t_max(5), t_step('path_id') as ?path, t_step(?x) as ?route, t_step('step_no') AS ?jump )
. FILTER ( ?y = <http://dbpedia.org/resource/HTML> && ?x = <http://dbpedia.org/resource/XML> )
}

Traversing Ontologies and (Sub)Classes; all subclasses of Person down the hierarchy : RESULTS
SELECT DISTINCT ?x WHERE
{
{ SELECT ?x ?y WHERE { ?x rdfs:subClassOf ?y } }
OPTION ( TRANSITIVE, T_DISTINCT, t_in(?x), t_out(?y), t_step('path_id') as ?path, t_step(?x) as ?route, t_step('step_no') AS ?jump, T_DIRECTION 2 )
FILTER ( ?y = <http://dbpedia.org/ontology/Person> )
}

Free text search, scores and IRI Ranks (v5 compatible)

Searching over labels, with text match scores and additional ranks for each iri / resource : RESULTS
SELECT ?s ?page ?label ?textScore ((?s)) as ?iriRank WHERE {
?s foaf:page ?page ; rdfs:label ?label . FILTER( lang(?label) = "en" ) .
?label bif:contains 'adobe and flash' option (score ?textScore ) .
}

Virtuoso 6.1 (Open Source Edition) released. For features & bug fix details see: link

Sounds melodramatic but the whole point of this text is to explain in real terms the effect it'll have on the every day web worker; the web developer, web designer, seo expert, internet marketer etc. So that you, my current or future friends and associates still have a job in a couple  of years; and I researched it so that I would still have a job in a few years (+ because I love this stuff!)

A bit about Linked Open Data (LOD).

LOD can easily be a huge, scary and new thing, overwhelming in so many ways with all this talk of a cloud, billions of bits of information in some part of the web that is separate to "us"; take one look at the diagrams of the linked open data cloud and you'll see those academic acronyms of scientific organisations, future thinking global entities publishing their specialised data - nothing about you and me with our little blogs, and moreover nothing about our clients websites.

Sure it's about getting massive amounts of data on the web, linked and open for use, but it's different to how you expect :)

Linked Open Data is simply about making the info we already put on the net (like this post) machine readable as well as human readable.

It *IS NOT* about creating some system to dump everything from our database in some weird format for a machine to read somewhere.

It *IS* about wrapping the data on a normal page in a bit of markup so that a computer knows what it is.

If you're writing about london you simply add a tiny bit of markup that says 'about="http://en.wikipedia.org/wiki/London"' - honestly that's it in real terms, the user reading your page knows its about London, England - and now a system like google knows that it's definitely about London, England too. In most cases though it's simpler; it's a case of saying this article is titled "x" and made by person "y" - that alone makes a huge difference to the net.

How LOD will change the web.

More Links! We're currently in the age of search, if you want something you search for it, to get more info you search again, and again and so on.

Link trust was at an all time low a few years ago, sure you'd click a navigation link on a site but not a link in a document, because it was probably to a popup, an advert, something you didn't want. Not now though, mainly thanks to bloggers with their in text links to other pages, the world has grown to trust the link again.

Linked Open Data will spawn a massive increase in related data on page, related resources, articles, images, videos and more. And thus many, many more links.

This means that people will search less, and explore more; ever increasingly.

It's unavoidable, and even if a website isn't enhanced with all this extra linked data, odds are the user will have a browser extension or app running that will show all the related information anyway - these technologies are already here and used - adoption *will* grow, no way out of it, change happens.

Info for specific sectors

This isn't the meant to be a full introduction or all encompassing, in fact nowhere near it - if you want the ins and outs of LOD then look elsewhere. This info is for the everyday Web Developer, Web Designer and SEO Specialist.

Web Designers (+ those who work with html)

To be honest I think this change might hit you guy's hardest; you see XHTML+RDFa is already here, it'll be massive soon (and don't go thinking HTML5 will get you out of it, RDFa will be in there too). In short XHTML+RDFa is xhtml as you know it, but with support for embedded RDF information, really it means a few new properties on elements that let you say what they are; in place FOAF, Dublin Core (DC) and the like. Any further description is outside the scope of this document ;)

What this means for you is that as well as having potentially a lot more to display on page (linked data) and lot's of UI challenges, you also now have to cater for this RDFa data in your templates. It's not like other W3C stuff which you can ignore, different to cross browser compatibility, if you leave it out or skip the RDFa stuff then the site will potentially be outside of the LOD network, traffic will drop and ultimately the site may as well not be "in" the web (might be a few years before that though) - so in many ways the end of ignorance and excuses.

You can currently slap out some HTML4, change the doctype, stick on jquery and make it "look" web 2.0 - and people will think it's web 2.0; with web 3.0 (the linked open data based net) you can't do that, it either is web 3.0 or isn't; there isn't a "web 3.0" look, just web 3.0 source.

Drupal 7 has RDFa support out of the box; within the year I bet every CMS & Blog will too; and if you make a new template with the RDFa cut out because you "don't know it", then I'm pretty sure it won't be long before your clients or employers cut you out; and we don't want that.

Further, if you don't - developers will be on your back big time & changing your source; or worse the SEO guys will be ;)

Web Developers
All of you need to know what triples are (subject-predicate-object), and URIs and CURIEs (not your normal URIs, URIs as Identifiers).

If you're going to be exposing data in your systems then you need to get used to mapping database properties through to RDF triples; that a user is a foaf:person with a foaf:name; that tags are ctags and dc:subjects, and that articles have a dc:title (keeping it simple for this).

If you're going to be consuming LOD data then you need to learn a bit more, RDF, SPARQL, Owl, ontologies and a bit more.

And if you want to get "in to" LOD in a big way, then go do it.

SEO Specialists
You need to know what the designers know, and you'll be changing from SEO specialists to data exploration optimizers or suchlike, focus will be on how you can make the data machine readable and get it linked in by the right services.. should be fun!

Further, you'll need to watch for how to get traffic to the sites, as mentioned search engine traffic will drop slowly over the next few months and years; with more focus going on "links" from related pages. As for the diggs & reddits, who knows how it'll effect traffic from them.

Summary

IMHO it's in all of our best interests to just get on with this, it will happen and the sooner YOU do it and convince your employers you have to make this move the better, companies can easily loose clients too if your competition is offering "web 3" and you aren't.

At no point have I seen a tech hit the web which could literally leave people behind if they don't jump on board; it's happened in other industries and now ours (remember VHS?).

The two questions most people / companies / clients will immediately raise..

1] We don't want to expose all our data for reasons X,Y&Z!

LOD isn't about exposing all the your data on the internet; it's about making the data you've already exposed on the internet in a more granular fashion, it's about making that data machine readable.

Presently you may have an article on a page with a title and author credit in HTML, in the future you would still have the same author and title, however they would be wrapped in markup that allows a machine to understand that "Joe Blogs" is a person who is the author of the article, and that the articles title is "I'm scared of exposing my data".

If you consider you're public facing web pages, everything on that page is already exposed, all we're doing here is describing what each bit of data is in a way we can all use.

2] Trust & Junk

One common misconception is that you have no control over the source of the data you pull from the "cloud", and that it could essentially be junk. However this couldn't be further from the truth, what we do is to find a source of data we trust that has their data exposed in a machine readable format, then query it for the exact information we want, and finally include or display it in our own system.

To illustrate, consider you wanted to reference the countries of the world with population in your system. Currently you would have to build  a database table, populate that data with country name and country population, then write some code to display that data. In this scenario you'd probably get the population data from a credible source such as wikipedia (copy and paste it in to your own database).

By using linked open data, you could treat the machine readable version of wikipedia (dbpedia) as your database table, query it instead and again write some code to display the data on you're own site.

You're displaying the same data, from the same trusted source; and you've selected which source you trust; it's not a case of just querying some cloud of data; it's a case of choosing which source(s) you want / trust and querying them.

As an additional bonus you don't need to worry about your information going out of date, as you're getting the data straight from source, the population of each country is updated on your site whenever it's updated on wikipedia.

Further, you don't need to worry about maintaining that list of countries, as in a single query you can pull out a list of all countries with each ones population, as the world grows and changes, so does your data.

Further still! once you've made the move to using some linked open data, all the data you could want is at your finger tips, let's say a decision is made to include 30 different bits of information about each country in your system. Consider that task for a minute - full system change, finding, collating and entering all that data; let alone maintaining it! Well, I'm sure you can guess the next bit, using LOD we can simply expand our original query to include the other bits of information we want, then display it - job done.

That's it.

Good Luck!

nathan

To me, the draw of linked data and the semantic web has always been exploration; the notion that even the most unskilled of publishers should be able to enrich their content via semi-automated software to the standards of a near perfect wikipedia article has always been the driving force. Additionally, content classification, relation, linkage, data centralization and the like are all major benefits which will make a vast difference to the usability of the web.

Search will always be a major part of the internet, at the moment we use search to find content on a specific subject, then search again to find more, and search again to find related or expanded info, help, facts, answers, whatever; however, in the future I hope to see search move to a less prominent role, one where we use search to find the most suitable "entry point" in to the web of linked data - and from there every other piece of related / expanded information is either on page, or a click (link) away.

Some major hurdles need to be jumped before we can get to that stage though, both through lack of organization and lack of appropriate software. Personally I have a mental blueprint / overview of what's needed (imho), and some very specific ideas on the software side, with any luck I'll get a chance to contribute + build some of this, we'll see.

Some thoughts of what's needed from my little brain.

Linked Data Ping
A central service API which is pinged by all software as it publishes information with machine accessible content. (Needed way before (x)HTML+RDFa takes off). Provides a stream of all recent pings to be consumed (xmpp pub-sub?).

Clustered Servers holding a centralized data GUID lookup and proxy.
In essence all resources on the net should be a linked pair of GUID to endpoint, each endpoint should contain a reference to the GUID, and each GUID should be a URI which redirects to the endpoint, endpoints change GUID/URI stays unique. In an ideal scenario when somebody creates a link to X resource or Y document, the publishing/controlling software should replace the endpoint with the GUID instead. This would also enable multiple other services such as centralized pingback, references, statistics etc.

Machine Readable Data Cache.
Together with the aforementioned services a high availability database of cache'd information should exist; in principal this would work by reading the stream of "Linked Data Pings", getting the GUID for the content and then retrieving all machine readable data and caching it. Much like the RDF data exposed through dbpedia, however for everything. Even if only a predefined subset of the common rdf vocabularies was stored and exposed it'd be enough to start, from there all other domain specific ontology could be retrieved by reading the endpoint itself.

Semantic CMS
Ideally we need a new breed of CMS, one that not only has simple FOAF and Dublin Core (~Drupal 7), but also support for full content enrichment using the aforementioned machine services; and provides a simple UI for manually exposing entities, events, facts etc. (Think highlight name in text, mark as Person with Name, system finds guid and builds relevant RDFa and we have another triple of linked data.)

The possibilities from this point are endless; if you're reading this document after all this has been made, then you'd see a whole host of in text links through to more information on each keyphrase, person, entity etc; you'd be aided by auto injection of sources, related reading, comments, further documents discussing the content here, in short you'd be exploring the net one click at a time, linked data all the way; not searching.

In summary (and very much imho), Linked Data is not for searching, it's for linking data - search was invented to address the issue that everything isn't linked, when it is then the link takes precedence again.

My only worry in all of this, is the idea that all rdf triples are fact, and true - already the major search engines are exposing rdfa data in summaries, 5* ratings on products and suchlike, the room for abuse will only get worse.

Thanks Georgi for placing the spark that clarified my current thoughts.

Finally, this isn't a biased opinion in anyway, or an endorsement, but to me openlink virtuoso, dbpedia, zemanta and open calais are leading the way and enabling all of this; together with the hard working folks contributing to the various linked W3C projects and specs. If only dbpedia/zemanta/calais would unify there uri's/guids/endpoints we'd be a lot further along.. (well I would ;).

Regards!

To save everything going to waste, here's a quick list of multiple handy resources which I've found (save you a bit of googling and time).

XHTML + RDFa and related techs.

• Wikipedia RDFa
• RDFa info
• W3C RDFa Primer, RDFa for HTML Authors, RDFa Syntax, RDFa Usage Scenarios
• Google RDFa Support help for webmasters.
• Drupal 7 RDFa Intro, RDF / RDFa Details
• ARC RDF/RDFa Classes for PHP
• OpenLink Virtuoso Open Source Database (RDBMS, ORDBMS, virtual database, RDF, XML, free-text, web application server, file server, sparql, ws-* etc)
• Notation3 on W3C, Wikipedia
• Turtle on Wikipedia, W3C, Authors Site
• N-Triples on Wikipedia, W3C
• GRDDL on Wikipedia, W3C Spec, W3C Primer, W3C Usage Scenarios

Semantic Web Resources

• Semantic Linked Data W3C ESW Wiki
• SOIC, FOAF, SKOS
• SPARQL Query Language, SPARQL Protocol, SPARQL Result Format, SPARQL Tutorial
• RDF Schema (RDFS) on Wikipedia, W3C RDFS Spec, Calais RDFS
• DBPedia Applications, and DBPedia Open Access
• Freebase Developer Docs, Freebase RDF
• Yahoo Term Extraction Service
• Zemanta Contextual Information Service Developer Documentation
• Open Calais Documentation
• AlchemyAPI
• Common Tag

Hopefully that'll be enough to get most people going!

Figured this one out years ago and have just never used it, you can display your rss feeds however you want using XSL to apply XHTML and CSS.

Take a look at mine here to see.

I'm not going to tell you too much about it, except for the fact that you need to pad out a chunk of atleast 512 bytes before the <rss> block in order for the xsl to be applied by browsers ;) check the source of my feed and you'll see a couple of comments - ahh something so simple that has stopped the entire world from skinning there rss and making it pretty.

should be fun to see what people do with this one!

Seeing as most of the audience of this blog will be using more recent browsers, I've opted for XSLT templates and small XML descriptors for each item - this saves me writing tonnes of html, saves the server some weight, saves you some bandwidth and let's me palm off all rendering on to the client.

Inkeeping with the spirit of experiments, one of the main reasons I've done this is to see how the xml documents get listed in google (if at all); and to check if analytics works with XSLT.

On this note, I had to do a little bit of fiddling with the recommended embed code from google to get it to work; if anybody else hits this problem here's a modified version.


<script src="http://www.google-analytics.com/ga.js" type="text/javascript"></script>
<script type="text/javascript">
try {
var pageTracker = _gat._getTracker("UA-3391780-6");
pageTracker._trackPageview();
} catch(err) {}
</script>


And as for the listings, well thats just a quick and dirty PHP 5 class which iterates the directories, loads them in to a dom document and spits out the xml - ultra simples.