Restarting Linked Data from scratch, part 1

nathan — Thu, 11 Mar 2010 07:41:51 +0000

I'm going out on a limb and starting my whole journey through Linked Data and "Web 3.0" again - in order to address the challenges many in the community are facing, and which are "blocking" me. I'm going to take everything I've learned so far and go right back to grass roots with linked data.

I'm primarily documenting this journey for my own benefit, for reference and to unload it from my brain; but hopefully it'll be of use to the wider community and any feedback will be massively appreciated.

Here goes, I'll start by analysing the web thus far:

The Web till now

The power and the success of the web so far, in my opinion, has come from four main things:

The URI
Hyperlink
The Resource
HTTP and it's RESTful design.

More importantly it's the combination of the four working together that makes the web so great, because, they let you cut out everything else and go straight to the resource you want. This is a point that we need to concentrate on for a minute.

Going straight to the resource

Back at the start of the web, this allowed people to (for the first time) jump from a resource in the bowels of one companies hierarchy straight to another resource in a different companies hierarchy - very much like reading a book, looking at the references and suddenly having the book or paper it references right there in your lap - amazing to say the least.

Skip forwards a few years and we have the search engines, suddenly simply by typing a few keywords you can jump right to a resource (page/image/...) anywhere on the web. Fast forward some more and we get to web 2.0 where again people are amazed every time direct access to a resource is exposed. Yup.. all your web 2.0 is just this simple principal..

An RSS feed, well it lets you read a resource (a post) outside the context of the website and inside lets say google reader. You can rip a resource (video) out of youtube and embed it anywhere you like. You can interact with a web application super fast thanks to targeting a resource directly with say ajax and only updating that resource rather than updating the entire view (page). You can interact with web applications using desktop clients because they let you access one resource at a time; and so on and so forth, virtually every improvement you see on the web comes down to that one thing, directly accessing a resource (and creating resources at a more granular level).

How we made the web faster

Negating the rather obvious upgrades in technology over the years, there is one primary thing that speeds up the web and virtually everything computer oriented, the cache.

Before all the web 2.0 stuff, resource caching was at an all time high and was making the web faster for all of us; caching at the resource level is enabled by HTTP and its RESTful design. Control Data allows us to limit how much information needs transferred over the web, request an image once and it gets transferred, request it a second time and thanks to caching and HTTP odds are very high that it won't get transferred again. When you consider that the average web page can easily have 30, 50, 300 images and static files embedded in it this is a huge speed increase, and frankly one we could not live without.

Skipping forwards to web 2.0 and the present day again, we've gone wild with caching; anybody who's been involved with a high traffic site will tell you that the only way to do it is to cache everything you can; from data in memory, through to code and op code caches. But this is only half the story.. a strange this has happened..

How we made the web slower

Simply, we forgot HTTP and a RESTful web somehow - that all important web caching whether it be at intermediate servers or in a web browser, it's forgotten.

To illustrate, if you view an image and then view it again, it'll be there instantly - why? because last-modified, etag and other control data is sent by great web servers like apache for static files, until you force a refresh or the file changes on the server you'll simply get a 304 response telling whatever cache down the line to use it's own copy instead. Now, try jumping on to a web page, even this one and you'll find the whole thing is reloaded, every time. I'd estimate that circa 80% of all pages you visit are fully reloaded every single time you see them, if not more.

Here's the reason - most pages are generated by scripts now, and something that goes unnoticed by most developers is that the web server (like apache) hand over *full* control to the language runtimes, and in turn to the developer. In other words, unless developers are calculating, receiving and sending control data for each response, and validating every http message in to their scripts, then most of the benefits of HTTP and RESTful design are completely lost; especially caching.

Here's a fact, whether you agree with it or not, to me it is a fact: the web has to be RESTful for it to work properly, whether it's a web of documents, or a web of data, or both.

Looking at the current state of Linked Data

Linked Data is amazing because it takes the big four I mentioned earlier (URI, Hyperlink, Resource, HTTP) to a new level; we create resources at the most granular level possible, assign them URIs, link them together with typed hyperlinks then expose them via HTTP.

Notice I didn't mention REST in there? that's because I (and I'm not the only one) don't feel that Linked Data is currently RESTful. And as we can learn from web 2.0, unless this is addressed we'll face major problems down the line. In addition, because of this lack of RESTful-ness I feel like the data isn't linked; simply using URIs from different datasets on both sides of a triple does not link those datasets, well not from a client perspective anyway.

To expand and refining the issues:

SPARQL Silos

Issue one, is that SPARQL and the servers with RDF stores which power it are positioned at the wrong side of the client / server relationship imho. Because each major dataset effectively has it's own server and access point (SPARQL interface) it means that when you query it, it can only return the Linked Data which it stores. This leaves us three options at the minute:

let that server pull in remote Linked Data and store it too (which makes the server fill up and slow down, and turns it in to a silo).
use one great big server that tries to store all the linked data (which feels like a silo all over again to me, not distributed at all).
Run our own server and only store limited data in it (limited.. and again a silo I guess)

If we moved SPARQL to the client side however, then all it would need is a starting point from which it could traverse the web of data, only pulling in what it needed for a query. This may sound slow but if all data was exposed as resources like it should be, and with control data so it could be cached, this slow down would soon disappear; lesson from web 1 and 2!

With regards the caching, this could happen at traditional intermediate caches within the internet and at ISPs, locally in client side triples stores (like a browsers cache) or the existing big servers that attempt to store all the linked data could be repositioned as linked data caches.

For example a small RDF document could simply delegate seeAlso http://datacac.he/http://subject.uri and that linked data cache could return back all the information it knows about the subject by returning the RDF results of a SPARQL describe. This alone would be a HUGE speed up, prevent silos and create a real web of data.

In addition, this would keep all linked data transferred through the web in RDF format, and thus machine readable and typed. At the minute we have lots and lots of SPARQL queries, which essentially are just untyped junk data that a machine couldn't possibly understand - SPARQL results remove all the goodness from RDF and give us something that is domain and developer specific, not re-usable. Think about that for a moment..

Clarification: I'm not saying SPARQL + RDF stores shouldn't be on the server side, they should as they are needed in most cases, I'm simply saying that the primary interface to linked data shouldn't be SPARQL over HTTP to a remote SPARQL endpoint. Rather we should be accessing RDF documents, or entities if you like via HTTP.

RESTful RDF

Issue two, the focus has been on getting data on the web, finding ways to link it, access it, store and query these vast datasets; and the work done thus far is amazing! But now that's handled it's time to go back to basics and find ways of both getting and publishing Linked Data RESTfully, at a granular per resource level.

This means handling RDF like ATOM, and essentially making atompub all over for RDF (as many are thinking and working on). I feel that regardless of what's implemented behind the interface, and whether triples stores and SPARUL are used, we still need to manage RDF / Linked Data in terms of documents and entities for it to be RESTful.

An additional issue raised by this is loosing the notion of a quad, g s p o, Named Graphs are vastly important to linked data, but we need to get named graphs in to triples and out of quads so we are always working with RDF through HTTP.

Also worth noting that temporal, provenance, multi-language, multiple representations etc will all need handled too; without using any HTTP extensions; no point half baking it or making the solution dependant on drafts - needs to work with the Universal Interface!

First Step

The above means one of the first challenges and things I'll try to tackle, is to find a way to fit a RESTful RDF publishing and exposing protocol in to the shared http space on a web server; taking in to account things like content negotiation, multiple representations, versions / time varying representations, and backwards compatibility with the current web.

note: I'm not going to define the protocols, plenty of more intelligent people than me are working on these things, just leverage a space where a full RESTful protocol can work in unison with the way we currently do things so that it's transparent to browsers and visible to linked data clients. This should then allow a stable test environment to try out different ways of doing things and test that the current web doesn't break.

To be continued.. often and frequently. I'm blocked on my current, v important project, and need to address these things.

webr3.org » Query languages