• a URI is associated with a thing by a group of agents/people as a name for that thing.
• some URIs are also associated with a set of representations over time by the dereferencing process.

Why were the representations made available for that URI?

• because somebody made a web page and then needed a uri to refer to it.
• because somebody named something with a uri and then wanted to provide information about it.
• because somebody made a web page about one specific thing and then needed a uri to refer to it and then the uri became commonly used to refer to the thing named.

That's a really minimal set of the different ways of looking at it, without getting in to any technical details at all, all three are really common cases of how people use URIs, the in-fighting is just people who've picked one of the three as being gospel, or technically required to make things work.

It's a social problem.

The httpRange-14 resolution picked the first of the above reasons as being the norm, and as requiring the least technical trade-offs. The resolution also accounted for the second case, with precedence given to the importance of having distinct names, rather than network performance or ease of implementation, again simply a design trade-off, one which prioritizes humans over machines.

I definitely cannot explain it any simpler than that.

Why? well the sem web community is packed full of the most technically skilled and decent people I've come across so far, so it can't be that, the tooling is pretty damn good, there's loads of data, most of the data's of a high quality, wanted by developers, and certainly more than usable. The concepts, theory and technical aspects are all solid as a rock. In short it's all good, apart from one rather important detail.

Our Linked Open Data isn't really open data, not in the eyes of the common web developer at least. To most web developers on the planet, open data is something they can get access to and use easily.

A good example of 'open data' for most developers, is the Twitter API.

Here's how a developer accesses it in PHP:

  $uri = 'http://api.twitter.com/1/statuses/show/7907258268647424.json';
  $tweet = json_decode(file_get_contents($uri));
  echo $tweet->user->description;

Here's how they access it in Javascript:

  uri = 'http://api.twitter.com/1/statuses/show/7907258268647424.json';
  $.getJSON(uri, function(tweet) {
    write( tweet.user.description );
  });

The reality of the matter is that you can't do this with Linked Open Data, and that's because you can't do it with RDF - and really, honestly, if it's not that simple, the masses won't use it, because, if it's not that simple, they can't use it.

The problems with the RDF formats

They're not perfect, and they are a very mixed bunch, for a change, let's look at the negatives.

RDF/XML
- Requires full XML tooling
- Can't read or write by hand
- butt ugly

Let's be honest, unless you have a full XML and RDF stack and you know what you're doing, RDF/XML is simply a no go zone.

RDFa
- Requires HTML/DOM/XML tooling
- An extension to a markup language, designed to augment annotated documents.

RDFa is great, but not for the general data use case, it's not a simple data interchange format like JSON, and you can't publish or consume it without specialist tooling, in fact it requires an even bigger, more complicated, stack than RDF/XML.

Turtle
- Requires a custom parser
- It's not yet seen as a data format by the masses.
- Doesn't have a registered media type.

If we're all honest, Turtle is the king of RDF serializations, it's small, powerful, easy to write and read, requires minimal tooling to parse. In fact, I'd quite happily say that Turtle is the best all round data format, period.

RDF/JSON and JSON-LD
- Square peg, round hole

You can't shoe horn RDF in to JSON, no matter how hard you try - well, you can, but you loose all the benefits of JSON in the first place, because the data is RDF, triples and not objects, rdf nodes and not simple values - I fear I'd better explain, quickly:

The benefit of JSON is that you can do the following:

  var u = tweet.user;                                // nested, simple objects
  write(u.message);                                  // simply "a string of text"
  if(u.geo_enabled) {                                // a boolean true
    var d = u.statuses_count * u.favourites_count;   // numbers..
  }

Anything more complicated than that and you've lost 95% of the benefits of JSON.

Here's the code to do that if(u.geo_enabled) with JSON/RDF:

  var tweet = rdf['http://example.org/tweet/12343'];
  var user = rdf[tweet['http://example.org/property/userid']];
  var geoenabled = user['http://example.org/property/geo_enabled'];
  if( Boolean(geoenabled.value) ) {

and JSON-LD:

  var tweet, user;
  for(o in jsonld) {
    if(jsonld[o]["@"] == 'http://example.org/tweet/12343') tweet = jsonld[o];
  }
  for(o in jsonld) {
    if(jsonld[o]["@"] == tweet['twit:userid']) user = jsonld[o];
  }
  if(user['twit:geo_enabled']) {

Remember, those two examples are only for the simple if line from the benefits of JSON example, can you even imagine all four lines?

Clearly, RDF in JSON is of little to no use to anybody, you can see plainly yourself, 95% of the benefits are lost and it's just another RDF serialization that's pretty much unusable without tooling. The only benefit JSON serializations of RDF have, are that you don't require an XML stack, which is quite a large benefit tbh.

The problem with RDF

I've been a little unfair there, you see the problem isn't with the serializations, we can't make a "better" serialization of RDF for general web developers, because the real problem is that the data's RDF, it's triples not simple objects, URIs rather than simple terms, RDF Nodes with a language or type, and not just simple values.

An array of RDF triples, or a structure of RDF, just simply isn't usable in most (all?) programming languages, by a typical web developer, without specialist tooling and libraries or APIs.

Am I saying RDF is bad? no of course not, it's awesomely brilliant in every way, it powers a paradigm shift and will have huge positive effects on the web and the human race. You know the score :)

What I am saying, is that we're not backwards compatible, we're not making our data open in formats which are usable by normal developers, developers who need and want the data, want the links, but not the semwebbery. Hell even most of us who are heavy sem web users only consider the ontology+reasoning side enabled by properties-with-uris some of the time, most of the time <http://xmlns.com/foaf/0.1/name> might as well just be "name".

Opening Linked Data

So, here's what we need to do, we need to just accept that although we publish linked data as RDF, we also need to publish the data as simple objects so the world can use the data.

Given the linked data:

  @prefix : <http://webr3.org/nathan#> .
  @prefix foaf: <http://xmlns.com/foaf/0.1/> .
  @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .

  :me a foaf:Person;
    foaf:age 29;
    foaf:holdsAccount [ foaf:accountName "webr3";
        foaf:homepage <http://twitter.com/webr3>;
        rdfs:label "Nathan's twitter account"@en ];
    foaf:homepage <http://webr3.org>;
    foaf:knows <http://example.com/bob#me>;
    foaf:name "Nathan";
    foaf:nick "webr3", "nath" .

  <http://example.com/bob#me> a foaf:Person;
    foaf:name "Bob" .

We also need to publish it like this:

{
  "http://webr3.org/nathan#me": {
    "a": "http://xmlns.com/foaf/0.1/Person",
    "age": 29,
    "holdsAccount": {
      "accountName": "webr3",
      "homepage": "http://twitter.com/webr3",
      "label": "Nathan's twitter account"
    },
    "homepage": "http://webr3.org",
    "knows": "http://example.com/bob#me",
    "name": "Nathan",
    "nick": [ "webr3", "nath" ]
  },
  "http://example.com/bob#me": {
    "a": "http://xmlns.com/foaf/0.1/Person",
    "name": "Bob"
  }
}

and now one can do: me.holdsAccount.label and get back a string, in any language.

What have we lost?
Well nothing in the grand scheme of things because we're still publishing the RDF in other formats via conneg, however in this specific serialization of the data: we've lost the properties, the .language and the .type, although basic types are still there, dates are detectable, numbers are supported, booleans are supported, everything else is just a PlainLiteral, a string.

What's still there?
The data, http names for things, follow your nose, rdf types ++usable-accessible-data in a web standard format. It's 3.5 if not 4.5 star data!

Other considerations
Probably be wise to allow direct access to a .json URI too, so people can simple-GET the data (in addition to exposing via conneg).

Should be an easy hit, any good reasons why not?

The other RDF Serializations

While we're here, there are a few other things that need tidied up, properly.

Turtle
Let's make it the standard RDF serialization, with a proper registered media types of application/turtle and text/turtle, fix any bugs in the spec (if any), and possibly allow an optional comma in those lists (1,2,3).

Let's just accept that RDFa is great, but sometimes you just want to embed a chunk of RDF in an HTML document, you know we all want to on occasion, people are shouting for it, it's easy to do, to deploy, doesn't break anything, and it's a really easy hit - so, in the Turtle standard spec pop a note that shows how to include it in an HTML document.

<script type="text/turtle">
  // turtle in here, as-is, no special encoding
</script>

RDF/XML
Leave it as is, let it run it's course naturally, and in many respects just forget it moving forwards, everybody supports it already, that won't change, but there's approximately zero need to keep pushing it.

RDFa
As awesome as it is, just see it for what it is, RDF in attributes, it's like the missing markup features of HTML, for annotating documents and describing the things described in the documents with annotations. It's not a simple data or RDF format, and it's not really suited to just dropping chunks of machine readable RDF in to an HTML document, Turtle in HTML will do that far better.

RDF/JSON and JSON-LD
Standardize a JSON serialization as a replacement / alternative to XML - but, admit and stipulate before hand that it's not usable as-is, or really writable and is arguably human readable. It simply needs to be a fast, unambiguous, optimized for the machine, RDF serialization - no bells and whistles for humans, no "12^^xsd:type" in a string - just something that you can JSON.parse and run circa 10-20 standardized lines of code over to get back an RDFGraph of RDFTriples.

Fin & end.

The specific issues are:

• Read-Write Linked Data
• Socially Aware Cloud Storage
• Levels of Abstraction: Net, Web, Graph

I'm yet to disseminate all that's changed, but they certainly are filled out and refined, remember folks the devils in the details!

Quite sure that I'll follow up with a bunch of notes, as will a few others - but for now, there's the heads up that it's time to do a bit of reading.

This is quite a dry subject, and I could be wrong (in fact in some ways I want to be proved wrong, this is how we learn), but do read on if you're interested.

Example

Over the past few months I've hit on a number of occasions where I was convinced I needed Named Graphs in order to address the task at hand.

A notable example is the scenario where using WebAccessControl and the ACL ontology, a system would have to figure out just who should be given access to a resource, and who should be denied.

In this example I'll cover the notion of ACL for "groups" in a linked data world.

The task at hand is to allow access if:
the graph serialized within the document obtained by dereferencing the URI of the group states the <webid#me> is a member.

Otherwise written as:
if we dereference <groups#admin> does the graph returned include the following { <groups#admin> sioc:has_member <webid#me> }

Or in SPARQL:

ASK
GRAPH <groups> {
  <groups#admin> sioc:has_member <webid#me>
}

In this example we *do not* want to dereference the users webid to see if the graph returned specifies that { <webid#me> sioc:member_of <groups#admin> } , or indeed consider the open world possibilities that another yet unknown graph could assert that the user is a member of our admin group, as that would breach security.

The ACL

To proceed with the example, consider the following ACL:

[] a acl:Authorization ;
	acl:accessTo <https://example.org/sensitive> ;
 	acl:agentClass :mygroup ;
 	acl:mode acl:Read .

:mygroup owl:equivalentClass [
 	a owl:Restriction ;
 	owl:hasValue <groups#admin> ;
 	owl:onProperty [ owl:inverseOf sioc:has_member ];
 	] .

The Problem

The problem proposed by this ACL is that any of the following four sets of triples would infer that <webid#me> would qualify as an instance of :mygroup (or a member of <groups#admin> if you prefer).

• <webid#me> sioc:member_of <groups#admin> .

• <webid#me> _:x <groups#admin> .
  _:x owl:inverseOf sioc:has_member .

• <groups#admin> sioc:has_member <webid#me> .

• <groups#admin> _:y <webid#me> .
  _:y owl:inverseOf sioc:member_of .

In other words, the ACL does not specify a "Named Graph" to query, and at the moment, no way exists to specify with (OWL or RDF) which "Named Graph" to query / trust.

This, point in case, is one example where I saw the need for Named Graphs in RDF and OWL.

Another way of looking at it

You will have noticed the notion of "Named Graphs" creeping in above, seems like a logical thing to say, especially when you consider that to process this ACL and grant access you'd probably use SPARQL, and specify a Named Graph to query over. However, much of what follows arose because I'd decided not to use SPARQL, and rather to code an ACL processor in my preferred language.

If you consider the situation, the ACL processor which decides if access should be granted or not, must implicitly "trust" the document which contains the serialized ACL graph. That is to say, that it must by extension trust any resources pointed to by said ACL, and if it doesn't then the ACL isn't fit for the purpose.

It's also important to note that "trust" is context specific, in this case we trust the resources pointed to by the ACL for the purpose of WebAccessControl.

One could then pretty quickly conclude that in this scenario the ACL processor already know's how to process the ACL, it must only use resources it trusts, therefore it must only allow access if the graph serialized within the document obtained by dereferencing the URI of the group states the <webid#me> is a member.

(because <groups#admin> is specified in the ACL, and thus by extension, trusted)

Named Graphs in SPARQL

The aforementioned logic would also apply if I was using SPARQL to process the ACL, it would equate to the ACL processor asking:

ASK
GRAPH <groups> {
  <groups#admin> sioc:has_member <webid#me>
}

But again this is very context specific to the example, let's consider for a moment that the URI for the group could have been a non-fragment URI, <groups/admin> for example.

This leads us to an important problem, when we dereference <groups/admin> it would have to 303 See Other through to a different URI, let's say <data/groups/admin> - which would then mean that the Named Graph to be used was <data/groups/admin> - this URI, you may note, we do not know when we are writing our ACL; so if we ASKed the above SPARQL, the results would always come back negative, since their is no GRAPH <groups>.

The URI of the Named Graph issue is compounded by modern web servers and publishing practises, because <data/groups/admin> could easily be content negotiated (or rewritten), thus giving various final URI's of <data/groups/admin> or <data/groups/admin.rdf> or <data/groups/admin.ttl> or <data/groups/admin.n3> and so forth. One could quite easily (and often does) end up with the same Graph repeated multiple times within a quad store, all under "different" "Named Graphs".

I'll expand on a possible way of addressing this problem further on.

Directionality

Previously I mentioned that the ACL processor didn't have a problem with the above ACL, because it by nature trusted all resources which were mentioned in the ACL graph. However, again this is very context specific.

Let's consider for a moment an inverted ACL, where we want to allow access if:
the graph serialized within the document obtained by dereferencing the URI of the users webid states that <webid#me> is a sioc:member_of <groups#admin>.

We don't know the users webid ahead of time when we write the ACL, so again we have no way of writing how to trust a resource - it is critical to note that even if RDF(2) did support the concept of Named Graphs, it still wouldn't address the situation because we wouldn't know the Named Graph ahead of time, in order to trust it!

If we now consider the following ACL:

[] a acl:Authorization ;
	acl:accessTo <https://example.org/sensitive> ;
 	acl:agentClass :mygroup ;
 	acl:mode acl:Read .

:mygroup owl:equivalentClass [
 	a owl:Restriction ;
 	owl:hasValue <groups#admin> ;
 	owl:onProperty sioc:member_of;
 	] .

The outcome of our previous logic concludes that again we should be querying the "trusted" resource <groups#admin>, which gives us another problem, that's not the resource we want to be asking in this scenario.

The only thing that remains, and I'll later argue the only thing that ever matters in a web of linked data, is direction.

If we analyse the first ACL closer, we can see that we ultimately used the direction inferred by the presence of owl:inverseOf to place <groups#admin> in the subject position, rather than the value/object position it could have been in, indicated by the presence of owl:hasValue. (bare with me).

In this example, we can use the strong semantics of owl:hasValue (and lack of owl:inverseOf) to place <groups#admin> in the value/object position, and thus our ACL processor can come to the outcome we want, which is to look for the a triple with the meaning { <webid#access> sioc:member_of <groups#admin> }, and that means dereferencing the URI in the subject position, in other words asking the graph serialized in the document returned by GETting <webid> if it contains such a triple.

I've applied some understanding to OWL that quite simply isn't there though, as I earlier stated both ACL examples could easily equate to looking for any one of those four sets of triples.

However, this is the point - machine understanding of data is in the domain of the machine, the application doing the processing. And "truth" or "trust" is entirely context specific.

I'm increasingly convinced that the combined context of the data in a graph and the context under which that graph is being queried, specifies or infers in which direction you want to be reading, and directionality can be determined with linked data by dereferencing whichever uri you place on the left / in the subject position.

I recently found that Tim Berners-Lee wrote about this in a blog post entitled Backward and Forward links in RDF just as important:

One meme of RDF ethos is that the direction one choses for a given property is arbitrary: it doesn't matter whether one defines "parent" or "child"; "employee" or "employer". This philosophy (from the Enquire design of 1980) is that one should not favor one way over another. One day, you may be interested in following the link one way, another day, or somene else, the other way.

Key here is the sentence "One day, you may be interested in following the link one way, another day, or somene else, the other way.", and that is exactly what all these examples are doing, following a link one way, or the other way.

To conclude this part, in every scenario thus far where I've thought I needed Named Graphs, it turns out that I in-fact needed directionality - and because I'm dealing with Linked Data, whatever I place in the subject position defines the URI which I need to dereference, and ultimately the Graph(s) which are considered when resolving the answer to the question being ASKed.

I'd thus suggest that "Named Graphs", do not exist in a web of data, they are needed in N3 and when using rules, because all data is often in a single file, however that is not the case for Linked Data, where we dereference.

Back to SPARQL and Named Graphs

Previously I mentioned the complications with the way we currently use named graphs in SPARQL and in our quad stores, where the URI we end up using could literally be, anything; and often we get duplicate data under different graphs.

To address this, I'd suggest that what we should be storing as the graph ?g value, is not some made up "named graph" but rather: the dereferenced URI which we initially requested.

• in the case of <group#admins> this would be <group>.
• in the case of <group/admins> this would be <group/admins>

To clarify, *never* the URI that a GET request finally resolves to, and *always* the initial dereferenced URI we requested.

The above ensure that we'd never have duplicate data in our quad stores again, that SPARQL queries including a FROM clause always dereferenced, that publishers and web server administrators were free to relocate and restructure their data, and ultimately make for a much nicer, healthier web of data.

Cool URIs don't change, and they wouldn't, just because the final document serializing a graph may move to a different URI, doesn't mean the original URI has to change.

Conclusion

Apologies for the length of the post, but I figured everything needed covered, in context. Simply put we need to focus less on Named Graphs (which IMHO aren't needed) and focus more on directionality. Every problem I've encountered thus far is covered by what Tim said years ago: "One day, you may be interested in following the link one way, another day, or somene else, the other way."

Comments?

OpenCalais and Zemanta usage details and service comparison.

The demo leverages OpenCalais in order to pick up references to things, which are returned in most cases as string literals; OpenCalais can also be configured to return back socialtags which give a broad stroke idea of what the document is about, again with string literal "tags". With regards the references (semantic metadata, Entities, Facts, Events etc.) which OpenCalais returns, whilst it is generally string literals, it also returns back vital Type and Relevance information, so in the case of "London" it will also assert that London is a City. Even in the case where it doesn't previously know what a thing is, it can work out that say "Frank Neverbeenheardofbefore" is a Person.

Zemanta is also leveraged, the primary difference between Zemanta and OpenCalais (and thus the need for both services) is that Zemanta focuses more on accurate tagging of text. Primarily though, Zemanta tags (again string literals) are meaningful tags which are commonly known and are referenced to either existing Linked Data identifiers such as http://dbpedia.org/resource/London and further information about the tag (or thing), in the case of the aforementioned London, then it will often also provide links to the wikipedia page for London, the official homepage to the city of London and a link to show the position of London on google maps.

I should point out that ever increasingly OpenCalais also returns back Linked Data too, for instance in the case of London they have given it an HTTP URI which can be dereferenced to retrieve more information about London. At a very crude estimation I would suggest that (depending on the subject matter) OpenCalais returns Linked Data URIs for about 15% of all references it finds to well known "things".

Weighing up the two services I couldn't say that one is better than the other, both have advantages and disadvantages, the only way to get a decent overall picture is to use both. for the benefits of feedback to both of these great services though, here is a general comparison:

note: none of these figures are from exact tests, they are from extensive developer usage of both services as I've used them both since they were made public.

Zemanta is generally 2x as fast for average texts (the size of this post for instance) and as much as 5x as fast for longer texts. Average for Zemanta being 0.7 to 2 seconds. Average for OpenCalais being 1.5 to 10 seconds. It may also be worth noting that the availability of Zemanta is somewhat higher than that of OpenCalais, perhaps 1 in 250 calls to OpenCalais will fail.

OpenCalais does a lot more heavy work than Zemanta though, and *really* semantically analyzes the text to figure out a wealth of information. In this respect the tables are completely turned and Zemanta consitently deals with providing a few high quality known tags; where as OpenCalais often provides at least 10x as much information about a given text, including relevance and type as mentioned before. OpenCalais also extracts Facts / Events, and further it can figure out that "Jim" is also "Jim Bob", and that Jim said X about Y on date D.

Generally you can trust the data from Zemanta 99% as it deals with "known" things, however due to this in some cases very new topics (such as IPad for the first few days after its announcement) remain unknown. Due to the nature of OpenCalais and it's dealing with the unknown you need to take more time to verify what it has returned, however when OpenCalais assigns a LinkedData identifier to something or provides more information you can 99.99% trust that it is entirely accurate.

It's worth noting that both of these services do different things though, and both do it extremely well, Zemanta "tags" and OpenCalais "semantically extracts information", in some respects I was hesitant about comparing the two, as in the context of what I'm doing both are needed and both are equal, however in different contexts both do different jobs and there is a need for people to select one over the other.

Out of all the competition though, I would highly recommend both Zemanta and OpenCalais over their respective competitors, and do hope that neither of these great services ever decide to target each others markets. (e.g. they compliment each other well and both do so well because they stick to what they are good at).

extractor.data.fm details

This demo deals primarily with figuring out what a document is about; in that it aims to provide back a list of:

• Categories
  A list of 1-5 dbpedia (and therefore wikipedia) categories which the provided document would be categorized under if it were a wikipedia article and had been categorized by a huma who was knowledgeable in the subject domain(s) of the text.
• General Topics
  A short list of the general and broad-strokes Subjects covered by the document, these can are distinct from the primary specific subjects covered and the categories, and in many ways can be seen as the most common intersections between the primary specific subjects discussed.
• Primary Subjects
  These are the specific subjects covered in the document, not just the things mentioned, but the things actively discussed within the document, the primary subject matter as it were.
• Related or Mentioned Subjects
  Whilst I've termed them "related" as in dcterms:related, these are simply things which have been detected in the document or text and which are not primary subjects; in many ways "mentions" may be a more appropriate term.

Out of the above list, the two services do the heavy lifting to give the demo it's Primary Subjects and Related Subjects; in short OpenCalais' SocialTags and Zemanta's Tags give us back our Primary Subjects. Whilst OpenCalais by way of the semantic extraction provide us with the Related Subjects, namely all those extracted semantics which have the Type of a real thing (not an IndustryTerm or Event) and which are not all ready a Primary Subject; additionally those extracted semantics which are not tags but have a relevance higher than a certain score are boosted up to be Primary Subjects too.

A primary and initial function of the demo is to associate the tags returned by both services together, and figure out when each is talking about the same thing; this is covered first by dealing with the linked data they return; where both services are talking about the same thing you simply know this unambiguously due to the nature of http URIs and them both being the "sameAs" each other. After this two chunks of unhandled data remain, Zemanta tags which have not determined to be the sameas OpenCalais ones; and OpenCalais semantics which we have a string literal name for and a type.

In step Openlink Virtuoso 6.1 (open source edition!) with most of dbpedia 3.4 loaded in to do the heavy lifting from here on; Virtuoso is a really powerful bit of kit and has replaced mysql/sql server/postgres, rdf store and web dav server in my typical server stack. The public lod and dbpedia endpoints really do no justice as to just how powerful and fast Virtuoso is, queries which take a few seconds on the public endpoint return in hundredths of a second on my local (low spec) server, and the comparative performance to the aforementioned RDBMS solutions is not to be sniffed at.

To handle the typed string literals from OpenCalais, I built a custom dbpedia lookup service (using sparql over the aforementioned Virtuoso + dbpedia setup) which tries to unambiguously determine the identifier for a string literal, if it is known; the results are pretty good and I'd safely say that it gets it right in 98% of cases. This essentially turns the remaining unknown string literals in to known Linked Data URIs, and as a side benefit gives the correct full Name for the thing identified along with the correct casing and obviously much more linked data.

Remaining now the demo has a few OpenCalais semantics which are still unknown, but we know the Type and have a name for the thing; and as URIs are given to things that can be Named, I simply mint my own uri's for these and specify the OpenCalais identifier as a "seeAlso" (to be future compatible with a time where they do associate there own hash uris through to linked data).

At this point the demo has all of the Primary Subjects and Related Subjects determined and where possible linked through to additional LinkedData and human readable web documents about the subjects.

Categorization

This is where the script comes in to it's own and really leverages virtuoso, up till this point it's all been about cleaning, validating, looking up, associating and suchlike.

Given that we now have linked data HTTP URIs for all the subjects we are dealing with, and in all Primary Subject cases we also have dbpedia.org URIs the demo can start to use some of Virtuoso's more powerful features. First point of call is to get the Category intersection of all primary subjects (including the inferred categories!) via a slightly complex transitive sparql query over the dbpedia dataset. From here the demo calculates a set of primary categories which the text is related to, then it finds the general category intersection (again including inferred categories) between the primary categories, and the primary subjects. with the results returned is a wealth of numerical information which the demo dually considers and can then infer which are the General Subjects and the Categories for the text.

At some point I'll cover this part of the script in more details and give some virtuoso specific transitive SPARQL queries for you to use in your own such creations, but for now the above will have to do.

Conclusion

This extractor demo is something I've been working on and trying to achieve for about 5 years, and whilst it is still early days it's the first time the technologies have been available to both make it possible, and to utilize the results correctly to achieve what I'm aiming for overall.

The overall goal is to create a system which allows users to simply drop in content, and the system "files" it in the correct categories, lists it under the correct subjects and interlinks it with other resource via typed links such as "related resources" and looser resource lists of "also mentioned here", further benefits of such a system are that you can accurately figure out what readers are interested in and promote new content to them, you can give users the option of content streams where they can watch specific subjects or combination of subjects to be notified of their "ideal" reading. On the flip side you can also identify users and contributers interests and expertise, and correlate these together (with geo-location) to suggest others users who they may wish to collaborate with, other organisations doing the same work in the same fields and many such uses. In reality I have much of this implemented in a site I've been working on for the last year, which is just being rolled out again, and the system works extremely well with huge benefits to all involved, the site you see deals with climate adaptation and both provides a service to the general adaptation community where they can share and find knowledge, and more importantly serves organisations working on critical issues by letting them see which people / organisations / projects are doing what, where and allows them to both co-ordinate efforts and perhaps more importantly not duplicate efforts and waste resources where it counts most. This has a positive impact on the worlds poorest nations and those suffering people who these organisations are trying to work with and help.

Back to the demo, and with the context described, the extractor.data.fm demo is a quick UI around an API which is in many ways the backbone of the aforementioned system. The API is used in a semi-automated way, where the data returned by it is verified in a UI by the content author / admins who remove any unambiguous data and then hit save, from there everything is automated again and the system functions as above.

I'm unsure whether this kind of system will ever be able to be fully automated (or whether its wise to allow this) as certain scenarios just can't be covered yet, a real life example of this is an initiative called "TEA", ambiguity at this level, and with entities which are unknown to systems or even the web of data, will always be an issue at some point, as things progress it may be they are only ambiguous once, on their first discovery, but that is still once; hence why this may always have to be a semi-automated process.

The Standard Way
This way essentially exports all SSL data, certs, client and server side if you read the notes has performance penalty.

SSLVerifyClient optional_no_ca
SSLVerifyDepth 1
SSLOptions +StdEnvVars
SSLOptions +ExportCertData


The Lighter Way
This way simply passes in the SSL_CLIENT_CERT in to the env REMOTE_USER and skips the rest which you don't use (for FOAF+SSL).

SSLVerifyClient optional_no_ca
SSLVerifyDepth 1
SSLUserName SSL_CLIENT_CERT


Tested and works very nicely (again, imho).

note: Enabling SSLOptions +FakeBasicAuth will overwrite this with the Subject from the client side certificate.

I'm assuming that if you are reading this, you know what linked data is, and REST as per the dissertation of Roy T. Fielding. If not go do some reading :)

Interface Constraints

REST is defined by four interface constraints:

1. identification of resources
2. manipulation of resources through representations
3. self-descriptive messages
4. hypermedia as the engine of application state.

From here I'll look at each of these four constraints and build up the approach as I go.

What a resource is

Quoting extensively from REST 5.2.1.1 Resources and Resource Identifiers:

The key abstraction of information in REST is a resource. Any information that can be named can be a resource: a document or image, a temporal service (e.g. "today's weather in Los Angeles"), a collection of other resources, a non-virtual object (e.g. a person), and so on. In other words, any concept that might be the target of an author's hypertext reference must fit within the definition of a resource...

A resource is a conceptual mapping to a set of entities, not the entity that corresponds to the mapping at any particular point in time...

The values in the set may be resource representations and/or resource identifiers...

A resource can map to the empty set, which allows references to be made to a concept before any realization of that concept exists...

The only thing that is required to be static for a resource is the semantics of the mapping, since the semantics is what distinguishes one resource from another...

What a representation is

Again, quoting extensively from REST 5.2.1.2 Representations:

... A representation is a sequence of bytes, plus representation metadata to describe those bytes. Other commonly used but less precise names for a representation include: document, file, and HTTP message entity, instance, or variant...

If the value set of a resource at a given time consists of multiple representations, content negotiation may be used to select the best representation for inclusion in a given message...

The data format of a representation is known as a media type

Identification of resources

To do this properly I need to identify some resources, so for this I'm going to work with "Something" :)

• "Something" - a resource, a non-virtual object

At any point in time I have a description of Something which has multiple representations in different mediatypes, all semantically matching or equivalent:

• "something.rdf" - representation of Something with mediatype RDF+XML
• "something.n3" - representation of Something with mediatype RDF+N3
• "something.en.html" - representation of Something, in english, with mediatype text/html
• "something.de.html" representation of Something, in german, with mediatype text/html

Each one of those representations is also a resource because they can be the target of a hyperlink. Of course by resource I mean a conceptual mapping to each of the things listed, and I haven't assigned URIs but will..

To be able to make this set of representations manageable and to indicate they are in a set, I'm going to add in another resource which is a collection of resources, which can be considered a set of these equivalent representations of Something at a fixed point in time. For the purpose of this exercise, that point in time is today.

• "Something-20100311" - a resource which is a collection of equivalent representations of Something on the 11th March 2010.

Additionally, for the sake of argument, I'm going to say that a new set of representations (or version) is added every day - to handle this I then need one more resource, a collection of resources, where each resource in the collection is itself a collection of resources (one of the aforementioned and including the example "Something-20100311"). This will give me a conceptual mapping which covers time, and therefore everything I could need.

• "Somethings" - a resource which is a collection of resources, see above for full description!

Finally, I'm going to add in two shortcut resources which have no representation and are simply conceptual maps to the first and most current sets of representations.

• "first" - a resource which always maps to the first collection of representations of Something.
• "latest" - a resource which maps to the most recent collection of representations of Something.

Giving the resources URIs

Now to assign some URIs for this use case, there is no set structure and I'm not going to define one because it is up to each server (or manager of) to control it's own URI space, but for the sake of this exercise I'll define mine as follows:


base: http://data.webr3.org
...
/d/Something
/rg/Somethings
/rg/Somethings/first
/rg/Somethings/latest
/rg/Somethings/Something-20100311
/rg/Somethings/Something-20100311/something.rdf
/rg/Somethings/Something-20100311/something.n3
/rg/Somethings/Something-20100311/something.en.html
/rg/Somethings/Something-20100311/something.de.html
...
/rg/Somethings/Something-20100305
/rg/Somethings/Something-20100305/something.rdf
/rg/Somethings/Something-20100305/something.n3
/rg/Somethings/Something-20100305/something.en.html
/rg/Somethings/Something-20100305/something.de.html
...


From the above you can see that every possible representation has its own URI, in addition every collection of equivalent representations has its own URI, as does the collection of all those collections; and so does "Something" our non virtual object.

Also we've exposed multiple resources which could also be RESTful CRUD access points operating on an atompub style protocol. Small sentence, big potential, will cover approaches and protocols in later posts.

The Key resource

The most important thing, which I haven't yet covered, is that we've exposed a key resource, namely /rg/Somethings. This is a resource at the top of the representation chain which can be used to expose content negotiation, be it server or agent driven (or a mix of both), and regardless of the mappings and levels of collection further down the line this can always be a single point of entry to get representations.

I'll cover just how in a moment, but for now something important.

Important

I've had to give a fixed example just to make some progress, but we have to remember that every system has different needs, in some cases it may be that there is only a single fixed representation for a resource, whilst in others each strand of representation (like something.de.html) may take it's own versioning / temporal path. This could indicate that a structure such as the following may be in order:

...
/d/Something
/rg/Somethings
/rg/Somethings/first
/rg/Somethings/latest
/rg/Somethings/Something-20100311
/rg/Somethings/Something-20100305
/rg/Somethings/Something-rdf
/rg/Somethings/Something-rdf/20100311.rdf
/rg/Somethings/Something-rdf/20100305.rdf
/rg/Somethings/Something-html-en
/rg/Somethings/Something-html-en/20100311.html
/rg/Somethings/Something-html-en/20100305.html
/rg/Somethings/Something-html-de
/rg/Somethings/Something-html-de/20100308.html
/rg/Somethings/Something-html-de/20100303.html


The above highlights that whilst we may have added more resources, the core resources are still the same; remember that they are "conceptual maps", meaning that Something-20100311 may "map" to the version of en-html on the 11th and de-html on the 8th, because the de version was written first, then translated to english and from there rdf and so forth, but they are all semantically equivalent, containing the same information even though they were created at different times.

The Conceptual Maps are as follows, from what I can tell this should always cover any use-case, no matter how complex.


Thing 1-1 CollectionOfCollections
CollectionOfCollections 1-* CollectionOfEquivalentRepresentations
CollectionOfEquivalentRepresentations 1-* Representation


aside:At times like this I wish I'd had a chance to study computer science so that I could express these things formally, so you'll have to make sense of it as best you can :( sorry.

Exposing via Content Negotiation

In my research so far, I've been able to figure out how to expose all of the aforementioned via HTTP, RESTfully using content negotiation in a manner which seems to be transparent to existing web browsers, but exposes all the information needed in a manner that is visible to machines; without using any additional extensions headers. As follows:

1 The client does a normal GET request on our "Something", notice that no content negotiation is happening yet, we are simply asserting via a 303 "that the requested resource does not have a representation of its own that can be transferred by the server over HTTP."

#Request
GET /d/Something HTTP/1.1
Host: data.webr3.org
Accept: text/html;q=0.5, application/rdf+xml


#Response
HTTP/1.1 303 See Other
Location: http://data.webr3.org/rg/Somethings


2The client does a GET on the URI we specified in the Location field, namely to our key resource that can be used for content negotiation over all the representations.

#Request
GET /rg/Somethings HTTP/1.1
Host: data.webr3.org
Accept: text/html;q=0.5, application/rdf+xml


#Response
HTTP/1.1 300 Multiple Choices
Location: http://data.webr3.org/rg/Somethings/latest
Content-Type: application/xhtml+xml
Content-Length: 17400


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML+RDFa 1.0//EN"
"http://www.w3.org/MarkUp/DTD/xhtml-rdfa-1.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
xmlns:foaf="http://xmlns.com/foaf/0.1/"
xmlns:dc="http://purl.org/dc/elements/1.1/"
version="XHTML+RDFa 1.0" xml:lang="en">
...


Here's where it gets interesting and clients can take different routes; first the route of the typical user agent:

User Agent Route


#Request
GET /rg/Somethings/latest HTTP/1.1
Host: data.webr3.org
Accept: text/html;q=0.5, application/rdf+xml


#Response
HTTP/1.1 307 Temporary Redirect
Location: http://data.webr3.org/rg/Somethings/Something-20100311


#Request
GET /rg/Somethings/Something-20100311 HTTP/1.1
Host: data.webr3.org
Accept: text/html;q=0.5, application/rdf+xml


#Response
HTTP/1.1 302 Found
Vary: Accept
ETag: W/"xyzzy"
Last-Modified: Wed, 11 Mar 2010 12:45:26 GMT
Content-Type: application/xhtml+xml
Content-Length: 17400
Content-Language: en
Content-Location: http://data.webr3.org/rg/Somethings/Something-20100311/something.en.html


<!DOCTYPE html...


First you can see that the user agent simply goes straight through to the most recent content and what they expect to see; which is nice, with additional Server driven content negotiation.

Further, we can see that full cache control is in there which as we know speeds up the net, and further still we have a rather nifty "weak" entity tag; this entity tag is shared by all representations which are semantically equal, and asserts they are equal via the entity tag. It's also worth noting that you could add this entity tag to your RDF graphs and further assert provenance which could come in very handy down the line for POST and PUT implementations.

To recap, common user agents just go straight through to the expected resource via server driven content negotiation and can take full advantage of cache / control data.

The Machine Route

Back at 2 the server returned a 300 Multiple Choices as soon as /rg/Somethings was requested. All important was that the entity returned was XHTML+RDFa (although this could have been Atom or similar..), which means we can give both a human and machine readable list of all our various representations, the "machine" can then select which one it finds most fitting. The choices could be expressed using any suitable ontology; and further both Alternative and Link headers could be added if publishers wished.

I think that covers it all, if there are any errors or things I've missed please do let me know asap; but for now that'll do me - it's verbose, but I like verbose - prove it works then optimise it later :)

I'm primarily documenting this journey for my own benefit, for reference and to unload it from my brain; but hopefully it'll be of use to the wider community and any feedback will be massively appreciated.

Here goes, I'll start by analysing the web thus far:

The Web till now

The power and the success of the web so far, in my opinion, has come from four main things:

• The URI
• Hyperlink
• The Resource
• HTTP and it's RESTful design.

More importantly it's the combination of the four working together that makes the web so great, because, they let you cut out everything else and go straight to the resource you want. This is a point that we need to concentrate on for a minute.

Going straight to the resource

Back at the start of the web, this allowed people to (for the first time) jump from a resource in the bowels of one companies hierarchy straight to another resource in a different companies hierarchy - very much like reading a book, looking at the references and suddenly having the book or paper it references right there in your lap - amazing to say the least.

Skip forwards a few years and we have the search engines, suddenly simply by typing a few keywords you can jump right to a resource (page/image/...) anywhere on the web. Fast forward some more and we get to web 2.0 where again people are amazed every time direct access to a resource is exposed. Yup.. all your web 2.0 is just this simple principal..

An RSS feed, well it lets you read a resource (a post) outside the context of the website and inside lets say google reader. You can rip a resource (video) out of youtube and embed it anywhere you like. You can interact with a web application super fast thanks to targeting a resource directly with say ajax and only updating that resource rather than updating the entire view (page). You can interact with web applications using desktop clients because they let you access one resource at a time; and so on and so forth, virtually every improvement you see on the web comes down to that one thing, directly accessing a resource (and creating resources at a more granular level).

How we made the web faster

Negating the rather obvious upgrades in technology over the years, there is one primary thing that speeds up the web and virtually everything computer oriented, the cache.

Before all the web 2.0 stuff, resource caching was at an all time high and was making the web faster for all of us; caching at the resource level is enabled by HTTP and its RESTful design. Control Data allows us to limit how much information needs transferred over the web, request an image once and it gets transferred, request it a second time and thanks to caching and HTTP odds are very high that it won't get transferred again. When you consider that the average web page can easily have 30, 50, 300 images and static files embedded in it this is a huge speed increase, and frankly one we could not live without.

Skipping forwards to web 2.0 and the present day again, we've gone wild with caching; anybody who's been involved with a high traffic site will tell you that the only way to do it is to cache everything you can; from data in memory, through to code and op code caches. But this is only half the story.. a strange this has happened..

How we made the web slower

Simply, we forgot HTTP and a RESTful web somehow - that all important web caching whether it be at intermediate servers or in a web browser, it's forgotten.

To illustrate, if you view an image and then view it again, it'll be there instantly - why? because last-modified, etag and other control data is sent by great web servers like apache for static files, until you force a refresh or the file changes on the server you'll simply get a 304 response telling whatever cache down the line to use it's own copy instead. Now, try jumping on to a web page, even this one and you'll find the whole thing is reloaded, every time. I'd estimate that circa 80% of all pages you visit are fully reloaded every single time you see them, if not more.

Here's the reason - most pages are generated by scripts now, and something that goes unnoticed by most developers is that the web server (like apache) hand over *full* control to the language runtimes, and in turn to the developer. In other words, unless developers are calculating, receiving and sending control data for each response, and validating every http message in to their scripts, then most of the benefits of HTTP and RESTful design are completely lost; especially caching.

Here's a fact, whether you agree with it or not, to me it is a fact: the web has to be RESTful for it to work properly, whether it's a web of documents, or a web of data, or both.

Looking at the current state of Linked Data

Linked Data is amazing because it takes the big four I mentioned earlier (URI, Hyperlink, Resource, HTTP) to a new level; we create resources at the most granular level possible, assign them URIs, link them together with typed hyperlinks then expose them via HTTP.

Notice I didn't mention REST in there? that's because I (and I'm not the only one) don't feel that Linked Data is currently RESTful. And as we can learn from web 2.0, unless this is addressed we'll face major problems down the line. In addition, because of this lack of RESTful-ness I feel like the data isn't linked; simply using URIs from different datasets on both sides of a triple does not link those datasets, well not from a client perspective anyway.

To expand and refining the issues:

SPARQL Silos

Issue one, is that SPARQL and the servers with RDF stores which power it are positioned at the wrong side of the client / server relationship imho. Because each major dataset effectively has it's own server and access point (SPARQL interface) it means that when you query it, it can only return the Linked Data which it stores. This leaves us three options at the minute:

• let that server pull in remote Linked Data and store it too (which makes the server fill up and slow down, and turns it in to a silo).
• use one great big server that tries to store all the linked data (which feels like a silo all over again to me, not distributed at all).
• Run our own server and only store limited data in it (limited.. and again a silo I guess)

If we moved SPARQL to the client side however, then all it would need is a starting point from which it could traverse the web of data, only pulling in what it needed for a query. This may sound slow but if all data was exposed as resources like it should be, and with control data so it could be cached, this slow down would soon disappear; lesson from web 1 and 2!

With regards the caching, this could happen at traditional intermediate caches within the internet and at ISPs, locally in client side triples stores (like a browsers cache) or the existing big servers that attempt to store all the linked data could be repositioned as linked data caches.

For example a small RDF document could simply delegate seeAlso http://datacac.he/http://subject.uri and that linked data cache could return back all the information it knows about the subject by returning the RDF results of a SPARQL describe. This alone would be a HUGE speed up, prevent silos and create a real web of data.

In addition, this would keep all linked data transferred through the web in RDF format, and thus machine readable and typed. At the minute we have lots and lots of SPARQL queries, which essentially are just untyped junk data that a machine couldn't possibly understand - SPARQL results remove all the goodness from RDF and give us something that is domain and developer specific, not re-usable. Think about that for a moment..

Clarification: I'm not saying SPARQL + RDF stores shouldn't be on the server side, they should as they are needed in most cases, I'm simply saying that the primary interface to linked data shouldn't be SPARQL over HTTP to a remote SPARQL endpoint. Rather we should be accessing RDF documents, or entities if you like via HTTP.

RESTful RDF

Issue two, the focus has been on getting data on the web, finding ways to link it, access it, store and query these vast datasets; and the work done thus far is amazing! But now that's handled it's time to go back to basics and find ways of both getting and publishing Linked Data RESTfully, at a granular per resource level.

This means handling RDF like ATOM, and essentially making atompub all over for RDF (as many are thinking and working on). I feel that regardless of what's implemented behind the interface, and whether triples stores and SPARUL are used, we still need to manage RDF / Linked Data in terms of documents and entities for it to be RESTful.

An additional issue raised by this is loosing the notion of a quad, g s p o, Named Graphs are vastly important to linked data, but we need to get named graphs in to triples and out of quads so we are always working with RDF through HTTP.

Also worth noting that temporal, provenance, multi-language, multiple representations etc will all need handled too; without using any HTTP extensions; no point half baking it or making the solution dependant on drafts - needs to work with the Universal Interface!

First Step

The above means one of the first challenges and things I'll try to tackle, is to find a way to fit a RESTful RDF publishing and exposing protocol in to the shared http space on a web server; taking in to account things like content negotiation, multiple representations, versions / time varying representations, and backwards compatibility with the current web.

note: I'm not going to define the protocols, plenty of more intelligent people than me are working on these things, just leverage a space where a full RESTful protocol can work in unison with the way we currently do things so that it's transparent to browsers and visible to linked data clients. This should then allow a stable test environment to try out different ways of doing things and test that the current web doesn't break.

To be continued.. often and frequently. I'm blocked on my current, v important project, and need to address these things.

I've chosen the rather old RFC 2616 (1999!) at this time rather than the upcoming HTTPbis because I feel it's important to know where you are coming from, and whilst many things about the Web of Data feel new, they are really age old principals and technologies which have never been used to their full potential. Further you won't be able to appreciate the refinements in HTTPbis if you don't know what it's refining.

Virtually everything from here on is just a snippet/quote or paraphrase of the RFC. Let's start with a simple one:

Why use HTTP?

HTTP is an application-level protocol for distributed, collaborative, hypermedia information systems. ... HTTP allows an open-ended set of methods and headers that indicate the purpose of a request. ... HTTP is also used as a generic protocol for communication between user agents and proxies/gateways to other Internet systems ... HTTP allows basic hypermedia access to resources available from diverse applications. source

I do fully recommend reading the entire RFC and the new HTTPbis, most questions can be answered by returning to these documents and reading what they say (it's all in the detail); here's some more info gleaned from the RFC:

The difference between POST and PUT, URIs as Identifiers, and URIs to identify more than just documents.

The fundamental difference between the POST and PUT requests is reflected in the different meaning of the Request-URI. The URI in a POST request identifies the resource that will handle the enclosed entity. That resource might be a data-accepting process, a gateway to some other protocol, or a separate entity that accepts annotations. In contrast, the URI in a PUT request identifies the entity enclosed with the request -- the user agent knows what URI is intended .. source

Using POST RESTfully for more than just form data.

The POST method is used to request that the origin server accept the entity enclosed in the request as a new subordinate of the resource identified by the Request-URI in the Request-Line. POST is designed to allow a uniform method to cover the following functions: Annotation of existing resources; ... Extending a database through an append operation. The actual function performed by the POST method is determined by the server and is usually dependent on the Request-URI. The posted entity is subordinate to that URI in the same way that ... a record is subordinate to a database. source

What to do if something is created as a result of a POST request.

If a resource has been created on the origin server, the response SHOULD be 201 (Created) and contain an entity which describes the status of the request and refers to the new resource, and a Location header. source

When to use a PUT request?

The PUT method requests that the enclosed entity be stored under the supplied Request-URI. source

How to handle a PUT

If the Request-URI refers to an already existing resource, the enclosed entity SHOULD be considered as a modified version of the one residing on the origin server.

If the Request-URI does not point to an existing resource, and that URI is capable of being defined as a new resource by the requesting user agent, the origin server can create the resource with that URI. If a new resource is created, the origin server MUST inform the user agent via the 201 (Created) response. If an existing resource is modified, either the 200 (OK) or 204 (No Content) response codes SHOULD be sent. source

if extra headers were sent?

Unless otherwise specified for a particular entity-header, the entity-headers in the PUT request SHOULD be applied to the resource created or modified by the PUT. source

and what if I want to save it somewhere other than the URI specified by the client?

If the server desires that the request be applied to a different URI, it MUST send a 301 (Moved Permanently) response. source

and if the PUT can't be done..

If the resource could not be created or modified with the Request-URI, an appropriate error response SHOULD be given that reflects the nature of the problem. source

can I use PUT with server side versioning?

A single resource MAY be identified by many different URIs. For example, an article might have a URI for identifying "the current version" which is separate from the URI identifying each particular version ... a PUT request on a general URI might result in several other URIs being defined by the origin server. source

how would I let a client know I implement server side versioning when they PUT?

If an existing resource is modified, either the 200 (OK) or 204 (No Content) response codes SHOULD be sent.. source

200 indicates a message body in the response ;)

and DELETE?
well it's short so you may as well read it all..

The DELETE method requests that the origin server delete the resource identified by the Request-URI. This method MAY be overridden by human intervention (or other means) on the origin server. The client cannot be guaranteed that the operation has been carried out, even if the status code returned from the origin server indicates that the action has been completed successfully. However, the server SHOULD NOT indicate success unless, at the time the response is given, it intends to delete the resource or move it to an inaccessible location.

A successful response SHOULD be 200 (OK) if the response includes an entity describing the status, 202 (Accepted) if the action has not yet been enacted, or 204 (No Content) if the action has been enacted but the response does not include an entity. source

note that a response can be 200, meaning you can return a response message (like i have X other versions here [list] or delete them all by clicking here [form input which POSTs to a service] ), or an RDF response that can be interpreted by a client to do the aforementioned :)

but can't I tunnel all actions through GET?

Safe Method .. GET .. SHOULD NOT have the significance of taking an action other than retrieval! source

edit: removed small section about URI vs URL! Do however see the comment from Michael which links to more information on the subject.

Thanks for reading :)
There is much more information in the RFC, but those were some nicer points I found useful and relevant to current Web of Data topics & discussions.

Very brief intro: SPARQL is a fantastic query language which works over RDF and thus Linked Data, Virtuoso amongst other things has a powerful QuadStore which can be queried via SPARQL, and Virtuoso's implementation of SPARQL + the extensive suite of extensions they have implemented makes it the most usable and powerful query langauge available (again, in my honest opinion). In short this combination was enough to make me drop normal RDBMS systems and never look back.

Rather than rambling on about how fantastic it is though; here are some Virtuoso specific sample SPARQL (+GEO) queries, which should hopefully wet your appetite and give you some inclination of what can be done.

Basic Geo Lookups

Things within 20km of New York City : RESULTS
SELECT DISTINCT ?resource ?label ?location
WHERE
{
<http://dbpedia.org/resource/New_York_City> geo:geometry ?sourceGEO .
?resource geo:geometry ?location ; rdfs:label ?label .
FILTER( bif:st_intersects( ?location, ?sourceGEO, 20 ) ) .
FILTER( lang(?label) = "en" )
}

Distance between New York City and London, England : RESULTS
SELECT (bif:st_distance(?nyl,?ll)) as ?distanceBetweenNewYorkCityAndLondon
WHERE
{
<http://dbpedia.org/resource/New_York_City> geo:geometry ?nyl .
<http://dbpedia.org/resource/London> geo:geometry ?ll .
}


Querying Time and Space

All Educational Institutions within 10km of Oxford, UK; ordered by date of establishment : RESULTS
SELECT DISTINCT ?thing as ?uri ?thingLabel as ?name ?date as ?established ?matchGEO as ?location
WHERE
{
<http://dbpedia.org/resource/Oxford> geo:geometry ?sourceGEO .
?resource geo:geometry ?matchGEO .
FILTER( bif:st_intersects( ?matchGEO, ?sourceGEO, 5 ) ) .
?thing ?somelink ?resource ; <http://dbpedia.org/ontology/established> ?date ; rdfs:label ?thingLabel . FILTER( lang(?thingLabel) = "en" )
} ORDER BY asc( ?date )

Historical cross section of events related to Edinburgh and the surrounding area (within 30km) during the 19th century : RESULTS
SELECT DISTINCT ?thing ?thingLabel ?dateMeaningLabel ?date ?matchGEO WHERE {
{
SELECT DISTINCT ?thing ?matchGEO
WHERE
{
<http://dbpedia.org/resource/Edinburgh> geo:geometry ?sourceGEO .
?resource geo:geometry ?matchGEO .
FILTER( bif:st_intersects( ?matchGEO, ?sourceGEO, 30 ) ) .
?thing ?somelink ?resource
}
}
{?property rdf:type owl:DatatypeProperty ; rdfs:range xsd:date } .
?thing ?dateMeaning ?date . FILTER( ?dateMeaning in( ?property ) ) . FILTER( ?date >= xsd:gYear("1800") && ?date <= xsd:gYear("1900") )
?dateMeaning rdfs:label ?dateMeaningLabel . FILTER( lang(?dateMeaningLabel) = "en" ) .
?thing rdfs:label ?thingLabel . FILTER( lang(?thingLabel) = "en" )
} ORDER BY asc( ?date )

Transitivity and Inference (v5 compatible)

Finding the shortest route between two "things" (HTML and XML in the example) : RESULTS
SELECT ?route ?jump WHERE
{
{ SELECT ?x ?y WHERE { ?x foaf:page ?xpage ; ?predicate ?y . filter( isURI(?y) ) } }
OPTION ( TRANSITIVE, T_DISTINCT, T_SHORTEST_ONLY, t_in(?x), t_out(?y), t_max(10), t_step('path_id') as ?path, t_step(?x) as ?route, t_step('step_no') AS ?jump )
. FILTER ( ?y = <http://dbpedia.org/resource/HTML> && ?x = <http://dbpedia.org/resource/XML> )
}


..and all routes between the two "things" : RESULTS
SELECT ?route ?path ?jump WHERE
{
{ SELECT ?x ?y WHERE { ?x foaf:page ?xpage ; ?predicate ?y . filter( isURI(?y) ) } }
OPTION ( TRANSITIVE, T_NO_CYCLES, t_in(?x), t_out(?y), t_max(5), t_step('path_id') as ?path, t_step(?x) as ?route, t_step('step_no') AS ?jump )
. FILTER ( ?y = <http://dbpedia.org/resource/HTML> && ?x = <http://dbpedia.org/resource/XML> )
}

Traversing Ontologies and (Sub)Classes; all subclasses of Person down the hierarchy : RESULTS
SELECT DISTINCT ?x WHERE
{
{ SELECT ?x ?y WHERE { ?x rdfs:subClassOf ?y } }
OPTION ( TRANSITIVE, T_DISTINCT, t_in(?x), t_out(?y), t_step('path_id') as ?path, t_step(?x) as ?route, t_step('step_no') AS ?jump, T_DIRECTION 2 )
FILTER ( ?y = <http://dbpedia.org/ontology/Person> )
}

Free text search, scores and IRI Ranks (v5 compatible)

Searching over labels, with text match scores and additional ranks for each iri / resource : RESULTS
SELECT ?s ?page ?label ?textScore ((?s)) as ?iriRank WHERE {
?s foaf:page ?page ; rdfs:label ?label . FILTER( lang(?label) = "en" ) .
?label bif:contains 'adobe and flash' option (score ?textScore ) .
}

Virtuoso 6.1 (Open Source Edition) released. For features & bug fix details see: link

Sounds melodramatic but the whole point of this text is to explain in real terms the effect it'll have on the every day web worker; the web developer, web designer, seo expert, internet marketer etc. So that you, my current or future friends and associates still have a job in a couple  of years; and I researched it so that I would still have a job in a few years (+ because I love this stuff!)

A bit about Linked Open Data (LOD).

LOD can easily be a huge, scary and new thing, overwhelming in so many ways with all this talk of a cloud, billions of bits of information in some part of the web that is separate to "us"; take one look at the diagrams of the linked open data cloud and you'll see those academic acronyms of scientific organisations, future thinking global entities publishing their specialised data - nothing about you and me with our little blogs, and moreover nothing about our clients websites.

Sure it's about getting massive amounts of data on the web, linked and open for use, but it's different to how you expect :)

Linked Open Data is simply about making the info we already put on the net (like this post) machine readable as well as human readable.

It *IS NOT* about creating some system to dump everything from our database in some weird format for a machine to read somewhere.

It *IS* about wrapping the data on a normal page in a bit of markup so that a computer knows what it is.

If you're writing about london you simply add a tiny bit of markup that says 'about="http://en.wikipedia.org/wiki/London"' - honestly that's it in real terms, the user reading your page knows its about London, England - and now a system like google knows that it's definitely about London, England too. In most cases though it's simpler; it's a case of saying this article is titled "x" and made by person "y" - that alone makes a huge difference to the net.

How LOD will change the web.

More Links! We're currently in the age of search, if you want something you search for it, to get more info you search again, and again and so on.

Link trust was at an all time low a few years ago, sure you'd click a navigation link on a site but not a link in a document, because it was probably to a popup, an advert, something you didn't want. Not now though, mainly thanks to bloggers with their in text links to other pages, the world has grown to trust the link again.

Linked Open Data will spawn a massive increase in related data on page, related resources, articles, images, videos and more. And thus many, many more links.

This means that people will search less, and explore more; ever increasingly.

It's unavoidable, and even if a website isn't enhanced with all this extra linked data, odds are the user will have a browser extension or app running that will show all the related information anyway - these technologies are already here and used - adoption *will* grow, no way out of it, change happens.

Info for specific sectors

This isn't the meant to be a full introduction or all encompassing, in fact nowhere near it - if you want the ins and outs of LOD then look elsewhere. This info is for the everyday Web Developer, Web Designer and SEO Specialist.

Web Designers (+ those who work with html)

To be honest I think this change might hit you guy's hardest; you see XHTML+RDFa is already here, it'll be massive soon (and don't go thinking HTML5 will get you out of it, RDFa will be in there too). In short XHTML+RDFa is xhtml as you know it, but with support for embedded RDF information, really it means a few new properties on elements that let you say what they are; in place FOAF, Dublin Core (DC) and the like. Any further description is outside the scope of this document ;)

What this means for you is that as well as having potentially a lot more to display on page (linked data) and lot's of UI challenges, you also now have to cater for this RDFa data in your templates. It's not like other W3C stuff which you can ignore, different to cross browser compatibility, if you leave it out or skip the RDFa stuff then the site will potentially be outside of the LOD network, traffic will drop and ultimately the site may as well not be "in" the web (might be a few years before that though) - so in many ways the end of ignorance and excuses.

You can currently slap out some HTML4, change the doctype, stick on jquery and make it "look" web 2.0 - and people will think it's web 2.0; with web 3.0 (the linked open data based net) you can't do that, it either is web 3.0 or isn't; there isn't a "web 3.0" look, just web 3.0 source.

Drupal 7 has RDFa support out of the box; within the year I bet every CMS & Blog will too; and if you make a new template with the RDFa cut out because you "don't know it", then I'm pretty sure it won't be long before your clients or employers cut you out; and we don't want that.

Further, if you don't - developers will be on your back big time & changing your source; or worse the SEO guys will be ;)

Web Developers
All of you need to know what triples are (subject-predicate-object), and URIs and CURIEs (not your normal URIs, URIs as Identifiers).

If you're going to be exposing data in your systems then you need to get used to mapping database properties through to RDF triples; that a user is a foaf:person with a foaf:name; that tags are ctags and dc:subjects, and that articles have a dc:title (keeping it simple for this).

If you're going to be consuming LOD data then you need to learn a bit more, RDF, SPARQL, Owl, ontologies and a bit more.

And if you want to get "in to" LOD in a big way, then go do it.

SEO Specialists
You need to know what the designers know, and you'll be changing from SEO specialists to data exploration optimizers or suchlike, focus will be on how you can make the data machine readable and get it linked in by the right services.. should be fun!

Further, you'll need to watch for how to get traffic to the sites, as mentioned search engine traffic will drop slowly over the next few months and years; with more focus going on "links" from related pages. As for the diggs & reddits, who knows how it'll effect traffic from them.

Summary

IMHO it's in all of our best interests to just get on with this, it will happen and the sooner YOU do it and convince your employers you have to make this move the better, companies can easily loose clients too if your competition is offering "web 3" and you aren't.

At no point have I seen a tech hit the web which could literally leave people behind if they don't jump on board; it's happened in other industries and now ours (remember VHS?).

The two questions most people / companies / clients will immediately raise..

1] We don't want to expose all our data for reasons X,Y&Z!

LOD isn't about exposing all the your data on the internet; it's about making the data you've already exposed on the internet in a more granular fashion, it's about making that data machine readable.

Presently you may have an article on a page with a title and author credit in HTML, in the future you would still have the same author and title, however they would be wrapped in markup that allows a machine to understand that "Joe Blogs" is a person who is the author of the article, and that the articles title is "I'm scared of exposing my data".

If you consider you're public facing web pages, everything on that page is already exposed, all we're doing here is describing what each bit of data is in a way we can all use.

2] Trust & Junk

One common misconception is that you have no control over the source of the data you pull from the "cloud", and that it could essentially be junk. However this couldn't be further from the truth, what we do is to find a source of data we trust that has their data exposed in a machine readable format, then query it for the exact information we want, and finally include or display it in our own system.

To illustrate, consider you wanted to reference the countries of the world with population in your system. Currently you would have to build  a database table, populate that data with country name and country population, then write some code to display that data. In this scenario you'd probably get the population data from a credible source such as wikipedia (copy and paste it in to your own database).

By using linked open data, you could treat the machine readable version of wikipedia (dbpedia) as your database table, query it instead and again write some code to display the data on you're own site.

You're displaying the same data, from the same trusted source; and you've selected which source you trust; it's not a case of just querying some cloud of data; it's a case of choosing which source(s) you want / trust and querying them.

As an additional bonus you don't need to worry about your information going out of date, as you're getting the data straight from source, the population of each country is updated on your site whenever it's updated on wikipedia.

Further, you don't need to worry about maintaining that list of countries, as in a single query you can pull out a list of all countries with each ones population, as the world grows and changes, so does your data.

Further still! once you've made the move to using some linked open data, all the data you could want is at your finger tips, let's say a decision is made to include 30 different bits of information about each country in your system. Consider that task for a minute - full system change, finding, collating and entering all that data; let alone maintaining it! Well, I'm sure you can guess the next bit, using LOD we can simply expand our original query to include the other bits of information we want, then display it - job done.

That's it.

Good Luck!

nathan

To me, the draw of linked data and the semantic web has always been exploration; the notion that even the most unskilled of publishers should be able to enrich their content via semi-automated software to the standards of a near perfect wikipedia article has always been the driving force. Additionally, content classification, relation, linkage, data centralization and the like are all major benefits which will make a vast difference to the usability of the web.

Search will always be a major part of the internet, at the moment we use search to find content on a specific subject, then search again to find more, and search again to find related or expanded info, help, facts, answers, whatever; however, in the future I hope to see search move to a less prominent role, one where we use search to find the most suitable "entry point" in to the web of linked data - and from there every other piece of related / expanded information is either on page, or a click (link) away.

Some major hurdles need to be jumped before we can get to that stage though, both through lack of organization and lack of appropriate software. Personally I have a mental blueprint / overview of what's needed (imho), and some very specific ideas on the software side, with any luck I'll get a chance to contribute + build some of this, we'll see.

Some thoughts of what's needed from my little brain.

Linked Data Ping
A central service API which is pinged by all software as it publishes information with machine accessible content. (Needed way before (x)HTML+RDFa takes off). Provides a stream of all recent pings to be consumed (xmpp pub-sub?).

Clustered Servers holding a centralized data GUID lookup and proxy.
In essence all resources on the net should be a linked pair of GUID to endpoint, each endpoint should contain a reference to the GUID, and each GUID should be a URI which redirects to the endpoint, endpoints change GUID/URI stays unique. In an ideal scenario when somebody creates a link to X resource or Y document, the publishing/controlling software should replace the endpoint with the GUID instead. This would also enable multiple other services such as centralized pingback, references, statistics etc.

Machine Readable Data Cache.
Together with the aforementioned services a high availability database of cache'd information should exist; in principal this would work by reading the stream of "Linked Data Pings", getting the GUID for the content and then retrieving all machine readable data and caching it. Much like the RDF data exposed through dbpedia, however for everything. Even if only a predefined subset of the common rdf vocabularies was stored and exposed it'd be enough to start, from there all other domain specific ontology could be retrieved by reading the endpoint itself.

Semantic CMS
Ideally we need a new breed of CMS, one that not only has simple FOAF and Dublin Core (~Drupal 7), but also support for full content enrichment using the aforementioned machine services; and provides a simple UI for manually exposing entities, events, facts etc. (Think highlight name in text, mark as Person with Name, system finds guid and builds relevant RDFa and we have another triple of linked data.)

The possibilities from this point are endless; if you're reading this document after all this has been made, then you'd see a whole host of in text links through to more information on each keyphrase, person, entity etc; you'd be aided by auto injection of sources, related reading, comments, further documents discussing the content here, in short you'd be exploring the net one click at a time, linked data all the way; not searching.

In summary (and very much imho), Linked Data is not for searching, it's for linking data - search was invented to address the issue that everything isn't linked, when it is then the link takes precedence again.

My only worry in all of this, is the idea that all rdf triples are fact, and true - already the major search engines are exposing rdfa data in summaries, 5* ratings on products and suchlike, the room for abuse will only get worse.

Thanks Georgi for placing the spark that clarified my current thoughts.

Finally, this isn't a biased opinion in anyway, or an endorsement, but to me openlink virtuoso, dbpedia, zemanta and open calais are leading the way and enabling all of this; together with the hard working folks contributing to the various linked W3C projects and specs. If only dbpedia/zemanta/calais would unify there uri's/guids/endpoints we'd be a lot further along.. (well I would ;).

Regards!