Why? well the sem web community is packed full of the most technically skilled and decent people I've come across so far, so it can't be that, the tooling is pretty damn good, there's loads of data, most of the data's of a high quality, wanted by developers, and certainly more than usable. The concepts, theory and technical aspects are all solid as a rock. In short it's all good, apart from one rather important detail.

Our Linked Open Data isn't really open data, not in the eyes of the common web developer at least. To most web developers on the planet, open data is something they can get access to and use easily.

A good example of 'open data' for most developers, is the Twitter API.

Here's how a developer accesses it in PHP:

  $uri = 'http://api.twitter.com/1/statuses/show/7907258268647424.json';
  $tweet = json_decode(file_get_contents($uri));
  echo $tweet->user->description;

Here's how they access it in Javascript:

  uri = 'http://api.twitter.com/1/statuses/show/7907258268647424.json';
  $.getJSON(uri, function(tweet) {
    write( tweet.user.description );
  });

The reality of the matter is that you can't do this with Linked Open Data, and that's because you can't do it with RDF - and really, honestly, if it's not that simple, the masses won't use it, because, if it's not that simple, they can't use it.

The problems with the RDF formats

They're not perfect, and they are a very mixed bunch, for a change, let's look at the negatives.

RDF/XML
- Requires full XML tooling
- Can't read or write by hand
- butt ugly

Let's be honest, unless you have a full XML and RDF stack and you know what you're doing, RDF/XML is simply a no go zone.

RDFa
- Requires HTML/DOM/XML tooling
- An extension to a markup language, designed to augment annotated documents.

RDFa is great, but not for the general data use case, it's not a simple data interchange format like JSON, and you can't publish or consume it without specialist tooling, in fact it requires an even bigger, more complicated, stack than RDF/XML.

Turtle
- Requires a custom parser
- It's not yet seen as a data format by the masses.
- Doesn't have a registered media type.

If we're all honest, Turtle is the king of RDF serializations, it's small, powerful, easy to write and read, requires minimal tooling to parse. In fact, I'd quite happily say that Turtle is the best all round data format, period.

RDF/JSON and JSON-LD
- Square peg, round hole

You can't shoe horn RDF in to JSON, no matter how hard you try - well, you can, but you loose all the benefits of JSON in the first place, because the data is RDF, triples and not objects, rdf nodes and not simple values - I fear I'd better explain, quickly:

The benefit of JSON is that you can do the following:

  var u = tweet.user;                                // nested, simple objects
  write(u.message);                                  // simply "a string of text"
  if(u.geo_enabled) {                                // a boolean true
    var d = u.statuses_count * u.favourites_count;   // numbers..
  }

Anything more complicated than that and you've lost 95% of the benefits of JSON.

Here's the code to do that if(u.geo_enabled) with JSON/RDF:

  var tweet = rdf['http://example.org/tweet/12343'];
  var user = rdf[tweet['http://example.org/property/userid']];
  var geoenabled = user['http://example.org/property/geo_enabled'];
  if( Boolean(geoenabled.value) ) {

and JSON-LD:

  var tweet, user;
  for(o in jsonld) {
    if(jsonld[o]["@"] == 'http://example.org/tweet/12343') tweet = jsonld[o];
  }
  for(o in jsonld) {
    if(jsonld[o]["@"] == tweet['twit:userid']) user = jsonld[o];
  }
  if(user['twit:geo_enabled']) {

Remember, those two examples are only for the simple if line from the benefits of JSON example, can you even imagine all four lines?

Clearly, RDF in JSON is of little to no use to anybody, you can see plainly yourself, 95% of the benefits are lost and it's just another RDF serialization that's pretty much unusable without tooling. The only benefit JSON serializations of RDF have, are that you don't require an XML stack, which is quite a large benefit tbh.

The problem with RDF

I've been a little unfair there, you see the problem isn't with the serializations, we can't make a "better" serialization of RDF for general web developers, because the real problem is that the data's RDF, it's triples not simple objects, URIs rather than simple terms, RDF Nodes with a language or type, and not just simple values.

An array of RDF triples, or a structure of RDF, just simply isn't usable in most (all?) programming languages, by a typical web developer, without specialist tooling and libraries or APIs.

Am I saying RDF is bad? no of course not, it's awesomely brilliant in every way, it powers a paradigm shift and will have huge positive effects on the web and the human race. You know the score :)

What I am saying, is that we're not backwards compatible, we're not making our data open in formats which are usable by normal developers, developers who need and want the data, want the links, but not the semwebbery. Hell even most of us who are heavy sem web users only consider the ontology+reasoning side enabled by properties-with-uris some of the time, most of the time <http://xmlns.com/foaf/0.1/name> might as well just be "name".

Opening Linked Data

So, here's what we need to do, we need to just accept that although we publish linked data as RDF, we also need to publish the data as simple objects so the world can use the data.

Given the linked data:

  @prefix : <http://webr3.org/nathan#> .
  @prefix foaf: <http://xmlns.com/foaf/0.1/> .
  @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .

  :me a foaf:Person;
    foaf:age 29;
    foaf:holdsAccount [ foaf:accountName "webr3";
        foaf:homepage <http://twitter.com/webr3>;
        rdfs:label "Nathan's twitter account"@en ];
    foaf:homepage <http://webr3.org>;
    foaf:knows <http://example.com/bob#me>;
    foaf:name "Nathan";
    foaf:nick "webr3", "nath" .

  <http://example.com/bob#me> a foaf:Person;
    foaf:name "Bob" .

We also need to publish it like this:

{
  "http://webr3.org/nathan#me": {
    "a": "http://xmlns.com/foaf/0.1/Person",
    "age": 29,
    "holdsAccount": {
      "accountName": "webr3",
      "homepage": "http://twitter.com/webr3",
      "label": "Nathan's twitter account"
    },
    "homepage": "http://webr3.org",
    "knows": "http://example.com/bob#me",
    "name": "Nathan",
    "nick": [ "webr3", "nath" ]
  },
  "http://example.com/bob#me": {
    "a": "http://xmlns.com/foaf/0.1/Person",
    "name": "Bob"
  }
}

and now one can do: me.holdsAccount.label and get back a string, in any language.

What have we lost?
Well nothing in the grand scheme of things because we're still publishing the RDF in other formats via conneg, however in this specific serialization of the data: we've lost the properties, the .language and the .type, although basic types are still there, dates are detectable, numbers are supported, booleans are supported, everything else is just a PlainLiteral, a string.

What's still there?
The data, http names for things, follow your nose, rdf types ++usable-accessible-data in a web standard format. It's 3.5 if not 4.5 star data!

Other considerations
Probably be wise to allow direct access to a .json URI too, so people can simple-GET the data (in addition to exposing via conneg).

Should be an easy hit, any good reasons why not?

The other RDF Serializations

While we're here, there are a few other things that need tidied up, properly.

Turtle
Let's make it the standard RDF serialization, with a proper registered media types of application/turtle and text/turtle, fix any bugs in the spec (if any), and possibly allow an optional comma in those lists (1,2,3).

Let's just accept that RDFa is great, but sometimes you just want to embed a chunk of RDF in an HTML document, you know we all want to on occasion, people are shouting for it, it's easy to do, to deploy, doesn't break anything, and it's a really easy hit - so, in the Turtle standard spec pop a note that shows how to include it in an HTML document.

<script type="text/turtle">
  // turtle in here, as-is, no special encoding
</script>

RDF/XML
Leave it as is, let it run it's course naturally, and in many respects just forget it moving forwards, everybody supports it already, that won't change, but there's approximately zero need to keep pushing it.

RDFa
As awesome as it is, just see it for what it is, RDF in attributes, it's like the missing markup features of HTML, for annotating documents and describing the things described in the documents with annotations. It's not a simple data or RDF format, and it's not really suited to just dropping chunks of machine readable RDF in to an HTML document, Turtle in HTML will do that far better.

RDF/JSON and JSON-LD
Standardize a JSON serialization as a replacement / alternative to XML - but, admit and stipulate before hand that it's not usable as-is, or really writable and is arguably human readable. It simply needs to be a fast, unambiguous, optimized for the machine, RDF serialization - no bells and whistles for humans, no "12^^xsd:type" in a string - just something that you can JSON.parse and run circa 10-20 standardized lines of code over to get back an RDFGraph of RDFTriples.

Fin & end.

v8 and server side ecma-262 (javascript) is a fantastic thought, and definitely the future imho (universal language).

To get going and start making server side js web applications there are two good approaches, v8cgi which aims more at letting you run js programs as a cgi app (like you do with php cgi) under apache and similar, seems to be picking up speed, nice approach (workable) - the other is node.js which is off to a great start as a pure server side environment where 'HTTP is a first class protocol' which just proves what a good start Ryan Dahl has made.

Briefly to touch on v8, it allows you expose C code to javascript, which is.. well perfect, would be good to see far more lib's exposed to js using tools like cproxyv8 and v8-juice - in particular I'm looking at you librdf (and an interesting sub-thought, apache and openssl).

Back to node.js, seems to be exciting already has a strong API to get some great demo's and projects off the ground, and very well aligned with the W3C work (or maybe what-wg..), specifically on the Web Worker API front; in fact there are already implementations of Web Sockets for node.js (and socket.io) - nodelint (jslint for node) is a handy addition, whilst some projects seem to be going all in, like Persevere.

As for adapting current code to work on the server side, you'll want to be looking in to the modules .. which.. well to be honest that's a new link I just found, and not what I was thinking off.. emm, I think that provides almost everything I need - might be about to jump back in with two feet!

The rest of this post was about to mention php.js, harmony framework (Converts PHP 5 code into Javascript), and then mascara and of course haxe which compiles to js amongst many other targets.

Might have to come back to this post.. for anybody who follow what I do, thus far I'd figured out that node.js is http+tls friendly, you can get a clients certificate (for foaf+ssl) and you can send application webid+ssl certs too, although up till now I hadn't found a way to generate certificate (let alone using spcak) but that could be wrong..

I've grouped all the links in to two main sections, and then sub-grouped by how they make sense in my head! :)

Web, HTTP and REST

Roy T. Fielding Dissertation - Architectural Styles and the Design of Network-based Software Architectures Of particular relevance and note are chapters 4-6 (many only ever read chapter 5 and miss the context + summary *needed* in chapters 4 and 6!)

• Chapter 4 - Designing the Web Architecture: Problems and Insights
• Chapter 5 - Representational State Transfer (REST)
• Chapter 6 - Experience and Evaluation

Roy T. Fielding - REST APIs must be hypertext-driven
Discussion on HTML5 and RESTful HTTP in browsers
Discussion on URIs Resources and Switching content types w/ REST angle (v good)

RFC 2616 HTTP/1.1 and the HTTPbis Working Group HTTP/1.1 update in parts:

1. Messaging
2. Semantics
3. Payload
4. Conditional
5. Range
6. Cache
7. Authentication

Linked Data and the Semantic Web

Linking Open Data Community Project
Linked Data Applications
Equivalence Mining and Matching Frameworks
Linked Data Browsers, Mashups and other Client Applications

Dataset Dynamics - On the Dynamics of Linked Datasets
Realizing a write-enabled Web of Data
Web Access Control (WAC) - a decentralized system for allowing different users and groups various forms of access to resources where users and groups are identified by HTTP URIs.
Discussion of the WAC vocabulary
Socially Aware Cloud Storage Design Note
Distributed Social Networking through Socially Aware Cloud Storage from TimBL
AWWSW - "Architecture of the World Wide Semantic Web" Task Force

SPARQL 1.1 Uniform HTTP Protocol for Managing RDF Graphs
A Linked Data Frontend for SPARQL Endpoints
RAP - RDF API for PHP V0.9.6
Inav the Terrible - An idea for posting RDF through HTTP.

Talis Changesets
Triplify Update Vocabulary

skos as atom

RFC 4287 - The Atom Syndication Format
RFC 5023 - The Atom Publishing Protocol
AtomPub Tombstones - The Atom "deleted-entry" Element
RFC 5005 - Feed Paging and Archiving
Versioning Link Relations - Link Relation Types for Simple Version Navigation between Web Resources

Named Graphs, Provenance and Trust
Accessing Site-Specific APIs Through Write-Wrappers From The Web of Data
Provenance Information in the Web of Data - LDOW 2009 paper
Using Reification To Extend RDF (historical reification approach)
RDF Policy-based URI Access Control for Content Authoring
The Open Provenance Model Core Specification (v1.1)
W3C Provenance Incubator Group

History of the Web 1945, 1980 through 1997 on W3
LEIRI - Legacy extended IRIs for XML resource identification The type of "URI" used in xml:base
CSHALS 2010 W3C Semanic Web Tutorial
Mindswap online RDF Converter
W3 online RDF Validator

Something all of us developers realise but it doesn't always smack home until certain scenarios arise - The Wall (as I'll call it for now) is the virtual line behind the UI which is where the developer / client separation of concerns occurs. The client wants to see one thing on a UI, click a few buttons, hit a few keys and see the UI change to another thing, this change indicates that we as programmers have done our job, behind that line is generally our concern.

Here's the scenario.. a few weeks back "we" started a full 3 tier app, the Applcation tier consisted of the three big layers, service layer -> domain model layer -> data source layer; everything implemented by the book with transfer objects, DAOs, business objects, a full language extension lib, utils, everything coded to interfaces and using factories.. you get the picture.. a lot of code. On the presentation tier we had single UI consisting of a full MVC framework with all the magic and goodness, the models of which talked to the service layer of the application tier (as it should).

A couple of hundred hours later we get our first full end to end done, a simple FAQ system.. possibly the most overkilled FAQ system ever, but still a great end to end test. This is where the wall becomes clear, because for all of that the client only saw a simple FAQ system.

At this point, as a developer you have to ask yourself why.. the FAQ system could have been developed in circa 1 hour using a few lines of PHP and simple mysql database + some echo's.

I don't have an answer to this, and don't think there is one... because if you think about all the immediate arguments for doing it the "proper" way.. maybe even the java way - you still have to think, what is more maintainable a few hundred files and a complex architecture full of design patterns, or three lines of PHP a "n00b" could maintain.

More over.. ultimately those few lines of code you could make the application with are pretty much the same few lines of code which aren't reusable when using the full application approach.

Finally you ask yourself.. if a good programmer introduces about 2.5 bugs per 100 lines.. then what requires more long term maintenance; 100 lines for a full FAQ system or 100 classes with 100 lines each.

Just a thought, that won't go anywhere - because i get paid to make these big applications, but when the pressures on and I've got a 3 hour deadline I do the 100 line thing.

no response needed, simply questioning oneself (again)

from the documentation: "This function returns a unique identifier for the object. This id can be used as a hash key for storing objects or for identifying an object."

spl_object_hash() actually returns an md5 hash of the internal pointer of an object.

PHP efficiently reuses internal pointers; meaning the second you remove an object the pointer is reused.

thus:

<?php
$transaction_ids = array();
for( $i=0; $i<10; $i++ ) {
 $transaction = (object)('transaction ' . $i);
 $hash = spl_object_hash( $transaction );
 $transaction_ids[] = $hash;
}
print_r( $transaction_ids );
?>

will output:

Array
(
[0] => 4b8051a089b9e8352a97af234d4478a7
[1] => 076b44c4ffe009b98c0ba3348459c57a
[2] => 4b8051a089b9e8352a97af234d4478a7
[3] => 076b44c4ffe009b98c0ba3348459c57a
[4] => 4b8051a089b9e8352a97af234d4478a7
[5] => 076b44c4ffe009b98c0ba3348459c57a
[6] => 4b8051a089b9e8352a97af234d4478a7
[7] => 076b44c4ffe009b98c0ba3348459c57a
[8] => 4b8051a089b9e8352a97af234d4478a7
[9] => 076b44c4ffe009b98c0ba3348459c57a
)

not even remotely unique, could lead to serious problems, and simply.. I'd forget using it if I was you.

ObjectConverter - one of those classes you won't see the point of until you hit the problem.

Problem: When you convert an object (instance of a class) to an array in PHP, only the public properties are included; often you need public, protected, private and inherited properties to be visible, indeed there isn't any real way to expose all properties of an object without adding a "toArray" method to class itself; simply there is no simple way to do this (private properties are private for a reason!).

Solution: ObjectConverter :) This is a utility class which will turn any given instance in to either an array or a bare stdClass instance; effectively exposing all properties so you can persist or transfer your objects easily.

Note: afaik no other "object to array" script can do this, and after playing with multiple approaches over the years I finally settled on this implementation which does *not* use reflection or any such heavy weight trickery.

Usage:
$array = ObjectConverter::toArray( $obj );
$stdClass = ObjectConverter::toStdClass( $obj );

Download: ObjectConverter PHP Script (php5 only)

The Point - Events are useful everywhere even when you don't have a UI, the main reason for building this was to create a kind of "hook" system for php. Often we create multi site systems where each implementation of that system has different client specific considerations; consider a system where one client wants all of there images to go to amazon s3 instead of being saved on the host like your other clients. With this system you can simply:

1. throw an "storeUpload" event which is cancelable
2. register a default event listener which listens for "storeUpload" events and provides default functionality, saving the upload and setting the location (ur) on the event.target
3. on one site register a custom event listener which listens for the "storeUpload" event, uploads to amazon s3, then cancels the event and set the event.target to the s3 url instead

as you can see, with this you can keep the codebase of your multisite application the same and simply hook in to events whereever you want.

the system consists of 2 public classes:

Event

The base class for all events, provides event type and cancelable; with typical usage you create a new event, then dispatch it when ready.

• __construct( $type , $cancelable=FALSE )
  create a new event listener - if you extend the Event class you must call this with parent::__construct( $type , $cancelable );

  $type - string, event type eg "debugEvent"
  $cancelable - boolean, provides cancelation functionality, so you can prevent further event listeners from firing thus in effect overriding them.

  throws InvalidArgumentException if $type is not a string

• cancel() : void
  cancels an event, only if it is cancelable
• isCancelable() : Boolean
• isCanceled() : Boolean
• getType() : string
• getTarget() : obj
  target always contains the instance which dispatched the event.

EventDispatcher

All classes which wish to dispatch events must inherit from EventDispatcher (like in most languages), this class provides:

• addEventListener( $type , $callable , $owner ) : void
  adds an event listener, this may be a function, a class method or a static class method.
  note: a current limitation is that event listener methods must be public

  $type - string, event type eg "debugEvent"
  $callable - string, the function or method which is to be called when event $type is dispatched
  $owner - mixed, used when $callable is a method, can be either an object instance, or the string name of a class for static methods.

  throws:
  InvalidArgumentException when type or callable isn't a string
  Exception('unknown method / function')

• removeEventListener( $type , $callable , $owner ) : void
  removes an event listener that has been added with addEventListener, this only removes a single event listener each call, so if you have 3 identical event listeners you will need to call this method 3 times to remove them all.

  arguments are the same as addEventListener - throws no exceptions, always returns void

• hasEventListener( $type ) : Boolean
  checks if any event listeners are registered for event $type; always returns boolean true or false.
• dispatchEvent( Event $event ) : void
  the obvious one, dispatches an event and notifies all listeners.

Notes:

The event dispatcher implements a function which hides all instance properties starting with __ from the serialize process, thus hiding the events and meaning your current serialization process will still work, additionally you can utilize it to hide any properties you don't want to be serialized.

I've tried to make the classes as simple, small and practical as possible, they execute rather quickly and shouldn't make any speed difference to your application - do however be aware that loosely coupling and entire system could allow coding errors to creep in, so maybe don't use it everywhere - but then PHP isn't strict anyways so..!

Download

You can download the PHP Event system here (3 classes + example, 4.2k) - hope somebody finds it useful!

Skippable Details

All this traffic is round robin'd to 2 apache2 servers (each is 2x 2.86 quad xeons w/ 4GB ram) and one backend db server of similar spec.

Over the years familiar problems rear there heads, the mysql "Too Many Connections" error, Apache running like a dog under stress etc.

This weekend I got the chance to rebuild the servers from scratch, and a chance to optimize everything - works very very well now!

Easy Optimization - Apache2

KeepAliveTimeout: Number of seconds to wait for the next request from the same client on the same connection.

Translate this to "Number of seconds to lock up a worker thread just incase another request comes in from the same client on the same connection" - in short a lot of browsers will request the page, keep the connection alive and use it to request images, css etc.

The default is 15 seconds; in my experience about 40-50% of connections will use KeepAlive, so if you have enough ram to support 500 requests before going in to swap memory, and you get 500 requests a second, by the time 3 seconds are past you've got a load of workers with RAM assigned doing nothing and all the hard work is getting done using swap memory instead, from this poitn on basically all the hard work is getting done using swap rather than ram, ya?

Thus, the simple fix is to knock KeepAliveTimeout right down - some people recommend 2 seconds, my personal preference is 4 seconds - works a charm for me :)

To analyse this yourself just check the server-status provided by apache (normally only viewable from localhost, easy way around this is to <?php echo file_get_contents('http://localhost/server-status'); ?>

Easy Optimization - MySQL Server

Other than the normal cnf values you should be optimizing [ key_buffer, sort_buffer_size, max_allowed_packet, thread_stack, thread_cache_size, max_connections, table_cache, query_cache_limit, query_cache_size ] - there is one more that is very important; namely wait_timeout.

wait_timeout - The number of seconds the server waits for activity on a noninteractive connection before closing it.

By default this value is very high, but often you get connections that just aren't closed properly, especially in the PHP apache world - if you ever hit that "Too many connections" error or see a lot of connections open for prolonged periods of time then this could / should be addressed.

Often you will see connections sitting idle for 10-15 seconds, for no good reason. Best to address it I'd say; and thus we lower wait_timeout to 5 seconds.

Connections are cleared faster, more resources free for other connections and all round its good. Remember one of mysql's strongest features is it's ability to open and close connections ultra-fast.

Summary

Honestly just check the your apache server status, see how many connections are open, how many of them are flagged K for keep alive then drop the keep alive time, take a note of processor usage / memory usage, drop the KeepAliveTimeout to 2, 4, 6, 8 and check it all again, you may be very suprised, it sure beats weeks of manual code optimizatios to make those scripts run faster.

Regards

Seeing as most of the audience of this blog will be using more recent browsers, I've opted for XSLT templates and small XML descriptors for each item - this saves me writing tonnes of html, saves the server some weight, saves you some bandwidth and let's me palm off all rendering on to the client.

Inkeeping with the spirit of experiments, one of the main reasons I've done this is to see how the xml documents get listed in google (if at all); and to check if analytics works with XSLT.

On this note, I had to do a little bit of fiddling with the recommended embed code from google to get it to work; if anybody else hits this problem here's a modified version.


<script src="http://www.google-analytics.com/ga.js" type="text/javascript"></script>
<script type="text/javascript">
try {
var pageTracker = _gat._getTracker("UA-3391780-6");
pageTracker._trackPageview();
} catch(err) {}
</script>


And as for the listings, well thats just a quick and dirty PHP 5 class which iterates the directories, loads them in to a dom document and spits out the xml - ultra simples.

Ralph Hauwert somehow managed to push 300k+ 3D particles around in realtime using alchemy, pixelbender and flash 10; I played around with the code a bit and made similar in pure as3, nothing magnificent.. a month later in a totally unrelated event (namely me getting thoroughly p'd off with PHP) I discovered haXe and set to work playing around with it.

Finally in another totally unrelated event I came across Joa Eberts AS3 only version of Ralphs earlier work; upon closer inspection I found it wasn't as fast as Ralph's alchemy version but completely awesome code non-the-less, to do this in pure AS3 is simply amazing.

As if you can't guess where this is going.. I thought I'd give it a go in "pure haXe" to see if one could replicate the speed of Ralphs version or maybe even top it. Sparing any further detail..

Here's haXe pushing 300k+ 3d particles around in realtime (source included)

and...

here's haXe pushing 600k particles (twice as many), and erm 1 Million (a bit stuttery but hell why not)

and.. (just for fun)

here's haXe pushing 300k particles in color, and 500k in color for good measure! - before downplaying the color versions, it's worth noting that each r,g,b value is independantly calculated and weighted as a float then combined to a render color right before updating the bitmap, so in effect it's not just x,y,z stored for each pixel, it's r,g,b as well - so 2x more number crunching + a bit more.

think I can safely say I love haXe and since doign this I've been going nuts doing lots of experiments using similar code :p

so, any doublts about haXe's perfomance with flash can be considered resolved imho.

regards, nathan