Its the end of the world as we know it (and I feel fine)
Today is the last day of trading for Dominux Consulting. Finished, done, shop closed, shutters down.
I started with Notes back in 1995 age 20 at my first job. I was doing general IT support and one day I was given a box with some floppy disks in it (Notes 4.0 I think) and told to install this thing on a server, roll it out to 80 people while relocating the company to a new office 40 miles away, the final instruction was “When you have done that, tell us what it is”. Very quickly I realised that it was not just an email platform. I had it doing web browsing (we didn’t have a TCP/IP network so it was a great way to get web pages over a Netware IPX network) and NNTP news fairly quickly and then the company bought in a Quality Management System, which we customised quite heavily. After a while I figured that programming was much more fun than answering the phone (I am still pretty dreadful at answering the phone) so moved on to work full time as a Notes programmer, I then spent 15 years working for various companies including Dominux Consulting, my own little one man band. That brings us to today, in 2010 and the end of the line. I have now dissolved Dominux and I am now working full time on Free and Open Source software at The Open Learning Centre. The work is very much like it always was, writing great applications, on a solid and professional platform to solve business problems. The platform has changed somewhat though, I now use Alfresco for document management, Joomla! for web content management, Wordpress for blogs and blog-like websites including the very cool votegeek.org.uk with all these running on the Free and Open Source Ubuntu operating system. I think in future I will end up doing more work on applications based on CouchDB which is installed on Ubuntu by default and rapidly becoming the preferred back end for opportunistic application development, the type of little application that Notes gets used for. I find it interesting that for many years I have been told I use the one crazy database in the world that doesn’t even support SQL, but we now have a crop of NoSQL databases with a reputation for simplicity and scalability. The “Not a Real Database” marketing problem has been solved in style by the likes of CouchDB, but Notes and Domino are just acknowledged as something people used to use that has a NoSQL architecture. Notes might not be dead yet and this might not be the year of the Linux desktop, but both are coming close, it is pretty hard to be certain about either.
The campaign to not bother to Open Source Domino
A chap called Ian Tree of Hadleigh Marshall recently wrote an open letter to Steve Mills who is head of IBM software group, suggesting that IBM should “Open Source” Domino. This has started a lot of people talking about the possibility or otherwise of this proposal. I am a big Free and Open Source software advocate, but I don’t think this is a particularly useful suggestion for a number of reasons.
Hard to do
Domino is old, with bits of code licensed from all over the place. The provenance of various bits of code might be hard to establish and the right to change the licensing rules over bits of the code may be unclear.
Nobody really wants to extend it
Domino is already quite extensible. The C API is published and there are stacks of extension points. If you set up your build environment right with the correct elderly version of a C compiler you can write reams of impenetrable C code to do things like add options to file-export a view to a .ods spreadsheet, or add a formula language function to do something like @dbcommand(“xpath”;bookstore;”/Bookstore/Book/Title”) to return the titles of all the books in an XML format string in the bookstore variable. You can use the java runner framework to plug in new URL commands to parse views server-side and return them in new and more sensible formats, perhaps exposing your Domino application as a XUL so it uses all the local UI components of the Mozilla applications.
These things can be done now, and I have had a stab at most of them. Trouble is it isn’t much fun.
Here be dragons
Domino is full of historical cleverness that was before it’s time. The date and time handling is near-perfect, it was certainly the first application I ever came across with time zone handling that worked. It understands half-zones, differential daylight savings rules, everything. The trouble is it stores date values internally in an 8 byte floating point number, with the integer part being the days from 31st December 1899 (you can put in negative values to go back to 100 A.D.) and the fractional part being the number of seconds from midnight. This is fairly efficient, covers a wide range of dates and finding the difference between two dates is just simple subtraction. I am not quite sure where it puts the zone information but it finds somewhere for it to go. So all very clever, but not how you would do it now. It makes no reference internally or externally to ISO 8601 date formats you can’t get an ISO8601 out of a notesdatetime object, or indeed load one in. OK, if Notes was Open Source someone could add 8601 methods to notesdatetime, but anyone doing deeper work, perhaps adding a caldav server, is going to hit up against these clever but not very modern dates all the time.
Domino is older than Unicode. In fact Lotus helped write Unicode and even had Kevin Cavanaugh on the board of directors of the Unicode Consortium. Domino uses Unicode, but also uses LMBCS. This is a clever format, very efficient (only uses multiple bytes when it needs it) but ultimately non standard which means that you need to constantly convert between LMBCS and unicode when doing C-API level stuff (like trying to write @dbcommand(“xpath”) for example).
Response documents are very very handy, but somewhat unusual as a concept, Replication conflicts even more so. There are better ways of handling conflicts than mixing them up with real records, put them in the database attached to the relevant record but in a different way, kind of like the edit history in a Wiki.
So what would I like to see?
Well the initial suggestion uses the term “Open Source” which is a bit woolly, it can mean many things, from disclosure of the source code, to putting the whole lot in an SVN repository under a GPL license and accepting patches. The term “Free Software” is a bit more specific and provides four types of freedom:
- The freedom to run the program, for any purpose (freedom 0).
- The freedom to study how the program works, and adapt it to your needs (freedom 1). Access to the source code is a precondition for this.
- The freedom to redistribute copies so you can help your neighbor (freedom 2).
- The freedom to improve the program, and release your improvements to the public, so that the whole community benefits (freedom 3). Access to the source code is a precondition for this.
I want all of these, I am not sure how many Ian Tree had in mind. I don’t think IBM are going to provide any of them based on Notes, and I wouldn’t really be that interested if they did. What I do want to see is a new rich client written in a sensible modern high level language talking to a sensible modern document oriented distributed database on the back end. This isn’t Notes any more, but it might be a python based client and a couchDB back end.
Steve Mills wrote some time ago that Domino needed to be stuck in the workshop for an overhaul. This was somewhat controversial at the time as Domino and Notes was quite good when he said it. There were bits that could have been refined and made better, certainly. The strategy from that point onward seemed to be to make things bigger and more complicated. To make things more complicated and slower on the client side they chucked in Eclipse. To slow things down on the server side and waste a lot of time they did the whole db2nsf thing. Then they chucked in Xpages which might sanitise web-only applications a bit, but totally destroys the notion of a develop-once application for rich and thin clients. Then they grafted on an elderly version of OpenOffice.org (the newest version which they didn’t have to publish the source – but bizarely won an Open Source award recently). Granted some people might be really happy with the new stuff flung into Notes. Personally I would have simplified things until it hurt. Notes went into the West Coast Customs workshop and now it has lots of spoilers and fins all over it. They are tightly welded on so there is no going back. The only way forward is to drive it or get a new car.
Almost a workspace
As many of you know, I have been playing with various small Linux based laptops which are going to change the world. One thing I have noticed on some of them is a familiar user interface. This is the Elonex One, it was launched last week in the UK and I was there to play with it, I even did a review. Here is the home screen of the One in tablet mode:
Now lets look at another small Linux laptop, this is the Asus EeePC which in the UK RM are calling the RM Minibook:
Does that remind you of anything? Let me give you a clue 
I couldn’t get Notes standard for Linux running on it (mainly because of the odd way IBM packaged the install I think) but this is Notes 8 basic for Windows running under Wine.
Of course no article about world changing Linux laptops would be complete without a picture of the One Laptop Per Child XO laptop so here it is:
No workspace on that one, but it is running CouchDB (yes, that URL does start with localhost)
Spot the diff
Wikipedia or wikis in general have a really really cool feature, the history tab. It shows the differences between two versions of a page down to the word, you can easily see what has been added, what has been removed and what has been changed. Behind all this is a diff algorithm. The diff algorithm used by Domino Wiki was written by John Resig and released under a creative commons license, it works really well but it is written in Javascript. This is great for web applications and it does work in the Notes client as Javascript to some extent, but I want a version in LotusScript.
Step one is to understand the algorithm.
Firstly, there are some redundant functions. Function randomColor() and function diffString2() can both go, they are a distraction. Function escape() is useful on the web, but irrelevant in the notes client so that can go. Function diffString() and function diff() are the ones that we need to examine.
There are no comments in the code and lots of short cryptic variable names, so lets rewrite it with comments to explain what it is up to.
function diffString( o, n ) {
//o and n are strings, the old string and the new string
//first trim the ends of the strings s+$ means whitespace (s) followed by end of input ($)
o = o.replace(/s+$/, ''); n = n.replace(/s+$/, '');
//now call the diff function and put the result in a variant called out. More on this later.
//the parameters of the diff function are two arrays, the question mark is a kind of shorthand if statement
//if the string is "" then it passes in an empty array [] else it does an @explode on the strings and passes in that.
var out = diff(o == "" ? [] : o.split(/s+/), n == "" ? [] : n.split(/s+/) ); var str = "";
// it is now going to make an array of all the whitespaces in the original strings
//this allows it to put back the correct type of whitespace (space, tab, newline etc.) in the correct position
//personally I don't care very much. I would be happy with all whitespaces ending up as " " in the output
var oSpace = o.match(/s+/g);
if (oSpace == null) {
oSpace = ["n"];
} else {
oSpace.push("n");
}
var nSpace = n.match(/s+/g);
if (nSpace == null) {
nSpace = ["n"];
} else {
nSpace.push("n");
}
// out is an object with various properties, we will work out the exact structure later.
if (out.n.length == 0) {
//if the new string is empty then the whole of the old string is in <del> tags
for (var i = 0; i < out.o.length; i++) {
str += '<del>' + escape(out.o[i]) + oSpace[i] + "</del>";
}
} else {
if (out.n[0].text == null) {
//this is another test for the new string being empty so more <del> tags
for (n = 0; n < out.o.length && out.o[n].text == null; n++) {
str += '<del>' + escape(out.o[n]) + oSpace[n] + "</del>";
}
}
//now we loop through word by word in the new string
for ( var i = 0; i < out.n.length; i++ ) {
if (out.n[i].text == null) {
//the line above looks a bit wrong, but the .text property of out.n[i] doesn't do what you think it does.
//if out.n[i].text is null then that means that the word in the new string has been inserted
str += '<ins>' + escape(out.n[i]) + nSpace[i] + "</ins>";
} else {
var pre = ""; //now append any words from the old list that followed the current word from the new list
//these are stored in the row property of out.n[i]
for (n = out.n[i].row + 1; n < out.o.length && out.o[n].text == null; n++ ) {
pre += '<del>' + escape(out.o[n]) + oSpace[n] + "</del>";
}
now we add the words that appeared in both lists. out.n[i] won't be populated for inserted words.
str += " " + out.n[i].text + nSpace[i] + pre;
}
}
}
//and hand back one string with markup for differences to the caller.
return str; }
I hope the comments help a bit, but it doesn’t totally make sense yet until we understand the structure of the “out” object. I used the sample text from John Resig’s article:
diffString(“The red brown fox jumped over the rolling log.”,“The brown spotted fox leaped over the rolling log);
after running it gives output like this:
The <del>red</del> brown <ins>spotted</ins> fox <del>jumped</del> <ins>leaped</ins> over the rolling log.
I used a stringify function to convert the “out” object to JSON code, so here is what the object looks like as JSON:
{
“o”:[
{"text":"The","row":0},
"red",
{"text":"brown","row":1},
{"text":"fox","row":3},
"jumped",
{"text":"over","row":5},
{"text":"the","row":6},
{"text":"rolling","row":7},
"log."
],
“n”:[
{"text":"The","row":0},
{"text":"brown","row":2},
"spotted",
{"text":"fox","row":3},
"leaped",
{"text":"over","row":5},
{"text":"the","row":6},
{"text":"rolling","row":7},
"log"
]
}
so out has 2 properties, n and o. They are both arrays of things, the things can either be scalar text or arrays.
The arrays have a property called “text” and a property called “row”
That is enough for now, I will dig into the diff function next which builds this out object, then I will try to reimplement the whole thing in Lotuscript.
