Uniscala Granite is a web application stack with the following distinctive features:

• a convenient, simple to use, yet powerful embedded object database: db40
• its own ‘static’ (explained below) dependency injection,  using Uniscala Contexts
• JQuery integration through Wiquery
• a module system for easy application assembly (not to be confused with Maven modules)
• a fully integrated security framework for flexible authentication and authorisation, designed to support traditional user accounts as well as more interesting scenarios such as OAuth 2.0 (we have an experimental but working Facebook login module)

In addition to Granite ‘Core’, there are currently these optional Granite modules:

• Granite Scheduler — a custom scheduler for scheduling business-level tasks, using Scala Actors under the covers
• Granite Mailer — an email sending service based on the Java Mail API
• Granite Users — basic user ID/password accounts stored in the primary database (under construction)

The Application

So how does this all go together? Like any Wicket application, there is a main application class configured in web.xml. In  Granite you typically use the Wicket application instance for configuring global security settings, registering the Granite modules that are to be loaded as the application starts up, and perhaps overriding other application members such as the ‘homePage’. The rest of the initialisation occurs in the modules. A Granite application must extend the Granite ‘App’ class (note that we avoid the class name ‘Application’ as it has special meaning in Scala), so you might do something like this:

import net.uniscala.action.ImplicitAction._
import net.uniscala.meta.text.Name
import net.uniscala.granite.{App=>GraniteApp}

class App extends GraniteApp {
  override val name = Name("Granite Examples")
  override val allowedAnonymousActions
    = super.allowedAnonymousActions ++ Set(View)
  override def init = {
    registerModule(ModuleA)
    registerModule(ModuleB)
    registerModule(new ModuleC(someParameters))
    super.init // MUST be called last
  }
}

The code above does three things: it give the application a name, it allows anonymous users to perform actions that require the generalised ‘View’ action, and it registers three modules (in our example, the first two modules are singletons and the last one is instantiated and takes some configuration parameters).

Modules

Modules are the high-level component in a Granite application. They have a simple lifecycle. Unlike component systems such as OSGI, modules cannot be started outside of application initialisation, and cannot be stopped until the application stops running. Instead, the more dynamic aspects of your application can be provided by the resources and services that you set up in the contexts.

A module’s lifecycle interface is essentially:

 def registerModule(app:App)
 def initModule(app:App)
 def startModule(app:App)
 def destroyModule(app:App)

in the order that they are called on a module. The ‘registerModule’ method can be used for registering other modules that a module depends on. Otherwise, initialisation is generally performed in the ‘initModule’ method. However, it is important to note that the main database is not available in the context until the ‘startModule’ method is called (we discuss contexts below). The first three methods are called during application startup and the last method during application shutdown.

Modules typically set up or configure resources for an application. Frequently, this consists of registering actions and views (both of which we will describe  below) and often also configuring the database for new types of data. Sometimes modules will add some data to the database the first time that they run.

Contexts

Most things in Granite are provided with a context, or more specifically, a ‘UniversalContext’ from the Uniscala Contexts API, which allows accesses to the resources in the context by code such as this:

context { s:MyService =>
  // use the resource 's'
}

We use the term ‘resource’ fairly loosely here. Often resources like ‘MyService’ are traits, and the context will expose a concrete implementation of the trait that has been registered in the context. For example, like this:

context.register[ServiceX](new MyServiceX)

where ‘ServiceX’ is a trait and MyServiceX implements that trait. Note it is fine to register the same instance under multiple traits if it implements, say, more than one service.

A context used in this way is equivalent to

context useOrFail { res:SomeResource =>
  // use the resource 'res'
}

and will throw a ‘ResourceUnavailableException’ if there is no resource with the interface ‘SomeResource’ registered in the context. There are also ‘use’ and ‘useOrElse’ methods on a context that can provide alternatives to failing if you are not sure the resource should be there. There is a lot more to the contexts API, but that can be the topic for a later article.

Granite actions receive a context by implementing the ‘MutableUniversalContextual’ trait, which adds a mutable  ‘context’ property to a class. The Granite framework will make sure the context is set before an action is invoked. Granite views may get a context in other ways — we’ll examine this when we come to discuss views.

There is a root context stored in the application — typically this is what you will encounter throughout a simple Granite application, but that should not be assumed.

I should mention that one of the commonest uses of a context is access to the database via a db4o ‘ObjectContainer’:

context { db:ObjectContainer =>
  db.store(someItem)
}

Note also that Granite’ internal machinery will make sure the database transaction is committed at the end of the request, which brings us to the database:

The Database

For convenience, Granite comes with an embedded object database provided by Versant’s open source db4o product. Granite’s core doesn’t strictly rely on that database being used, and you could use alternative or additional databases for your application.. We already have a client project being demoed which is a reporting engine that mainly accesses logs in the MySQL database of another application. I can also envisage scenarios using more than one db4o databases — perhaps even one per user. Granite does not specifically provide extra support for any of these situations (although contexts are great places for connection pools — I will do an article on this soon).

Granite does provide some sugar coating for db4o. Firstly, it lazily provides a new ObjectContainer per request cycle, automatically committing that container at the end of a successful request. If an exception occurs during the request cycle though, the container is rolled back instead of being committed. Granite’s db4o operates in db4o’s so-called ‘embedded client-server’ mode.

Another convenience is the Uniscala DB4O Utilities library (which can be used outside of Granite). Quite often you want to use queries as Wicket models, but db4o’s SODA queries are not serializable. Thus we have provided our own query API, found in the ‘net.uniscala.db4o.query’ package, that allows serialisable queries to be created and later applied to a db4o database, and has a more FP flavour:

import net.uniscala.db4o.query.Query
val lookups = context { db:ObjectContainer =>
  Query[MyPOJO] in db
}

where ‘lookups’ is a sequence of lightweight (ObjectContainer)=>MyPOJO functions (they actually wrap db4o IDs). If you want to get the actual objects in one step:

val pojos:Seq[MyPOJO] = context { db:ObjectContainer =>
  Query[SomePOJO] in db map (_(db))
}

Db4o is used in a specific way for Granite’s default database, and YOU NEED TO KNOW THIS is for any non-trivial Granite application:

Firstly, transparent activation (TA) is turned on. This means that activation depth is unlimited for objects that are not aware of TA. Why do we do this? We don’t want to worry about fiddling with activation depth — we consider that process fraught with potential runtime errors. Manually managing activation depth means continually tracking the internal structure of the classes involved, often negating the advantages of OO encapsulation. It is bad enough having to do this for indexes, but at least the consequence of a bad index is potential for slow, unoptimised queries, not a horrendous runtime exception, or worse, an undetected error in program logic.

Secondly, for those that like it, there is a problem with code instrumentation to automatically add TA awareness to classes: last time I checked (Scala 2.7.x), it didn’t work with byte code generated by the Scala compiler. So, you ask, how are we preventing the wildfire of TA from pulling too much out of the database? We use what I like to think of as a ‘fire break’ strategy: judicious use of TA-aware relationship classes, typically based on one of our TA-aware base classes, such as:

class Membership extends ActivatableRelationship2[User,Group]

My rule of thumb is to use ordinary object references where you are linking to something that would be considered a part of the object (unless it is huge), and use our TA-aware relationships for anything less strong. In fact, you may find this leads to better extensibility, as relationship classes can be extended and evolve over time, whereas a plain object reference can’t. You could also use the TA-aware classes provided by the db4o API. We are still investigating this area. It may be that in Scala 2.8.0 or the upcoming 2.8.1, instrumentation will work, although I’m not that keen on byte code augmentation, so I may not use it anyway.

Db4o’s Scala limitations also appear to limit optimisation of native queries, so for the time being we are only recommending use of the db4o SODA query API when writing queries in Scala. Native queries can still be used, but Scala confuses db4o’s query optimisation and you will most likely end up with unoptimised queries which could make your application unexpectedly slow.

Overall, db4o integration is not a good as we would like, but hopefully we can address some of these issues over time. Even with the added difficulties mentioned above, it is still simpler and faster than using an ORM in our opinion.

Configuration of the db4o database can be done during module initialisation:

context { dbConfig:ServerConfiguration =>
  dbConfig.common.objectClass(classOf[Book]).objectField("title").indexed(true)
}

Actions

High-level user actions are modelled by actions in the Granite framework. They also provide a convenient point to attach security configuration. There are two broad categories of actions: global actions, and action on a particular business object: item actions. Ultimately, actions that do something useful extend net.uniscala.action.ConcreteAction:

package net.uniscala.action
trait ConcreteAction[+R] extends Action {
  def doAction:R
}

There are quite a few base classes provided by Granite and the Uniscala libraries for various types of action. For example:

• global actions are quite often associated with some type of business object, and so will extend net.uniscala.action.TypeAction.
• actions that implement MutableUniversalContextual will be given a context by Granite each time they are performed
• traits like net.uniscala.granite.action.ContextualTypeAction provide a basis for the above
• there is also a ContextualItemAction base class (ItemAction is a sub-trait of TypeAction)
• in the package net.uniscala.granite.db4o.action, the abstract Create, Delete, Edit, QueryItems and View classes provide convenient base classes for db4o CRUD operations, and also mix in security declarations (see Security below): extending these actions may be all that is needed for simple Granite applications

Global and item actions can be registered during module initialisation:

context { actions:ActionRegistry =>
  actions.registerAction[AddBook]
  actions.registerAction[ListBooks]
}
context { itemActions:ItemActionRegistry =>
  itemActions.registerItemActionFor[Book,ViewBook](lookup => new ViewBook(lookup))
  itemActions.registerItemActionFor[Book,EditBook](lookup => new EditBook(lookup))
  itemActions.registerItemActionFor[Book,DeleteBook](lookup => new DeleteBook(lookup))
}

which means they can appear in menus and and against business objects in listings.

Views

A view in Granite is a Wicket component that provides a web user interface to allow a user to perform an action. View are registered during module initialisation:

context { views:ViewRegistry =>
  views.registerView((wicketId, action:AddBook) => new AddBookForm(wicketId, action))
  views.registerView((wicketId, action:ListBooks) => new QueryItemsView[Book](wicketId, action))
  views.registerView((wicketId, action:ViewBook) => new BookView(wicketId, action))
  views.registerView((wicketId, action:EditBook) => new EditBookForm(wicketId, action))
  views.registerView((wicketId, action:DeleteBook) => new DeleteConfirmationView[Book](wicketId, action))
 }

Security

Security in Granite is still under development, and I will only touch on it in a shallow way here — both because it may change ,and because this article is already long enough! I have tried to design a security API that is less jargon filled and more intuitive than most. It probably deserves an article of its own at some stage.

Concrete actions that require security extends net.uniscala.auth.Secured:

trait Secured {
  def requiredActions:Set[Action]
}

The ‘requiredActions’ method typically returns a set of one or more net.uniscala.action.ImplictAction objects (the name ‘ImplictAction’ has nothing to do with Scala implicits). Implicit actions are actions that don’t do actually anything in an application, but instead represent an action: they include ”Create’, ‘View’, ‘Update’ and ‘Delete’ (note ‘Edit’ is the set  composed of ‘View’ and ‘Update’). Also there is ‘Add’ and ‘Remove’ for actions on collections. These are evaluated by an net.uniscala.auth.Authorizer (not shown here) which is located on the business object or through the context of the business object, action or view in question to see if the action is permitted (the way and order in which this is located  is under review, so this is deliberately vague). If no Authorizer is found then ultimately Granite falls back to using one of two methods in the App base class, which are by default implemented like this:

def allowedActions:Set[Action] = allowedAnonymousActions
def allowedAnonymousActions:Set[Action] = Set(WicketAction.ViewPage)

The first method is used for authenticated users only. The ‘WicketAction.ViewPage’ action simply represents instantiation (but not rendering) of a Wicket page. Thus the default is very restrictive (by design), and does not to allow any action that requires the previously mentioned implicit actions (View, Edit, etc.). In many cases, you will want to override this.

I am currently looking at various ways of grouping actions and/or creating “secure zones”, such as an admin area for all admin actions, with their own authorisers (that might for example use roles) in order to allows a range of security measures across Granite applications .. but this is a good topic for another article.

** NOTE: There is updated code in the comments.

If you’ve followed all of the steps in the guide for using the Maven archetype, you should already have an empty application called ‘myapp’ that will run under Jetty — we’ll use this as  a starting point for our example.

We will (roughly) be recreating the simple book database example that comes with Granite (in the ‘granite-example’ module), although it will run standalone, rather than being part of a larger  application. Refer to the book module in ‘granite-example’ in SVN for the latest version (and typo free code!).

Before we start, let’s quickly look at what we have in our application’s directory:

pom.xml
src/main/resources/log4j.xml
src/main/webapp/WEB-INF/web.xml
src/main/scala/net/example/App.scala
src/main/scala/net/example/Module.scala
src/main/webapp/static/...

The file pom.xml, log4j.xml and web.xml should be familiar in concept to any Maven-aware, Java web application developer. Wicket users will know that web.xml contains the name of the Wicket application class, in this case ‘net.example.App’, and other key Wicket configuration parameters. To this, Granite adds the location of the DB4o database relative to the servlet context path, and the user name and password of Granite’s ‘super user’ – a user that is always granted all permissions:

<init-param>
  <param-name>rootDatabaseFilePath</param-name>
  <param-value>root.db4o</param-value>
</init-param>
<init-param>
  <param-name>superName</param-name>
  <param-value>super</param-value>
</init-param>
<init-param>
  <param-name>superPassword</param-name>
  <param-value>granite456</param-value>
</init-param>

In any serious application, to make your application secure, you must make sure that the super user credentials are changed from the default. For this example though, it is fine to leave the file as is.

The ‘App.scala’ file contains the Wicket application:

class App extends GraniteApp {
  override def name = Name("My Application")
  override def init = {
    registerModule(Module)
    super.init
  }
}

There is not much going on here except registering a Granite module. Here is the module (I’ll omit a bunch of code comments and uninteresting methods and fields):

case object Module extends GraniteModule {
  override val name = Name("myapp")
  override def initModule(app:GraniteApp) = {
    super.initModule(app)
    // ...
  }
  override def startModule(app:GraniteApp) = {
    super.startModule(app)
    // ...
  }
}

Modules are high level components in Granite, and should not be confused with Maven modules. A simple application will often only have one module. The ‘initModule’ and ‘startModule’ methods represent the two stages in a module’s life cycle, both of which occur during the start-up of the application. Most of the action happens in the ‘initModule’ method, where we register the module’s resources and services and may configure resource and services from other parts of the application. Any modules used in the application must be registered first in App. Hopefully this will become clearer for the reader as we go through a concrete example.

Step 1: domain objects

Here is the Book class were a going to be using to store book data with:

import net.uniscala.meta.text.{Name, Named}

@serializable
class Book(var title:String, var author:String, var description:String) extends Named[Book] {

  def this() = this("", "", "")
  val created:Long = System.currentTimeMillis
  override def name = Name(title)
}

‘Name’ and ‘Named’ are part of the Uniscala Metadata API which is used by Granite views in listings and such — but you don’t need to worry about these at just yet. The main point is that the actual fields are nice simple types, which are easiest to store in DB4O.

Step 2: actions

The next thing to think about are the actions you want your application to perform. In our case, we want fairly standard CRUD operations: list all books, view a book, add a book, edit a book, and delete a book. Actions are used in Granite to perform desired actions, and also to generate menus and control security. Here are the actions we want:

import net.uniscala.granite.db4o.action.{QueryItems, View, Create, Edit, Delete}
import net.uniscala.ctx.UniversalContext

@serializable
class ListBooks extends QueryItems[Book]

@serializable
class ViewBook(lookup:UniversalContext=>Book) extends View[Book](lookup)

@serializable
class AddBook extends Create[Book] {
  override def createItem = new Book
}

@serializable
class EditBook(lookup:UniversalContext=>Book) extends Edit[Book](lookup)

@serializable
class DeleteBook(lookup:UniversalContext=>Book) extends Delete[Book](lookup)

The ‘UniversalContext’ references above are a context that actions are given before they are executed. They allow actions to access resources and services in Granite, such as access to a database. In this case, we are using the context to find the items that we are acting on — typically by pulling them out of a database, as is done under the covers of the View, Edit, Delete and QueryItems superclasses above. Instances of actions will be stored in Wicket components, so they need to be serializable. Actions are registered in the module, but we will show that further below.

Step 3: views

Lastly, we need views for our actions. Views in Granite are Wicket components, usually panels, that enable the user to perform actions through the web UI. Here we show the views we’ll need:

import org.apache.wicket.markup.html.basic.Label
import org.apache.wicket.markup.html.form.{Button, Form, TextField}
import org.apache.wicket.markup.html.panel.Panel
import org.apache.wicket.model.PropertyModel

import org.joda.time.DateTime
import org.joda.time.format.DateTimeFormat

import net.uniscala.action.TypeAction
import net.uniscala.granite.db4o.action.{Create, Edit, View}
import net.uniscala.granite.model.ContextualItemModel

class BookView(wicketId:String, action:View[Book]) extends Panel(wicketId) {
  def model = new ContextualItemModel(ctx => action.item)
  setDefaultModel(model)

  {
    add(new Label("title", book.title))
    add(new Label("author", book.author))
    add(new Label("description", book.description))
    val format = DateTimeFormat.mediumDateTime
    val added = new DateTime(book.created)
    add(new Label("added", format.print(added)))
  }

  def book:Book = getDefaultModelObject.asInstanceOf[Book]
}

abstract class BookForm(wicketId:String, val action:TypeAction[Book,_]) extends Panel(wicketId) {
  def model:ContextualItemModel[Book]
  setDefaultModel(model)
  def book:Book = model.getObject

  val form = new Form("form")
  form.add(new TextField("title", new PropertyModel[Book](book, "title")))
  form.add(new TextField("author", new PropertyModel[Book](book, "author")))
  form.add(new TextField("description", new PropertyModel[Book](book, "description")))
  form.add(new Button("save") {
    override def onSubmit = BookForm.this.onSubmit
  })
  add(form)

  def onSubmit:Unit = action.doAction
}

class AddBookForm(wicketId:String, action:Create[Book]) extends BookForm(wicketId, action) {
  override def model = new ContextualItemModel(ctx => action.item)
}

class EditBookForm(wicketId:String, action:Edit[Book]) extends BookForm(wicketId, action) {
  override def model = new ContextualItemModel(ctx => action.item)
}

A ‘ContextualItemModel’ is a loadable-detachable model which can receive a context and can for example loads items out of a database. Note that the view wraps the action, which is why actions need to be serializable. Like actions, views are also registered in the module , as we show below. We also want views for listing all books and deleting books, be we can use some views that Granite supplies for these, as you can see in the Module code below.

There also has to be Wicket HTML markup files for our views, as is the Wicket way, but these are straightforward (you need to know Wicket!) Here is the HTML for BookView:

<wicket:panel>
<p>
 <b><span wicket:id="title">title</span></b>
 by <em><span wicket:id="author">author</span></em> &mdash;
 <span wicket:id="description">description here</span>
</p>
<p style="font-size:50%; color: gray;">
 entry added: <span wicket:id="added">added date</span>
</p>
</wicket:panel>

and the HTML for BookForm:

<wicket:panel>
<form wicket:id="form" action="#">
 <label>Title: </label><input wicket:id="title" size="60" /><br />
 <label>Author: </label><input wicket:id="author" size="30" /><br />
 <label>Description:<br />
 </label><input wicket:id="description" size="100" /><br />
 <input type="submit" wicket:id="save" value="Save" /><br />
</form>
</wicket:panel>

Step 4: wire it up

Below is the complete Module class. As well as wiring up the actions and views, we also demonstrate a little DB4O configuration by pulling out DB4O database’s ServerConfiguration from the context and adding some indexes — this is just for show as we don’t have any search features in our application yet. However, if we have search, these (entirely optional) indexes would  optimise searching by title and author.

We similarly access the registries for actions and views and add entries there. Note that there are two types of actions: global actions and item actions. Item actions are actions done to an ‘item’ — typically some kind of business object (a book in our case).

import com.db4o.cs.config.ServerConfiguration

import net.uniscala.meta.text.Name
import net.uniscala.granite.{App=>GraniteApp}
import net.uniscala.granite.module.{Module=>GraniteModule}
import net.uniscala.granite.action.{ActionRegistry, ItemActionRegistry}
import net.uniscala.granite.view.ViewRegistry
import net.uniscala.granite.db4o.view.{DeleteConfirmationView, QueryItemsView}

case object Module extends GraniteModule {
  override val name = Name("Book Example")
  override def initModule(app:GraniteApp) = {
    super.initModule(app)
    context { dbConfig:ServerConfiguration =>
      val common = dbConfig.common
      val bookClass = common.objectClass(classOf[Book])
      bookClass.objectField("title").indexed(true)
      bookClass.objectField("author").indexed(true)
  }
  context { actions:ActionRegistry =>
    actions.registerAction[AddBook]
    actions.registerAction[ListBooks]
  }
  context { itemActions:ItemActionRegistry =>
    itemActions.registerItemActionFor[Book,ViewBook](
      lookup => new ViewBook(lookup))
    itemActions.registerItemActionFor[Book,EditBook](
      lookup => new EditBook(lookup))
    itemActions.registerItemActionFor[Book,DeleteBook](
      lookup => new DeleteBook(lookup))
  }
  context { views:ViewRegistry =>
    views.registerView((wicketId, action:AddBook)
      => new AddBookForm(wicketId, action))
    views.registerView((wicketId, action:ListBooks)
      => new QueryItemsView[Book](wicketId, action))
    views.registerView((wicketId, action:ViewBook)
      => new BookView(wicketId, action))
    views.registerView((wicketId, action:EditBook)
      => new EditBookForm(wicketId, action))
    views.registerView((wicketId, action:DeleteBook)
      => new DeleteConfirmationView[Book](wicketId, action))
    }
  }
}

Step 5: run it!

You can now compile your application and run it through Jetty. However, because of Granite’s default security configuration, you wont see anything interesting until you login as ‘super’ through the special URL http://localhost:8080/login?super=1. After logging in (default: super/granite456) you should  see the global actions defined for your application laid out in a simple default  listing:

From there you can add a book:

and list books:

and from the listing, corresponding to the icons, view, edit or delete books (note: the order of icons above differs, and is not the best):

Although Granite is not released yet, I hope brave explorers who are looking at it will find the above information a useful first step. How all of this actually works underneath is a topic for a future post. There is still work to be done in polishing the views and how actions are found and flow from one to the other, and also reducing boilerplate code. For example, we have some incomplete and experimental code for view generation based on type-safe schemas that are hand coded or generated by reflection (or both) that could be suitable for basic user interfaces, and could eliminate the view code in our example above.

Granite is a lightweight framework for the rapid development of web applications. It is based on the very cool and richly featured Apache Wicket web framework. Granite uses an embedded object database that avoids the need for SQL or Object-Relational Mappers (ORMs), and, in the Wicket tradition, is proud of, if not smug about, its distinct lack of external XML configuration files.

Granite is not designed to build large ultra-scalable systems such as Facebook and Twitter, although conceivably it could be a component in a system like that (I sometimes dream of a world where everyone has their own database they can take with them from one application to another). It is more aimed at modest membership databases, staff directories, online shops, and so on.

My prior experience with writing applications for Zope, and Plone, which runs on Zope, was the beginning of Granite. Zope has its own object database, ZODB, which is how I came to be addicted to this kind of persistent storage. Settling on one particular database rather than trying to abstract over many kinds also has certain advantages when it comes to both simplicity and taking full advantage of the features of a database.

The object database that we are using for Granite is Versant’s DB4O, licensed under their dOCL license. Prior to the dOCL, we have been using GPL for our own components, but I believe the dOCL allows us to change to Apache 2.0. If we find the licensing intractable, we will switch to a different object database, such as NeoDatis (LGPL).

Limitations in ZODB, and a desire for type-safe code is what led to a change from the Python-Zope world to the Java world … and the Scala world, because that is what we are coding in. Yes, Scala is the final part of the Granite trinity (Wicket-DB4O-Scala), because only Scala’s expressive power could tempt me away from Python and back to the land of type-safe languages.

What else to say? We really only have one simple example at the moment, and some tectonic plates of architecture are still adrift, especially with regard to the security components. There is also very little internationalisation so far – we really need an expert to guide us here. However, a client project of ours is using Granite for some things and is almost at completion, and we have a number of internal projects on the go that depend on Granite. As well as the example, there is a “quick start” guide available to shows how to check out and use the code. There is also a development Maven repository available, but it is not yet updated often enough (or tested) to recommend that developers use it. There will be further announcements on this soon.

I invite you to check out Granite and play with it. I apologise for the lack of general documentation, but more is coming, especially if people prompt me for it . I’m always open to discussion and suggestions, and nothing is set in stone yet. I intend to publish more entries on the finer details of how Granite hangs together, as well as some detail on the reusable components (the Uniscala libraries) that can be used outside of Granite that I hope some will find useful.

Cost

“open source software requires a certain level of expertise in order to manage content”

Any CMS laden with worthwhile features requires training for proper use. The authors fail to explain that this equally applies to proprietary systems. The same can be said for other issues listed in the Cost section: the cost of support and building custom infrastructure. There is a vague implications that consumers will be “battling with service/support issues” and an unsubstantiated claim that open source systems have a “general inability to scale”, which is at odds with the vast array of open source software that powers much of the Internet’s inner workings, as well as gargantuan open source-based systems such as Facebook, Flickr and Twitter.

The pinnacle of non-disclosure in the Cost section is this:

“Notably, open source software providers are increasingly charging for add-ons, additional services and integration.”

Is coreDNA saying they won’t charge for additional services, integration, or custom add-ons? A little too good to be true I suspect. For Plone, I have never needed to purchase add-on modules for clients, since free ones abound. It is true that we have been paid to customise modules for clients, or even produce new custom components, but the same would be true for proprietary systems such as coreDNS, and your choice of developer would be vastly more limited, and thus the expense likely to be much higher. Additionally, in an open source ecosystem, with hundreds or thousands of developers, the chance that someone has already done what your client needs is much higher.

Service and Support

“Open source software relies on its online community network to deliver support via forums and blogs. While there are massive, loyal and engaged online communities that users can turn to, time-poor consumers of today are familiar with the immediate service and support that enables issues to be resolved in a timely manner, and these communities cannot guarantee the high level of responsive service and support proprietary software can offer.”

The larger open source projects are typically backed by foundations and corporations, and are fertile zones for support service organisation to spring up who are eager to be paid to provide exactly what the above statement suggests you can’t get: timely and responsive service and support. That’s one of the things I do for a living. It is well documented that larger open source projects receive significantly more contributions from corporations and businesses than from individuals. The service is there, but if you need more than support from online forums, then you might need to pay for it.

Innovation

“customized changes to the original source code limit the future support and growth of the software”

The same applies to proprietary software, except you might not have the option of modifying the source at all, and be left without the feature that you needed. Obviously, you need to look at the impact of modifying the original source code before you leap in. However, this situation is fairly rare. It is much more common to add custom layers, plugins or modules on top of the original source, without modifying it. Thus your changes are tightly contained in scope, and easier to manage into the future. It is prudent to discuss future maintenance with your developer before going down this path.

Usability

Open source developers might need to sit down before they read this one.

“Open source software is generally developer-centric, and without system administration experience or the knowledge required to manipulate programming language, use of the software and ability to fix errors as they arise is often limited to those with technical expertise.”

Again, untrue. Many of the foremost experts in computer science, systems administration and software engineering, live on the cutting edge of  open source development. The Internet’s infrastructure is mostly built with open source, as are most new programming languages such as Scala, Groovy, Python and Clojure. Popular non-Windows operating systems are also constructed from open source components: Android, Ubuntu and large parts of MacOS  for example.

“Open source software has been highly criticized for its lack of usability, as generally, the technology is not reviewed by usability experts and does not cater to the vast majority of computer users.”

The authors should do some research here. Plone, for example, has usability experts on its core development team.

“Furthermore, open source software does not legally require documentation such as user manuals or guides, hindering the creation of such tools.”

Is proprietary software legally required to be documented? Documentation can be a weak point of open source projects,especially new or small ones. On the other hand, books on Plone are coming out all the time. Plone has an official online user manual, and sites like Plone TV also provide video tutorials and the like. There are many other source of information in abundance, and also online forums and chat rooms if you are stumped. For mission critical projects, you should also think about having an ongoing contract with a Plone-friendly provider for occasional maintenance, training and general assistance.

Security

Everyone should be concerned about security. It is well documented that security by obscurity – that is, assuming that because your code is hidden it is more secure – is an untrue and dangerous assumption. Security guru Bruce Schneier has this to say:

“As a cryptography and computer security expert, I have never understood the current fuss about the open source software movement. In the cryptography world, we consider open source necessary for good security; we have for decades. Public security is always more secure than proprietary security. It’s true for cryptographic algorithms, security protocols, and security source code. For us, open source isn’t just a business model; it’s smart engineering practice.” – Bruce Schneier (1999)

As a prior principal engineer in the area of software security, I must agree, and declare that coreDNA’s series of scare statements about open source and lack of expertise, trojans, back doors, and lack of peer-review or validation are bunk, and more likely prevalent in proprietary software than open source in my experience.

Conclusion

To conclude, keep in mind that you are locking yourself into a proprietary vendor once you select them as your CMS provider. Unlike open source systems, you generally can’t get independent development companies  in to add custom features that you want – particularly for integrating to other systems which may change or come and go over time. This can vastly reduce the options for your future overall infrastructure, and you may find it difficult or impossible to  extract your data later should you want to change to another solution. Trying to get back to an open source solution if you are dissatisfied may be a very expensive exercise.

$ ls -lh ~/VirtualBox/VDI/MS\ Windows\ 7.vdi
-rw------- 1 sam sam 8.0G 2010-07-21 19:44 /home/sam/VirtualBox/VDI/MS Windows 7.vdi

Firstly, we can use ‘split’ to move the file in smaller chunks. I chose 500 MB chunks:

$ split -a 4 -d -b 500M  ~/VirtualBox/VDI/MS\ Windows\ 7.vdi MS\ Windows\ 7.vdi.part

Here is the result, with nice numeric suffixes:

$ ls -lh
total 8.0G
-rw-r--r-- 1 sam sam 500M 2010-07-29 11:40 MS Windows 7.vdi.part0000
-rw-r--r-- 1 sam sam 500M 2010-07-29 11:40 MS Windows 7.vdi.part0001
-rw-r--r-- 1 sam sam 500M 2010-07-29 11:40 MS Windows 7.vdi.part0002
-rw-r--r-- 1 sam sam 500M 2010-07-29 11:41 MS Windows 7.vdi.part0003
-rw-r--r-- 1 sam sam 500M 2010-07-29 11:41 MS Windows 7.vdi.part0004
-rw-r--r-- 1 sam sam 500M 2010-07-29 11:41 MS Windows 7.vdi.part0005
-rw-r--r-- 1 sam sam 500M 2010-07-29 11:42 MS Windows 7.vdi.part0006
-rw-r--r-- 1 sam sam 500M 2010-07-29 11:42 MS Windows 7.vdi.part0007
-rw-r--r-- 1 sam sam 500M 2010-07-29 11:43 MS Windows 7.vdi.part0008
-rw-r--r-- 1 sam sam 500M 2010-07-29 11:43 MS Windows 7.vdi.part0009
-rw-r--r-- 1 sam sam 500M 2010-07-29 11:43 MS Windows 7.vdi.part0010
-rw-r--r-- 1 sam sam 500M 2010-07-29 11:44 MS Windows 7.vdi.part0011
-rw-r--r-- 1 sam sam 500M 2010-07-29 11:44 MS Windows 7.vdi.part0012
-rw-r--r-- 1 sam sam 500M 2010-07-29 11:44 MS Windows 7.vdi.part0013
-rw-r--r-- 1 sam sam 500M 2010-07-29 11:45 MS Windows 7.vdi.part0014
-rw-r--r-- 1 sam sam 500M 2010-07-29 11:45 MS Windows 7.vdi.part0015
-rw-r--r-- 1 sam sam 166M 2010-07-29 11:45 MS Windows 7.vdi.part0016

Now generate the checksums inside the same directory:

$ sha1sum MS\ Windows\ 7.vdi.part00* > SHA1SUM
$ cat SHA1SUM
a87ce194e11535f3fd61e3942c527287d0c41a65  MS Windows 7.vdi.part0000
cd20735aafccdcf24bf1694dbd6f9711805d6180  MS Windows 7.vdi.part0001
4bb1aac9de6f58be99f68acfb19affad9478d199  MS Windows 7.vdi.part0002
201d2a430a6925f2d138ae1e9e6e76278b3951dd  MS Windows 7.vdi.part0003
91a18a9c0d33ffe5528b04162ea780615a0ee020  MS Windows 7.vdi.part0004
49f4dafd2a048ea0b0eed15224e4210c565c0ce8  MS Windows 7.vdi.part0005
9e647219587ee40a8c29c875be8134fe1e83a7f7  MS Windows 7.vdi.part0006
f4608431fbe6ba46141519c456a96019d2b78dc2  MS Windows 7.vdi.part0007
72d48624aaf3763fafa3575a531620e2ab3fa2a6  MS Windows 7.vdi.part0008
68c8ba1474b068242d1106446284d4f2bc4074a8  MS Windows 7.vdi.part0009
c28f1f00786a70da9df50ed88561c3444f5441bd  MS Windows 7.vdi.part0010
3f0ca7ab71efb670347a702258149b8e5a834893  MS Windows 7.vdi.part0011
98f4708e2b86f0bb0e0e5b59ea5beb52e104f195  MS Windows 7.vdi.part0012
fd87b77ee332f822bd216cf24bde861514c45e71  MS Windows 7.vdi.part0013
727ded9f08a0ede8ed9a5011d75308f093e44162  MS Windows 7.vdi.part0014
386f9c4168c137df52cd040ef24514d62fcf7e42  MS Windows 7.vdi.part0015
85607ffb0fb310bdb2d590475d8df40b8fb89637  MS Windows 7.vdi.part0016

You ca now copy this directory (by flash drive or networks) and check the files afterwards by running ‘sha1sum’ inside the directory thus:

$ sha1sum --check SHA1SUM
MS Windows 7.vdi.part0000: OK
MS Windows 7.vdi.part0001: OK
MS Windows 7.vdi.part0002: OK
MS Windows 7.vdi.part0003: OK
MS Windows 7.vdi.part0004: OK
MS Windows 7.vdi.part0005: OK
MS Windows 7.vdi.part0006: OK
MS Windows 7.vdi.part0007: OK
MS Windows 7.vdi.part0008: OK
MS Windows 7.vdi.part0009: OK
MS Windows 7.vdi.part0010: OK
MS Windows 7.vdi.part0011: OK
MS Windows 7.vdi.part0012: OK
MS Windows 7.vdi.part0013: OK
MS Windows 7.vdi.part0014: OK
MS Windows 7.vdi.part0015: OK
MS Windows 7.vdi.part0016: OK

… and finally concatenate the results into the original file:

$ cat MS\ Windows\ 7.vdi.part00* > /tmp/MS\ Windows\ 7.vdi