Projects:Nathan Marz

Software Plugins and Extensions

Overview

Software is a highly malleable entity. In today’s Agile world, software is constantly changing in response to new and changed requirements. The ability to quickly adapt software to these pressures is essential. Software extensibility and pluggability are two techniques, which when done correctly, can both increase the productivity of a software project and also provide a better experience for the end-user. I will explore a number of case studies in this document of highly extensible software and their different approaches towards achieving extensibility. I will analyze how and when extensibility is a good design decision in a software project.

What is the difference between an Extension and a Plugin?

A software extension in an entity that adds significant unintended functionality to the software.

A software plugin is an entity that adds intended functionality to a plugin in a controlled manner. An example of this would be adding code to handle a new image type in Firefox.

Sometimes there is a gray area between these two distinctions.

Case Studies

Eclipse

Background

Eclipse is an open-source project originally developed by IBM. It was originally intended to be a platform for creating software development tools. The flagship product built on top of Eclipse is a popular Java IDE. Eclipse also consists of a plethora of other tools, such as:

Integrated Source Control (CVS and SVN)
CDT – C/C++ Development Tooling
EMF – Eclipse Modeling Framework

Recently, Eclipse has evolved towards being a more generic framework for “Rich Client Platforms” – supporting the construction of any application rather than just software development tools. Examples of applications built this way include:

Azureus – A Bittorent client
Lotus Notes 8.0 – Collaborative application for business tasks

Eclipse was a successor to IBM VisualAge family of IDE’s. These developers thereby had significant experience in developing IDE’s and software development tools.

Eclipse Architecture

Eclipse itself is a small core that consists of a protocol for creating plugins and a runtime that manages the plugins. The plugin runtime activates and loads plugins when they are in use and can deactivate plugins when they are not in use to reduce the memory footprint. Plugins build on top of each other, so an application built on Eclipse can contain a deep hierarchy of plugins.

A plugin is defined by two files:

META-INF/MANIFEST.MF: This file specifies meta-data about the plugin, such as what plugins it is dependent on for functionality, the plugin name, and the name of the class to use to manage the plugin’s lifetime.

plugin.xml: In this file, a plugin specifies what other plugins it is adding functionality to (“extension”) and also specifies how functionality can be added to this plugin (“extension-point”).

Plugin definitions are very rigid and explicit. Consider the following example of a plugin.xml:

This plugin extends two points on the “org.eclipse.ui” plugin. The first point it extends is called “popupMenus”, which lets it add new menu items to various popup menus in the Eclipse UI. The second plugin it extends is “views”, which allows it to add a new view to the “view container” at the bottom of the Eclipse UI.

When a plugin defines an extension-point to be built upon by other plugins, it defines an XML-schema that extensions on that point must implement. The example above shows two extensions implementing completely different schemas. Notice in the example how each extension specifies a Java class at some point to provide functionality. Although it can’t be seen through the example, the extension-points defined by “org.elipse.ui” define specific interfaces for those classes to implement. Everything is very rigid and structured.

A plugin can programmatically ask the plugin runtime for all the classes which implement one of its extension-points. The functionality of “extensibility” is thereby yielded this way: the plugin defining the extension-points decides how and where to use the data and code provided by its extensions. The Eclipse model is extensibility through pluggability.

Extending Eclipse

I constructed an Eclipse plugin that will count the number of lines in every .java file in a project and also give the line total. This is a feature that is noticeably lacking in the Java IDE distribution of Eclipse. The UI for this plugin looks like:

The plugin implementation was relatively straightforward once I was able to find all the information I needed to understand how to build it. However, that is the main problem with developing Eclipse plugins. A lot of time is spent looking through documentation trying to figure out how things work.

For example, at first I was going to make a plugin that would color editor tabs after they had been inactive for a long period of time. It took a long time to figure out how to access the tab elements (PlatformUI.getWorkbench().getActiveWorkbenchWindow().getActivePage().getEditorReferences()), and Google queries to find example code on how to do these things were not terribly effective. As it turned out, it was not possible to change the color of the tabs – an example of how Eclipse is not arbitrarily extensible. There is a high learning curve to building Eclipse plugins, especially as compared to Mozilla Firefox as will be seen later.

Some ways that Eclipse helps ease the learning process is through its Plugin Development Environment (PDE), which is a set of plugins that allow a programmer to develop plugins for Eclipse within Eclipse. These include visual editors for the plugin.xml and MANIFEST.MF files. Also, it includes useful template projects for getting started fast. For example, there is a template that will create a plugin project which contains a toolbar button and a menu item.

Analysis

Eclipse is a heavyweight way of creating extensibility. It is not true extensibility as defined in the beginning of this wiki, however. Rather, it provides a framework such that plugins can export API’s such that they can be plugged into. Eclipse is not arbitrarily extensible. However, functionality can be added to Eclipse in a myriad of ways, as can be seen by the plethora of projects out there.

Ultimately, the heavyweight model is essential in the Eclipse design. With the functionality of an Eclipse application spread out over so many plugins, there needs to be a robust, rigid way that the plugins interact. If plugins could interact with each other in arbitrary ways, errors would be hard to diagnose and maintenance would be extremely difficult.

Mozilla Firefox

Mozilla Application Framework

Firefox is built on a set of technologies called the “Mozilla Application Framework”. The five most important technologies from this framework are:

XUL – XML-based user interface markup language. Similar to “HTML”, but for making desktop applications.
XBL – XML-based markup language for defining new widgets to be used within XUL.
CSS – Same as the internet technology. This is used to add style and behavior information to the application.
Javascript – Used to control the behavior of the majority of the UI. Integrates with XUL very similar to how it integrates with HTML in web applications.
XPCOM – Like “COM”, this technology allows for integration of different components written in different languages. For example, XPCOM makes it easy to integrate a networking library written in C with a Javascript script.

The Mozilla Application Framework was developed over a period of years before the development of Firefox started. It evolved during the development of the original Mozilla browser.

Mozilla Firefox

Firefox was developed using the Mozilla Application Framework. The majority of the interface is coded using XUL, XBL, and Javascript. The core Firefox file is called “browser.xul”, and this file defines the Firefox interface. Here is a sample portion from the file:

This code defines the search bar on the top right of the Firefox browser.

This is XUL code. However, “searchbar” is not an XUL widget. Typical XUL widgets are buttons, menus, panes, and many more – exactly what is expected in a UI package. The beauty of XUL is the ability to define new widgets and use them exactly like built-in widgets. This is done using CSS and XBL. Here is the CSS code defining the search bar:

The CSS code attaches a “binding” to all elements named “searchbar”. This binding defines the location of the XBL code which defines the content and behavior of the searchbar element. “search.xml” is the name of the file that the binding is contained within, and “#searchbar” indicates to use the binding in the file with the id “searchbar”.

Here is a portion of the XBL code defining the searchbar element:

The “content” tag indicates what the element consists of. This binding defines a horizontal box (“hbox”), and places a button and textbox inside. The button is a built in XUL type, and it is configured to bring up a popup menu as indicated by the picture.

Notes on Firefox

As the previous example showed, the Firefox architecture is very intuitive. XUL/XBL/CSS/Javascript makes it easy to rapidly construct sophisticated user interfaces. The separation of concerns in the Firefox architecture is excellent: all the style information is in the CSS, the XUL defines the layout of the elements, and Javascript defines the behavior. As will be shown, this architecture is highly extensible: any functionality imagined can be added to Firefox via extensions. However, it is easy to see that Firefox was designed this way solely because it was the easiest way to write a web browser. Extensibility was a side benefit. In fact, one of the goals of the Firefox browser was to avoid feature-creep and create a slim application see article.

Extending Mozilla Firefox

I created an extension to Firefox to provide functionality for managing lots of tabs at once. If a tab has been inactive for awhile, the extension colors the tab dark gray. Additionally, the extension adds a button to the toolbar which can be pressed to close all inactive tabs. A tab is considered inactive when it has been deselected for 20x longer than it has been selected. Additionally, the extension adds functionality so that when a tab is double-clicked, it is set to be active.

An extension is defined by two files:

install.rdf: provides meta-data about the extension like its description, name, and version number.

chrome.manifest: Provides instructions to Firefox on the locations of the extension content.

Here is the chrome.manifest for the “Inactive Tabs Closer”:

content inactive_tabs content/
style chrome://global/content/customizeToolbar.xul chrome://inactive_tabs/content/button.css
overlay chrome://browser/content/browser.xul chrome://inactive_tabs/content/overlay.xul

The first line tells Firefox where to find the implementation of the extension. The second line adds style information to the toolbar interface code which is used to style the button which is added. The third line adds an “overlay” to browser.xul, the main file defining Firefox’s interface. An overlay defines what and where to place additional UI code within another XUL file.

Here is the content of the extension’s “overlay.xul”, which contributes to the browser interface:

<overlay id="custombutton-overlay"
  xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">
<script type="application/javascript;version=1.7" src="close_tabs.js"/>
<toolbarpalette id="BrowserToolbarPalette">
  <toolbarbutton id="custom-button-1"/>
  </toolbarpalette>
<toolbarbutton id="custom-button-1"
  label="Clean tabs"
  tooltiptext="Closes unused tabs"
  oncommand="closeTabs.cleanUpTabs();"
  class="toolbarbutton-1 chromeclass-toolbar-additional custombutton"
  >
  </toolbarbutton>
</overlay>

This overlay adds two elements to browser.xul. The first element it adds is a toolbarpalette. The id of the toolbarpalette is the same as that which already exists in “browser.xul”. Firefox will thereby add the content of the overlay’s toolbarpalette to the toolbarpalette already in “browser.xul”. Likewise, the second element defined, the toolbarbutton, will contribute it’s content to the toolbarbutton just defined.

Finally, the overlay also includes a Javascript script. This script provides all the behavior of the extension. Here are some samples from that script:

Sample 1:

window.addEventListener('load', function() {
	closeTabs.init();
}, false);

Sample 2:


this.cleanUpTabs = function() {
	var tabs = gBrowser.tabContainer.childNodes;
	var selected = gBrowser.tabContainer.selectedItem;
	for(let j=0;j<tabs.length;j++) {
		mytab = tabs[j];
			if(mytab.hasAttribute("ctactive")&&mytab.hasAttribute("cttime")&&mytab!=selected) {
				time = parseInt(mytab.getAttribute("cttime"));
				active = parseInt(mytab.getAttribute("ctactive"));
				if(time>=temp.cutoff && 1.0*active/time <= 1.0/temp.ratio) {
					gBrowser.removeTab(mytab);
					j--;
				}
			}
		}

	};

The first sample calls initialization code to be run when the window loads up. The initialization code adds some event listeners to the Firefox tabs along with registering a function to be called every few seconds to perform the extension logic.

The second sample is the code that is called whenver a user pushes the “Close Tabs” button that the extension adds to the toolbar. “gBrowser” is a global variable created by the Firefox core code which is the head of the DOM representing the browser. It is possible to access any part of the UI from this variable. Note how the code accesses attributes of the tab called “cttime” and “ctactive”. These attributes are created and managed dynamically by the extension – a tribute to the flexibility of Javascript and DOM. These technologies make development of extensions very rapid.

Analysis of Firefox Extensibility model

Firefox is infinitely extensible. Almost any piece of it can be modified or added to. Additionally, creating these extensions is easy and rapid due to the flexible natures of the technologies involved.

On the other hand, building extensions for Firefox encourages hacking. For example, adding arbitrary attributes to XUL elements is not the most robust thing to do. If a new version of Firefox comes out which adds the same attribute names to those XUL elements, the extension will all of a sudden break the new version. Additionally, the complete freedom offered by the extension model allows an extension to override any behavior of Firefox. Unlike Eclipse, extensions in Firefox have no bounds. Additionally, multiple extensions can conflict with one another, producing unexpected behavior.

Overall, however, the Firefox extension model must be rated as fantastic. Unlike Eclipse, only a small portion of Firefox functionality comes from extensions. Users pick and choose which extensions they want to install. Because of this property, robustness is not that critical of a property. It is okay if there is a risk of extensions conflicting, since the probability of it happening is low. When it does happen, users should be able to easily diagnose the problem since they chose what extensions to install and know what each does.

If Firefox were architected like Eclipse, as a series of hundreds of extensions, its extension model would be terrible. Clearly, not being able to restrict the scope of what an extension can touch would make it nearly impossible to comprehend what is happening in the code.

Another great property of Firefox’s extension model is the ease with which an extender can find the information he or she needs to extend the Firefox UI. Firefox’s source code is self-documenting. If one wants to add a button to Firefox’s status bar, one needs only read “browser.xul” until one finds the id of the panel to contribute an overlay to. The utility grep is surprisingly effective in mining the Firefox source for relevant code for an extension.

Finally, the fact that Firefox uses technology very similar to DHTML makes the learning curve relatively low, especially as compared to Eclipse.

bddbddb Case Study

Background

This case study is the result of interviews done with Monica Lam and John Whaley in May, 2007, along with John Whaley’s PHD dissertation.

bddbddb is an implementation of Datalog built mainly to be a framework for building program analyses. Program analyses are difficult to write because so much work needs to go into tuning them for performance. bddbddb uses binary decision diagrams as its internal representation, which is an extremely efficient way to represent and operate on data. bddbddb makes it much easier to create program analyses because it abstracts away much of the performance concerns leaving the programmer free to focus on the specification of the program analysis itself.

Development

bddbddb started as a project to build a scalable context-sensitive pointer analysis. BDD’s were chosen as the representation with the premise that if one could represent the data involved in the analysis efficiently, then one could build an efficient analysis. However, it was not so easy as that. John Whaley spent a year improving the performance of the code until it was fast enough. However, he had to put so many complicated “performance hacks” into the code that the code ended up being tedious, error-prone, and generally hard to work with. He did not want to have to repeat all this work for each program analysis he wanted to do, so he decided to build a platform from which he could encode analyses easily that would abstract away the lower level BDD code.

At first, the platform was going to be a custom language for specifying program analyses. However, through research and discussion with his colleagues, Whaley realized that the right language to use was Datalog since it had the semantics he was looking for. Additionally, he could take advantage of the huge amount of work already put into Datalog in prior work.

The implementation from Datalog to BDD’s took a few months. Whaley attributes the rapidness of the implementation to his one year experience building the pointer analysis. He stated that implementing Datalog to BDD would have been “impossible” without his prior experience – it was too enormous of a task to take all at once. Because of its generality, the Datalog implementation is more difficult than only implementing the pointer analysis directly, because logic specific to the pointer analysis cannot be used.

Without the initial experience of building an analysis directly using low-level bdd code, Whaley would not have known what would be an effective language for specifying an analysis. Even worse, since it was a research project, he did not even know if the BDD approach would succeed in the end.

Analysis

bddbddb can be categorized as a pluggable system, where the plugin is a Datalog program specifying an analysis. The most important conclusion from this case study is the following: building the pluggable framework directly, instead of implementing solely context-sensitive pointer analysis, would have been a disaster since the lack of experience would have prevented Whaley from knowing what to build or how to build it.

SUIF2 Case Study

The information in this case is the result of an interview with Monica Lam in May, 2007.

SUIF2 was a project to build an infrastructure for compiler research. It is an example of how being over-zealous in creating extensibility can hurt a project. Part of the problem was that it was an infrastructure for the sake of building an infrastructure. Exactly how it would be used was not clearly defined, so a lot of time was spent building components that did not necessarily produce value in the end.

Conclusions

Stong Understanding of the Problem Domain

The first notable observation to make in the case studies that were successful is that the developers in each had a very strong understanding in the problem domain. In the cases shown (Eclipse, Firefox, bddbddb), they achieved this understanding through prior experience in the domain. The Eclipse developers came from the VisualAge IDE, the Firefox developers inherited the Mozilla Application Framework and had worked on the previous Mozilla browsers, and John Whaley wrote BDD code for a year before deciding to make a pluggable platform.

Having a strong understanding of the problem domain in building an extensible system is necessary for two reasons:

Need to understand how the system will be used in practice – What are the requirements?
Need to understand the difficulty involved between various designs – Is extensibility worth the cost of building it?

For large, complex projects, it seems that gaining experience in the problem domain is the only way to answer these questions.

Feedback during Development

Feedback on a software project is critical for many reasons, and it is especially crucial if the project is designed to be extensible. This is because of the need to understand how the system will be built upon in practice – what is needed in the infrastructure, what is not needed, and what can be made easier to use. The Eclipse core developed and evolved because of feedback received as more and more plugins were created on top of it. The Java IDE developed on Eclipse was developed along with the runtime. Additionally, version 3.0 of Eclipse had a major overhaul of the runtime, adopting an OSGi standard. As people used the Eclipse platform, deficiencies in plugin management were noticed and addressed.

bddbddb is another example of a project which benefited a lot from feedback. It’s usability as a platform increased considerably due to feedback from people using the system.

Method of Extensibility Depends on the Application

As shown, Eclipse and Firefox have polar opposite extensibility models. Additionally, they both have an excellent extensibility model for their problem domains. Eclipse plugins need to be able to build on each other and exist in a large environment of plugins, while Firefox extensions are small add-ons to the core program. Firefox is able to use a flexible, easy-to-use system as its extension mechanism, while Eclipse needs a more heavyweight, controlled, robust approach. Two factors to balance when designing an extensibility system are ease of use and robustness. The way these two factors can interact can be seen in the differences between Firefox and Eclipse.

Future Work

There are a number of interesting areas of future work in this area. The most striking problem area in designing a system is that it is dangerous to make it extensible because of a lack of deep understanding of the problem domain. Yet, the project may potentially greatly benefit from extensibility. Restarting and redesigning a project after months of work is not cheap. How can projects be identified early as being good candidates for extensible systems? What properties should that extensible system have to an extender (i.e., ease of use, robustness)? Expanding upon work in design patterns could potentially help in this area.

Another interesting area of future work I discovered was that of self-documenting code. Mozilla Firefox code is easy to comprehend and needs no additional explanation. The code is concise, at a high level of abstraction, and has a regular structure. This is different from Eclipse, which needs a considerable amount of external documentation to understand how to extend upon it. Self-documenting code is a major boon as it eliminates the need to write external documentation, which can be time-consuming. Additionally, it allows someone to understand how every piece of the program works, something that is not possible using external documentation. What properties must a code-base have to be considered “self-documenting”?