As the saying goes…all good things must end. As of today, my blog is shutting down!

I do intend to leave all content on-line so you’ll always be able to see a list of all 134 posts, but there won’t be any new posts for the foreseeable future. I’d like to thank everyone that has taken the time to read my often times long-winded posts. I hope you’ve found them useful.

For the past few years, I’ve written almost exclusively about modularity and OSGi (with a few other topics sprinkled in on occasion). To that end, I’ve decided to pursue a long overdue book project, tentatively titled Patterns of Modular Architecture (POMA). I hope you’ll decide to join me in my journey. Also, be sure to follow the book updates on Twitter.

So as to not disappoint those who have come to expect something a bit more verbose, I have managed to cobble together a few words explaining my decision in more detail. So if you want to know more, read on!

Story Behind the Story

Over the past few years of blogging, I’ve focused considerable energy on espousing the virtues of modularity and OSGi. From conceptual posts that discuss the important architectural benefits of modularity to simple examples that illustrate the benefit. From the beginning to the end, I’ve written about the benefits of modular architecture at a conceptual level, practical level, and have even provided some concrete examples. To the chagrin of some, I’ve even discussed the challenges that lie ahead. Of course, modularity and OSGi aren’t the only topics I write about. I’ve also written about agility, IT labor, metrics, and more. At some point though, the topics always came back to modularity and OSGi.

In the time that I’ve been focused on modularity and OSGi, the number of folks that access the content on this site have gone from less than 1000 visitors per month to more than 10,000, culminating in more than 100,000 pages served up and over a quarter million hits to the site each month. The blog was listed as one of the Top 200 Blogs for Developers, a Top Analyst Blog, and many of my posts are syndicated on JavaLobby. While still small by many standards, I recognized progress. And in general, the feedback I’ve received from the community is positive, though there is no doubt that at times, I’ve struck some nerves.

For those that read my posts, you already know that most tend to span many paragraphs. They require extensive writing time, considerable editing time, and careful review. It’s taken me hours to author many of them. And all of the content on this site has always been licensed under a Creative Commons Attribution-Noncommercial-Share Alike license. I vowed to never include any noise on the site, such as Google AdSense. It’s always been an earnest attempt to help developers improve the systems they create. A labor of love, you might say.

A while back, I introduced the modularity patterns, but didn’t elaborate much on the patterns themselves. Alas, that time has come. I’ve quickly realized that I do not have the bandwidth to continue with my long-winded posts and write a book. I’ve decided to pursue the latter and am hopeful that a book on modular architecture will have a far greater, longer lasting impact that serves as a capable ambassador for modular architecture. So while my blog may be silent, I have every intent to continue my advocacy of modular architecture, albeit in a way that I hope is more impactful.

I’ve decided to adopt an open and transparent approach to writing the book. The book’s website illustrates the book in it’s current form - a rough and incomplete draft to be sure. But I hope you’ll take the time to check it out, and offer any feedback you might have. I’ve created a Reviewers page that provides some guidance on the type of feedback I hope to receive. Also, be sure to follow the updates on Twitter.

By the way, this does not mean that I’ll necessarily cease my blogging activities altogether. In the past, I’ve published some articles to the APS Blog. Soon, the APS blog will also be shut down, and we’ll begin to blog on the Gartner Blog Network (GBN). So be sure to checkout GBN on occasion!

Thank you!

Recently, I questioned whether OSGi and modularity would succeed in penetrating the enterprise. But what I really meant to question is whether OSGi will have the disruptive impact of which it’s capable. I asserted that if OSGi does succeed, it won’t be based on the technical merits of OSGi. It’ll be because of something else. Something trendy. Something fashionable.

Now, I could be wrong. OSGi adoption is certainly increasing, albeit slowly. And as case studies begin to emerge that tout the cost reduction, improved responsiveness, and time-to-market advantages of OSGi, adoption will likely continue to rise.

But adoption is one thing, disruption another, and I still have this nagging sensation that serves as cause for pause. What if something trendier, more fashionable surfaces, and OSGi is pushed into the backwaters? Will it really have the impact it’s capable of? You know…an “iPhone-esque” impact that raises the bar and redefines an industry. Not just evolutionary, but truly disruptive.

OSGi As Enabler

For OSGi to cross the chasm, it must enable something big that a business wants to buy. Maybe cost reduction, improved responsiveness and time-to-market benefits will be enough. But that’s easily perceived by many as a rather evolutionary impact. Not really disruptive.

Quite possibly OSGi will flourish in the data center, as organizations seek more adaptable platforms that lend them these benefits. But this doesn’t necessarily guarantee that development teams will leverage OSGi to build systems with a modular architecture. Because leveraging a platform built atop OSGi is separate from building modular software systems, even though OSGi enables both.

Without something trendy that promises real benefits, it’ll get brushed under the carpet like what has happened to many technologically superior solutions. Perhaps it’s already happening, with all of the hype surrounding the cloud. Or maybe it’ll be the cloud that helps OSGi cross the chasm. OSGi doesn’t obviously enable the cloud, but OSGi can enable the dynamic and adaptive runtime benefits of the cloud.

But again, this doesn’t mean that teams will begin leveraging OSGi to design modular software. It only means that the platform itself is adaptable. Of course, there is benefit in that. But if that’s the route that is taken, teams will still continue to develop monolithic applications that lack architecturally resiliency, and the full benefit of OSGi (and modularity) will not be realized.

The Disruption

OSGi has the potential to have have a much broader impact, affecting everyone from the developer to those working in the data center.  So what might this trend be that will propel OSGi to stardom?

OSGi enables ecosystems!

Now, you’re thinking I’ve gone on the deep end, perhaps? But give me a chance…let me explain.

A Bit of [Recent] Platform History

To start, I want to take a brief walk down memory lane. Not too far back though, but far enough so that we can see how important the ecosystem is in today’s most successful platforms. And these platforms span a range of markets, from mobile to social media. But each are successful in large part due to a thriving ecosystem.

Apple

In 2007, Apple released their first generation iPhone. Without question, the device revolutionized the mobile phone industry. While the device offers a great user experience that has certainly played a role in its surging popularity, people flock to the iPhone today because of the wealth of applications available. Yeah, “there’s an app for that.”

Apple, recognizing the power of this ecosystem, now delivers the iPad. With fewer preinstalled applications than the iPhone, Apple is counting on the ecosystem to drive adoption. The more consumers who flock to the device, the more developers who flock to the platform to deliver applications. As more applications become available, consumers will buy more iPads. The ecosystem fuels itself. Apple has simply provided the environment for the ecosystem to thrive.

Eclipse

In 2003, the Eclipse team was thinking of ways to make Eclipse more dynamic. Their decision to use OSGi to create a rich client platform that supports plug-in architecture was the first step toward the resulting ecosystem we know today. One of the reasons developers use Eclipse is because there are an abundance of plug-ins available that allow them to do their jobs more effectively. Other developers create Eclipse plug-ins because Eclipse is a popular IDE used by many developers. Again, the ecosystem fuels itself. The Eclipse team provided the environment that allows today’s Eclipse ecosystem to thrive.

If you’re interested, you can read more about the history of Eclipse and OSGi.

Hudson

Arguably, Hudson is today’s most popular continuous integration server. But it hasn’t always been. Before Hudson was CruiseControl. And while CruiseControl did help development teams get started on their path toward continuous integration, it was also unwieldy to use in many ways. With Hudson’s plug-in architecture, developers have the ability to extend the tool in ways the original creator couldn’t imagine or couldn’t find the time to do himself. Kohsuke created Hudson and gave the development community a new platform for continuous integration. With its plug-in architecture though, he also provided an environment that allows the Hudson ecosystem to thrive.

Social Media

Facebook. MySpace. Twitter. Each are examples of social media tools with a strong developer community that creates extensions to the platform that users can leverage to enhance the experience. Facebook Developers. MySpace Developer Platform. Twitter API. Each allows the ecosystem to thrive.

And Others

These are just a few examples. It’s easy to find other platforms with similar ecosystems, as well. The ease with which Wordpress themes and plug-ins can be developed and used to enhance a Wordpress website is another example. In fact, many content management systems provide similar capabilities. A large reason why the Firefox web browser has emerged as the preferred web browser is the ease with which add-ons can be installed that extend the capabilities of the browser. The Atlassian Plugin Framework is another example that uses OSGi, and platforms such as Force.com and SharePoint have built (or are trying to build) similar ecosystems.

The Power of Ecosystems

Aside from Eclipse (and Atlassian), none of these other platforms leverage OSGi. Yet each are wildly successful because of two reasons:

• An environment was created that allowed an ecosystem to form and flourish. This environment includes a platform and a marketplace.
• A group of customers and developers converged on the marketplace and fueled growth of the platform. The result is a self-sustaining ecosystem.

If you look at many of the more popular platforms that have emerged over the past decade, they tend to possess a similar characteristic - a community of individuals dedicated to providing great solutions leveraging the foundation of the platform. OSGi and modularity enables ecosystems on the Java platform.

Ecosystems and the Two Faces

I’ve already talked about the two faces of OSGi - the runtime model and the development model. I’ve also explained how one could possibly see widespread adoption while the other has little impact. A strong ecosystem surrounding OSGi and modularity must leverage both. Developers would create reusable modules, implying they are designing modular software. For development teams to leverage these modules, they must be using a platform that supports the runtime model.

CBD Has Already Had Its Day, You Say?

Now some of you might be arguing that this sounds a lot like the component based development (CBD) fad of the 1990’s. True…to an extent. Certainly these ideas are not new. But there are also some striking differences between that which OSGi enables and the CBD fad that has come and largely gone, or whose promise was never fully realized.

Foremost, the CBD fad was focused almost exclusively on visual components, such as ActiveX. While some attempted to create components for the Java platform, the movement largely failed to go mainstream. Instead, Java EE grew in popularity and for a number of years, garnered everyone’s attention. Why did this  happen?

IMHO, the answer is fairly simple. Even though numerous marketplaces emerged that allowed the consumer and producer to come together to buy and sell components, there was never a suitable component execution environment. That is, an environment that would support dynamic deployment, support for multiple versions, dependency management, and, in general, complete control over all components currently executing within the environment. ActiveX components did have an execution environment (though did not support each of these capabilities), but Java did not. Today, in OSGi, Java has the requisite execution environment!

The Ecosystem

It’s easy find holes in this idea. To explain why it cannot work. Yet, it’s happening elsewhere, so why not on the server…in the enterprise? Certainly there are a variety of different ways such an ecosystem could manifest itself. Possibly multiple ecosystems emerge like what we see in the mobile market today.

But for a moment, imagine the world where you have the ability to easily assemble a platform from pre-built infrastructure modules that exactly meet the demands of your application. You might purchase these modules, you might choose to use open source modules, or you might build them yourself. For those you don’t build, you obtain from a module marketplace, possibly deploying them to your [cloud] environment.

And when you choose to use a module, it’s dynamically deployed to your environment. The modules it depends upon? You’re given the option to purchase and deploy them, as well. You develop your software modules using the sound principles and patterns of modular design to ensure loose coupling and high cohesion. As you roll out your business solution modules, you simultaneously deploy the additional infrastructure modules that are needed.

In this marketplace, modules are sourced by multiple vendors. Some large. Some small. Neither the stack, nor your applications, are monolithic beasts. Instead, they are a composition of collaborating software modules. Your infrastructure isn’t necessarily tied to a specific vendor solution. The option always exists for organizations to purchase modules from different providers, easily swapping one provider module out with another.

The ecosystem flourishes. Developers flock to sell their latest creation. Organizations seek to add amazing capabilities to their rightsized environment at a fraction of the cost compared to what they are accustomed to today. The business benefits are real. The technical advantages are real. And the resulting ecosystem is sustainable.

A successful ecosystem demands both the runtime model and development model. And today, OSGi is the only standard technology that will allow this type of successful ecosystem to form on the Java platform. But will it happen? We may have a ways to go, but it sure would be cool! And it would be a shame if we lost this opportunity.

Thoughts?

Note: If you’re interested in ecosystems more generally, you might want to see the great TED talk by Dan Barber, “How I fell in love with a fish.” It’s very entertaining, informative, and worth 20 minutes of your time. His discussion on sustainability is truly fascinating!

Image Source: http://en.wikipedia.org/wiki/File:Blue_Linckia_Starfish.JPG

It seems my recent post on OSGi has ruffled a few feathers. I’ve also received a few personal e-mails suggesting that I have missed the mark because OSGi is a significant component of the next generation application platform. Major vendors such as IBM are building OSGi into their products, and other leading edge vendors leverage OSGi as an enabling technology upon which their technical architecture is built.

My intent wasn’t to indict OSGi, so I was pretty surprised with the defensive responses I received…it’s a great technology and the benefits of modularity are real. As I’ve mentioned, and as many already know, I’m an ardent supporter of OSGi and modularity.

Based on the type of the feedback I’ve gotten on my previous post though, I’ve come to the realization that various constituencies have different perspectives that is contributing to the perception of OSGi adoption. Let me explain.

Differing Perspectives

In my post, The Two Faces of Modularity & OSGi, I talk about the runtime model and the development model. There are significant advantages to each. But interestingly, it’s possible for one to succeed while the other never sees widespread adoption. That is, OSGi might lie at the core of the next generation platform, but it doesn’t necessarily imply that enterprise developers will leverage it to build applications with a modular architecture.

Today, most vendors emphasize the runtime model, touting the cost savings, improved responsiveness, and flexibility of a dynamic platform. So I can understand why a lot of folks have reached out to explain to me the error of my ways. From their perspective, OSGi is flourishing.

An excellent example is the Eclipse platform, which released Eclipse 3.0 in June of 2004. However, I’d wager that only a fraction of Java developers using the Eclipse IDE know that OSGi is the major enabling technology upon which Eclipse is built. Instead, they use Eclipse, install plug-ins, but know very little about the underpinnings of the platform.

From a development perspective I’m incredibly interested in the benefits of modular architecture, and I want tools that reduce the accidental complexity of the development model. However, these tools don’t exist today, and developing enterprise server-side Java software applications leveraging OSGi is painful. So while vendors are leveraging OSGi for various platform benefits, as long as they encourage the traditional deployment model and the dearth of tools persist, enterprise developers will not use OSGi to build applications with a modular architecture. We don’t have it today, and we won’t see it in the future until this changes.

Parting Thoughts

I do believe that OSGI has the potential to be a major component of the next generation application platform. Though there are no guarantees, and I still have concerns. Let’s take SpringSource as an example. After seeing almost zero adoption of dm Server, they donated it to the Eclipse Foundation. And the recent VMForce announcement makes nary a mention of OSGi. Instead, tc Server provides the execution environment. So while some might argue that the runtime platform benefits of OSGi are real, there is a far more significant trend called the cloud that provides many of the same advantages. And the cloud certainly doesn’t need OSGi. Though OSGi could thrive in the cloud.

I’m also not convinced that the benefits of modularity will be realized by enterprise development teams. Will enterprise developers leverage OSGi and modularity as a core component of next generation architecture? I hope so, but I don’t believe it’s the slam dunk that many believe it is. Modularity is important. The benefits are real. It helps teams overcome a serious challenge when developing large enterprise software systems. But we need more developer advocacy surrounding OSGi and the benefits of modular architecture. Software is a fashion industry, and we must attach modularity to something trendy and more aggressively advocate adoption.

It is what it is. And unfortunately, that’s just the way it is.

Developing enterprise software is hard work. Bureaucracy. Large teams. Legacy systems. The list of challenges is immense. But enterprise development can also offer tremendous opportunity and reward. It can even be fun! I hope that at some point, you have the opportunity to experience this feeling.

The Exciting Enterprise

I worked here once. In the exciting enterprise. Working here was different. It was actually exciting. What made it different? Well, it was the process. Exciting process? What’s that? It’s the process that makes software development fun again, productive, meaningful, and satisfying.

So what kind of process is this, you ask? Iterative? Agile? RUP? XP? Scrum? Kanban? Hell no! We never talked process lingo. We didn’t care how agile people thought we were. We didn’t want to get caught up in the bureaucracy and political mayhem surrounding software process improvement. There was work to be done. That’s what we talked about. And then we did what we needed to do to get it done.

Not Always Easy

Working in the exciting enterprise wasn’t always easy though. It required stamina, determination, and discipline. There were lots of people who didn’t appreciate how we went about our business. We didn’t submit weekly status reports. We didn’t have the Gantt chart. Actually, we didn’t have a lot of things. But we had what we needed. To us, it all seemed like common sense. Of course, we had a very experienced team, and we knew what worked and what didn’t.

Plan driven? Predictive? Estimates? Models? Oh yeah…we had it all. We had a 18 month project road map. It was quite a plan. Took a small group of us about 1.5 days to develop. All on a single spreadsheet. Showed all the major systems that we’d retire right along with the new functionality that was going to come online. Probably could have gotten another three months of project time if we wanted to do more planning. But we had code to write.

Oh That Build

Code? But what about the requirements? Yep. Gathered them as we went. And when developers had questions, they asked them. Right to the customers face too. Blasphemy! The business analyst working with the developers. Those BAs were awesome too. They worked real hard to clear any confusion. Made sure developers always had what they needed. And we got ‘em as quickly as we could handle them. A steady stream of requirements flowing in and right back out as an executable piece of software.

Oh sure, it wasn’t always easy. We had lots of important checkpoints along the way to make sure we were on the right track. Weekly checkpoints. Daily checkpoints. Hourly checkpoints. Developers and customers sitting in a room together to see what we’d gotten done the past week. Continuous deployment to an environment where folks could actually use the software. Hourly builds that made sure we never strayed too far from a working system.

I still think about that build. Actually, it was more than a build. It was a piece of frickin’ art. That build was the coolest piece of process I’ve ever seen. It was staged. It was fast. It did pretty much anything we asked it to do. It never got in our way. It just went on it’s merry way, building our system. Hourly. Every hour. Automatically. It was the glue that held the team together as we grew in size from a fledgling crew of six developers to upwards of 100 at times.

We protected that build. When somebody cause it to fail, they’d feel the wrath. Eventually, they all grew to love what it could do. Honestly, what choice did we have if we wanted to ensure we could have weekly checkpoints throughout the process? The only way to pull it off was to emphasize software that works. It always had to work. ALWAYS! Guard the source. The prime directive - software that works!

The Source

Now don’t get me wrong. We had documentation. We had lots of documentation. Some of it was pretty nice looking stuff too. But we weren’t afraid to let it go. Let it serve it’s purpose and then move onto what was real important…the source code. What else really matters, anyway? All the pretty documents, models and plans don’t amount to a hill of beans if the source code isn’t provably correct. That was our focus. It was the teams focus. And we guarded that source code very closely.

Improving the Right Process

I look back now. An 18 month project. A team that clicked. Not just a development team. A team. Including the customers. Honestly, we didn’t just build a piece of software, though. We improved the process. We automated it. We optimized it. No, I’m not talking about the software process. I mean the business process! That’s what made it so cool…

Granularity is the extent to which a system is broken down into it’s behavioral entities. Coarse-grained entities tend to be richer in behavior than fine-grained entities. Because coarse-grained entities do more, they tend to be larger than fine-grained entities, and are easier to use. In general, a fine-grained entity is more reusable than a coarse-grained entity because it does a little bit less. This is pretty natural. If it does less, then it’s more likely to apply across a wider variety of usage scenarios. I spent some time discussing this tension in Reuse: Is the Dream Dead?

Unfortunately, while there are a lot of patterns, principles, and idioms that offer really good guidance surrounding many aspects of architecture and design, there isn’t a lot of guidance surrounding granularity. Let’s start with a really simple example that illustrates the challenge.

A Really Simple Example

Consider the following save method:

public class Person {
   private String firstName;
   private String lastName;
   private String ssNumber;
   public void save() {
      ValidationErrors errors = this.validate();
      if (errors != null) {
         //handle the errors somehow
      } else {
         //save to the database.
      }
   }
   private ValidationErrors validate() {
      //perform validation
   }
}

As can be seen, the save method invokes the validate method before saving the information to the database. Generally speaking, this makes saving the information easier (and possibly safer) since whatever invokes save doesn’t have to worry about validating. And this seems to be a good thing.

Yet, what happens when a new consumer of this method wants to save the information but apply a slightly different set of rules than the current validation method provides? Here’s where the existing save method is simply too coarse-grained. It does just a little bit too much. We’ve made it easier to use, but also less reusable.

There are quite a few variations I could go with at this point to improve the design. One would be to make the validate method public, and have clients invoke the validate method before invoking save. Then, if validate didn’t do what it was supposed to, clients could always opt out of invoking the validate method and apply their own validation before invoking save. Certainly less safe though because validation is optional.

Yet another alternative might be the following:

public class Person {
   private String firstName;
   private String lastName;
   private String ssNumber;
   public void save(Validator validator) {
      ValidationErrors errors = validator.validate();
      if (errors != null) {
         //handle the errors somehow
      } else {
         //save to the database.
      }
   }
}

This seems to offer the best of both worlds. I now require a Validator be passed in before I can save, and if the Validator is an interface, I can certainly allow clients to pass in the Validator they need. Of course, I could easily pass in a NOP validator.

Whatever…there are lots of different options, and each has their own set of tradeoffs. I’d like to avoid extensive debate surrounding which approach to save and validation is best and instead focus on the main point here, which is that achieving the right level of granularity for an entity can be very challenging. It requires more than just looking at the problem from a code level viewpoint and demands that we possess contextual information that will help us answer the question, “What’s the right level of granularity?”

Bring It Up a Level

The code above was a pretty simple example that illustrates the challenges of granularity at the method level. But the same challenges exist when developing classes, packages, modules and services. I discussed another, much higher level example, in the post on architecture all the way down. So what I really want to know is, “How do I determine the appropriate level of granularity?”

This is the million dollar question. Because granularity is a significant inhibitor to creating reusable software that’s easy to use. (Oh, and managing dependencies too.) If something does too much, it’s less reusable. If something does too little, it’s more difficult to reuse.

In the past, I’ve used the diagram at right (click to enlarge) to help illustrate one view of granularity. As can be seen, services are more coarse-grained than modules which are more coarse-grained than packages which in turn are slightly more coarse-grained than classes. This begins to help answer the question, “What’s the right level of granularity?”

If I’m concerned that a service I create is simply too coarse-grained and fails to maximize its reuse potential, I can break the behaviors of the service out into modules that are a bit finer grained and more reusable (of course, one might consider doing this in general). Then I can compose the services from the modules and reuse the modules across services. The result is different entities at different levels of granularity that lends tremendous flexibility to how I compose, use, and reuse software entities.

This provides some guidance on the level of granularity at which different types of software entities should be defined. However, it still doesn’t offer enough guidance to determine the right level of granularity for the save method in our example above.

Another Dimension

There’s another dimension that is relevant, and that ties in nicely (though in a rather subtle way) to the initial coding example. It has to do with the way we traditionally layer our software. The diagram at right illustrates this (click to enlarge). As we can see, as we move down the layered hierarchy, entities become finer grained.

While this isn’t absolute, and depends on your architectural principles and style, entities in higher level layers tend to be more coarse-grained than entities in lower level layers because they are an amalgamation of their own behavior and the behavior of entities in the lower level layers.

Armed with this information, we can determine the right level of granularity for the save method as long as we understand in which layer the code containing the save method lives. If it’s in the data access layer, and we place architectural constraints on business rules living in the data access layer, then we shouldn’t have any validation code that lives in the data access layer. If the save method lives in a class in the domain layer, however, it may be suitable for the save method to contain validation code in the domain layer and invoke another method in the data access layer that actually performs the save operation.

The Complete Picture

At this point we can make the following general statements:

• Entities in higher level layers are more coarse-grained than entities in lower level layers.
• Services are coarser than modules which are coarser than packages which are coarser than classes.

And if we combine these two ideas into a single thought, and overlay the two diagrams, we can begin to visualize the level of granularity for different types of software entities.

Certainly this doesn’t offer hard guidance. Realistically, there are very few architecture and design principles and patterns that are universally applicable. In fact, I’m not certain you’d ever create a “presentation service.” But certainly you might create a data service that is composed of multiple data access modules. And you might have separate data services that reuse a data access module. And you might have a business service that uses a data service and is also composed of multiple domain modules. But you definitely do not want a data access module that invokes a business service, nor a business service that references a presentation module.

So this general guidance can serve as useful information to help answer our question, “How do I determine the appropriate level of granularity for a software entity?” And it can also serve as guidance when establishing architectural principles and constraints that help determine where software entities with specific behaviors should reside. Because without some scheme that will help determine the appropriate level of granularity, it’s hopeless to imagine that we’ll be able to design reusable software entities that are also easy to use.

I’d like to take a short moment to offer an additional perspective to my discussion on OSGi: Complexity is NOT the Problem. I believe this perspective adds clarity to that previous discussion, as well. All initiated thanks to a tweet, which summed up the situation in much less than 140 characters.

So over the weekend, I turned to the essay that said tweet refers, and reviewed the essence and the accidents. From the article, I quote Mr. Brooks:

“…to see what rate of progress we can expect in software technology, let us examine its difficulties. Following Aristotle, I divide them into essence - the difficulties inherent in the nature of the software - and accidents - those difficulties that today attend its production but that are not inherent.”

This leads me, pretty clearly, to the following simple conclusion.

Today, development teams leveraging OSGi to build server-side applications are burdened by accidental complexity. As platforms and tools mature, the accidental complexity of OSGi will be reduced to near zero. Modularity though, enabled and enforced by OSGi, attacks the essence.

If the difference between essential and accidental complexity isn’t quite clear, I highly recommend you take a few moments and read Mr. Brook’s essay (linked above). And then, if you can find a copy of the book’s 2nd edition, take a look at Chapter 17, “No Silver Bullet” Refired (Note: If anyone can find the Refired essay online, please post the link in comments).

Throughout these two discussions, you’ll find subtle hints extolling the virtues of modularity. But only do this if you’re willing to exercise your brain with thought, because the connection you’ll discover might just be transformational!

The editors at SD Times have proclaimed that OSGi is too complex. Unfortunately, they miss the mark, and the following statement taken from the article is misleading.

We believe that OSGi, rather than simplifying server application development, has actually made it more complex, and there aren’t sufficient benefits to justify the added complexity.

This statement should have been tempered with context. It is not universally applicable, though is made to sound as if it were. OSGi may face challenges, but complexity of the OSGi framework is not one of the long-term inhibitors to adoption.

Technology Adoption Lifecycle

All technology goes through various phases of adoption. A lack of platform support and tooling may be inhibiting adoption today, but that is today’s problem, and one that is being dealt with. And there is a stark difference between complexity brought on by lack of tooling and platform support versus complexity associated with an ill-conceived idea. OSGi suffers today from lack of enterprise-class tooling and platform support. OSGi is not an ill-conceived idea.

Whenever an organization adopts a new technology, they’ll undoubtedly face some degree of initial complexity. New technology takes time to learn. As the innovators and early adopters continue to use OSGi, tooling is going to get better and developing modular applications using OSGi is going to get easier.

The question you must answer is whether the pain of adoption today is worth the gain of a modular architecture. There are many factors to consider when answering this question, but as the technology matures, the question will be easier to answer. As I’ve alluded to, other factors that have little to do with OSGi’s technical benefits will likely determine if it’s is able to cross the chasm.

Long Term Value

But the article’s biggest flaw is in stating:

And what’s the benefit again? Enterprise developers have written many, many server-side Java applications without using OSGi.

It’s important to understand its value over time. Let me pose some simple questions.

• Is Spring too complex? It is far easier to create an application with concrete class references than it is to use dependency injection and abstractions.
• Is Hibernate too complex? It is far easier to use JDBC than it is an ORM framework.
• Is unit testing too costly? It is far easier to forego unit testing than it is to create a robust suite of tests.

Yet each contribute to more flexible software systems, and in that, there is significant benefit. So it is with OSGi and modularity, as well.

Leveraging OSGi to design modular applications will require us to learn new concepts, tools, and design paradigms, but there is no doubt that modularity has a tremendous upside. This makes the SD Times article paradoxical. OSGi does not increase complexity; the very benefit of modularity is that it helps us manage complexity.

More on Benefits

If you’re interested in exploring some of the benefits of modularity, you can start with the following:

• Architecture all the Way Down - The important role of modularity in architecture.
• Modularity by Example - A simple visual example illustrating the benefits of modularity.
• Agile Architecture, Lean Principles - Comparing my past thoughts on agile architecture to the Lean Principles of Software Development.
• Modularity & Architecture - A response to the entry on eliminating architecture.
• Eliminate Architecture - Discusses the goal of architecture and how to eliminate the impact and cost of change.
• Agile Architecture - My views on agile architecture and the natural architectural shifts that occur throughout the development lifecycle.
• Agile Architecture Requires Modularity - Discusses the role of modularity in agile architecture.
• On SOLID Principles and Modularity - Discusses where you need flexibility in architecture.
• Two Faces of Modularity and OSGi - Introduces the need for patterns and tools to help design more flexible and modular architecture.
• Reuse: Is the Dream Dead - Discusses the tension between reuse and use.
• Modularity Patterns - Presents 19 modularity patterns that help ease the tension between
  reuse and use while making a software system easier to understand,
  maintain, and extend.

I want to be careful here, but also very clear - I advocate OSGi and modularity, though am concerned about its future. In some circles, OSGi is hailed as a disruptive technology; in others, it lives in relative obscurity. There are some indications that adoption is growing, and possibly at some point, it’ll catch like wildfire. But right now, OSGi is the most widely acclaimed technology that nobody is using.

Maybe I’m too harsh. Or too impatient. Maybe I need to give it more time. I know some will argue that people are using OSGi, and they’re right. Some are. But of the more than 6 million Java developers worldwide, those using OSGi and designing modular applications is only a very small fraction.

So what’s the problem? What’s needed to drive adoption? How will OSGi and modularity succeed?

What’s In a Name?

One of the greatest inhibitors to adoption has been platform support. Until recently, few platforms exposed the virtues of OSGi to the enterprise developer. That’s beginning to change, and should help drive adoption. Another obstacle is tooling, which I discussed a while back when lamenting that one of the biggest barriers to adoption is no migration path. With better tools, we’re beginning to overcome that challenge.

But will it be enough? Will platform support and great tools help drive adoption? Sadly, possibly not. And part of the problem, I believe, has very little to do with the technology, and much more to do with its image.

So, what’s in a name?

OSGi is the dynamic module system for Java!

Modularity - Old and Stodgy

Recently, I commented on a recent conversation I had with a conference organizer and his reluctance to accept my speaking proposal on modularity. The general sentiment was that it’s an old, boring concept that can be traced back 40 years! He was interested in my talk on agility, but I explained that given the choice, I’d prefer to speak on modularity.

Currently, I’m participating in a similar conversation with the organizers of the Agile 2010 conference on a few sessions I’ve proposed on modularity. And now, Peter writes about a similar experience. This sends an important message to the OSGi community, and highlights an important challenge surrounding modularity. The message: Modularity is boring! The problem: The importance of modularity, and its support as a first class citizen on the Java platform, is not well communicated nor understood.

Not a Technology Problem

This is a serious problem that requires attention. In his post, Peter suggested that micro-services may be a more apt term for OSGi services, so as not to confuse the term with web services. He even hints at proposing a new JSR that would introduce the idea of micro-services as a new primitive on the Java platform. This is a good start, and micro-services isn’t a bad idea, but more needs to be done. The problem is not that OSGi isn’t a great technology with real benefits, the problem is that nobody cares about modularity.

Possibly it isn’t that they don’t care, but instead that modularity won’t sell. Nobody wants to buy a 40 year old technology, they want to buy the next great thing. And without question, developing modular software is not a new idea. But in general, the ideas surrounding SOA were not new either. Yet, without question, SOA has garnered significantly more attention than OSGi over the past decade. SOA has had its day in the sun; OSGi and modularity have not.

This is not a technology problem. The real problem is that OSGi doesn’t have a clearly articulated identity that explains why an organization must be using it, and organizations will adopt OSGi only if its adoption can be tied to real business benefits. Just like object-orientation in the 1990’s and SOA in the 2000’s, each were sold with a wonderful accompaniment of benefits that would reduce cost, speed delivery, model the real world, blah, blah, and blah. Whether they did or did not help organizations realize these benefits is arguable. Whether each were widely adopted is not!

Crossing the Chasm

OSGi has the potential to be this decade’s most disruptive technology. Unfortunately, the virtues of OSGi run the very real risk of never coming to fruition if it’s unable to cross the chasm. And sadly, it can never cross the chasm unless we find an effective way to sell, or reinvent, modularity.

Foremost, OSGi, modularity, and micro-services do not represent the paradigm shift that’s necessary. Too often, we talk about OSGi in terms of its technical merits alone. Unfortunately, that’s not enough. If OSGi is to cross the chasm, it is going to require something different. Something that people can really latch onto. Something that gets them excited. Modularity isn’t enough. Nor is plug-in architecture. Nor any other technically superior thing that OSGi is.

Naturally, this is only my perspective. My opinion. But OSGi has been a widely acclaimed technology for several years, and it has achieved very little enterprise adoption. While the move by SpringSource to donate dm Server to the Eclipse Foundation was hailed as a great contribution to open source, it’s also a clear indication that adoption of dm Server was gaining little traction.

Now, it is possible that an 800 pound gorilla could throw their weight behind OSGi and modularity that will help drive enterprise adoption. Maybe IBM’s move to expose the virtues of OSGi to the enterprise will be enough. With other vendors following suit, and up and coming vendors such as Paremus on the rise, that just might be enough. But if it is enough, I still believe the enterprise isn’t going to buy OSGi and modularity for the reasons we believe they should, they’ll buy something else very trendy in which OSGi and modularity plays a central role. Something new. Something fashionable. Because as I recall Ivar Jacobson alluding to at a keynote a couple of years ago,

The software industry is more driven by fashion than the fashion industry.

Sad. But true! And hopefully when it happens it won’t dilute the value of OSGi and modularity.

Ignore my Rant if You Want

OSGi is a great technology solution. But we also recognize that technologically superior solutions often fail to win. One of the most popular stories is of the video tape format wars. So my little pontification here should in no way be interpreted as questioning whether OSGi is capable, but instead whether OSGi will.

You can choose to ignore me if you’d like, scoff at me, or call this a rant. Maybe I’m just impatient. Either way, I’ll still continue to proclaim the virtues of OSGi and modularity. I believe in it…firmly! But the nagging feeling I have surrounding its ability to cement its place in the world of enterprise software development will continue to tug at me. And I wonder…will it ever succeed? And if so, how? And…when?

What do you think?

Often times, I hear folks exclaim that they’ve been on “40 development projects over the past 10 years.” Or, “70 projects spanning a 20 year career.” They say this as if it’s some badge of honor. But I’m not so sure that it is.

Of course, there is value in widespread project exposure. Gaining experience with new technologies. Experiencing the dynamics of different teams. Understanding the challenges surrounding different organizational cultures. Each project is unique, bringing its own set of experiences, and these experiences are all incredibly valuable.

But when I see that someone has jumped from one project to another, it also leaves me wondering! Have they ever stuck around long enough on any single project to see their decisions through to the end? Have they ever had to live with the decisions they’ve made? An amazing compilation of knowledge is obtained by not only participating in the early stages of development, but also in maintaining the software system after it’s been released. To name just a few:

• Dealing with the challenges of Phase 2, while also maintaining Phase 1.
• Managing multiple branches of the software system. Merging those branches back together. Creating new branches.
  
• A critical production issue arising about the same time you need to start heading out for that important customer demo.
• Attempting to change a piece of code that hasn’t been touched in 10 months.
• Watching the software system grow from nothing to a system that’s more than 500,000 lines of code. Keeping the build performant. Keeping the build working!
• Being forced to live with seemingly meaningless early design decisions that haunt you over time.
• Trying to upgrade versions of third party libraries while development is ongoing. Simply recognizing the right time to upgrade.

And there is so much more. Priorities tugging you in five different directions at one time. If you’ve had the luxury of living with a system (and the mistakes) you created, you’ll realize that there are very few, if any, decisions that shouldn’t be given conscious thought.

When an individual sticks with a project for a long time, they realize the importance of maintaining a clean design, a robust suite of unit tests, and how they package their software system. There is significant knowledge gained by sticking with a project for a long period of time. Perhaps the next time you hire a developer, it might be wise to ask them, “What’s the longest you’ve spent on any given project?” That may be more important than the number of projects they’ve been on.

Somewhere along the way, I picked up a small piece of advice that has helped serve as a valuable guide. There is a big difference between ten years of experience and one year of experience ten times. When you jump from project to project, never living with the decisions you’ve made, you have lost an opportunity to learn what works well and what does not.

I’ve owned Macs for about four years now. I use them daily. One of the biggest complaints I hear surrounding Macs is that they’re more expensive than their PC counterparts. This recent article in CIO examining total cost of ownership (TCO) for businesses using Macs left me wondering. For personal use, are Macs really that much more expensive than PCs, especially when factoring in TCO? Unfortunately, it’s not so black and white. Let’s look at some numbers.

For this little exercise, I’ve decided to compare sample TCO over a period of time (ie. 2 years) for a Mac and a PC. This includes the initial purchase cost, ongoing support costs, and software purchases. It does not include any personal time that you would need to devote to maintaining and managing the device, such as reinstalling the operating system, cleaning up unused applications, dealing with anti-virus software and configuration, and general troubleshooting tasks.

I’ve made my most valiant effort to present an unbiased view and have arguably given favor to the PC. It would have been easy to close the price gap much more quickly if I were to toss in a few not-so-easy to measure items, such as the likelihood that you’ll have to take your PC to a professional to have it saved. But I’m sticking with hard numbers; the stuff that’s easy to measure. Let’s get started.

The Initial Purchase

I picked two models, fairly equal in their capabilities - a Dell Inspiron 15 and a 15″ MacBook Pro. When purchasing the Dell, I had to configure it so it had specs equal to the Mac. Let’s look at purchasing and configuring the Mac first.

The MacBook Pro

The 15″ MacBook Pro we’re going to purchase comes already loaded, so we aren’t going to make many customizations to it. We’re going to go with the entry level model, which prices out at $1699. The general specs include a 2.53 GHz Intel Core 2 duo processor with 4 GB of RAM. All we’re going to do is increase the hard drive to 320 GB, which adds $50 to the price, and purchase iWork, which adds $49 (Pages, Numbers, and Keynote). For those not familiar with iWork, this is what we’ll use on the Mac instead of Microsoft Office.

Total price for the MacBook Pro: $1798.00.

The Dell Inspiron 15

Keep in mind that the Inspiron is under Dell’s “Home” category. What this generally means is that it comes pre-loaded with a whole bunch of useless trial software that expires in either 30, 60, or 90 days. After that initial trial period, you’ll have to purchase a license to continue using it. For a business, you’d probably purchase the Latitude to avoid all of this unnecessary garbage. To keep the price down, I’m going to go with the Inspiron 15. See…told you I was giving favor to the PC!

To get a somewhat even comparison, I’m going to have to customize this thing to bring it up to equal specs as that of the Mac we’re going to purchase. The initial cost of the base machine is $499. But without any further customizations, that machine doesn’t offer much horsepower. First, I’ve got to add a Core 2 duo processor clocking in at 2.53 GHz. This adds $225 to the price.  We do need the wireless -N card, which adds $40. Why would this not be standard? Odd!

Next is Microsoft Office Home and Student 2007, which adds $119. We need this if we want to create spreadsheets, documents, or presentations. I recognize I could have gone with OpenOffice, but I find that most people choose Microsoft Office, so that’s what I’m doing here. Since we’re running Windows, we need anti-virus software. With the Dell model, there’s an option to purchase a 36 month license of McAfee. I’m going to jump on that because it only adds $40 to the price.

At this point, we’re finished with the Dell. There are a few things we could have done to even out the playing field a bit more. For an additional $45, we could have upgraded to the 9-cell battery that would have given us up to 8 hours of battery life. Keep in mind the Mac has a battery that gives us about 7 hours. Additionally, the version of Microsoft Office we purchased doesn’t come with Outlook, so we’ll still need to find an e-mail client once we get our laptop. But I just couldn’t see spending another $160 to purchase the Small Business edition that comes equipped with Outlook. The Mac will have Mail.app preinstalled for us, which is a nice e-mail client with Exchange integration. Already, the cost of software licensing is a sign of things to come that’s going to factor heavily into TCO.

Total price for the Dell: $883.

The Software

While these are each good machines, to get great things done requires software. Each of us have different needs surrounding the software we use, so I’ll offer up a few samples based on what I use. It won’t take long to get a clear understanding of the significant impact this has on TCO. Choose your own software stack, and see how it adds up. Then, let me know if you’re findings are consistent with mine.

First, I want to purchase software that allows me to create some nice drawings. For the Mac, I’m going to purchase OmniGraffle, which comes in two versions, standard ($99.95) and professional ($199.95). For Windows, Visio is a good option, which also comes in two different versions, standard ($259.95) and professional ($559.95). Let’s go the cheaper route on each. This brings the total price for the Mac to $1897.95 and the price of the Dell to $1082.95. Note: If I had purchased the professional version of each, the TCO at this point would be 1997.95 and 1382.95, respectively.

Next, I want some screen capture software so I can create some short training videos for online publication. For this, we’ll use Camtasia. The Mac edition of Camtasia prices out at $99.00, and the Windows version comes in at a rather lofty $299.00. TCO for the Mac is now at $1996.95 and the Dell is at $1381.95.

See a pattern developing here yet?

Keeping this post at a reasonable length, I’m only going to make one more purchase. At some point throughout my time as owner of one of these wonderful products, it’s likely that a new version of an operating system is going to arrive. And I’m going to want to upgrade. When Snow Leopard for Mac hit the market, the cost to upgrade was a paltry $9.95. Alternatively, the price to upgrade to Windows 7 Home Premium comes in around $119.95. In fact, a full license of Snow Leopard is only $29.95. A full license of Windows 7 Home Premium? $199.99! Again, the pattern is developing. TCO for the Mac is now $2006.90, while the Dell is at $1501.90.

Additional Analysis

At this point, the Mac is still slightly more expensive. But an initial price difference of $1200 was quickly dwindled down to just over $500 by the time we were finished. Clearly, the stark difference in the initial price is quite different from TCO. I could easily keep going, purchasing software that I want and need, and each time find that the Mac edition of the software comes in priced far below the Windows version. But I think I’ve illustrated the pattern here. The cost of software for Windows is more expensive than corresponding software for the Mac, and throughout the lifetime of ownership, TCO for Windows will approach, if not exceed, that of the Mac. For instance, Apple offers the option of purchasing a 5-license family pack of iWork for a mere $20 more than a single copy. A 5-license copy of Office is going to be quite expensive ($595).

Additionally, I have not broached the subject surrounding the time you’ll spend maintaining the Windows machine to keep it running smoothly. In four years on a Mac, I haven’t devoted any effort to tuning the machine for performance. It just works. After about six months on a Windows machine, I find that I’m always tinkering with it to maintain optimal performance. Since I can’t put an accurate price tag on this, I haven’t discussed it. But it is significant. Ok, I’m a little biased here, aren’t I?

Finally, I could have gone with the 13″ MacBook Pro with the exact same specs (just a smaller display) as the Inspiron 15 for an entry price of $1448.00. And the 13″ Mac is a great machine. After purchasing the software I want, the price for the 13″ Mac comes to 1656.90. That’s only $155.00 more for the Mac!

The Clear Winner Is…

The winner…you decide! All too often we compare Mac and PCs based on initial cost, failing to factor in the longer term TCO. When comparing Apples-to-Apples (errr…I mean PCs) though, the difference isn’t as stark. If you want a low end machine with a short lifespan, Dell (and many other PC manufacturers using Windows) offer this, including their line of netbooks.

It is possible to purchase a Dell for under $500. But remember, you get what you pay for. And side-by-side, machines of equal horsepower and capability are going to result in similarly equal TCO. By the time you spec out the machines evenly, and factor in TCO, the price of a Mac is arguably equal too, if not less than, that of a Windows PC. Either way, I know what my choice is!

Last week at OSGi DevCon, I attended Peter Krien’s Advanced OSGi tutorial. The tutorial focused on OSGi services, and I had just come off a rather lengthy e-mail discussion with Peter on a similar topic. So this was pretty good timing. I want to share with you one aspect of the discussion, and its relationship to architecture. It’s quite simple, though makes an important point.

In the past, I’ve talked about the importance of designing the joints of a system. OSGi services represent these joints, since services are essentially a module’s published interface. In general, if the implementation details of a module are well encapsulated, then implementation details can change without impacting the rest of the system. In other words, we minimize that nasty ripple affect of change if we confine change to a single module. But changes that span modules is nasty. Let’s take a few visuals to illustrate the point here.

Where’s the Architecture

Commonly, we visualize the module structure of a software system using a simple diagram similar to what’s shown at right (click to enlarge). We see the modules and the relationships between them. Of course, the relationships between modules represent the joints. Pretty simple stuff! Unfortunately, our attention is immediately drawn to the modules themselves, but not so much the joints.

And we really must be much more concerned about the joints than we are the implementation details because changes in the joints of the system are more complex and costly. This is where we need flexibility and it’s also where we need stability. Emphasizing module implementation is important, but flexibility in the joints is more important.

Illustrating the Joints

In the session, Peter annotated the diagram using a new convention, which I’ve depicted in the diagram at right (click to enlarge). The red triangles more clearly illustrate the joints. Here, our attention is drawn more toward the design constructs that span module boundaries. No longer is emphasis placed on the implementation details, but instead has moved to focus on the relationships between the modules. And the direction of the red triangle also helps illustrate the direction of the dependency relationship between the modules. In other words, these red triangles are the services.

The Real Architecture

Now, because the module itself represents behavior that is encapsulated, we can remove the modules from the diagram altogether and show only the joints, as shown at right (click to enlarge). This is where the impact and cost of change is most significant. And this is where we really need to ensure we have the greatest flexibility. These are the decisions that are most architecturally significant.

A Short Digression

Now I don’t think it’s necessarily important to start drawing architecture diagrams this way. In fact, I’ve never felt that it’s critically important to draw these pretty diagrams in the first place. While diagrams (and documentation) can be useful, they are little more than a pimple on software development’s ass. There are much more important things to worry about than diagrams. But I digress, and in general, this simple exercise helps recognize the importance of designing the joints of the software system.

The Real Point

Of course, while this example illustrates the point using modules, it’s not just the joints between modules that are important, but the joints at many different levels of the system, including applications, services, packages, and classes. In other words, the point where two entities connect is where the real architecture lives.

After the OSGi DevCon keynote, I had a few folks stop by and ask me for the slides. They are now available on slideshare. Usually, when browsing the slides without discussion, the lack of context can cause some confusion. So if you have any questions regarding the content, please feel free to reach out. Enjoy!

I’ll be at OSGi DevCon next week, having been given the opportunity to present the keynote session on Tuesday. The conference is co-hosted with JAX London. I thought I’d take a moment before jumping on the flight to offer a preview of what I’ll be speaking about.

The Cost of Complexity

The title of the session is OSGi in the Enterprise - Agility, Modularity, and Architecture’s Paradox. We all know that OSGi isn’t a new technology. It’s been around for over a decade. But the recent surge in interest in modularity on the Java platform is overdue. The complexity of software is increasing exponentially. Did you know:

• In 1990, there were 120 billion lines of code
• in 2000, there were 250 billion lines of code
• The number of lines of code doubles every 7 years
• 50% of development time is spent understanding code
• 90% of software cost is maintenance and evolution

Let’s put this in perspective using the visual at right (click to enlarge), where we can see that the amount of code doubles every seven years. Pretty staggering! But let’s put this in perspective for what it means over the course of the next seven years, as well. It means that between 2010 and 2017, we’ll produce more code than the total amount of code ever written combined! That’s staggering!

I decided to do a bit of research to see if I could find examples that supported these claims. I did:

• Since the Spring framework was released in 2002, the number of lines of code has grown more than 500% through the release of Spring framework 2.5 in 2008.
• FreeBSD contained roughly 8 million lines of code in 2002. In 2009, it was close to 16 million.
• In 2004, the Linux Kernel contained approximately 6 million lines of code. In 2009, it was around 12 million.

I’m sure many of us have experienced this phenomenon. Systems tend not to get smaller over time. And we also recognize that in almost every way, larger software systems are inherently more difficult to design, build, manage, and maintain than are smaller software systems.

Yet, such phenomenal growth is desirable. We can only hope that the systems we create are in such high demand that we have the opportunity to grow them over time. A software system that doesn’t change, dies. Evolution is a big deal! As Lehman’s law suggests:

As a system evolves, its complexity increases unless work is done to maintain or reduce it.

Add it all up, and there are some key takeaways here. We need something that will help us understand complex systems. We need something that help manage the complexity. We need something that will help ease maintenance. We need something that will help us deal with the natural evolution of software systems. We need something that will allow us deal with the natural architectural shifts that occur as a system grows to accommodate demand. For a long time, a central ingredient has been missing. But not for much longer, because the enterprise will get its OSGi!

For the rest of the story, see you in London.

Source of statistical information above: http://users.jyu.fi/~koskinen/smcosts.htm

It’s housecleaning time again, and like last time, I stumbled across an article I wrote back in 2006 that I don’t believe ever reached publication (at least, I don’t think it did…how am I expected to remember what I did in 2006?). For the most part, I’ve left it in its original state, except that I removed the Agile Manifesto and 12 supporting principles. There are easily enough found on the Agile Manifesto website, and the article is long enough without this duplication. The wordle at right shows the most common words used in this document (click to enlarge). Here, in it’s otherwise unadulterated glory, is Agile: A New Era of Software Development.

Agile: A New Era of Software Development

Embrace Change

Writing code is easy, but developing software is hard. While syntax errors are common, their severity pales in comparison to the logic flaws inherent in many software systems. The difficulty in software development is not writing code or applying a certain technology stack. Instead, the challenge lies in the specification and design of the software itself.  Therein lies the essential complexity of software development, an idea introduced by Frederick Brooks in his 1987 article titled, “No Silver Bullet” [Brooks]. The most difficult aspect of software development is deciding what, exactly, needs to be built.

There is certainly evidence backing this claim. The original Chaos Report shows the top three impediments to a successful development effort are lack of user input, incomplete requirements and specifications, and changing requirements and specifications [CHAOS]. No other activity, if done incorrectly, stands to compromise the system more than incorrect requirement specifications.

It might not be so difficult were software a concrete entity, existing in a world where we could easily visualize it’s structure and behavior, allowing us to more reliably assess and share the impact of change. But software is a highly conceptual, invisible construct. It is considered infinitely malleable by those not intimately familiar with the conceptual complexity of it’s structure and behavior. The contractor building your home would look at you with incredulous disbelief if you suggested that the house he has 90% complete no longer met your needs, and you asked that he move walls. Or imagine how ridiculous it would sound to suggest that a new third floor be inserted to a 100 story skyscraper. Physicists labor on with firm belief that there exist an underlying set of unifying principles to serve as guidance. Or at least, there are laws of physics that we hold to be true. There are no such rules or principles that guide software development. We are left with the imagination and dreams of our clients, and they demand and deserve rapid response to change.

We have made valiant attempts at conformity. Ceremonial processes attempting to define standardized activities that guide the development process have failed, however. We cannot define detailed up-front requirements specifications and expect them to survive the development lifecycle intact. We cannot establish an initial design of the conceptual construct and expect the structure to go unscathed throughout the process of construction. Software development is an error prone human activity involving experts with varying backgrounds and skills who must come together and attempt to communicate uniformly, working as a team toward a common goal. While tools and process do help, we must also accept that change is expected. We cannot treat change as evil. Instead, the tools and process used must allow us to accommodate change, treating it as an inherent part of software development. Changing requirements is a rule of our game. The software we develop must be malleable and adaptive to change, and the process we use must embrace change.

We often draw comparisons between software development and various manufacturing processes. As Larman points out, however, manufacturing is a predictive process [Larman]. Herein lies one of the greatest differences between software development and the manufacturing processes to which we often draw comparisons. Manufacturing is a repeatable activity, with high rates of near-identical creation where a consistent product is produced in assembly line fashion. Little change is expected, making it possible to reliably estimate and predict the outcome. Software development is much more like new product development, where evolutionary specifications and adaptive planning is necessary to deal with the many unknowns that lie ahead.

Agile Principles

In early 2001, a small group of industry experts converged in Utah to discuss alternatives to heavy, document driven software development methods. Emerging from this meeting was the Agile Manifesto, a symbolic proclamation endorsing the virtues of a lighter, more flexible, people-oriented approach to software development, giving birth to the agile software development movement. (Since this is already a long article, I’ve snipped the manifesto and principles, which were included in the original version. If you’re interested, you can view the manifesto and its 12 principles on the Agile Manifesto website.)

<Snipped the Manifesto for Agile Software Development and 12 Principles>

The ideas behind these 12 principles are simple, and contain no hidden messages. Of course, there are different techniques embodied within various agile processes that support these principles. The one certainty is that agile teams definitely work differently from their less agile peers. They recognize there is one end goal - to create a working, functional software product. With that in mind, they work very closely with project stakeholders throughout the development lifecycle, knowing it is the stakeholders who possess the knowledge the system must embody. Agile teams work very hard to deliver working software iteratively and incrementally, and they adopt techniques representative of that ideal.

Agile project managers tend to favor intense communication and collaboration over heavy documentation. Empowering team members to make decisions enables responsiveness to change. Facilitating and negotiating requirements scope provides important feedback, helping plan future iterations, where each iteration produces a deliverable that can be shared with clients and stakeholders. Instead of forcing the team to follow a predictive project plan, agile project managers are more opportunistic. They prioritize features based on stakeholder feedback, and make adjustments as the iterations progress. Concurrent and parallel activities are favored over a phased approach. Agile project managers tend to guide the team instead of manage the team, and strongly discourage unnecessary overhead.

Agile developers work with a similar set of goals, knowing functional software must be delivered early and often. They work judiciously to grow a code base built upon a solid foundation, where each day represents a step forward. They integrate frequently, and do not tolerate failed builds. A rich suite of tests provide the courage necessary to respond to change when the need arises. They avoid the notion of code ownership, empowering other developers to make improvements to any component of the software product.

A common misconception is that agile processes discourage all documentation. This is untrue. Agile processes discourage unnecessary documentation, favoring collaboration as the preferred technique. Instead of using documentation to drive communication, agile processes favor face-to-face communication. Documents are encouraged by agile processes, so long as the need is immediate and significant.

Transitioning to Agile

Agile software development is based upon the fundamental premise that we must drive and respond to change quickly. The Agile Manifesto and 12 supporting principles serve this premise well. Advocates of agility claim speedier delivery of software, software with more business value, increased team productivity, higher quality systems, and a more enjoyable development experience. I believe each of these to hold true. Agile teams not only welcome change, they are able to respond to change at all levels of development. A project manager might discuss a changing requirement with a business client, empower a business analyst to schedule a meeting with the client to discuss further details, while a developer assesses the impact of change knowing she has the courage to accommodate the request because of the rich suite of unit tests in place.

Saying you’ll be more responsive to change and creating an environment that embraces change are separate beasts, however. Practicing agility is hard work, especially if your team is accustomed to more traditional approaches. As with many things new and unfamiliar, some resistance will no doubt arise by those who aren’t fully convinced. Agile projects differ greatly from their less agile counterparts, and skeptics will have many opportunities to express their discontent. As someone experimenting with agility, you may even have doubts. But don’t be discouraged, and give your agile transition the time it deserves.

One of the most significant changes you may experience is a feeling that you’ve been thrust into a chaotic nightmare. I doubt it’s unusual to feel this way. You’ve lost the security of detailed requirements specification and user sign-off. You are writing code without the comfort of knowing exactly what your stakeholders want. The detailed plans that have served as your security blanket on past projects no longer exist. And the celebrations accompanying completion of your various phase milestones are gone. Of course, these were all false comforts anyway. Stakeholders always changed their minds. Your detailed requirements and plans were outdated as quickly as they were completed.

Instead, you’re now working in shorter iterations with vague requirements. Initial releases early in the lifecycle may be completely thrown away. Your first few weeks may seem wasted, with little or no valuable artifacts produced. Naysayers will immediately come forward and cite the lack of progress. Previously, those first few weeks or months were spent producing very detailed requirement specifications and beautiful design models. But don’t give up yet. In that previous world, you were only delaying risk and postponing integration, avoiding the most difficult aspect of software development until the end of the lifecycle. Now you’re attacking risk early, prioritizing features, and working hard to develop a functional piece of software as early as possible. Progress may not be at breakneck speeds, but you are learning a tremendous amount about the requirements of the system, and your velocity is sustainable. Additionally, you are also performing a number of other important lifecycle activities.

Depending on the level of ceremony and bureaucracy within your organization, you will experience varying degrees of success when adopting agile techniques. As with any new technology adoption, it’s best to phase the transition. Some agile techniques are easier to adopt than others, and some serve as valuable catalysts to adopting additional techniques in the future. Don’t attempt to completely redefine how you work. It’s relatively easy to phase the agile transition, and you’ll want to adopt those principles that offer you the greatest initial reward.

For instance, if you’re struggling to produce quality software at a consistent rate, implementing a continuous integration strategy will help you frequently verify the quality of your work. In addition to the comfort of knowing you have a product always in a functional state, the ability to share the product with clients using functional demos and prototypes will tighten the feedback loop and offer valuable insight to the client’s perception of the software. In a number of cases, I’ve found this to be valuable in identifying subtle requirements that can be difficult to identify in other requirements elicitation venues.

Empirical Evidence

In recent years, there has been a significant amount of research comparing agile development methods to their waterfall counterpart. In Agile and Iterative Development: A Manager’s Guide, Craig Larman illustrates the advantage of agile development through detailed analysis of multiple studies[Larman]. The compilation of his results are illustrated below.

A study by Alan MacCormack at Harvard Business School explored whether evolutionary development techniques yielded better results than the waterfall model. The study included applications ranging from application software to embedded systems, with median values of nine developers spanning a 14 month development cycle.

A key conclusion of the study, in which 75% of participants used agile techniques compared to 25% using waterfall, explained releasing software earlier, rather than later, contributed to a lower defect rate and higher productivity. There was little evidence showing that a detailed design specification resulted in a lower defect rate, however, reviews with peers did help in reducing the rate of defects. The study found that iterative and agile practices have a significant impact on defect and productivity factors, as indicated by the following points.

• Releasing a system with 20% of the functionality complete is associated with a decrease in the defect rate of 10 defects per month per million lines of code as compared to waiting to release a product until 40% of the functionality is complete, and an increase in productivity of eight more lines of source code per person-day.
• Continuous Integration, the idea of integrating and testing code as it is released to your source code repository, resulted in a decrease in the defect rate of 13 defects per month per million lines of code, and an increase in productivity of 17 lines of source code per person-day.

The study also found four practices that were consistently used by the most successful development teams. The first two are deeply embedded in the ideals of agile software development.

• Releasing early and often to project stakeholders, using an iterative lifecycle.
• Continuous integration, with daily builds including regression testing.
• Teams with broad experience delivering multiple projects.
• Careful attention to modular and loosely coupled, componentized architectures.

In a separate study [Shine], 88% of organizations cited improved productivity when using agile methods, and 84% cited improved productivity. 49% stated that the cost of development was less when using agile methods. Additionally, 83% claimed increased business satisfaction and 26% claimed significantly better satisfaction. Another study by Boehm and Papaccio [Boehm] discovered that a typical project experiences a 25% change in requirements, while yet another [Johnson] showed  that 45% of features were never used.

There have also been many research efforts devoted exclusively to the analysis of waterfall methods. Below is a summary of these findings, taken from a variety of studies.

• Scope management related to detailed up-front requirements was a significant contributing factor of failure [Thomas].
• The U.S. Department of Defense (DoD), when following a waterfall lifecycle, experienced a 75% failure rate [Jarzombek]. This resulted in the DoD adopting a more iterative and agile approach.
• On a study including 400 waterfall projects, only 10% of the code was deployed. Only 20% of code deployed was used. The main factors included changing and misunderstood requirements [Cohen].

As these studies clearly illustrate, there is significant evidence showing that agile and iterative techniques offer significant advantages over the waterfall model of development. In fact, for larger projects, the statistics supporting agility were even more pronounced.

Conclusion

There are a variety of agile processes available to choose from, and each abide by the spirit of the manifesto and it’s 12 supporting principles. The agile movement and it’s supporters recognize that software development is a human (though not always humane) activity. Instead of forcing process on people, agile methods allow process conformance to people. Good people, working toward a common goal, can achieve great things will little ceremonial process, assuming you give them an environment that empowers them. Solid empirical evidence backs this claim. And if the quality of people is in question, it’s doubtful that any process will produce success.

References

• [Alliance]. The Agile Alliance. Manifesto for Agile Software Development. 2001. http://www.agilemanifesto.org
• [Boehm]. Boehm, B, and Papaccio, P. Understanding and Controlling Software Costs. IEEE Transaction on Software Engineering. October 1988.
• [Brooks]. Brooks, Frederick. No Silver Bullet: Essence and Accidents of Software Engineering. 1987.
• [CHAOS]. The Standish Group International, Inc. The CHAOS Report. 1995.
• [Cohen]. Cohen, D., Larson, G., and Ware, B. Improving Software Investments through Requirements Validation. IEEE 26th Software Engineering Workshop. 2001.
• [Jarzombek]. Jarzombek, J. The 5th Annual JAWS S3 Proceedings. 1999.
• [Johnson]. Johnson, J. Keynote speech, XP 2002, Sardinia, Italy. 2002.
• [Larman]. Larman, Craig. Agile and Iterative Development: A Manager’s Guide. Addison-Wesley, 2004.
• [MacCormack]. MacCormack, A. Product-Development Practices That Work. MIT Sloan Management Review. 2001.
• [Shine]. Corporate Report. Agile Methodologies Survey Results. Shine Technologies Pty Ltd. Victoria, Australia. 2003.
• [Thomas]. Thomas, M. IT Projects Sink or Swim. British Computer Society Review. 2001.

Story of the Concorde

On July 25th, 2000, flight 4590 crashed. It was the first, and only, crash of the famed Concorde. Eventually, it would lead to retirement for the amazing aircraft. Investigators spent countless hours poring over the wreckage, and placed blame on a piece of runway debris that slashed the tire. A piece of that tire struck one of the fuel tanks, causing it to rupture and the plane caught fire. Case closed, right? Not so fast.

Surely a small piece of runway debris shouldn’t bring down a commercial airliner! As it turns out, there is quite a bit of contention among experts surrounding other factors that may have contributed to the crash. Some argue that it was a complex chain of events, all coming together, that brought down the aircraft. The plane was missing a spacer between the two wheels, had too much fuel in the tank, attempted to takeoff in unstable conditions, and was overweight. The flight was also delayed, causing angst among the flight crew. Finally, a required daily runway inspection was not performed.

Perhaps, if the runway inspection had been performed, the piece of debris would have been spotted. Or had the aircraft not been overfueled, the piece of tire may not have caused an increase in fuel tank pressure that some say caused the tank to burst. Had the spacer been installed, it’s possible the tire would have never burst. A series of small events, each contributing in their own way, to the fatal crash.

No Matter How Small

The story reminds me of the importance of attention to detail in software development. Of how the aggregate of all of the small, seemingly insignificant decisions we make on a continuous basis can have long-term consequences on the future of the software system. Possibly even your organization. Every time you design a class, define a variable, write a test, create a package, build a module, modify a method, or make a design decision, you are affecting the future of your system in some unsuspecting way.

What may seem insignificant today, can have a detrimental affect tomorrow. Really, there are no small, insignificant decisions in software development. I’m reminded of how important it is to make conscious decisions that are given careful thought, no matter how small. It also reminded me of a poem on build automation and continuous integration that I read a while back on the Test Early blog.

FOR WANT OF A BUILD

For want of a build, a test case was not executed
For want of test case, a defect was not detected
For want of a defect report, a bad release was promoted
For want of a good release, a strategic customer was lost
For want of a customer, a development team was reduced
For want of developers, a product stagnated
For want of a product, a company was lost
And all for the want of a build…

I’ve spent some time thinking a bit more deeply about a few of my recent posts on software architecture, and have come to the following revelation.

Architects architect architecture!

Don’t let the triviality of this statement undermine its depth. While each of the three words are variations of the same thing, each have different contextual meaning. Let’s tease the statement apart to see what I mean.

• Architects - Humans create software architecture, and for architecture to be effective, we must also be able to understand the architecture. In Eliminate Architecture, I cited a definition of architecture that introduces the social dimension. Architects signify the social pillar.
• Architect - The way we arrive at architecture is through some process or series of steps. We might create diagrams or software architecture documents. We might write a little code (proofs, spikes, prototypes) to determine the viability of architecture. There are many different activities we perform as we create the architecture. Architect signifies the process pillar.
• Architecture - In Modularity and Architecture, I offered up a few industry definitions of architecture. Common keywords that span definitions include components, composition, interfaces, subsystems, and structure. Architecture signifies the technology pillar.

To ensure balance, we must give attention to each of the three pillars. Additionally, each pillar is related to the other. For instance, ignoring the social pillar impacts the other two in unexpected ways.

The Social Pillar

Turtles and Architecture generally focused on the social pillar of software architecture, but also talked a bit about how the technology pillar can increase understanding, visibility, and transparency. The general sentiment is that architects must focus on more than just concepts and developers must focus on more than just code. There is a middle ground that demands attention, as well.

I used a visual similar to what’s at right (click to enlarge) to illustrate the middle ground. It also illustrates how aspects of the technology pillar can help increase understanding and transparency of architecture. Increased understanding of the architecture hopefully leads to improved structural quality (technology pillar) and transparency of the process (process pillar).

The Technology Pillar

Architecture All the Way Down primarily focused on the technology pillar. The visual at right (click to enlarge) illustrates this view. Again, we see the huge gap that exists if we fail to emphasize the importance of modularity. The rightmost portion illustrates how modularity fills the gap - architecture all the way down. Of course, other gaps are also created if we ignore any of the other aspects, such as class design. Note that as we move from services to modules to packages and classes, we increase the granularity along the way. Our classes should not be as fine-grained as our modules, nor our modules as fine-grained as our services.

Additionally, each entity solves a different type of problem (or provides a different type of advantage), as illustrated by the bars at the bottom. All are units of composition, but only services and modules are units of deployment. Services are reused through distributed computing, whereas module reuse is constrained by process boundaries. The technology pillar affects the other pillars. An inflexible rigid architecture makes it difficult for people to understand and communicate (social pillar) and inhibits how quickly we’re able to respond to change (process pillar).

The Process Pillar

The process pillar is one that I’ve not spent much time discussing. Certainly, it’s important though, and includes various aspects like deferring commitment to significant architectural decisions, evolutionary and emergent architecture, and reversibility. The visual at right illustrates the process pillar (click to enlarge). It’s not as descriptive as the other diagrams, I admit.  Anyone have something better that illustrates the process pillar?

I did talk a little about these ideas in Agile Architecture, Lean Principles. But certainly more needs to be fleshed out surrounding the process pillar. This tends to be where most spend their time when discussing agile architecture, but the other pillars are certainly important. The process pillar affects the other pillars. A bad process accompanied by bad practices results in an inflexible architecture (technology pillar) that noone is able to understand (social pillar).

For those that follow this blog, but not the APS blog, I’ve just published a synopsis of modularity on the Java platform. With a lot of momentum continuing to build around OSGi, we’re still in a state of limbo surrounding a standard module system on the Java platform. Some might argue that OSGi has already won - it is the defacto standard module system. Possibly.

Oracle has offered us very little direction on their future plans. There was little fanfare surrounding WebLogic dm Server, with only sparse references and no official word from Oracle. I’m hopeful we can expect to hear some news of the direction they intend to take in their upcoming webcast, though I’m not expecting them to make any significant statements at such a fine level of technical detail. Coincidentally, Apple is holding their own event that very same day, and is expected to announce their new tablet. Let your imagination run wild. Those two events should make for a very exciting day.

Regardless, I’ve had quite a few conversations over the past several months surrounding modularity and OSGi. I have my suspicions on Oracle’s intent, though am reticent to share at this time. Read the APS post titled, Java Modularity - Time to Set Sail.

I’ve been thinking a bit more about the list of disruptive technologies on Richard’s list, and then I watched Dan Pink’s TED talk on the surprising science of motivation. There must be some relationship between the list, which is comprised of almost all open source software products, and Dan’s assertion that the 20th century reward system won’t work for the cognitive tasks performed by workers in the 21st century. What is it, though?

We develop open source software to scratch a personal itch, ease the pain in performing a certain type of task, or create a more compelling alternative to a commercial product. While the professional open source model has emerged the last few years as a way for the open source community to create a sustainable business model, none of the open source technologies on the list were initially developed that way. They were developed in response to need. Ant because Java lacked a good build system. Spring because Java EE was cumbersome and bloated. JUnit to help increase quality. In many cases, these tools have grown to become defacto standard technologies widely used by enterprise development teams.

In Dan’s mind, the new incentive program within organizations must revolve around three things - autonomy, opportunity, and purpose. We must be given autonomy, or the empowerment to make our own decisions. We must be given the opportunity to master something that matters to us. And we must be given purpose, which is the desire to do what we do in the service of something larger than ourselves. Dan notes that financial incentive is also important, but is not the decisive factor in what motivates us.

There are a lot of ways to connect the dots between the open source products so prevalent on Richard’s list and Dan’s point about incentives. I’ll allow you the opportunity to connect these dots any way you wish. A few that immediately come to mind for me include the way commercial software is sold, corporate incentive programs, empowering developers, and flaws in corporate culture. But here’s something else to chew on.

In most of the cases, it was the developer who spurred adoption of the most disruptive application development technologies of the last decade. We weren’t motivated to develop or adopt these great products because of financial incentive or a reward system. Sure, in some cases that’s a positive side affect, but it is not the force that motivates. Instead, we were motivated because they make our jobs a little bit easier and our software a little bit better.

Developers are fighting like hell to create better software. While a lot of commercial vendors are selling shelfware (they aren’t selling it to the developer, mind you), developers are driving adoption of the technologies that are making a difference. Developers seek autonomy,  opportunity, and purpose. Given corporate culture, it’s not always easy to find. But if the last decade is any indication, we are finding it, developers do have a voice, and that voice is being heard.

Tight coupling between modules is a bad idea, and the worst form of coupling is cyclic dependencies between modules. Fortunately, there are a few techniques we can use to break the cycles. They are Callback, Escalation, and Demotion, and I’m going to walk through some examples that show each of them in action.

Then, once the dependencies are broken, we’ll look at two more techniques that allow us to invert and eliminate the relationship altogether. The code for each of the samples can be found in the edcie project on my Google code repository. Each example includes a build script and a simple test case. To execute them though, you’ll need GraphViz if you want to use JarAnalyzer. To invoke the build scripts without invoking JarAnalyzer, you can simply type:

ant compile

Keep in mind that each variation of the system has the exact same behavior!

The Example

The example we’re going to use to drive the remainder of our discussion is incredibly simple. We’ve got a Customer and a Bill class that we’re going to bundle into two separate modules - cust.jar and bill.jar. There’s also a test case called PaymentTest that serves as a sample client to drive the interactions between the two classes. The test case is bundled into the billtest.jar module. The initial class diagram is seen at right. Note the bi-directional relationship between the two classes.

As we progress, we’ll add more classes and abstractions to the system to help increase the modularity. Additionally, we’re going to use JarAnalyzer to illustrate the relationships between the modules and also help us assess the quality of our design. The module structure is below, as generated by JarAnalyzer. You can see how to use JarAnalyzer in the build by reviewing the build file. Again, our goal is to break the cyclic dependency between cust.jar and bill.jar, and we’re going to look at three different ways to do this before moving on to examine different ways to massage acyclic module relationships.

Initial Module Structure with Cyclic Dependencies

Escalation

The first technique we’re going to apply is called Escalation. With Escalation, we break the cyclic dependencies by escalating the cause of the dependency to a higher level entity. Before we do that, we need to more fully understand why a cyclic dependency exists in this example. This reason follows:

A Customer has a list of Bill instances. When the pay method on Bill is invoked, the Bill needs to determine if a discount should be applied. The discount is a product of the Customer the bill belongs to, not necessarily the Bill. Therefore, the Bill class calls a method on Customer to determine the appropriate discount amount. Think of it this way…The Customer represents a payee and we negotiate a discount with each payee. The calculation of this discounted amount is encapsulated within the Customer.

To break this dependency, we want to escalate the cause of the dependency up to a higher level class - the CustomerMediator. The mediator now encapsulates calculation of the discount and passes that to the bill class. The best way to see this change is to look at the modified PaymentTest class. Now, I’ve modified the build script and have bundled the mediator into it’s own module, as shown below. If you dig a bit more deeply into the class structure, you’ll wonder why I didn’t just pass the discount amount from Customer into Bill. Don’t worry about that. This example is slightly contrived because escalation isn’t the best way to solve this type of problem. The key takeaway is that we’ve escalated the dependency up to the mediator.jar bundle, breaking the cyclic dependency.

Escalating the Cause of the Cyclic Dependency

Demotion

A slightly better way to solve this particular type of cyclic dependency (where we have a true composite relationship between Customer and Bill) is to use demotion. With demotion, we push the cause of the dependency to a lower level module. Exactly the opposite of escalation. We do this by introducing a DiscountCalculator class that will be passed into the Bill class. Our modified PaymentTest class will create the calculator and pass it in for us. The Customer class will serve as the factory for the DiscountCalculator, since it’s the Customer that knows the discount that must be applied. The new class structure can be seen at right.

Now we’ll modify our build script to create an additional calc.jar bundle which will contain the DiscountCalculator class. Our resulting module structure is shown below.

Demoting the Cause of the Cyclic Dependency

Already you can see how this is a more natural solution than escalation for this particular type of cyclic dependency problem.  What’s the key difference you might ask? With escalation, notice how I would be able to deploy the cust.jar and bill.jar modules independently. While demotion is a more natural solution in this situation, it also means that to deploy bill.jar or cust.jar, I must also deploy calc.jar. The right solution is always going to be contextual and the ideal solution is likely to shift throughout the development lifecycle.

Callback

Using a Callback is similar to the Observer pattern. With this approach, we’ll refactor our DiscountCalculator class to an interface, and then modify the Customer class to implement this interface. This new class structure can be seen at right.

As it happens in this specific situation, using a Callback represents a combination of demotion and our initial solution. We’ll go back to passing the Customer into the Bill, but will pass it in as a DiscountCalculator type. Whereas in the Demotion example we bundled the DiscountCalculator in a separate module, we’ll now just include it in our bill.jar module. Note that putting the DiscountCalculator in the cust.jar module would introduce the cyclic dependency we’re trying to get rid of. The new module structure, which resembles our original version minus the cyclic dependency, is shown below.

Using a Callback to Eliminate the Cyclic Dependency

Inverting Relationships

Now we’re going to play around a bit with the module relationships. While Callback seems like the most logical solution, what if we wanted to use the cust.jar module without the bill.jar module? Callback, as it’s implemented, doesn’t allow us to do this. But with a bit of trickery, I can actually invert the relationship between the cust.jar and bill.jar modules.

I start by refactoring the Bill class to an interface. Then, to avoid split packages (where classes in the same package are bundled into separate modules), I move the Bill class into the same package as the Customer class. The new class diagram is shown at right, and the inverted module structure is shown below.

Inverted Module Structure

Eliminating Relationships

Inverting the relationships allows us to deploy the cust.jar module independent of the bill.jar module. Again, it’s all about need. But I’d like to explore another option based on another important need - the ability to test modules independently. Before inverting the relationships, I am able to test the bill.jar module independently. After inverting the relationships, I can test the cust.jar module independently. But what if I want to test (or deploy) both modules independently? To do this, I need to completely eliminate the relationship altogether.

As it turns out, because I’ve got a pretty flexible class structure after I inverted the relationships (lot’s of abstract coupling), I can do this by simply bundling the two interfaces - Bill and DiscountCalculator - into a separate module. No other coding changes required. I start by moving them to a new package called base. Then, I modify my build script to bundle these two interfaces into a separate base.jar module, and we have successfully eliminated the relationship between the bill.jar and cust.jar modules, as shown below.

Eliminating Relationships Between Modules

Wrapping Up

We’ve come a long ways from our original version of the system. Two modules with a cyclic relationship to a module structure where the original modules don’t have any relationship to each other. This means these modules can be tested and deployed independently. If you follow this blog, you know I’ve written plenty about the tradeoffs between flexibility and complexity, use and reuse, and many other architectural and design challenges. I hope this little exercise has helped drive some of those concepts home.

As a final note, to explore some of these design decisions a bit more deeply, and to examine the tradeoffs a bit more objectively, I encourage you to run the builds for each of the projects and examine the dependencies.html file in the stats directory. To do this, you’ll need to make sure JarAnalyzer is executed, which requires GraphViz. As you’ll see when comparing the initial version to the final version, we made considerable progress in improving the quality of the design.

2010 is going to be a big year for modularity on the Java platform. So based on recent industry trends, and as momentum continues to build around modularity on the Java platform:

I hereby officially declare 2010 as the year of modularity on the Java platform.

So that’s a relatively silly statement, and I really don’t have any authority to make such sweeping declarations. But the statement is backed by some pretty serious trends. Here are a few nuggets to chew on that has led me to believe modularity is going to be bigger in 2010 than ever before!

Between 2007 and 2009, we saw amazing momentum build around OSGi. GlassFish, Jetty, Spring DM, SpringSource dm Server, Paremus Service Fabric, Nimble, PAX, Sigil, Aries, WebSphere, JBoss, Maven, and of course Eclipse are just a few products that leverage OSGi. The list goes on. The number of 3rd party frameworks that have been “osgi-ified” has also grown substantially. And whether you love it or hate it, let’s not forget about Jigsaw and JSR-294.

But there has been a nagging problem…an elephant in the room you might say…that’s hindered widespread adoption. Without gluing together a platform, assembling it yourself, or adopting an entirely new platform, aside from the vendors who were already exposing the virtues of OSGi in their products, the majority of enterprise developers were unable to leverage any of it to build modular enterprise applications! But in 2010, that’s all going to change.

Modularity is coming to the enterprise and you’re going to be able to use it to build modular applications. IBM has already made this announcement. I suspect others will follow suit. For those using SpringSource dm Server or Infiniflow, you’re probably already doing it. With the major platform vendors showing their support for a technology that’s already embedded in products such as dm Server and Infiniflow, I suspect these platforms will see an increase in market penetration through 2010 and beyond. And I suspect you can expect some big announcements surrounding these products soon. Finally, let’s not forget, if it can avoid another delay, JDK 7 (with support for modularity) will be released in September.

Whereas prior to 2010, enterprise developers could only dream about building modular applications (or work very hard to do so), modularity is going to take center stage. And since modularity lies at the heart of the platform, it doesn’t matter what language you use. Neil showed us how to do it with Scala way back in 2007. And Groovy does OSGi, as does Clojure. Yep, so does Scala. So no matter which language you use, you’ll be able to take advantage of modularity on the Java platform.

In 2010 modularity on the Java platform is going to be a game changer…a disruptor…that reaches from the developer to the data center. As 2010 progresses, more and more developers will be touched by modularity. It’s not something we should ignore. Really, it’s not something we can ignore anymore.

Thanks

This is my last post of the year, and I want to take a moment to thank everyone that spends their precious time reading my long-winded entries. In 2009, I saw a four fold increase in blog traffic over 2008, and December 2009 saw a five fold increase in traffic over January 2009. Out of a total of 74 posts this year, more than half were related to modularity. I’d like to think the increase in traffic is related to the topics I’ve been writing about. Modularity is something that folks are interested in. But we still have a ways to go.

Future of Modularity

Interestingly, I was speaking to a conference organizer yesterday, and I sensed his mild surprise when I began to talk about modularity. He noted that modularity is certainly not a new concept. I agreed. It’s an idea that has been around since at least the early 1970’s when David Parnas published his essay. But things are different now. Until recently, modularity was not a first class concept on the Java platform. If we wanted to develop software with a modular architecture, we were required to do so without much framework or platform support. In fact, we hadn’t even identified any standard definition of module on the platform.

That’s changing. As the application platform vendors continue to bake OSGi into their products, it’s clear that a module is a JAR file and there’s a framework to back it up. In 2010 modularity on the Java platform will gain considerable visibility in our industry. The stackless stack is coming to fruition, and it’s a game changer…a disruptor…that reaches from the developer to the data center. It’s not something we should ignore. Really, it’s not something we can ignore anymore.  So, as we close out one decade and usher in another, modularity will celebrate at least its 40th birthday. Maybe we’re finally starting to get it.

All Posts Page

A final note before closing out 2009. For those interested, I’ve created a custom Wordpress page that shows a summary of all posts published on this blog. Enjoy!

I’m excited to be given the opportunity to present the keynote at OSGi DevCon London on February 23, 2010. The event is hosted in conjunction with JAX London. The sessions for the conference look great, with a great lineup of speakers and 15 sessions, including two in-depth tutorials by Neil Bartlett and Peter Kriens, devoted exclusively to OSGi.

Here’s the abstract for my session titled OSGi in the Enterprise: Agility, Modularity, and Architecture’s Paradox.

Attempts to architect more flexible software often results in the opposite - brittle software fraught with complexity. Something is missing. Complexity is the beast we must tame, and modularity is part of the answer. In this keynote, we’ll examine the challenges of traditional approaches to software architecture, probe the paradox of architecture, and explore the inextricable link between structural and temporal architectural decisions. From the highest level applications and services to the code that exists in the bowels of the system, we expose the advantages of a modular architecture. Come learn new ways that large software systems can be organized to increase flexibility, reusability, adaptability, maintainability, testability, and extensibility. Come discover the importance of OSGi in the Enterprise.

Yes, come discover the important of OSGi in the Enterprise!

Note: This is a re-post, with slight modifications, from an entry in October 2007. And now, two years later, we’re just about there! For the abridged version of this post, focus on the text in bold. You can see the original version here.

Design Rot

We’ve all experienced that sinking feeling when maintaining a piece of crappy software. Has my change broken the system in some unintended way? What is the ramification of my change on other parts of the system? If you’re lucky, and the system has a robust suite of unit tests, they can offer some support in proving your work. In practice, however, few systems have thorough automated test coverage. Mostly we’re alone, left to verify our changes as best as possible. We might privately criticize the original developers for creating such garbage. It certainly lends a plausble excuse in explaining why the maintenance effort is so costly or time-consuming. Or it might serve as the basis upon which we recommend a re-write. But mostly, we should wonder how it happened.

For sure, most software doesn’t start out this way. Most software starts out clean, with a clear design strategy. But as the system grows over time, strange things begin to happen. Business rules change. Deadline pressures mount. Test coverage slips. Refactoring is a forgotten luxury. And the inherent flaws present in every initial design begin to surface. Reality has proven that few enterprise development teams have the time or resources to fix a broken design. More often, we are left to work within the constraints of the original design. As change continues, our compromises exacerbate the problem. The consequence of rotting design is seen throughout the enterprise on a daily basis. Most apparent is the affect on software maintenance. But rotting design leads to buggy software and performance degradation, as well. Over time, at least a portion of every enterprise software system experiences the problem of rotting design. A quote from Brook’s sums it well:

All repairs tend to destroy the structure, to increase the entropy and disorder of the system. Less and less effort is spent on fixing the original design flaws; more and more is spent on fixing flaws introduced by earlier fixes. As time passes, the system becomes less and less well-ordered. Sooner or later the fixing ceases to gain any ground. Each forward step is matched by a backward one. Although in principle usable forever, the system has worn out as a base for progress.

The most obvious question is, “How do we prevent rotting design?” Unfortunately, rotting design is not preventable, only reducable. Of the past ten years, the design patterns movement has provided insight to the qualities of good design. Dissecting design patterns reveals many important design principles that contribute to more resilient software design. Favor object composition over class inheritance, and program to an interface, not an implementation, are two examples. Of the 23 patterns in the GOF book, all adhere to these fundamental statements. Alone however, design patterns that emphasize class structure are not enough to help reduce rotting design.

Reducing Rot

Most patterns emphasize class design, and present techniques that can be used in specific contexts to minimize dependencies between classes. Teasing apart the underlying goal of most patterns shows us that each aim to manage the dependencies between classes through abstract coupling. Conceptually, classes with the fewest dependencies are highly reusable, extensible, and testable. The greatest influence in reducing design rot is minimizing unnecessary dependencies. Yet enterprise development involves creating many more entities beyond only classes. Teams must define the package structure in which those classes live, and the module structure in which they are deployed. Increasing the survivability of your design involves managing dependencies between all software entities - classes, packages, and modules.

But if minimal dependencies were the only traits of great design, developers would lean towards creating very heavy, self-contained software entities with a rich API. While these entities might have minimal dependencies, extreme attempts to minimize dependencies results in excessive redundancy across entities with each providing its own built-in implementation of common behavior. Ironically, avoiding redundant implementations, thereby maximizing reuse, requires that we delegate to external entities, increasing dependencies. Attempts to maximize reuse results in excessive dependencies and attempts to minimize dependencies results in excessive redundancy. Neither is ideal, and a gentle balance must be sought when defining the behavior, or granularity, of all software entities - classes, packages, and modules. For more on the use/reuse paradox, see Reuse: Is the Dream Dead?

Software design is in a constant quandary. Any single element key to crafting great designs, if taken to its individual extreme, results in directly the opposite - a brittle design. The essential complexity surrounding design is different for every software development effort. The ideal design for a software system is always the product of it’s current set of behavioral specifications. As behavior changes, so too must the granularity of the software entities and the dependencies between them. The most successful designs are not characterized by their initial brilliance, but instead through their ability to withstand the test, and evolve over the course, of time. As the complexity of software design is an essential complexity surrounding software development, our hopes lie with technologies and principles that help increase the ability of your design to survive. Such is the reason why agile architecture is so important.

A Promising Future

I’m hopeful that all software developers have experienced the pleasure of a design that, through the course of time, has withstood the test of time. Unfortunately, many enterprise development teams have too few of these experiences. Likewise, few enterprise development teams devote adequate effort to package and module design. It’s unreasonable to believe that even the most flexible class structure can survive should the higher level software entities containing those classes not exhibit similarily flexible qualities. The problems are rampant. Increased dependencies between packages and modules inhibit reusability, hinder maintenance, prevent extensibility, restrict testability, and limit a developer’s ability to understand the ramification of change.

Services offer some promise to remedy our failures with object-oriented development. Yet, while services may offer tangible business value, within each awaits a rotting design. There exists a world between class design and web services that deserves more exploration, and as an industry, we are beginning to notice. OSGi is a proven module system for the Java platform, while Jigsaw aims to modularize the JDK. JSR-294 aims to improve modularity on the Java platform. While some friction might exist between the constituencies involved, it’s only because they too recognize that something has been missing, and are passionate about fixing the problem. Of course, it doesn’t stop there. A plethora of application servers and tools are also including support for modularity using OSGi, which has grown into the defacto standard module system on the Java platform.

All aim to help manage the complexity, from design through deployment, of enterprise software development. With each, new practices, heuristics, and patterns will emerge that increase the ability of a design to grow and adapt.

Certainly, there are many landmark books in software development that have shaped our industry. Design Patterns: Elements of Reusable Object-Oriented Software (the GOF book) is one really good example. Unfortunately, a good share of people don’t have the opportunity to read some of these great works because, well…it can get expensive pretty quickly stocking a bookshelf. But there exists a treasure trove of published content available online that is equally impactful. Here, in no particular order, are 5 essays that have helped shaped our industry, for better or worse.

• Cathedral and the Bazaar by Eric Raymond - Discusses the evolution of Linux and provides amazing insight to lessons learned.
• Code as Design by Jack Reeves - Presents the notion that programming is fundamentally a design activity and that the only final and true representation of “the design” is the source code itself.
• Managing the Development of Large Software Systems (pdf) by Winston Royce - Paper widely regarded as that which gave birth to the waterfall development lifecycle.
• No Silver Bullet by Frederick Brooks - We’re still looking, but as this paper points out, there is no silver bullet. The essential complexity Brook’s speaks of is largely why we continue to struggle with the same problems today that we did a decade ago.
• On the Criteria to be Used in Decomposing Systems into Modules by David Parnas - Discusses the important design decisions that impact how we modularize our software systems. Important because modularity is coming to the Java platform, and we need to know how to use it effectively.

I’ll give an honorable mention to Design Principles and Design Patterns by Bob Martin, which discusses key principles of object-oriented design. Many of the patterns in the GOF book adhere to these principles.

These essays are sure to provide a positive and lasting influence. But I’m sure there are more. What am I missing? What do you consider the most impactful software development essays? What would you add to this list?

In Turtles and Architecture, I talked about how important it is that we “architect all the way down”. It helps increase transparency and understanding between developers and architects by emphasizing a lot of the middle ground that noone ever seems to focus on. It’s just another reason why modularity is so important. I used the diagram at right to illustrate the point (click to enlarge). Look at the huge gap that exists if we don’t focus on the stuff highlighted by the gray bar in the middle.

One reason I like this visual is that it illustrates the social aspect of software architecture. Yet, there are other significant advantages to architecture all the way down that we haven’t explored yet. Another is structural flexibility.

Structural Flexibility - Different Entities, Different Purpose

Another benefit of module design in filling that middle ground is that modules can offer different benefits than classes and services. The diagram at right (click to enlarge) illustrates some of the capabilities of different types of entities.

For example, classes are pretty easily reused within an application, but because classes aren’t a unit of deployment, it’s difficult to use them across applications. Of course, intra-application reuse is a sweet spot of services, since they’re method of invocation is through some distributed protocol (SOAP, HTTP, even RMI/IIOP). Yet because services are invoked remotely, we typically like to make them coarser-grained because of the performance implications of distributed requests. So this begs the question - If a service is too coarse-grained to reuse (ie. it does more than what we want), how do I reuse some desirable behavior across applications? Well, without modules, our only other choice is the class. Since a class can’t be reused intra-process, we do one of either two things. Expose it as a service or copy the class (ie. the code). Given the circumstances, neither of these may be ideal. Another option is desirable.

Modules represent that other option. They are a finer level unit of granularity than services, and are a unit of deployment. Since each of these different types of entities are units of composition, we have tremendous flexibility in how we assemble applications. Possibly more important though is our increased ability to accommodate the architectural shifts that naturally occur as requirements evolve. Let’s look at a more concrete example.

A Bit More Concretely

Let’s say I have a business function called Pay Bill for which I develop a web service that can be invoked by many different consumers. That service happens to be relatively coarse-grained, and performs all the steps involved in paying the bill. These happen to include the following:

• audit bill - apply a discount to the bill based on payee
• check for duplicate - ensure the bill hasn’t already been paid
• remit payment - cut the check
• reconcile payment - reconcile with accounts payable financials

This seems reasonable. We have a nice little service that we can reuse any time we want to pay a bill. Unfortunately, the real world often gets in the way of our idealistic solutions. In fact, there are two problems that will eventually surface, and modularity benefits both scenarios. Let’s start by looking at the first scenario.

What should I do when a different scenario arises that demands I follow a slightly modified Pay Bill function?

As part of the remit step, I have a new payee that demands electronic payment. This is pretty easy actually. I simply modify the service to support electronic payments and then configure the service to context for that specific payee. So how does modularity help here?

If the service is composed of modules, it’s going to be much easier for me to understand the structure of the service, assess the impact of what’s it’s going to cost to change the service, and then introduce a new module (or modify the existing module) to provide the new capability. Without modules, I’m simply wading through the code trying to figure out all of these things. Now, the 2nd scenario.

What should I do when I want to reuse just one step of the Pay Bill function?

Let’s say another new requirement emerges. Whereas traditionally bills were entered by data entry personnel, we now have to support electronic delivery of bills. We also know that bills delivered electronically are often duplicates. It’s just one of those business things, you know? If we don’t pay the bill on the day it’s received, the billing party sends us the bill again, asking for payment. So we need to check for duplicates before we save the bill to the database and prepare it for processing. What do we do?

We could take the same approach as before and modify the Pay Bill service so that the duplicate check could be invoked separately from the higher level pay bill function. But that’s a bastardized design solution. Why? We are exposing behavior of the Pay Bill service that shouldn’t be exposed. The API is coarse-grained in some areas and fine-grained in others.

Maybe exposing the finer-grained capabilities of the Pay Bill function isn’t a severe compromise, but it is a compromise nonetheless. And we are forced to compromise because of architectural inflexibility. We don’t have architecture all the way down, and are therefore left with limited choice as to how we support the new requirement. We can either modify the service to reuse what we know is already encapsulated within the service, or copy the code to create something new. But those are the two options we have, and neither may be ideal.

As we continue to hack the system in this manner, with a multitude of other similar changes, the design will rot. Eventually, our Pay Bill service is transformed into a utility that performs all general-purpose bill functions. It’s API is a mix of coarse-grained and fine-grained operations. It’s become the monolith that we’re trying to avoid. While the Pay Bill service is still pretty reusable (it does everything now), it isn’t that usable. That tension between reuse and use surfaces again.

Our decision to modify the Pay Bill service to expose the duplicate check was driven by one thing - ease. It was our easiest option. Really, it was our only option. But it isn’t the best option.

If we architect all the way down, we have another option. A Pay Bill service composed of modules that audit the bill, check for duplicates, remit payment, and reconcile the payment means we can choose the desirable solution over the easiest short term choice. We can avoid the long term compromises that degrade architectural integrity.

Shown at right (click to enlarge), we see the service composed of modules. If we have a check for duplicates module, we can simply reuse that module in the new service or application that’s going to process electronic bills. Or we might expose the capabilities of the check for duplicates module as a separate service. We have multiple reuse entry points, as well as different levels of granularity through which our software entities can be managed, deployed, built, and much more.

To Conclude

My point here isn’t to debate the original design nor the decisions made to reuse the check for duplicates functionality in another service or application. Certainly, debating architectural and design decisions is a favorite past-time of developers. There are a variety of different ways that we can support new requirements as they emerge. Some are better than others and all are contextual.

The gist of my point is that architecture all the way down gives us options, and these options help maintain architectural integrity. They increase our options when making decisions and allow our system to accommodate unforeseen architectural shifts. I can modify an existing application or service to give me what I need. Or I can reuse an existing module that’s already available for me. Or I can compose a new service from an existing set of modules. Or I can break apart existing modules to create new modules that result in new services. And I can do a lot of this refactoring without significant impact on the existing code. This was an important point illustrated in the series of posts on Applied Modularity.

In other words, architecture all the way down helps increase architectural agility, and modularity is a key ingredient.

All too often, software process improvement initiatives fail. In a recent post discussing SEMAT, Ralph Johnson provided some words of wisdom that serve as a wonderful guide to any team about to embark on that much vaunted software process improvement initiative.

The state of the practice in software development is pretty dismal. Some groups do a great job, but most do not.  As I tell the students in my software engineering course, if you manage requirements, make sure the developers talk to each other, release working code regularly, have some sort of a systematic testing process, use build and version control tools, and periodically stop and see how you are doing and how you can improve, you will be better than 90% of the groups out there. Of course, I could be exaggerating.  Maybe it is only better than 75%.

I suppose that pretty much sums it up! Amazing how difficult we tend to make things though, heh?

There have been a few folks who believe that OSGi isn’t something that enterprise developers want. I’ve heartily disagreed with this, and have stated that the enterprise wants it’s OSGi. In The Two Faces of Modularity and OSGi, I stated:

While there is some dispute over whether vendors will expose the virtues of OSGi to the enterprise, eventually they will (SpringSource dm Server currently does). They will because the enterprise stands to gain considerably from a more modular architecture. There is benefit in a world void of classpath hell and monolithic applications. There is benefit in modularity, and eventually the enterprise will ask for OSGi. In the end, the enterprise will get what they’re asking for…

Well, it looks like that day is finally arriving. IBM recently announced the IBM Websphere Application Server V7 OSGi Application Open Alpha Program. From the program’s homepage, I quote the following:

The OSGi Applications Open Alpha brings the modularity, dynamism, and versioning of the OSGi service platform to enterprise application developers. It delivers a simple to use, lightweight programming model for web applications that combines the standard Blueprint component model with familiar Java enterprise technologies. TheOSGi Applications Open Alpha enables enterprise applications to be deployed to the WebSphere Application Server as collections of OSGi bundles, leveraging WebSphere platform enterprise qualities of service to provide the most complete and robust enterprise server for modular web applications. In addition to being easy to use, theWebSphere OSGi applications implementation addresses many of the challenges of developing and maintaining extensible Web applications through OSGi technology benefits of modularity, dynamism, and versioning.

Additionally, a post to the IBM developerWorks forum discusses some of the capabilities, which includes:

• a built-in bundle repository which can host common and versioned bundles shared between multiple applications so that each application does not deploy its own copy of each common library.
• a standardized (by the OSGI Alliance) and IBM-supported evolution of the Spring framework called the Blueprint component model. This provides the declarative assembly and simplified unit test of the Spring framework but in a standardized form that is provided as part of the WAS runtime rather than being a 3rd party library deployed as part of the application
• familiar Java EE programming model with the option of deploying web applications as a set of versioned OSGi bundles with dynamic lifecycle.
• full integration with WAS Admin and WAS platform enterprise qualities of service

This is great for enterprise developers who are tied to the Websphere Applicatin Server (WAS) platform. But don’t confuse WAS support of OSGi with WAS being a next generation application platform. Other vendors have done a great job providing platforms built atop OSGi, including SpringSource and Paremus.

And don’t confuse support of OSGi as a promise that you’ll immediately realize the benefits of modularity. We’ve made that mistake before with other technologies. There is still a lot to learn about how to design more modular software applications. While WAS support for OSGi brings many of the runtime and management benefits of modularity to the enterprise, it’s still our responsibility to craft systems with a modular architecture.

I’m still unsure of the actual release date for WAS V7, and I don’t have information surrounding all the prickly details of OSGi in WAS 7 and the Open Alpha program. But I intend to find out, and have already reached out to IBM to gather more details. There is still more to learn, so stay tuned. Maybe…just maybe…2010 will be the year that OSGi (and modularity) hits the enterprise.

We’ve completed our four part series on Applied Modularity, and I wanted to put a final wrap on the posts by highlighting a few things that may not be obvious. First, a brief review on the series.

• In Part 1, we introduced the sample application and applied PhysicalLayers pattern to separate our logical layers out into physical layers.
• In Part 2, we applied the AbstractModules (second refactoring) and AcyclicRelationships (third refactoring) patterns.
• In Part 3, we applied the SeparateAbstractions (fourth refactoring) pattern.
• In Part 4, we applied the CollocateException (fifth refactoring), IndependentDeployment (sixth refactoring), and ImplementationFactory (sevent refactoring) patterns.

Through this series of refactorings, we made considerable progress in modularizing the application using a few of the modularity patterns. The diagram at right (click to enlarge) illustrates the progress we made. We started with everything bundled into a single WAR file and wound up with a highly modularized system that satisfied the evolutionary requirements. Aside from the many advantages we spoke about in each post, I want to take a moment to explore a few other thoughts.

A Note On Module Testing

If you’ve explored (and built) the system by getting the code from the Google code repository, you’ll notice that there are a corresponding set of test modules for each module that we’ve created. These can be found in the bin directory (shown at right). Like we do with unit testing, I’ve tried to create a test component for each module in the system. Unfortunately, there’s a flaw in the billtest.jar module.

Similar to unit testing, where we create mocks and stubs to avoid undesirable dependencies, a test module shouldn’t pull in other modules that contain implementation classes. Instead, we should create mocks or stubs to avoid this situation. In other words, a test module should only be dependent on the same set of modules as the module it’s testing. Unfortunately, the billtest.jar module breaks this rule by leveraging the the AuditFacade implementations. That means the billtest.jar module is also dependent on the audit1.jar and audit2.jar modules, but the bill.jar module is not. So billtest.jar is really a module integration test, not a module unit test. It could easily be fixed by creating a mock AuditFacade implementation that lived in the billtest.jar module.

This begs another question….

How do we keep track of module relationships so that we recognize when something bad like this happens?

Even for small systems, without a module system like OSGi, it can be incredibly challenging.

A Note On Managing Modules

Modularizing a system on a platform that doesn’t support modularity is challenging. Hell, modularizing a system on a platform that does support modularity is challenging! One of the greatest challenges is in managing module dependencies. Tracking the dependencies between modules is really quite difficult.

This is where module systems like OSGi really help by enforcing the declared module dependencies. In plain ole Java today, there is no notion of module so there is nothing to help enforce modularity. And the first unwanted dependency that creeps into our system compromises architectural integrity. This is where JarAnalyzer can be helpful. By incorporating JarAnalyzer into my build script, I’m able to more easily manage the dependencies between modules.

JarAnalyzer has two output formats. The first is a GraphViz compliant dot file that can be easily converted to an image showing module relationships. The image at right (click to enlarge), which includes the test modules, clearly illustrates the problem with the billtest.jar module discussed above.

As can be seen, the bill.jar module has only a single outgoing dependency on the auditspec.jar module. So the module that tests the bill.jar module should not be dependent on any other modules, either. However, if you look at the billtest.jar module, you’ll see that it depends upon the audit1.jar and audit2.jar modules. So instead of using a mock or stub to test the bill.jar module, I got lazy and used the various AuditFacade implementations. Look at a few of the other modules, and you’ll discover that none include additional dependencies beyond the dependencies already present within the modules they test.

The second output format for JarAnalyzer is an html file that provides some key design quality metrics, as well as listing the dependencies among modules. Essentially, it’s a textual view of the same information provided by the visual diagram. I’ve included the Summary header of the JarAnalyzer report for the system below (click to enlarge). You can also browse the complete JarAnalyzer HTML report for the final version of the system. There is also a version that omits the test modules.

Look at the auditspec.jar module. Note that it has 8 incoming dependencies (afferent coupling) and 0 outgoing dependencies (efferent coupling). It’s abstractness is 0.67 and Instability is 0.00. This is a pretty good sign. Why? It’s instability is very low, implying it’s highly resistant to change. It possesses this resistance to change because of the large number of incoming dependencies. Any change to this module may have serious implications (ie. the ripple effect of change). But because it’s quite abstract, it’s less likely to change than a module with a lot implementation classes. The Distance for the module is 0.33 (ideal is 0.00), so we’re not far from where we ideally want to be.

In case you’re wondering about all these metrics I’m rambling about, you might want to take a look at the Martin Metrics. In general, without a utility like JarAnalyzer (or a module framework like OSGi), it would have been incredibly difficult to manage the modules composing this system.

A Note on Module Reuse

The reuse/release equivalency principles states that the unit of reuse is the unit of release. Modules are a unit of release, and therefore are a unit of reuse. Naturally, the devil is in the details, and we’re going to discuss these details here.

In Reuse: Is the Dream Dead, I spoke of the tension between reuse and use. That tension is evidently at play here. Earlier versions of the system had coarser-grained modules that were easier to use but more difficult to reuse. As we progressed, we broke these coarser-grained modules out into finer-grained modules, increasing their reusability but decreasing their ease of use. A perfect example of this is the bill.jar module. In the final version, it was quite reusable, since it was only dependent on the auditspec.jar module. However, this came at the price of useability.

To elaborate a bit more. In Part 4, the sixth refactoring, we decoupled the bill.jar and financial.jar modules so the two could be deployed independently (ie. increase reuse). But the runtime structure still has some dependencies. In order to reuse bill.jar, we need a BillPayer type. While an alternative BillPayer implementation could be created, the existing implementation is the BillPayAdapter in the mediator.jar module, which also has a relationship to the financial.jar module. This means that to use the bill.jar module without the mediator.jar and financial.jar modules would require a new consuming module to implement the BillPayer interface.

So what do we do if we want to break this runtime coupling? We should move the pay method on the Bill up to the BillPayAdapter class, and get rid of the BillPayer interface. Now the Bill class has no dependency on the BillPayer interface, but it also can’t make payments. Every action has an equal an opposite reaction, heh?

A Note on The Build

The build was a key element in helping enforce modularity (note: JarAnalyzer helped me manage module relationshps; the build enforced module relationships). Even a framework such as OSGi is only going to manage module relationships at runtime. I talk a bit more about this concept in The Two Faces of Modularity & OSGi, and it’s why we need really good tools that help us design more modular software. It’s our responsibility to craft the modules, and the build is one way to help put in place a system of checks and balances that help enforce modularity before discovering at runtime that one module has a relationship to another. In Part 2, as part of the third refactoring, we refactored our build script to a levelized build. Here’s the before and after build script.

This means that as we build each module, we include only the required modules in the build classpath. This is more easily explained by examining the build script for the final version, where you can clearly see what I’m talking about. Look at line 40. When we build the auditspec.jar module, we include nothing else in the build classpath because the auditspec.jar module doesn’t require anything. Now look at line 60, where we build the audit1.jar module. The auditspec.jar module built in the previous step is included in the classpath. This pattern recurs throughout the remainder of the script. Introducing a module dependency that violates the dependency structure enforced by the build results in a failed build.

A Note on Object Orientation

The way we managed, massaged, and modified module relationships was through OO techniques. By introducing interfaces and abstraction and allocating them to their respective modules, we were able to significantly change the module structure of the system. While we used OO to do this, OO is not a prerequisite. We could just as easily have used other techniques, such as aspects (AOP).

The Final Wrap

If you’ve made it this far through the tutorial, you’ve done well. Overall, this was a pretty lengthy and involved tutorial. In fact, I only touched briefly on all that I really had to say. Yeah, you’ve seen the abridged version here! I think I could pretty easily fill a book with the rest. Hmmm…

I use this same system and examples in quite a few of my talks on architecture and modularity. If you have questions or suggestions, feel free to drop me a line via the comments or send me an e-mail (hint: look on the About Page). Or you can track me down at a conference, as I’m always happy to discuss topics related to modularity, architecture, and agility.

I have one final deliverable. As promised way back in Part 1, I intend to show an OSGi-ified version of the system. That will follow shortly.

In part 3, we applied the SeparateAbstractions pattern so that we could independently manage the two different AuditFacade interface implementations. The result was the module relationships shown at right (click to enlarge). The code can for this version of the system can be found in the SeparateAbstractions project. Also, be sure to check out Part 1, where we introduced our initial version of the system, and Part 2 where we applied a couple of the modularity patterns. Now, we’re going to focus on the financial.jar module and try to decouple bill.jar from financial.jar so the two are completely independent of each other.

I think you’ll really like this one…But first, some additional housekeeping.

Fifth Refactoring

Before we decouple the bill.jar module from the financial.jar module, we’re going to talk about exceptions. In general, we need to answer the question.

Where do exceptions belong?

In the CollocateExceptions project, we introduce an AuditException that is thrown if an error is encountered. The CollocateExceptions pattern says that exceptions should be close to the classes (or interfaces) that throw them. Since the AuditFacade interface throws the exception (shown below), we should put the AuditException in the same module as the AuditFacade interface, which is the auditspec.jar module. Note that if we put the exception anywhere else, we’d create an unwanted dependency between modules.

Sixth Refactoring

Ok, the previous refactoring was pretty simple, and it’s time to move on. Here’s the need, and I think you’ll find the solution quite interesting.

A new requirement has emerged, and we need to use the bill.jar module in another system. In this other system, we don’t need the financial.jar module to make the payment. We just want to use the bill.jar functionality. So, what do we do?

First, recall our initial version of the class structure of the system. In the third refactoring in Part 2, we introduced a Payable interface, and then created the financial.jar module. After we finished Part 3, we were left with the module structure shown at right (click to enlarge). Again, our goal here is to eliminate the dependency between the bill.jar and financial.jar modules so that we can deploy the bill.jar module without the financial.jar module. We’re going to apply a little trick called escalation. Essentially, we’re going to escalate the dependency to a higher level module. How? First, let’s take a look at the new class structure that’s going to allow us to remove the module dependency.

As shown at right (click to enlarge), we see the new class structure that’s going to allow us to decouple the bill.jar and financial.jar modules. Things are starting to get a bit tricky here, so before we allocate these classes to their respective modules, let’s walk through the steps to illustrate how these classes will callaborate to give us what we need.

We start by passing a BillPayAdapter instance to the Bill as a BillPayer type. The BillPayAdapter is going to manage the relationship between Bill and Payment. When the pay method on Bill is called, it will turn around and invoke the generateDraft method on BillPayAdapter. The BillPayAdapter invokes the generateDraft method on Payment, passing itself in as a Payable. This allows the Payment to callback on the Payable and invoke the appropriate method (either getAmount or getAuditedAmount) on BillPayAdapter. The BillPayAdapter can now query the Bill to obtain the audited amount and pass it back to the payment. Once the Payment has this amount, it can make the payment. Whew! A fairly complex callaboration, but enough to ensure we can decouple the bill.jar module from the financial.jar module. The code for the new BillPayAdapter class, which mediates (hmm..possibly naming it a mediator would have been better, huh?) the relationship between Bill and Payment, is shown below.

Now it’s time to allocate these classes to the appropriate modules. The key decision here is where we place the BillPayAdapter. Because this controls the Bill and Payment callaboration, we can’t put it in either of those two modules. In fact, we’re going to create a new module that contains the BillPayAdapter. The new module, which we’ll call billpay.jar, will depend on both the bill.jar and financial.jar modules, as shown at right (click to enlarge). The code for this version can be found in the IndependentDeployment project.

For the most part, our system is complete now. But we do have one nasty little problem that we haven’t solved yet. We have two different AuditFacade implementations, but the current system hardcodes creation of AuditFacade1 within the AuditAction class. All this flexibility compromised by a single line of code. Unnerving how that happens! We really do have to architect all the way down.  Anyway, let’s fix this problem.

Seventh Refactoring

The problem of creating instances is pretty common, so the solution is relative easy. There are a lot of different ways to do this, but we’re going to take the easiest route (and not the most flexibile mind you) by creating an AuditFacadeFactory class, which can be found in the ImplementationFactory project. Now, instead of the AuditAction creating the AuditFacade implementation, it invokes the factory and is returned an instance, as shown below.

Wrapping Up and Getting Ready for the Postmortem

We’ve made considerable progress. Amazing really. From a single monolithic application to a fully modularized architecture with considerable flexibility. But there are a lot of very interesting takeaways that we haven’t talked about yet. A number of positive side affects have resulted that aren’t immediately obvious. In our final post in this series, we’ll take a more in-depth look at the impact of modularity.

In Part 1, we introduced the system, and broke each layer out into separate modules. In Part 2, we applied two refactorings using two different modularity patterns - AbstractModules and AcyclicRelationships. But we were still left with a problem, which we’ll solve here.

Fourth Refactoring

After the third refactoring, we were left with the modules shown at right (click to enlarge). But this doesn’t entirely solve our problem of how we deploy a new AuditFacade implementation, especially if we want to avoid redeploying the audit.jar module and everything in it. Additionally, because the two implementations of AuditFacade live in the same module, we can’t rip one out (ie. uninstall) when we add the other. In other words, because they are in the same module, they can only be managed together.

To solve this tricky little challenge, we’re going to apply the SeparateAbstractions pattern, which states that we should separate abstractions from the classes that realize them. If you’ll recall from part 2, we applied this pattern when allocating the Auditable interface to the audit.jar module, but we didn’t apply it to the AuditFacade1 class. So we’ll apply it to the AuditFacade1 implementation, as well as a new AuditFacade2 implementation. First, we have to answer the following question.

Where do we put the AuditFacade interface so that new implementations of the interface can be managed separately from the interface and other implementations?

Our answer can be found by looking at the module structure shown at right (click to enlarge). Note that I’ve also included the financial.jar module in this diagram, which was actually introduced in Part 2 when we introduced the Payable interface. We’ll use this soon, but I digress.

Anyway, we separate the AuditFacade interface out into its own module - auditspec.jar. Each of the implementations (AuditFacade1 and AuditFacade2) are also placed in their own modules. The code for this solution can be found in the SeparateAbstractions project. No real coding changes are necessary, other than we’ve added a new class - the AuditFacade2 implementation. Note that we’ve also separated each implementation out into different packages to avoid splitting packages across modules. After adding this new class, we modified our build script to allocate the classes to the appropriate modules.

A Note on the Benefit of OSGi

Now, we have an overall increase in architectural flexibility. Our AuditFacade implementation classes are allocated to separate modules, allowing us to manage the two independently. From a maintenance perspective, we can work on each of the implementations separately, resting assured that changes to one implementation won’t negatively impact changes to the other. We can also test each independently and reuse each independent of the other.

This change also increases the resiliency of the bill.jar module since it’s no longer tightly coupled to any AuditFacade implementation. We can test the bill.jar module using only the auditspec.jar module by creating a mock implementation of AuditFacade for testing purposes. While the billpay.jar module is still dependent on the financial.jar module, we’re going to solve that problem in the next refactoring. In general, we have completely eliminated the dependencies between the bill.jar module and the AuditFacade implementations in the audit1.jar and audit2.jar modules. Remember, at the beginning of Part 2, these modules containing the bill and audit behavior had a cyclic relationship.

While this example doesn’t leverage OSGi, it is important to point out the benefit that OSGi can provide here. Because we don’t have a runtime module system we still need to deploy these modules within the WAR file. The presence of OSGi, however, brings the same degree of flexibility to the runtime that we have at development time (for more information, see The Two Faces of Modularity & OSGi). With OSGi, I would be able to install and uninstall the audit1.jar and audit2.jar modules without redeploying the entire system. When we’re finished with all of our refactorings, we’ll “osgi-ify” the system, deploy it to Tomcat, and realize this benefit.

Wrapping Up and Getting Ready for Part 4

So we’ve made quite a bit of progress from our initial version of the system that completely lacked modularity. By breaking the system out into modules, we ease the maintenance effort and increase overall system flexibility. In part 4, we’re going to to turn our attention to the financial.jar module that focuses on payment. Recall that in Part 2 (when we applied the AcyclicRelationships pattern to the AcyclicRelationships version of the project) , we separated the Payment class out into a separate module and decoupled Bill and Payment through the Payable interface. Part 4 will examine how we can decouple the bill.jar module from the financial.jar module so bill.jar can be managed, tested, and deployed independent of financial.jar. Stay tuned!

See Part 1 for an overview of how we got started. In this post, we’re going to take the modularization of our application a step further. We’re going to apply two refactorings using two different modularity patterns - AbstractModules and AcyclicRelationships. First, we’re going to separate the bill and audit functionality out into separate modules so we can manage (develop, deploy, etc.) them independently. Second, we’re going to remove the cyclic dependency between these two modules.

A Quick Review

Before we get started with this second installment, let’s review what we did in Part 1. Initially, our billing application was all packaged and deployed in a single WAR file. Since we had a conceptually layered application, it was pretty easy to separate the UI layer from the business object and data access layer. Not sure we explicitly pointed it out in Part 1, but that refactoring essentially applied the PhysicalLayers pattern. After doing this, we were left with the system at right (click to enlarge).

Second Refactoring

Recall the initial class diagram, where we had a Bill class with a bi-directional relationship to the AuditFacade class. This design has two fundamental flaws. The Bill is tightly coupled to the concrete AuditFacade class and the relationship is bi-directional. Bad all around! This can be seen in the following code snippet illustrating the Bill’s audit method.

Notice that the audit method actually creates the AuditFacade, calls the audit method, and passes a reference to Bill. Ugly. Let’s clean this up a little bit. While there are obvious technology reasons why we need to clean this up, there is also a motivating business force.

The system needs to go live with the current vendor’s auditing system, but the business has indicated that they aren’t renewing the contract with the vendor and are in ongoing negotiation with another vendor. The contract expires in 6 months, but we deliver the initial version of the system in 3 months.

So, 3 months after deployment, we know we’ll need to swap out auditing systems. What to do?

The first half of this solution can be found in the AbstractComponents project, where we apply the AbstractModules pattern which states that we should depend upon the abstract elements of a module. So we’re going to refactor the AuditFacade to an interface and create a separate AuditFacade1 implementation. This solves the first half of our problem - the tight coupling between the Bill and AuditFacade implementation. The result is the class diagram shown at right (click to enlarge). Take a look at the refactored Bill class, and you’ll see that the AuditFacade interface is now passed into the Bill, allowing us to swap out AuditFacade implementations. Now we need to decide how to modularize the system.

If we take this flexible class structure and continue to deploy in the single bill.jar module, we have the flexibility to swap out AuditFacade implementations at the class level, but we’re still required to package everything up into a single bundle and deploy it together. So what we really must do is separate the audit functionality out into separate modules from the bill functionality. Separating the AuditFacade interface and AuditFacade1 implementation out into separate modules results in the diagram at right (click to enlarge). We can also see this change reflected in the build file that packages up these modules. Here’s where the bi-directional relationship between Bill and the AuditFacade interface (and AuditFacade1 implementation) rears it’s ugly head. We have a cyclic dependency between our bill.jar and audit.jar modules. We need to fix this problem.

Third Refactoring

The second half of this solution can be found in the AcyclicRelationships project. Now, we need to remove the cyclic dependency between the modules, and we’ll apply the AcyclicRelationships pattern which states that module relationships must be acyclic. To do this, we’ll introduce an additional abstraction, called Auditable, that our Bill class implements. Upon applying this little trick, we can see that we have now removed the bi-directional relationship between Bill and the AuditFacade interface, as shown at right (click to enlarge).

Whereas previously the AuditFacade accepted a Bill to the audit method, it now accepts an Auditable type. The new AuditFacade interface can be seen in the code snippet below.

The Bill will pass itself to the AuditFacade interface as an Auditable type. The key element at this point is how we allocate these classes to their respective modules. Here’s a simple rule to abide by when determining how to allocate classes and interfaces to modules.

Interfaces should be closer to the classes that use them, and farther away from the classes that implement them.

We’ll talk more about this in the next post, because we build on this idea to address another problem that has surfaced. But applying this rule now, it’s clear that Auditable should be bundled with the AuditFacade interface, not in the same module as the Bill class.

While we haven’t talked about the new financial.jar module, we applied a similar refactoring to the Payment class by implementing a Payable interface, and breaking it out into a separate module. Allocation of classes to modules is done at build time, so after modifying our build script, the result is the modules shown at right (click to enlarge).

As I’m sure you’ve noticed at this point, we broke our rule above and bundled the AuditFacade1 implementation close to the interface it implements (btw, this is the other problem that has surfaced). If we apply our rule in this situation, we’d bundle the AuditFacade interface and Bill class in the same bundle, which certainly wouldn’t work because it would result in the cyclic dependency between the bill.jar and audit.jar modules that we’ve worked so hard to remove. Alas, this is the focus of our next refactoring, so we’re going to wait to tackle this challenge.

Wrapping Up and Getting Ready for Part 3

This post involved two refactorings. The first was to separate the bill and audit functionality out into separate modules. The second was to remove the cyclic dependency between the bill and audit modules.This offers us decent flexibility. We can introduce a new AuditFacade implementation, knowing that we’d only need to rebuild the audit.jar module. We can test the audit.jar module independent of any other module because it doesn’t have any outgoing dependencies. And if we wanted to, we could deploy the audit functionality separately (ie. we can reuse it elsewhere). So overall, some decent progress.

But some problems remain. If we really want the ability to swap out Audit systems, bundling the AuditFacade interface and AuditFacade1 implementation into the same module doesn’t give us the flexbility we need. While we can easily create a new AuditFacade implementation and allocate it to the audit.jar, this requires us to redeploy audit.jar unnecessarily. In the next post, we’ll explore how we can change the module structure to allow a new AuditFacade implementation while also allowing the existing AuditFacade1 implementation to be removed from the system.

This is my first in a series of posts that shows how we can modularize Java applications today. A few notes before we get started. First, the examples are all core Java, with no OSGi. I’ll explain why in just a moment. Second, I’ve applied these techniques on real projects, so I know that they work.

Why No OSGi?

So why didn’t I use OSGi? Good question. One would think that if I’m going to modularize my system, I’d want to use a module framework. There are a couple of reason I chose not to.

• While OSGi is a module system, OSGi will not help you design more modular software. Among other things, designing modular software requires that we understand the weight and granularity of individual modules, and use the right techniques to manage the coupling between modules. In other words, designing good software is our job. Tools and technologies may help, but make no guarantee. I spend more time talking about this in The Two Faces of Modularity & OSGi.
• Most of us aren’t able to leverage OSGi today because the platforms and langauges we use don’t support it. I want to use the same tools and techniques that we can leverage in the enterprise right now. This is where the modularity patterns, as well as a few other tools, will help.

In general, I really want to focus on the design paradigm, not the tools and technologies. So instead of answering, “How do I use OSGi?”, I want to focus on “How do I modularize my system?”. Hopefully, that resonates with you. For the curious, if you do want to see the difference that OSGi makes, I have OSGi-ified the final version. But you’ll have to wait.

Let’s Get Started - The System

Interestingly, I’ve found that when designing modular software, it’s tough to identify the modules early in the lifecycle. Instead, shifts typically occur, and as things unfold, the modules become more apparent as development progresses. With a SOLID OO design, it’ll make it much easier to move things around and create new modules. So to start, while the system is small, I favor larger (coarser-grained and heavier-weight) modules.

As specific needs emerge, we’ll break larger modules out into a bunch of smaller (finer-grained and lighter-weight) modules that address specific needs (both functional and non-functional requirements). If you’re interested in the abstract essence of what I’m referring to here, you should read some of my prior blog posts (Agile Architecture might be a good place to start). It’s a deep and very interesting topic, and impacts how we understand software, maintain the system, reuse software entities, and more. Or, you can wait and see what I’m talking about, as we’ll experience this phenomenon as we move through the exercise.

Here’s a simple, high-level description of the system we’ll develop. It’s the common bill payment sample system often used.

We’ve been asked to develop a system to handle payment of of bills. Prior to paying the bill, the system should apply a discount to the bill in an amount that has been negotiated with the payee (we call this the process of auditing the bill). Applying this discount is a fairly complex process, and a 3rd party vendor has been commissioned that will apply this discount. Additionally, we must integrate with a legacy financials system that must be fed payment information for reconciliation.

We’ll flesh out additional details as development progresses.

Version One

The initial version for this system uses Struts as the web framework, and packages everything into a single WAR file. The initial class diagram can be seen at right (click to enlarge). It’s not a complete class diagram, but it does show the main abstractions. I’ve greatly simplified the system for purposes of example.

As you can see, there are some Action and ActionForm classes that leverage Struts. There are also a couple of JSP you don’t see here, but will if you take a look at the project. A Customer has a list of Bills, and each Bill has a reference to an AuditFacade and Payment class. The AuditFacade integrates with the 3rd party vendor software that applies the discount, and the Payment class integrates with the legacy financials system. Of particular interest, note the bi-directional relationship between Bill and these classes - a sure sign of a problem that will haunt us later. But don’t worry, we’ll fix it.

For the sake of simplicity, I’ve hardcoded the database into the data access layer, which is represented by the Loader interfaces. This way, you can experiment with the system without running any DDL scripts to create the database. If you really want to, feel free to add a real database backend. It wouldn’t be very difficult, but it’s not what I want to focus on here. The code for this initial version can be found in my Google Code Repository.

First Refactoring

So packaging everything up into a single WAR file for deployment certainly isn’t modular. In most systems we develop, we try to design layers that encapsulate specific behaviors and isolate certain types of change. Typical layers include a UI layer, a business or domain object layer, and a data access layer. In this system, we have these three layers. The Struts action and Form classes represent a part of the UI layer, which is shown in red. The Customer, Bill, Name, AuditFacade, and Payment form the business object layer, and the Loader classes form the data access layer. These classes are shown in blue. Now, here’s a key statement that you need to take with you.

If I truly have a layered system, then I should be able to break out each layer into a separate module where modules in the upper layers depend on modules in lower layers, but not vice versa.

If you try this, it’s likely you’ll find it’s not so easy. The takeaway here? Most development teams feel they have a layered architecture, but in reality, they don’t because somewhere deep within the bowels of the system lies an import or reference to a class higher up in the food chain that we aren’t aware of.

In fact, if I really do have a layered system, then I shouldn’t have to change anything other than my build script to break the layers out into separate JAR files. If I do have to change more than a build script, then I didn’t have a layered system to begin with, and I should perform some architectural refactoring to clean things up. Anyway, the end result is relatively simple to understand. No code changes. Only a build script change. And the structure shown at right is the result (click to enlarge). You can view the refactored project in the PhysicalLayers project, which includes the change to the build script on line 41.

Wrapping Up and Getting Ready for Part 2

This first refactoring was quite simple, but has significant implications. Foremost, it proves that my class level architecture was pretty decent. I was able to break the system out into modules for the various layers without having to change a bunch of code. Really, that’s the reason why it was so simple…because the design was decent. Had the design been shoddy, it would have been significantly more difficult pulling off this refactoring. Trust me!

Yet, as we’ll see, the existing design may meet the needs of today, but it’s going to have to evolve as change emerges. In part 2 of this series, we’ll take a look at what we need to do to integrate with another auditing system, and how modularity can help us do this. And as we progress, the amazing transformation of a system lacking modularity to a highly modularized version will unfold. Stay tuned!