Simple and Elegant

Tue, 13 Apr 2010 07:57:59 -0400

The best software designs are often the simplest, even when solving the most complex problems. This can be a challenge when building distributed software systems, but simplicity should still be the goal. In fact, much innovation in the software space is the result of trying to simplify something otherwise complex. For instance, the web makes it easier to navigate the intricacies of the Internet. I recall the days before the web, manually navigating newsgroups and directory trees via anonymous FTP. Web services extended this simplicity to inter application communication, an otherwise complex topic.

However, it seems that complexity still finds its way into our lives, either as a technology consumer or as a software engineer. I've always tried to avoid this complexity, but I've seen others look at it almost as a selling point. I disagree. In fact, I think complexity is a sign of weakness.

In my opinion, if a software package is hard to install, configure, or to otherwise get up and running, I avoid it. I'm not being arrogant, I'm speaking from experience; I've used plenty of overly complex systems, and I've certainly built my share as well. I admit it. That's why I take such a strong view against complexity. Therefore, I strive for simplicity.

Eric's first rule: Simplicity Equals Elegance

I find that simple software design often ensures success, but it also helps to maintain the sanity of those who use that software. How much time have you spent wrestling to get a web server, application server, or database server up and running, and configured properly, for example? It's no accident that the main goal of Java EE 6 was to simplify Java EE development and deployment. Simplicity is also why thousands of developers have moved to other languages and environments such as Ruby to build web applications.

But when you need to connect more than one computer together, and arrange to have software communicate and coordinate activities between them, a certain level of complexity is unavoidable. For instance, a network stack consists of many low-level software components coordinating bits of data transmitted by fluctuations of electricity over a wire and though a network interface card. There's very little simple about that, even at a high level. But computer science offers solutions to help here as well.

Abstraction, or "Just wrap it!"

As a user or developer of network software, life is relatively simple today because operating systems and software libraries provide a nice facade to work with. Behind these facades may lie a level of complexity you're not ready or willing to deal with, but thanks to clever design you may never have to. This is called abstraction, where you gain the ability to treat an entire system of seething complexity as a simple box in a diagram, or entity in your mind. Not everyone has the ability to abstract to the level required to deal with such things. Just as the most famous artists have used it to amaze us with their work, the best computer scientists use it to solve complex problems with (sometimes only apparent) simplicity. This is what computer scientists often refer to as elegance.

Therefore, I propose that the transitive of my first rule of design is also true:
Eric's second rule: Elegance equals simplicity.

I once worked with a developer who used to talk about a fictional product he dreamed of creating called "Wrap App." This revolutionary tool would offer the ultimate in abstraction and wrap every possible problem in a veil of simplicity. With it, new applications could be created instantly with a click of a button. Awesome vision and goal! It's not the sort of cerebral challenge most developers crave; but from a business point of view it's the brass ring. But even I'll admit, this goal may involve too much simplicity to ever be a reality.

Turn it Inside-Out

Simple and elegant software design should go beyond just the code that makes the system work, it should extend to the user interface as well. This may be obvious for applications with a graphical interface, for which there are ample guidelines to good design, but it also applies to applications with more abstract interfaces: the tools and systems that we use to develop and deploy other software. For instance, why has FTP lasted so long in a world where constant software upgrades are a fact of life? Because it just works, out of the box. This is how all software should be. Don't make installation and configuration an afterthought.

This leads us to my next rule:
Eric's third rule: Turn your design inside out.

A good internal design should be turned inside out, and it's usage should be as elegant as it's inner workings. If it's pretty on the inside, it should at least be as pretty on outside. The inverse is true for those whose focus may be on the outside (i.e. graphic artists). If you're a UI designer and also a programmer, try not to sacrifice the elegance of your software's internal design for the sake of quickly iterating the UI. I've seen far too many prototypes (with hacked code never meant to survive) go on to become the actual product. Again, I speak from experience; this is another software crime I'm guilty of.

Conclusion

Although there are countless rules, guidelines, and methodologies to help achieve good software design, the best ones are those that lead to the greatest success. How you measure success may vary: high performance, stability, robustness, scalability, ease of code modification, and so on. But I've found that my three simple high-level rules for software design are always a good start:

1) Simplicity equals elegance: strive for the simplest solutions involving fewest "moving pieces".
2) Elegance equals simplicity: apply abstraction to simplify even the toughest of problems.
3) Turn your design inside-out: make your software simple and elegant to use as well.

Enterprise Mash-Ups Matter

Thu, 1 Apr 2010 23:12:26 -0400

Software re-use is an over-used phrase from the last century that never really lived up to its promise. Perhaps that’s because the real value isn’t in reusing lines of code, but the data created by its users instead. In recent years, web 2.0 mashups—web applications that combine data from two or more sources—have become popular in the consumer space. For instance, combining Google Maps with vacation data and photos on Flickr makes for an interesting mashup. Of course, to do this, you don’t need to touch a line of Google’s code. All you need is access to its data; and to take a vacation, of course.

Many popular sites, such as Amazon, Flickr, Google, and YouTube, expose data via APIs especially for mashups. Other forms of data come from RSS feeds, and screen scraping. Often, the client portion of the application includes HTML and JavaScript for a dynamic, Ajax-based, experience. In some cases, the mashup of the data sources can be done completely with Ajax, with no need to deploy server-side application code. In other cases, a web application with server-side code is used, and may be combined with Ajax. The elegance of mahups is that there really are no rules, except that you shouldn’t violate copyright laws.

Mashups in the Enterprise

An enterprise application is one that is used in a corporate atmosphere to drive key business processes or serve customers, where security and reliability are crucial. Given its popularity in the consumer space, some say it was only a matter of time before mashups entered the enterprise space. In my opinion, they were actually conceived in the enterprise space. After all, web services were in vogue back in 2000, well before a single consumer mashup site hit the web.

In fact, any application based on a service-oriented architecture (SOA) is technically a mashup. SOA, clearly an enterprise endeavor, is the tool behind the promise of enterprise mashups. RSS feeds and the broadcasting of data via publish/subscribe protocols have been in use in the enterprise for even longer. And since many enterprise applications have become web-based since 2000, they satisfy that requirement as well.

However, the differences between consumer and enterprise mashups go beyond their target audiences; there are additional requirements for the enterprise. For instance, you need a framework that combines corporate data from SOA components, Ajax-enabled web interfaces, and built-in features that satisfy security, reliability, and corporate compliance requirements. It also needs to be based on standards.

Dynamic Business Applications

Perhaps the most important difference is that enterprise mashups need to be more dynamic; they need to provide full-cycle access to data. Users should be able to add, modify, move, contribute to, or somehow transform and aggregate the data the application uses. This two-way data cycle, combined with the traditional Internet protocols and Ajax capabilities of modern web 2.0 applications, results in what some call the Dynamic Business Application.

The tools for such applications have been around for some time. For example, Microsoft BizTalk was originally launched in the late 1990’s. Its goal was to allow developers to easily gather data, transform it, and orchestrate users’ activities using that data. That sounds like a mashup to me. Apache Cocoon provides similar orchestration features, and when combined with the information pipeline of an enterprise service bus, you have a solid foundation for dynamic business application development and deployment.

Enterprise Mashups in Action

There are many types of corporate applications that make sense as an enterprise mashup. Some examples that come to mind are:

• Management dashboards: whenever resources need to be managed remotely by individual operators, dashboard applications are used. Examples are network management, fleet management, power plants, railroads, and so on. Data from many locations needs to be aggregated and displayed remotely.

• Financial applications: data from multiple sources needs to be combined for investors, traders, and business people to make decisions. Combining data in unique ways, with proprietary algorithms, is a selling point. Mashups make it quick and easy to build and deploy new features.

• Call centers: typically, call centers rely on taking live data from a caller, using it to search a database for information (perhaps to solve a problem), and then add it back to the database for logging, escalation, or follow-up.

• Employee resources: corporations are beginning to combine their internal applications and associated data to build web-based applications for shared calendaring, name lookup, conference room booking, and corporate travel. These applications may have existed separately before, but data was duplicated across. Enterprise mashups allow them to share and combine data for increased value.

• Map-based applications: anyone who needs to combine statistics with location data (photographers, police officers, travel agents, fleet operators, reporters, and so on) can benefit from mashups that combine their unique data with Google maps, for instance.

Of course, this list isn’t complete; it’s limited only by the infinite needs of corporations globally, and the ingenuity of developers and other business-focused individuals.

Take Reuters Knowledge as an example of an enterprise mashup application in action (I worked on this project from its inception). This application combines multiple internal Thomson Reuters data sources with some external ones, aggregates the data in unique ways, and makes it available through a web interface designed for investment bankers. All that’s needed for the user is a supported web browser, and a broadband Internet connection.

The Future of Mashups

Viewing an enterprise mashup as a dynamic business application goes a long way to demonstrating its usefulness behind the corporate firewall. The goals of just about every corporation include the reuse of application logic and data, the need for two-way data communication, and the desire to quickly and easily provide new applications and features to its internal and external users. Enterprise mashups, a.k.a. dynamic business applications, meet these goals head on.

For more information on enterprise mashups, including sample applications, see the following:

• Mashup definition on Wikipedia
• Reuters Knowledge
• Chicago Police Department crime viewer
• Microsoft BizTalk Server
• Cocoon
• Enterprise Service Bus
• Google APIs (for maps and more)
• Dynamic Business Applications

Federated Software Architecture

Wed, 17 Sep 2008 22:15:07 -0400

I've worked in large companies, and I've worked in small companies. However, a common problem I've seen in even small companies is duplicated effort. For instance, a product group will kick off a new project based on an idea, a competitive product, or market research, and a development group will go off to design and build it. Often, the design and development is done in a silo, or with limited visibility to the rest of the organization. The unfortunate result is that portions of the system being designed may already exist, but are instead redesigned and rebuilt because no one on the team knew about them.

One attempt I've witnessed to avoid this situation is the use of a global architecture. In this model, a group of architecture masters define golden designs, using a pre-defined set of technologies that no one can stray from, and they cast them down from their ivory towers for all the mindless developer citizens to follow. Obviously, in the company I witnessed this at, it didn't work, and it wasn't enjoyable to be a part of. There was no room for creativity or feedback from the individual development groups.

Instead, borrowing from the construction industry, what I propose is more of a building code for software design, complete with the need to acquire "building permits" for software construction. For companies with multiple, distributed, development groups, what I feel is also needed is a company-wide group that oversees, but does not necessarily dictate or govern, software design direction. I call this group of people the Design Support Group (DSG) .

Building Permits

A company-wide DSG would put procedures in place that would require development groups to get what is the equivalent a “building permit.” If this is done correctly, it should also relieve project managers of a lot of the work in getting a project off the ground.

Once defined and agreed upon by all development groups, the “building permit” procedure is a process by which a project must pass its plans. This is called the DSG review procedure, and it includes details such as development languages to be used, basic architectural principals, and hardware, software, and OSs to be deployed. The DSG would also offer its suggestions as to how to design and build the product according to its expertise and experience. “Permits” would be granted so long as the project team plans to build the product according to a flexible "building code."

In the end, given that all software design passes through the DSG, people will be alerted to the possibility of duplicated effort, and it can be avoided before too much time is wasted. Additionally, new and creative solutions can be shared (instead of stifled) for the good of all future development efforts. Overall, the DSG should operate in a flexible way, offering assistance (not roadblocks) to designers and developers.

Management Relief

The DSG would also help project managers maintain project scope, and prevent feature creep. By holding periodic “building permit” meetings in the course of product development, the DSG would ensure that all procedures are being followed. It is at this point that the DSG would help to prevent non-approved features from being implemented.

As a result, project management would then have outside, corporate, support in not being forced into over-committing to added features late in the process. This isn't meant to serve as resistance to customer needs or new feature ideas; it's simply a procedure to ensure that all new feature ideas raised during the course of development get due diligence in terms of scope, design, scheduling, and resourcing.

These additional features / requirements should be captured and funneled into their own, separate, project or release (see my Bus Stop Releases blog here). You now have an iterative process that incrementally builds a product in phases. These phases would theoretically be much smaller in size than traditional project development phases as they maintain their scope.

Respect

Depending upon the size of your organization, the DSG can be structured such that there are satellite DSGs local to each development group or geographic region. A central DSG Project Office can be set up to unify the satellite DSGs, and to coordinate their efforts. Together, permits would be granted, and standards and procedures would be conceived and put in place. The most important aspect of this is that the local satellite DSGs should be made up of knowledgeable, experienced, and respected individuals.

Conclusion

Is it possible to implement a company-wide DSG, with a standard architecture and procedures? I think so. As a matter of fact, I think it’s necessary for companies that maintain distributed development groups. Therefore, instead of corporate architecture and technology mandates, pre-approved system designs, and stifled technology choices, think about implementing a procedure that maintains control and avoids duplication, but also allows creativity in design and development.

What does your company do to regulate software development, and avoid duplication and rogue software systems? Write back in the comments section below and share with your fellow readers.

Bus Stop Releases

Sat, 6 Sep 2008 17:44:50 -0400

Bus Stop Releases

I often hear of development groups doing weekly or nightly builds, software releases based on feature sets, or releases in the form of updates (such as Sun with Solaris 10 and Java SE 6, and Apple with OS enhancements). What used to be called patches in the past, and often reserved for bug fixes, are now generally accepted as vehicles to deliver enhancements to end-user software. Most system OS vendors do this, including Microsoft's Windows (all versions), Apple's OS X, iPhone, and even the iPod.

What I don't hear a lot about is the concept of a bus stop release when new software versions are developed. I've been part of development groups where we had release cycles that went out 12 months in some cases. In contrast, other projects I've been involved with had release cycles measured in weeks. In all of these cases, there were usually lists off features that were mandated for inclusion in each release, and if a feature(s) took longer to complete than anticipated, the entire release was delayed.

This can make sense in some cases, but in many cases, it wastes internal resources, and potentially keeps anxious customers waiting for the other, already complete, features. Instead, I propose the bus stop release paradigm. Here, releases are scheduled by date according to a bigger-picture view of what each release is about. When each date approaches, there will invariably be a feature or two that aren't ready. When that happens, it's just too bad; the release is deployed anyway.

We called these "bus stop releases" because it works like a bus on a schedule. When the bus comes, the features that are ready and at the bus stop get on the bus for delivery to customers (or onto web servers). The late ones make the next bus (release). When we first deployed this paradigm, we were concerned that late features would cause backlogs and affect each of the later releases. However, we realized that in reality, the previous method of delaying an entire release because of one or two late features impacted future releases even more. One or two deferred features (which were mostly complete when they missed the bus) had much less impact on the future schedule.

In the end, we found that bus stop releases kept the entire development organization (which includes more than just developers, such as the managers, testers, the build group, tech writers, and customer support) utilized more efficiently, it kept our customers happier as they could expect (and plan for) software updates more regularly and predictable, and it helped us realistically schedule future releases as we could measure more precisely why feature delays occurred in the first place.

What release cycle does your organization use? How do you handle delays and missed schedules? Write back in the comments section and share your experiences.

Happy coding!
-EJB

Requirements Traceability Matrix

Fri, 1 Aug 2008 13:13:07 -0400

When embarking on a new software project, there are many development methodologies that you can follow. Of course, there's the classic waterfall model, where you define all of the requirements, then you design the entire system, then you code the entire system, then you test the entire system, and you don't release it until all of the features are complete. This makes sense on paper, but in practice it's inflexible, and impractical as there's no room for change along the way.

Iterative Development

An alternative is to use an iterative approach, where you break the overall project into sub-projects (possibly by requirement) and you create a pipeline for each. For instance, while one requirement is nailed down and being coded, others might still be in the works, others might already be in testing, and others might loop back around to requirements definition even though coding already started for them. It's truly iterative, in that you go through the "define / code / test" cycle multiple times, even for the same requirement(s).

From this concept, Model-driven Engineering (MDE) was born, with its variants such as Rational's Unified Process based on UML, and the OMG's Model-driven Architecture (MDA). I particularly like UML as it helps me visualize the components in a large software system, and it goes a long way to communicating a system's design with others. In my opinion, a handful of good UML diagrams tell me more about a system then dozens of written pages.

CMM

The Capability Maturity Model (CMM), defined by the Software Engineering Institute (SEI), defines a five-level process to software development. It describes the steps that a software development team should follow to attain a repeatable, successful, software development process. It also defines five possible levels that a development organization might be in, level 1 being the base level. After being assessed on certain points, such as how well documented requirements, design, and testing are, as well as how well progress is measured, an organization is deemed to be at one of the five levels.

Some developers balk at all of the formality involved with CMM, and the process-heavy approach they're often required to follow. Others like the structure and repeatability it fosters. However, most developers will agree that some process and documentation are small prices to pay to avoid the problems and possible project cancelations that might result without them. Perhaps the biggest strength of CMM is that it's flexible enough to work with any software methodology.

Agile Development and XP

Agile Software Development is a methodology that encompasses other software development methodologies. More importantly, it combines software development process with an iterative approach, group collaboration, adaptability, and software life-cycle issues. After all, a software system's life doesn't end after it exits testing. The need to deal with full life-cycle issues (i.e. customer support, patch deployment, and capturing customer feedback) is often underestimated.

Extreme Programming (XP), made popular by Kent Beck from his work at Chrysler, is form of Agile Programming. Its goals are to consider the humanity in software development just as much as the process. It focuses equally on the social side of programming, setting it out to make it as enjoyable as it is successful. The idea is that happy developers are productive developers. As a result, the methodology combines process with style, discipline, communication, feedback, and basic teamwork.

The SOYP

Unfortunately, all too often, most development teams use the SOYP methodology; otherwise known as the Seat Of Your Pants process (or lack thereof). I've found that it's best to follow some sort of process. Even if you don't choose to follow one strictly, you'll find more success if you follow portions of the methodologies defined above.

For instance, I've combined the use of UML diagrams, with the small group communication techniques found in XP, and the need to document and execute unit tests, without adopting all of the related details of the Unified Process, or XP, or CMM. On many projects and with most development teams, this can lead to more success than going too far in either direction (SOYP versus strict process). However, I've found one component critical to success in all of these projects; the RTM.

Requirements Traceability Matrix (RTM) Defined

Regardless of the process followed on the projects I've been involved in, I've always introduced the concept of the requirements traceability matrix (RTM). This is implemented as a simple spreadsheet, with each row of the spreadsheet dedicated to a requirement to be implemented. Included are spreadsheet columns for the following:

-A unique requirements identifier (i.e. a number)
-A description of the requirement
-A pointer to a design document for the requirement
-A pointer to the unit test document for the requirement
-A pointer to the test results after testing the requirement
-A color-coded (i.e. red, yellow, or green) status column for the requirement
-A column for sign-off for the requirement's inclusion in the next release

Each associated document, and all communication about the project (email, and so on) refer back to the requirements by requirement ID. This provides good two-way reconciliation between the RTM, and the set of design, test, and test-result documents. Meetings to measure project status and development progress always revolve around the RTM, and the requirements IDs contained within.

I've found that this simple form of project accounting helps to keep the project organized, and helps to focus all of the stakeholders. It also helps to ensure that nothing falls through the cracks. For instance, if a design document is missing for a particular requirement/feature, it will be obvious when you look at the spreadsheet. If the implementation of a particular requirement is not going well, the status column will show this (and you can sort them by status if needed).