Thursday, December 20, 2007

Software Design Principles

Rapidly changing customer requirements and accelerating technological advancements are becoming increasingly more complex. Faced with intensifying competition and tightening operating budgets, a growing number of companies can no longer tolerate the escalating costs and disappointing return on investment of legacy applications. Many of these organizations are now seeking to standardize platform that can more easily and cost-effectively meet their business needs. They are also interested in thinking about the future and selecting the right development and delivery platform to support their future requirements.

The software systems should be complex, but some of them are still not complex. The largely forgettable applications that are specified, constructed, maintained, and used by the same person, usually the amateur programmer or the professional developer working in isolation. Not every systems are crude and inelegant and it's doesn't mean to belittle their creators. Such systems tend to have a very limited purpose and a very short life span.

The distinguishing characteristic of enterprise software is that it is intensely difficult, if not impossible, for the individual developer to comprehend all the subtleties of its design. Stated in blunt terms, the complexity of such systems exceeds the human intellectual capacity. This complexity is an essential property of all large software systems.

Problems which we are trying to solve

One of the biggest challenges today for product managers is clarifying the role of product management. In software companies, this is especially challenging as development teams are adopting more agile or iterative development methodologies to meet the increased demands to develop more code faster.

The strategic role of product management is president of the product. Running the product as a business. Taking a marketdriven approach to product management. Making decisions about the product to increase adoption across the market, delight customers, and ultimately deliver profitable products. Even if your product is “free,” it should contribute to profits somewhere in the company.

But instead of being “president of the product,” product managers often feel like “janitor of the product.” Cleaning up after everyone instead of leading the team. Picking out icons, writing error messages, answering 30 questions a day because the developers don’t want written requirements.

On the surface, the problem seems to be that product management and development are each trying to wrestle control from the other. Product management is frustrated at what appears to be a lack of commitment from development. Development is frustrated because product management won’t commit to what they want and keep changing their minds. Even customers don’t know what they want.

Software development involves taking abstract requirements and building a system to satisfy those requirements. Going from the abstract concepts to running software is a type of invention - the development team comes up with a solution to meet the business need. However, there are multiple possible solutions that can conceivably meet the business needs. How do you determine which is a better solution than the other? The project manager should determine the better solution and communicate that to the team correctly will deliver business value as well.

A further complication is that the requirements of a software system often change during its development, largely because the very existence of a software development project alters the rules of the problem. Seeing early products, such as design documents and prototypes, and then using a system once it is installed and operational are forcing functions that lead users to better understand and articulate their real needs. At the same time, this process helps developers master the problem domain, enabling them to ask better questions that illuminate the dark corners of a system’s desired behavior.

This external complexity usually springs from the “communication gap” that exists between the customer of a system and its developers. Customer generally find it very hard to give precise expression to his needs in a form that developers can understand. In some cases, customer may have only vague ideas of what hi want in a software system. Customer and developers have different perspectives on the nature of the problem and make different assumptions regarding the nature of the solution. Actually, even if customer had perfect knowledge of his needs, developers currently have few instruments for precisely capturing these requirements. The common way to express requirements is with large volumes of text, occasionally accompanied by a few drawings. Such documents are difficult to comprehend, are open to varying interpretations, and too often contain elements that are designs rather than essential requirements.

A company of the software can start to earn money earlier if it is a commercial product. Reducing time brings more value to the company because company begin to use the product earlier. Some early use out of a subset of functionality can increase business value. That help company release early and often provide business value to customers who are concerned with time to market.

Products that get to market faster have the potential of getting market feedback earlier. So there is an opportunity for the team to increase the product’s usefulness to the customer by frequently incorporating concrete feedback. That help your team take advantage of this information will also increase the value to market.

Because a large software system is a capital investment, project manager cannot afford to scrap an existing system every time its requirements change. Planned or not, systems tend to evolve over time, a condition that is often incorrectly labeled software maintenance. To be more precise, it is maintenance when we correct errors; it is evolution when we respond to changing requirements; it is preservation when we continue to use extraordinary means to keep an ancient and decaying piece of software in operation. Unfortunately, reality suggests that an inordinate percentage of software development resources are spent on software preservation.

Project activities

Here’s the work that needs to get done:
1. Problem definition
2. Analysis and design
3. Build and test

These are the necessary phases to building good software, regardless of the development methodology used. Depending on which approach team are use, the level of written documents and the scope of what is done will change. With waterfall, we will do a lot of work in each phase before moving to the next phase. The written documents will be more detailed. With agile, there is less written documentation and the scope for each phase is smaller.

The first step in any software design process is to define the process and exactly what it is that the customer actually wants. While this may seem to be a simple and obvious step (and in fact, is often skipped by other companies), this to be one of the most important steps in the design and development process. It is through this process that Software Principles and customer come to a mutual understanding and agreement of what will be delivered.

The design of the necessary protocols and software could be guided by the following principles:
  1. The software should be loosely coupled. Given uncertainties such as the volatility of the technologies involved, it is likely that middleware will go through a rapid evolution in the next few years. It is important to realize that the architecture of a complex system is a function of its components as well as the hierarchic relationships among these components.
  2. Software deployments should demonstrate early wins. Given the political aspects of middleware deployment, it will be very useful to show immediate benefits of early components. This will help motivate the significant institutional investments that will be required. Individual components should have value in themselves as well as in concert.
  3. Make software as economically and technically cheap as possible. IT organizations in higher education have limited resources. Financial stresses and employee retention issues suggest keeping software and expertise costs low. The choice of what components in a system are primitive is relatively arbitrary and is largely up to the discretion of the observer of the system. What is primitive for one observer may be at a much higher level of abstraction for another. Common Patterns should be used. Many complex systems are implemented with an economy of expression.(In other words, complex systems have common patterns.)
  4. The software systems should accommodate the distinctive aspects of higher education. The higher education IT environment has a number of special characteristics, such as the migratory workstation habits of students, traditions of academic freedom and privacy, and the legal requirements of public institutions. Middleware solutions must accommodate these characterisitics. (Stable Intermediate Forms.Complex systems tend to evolve over time. Complex systems will evolve from simple systems much more rapidly if there are stable intermediate forms than if there are not. A complex system that works is invariably found to have evolved from a simple system that worked. . . . A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system. As systems evolve, objects that were once considered complex become the primitive objects on which more complex systems are built. Furthermore, we can never craft these primitive objects correctly the first time: We must use them in context first and then improve them over time as we learn more about the real behavior of
    the system.)
  5. The software should be easy to use. End users prefer natural naming and intuitive tools. Users may not be able to handle complexity in management of middleware components or personal data.

The fundamental task of the software development team is to engineer the illusion of simplicity - to shield users from this vast and often arbitrary external complexity. Certainly, size is no great virtue in a software system. Strive to write less code by inventing clever and powerful mechanisms that give us this illusion of simplicity, as well as by reusing frameworks of existing designs and code. However, the sheer volume of a system’s requirements is sometimes inescapable and forces us either to write a large amount of new software or to reuse existing software in novel ways.

The customer’s ability to see the true state of the project as it progresses. This is important because it allows the customer to steer the software project and also manage risks and expectations. On the other hand, software practices that increase visibility will allow customers to get the most benefit throughout the project development cycle and engender trust and cooperation with customers. The software developing should be faster, better, cheaper. That’s what company must all do to survive. This business value is about building the system for less. Just a few decades ago, assembly language programs of only a few thousand lines of code stressed the limits of our software engineering abilities. Today, it is not unusual to find delivered systems whose size is measured in hundreds of thousands or even millions of lines of code (and all of that in a highorder programming language, as well). No one person can ever understand such a system completely. Even if decompose our implementation in meaningful ways, still end up with hundreds and sometimes thousands of separate modules. This amount of work demands that use a team of developers, and ideally use as small a team as possible.

Start with a small team and get experience in adoption. Most small teams that get appropriate coaching make the transition to Agile practices well. After your first successful adoption project, ramp up to more people and more projects. Take periodic readings of your business values —they will change with successful adoption. However, no matter what its size, there are always significant challenges associated with team development. Having more developers means more complex communication and hence more difficult coordination, particularly if the team is geographically dispersed, as is often the case. With a team of developers, the key management challenge is always to maintain a unity and integrity of design.

Longer product lifetime directly affects the product’s Return on Investment (ROI). Unfortunately, software tends to age poorly. Maintenance becomes more difficult and it acquires inertia. Companies that support multiple aging versions of a product spend a large amount of effort keeping those products alive and then finally have to discontinue support because of the cost.

For many product companies this is an important business value to address. Many of the Agile development practices will improve the maintainability and flexibility of the code base that, in turn, increases the ability of the development team to keep the product alive. These practices that directly and indirectly increase product lifetime have business value to the customer.

Remember that the goal of software development is to provide value to the customer. You should work with your customer to really understand his or her needs. Once you have a prioritized list of business values that are important to your customer, spread the knowledge. Make sure your team is aware of that information.

You should work hard to determine and prioritize your organization's bussines values. Do not make these decisions alone. Involve your customer so that he understand why you are making these changes. Make these decisions with business value in mind. If your adoption is driven by customer needs, and you track your progress in that area (even if subjectively), you may get their support. If you include them in your reviews instead of hiding your faults they may start to trust you and work with you. If you deliver improved results then you’ll have raging fans.

Use the feedback from those retrospectives to modify your development process. Tweak the practices. Drop the ones that don’t work. Adopt new ones to complete a cluster of practices.

Roles in team

A key success factor to move through the phases is defining who does what. Unless one person is doing everything, you have to divide up the work. With two guys in a garage, one person might define the problem, and the other might analyze, design and build. Maybe the first guy tests. But each knows which part to do because they talk it through. It might alternate from task to task, but whenever there is more than one person, role definition and communication are essential.

The team performs the actual work of problem solvers and designers. The team normally consists of 5-9 people – a group size that experience and research has shown to be best for this type of work.

The team members decide how the work is arranged and how assignments are distributed. There are no set project roles – everyone should be able to swap tasks with another member. Naturally, this does not prevent individual members from being experts in a field.

One of the team members is team leader. The team leader meets with the team every day in brief meetings. When someone outside the project has an important issue to discuss with the team, the team leader tries to ensure that the designers are disturbed as little as possible in their work. The team leader always adopts team to the work.

Engineering as a Science and an Art

The role of the engineer as artist is particularly challenging when the task is to design an entirely new system. Especially in the case of reactive systems and systems for command and control, we are frequently asked to write software for an entirely unique set of requirements, often to be executed on a configuration of target processors constructed specifically for this system. In other cases, such as the creation of frameworks, tools for research in artificial intelligence, or information management systems, we may have a well-defined, stable target environment, but our requirements may stress the software technology in one or more dimensions. For example, we may be asked to craft systems that are faster, have greater capacity, or have radically improved functionality. In all these situations, we try to use proven abstractions and mechanisms as a foundation on which to build new complex systems. In the presence of a large library of reusable software components, the software engineer must assemble these parts in innovative ways to satisfy the stated and implicit requirements, just as the painter or the musician must push the limits of his or her medium.

The Meaning of Design

In every engineering discipline, design encompasses the disciplined approach used to invent a solution for some problem, thus providing a path from requirements to implementation. In the context of software engineering, the purpose of design is to construct a system that:
  • Satisfies a given (perhaps informal) functional specification
  • Conforms to limitations of the target medium
  • Meets implicit or explicit requirements on performance and resource usage
  • Satisfies implicit or explicit design criteria on the form of the artifact
  • Satisfies restrictions on the design process itself, such as its length or cost, or the tools available for doing the design

Design involves balancing a set of competing requirements. The products of design are models that enable us to reason about our structures, make trade-offs when requirements conflict, and in general, provide a blueprint for implementation.

The Importance of Model Building

The building of models has a broad acceptance among all engineering disciplines, largely because model building appeals to the principles of decomposition, abstraction, and hierarchy. Each model within a design describes a specific aspect of the system under consideration. As much as possible, project manager seek to build new models upon old models in which we already have confidence. Models give us the opportunity to fail under controlled conditions. Evaluate each model in both expected and unusual situations, and then alter them when they fail to behave as we expect or desire. You have found that in order to express all the subtleties of a complex system, must use more than one kind of model. For example, when designing a personal computer, an electrical engineer must take into consideration the component-level view of the system as well as the physical layout of the circuit boards. This component view forms a logical picture of the design of the system, which helps the engineer to reason about the cooperative behavior of the components. The board layout represents the physical packaging of these components, constrained by the board size, available power, and the kinds of components that exist. From this view, the engineer can independently reason about factors such as heat dissipation and manufacturability. The board designer must also consider dynamic as well as static aspects of the system under construction. Thus, the electrical engineer uses diagrams showing the static connections among individual components, as well as timing diagrams that show the behavior of these components over time. The engineer can then employ tools such as oscilloscopes and digital analyzers to validate the correctness of both the static and dynamic models.

Summary

Here are the minimum activities to have a viable approach:

  • Build trust. Share methodologies. Learn each other’s terminology, processes, artifacts, principles, and goals.
  • Communicate the vision. Who are you building the product for, why would they want to buy it, and what key problems will you solve? Speak in market facts, not opinions.
  • Clarify roles. Do a gap analysis of the Product Management role. Identify roles and responsibilities on the Development project.
  • Prioritize the work. To participate in what is being built, you need to have a quantified, prioritized list of problems to solve. Feed this to Development in the form of personas, goals, problems, use scenarios, and requirements. Work together on the scheduling challenges to fit the work into agile, iterative pieces, but don’t get dragged into the subatomic level.
  • Software is inherently complex; the complexity of software systems often exceeds the human intellectual capacity.
  • The task of the software development team is to engineer the illusion of simplicity.
  • Complexity often takes the form of a hierarchy; it is useful to model both the “is a” and the “part of” hierarchies of a complex system.
  • Complex systems generally evolve from stable intermediate forms.
  • There are fundamental limiting factors of human cognition; you can address these constraints through the use of decomposition, abstraction, and hierarchy.
  • Complex systems can be viewed by focusing on either things or processes; there are compelling reasons for applying object-oriented decomposition, in which we view the world as a meaningful collection of objects that collaborate to achieve some higher-level behavior.
  • Object-oriented analysis and design is the method that leads us to an objectoriented decomposition; object-oriented design uses a notation and process for constructing complex software systems and offers a rich set of models with which we may reason about different aspects of the system under consideration.

Wednesday, December 5, 2007

OSGi based Web application

I decide to develop OSGi based web application. Quick investigation bring me to Equinox Servlet Container. Because I didn't decide jet which OSGi implementation I'd like to use, I created my own servlet container. My servlet container is identical to Equinix implementation, but there are some changes and I think in the future there could be more.

Peter Kriens
Well, just as obviously, people's requirements differ. Equinox is optimized for very large systems, uses lots of caching, but is therefore much bigger than Knopflerfish or Apache Felix. The best part about OSGi technology is that it still strives to write applications once and run anywhere, and it is really getting closer. An independent specification allows different implementations that optimize for a certain area.

The next decision was using Maven to setup the servlet bridge for Tomcat. It has borrowed much more time, than I thought. But hear is the results of my pom.xml. So the maven could deploy my application at a servlet container (e. g. Tomcat, Jetty, ...)

The advantage of using OSGi is a strong modularity and your can use it at different solutions. First of all I would like to create Google Web Toolkit (GWT) bundle and use it with such JE frameworks as Spring and Hibernate. The last two already have the bridge on OSGi. Actually I don't know does the GWT have some bridge or can I just wrap it.