Recent reading

Continuous Integration

2006-05-19T14:15:00.000+07:00

Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. This article is a quick overview of Continuous Integration summarizing the technique and its current usage.

I vividly remember one of my first sightings of a large software project. I was taking a summer internship at a large English electronics company. My manager, part of the QA group, gave me a tour of a site and we entered a huge depressing warehouse stacked full with cubes. I was told that this project had been in development for a couple of years and was currently integrating, and had been integrating for several months. My guide told me that nobody really knew how long it would take to finish integrating. From this I learned a common story of software projects: integration is a long and unpredictable process.

But this needn't be the way. Most projects done by my colleagues at ThoughtWorks, and by many others around the world, treat integration as a non-event. Any individual developer's work is only a few hours away from a shared project state and can integrated back into that state in minutes. Any integration errors are found rapidly and can be fixed rapidly.

This contrast isn't the result of an expensive and complex tool. The essence of it lies in the simple practice of everyone on the team integrating frequently, usually daily, against a controlled source code repository.

When I've described this practice to people, I commonly find two reactions: "it can't work (here)" and "doing it won't make much difference". When people find that it's much easier than it sounds, and see that it makes a huge difference to development. Thus the third common reaction is "yes we do that - how could you live without it?"

The term 'Continuous Integration' originated with the Extreme Programming development process, as one of its original twelve practices. When I started at ThoughtWorks, as a consultant, I encouraged the project I was working with to use the technique. Matthew Foemmel turned my vague exhortations into solid action and we saw the project go from rare and complex integrations to the non-event I described. Matthew and I wrote up our experience in the original version of this paper, which has been one of the most popular papers on my site.

See Related Article: Continuous Integration (original version)

The original article on Continuous Integration from this website. If you are following most links, this is the article they intended you to read (although I think the new one is better). It describes the experiences that we went through as Matt helped put together continuous integration through a project at ThoughtWorks in the early 00's.

Although Continuous Integration is a practice that requires no particular tooling to deploy, we've found that it is useful to use a Continuous Integration server. The best known such server is CruiseControl, an open source tool originally built by several people at ThoughtWorks and now maintained by a wide community. The original CruiseControl is written in Java but is also available for the Microsoft platform as CruiseControl.net.

Building a Feature with Continuous Integration

The easiest way for me to explain what CI is and how it works is to show a quick example of how it works with the development of a small feature. Let's assume I have to do something to a piece of software, it doesn't really matter what the task is, for the moment I'll assume it's small and can be done in a few hours. (We'll explore longer tasks, and other issues later on.)

I begin by taking a copy of the current integrated source onto my local development machine. I do this by using a source code management system by checking out a working copy from the mainline.

The above paragraph will make sense to people who use source code control systems, but be gibberish to those who don't. So let me quickly explain that for the latter. A source code control system keeps all of a project's source coded in a repository. The current state of the system is usually referred to as the mainline. At any time a developer can do a controlled copy of the mainline onto their own machine, this is called checking out. The copy on the developer's machine is called a working copy. (Most of the time you actually update your working copy to the mainline - in practice it's the same thing.)

Now I take my working copy and do whatever I need to do to complete my task. This will consist of both altering the production code, and also adding or changing automated tests. Continuous Integration assumes a high degree of tests which are automated into the software: a facility I call self-testing code. Often these use a version of the popular XUnit testing frameworks.

Once I'm done (and usually at various points when I'm working) I carry out an automated build on my development machine. This takes the source code in my working copy, compiles and links it into an executable, and runs the automated tests. Only if it all builds and tests without errors is the overall build considered to be good.

With a good build, I can then think about committing my changes into the repository. The twist, of course, is that other people may, and usually have, made changes to the mainline before I get chance to commit. So first I update my working copy with their changes and rebuild. If their changes clash with my changes, it will manifest as a failure either in the compilation or in the tests. In this case it's my responsibility to fix this and repeat until I can build a working copy that is properly synchronized with the mainline.

Once I have made my own build of a properly synchronized working copy I can then finally commit my changes into the mainline, which then updates the repository.

However my commit doesn't finish my work. At this point we build again, but this time on an integration machine based on the mainline code. Only when this build succeeds can we say that my changes are done. There is always a chance that I missed something on my machine and the repository wasn't properly updated. Only when my committed changes build successfully on the integration is my job done. This integration build can be executed manually by me, or done automatically by CruiseControl.

If a clash occurs between two developers, it is usually caught when the second developer to commit builds their updated working copy. If not the integration build should fail. Either way the error is detected rapidly. At this point the most important task is to fix it, and get the build working properly again. In a Continuous Integration environment you should never have a failed integration build stay failed for long. A good team should have many correct builds a day. Bad builds do occur from time to time, but should be quickly fixed.

The result of doing this is that there is a stable piece of software that works properly and contains few bugs. Everybody develops off that shared stable base and never gets so far away from that base that it takes very long to integrate back with it. Less time is spent trying to find bugs because they show up quickly.

Practices of Continuous Integration

The story above is the overview of CI and how it works in daily life. Getting all this to work smoothly is obviously rather more than that. I'll focus now on the key practices that make up effective CI.

Maintain a Single Source Repository.

Software projects involve lots of files that need to be orchestrated together to build a product. Keeping track of all of these is a major effort, particularly when there's multiple people involved. So it's not surprising that over the years software development teams have built tools to manage all this. These tools - called Source Code Management tools, configuration management, version control systems, repositories, or various other names - are an integral part of most development projects. The sad and surprising thing is that they aren't part of all projects. It is rare, but I do run into projects that don't use such a system and use some messy combination of local and shared drives.

So as a simple basis make sure you get a decent source code management system. Cost isn't an issue as good quality open-source tools are available. The current open source repository of choice is Subversion. (The older open-source tool CVS is still widely used, and is much better than nothing, but Subversion is the modern choice.) Interestingly as I talk to developers I know most commercial source code management tools are liked less than Subversion. The only tool I've consistently heard people say is worth paying for is Perforce.

Once you get a source code management system, make sure it is the well known place for everyone to go get source code. Nobody should ever ask "where is the foo-whiffle file?" Everything should be in the repository.

Although many teams use repositories a common mistake I see is that they don't put everything in the repository. If people use one they'll put code in there, but everything you need to do a build should in there including: test scripts, properties files, database schema, install scripts, and third party libraries. I've known projects that check their compilers into the repository (important in the early days of flaky C++ compilers). The basic rule of thumb is that you should be able to walk up to the project with a virgin machine, do a checkout, and be able to fully build the system. Only a minimal amount of things should be on the virgin machine - usually things that are large, complicated to install, and stable. An operating system, Java development environment, or base database system are typical examples.

You must put everything required for a build in the source control system, however you may also put other stuff that people generally work with in there too. IDE configurations are good to put in there because that way it's easy for people to share the same IDE setups.

One of the features of version control systems is that they allow you to create multiple branches, to handle different streams of development. This is a useful, nay essential, feature - but it's frequently overused and gets people into trouble. Keep your use of branches to a minimum. In particular have a mainline: a single branch of the project currently under development. Pretty much everyone should work off this mainline most of the time. (Reasonable branches are bug fixes of prior production releases and temporary experiments.)

In general you should store in source control everything you need to build anything, but nothing that you actually build. Some people do keep the build products in source control, but I consider that to be a smell - an indication of a deeper problem, usually an inability to reliably recreate builds.

Automate the Build

Getting the sources turned into a running system can often be a complicated process involving compilation, moving files around, loading schemas into the databases, and so on. However like most tasks in this part of software development it can be automated - and as a result should be automated. Asking people to type in strange commands or clicking through dialog boxes is a waste of time and a breeding ground for mistakes.

Automated environments for builds are a common feature of systems. The Unix world has had make for decades, the Java community developed Ant, the .NET community has had Nant and now has MSBuild. Make sure you can build and launch your system using these scripts using a single command.

A common mistake is not to include everything in the automated build. The build should include getting the database schema out of the repository and firing it up in the execution environment. I'll elaborate my earlier rule of thumb: anyone should be able to bring in a virgin machine, check the sources out of the repository, issue a single command, and have a running system on their machine.

Build scripts come in various flavors and are often particular to a platform or community, but they don't have to be. Although most of our Java projects use Ant, some have used Ruby (the Ruby Rake system is a very nice build script tool). We got a lot of value from automating an early Microsoft COM project with Ant.

A big build often takes time, you don't want to do all of these steps if you've only made a small change. So a good build tool analyzes what needs to be changed as part of the process. The common way to do this is to check the dates of the source and object files and only compile if the source date is later. Dependencies then get tricky: if one object file changes those that depend on it may also need to be rebuilt. Compilers may handle this kind of thing, or they may not.

Depending on what you need, you may need different kinds of things to be built. You can build a system with or without test code, or with different sets of tests. Some components can be built stand-alone. A build script should allow you to build alternative targets for different cases.

Many of us use IDEs, and most IDEs have some kind of build management process within them. However these files are always proprietary to the IDE and often fragile. Furthermore they need the IDE to work. It's okay for IDE users set up their own project files and use them for individual development. However it's essential to have a master build that is usable on a server and runnable from other scripts. So on a Java project we're okay with having developers build in their IDE, but the master build uses Ant to ensure it can be run on the development server.

Make Your Build Self-Testing

Traditionally a build means compiling, linking, and all the additional stuff required to get a program to execute. A program may run, but that doesn't mean it does the right thing. Modern statically typed languages can catch many bugs, but far more slip through that net.

A good way to catch bugs more quickly and efficiently is to include automated tests in the build process. Testing isn't perfect, of course, but it can catch a lot of bugs - enough to be useful. In particular the rise of Extreme Programming (XP) and Test Driven Development (TDD) have done a great deal to popularize self-testing code and as a result many people have seen the value of the technique.

Regular readers of my work will know that I'm a big fan both both TDD and XP, however I want to stress that neither of these approaches are necessary to gain the benefits of self-testing code. Both of these approaches make a point of writing tests before you write the code that makes them pass - in this mode the tests are as much about exploring the design of the system as they are about bug catching. This is a Good Thing, but it's not necessary for the purposes of Continuous Integration, where we have the weaker requirement of self-testing code. (Although TDD is my preferred way of producing self-testing code.)

For self-testing code you need a suite of automated tests that can check a large part of the code base for bugs. The tests need to be able to be kicked off from a simple command and to be self-checking. The result of running the test suite should indicate if any tests failed. For a build to be self-testing the failure of a test should cause the build to fail.

Over the last few years the rise of TDD has popularized the XUnit family of open-source tools which are ideal for this kind of testing. XUnit tools have proved very valuable to us at ThoughtWorks and I always suggest to people that they use them. These tools, pioneered by Kent Beck, make it very easy for you to set up a fully self-testing environment.

XUnit tools are certainly the starting point for making your code self-testing. You should also look out for other tools that focus on more end-to-end testing, there's quite a range of these out there at the moment including FIT, Selenium, Sahi, Watir, FITnesse, and plenty of others that I'm not trying to comprehensively list here.

Of course you can't count on tests to find everything. As it's often been said: tests don't prove the absence of bugs. However perfection isn't the only point at which you get payback for a self-testing build. Imperfect tests, run frequently, are much better than perfect tests that are never written at all.

Everyone Commits Every Day

Integration is primarily about communication. Integration allows developers to tell other developers about the changes they have made. Frequent communication allows people to know quickly as changes develop.

The one prerequisite for a developer committing to the mainline is that they can correctly build their code. This, of course, includes passing the build tests. As with any commit cycle the developer first updates their working copy to match the mainline, resolves any conflicts with the mainline, then builds on their local machine. If the build passes, then they are free to commit to the mainline.

By doing this frequently, developers quickly find out if there's a conflict between two developers. The key to fixing problems quickly is finding them quickly. With developers committing every few hours a conflict can be detected within a few hours of it occurring, at that point not much has happened and it's easy to resolve. Conflicts that stay undetected for weeks can be very hard to resolve.

The fact that you build when you update your working copy means that you detect compilation conflicts as well as textual conflicts. Since the build is self-testing, you also detect conflicts in the running of the code. The latter conflicts are particularly awkward bugs to find if they sit for a long time undetected in the code. Since there's only a few hours of changes between commits, there's only so many places where the problem could be hiding. Furthermore since not much has changed you can use diff-debugging to help you find the bug.

My general rule of thumb is that every developer should commit to the repository every day. In practice it's often useful if developers commit more frequently than that. The more frequently you commit, the less places you have to look for conflict errors, and the more rapidly you fix conflicts.

Frequent commits encourage developers to break down their work into small chunks of a few hours each. This helps track progress and provides a sense of progress. Often people initially feel they can't do something meaningful in just a few hours, but we've found that mentoring and practice helps them learn.

Every Commit Should Build the Mainline on an Integration Machine

Using daily commits, a team gets frequent tested builds. This ought to mean that the mainline stays in a healthy state. In practice, however, things still do go wrong. One reason is discipline, people not doing an update and build before they commit. Another is environmental differences between developers' machines.

As a result you should ensure that regular builds happen on an integration machine and only if this integration build succeeds should the commit be considered to be done. Since the developer who commits is responsible for this, that developer needs to monitor the mainline build so they can fix it if it breaks. A corollary of this is that you shouldn't go home until the mainline build has passed with any commits you've added late in the day.

There are two main ways I've seen to ensure this: using a manual build or a continuous integration server.

The manual build approach is the simplest one to describe. Essentially it's a similar thing to the local build that a developer does before the commit into the repository. The developer goes to the integration machine, checks out the head of the mainline (which now houses his last commit) and kicks off the integration build. He keeps an eye on its progress, and if the build succeeds he's done with his commit. (Also see Jim Shore's description.)

A continuous integration server acts as a monitor to the repository. Every time a commit against the repository finishes the server automatically checks out the sources onto the integration machine, initiates a build, and notifies the committer of the result of the build. The committer isn't done until she gets the notification - usually an email.

At ThoughtWorks, we're big fans of continuous integration servers - indeed we led the development of CruiseControl and CruiseControl.net, the widely used open-source CI servers. ThoughtWorkers like Paul Julius, Jason Yip, and Owen Rodgers are still active committers to these open source projects. We use CruiseControl on nearly every project we do and have been very happy with the results.

Not everyone prefers to use a CI server. Jim Shore gave a well argued description of why he prefers the manual approach. I agree with him that CI is much more than just installing Cruise Control. All the practices here need to be in play to do Continuous Integration effectively. But equally many teams who do CI well find Cruise Control a helpful tool.

Many organizations do regular builds on a timed schedule, such as every night. This is not the same thing as a continuous build and isn't enough for continuous integration. The whole point of continuous integration is to find problems as soon as you can. Nightly builds mean that bugs lie undetected for a whole day before anyone discovers them. Once they are in the system that long, it takes a long time to find and remove them.

A key part of doing a continuous build is that if the mainline build fails, it needs to be fixed right away. The whole point of working with CI is that you're always developing on a known stable base. It's not a bad thing for the mainline build to break, although if it's happening all the time it suggests people aren't being careful enough about updating and building locally before a commit. When the mainline build does break, however, it's important that it gets fixed fast.

When teams are introducing CI, often this is one of the hardest things to sort out. Early on a team can struggle to get into the regular habit of working mainline builds, particularly if they are working on an existing code base. Patience and steady application does seem to regularly do the trick, so don't get discouraged.

Keep the Build Fast

The whole point of Continuous Integration is to provide rapid feedback. Nothing sucks the blood of a CI activity more than a build that takes a long time. Here I must admit a certain crotchety old guy amusement at what's considered to be a long build. Most of my colleagues consider a build that takes an hour to be totally unreasonable. I remember teams dreaming that they could get it so fast - and occasionally we still run into cases where it's very hard to get builds to that speed.

For most projects, however, the XP guideline of a ten minute build is perfectly within reason. Most of our modern projects achieve this. It's worth putting in concentrated effort to make it happen, because every minute you reduce off the build time is a minute saved for each developer every time they commit. Since CI demands frequent commits, this adds up to a lot of time.

If you're staring at a one hour build time, then getting to a faster build may seem like a daunting prospect. It can even be daunting to work on a new project and think about how to keep things fast. For enterprise applications, at least, we've found the usual bottleneck is testing - particularly tests that involve external services such as a database.

Probably the most crucial step is to start working on setting up a staged build. The idea behind a staged build (also known as build pipeline) is that there are in fact multiple builds done in sequence. The commit to the mainline triggers the first build - what I call the commit build. The commit build is the build that's needed when someone commits to the mainline. The commit build is the one that has to be done quickly, as a result it will take a number of shortcuts that will reduce the ability to detect bugs. The trick is to balance the needs of bug finding and speed so that a good commit build is stable enough for other people to work on.

Once the commit build is good then other people can work on the code with confidence. However there are further, slower, tests that you can start to do. Additional machines can run further testing routines on the build that take longer to do.

A simple example of this is a two stage build. The first stage would do the compilation and run tests that are more localized unit tests with the database completely stubbed out. Such tests can run very fast, keeping within the ten minute guideline. However any bugs that involve larger scale interactions, particularly those involving the real database, won't be found. The second stage build runs a different suite of tests that do hit the real database and involve more end-to-end behavior. This suite might take a couple of hours to run.

In this scenario people use the first stage as the commit build and use this as their main CI cycle. The second-stage build is a secondary build which runs when it can, picking up the latest good commit build for further testing. If the secondary build fails, then this doesn't have the same 'stop everything' quality, but the team does aim to fix such bugs as rapidly as possible, while keeping the commit build running. Indeed the secondary build doesn't have to stay good, as long as each known bug is identified and dealt with in a next few days.

If the secondary build detects a bug, that's a sign that the commit build could do with another test. As much as possible you want to ensure that any secondary build failure leads to new tests in the commit build that would have caught the bug, so the bug stays fixed in the commit build. This way the commit tests are strengthened whenever something gets past them. There are cases where there's no way to build a fast-running test that exposes the bug, so you may decide to only test for that condition in the secondary build. Most of time, fortunately, you can add suitable tests to the commit build.

This example is of a two-stage build, but the basic principle can be extended to any number of later builds. The builds after the commit build can also be done in parallel, so if you have two hours of secondary tests you can improve responsiveness by having two machines that run half the tests each. By using parallel secondary builds like this you can introduce all sorts of further automated testing, including performance testing, into the regular build process. (I've run into a lot of interesting techniques around this as I've visited various ThoughtWorks projects over the last couple of years - I'm hoping to persuade some of the developers to write these up.)

Test in a Clone of the Production Environment

The point of testing is to flush out, under controlled conditions, any problem that the system will have in production. A significant part of this is the environment within which the production system will run. If you test in a different environment, every difference results in a risk that what happens under test won't happen in production.

As a result you want to set up your test environment to be as exact a mimic of your production environment as possible. Use the same database software, with the same versions, use the same version of operating system. Put all the appropriate libraries that are in the production environment into the test environment, even if the system doesn't actually use them. Use the same IP addresses and ports, run it on the same hardware.

Well, in reality there are limits. If you're writing desktop software it's not practicable to test in a clone of every possible desktop with all the third party software that different people are running. Similarly some production environments may be prohibitively expensive to duplicate (although I've often come across false economies by not duplicating moderately expensive environments). Despite these limits your goal should still be to duplicate the production environment as much as you can, and to understand the risks you are accepting for every difference between test and production.

If you have a pretty simple setup without many awkward communications, you may be able to run your commit build in a mimicked environment. Often, however, you need to use test doubles because systems respond slowly or intermittently. As a result its common to have a very artificial environment for the commit tests for speed, and use a production clone for secondary testing.

I've noticed a growing interest in using virtualization to make it easy to put together test environments. Virtualized machines can be saved with all the necessary elements baked into the virtualization. It's then relatively straightforward to install the latest build and run tests. Furthermore this can allow you to run multiple tests on one machine, or simulate multiple machines in a network on a single machine. As the performance penalty of virtualization decreases, this option makes more and more sense.

Make it Easy for Anyone to Get the Latest Executable

One of the most difficult parts of software development is making sure that you build the right software. We've found that it's very hard to specify what you want in advance and be correct; people find it much easier to see something that's not quite right and say how it needs to be changed. Agile development processes explicitly expect and take advantage of this part of human behavior.

To help make this work, anyone involved with a software project should be able to get the latest executable and be able to run it: for demonstrations, exploratory testing, or just to see what changed this week.

Doing this is pretty straightforward: make sure there's a well known place where people can find the latest executable. It may be useful to put several executables in such a store. For the very latest you should put the latest executable to pass the commit tests - such an executable should be pretty stable providing the commit suite is reasonably strong.

If you are following a process with well defined iterations, it's usually wise to also put the end of iteration builds there too. Demonstrations, in particular, need software that whose features are familiar, so then it's usually worth sacrificing the very latest for something that the demonstrator knows how to operate.

Everyone can see what's happening

Continuous Integration is all about communication, so you want to ensure that everyone can easily see the state of the system and the changes that have been made to it.

One of the most important things to communicate is the state of the mainline build. If you're using CruiseControl there's a built in web site that will show you if there's a build in progress and what was the state of the last mainline build. Many teams like to make this even more apparent by hooking up a continuous display to the build system - lights that glow green when the build works, or red if it fails are popular. A particularly common touch is red and green lava lamps - not just do these indicate the state of the build, but also how long it's been in that state. Bubbles on a red lamp indicate the build's been broken for too long. Each team makes its own choices on these build sensors - it's good to be playful with your choice (recently I saw someone experimenting with a dancing rabbit.)

If you're using a manual CI process, this visibility is still essential. The monitor of the physical build machine can show the status of the mainline build. Often you have a build token to put on the desk of whoever's currently doing the build (again something silly like a rubber chicken is a good choice). Often people like to make a simple noise on good builds, like ringing a bell.

CI servers' web pages can carry more information than this, of course. CruiseControl provides an indication not just of who is building, but what changes they made. Cruise also provides a history of changes, allowing team members to get a good sense of recent activity on the project. I know team leads who like to use this to get a sense of what people have been doing and keep a sense of the changes to the system.

Another advantage of using a web site is that those that are not co-located can get a sense of the project's status. In general I prefer to have everyone actively working on a project sitting together, but often there are peripheral people who like to keep an eye on things. It's also useful for groups to aggregate together build information from multiple projects - providing a simple and automated status of different projects.

Good information displays are not only those on a computer screens. One of my favorite displays was for a project that was getting into CI. It had a long history of being unable to make stable builds. We put a calendar on the wall that showed a full year with a small square for each day. Every day the QA group would put a green sticker on the day if they had received one stable build that passed the commit tests, otherwise a red square. Over time the calendar revealed the state of the build process showing a steady improvement until green squares were so common that the calendar disappeared - its purpose fulfilled.

Automate Deployment

To do Continuous Integration you need multiple of environments, one to run commit tests, one or more to run secondary tests. Since you are moving executables between these environments multiple times a day, you'll want to do this automatically. So it's important to have scripts that will allow you to deploy the application into any environment easily.

A natural consequence of this is that you should also have scripts that allow you to deploy into production with similar ease. You may not be deploying into production every day (although I've run into projects that do), but automatic deployment helps both speed up the process and reduce errors. It's also a cheap option since it just uses the same capabilities that you use to deploy into test environments.

If you deploy into production one extra automated capability you should consider is automated rollback. Bad things do happen from time to time, and if smelly brown substances hit rotating metal, it's good to be able to quickly go back to the last known good state. Being able to automatically revert also reduces a lot of the tension of deployment, encouraging people to deploy more frequently and thus get new features out to users quickly. (The ruby on rails community developed a tool called Capistrano that is a good example of a tool that does this sort of thing.)

In clustered environments I've seen rolling deployments where the new software is deployed to one node at a time, gradually replacing the application over the course of a few hours.

See Related Article: Evolutionary Database Design

A common roadblock for many people doing frequent releases is database migration. Database changes are awkward because you can't just change database schemas, you also have to ensure data is correctly migrated. This article describes techniques used by my colleague Pramod Sadalage to do automated refactoring and migration of databases. The article is an early attempt the capture the information that's described in more detail by Pramod and Scott Amblers book on refactoring databases[ambler-sadalage].

A particularly interesting variation of this that I've come across with public web application is the idea of deploying a trial build to a subset of users. The team then sees how the trial build is used before deciding whether to deploy it to the full user population. This allows you to test out new features and user-interfaces before committing to a final choice. Automated deployment, tied into good CI discipline, is essential to making this work.

Benefits of Continuous Integration

On the whole I think the greatest and most wide ranging benefit of Continuous Integration is reduced risk. My mind still floats back to that early software project I mentioned in my first paragraph. There they were at the end (they hoped) of a long project, yet with no real idea of how long it would be before they were done.

The trouble with deferred integration is that it's very hard to predict how long it will take to do, and worse it's very hard to see how far you are through the process. The result is that you are putting yourself into a complete blind spot right at one of tensest parts of a project - even if you're one the rare cases where you aren't already late.

Continuous Integration completely finesses this problem. There's no long integration, you completely eliminate the blind spot. At all times you know where you are, what works, what doesn't, the outstanding bugs you have in your system.

Bugs - these are the nasty things that destroy confidence and mess up schedules and reputations. Bugs in deployed software make users angry with you. Bugs in work in progress get in your way, making it harder to get the rest of the software working correctly.

Continuous Integrations doesn't get rid of bugs, but it does make them dramatically easier to find and remove. In this respect it's rather like self-testing code. If you introduce a bug and detect it quickly it's far easier to get rid of. Since you've only changed a small bit of the system, you don't have far to look. Since that bit of the system is the bit you just worked, it's fresh in your memory - again making it easier to find the bug. You can also use diff debugging - comparing the current version of the system to an earlier one that didn't have the bug.

Bugs are also cumulative. The more bugs you have, the harder it is to remove each one. This is partly because you get bug interactions, where failures show as the result of multiple faults - making each fault harder to find. It's also psychological - people have less energy to find and get rid of bugs when there are many of them - a phenomenon that the Pragmatic Programmers call the Broken Windows syndrome.

As a result projects with Continuous Integration tend to have dramatically less bugs, both in production and in process. However I should stress that the degree of this benefit is directly tied to how good your test suite is. You should find that it's not too difficult to build a test suite that makes a noticeable difference. Usually, however, it takes a while before a team really gets to the low level of bugs that they have the potential to reach. Getting there means constantly working on and improving your tests.

If you have continuous integration, it removes one of the biggest barriers to frequent deployment. Frequent deployment is valuable because is allows your users to get new features more rapidly, to give more rapid feedback on those features, and generally become more collaborative in the development cycle. This helps break down the barriers between customers and development - barriers which I believe are the biggest barriers to successful software development.

Introducing Continuous Integration

So you fancy trying out Continuous Integration - where do you start? The full set of practices I outlined above give you the full benefits - but you don't need to start with all of them.

There's no fixed recipe here - much depends on the nature of your setup and team. But here are a few things that we've learned to get things going.

One of the first steps is to get the build automated. Get everything you need into source control get it so that you can build the whole system with a single command. For many projects this is not a minor undertaking - yet it's essential for any of the other things to work. Initially you may only do build occasionally on demand, or just do an automated nightly build. While these aren't continuous integration an automated nightly build is a fine step on the way.

Introduce some automated testing into you build. Try to identify the major areas where things go wrong and get automated tests to expose those failures. Particularly on an existing project it's hard to get a really good suite of tests going rapidly - it takes time to build tests up. You have to start somewhere though - all those cliches about Rome's build schedule apply.

Try to speed up the commit build. Continuous Integration on a build of a few hours is better than nothing, but getting down to that magic ten minute number is much better. This usually requires some pretty serious surgery on your code base to do as you break dependencies on slow parts of the system.

If you are starting a new project, begin with Continuous Integration from the beginning. Keep an eye on build times and take action as soon as you start going slower than the ten minute rule. By acting quickly you'll make the necessary restructurings before the code base gets so big that it becomes a major pain.

Above all get some help. Find someone who has done Continuous Integration before to help you. Like any new technique it's hard to introduce it when you don't know what the final result looks like. It may cost money to get a mentor, but you'll also pay in lost time and productivity if you don't do it. (Disclaimer / Advert - yes we at ThoughtWorks do do some consultancy in this area. After all we've made most of the mistakes that there are to make.)

Final Thoughts

In the years since Matt and I wrote the original paper on this site, Continuous Integration has become a mainstream technique for software development. Hardly any ThoughtWorks projects goes without it - and we see others using CI all over the world. I've hardly ever heard negative things about the approach - unlike some of the more controversial Extreme Programming practices.

If you're not using Continuous Integration I strongly urge you give it a try. If you are, maybe there are some ideas in this article that can help you do it more effectively. We've learned a lot about Continuous Integration in the last few years, I hope there's still more to learn and improve.

Acknowledgments

First and foremost to Kent Beck and my many colleagues on the Chrysler Comprehensive Compensation (C3) project. This was my first chance to see Continuous Integration in action with a meaningful amount of unit tests. It showed me what was possible and gave me an inspiration that led me for many years.

Thanks to Matt Foemmel, Dave Rice, and everyone else who built and maintained Continuous Integration on Atlas. That project was a sign of CI on a larger scale and showed the benefits it made to an existing project.

Paul Julius, Jason Yip, Owen Rodgers, Mike Roberts and many other open source contributors have participated in building some variant of Cruise Control. Although the tool isn't essential, many teams find it helpful. Cruise has played a big part in popularizing and enabling software developers to use Continuous Integration.

One of the reasons I work at ThoughtWorks is to get good access to practical projects done by talented people. Nearly every project I've visited has given tasty morsels of continuous integration information.

Significant Revisions

01 May 06: Complete rewrite of article to bring it up to date and to clarify the description of the approach.

10 Sep 00: Original version published.

URL Rewriting in ASP.NET

2006-02-26T18:30:00.000+07:00

Scott Mitchell

4GuysFromRolla.com

March 2004

Applies to:

Microsoft® ASP.NET

Summary: Examines how to perform dynamic URL rewriting with Microsoft ASP.NET. URL rewriting is the process of intercepting an incoming Web request and automatically redirecting it to a different URL. Discusses the various techniques for implementing URL rewriting, and examines real-world scenarios of URL rewriting. (31 printed pages)

Download the source code for this article.

Introduction

Common Uses of URL Rewriting

What Happens When a Request Reaches IIS

Implementing URL Rewriting

Building a URL Rewriting Engine

Performing Simple URL Rewriting with the URL Rewriting Engine

Creating Truly "Hackable" URLs

Conclusion

Related Books

Introduction

Take a moment to look at some of the URLs on your website. Do you find URLs like http://yoursite.com/info/dispEmployeeInfo.aspx?EmpID=459-099&type=summary? Or maybe you have a bunch of Web pages that were moved from one directory or website to another, resulting in broken links for visitors who have bookmarked the old URLs. In this article we'll look at using URL rewriting to shorten those ugly URLs to meaningful, memorable ones, by replacing http://yoursite.com/info/dispEmployeeInfo.aspx?EmpID=459-099&type=summary with something like http://yoursite.com/people/sales/chuck.smith. We'll also see how URL rewriting can be used to create an intelligent 404 error.

URL rewriting is the process of intercepting an incoming Web request and redirecting the request to a different resource. When performing URL rewriting, typically the URL being requested is checked and, based on its value, the request is redirected to a different URL. For example, in the case where a website restructuring caused all of the Web pages in the /people/ directory to be moved to a /info/employees/ directory, you would want to use URL rewriting to check if a Web request was intended for a file in the /people/ directory. If the request was for a file in the /people/ directory, you'd want to automatically redirect the request to the same file, but in the /info/employees/ directory instead.

With classic ASP, the only way to utilize URL rewriting was to write an ISAPI filter or to buy a third-party product that offered URL rewriting capabilities. With Microsoft® ASP.NET, however, you can easily create your own URL rewriting software in a number of ways. In this article we'll examine the techniques available to ASP.NET developers for implementing URL rewriting, and then turn to some real-world uses of URL rewriting. Before we delve into the technological specifics of URL rewriting, let's first take a look at some everyday scenarios where URL rewriting can be employed.

Common Uses of URL Rewriting

Creating data-driven ASP.NET websites often results in a single Web page that displays a subset of the database's data based on querystring parameters. For example, in designing an e-commerce site, one of your tasks would be to allow users to browse through the products for sale. To facilitate this, you might create a page called displayCategory.aspx that would display the products for a given category. The category's products to view would be specified by a querystring parameter. That is, if the user wanted to browse the Widgets for sale, and all Widgets had a had a CategoryID of 5, the user would visit: http://yousite.com/displayCategory.aspx?CategoryID=5.

There are two downsides to creating a website with such URLs. First, from the end user's perspective, the URL http://yousite.com/displayCategory.aspx?CategoryID=5 is a mess. Usability expert Jakob Neilsen recommends that URLs be chosen so that they:

Are short.

Are easy to type.

Visualize the site structure.

"Hackable," allowing the user to navigate through the site by hacking off parts of the URL.

I would add to that list that URLs should also be easy to remember. The URL http://yousite.com/displayCategory.aspx?CategoryID=5 meets none of Neilsen's criteria, nor is it easy to remember. Asking users to type in querystring values makes a URL hard to type and makes the URL "hackable" only by experienced Web developers who have an understanding of the purpose of querystring parameters and their name/value pair structure.

A better approach is to allow for a sensible, memorable URL, such as http://yoursite.com/products/Widgets. By just looking at the URL you can infer what will be displayed—information about Widgets. The URL is easy to remember and share, too. I can tell my colleague, "Check out yoursite.com/products/Widgets," and she'll likely be able to bring up the page without needing to ask me again what the URL was. (Try doing that with, say, an Amazon.com page!) The URL also appears, and should behave, "hackable." That is, if the user hacks of the end of the URL, and types in http://yoursite.com/products, they should see a listing of all products, or at least a listing of all categories of products they can view.

Note For a prime example of a "hackable" URL, consider the URLs generated by many blog engines. To view the posts for January 28, 2004, one visits a URL like http://someblog.com/2004/01/28. If the URL is hacked down to http://someblog.com/2004/01, the user will see all posts for January 2004. Cutting it down further to http://someblog.com/2004 will display all posts for the year 2004.

In addition to simplifying URLs, URL rewriting is also often used to handle website restructuring that would otherwise result in numerous broken links and outdated bookmarks.

What Happens When a Request Reaches IIS

Before we examine exactly how to implement URL rewriting, it's important that we have an understanding of how incoming requests are handled by Microsoft® Internet Information Services (IIS). When a request arrives at an IIS Web server, IIS examines the requested file's extension to determine how handle the request. Requests can be handled natively by IIS—as are HTML pages, images, and other static content—or IIS can route the request to an ISAPI extension. (An ISAPI extension is an unmanaged, compiled class that handles an incoming Web request. Its task is to generate the content for the requested resource.)

For example, if a request comes in for a Web page named Info.asp, IIS will route the message to the asp.dll ISAPI extension. This ISAPI extension will then load the requested ASP page, execute it, and return its rendered HTML to IIS, which will then send it back to the requesting client. For ASP.NET pages, IIS routes the message to the aspnet_isapi.dll ISAPI extension. The aspnet_isapi.dll ISAPI extension then hands off processing to the managed ASP.NET worker process, which processes the request, returning the ASP.NET Web page's rendered HTML.

You can customize IIS to specify what extensions are mapped to what ISAPI extensions. Figure 1 shows the Application Configuration dialog box from the Internet Information Services Administrative Tool. Note that the ASP.NET-related extensions—.aspx, .ascx, .config, .asmx, .rem, .cs, .vb, and others—are all mapped to the aspnet_isapi.dll ISAPI extension.

Figure 1. Configured mappings for file extensions

A thorough discussion of how IIS manages incoming requests is a bit beyond the scope of this article. A great, in-depth discussion, though, can be found in Michele Leroux Bustamante's article Inside IIS and ASP.NET. It's important to understand that the ASP.NET engine gets its hands only on incoming Web requests whose extensions are explicitly mapped to the aspnet_isapi.dll in IIS.

Examining Requests with ISAPI Filters

In addition to mapping the incoming Web request's file extension to the appropriate ISAPI extension, IIS also performs a number of other tasks. For example, IIS attempts to authenticate the user making the request and determine if the authenticated user has authorization to access the requested file. During the lifetime of handling a request, IIS passes through several states. At each state, IIS raises an event that can be programmatically handled using ISAPI filters.

Like ISAPI extensions, ISAPI filters are blocks of unmanaged code installed on the Web server. ISAPI extensions are designed to generate the response for a request to a particular file type. ISAPI filters, on the other hand, contain code to respond to events raised by IIS. ISAPI filters can intercept and even modify the incoming and outgoing data. ISAPI filters have numerous applications, including:

Authentication and authorization.

Logging and monitoring.

HTTP compression.

URL rewriting.

While ISAPI filters can be used to perform URL rewriting, this article examines implementing URL rewriting using ASP.NET. However, we will discuss the tradeoffs between implementing URL rewriting as an ISAPI filter versus using techniques available in ASP.NET.

What Happens When a Request Enters the ASP.NET Engine

Prior to ASP.NET, URL rewriting on IIS Web servers needed to be implemented using an ISAPI filter. URL rewriting is possible with ASP.NET because the ASP.NET engine is strikingly similar to IIS. The similarities arise because the ASP.NET engine:

Raises events as it processes a request.

Allows an arbitrary number of HTTP modules handle the events that are raised, akin to IIS's ISAPI filters.

Delegates rendering the requested resource to an HTTP handler, which is akin to IIS's ISAPI extensions.

Like IIS, during the lifetime of a request the ASP.NET engine fires events signaling its change from one state of processing to another. The BeginRequest event, for example, is fired when the ASP.NET engine first responds to a request. The AuthenticateRequest event fires next, which occurs when the identity of the user has been established. (There are numerous other events—AuthorizeRequest, ResolveRequestCache, and EndRequest, among others. These events are events of the System.Web.HttpApplication class; for more information consult the HttpApplication Class Overview technical documentation.)

As we discussed in the previous section, ISAPI filters can be created to respond to the events raised by IIS. In a similar vein, ASP.NET provides HTTP modules that can respond to the events raised by the ASP.NET engine. An ASP.NET Web application can be configured to have multiple HTTP modules. For each request processed by the ASP.NET engine, each configured HTTP module is initialized and allowed to wire up event handlers to the events raised during the processing of the request. Realize that there are a number of built-in HTTP modules utilized on each an every request. One of the built-in HTTP modules is the FormsAuthenticationModule, which first checks to see if forms authentication is being used and, if so, whether the user is authenticated or not. If not, the user is automatically redirected to the specified logon page.

Recall that with IIS, an incoming request is eventually directed to an ISAPI extension, whose job it is to return the data for the particular request. For example, when a request for a classic ASP Web page arrives, IIS hands off the request to the asp.dll ISAPI extension, whose task it is to return the HTML markup for the requested ASP page. The ASP.NET engine utilizes a similar approach. After initializing the HTTP modules, the ASP.NET engine's next task is to determine what HTTP handler should process the request.

All requests that pass through the ASP.NET engine eventually arrive at an HTTP handler or an HTTP handler factory (an HTTP handler factory simply returns an instance of an HTTP handler that is then used to process the request). The final HTTP handler renders the requested resource, returning the response. This response is sent back to IIS, which then returns it to the user that made the request.

ASP.NET includes a number of built-in HTTP handlers. The PageHandlerFactory, for example, is used to render ASP.NET Web pages. The WebServiceHandlerFactory is used to render the response SOAP envelopes for ASP.NET Web services. The TraceHandler renders the HTML markup for requests to trace.axd.

Figure 2 illustrates how a request for an ASP.NET resource is handled. First, IIS receives the request and dispatches it to aspnet_isapi.dll. Next, the ASP.NET engine initializes the configured HTTP modules. Finally, the proper HTTP handler is invoked and the requested resource is rendered, returning the generated markup back to IIS and back to the requesting client.

Figure 2. Request processing by IIS and ASP.NET

Creating and Registering Custom HTTP Modules and HTTP Handlers

Creating custom HTTP modules and HTTP handlers are relatively simple tasks, which involve created a managed class that implements the correct interface. HTTP modules must implement the System.Web.IHttpModule interface, while HTTP handlers and HTTP handler factories must implement the System.Web.IHttpHandler interface and System.Web.IHttpHandlerFactory interface, respectively. The specifics of creating HTTP handlers and HTTP modules is beyond the scope of this article. For a good background, read Mansoor Ahmed Siddiqui's article, HTTP Handlers and HTTP Modules in ASP.NET.

Once a custom HTTP module or HTTP handler has been created, it must be registered with the Web application. Registering HTTP modules and HTTP handlers for an entire Web server requires only a simple addition to the machine.config file; registering an HTTP module or HTTP handler for a specific Web application involves adding a few lines of XML to the application's Web.config file.

Specifically, to add an HTTP module to a Web application, add the following lines in the Web.config's configuration/system.web section:

<httpModules>     <add type="type" name="name" />  </httpModules>

The type value provides the assembly and class name of the HTTP module, whereas the name value provides a friendly name by which the HTTP module can be referred to in the Global.asax file.

HTTP handlers and HTTP handler factories are configured by the <httpHandlers> tag in the Web.config's configuration/system.web section, like so:

<httpHandlers>     <add verb="verb" path="path" type="type" />  </httpHandlers>

Recall that for each incoming request, the ASP.NET engine determines what HTTP handler should be used to render the request. This decision is made based on the incoming requests verb and path. The verb specifies what type of HTTP request was made—GET or POST—whereas the path specifies the location and filename of the file requested. So, if we wanted to have an HTTP handler handle all requests—either GET or POST—for files with the .scott extension, we'd add the following to the Web.config file:

<httpHandlers>     <add verb="*" path="*.scott" type="type" />  </httpHandlers>

where type was the type of our HTTP handler.

Note When registering HTTP handlers, it is important to ensure that the extensions used by the HTTP handler are mapped in IIS to the ASP.NET engine. That is, in our .scott example, if the .scott extension is not mapped in IIS to the aspnet_isapi.dll ISAPI extension, a request for the file foo.scott will result in IIS attempting to return the contents of the file foo.scott. In order for the HTTP handler to process this request, the .scott extension must be mapped to the ASP.NET engine. The ASP.NET engine, then, will route the request correctly to the appropriate HTTP handler.

For more information on registering HTTP modules and HTTP handlers, be sure to consult the <httpModules> element documentation along with the <httpHandlers> element documentation.

Implementing URL Rewriting

URL rewriting can be implemented either with ISAPI filters at the IIS Web server level, or with either HTTP modules or HTTP handlers at the ASP.NET level. This article focuses on implementing URL rewriting with ASP.NET, so we won't be delving into the specifics of implementing URL rewriting with ISAPI filters. There are, however, numerous third-party ISAPI filters available for URL rewriting, such as:

ISAPI Rewrite

IIS Rewrite

PageXChanger

And many others!

Implementing URL rewriting at the ASP.NET level is possible through the System.Web.HttpContext class's RewritePath() method. The HttpContext class contains HTTP-specific information about a specific HTTP request. With each request received by the ASP.NET engine, an HttpContext instance is created for that request. This class has properties like: Request and Response, which provide access to the incoming request and outgoing response; Application and Session, which provide access to application and session variables; User, which provides information about the authenticated user; and other related properties.

With the Microsoft® .NET Framework Version 1.0, the RewritePath() method accepts a single string, the new path to use. Internally, the HttpContext class's RewritePath(string) method updates the Request object's Path and QueryString properties. In addition to RewritePath(string), the .NET Framework Version 1.1 includes another form of the RewritePath() method, one that accepts three string input parameters. This alternate overloaded form not only sets the Request object's Path and QueryString properties, but also sets internal member variables that are used to compute the Request object's values for its PhysicalPath, PathInfo, and FilePath properties.

To implement URL rewriting in ASP.NET, then, we need to create an HTTP module or HTTP handler that:

Checks the requested path to determine if the URL needs to be rewritten.

Rewrites the path, if needed, by calling the RewritePath() method.

For example, imagine that our website had information each employee, accessible through /info/employee.aspx?empID=employeeID. To make the URLs more "hackable," we might decide to have employee pages accessible by: /people/EmployeeName.aspx. Here is a case where we'd want to use URL rewriting. That is, when the page /people/ScottMitchell.aspx was requested, we'd want to rewrite the URL so that the page /info/employee.aspx?empID=1001 was used instead.

URL Rewriting with HTTP Modules

When performing URL rewriting at the ASP.NET level you can use either an HTTP module or an HTTP handler to perform the rewriting. When using an HTTP module, you must decide at what point during the request's lifecycle to check to see if the URL needs to be rewritten. At first glance, this may seem to be an arbitrary choice, but the decision can impact your application in both significant and subtle ways. The choice of where to perform the rewrite matters because the built-in ASP.NET HTTP modules use the Request object's properties to perform their duties. (Recall that rewriting the path alters the Request object's property values.) These germane built-in HTTP modules and the events they tie into are listed below:

HTTP Module	Event	Description
FormsAuthenticationModule	AuthenticateRequest	Determines if the user is authenticated using forms authentication. If not, the user is automatically redirected to the specified logon page.
FileAuthorizationMoudle	AuthorizeRequest	When using Windows authentication, this HTTP module checks to ensure that the Microsoft® Windows® account has adequate rights for the resource requested.
UrlAuthorizationModule	AuthorizeRequest	Checks to make sure the requestor can access the specified URL. URL authorization is specified through the <authorization> and <location> elements in the Web.config file.

Recall that the BeginRequest event fires before AuthenticateRequest, which fires before AuthorizeRequest.

One safe place that URL rewriting can be performed is in the BeginRequest event. That means that if the URL needs to be rewritten, it will have done so by the time any of the built-in HTTP modules run. The downside to this approach arises when using forms authentication. If you've used forms authentication before, you know that when the user visits a restricted resource, they are automatically redirected to a specified login page. After successfully logging in, the user is sent back to the page they attempted to access in the first place.

If URL rewriting is performed in the BeginRequest or AuthenticateRequest events, the login page will, when submitted, redirect the user to the rewritten page. That is, imagine that a user types into their browser window, /people/ScottMitchell.aspx, which is rewritten to /info/employee.aspx?empID=1001. If the Web application is configured to use forms authentication, when the user first visits /people/ScottMitchell.aspx, first the URL will be rewritten to /info/employee.aspx?empID=1001; next, the FormsAuthenticationModule will run, redirecting the user to the login page, if needed. The URL the user will be sent to upon successfully logging in, however, will be /info/employee.aspx?empID=1001, since that was the URL of the request when the FormsAuthenticationModule ran.

Similarly, when performing rewriting in the BeginRequest or AuthenticateRequest events, the UrlAuthorizationModule sees the rewritten URL. That means that if you use <location> elements in your Web.config file to specify authorization for specific URLs, you will have to refer to the rewritten URL.

To fix these subtleties, you might decide to perform the URL rewriting in the AuthorizeRequest event. While this approach fixes the URL authorization and forms authentication anomalies, it introduces a new wrinkle: file authorization no longer works. When using Windows authentication, the FileAuthorizationModule checks to make sure that the authenticated user has the appropriate access rights to access the specific ASP.NET page.

Imagine if a set of users does not have Windows-level file access to C:\Inetput\wwwroot\info\employee.aspx; if such users attempt to visit /info/employee.aspx?empID=1001, then they will get an authorization error. However, if we move the URL rewriting to the AuthenticateRequest event, when the FileAuthorizationModule checks the security settings, it still thinks the file being requested is /people/ScottMitchell.aspx, since the URL has yet to be rewritten. Therefore, the file authorization check will pass, allowing this user to view the content of the rewritten URL, /info/employee.aspx?empID=1001.

So, when should URL rewriting be performed in an HTTP module? It depends on what type of authentication you're employing. If you're not using any authentication, then it doesn't matter if URL rewriting happens in BeginRequest, AuthenticateRequest, or AuthorizeRequest. If you are using forms authentication and are not using Windows authentication, place the URL rewriting in the AuthorizeRequest event handler. Finally, if you are using Windows authentication, schedule the URL rewriting during the BeginRequest or AuthenticateRequest events.

URL Rewriting in HTTP Handlers

URL rewriting can also be performed by an HTTP handler or HTTP handler factory. Recall that an HTTP handler is a class responsible for generating the content for a specific type of request; an HTTP handler factory is a class responsible for returning an instance of an HTTP handler that can generate the content for a specific type of request.

In this article we'll look at creating a URL rewriting HTTP handler factory for ASP.NET Web pages. HTTP handler factories must implement the IHttpHandlerFactory interface, which includes a GetHandler() method. After initializing the appropriate HTTP modules, the ASP.NET engine determines what HTTP handler or HTTP handler factory to invoke for the given request. If an HTTP handler factory is to be invoked, the ASP.NET engine calls that HTTP handler factory's GetHandler() method passing in the HttpContext for the Web request, along with some other information. The HTTP handler factory, then, must return an object that implements IHttpHandler that can handle the request.

To perform URL rewriting through an HTTP handler, we can create an HTTP handler factory whose GetHandler() method checks the requested path to determine if it needs to be rewritten. If it does, it can call the passed-in HttpContext object's RewritePath() method, as discussed earlier. Finally, the HTTP handler factory can return the HTTP handler returned by the System.Web.UI.PageParser class's GetCompiledPageInstance() method. (This is the same technique by which the built-in ASP.NET Web page HTTP handler factory, PageHandlerFactory, works.)

Since all of the HTTP modules will have been initialized prior to the custom HTTP handler factory being instantiated, using an HTTP handler factory presents the same challenges when placing the URL rewriting in the latter stages of the events—namely, file authorization will not work. So, if you rely on Windows authentication and file authorization, you will want to use the HTTP module approach for URL rewriting.

Over the next section we'll look at building a reusable URL rewriting engine. Following our examination of the URL rewriting engine—which is available in this article's code download—we'll spend the remaining two sections examining real-world uses of URL rewriting. First we'll look at how to use the URL rewriting engine and look at a simple URL rewriting example. Following that, we'll utilize the power of the rewriting engine's regular expression capabilities to provide truly "hackable" URLs.

Building a URL Rewriting Engine

To help illustrate how to implement URL rewriting in an ASP.NET Web application, I created a URL rewriting engine. This rewriting engine provides the following functionality:

The ASP.NET page developer utilizing the URL rewriting engine can specify the rewriting rules in the Web.config file.

The rewriting rules can use regular expressions to allow for powerful rewriting rules.

URL rewriting can be easily configured to use an HTTP module or an HTTP handler.

In this article we will examine URL rewriting with just the HTTP module. To see how HTTP handlers can be used to perform URL rewriting, consult the code available for download with this article.

Specifying Configuration Information for the URL Rewriting Engine

Let's examine the structure of the rewrite rules in the Web.config file. First, you'll need to indicate in the Web.config file if you want perform URL rewriting with the HTTP module or the HTTP handler. In the download, the Web.config file contains two entries that have been commented out:

<!--  <httpModules>     <add type="URLRewriter.ModuleRewriter, URLRewriter"           name="ModuleRewriter" />  </httpModules>  -->    <!--  <httpHandlers>     <add verb="*" path="*.aspx"           type="URLRewriter.RewriterFactoryHandler, URLRewriter" />  </httpHandlers>  -->

Comment out the <httpModules> entry to use the HTTP module for rewriting; comment out the <httpHandlers> entry instead to use the HTTP handler for rewriting.

In addition to specifying whether the HTTP module or HTTP handler is used for rewriting, the Web.config file contains the rewriting rules. A rewriting rule is composed of two strings: the pattern to look for in the requested URL, and the string to replace the pattern with, if found. This information is expressed in the Web.config file using the following syntax:

<RewriterConfig>     <Rules>     <RewriterRule>        <LookFor>pattern to look for</LookFor>        <SendTo>string to replace pattern with</SendTo>     </RewriterRule>     <RewriterRule>        <LookFor>pattern to look for</LookFor>        <SendTo>string to replace pattern with</SendTo>     </RewriterRule>     ...     </Rules>  </RewriterConfig>

Each rewrite rule is expressed by a <RewriterRule> element. The pattern to search for is specified by the <LookFor> element, while the string to replace the found pattern with is entered in the <SentTo> element. These rewrite rules are evaluated from top to bottom. If a match is found, the URL is rewritten and the search through the rewriting rules terminates.

When specifying patterns in the <LookFor> element, realize that regular expressions are used to perform the matching and string replacement. (In a bit we'll look at a real-world example that illustrates how to search for a pattern using regular expressions.) Since the pattern is a regular expression, be sure to escape any characters that are reserved characters in regular expressions. (Some of the regular expression reserved characters include: ., ?, ^, $, and others. These can be escaped by being preceded with a backslash, like \. to match a literal period.)

URL Rewriting with an HTTP Module

Creating an HTTP module is as simple as creating a class that implements the IHttpModule interface. The IHttpModule interface defines two methods:

Init(HttpApplication). This method fires when the HTTP module is initialized. In this method you'll wire up event handlers to the appropriate HttpApplication events.

Dispose(). This method is invoked when the request has completed and been sent back to IIS. Any final cleanup should be performed here.

To facilitate creating an HTTP module for URL rewriting, I started by creating an abstract base class, BaseModuleRewriter. This class implements IHttpModule. In the Init() event, it wires up the HttpApplication's AuthorizeRequest event to the BaseModuleRewriter_AuthorizeRequest method. The BaseModuleRewriter_AuthorizeRequest method calls the class's Rewrite() method passing in the requested Path along with the HttpApplication object that was passed into the Init() method. The Rewrite() method is abstract, meaning that in the BaseModuleRewriter class, the Rewrite() method has no method body; rather, the class being derived from BaseModuleRewriter must override this method and provide a method body.

With this base class in place, all we have to do now is to create a class derived from BaseModuleRewriter that overrides Rewrite() and performs the URL rewriting logic there. The code for BaseModuleRewriter is shown below.

public abstract class BaseModuleRewriter : IHttpModule  {     public virtual void Init(HttpApplication app)     {        // WARNING!  This does not work with Windows authentication!        // If you are using Windows authentication,         // change to app.BeginRequest        app.AuthorizeRequest += new            EventHandler(this.BaseModuleRewriter_AuthorizeRequest);     }       public virtual void Dispose() {}       protected virtual void BaseModuleRewriter_AuthorizeRequest(       object sender, EventArgs e)     {        HttpApplication app = (HttpApplication) sender;        Rewrite(app.Request.Path, app);     }       protected abstract void Rewrite(string requestedPath,        HttpApplication app);  }

Notice that the BaseModuleRewriter class performs URL rewriting in the AuthorizeRequest event. Recall that if you use Windows authentication with file authorization, you will need to change this so that URL rewriting is performed in either the BeginRequest or AuthenticateRequest events.

The ModuleRewriter class extends the BaseModuleRewriter class and is responsible for performing the actual URL rewriting. ModuleRewriter contains a single overridden method—Rewrite()—which is shown below:

protected override void Rewrite(string requestedPath,      System.Web.HttpApplication app)  {     // get the configuration rules     RewriterRuleCollection rules =        RewriterConfiguration.GetConfig().Rules;       // iterate through each rule...     for(int i = 0; i < rules.Count; i++)     {        // get the pattern to look for, and         // Resolve the Url (convert ~ into the appropriate directory)        string lookFor = "^" +           RewriterUtils.ResolveUrl(app.Context.Request.ApplicationPath,           rules[i].LookFor) + "$";          // Create a regex (note that IgnoreCase is set...)        Regex re = new Regex(lookFor, RegexOptions.IgnoreCase);          // See if a match is found        if (re.IsMatch(requestedPath))        {           // match found - do any replacement needed           string sendToUrl =   RewriterUtils.ResolveUrl(app.Context.Request.ApplicationPath,               re.Replace(requestedPath, rules[i].SendTo));             // Rewrite the URL           RewriterUtils.RewriteUrl(app.Context, sendToUrl);           break;      // exit the for loop        }     }  }

The Rewrite() method starts with getting the set of rewriting rules from the Web.config file. It then iterates through the rewrite rules one at a time, and for each rule, it grabs its LookFor property and uses a regular expression to determine if a match is found in the requested URL.

If a match is found, a regular expression replace is performed on the requested path with the value of the SendTo property. This replaced URL is then passed into the RewriterUtils.RewriteUrl() method. RewriterUtils is a helper class that provides a couple of static methods used by both the URL rewriting HTTP module and HTTP handler. The RewriterUrl() method simply calls the HttpContext object's RewriteUrl() method.

Note You may have noticed that when performing the regular expression match and replacement, a call to RewriterUtils.ResolveUrl() is made. This helper method simply replaces any instances of ~ in the string with the value of the application's path.

The entire code for the URL rewriting engine is available for download with this article. We've examined the most germane pieces, but there are other components as well, such as classes for deserializing the XML-formatted rewriting rules in the Web.config file into an object, as well as the HTTP handler factory for URL rewriting. The remaining three sections of this article examine real-world uses of URL rewriting.

Performing Simple URL Rewriting with the URL Rewriting Engine

To demonstrate the URL rewriting engine in action, let's build an ASP.NET Web application that utilizes simple URL rewriting. Imagine that we work for a company that sells assorted products online. These products are broken down into the following categories:

Category ID	Category Name
1	Beverages
2	Condiments
3	Confections
4	Dairy Products
...	...

Assume we already have created an ASP.NET Web page called ListProductsByCategory.aspx that accepts a Category ID value in the querystring and displays all of the products belonging to that category. So, users who wanted to view our Beverages for sale would visit ListProductsByCategory.aspx?CategoryID=1, while users who wanted to view our Dairy Products would visit ListProductsByCategory.aspx?CategoryID=4. Also assume we have a page called ListCategories.aspx, which lists the categories of products for sale.

Clearly this is a case for URL rewriting, as the URLs a user is presented with do not carry any significance for the user, nor do they provide any "hackability." Rather, let's employ URL rewriting so that when a user visits /Products/Beverages.aspx, their URL will be rewritten to ListProductsByCategory.aspx?CategoryID=1. We can accomplish this with the following URL rewriting rule in the Web.config file:

<RewriterConfig>     <Rules>        <!-- Rules for Product Lister -->        <RewriterRule>           <LookFor>~/Products/Beverages\.aspx</LookFor>           <SendTo>~/ListProductsByCategory.aspx?CategoryID=1</SendTo>        </RewriterRule>        <RewriterRule>     </Rules>  </RewriterConfig>

As you can see, this rule searches to see if the path requested by the user was /Products/Beverages.aspx. If it was, it rewrites the URL as /ListProductsByCategory.aspx?CategoryID=1.

Note Notice that the <LookFor> element escapes the period in Beverages.aspx. This is because the <LookFor> value is used in a regular expression pattern, and period is a special character in regular expressions meaning "match any character," meaning a URL of /Products/BeveragesQaspx, for example, would match. By escaping the period (using \.) we are indicating that we want to match a literal period, and not any old character.

With this rule in place, when a user visits /Products/Beverages.aspx, they will be shown the beverages for sale. Figure 3 shows a screenshot of a browser visiting /Products/Beverages.aspx. Notice that in the browser's Address bar the URL reads /Products/Beverages.aspx, but the user is actually seeing the contents of ListProductsByCategory.aspx?CategoryID=1. (In fact, there doesn't even exist a /Products/Beverages.aspx file on the Web server at all!)

Figure 3. Requesting category after rewriting URL

Similar to /Products/Beverages.aspx, we'd next add rewriting rules for the other product categories. This simply involves adding additional <RewriterRule> elements within the <Rules> element in the Web.config file. Consult the Web.config file in the download for the complete set of rewriting rules for the demo.

To make the URL more "hackable," it would be nice if a user could simply hack off the Beverages.aspx from /Products/Beverages.aspx and be shown a listing of the product categories. At first glance, this may appear a trivial task—just add a rewriting rule that maps /Products/ to /ListCategories.aspx. However, there is a fine subtlety—you must first create a /Products/ directory and add an empty Default.aspx file in the /Products/ directory.

To understand why these extra steps need to be performed, recall that the URL rewriting engine is at the ASP.NET level. That is, if the ASP.NET engine is never given the opportunity to process the request, there's no way the URL rewriting engine can inspect the incoming URL. Furthermore, remember that IIS hands off incoming requests to the ASP.NET engine only if the requested file has an appropriate extension. So if a user visits /Products/, IIS doesn't see any file extension, so it checks the directory to see if there exists a file with one of the default filenames. (Default.aspx, Default.htm, Default.asp, and so on. These default filenames are defined in the Documents tab of the Web Server Properties dialog box in the IIS Administration dialog box.) Of course, if the /Products/ directory doesn't exist, IIS will return an HTTP 404 error.

So, we need to create the /Products/ directory. Additionally, we need to create a single file in this directory, Default.aspx. This way, when a user visits /Products/, IIS will inspect the directory, see that there exists a file named Default.aspx, and then hand off processing to the ASP.NET engine. Our URL rewriter, then, will get a crack at rewriting the URL.

After creating the directory and Default.aspx file, go ahead and add the following rewriting rule to the <Rules> element:

<RewriterRule>     <LookFor>~/Products/Default\.aspx</LookFor>     <SendTo>~/ListCategories.aspx</SendTo>  </RewriterRule>

With this rule in place, when a user visits /Products/ or /Products/Default.aspx, they will see the listing of product categories, shown in Figure 4.

Figure 4. Adding "hackability" to the URL

Handling Postbacks

If the URLs you are rewriting contain a server-side Web Form and perform postbacks, when the form posts back, the underlying URL will be used. That is, if our user enters into their browser, /Products/Beverages.aspx, they will still see in their browser's Address bar, /Products/Beverages.aspx, but they will be shown the content for ListProductsByCategory.aspx?CategoryID=1. If ListProductsByCategory.aspx performs a postback, the user will be posted back to ListProductsByCategory.aspx?CategoryID=1, not /Products/Beverages.aspx. This won't break anything, but it can be disconcerting from the user's perspective to see the URL change suddenly upon clicking a button.

The reason this behavior happens is because when the Web Form is rendered, it explicitly sets its action attribute to the value of the file path in the Request object. Of course, by the time the Web Form is rendered, the URL has been rewritten from /Products/Beverages.aspx to ListProductsByCategory.aspx?CategoryID=1, meaning the Request object is reporting that the user is visiting ListProductsByCategory.aspx?CategoryID=1. This problem can be fixed by having the server-side form simply not render an action attribute. (Browsers, by default, will postback if the form doesn't contain an action attribute.)

Unfortunately, the Web Form does not allow you to explicitly specify an action attribute, nor does it allow you to set some property to disable the rendering of the action attribute. Rather, we'll have to extend the System.Web.HtmlControls.HtmlForm class ourselves, overriding the RenderAttribute() method and explicitly indicating that it not render the action attribute.

Thanks to the power of inheritance, we can gain all of the functionality of the HtmlForm class and only have to add a scant few lines of code to achieve the desired behavior. The complete code for the custom class is shown below:

namespace ActionlessForm {    public class Form : System.Web.UI.HtmlControls.HtmlForm    {       protected override void RenderAttributes(HtmlTextWriter writer)       {          writer.WriteAttribute("name", this.Name);          base.Attributes.Remove("name");            writer.WriteAttribute("method", this.Method);          base.Attributes.Remove("method");            this.Attributes.Render(writer);            base.Attributes.Remove("action");            if (base.ID != null)             writer.WriteAttribute("id", base.ClientID);       }    }  }

The code for the overridden RenderAttributes() method simply contains the exact code from the HtmlForm class's RenderAttributes() method, but without setting the action attribute. (I used Lutz Roeder's Reflector to view the source code of the HtmlForm class.)

Once you have created this class and compiled it, to use it in an ASP.NET Web application, start by adding it to the Web application's References folder. Then, to use it in place of the HtmlForm class, simply add the following to the top of your ASP.NET Web page:

<%@ Register TagPrefix="skm" Namespace="ActionlessForm"      Assembly="ActionlessForm" %>

Then, where you have <form runat="server">, replace that with:

<skm:Form id="Form1" method="post" runat="server">

and replace the closing </form> tag with:

</skm:Form>

You can see this custom Web Form class in action in ListProductsByCategory.aspx, which is included in this article's download. Also included in the download is a Visual Studio .NET project for the action-less Web Form.

Note If the URL you are rewriting to does not perform a postback, there's no need to use this custom Web Form class.

Creating Truly "Hackable" URLs

The simple URL rewriting demonstrated in the previous section showed how easily the URL rewriting engine can be configured with new rewriting rules. The true power of the rewriting rules, though, shines when using regular expressions, as we'll see in this section.

Blogs are becoming more and more popular these days, and it seems everyone has their own blog. If you are not familiar with blogs, they are often-updated personal pages that typically serve as an online journal. Most bloggers simply write about their day-to-day happenings, others focus on blogging about a specific theme, such as movie reviews, a sports team, or a computer technology.

Depending on the author, blogs are updated anywhere from several times a day to once every week or two. Typically the blog homepage shows the most recent 10 entries, but virtually all blogging software provides an archive through which visitors can read older postings. Blogs are a great application for "hackable" URLs. Imagine while searching through the archives of a blog you found yourself at the URL /2004/02/14.aspx. Would you be terribly surprised if you found yourself reading the posts made on February 14th, 2004? Furthermore, you might want to view all posts for February 2004, in which case you might try hacking the URL to /2004/02/. To view all 2004 posts, you might try visiting /2004/.

When maintaining a blog, it would be nice to provide this level of URL "hackability" to your visitors. While many blog engines provide this functionality, let's look at how it can be accomplished using URL rewriting.

First, we need a single ASP.NET Web page that will show blog entries by day, month, or year. Assume we have such a page, ShowBlogContent.aspx, that takes in querystring parameters year, month, and day. To view the posts for February 14th, 2004, we could visit ShowBlogContent.aspx?year=2004&month=2&day=14. To view all posts for February 2004, we'd visit ShowBlogContent.aspx?year=2004&month=2. Finally, to see all posts for the year 2004, we'd navigate to ShowBlogContent.aspx?year=2004. (The code for ShowBlogContent.aspx can be found in this article's download.)

So, if a user visits /2004/02/14.aspx, we need to rewrite the URL to ShowBlogContent.aspx?year=2004&month=2&day=14. All three cases—when the URL specifies a year, month, and day; when the URL specifies just the year and month; and when the URL specifies only the yea—can be handled with three rewrite rules:

<RewriterConfig>     <Rules>        <!-- Rules for Blog Content Displayer -->        <RewriterRule>           <LookFor>~/(\d{4})/(\d{2})/(\d{2})\.aspx</LookFor>           <SendTo>~/ShowBlogContent.aspx?year=$1&amp;month=$2&day=$3</SendTo>        </RewriterRule>        <RewriterRule>           <LookFor>~/(\d{4})/(\d{2})/Default\.aspx</LookFor>           <SendTo><![CDATA[~/ShowBlogContent.aspx?year=$1&month=$2]]></SendTo>        </RewriterRule>        <RewriterRule>           <LookFor>~/(\d{4})/Default\.aspx</LookFor>           <SendTo>~/ShowBlogContent.aspx?year=$1</SendTo>        </RewriterRule>     </Rules>  </RewriterConfig>

These rewriting rules demonstrate the power of regular expressions. In the first rule, we look for a URL with the pattern (\d{4})/(\d{2})/(\d{2})\.aspx. In plain English, this matches a string that has four digits followed by a forward slash followed by two digits followed by a forward slash, followed by two digits followed by .aspx. The parenthesis around each digit grouping is vital—it allows us to refer to the matched characters inside those parentheses in the corresponding <SendTo> property. Specifically, we can refer back to the matched parenthetical groupings using $1, $2, and $3 for the first, second, and third parenthesis grouping, respectively.

Note Since the Web.config file is XML-formatted, characters like &, <, and > in the text portion of an element must be escaped. In the first rule's <SendTo> element, & is escaped to &amp;. In the second rule's <SendTo>, an alternative technique is used—by using a <![CDATA[...]]> element, the contents inside do not need to be escaped. Either approach is acceptable and accomplishes the same end.

Figures 5, 6, and 7 show the URL rewriting in action. The data is actually being pulled from my blog, http://ScottOnWriting.NET. In Figure 5, the posts for November 7, 2003 are shown; in Figure 6 all posts for November 2003 are shown; Figure 7 shows all posts for 2003.

Figure 5. Posts for November 7, 2003

Figure 6. All posts for November 2003

Figure 7. All posts for 2003

Note The URL rewriting engine expects a regular expression pattern in the <LookFor> elements. If you are unfamiliar with regular expressions, consider reading an earlier article of mine, An Introduction to Regular Expressions. Also, a great place to get your hands on commonly used regular expressions, as well as a repository for sharing your own crafted regular expressions, is RegExLib.com.

Building the Requisite Directory Structure

When a request comes in for /2004/03/19.aspx, IIS notes the .aspx extension and routes the request to the ASP.NET engine. As the request moves through the ASP.NET engine's pipeline, the URL will get rewritten to ShowBlogContent.aspx?year=2004&month=03&day=19 and the visitor will see those blog entries for March 19, 2004. But what happens when the user navigates to /2004/03/? Unless there is a directory /2004/03/, IIS will return a 404 error. Furthermore, there needs to be a Default.aspx page in this directory so that the request is handed off to the ASP.NET engine.

So with this approach, you have to manually create a directory for each year in which there are blog entries, with a Default.aspx page in the directory. Additionally, in each year directory you need to manually create twelve more directories—01, 02, …, 12—each with a Default.aspx file. (Recall that we had to do the same thing—add a /Products/ directory with a Default.aspx file—in the previous demo so that visiting /Products/ correctly displayed ListCategories.aspx.)

Clearly, adding such a directory structure can be a pain. A workaround to this problem is to have all incoming IIS requests map to the ASP.NET engine. This way, even if when visiting the URL /2004/03/, IIS will faithfully hand off the request to the ASP.NET engine even if there does not exist a /2004/03/ directory. Using this approach, however, makes the ASP.NET engine responsible for handling all types of incoming requests to the Web server, including images, CSS files, external JavaScript files, Macromedia Flash files, and so on.

A thorough discussion of handling all file types is far beyond the scope of this article. For an example of an ASP.NET Web application that uses this technique, though, look into .Text, an open-source blog engine. .Text can be configured to have all requests mapped to the ASP.NET engine. It can handle serving all file types by using a custom HTTP handler that knows how to serve up typical static file types (images, CSS files, and so on).

Conclusion

In this article we examined how to perform URL rewriting at the ASP.NET-level through the HttpContext class's RewriteUrl() method. As we saw, RewriteUrl() updates the particular HttpContext's Request property, updating what file and path is being requested. The net effect is that, from the user's perspective, they are visiting a particular URL, but actually a different URL is being requested on the Web server side.

URLs can be rewritten either in an HTTP module or an HTTP handler. In this article we examined using an HTTP module to perform the rewriting, and looked at the consequences of performing the rewriting at different stages in the pipeline.

Of course, with ASP.NET-level rewriting, the URL rewriting can only happen if the request is successfully handed off from IIS to the ASP.NET engine. This naturally occurs when the user requests a page with a .aspx extension. However, if you want the person to be able to enter a URL that might not actually exist, but would rather rewrite to an existing ASP.NET page, you have to either create mock directories and Default.aspx pages, or configure IIS so that all incoming requests are blindly routed to the ASP.NET engine.

Related Books

ASP.NET: Tips, Tutorials, and Code

Microsoft ASP.NET Coding Strategies with the Microsoft ASP.NET Team

Essential ASP.NET with Examples in C#

Works consulted

URL rewriting is a topic that has received a lot of attention both for ASP.NET and competing server-side Web technologies. The Apache Web server, for instance, provides a module for URL rewriting called mod_rewrite. mod_rewrite is a robust rewriting engine, providing rewriting rules based on conditions such as HTTP headers and server variables, as well as rewriting rules that utilize regular expressions. For more information on mod_rewrite, check out A User's Guide to URL Rewriting with the Apache Web Server.

There are a number of articles on URL rewriting with ASP.NET. Rewrite.NET - A URL Rewriting Engine for .NET examines creating a URL rewriting engine that mimics mod_rewrite's regular expression rules. URL Rewriting With ASP.NET also gives a good overview of ASP.NET's URL rewriting capabilities. Ian Griffiths has a blog entry on some of the caveats associated with URL rewriting with ASP.NET, such as the postback issue discussed in this article. Both Fabrice Marguerie (read more) and Jason Salas (read more) have blog entires on using URL rewriting to boost search engine placement.

About the author

Scott Mitchell, author of five books and founder of 4GuysFromRolla.com, has been working with Microsoft Web technologies for the past five years. Scott works as an independent consultant, trainer, and writer. He can be reached at mitchell@4guysfromrolla.com or through his blog, which can be found at http://ScottOnWriting.NET.

A Matter of Context

2006-02-26T18:21:00.000+07:00

A Matter of Context

Susan Warren
Microsoft Corporation

January 14, 2002

One of the most common problems with writing Web applications is letting your code know the context in which it's being executed. Let's look at a simple example—personalizing a page—that illustrates this problem:

Please sign in.

vs.

Welcome Susan!

Seems simple enough, but even this tiny bit of Web UI requires a couple of bits of information that will vary each time the page is requested. I'll need to know:

1. Is the user signed in?

2. What is the user's display name?

More generally, what is the unique context each time the page is requested? And how can I write my code so that it takes this information into account?

In fact, due to the stateless nature of HTTP, there are many different pieces of context a Web application might need to track. When a user interacts with a Web application, the browser sends a series of independent HTTP requests to the Web server. The application itself has to do the work of knitting these requests into a pleasing experience for the user and knowing the context of the request is critical.

ASP introduced several intrinsic objects like Request and Application to help track the context for an HTTP request. ASP.NET takes the next step and bundles these objects, plus several additional context-related objects into an extremely handy intrinsic object called Context.

Context is an object of type System.Web.HttpContext. It is exposed as a property of the ASP.NET Page class. It's also available from user controls and your business objects (more on that later). Here's a partial list of the objects rolled up by HttpContext:

Object
Description

Application
A key/value pair collection of values that is accessible by every user of the application. Application is of type System.Web.HttpApplicationState.

ApplicationInstance
The actual running application, which exposes some request processing events. These events are handled in Global.asax, or an HttpHandler or HttpModule.

Cache
The ASP.NET Cache object, which provides programmatic access to the cache. Rob Howard's ASP.NET Caching column provides a good introduction to caching.

Error
The first error (if any) encountered while processing the page. See Rob's Exception to the Rule, Part 1 for more information.

Items
A key-value pair collection that you can use to pass information between all of the components that participate in the processing of a single request. Items is of type System.Collections.IDictionary.

Request
Information about the HTTP request, including browser information, cookies, and values passed in a form or on the query string. Request is of type System.Web.HttpRequest.

Response
Settings and content for creating the HTTP response. Request is of type System.Web.HttpResponse.

Server
Server is a utility class with several useful helper methods, including Server.Execute(), Server.MapPath(), and Server.HtmlEncode(). Server is an object of type System.Web.HttpServerUtility.

Session
A key/value pair collection of values that are accessible by a single user of the application. Application is of type System.Web.HttpSessionState.

Trace
The ASP.NET Trace object, which provides access to tracing functionality. See Rob's Tracing article for more information.

User
The security context of the current user, if authenticated. Context.User.Identity is the user's name. User is an object of type System.Security.Principal.IPrincipal.

If you're an ASP developer, some of the objects above will look quite familiar. There are a few enhancements, but for the most part, they work exactly the same in ASP.NET as in ASP.

Context Basics

Some of the objects in Context are also promoted as top-level objects on Page. For example, Page.Context.Response and Page.Response reference the same object so the following code is equivalent:

[Visual Basic® Web Form]

Response.Write ("Hello ")
Context.Response.Write ("There")

[C# Web Form]

Response.Write ("Hello ");
Context.Response.Write ("There");

You can also use the Context object from your business objects. HttpContext.Current is a static property that conveniently returns the context for the current request. This is useful in all kinds of ways, but here's a simple example of retrieving an item from the cache in your business class:

[Visual Basic]

' get the request context
Dim _context As HttpContext = HttpContext.Current

' get dataset from the cache
Dim _data As DataSet = _context.Cache("MyDataSet")

[C#]

// get the request context
HttpContext _context = HttpContext.Current;

// get dataset from cache
DataSet _data = _context.Cache("MyDataSet");

Context in Action

The Context object provides The Answer to several common ASP.NET "How Do I ...?" questions. Perhaps the best way to communicate just how valuable this gem can be is to show it in action. Here are a few of the best Context tricks I know.

How Do I Emit an ASP.NET Trace Statement From My Business Class?

Answer: Easy! Use HttpContext.Current to get the Context object, then call Context.Trace.Write().

[Visual Basic]

Imports System
Imports System.Web

Namespace Context

' Demonstrates emitting an ASP.NET trace statement from a
' business class.

Public Class TraceEmit

Public Sub SomeMethod()

' get the request context
Dim _context As HttpContext = HttpContext.Current

' use context to write the trace statement
_context.Trace.Write("in TraceEmit.SomeMethod")

End Sub

End Class

End Namespace

[C#]

using System;
using System.Web;

namespace Context
{
// Demonstrates emitting an ASP.NET trace statement from a
// business class.

public class TraceEmit
{

public void SomeMethod() {

// get the request context
HttpContext _context = HttpContext.Current;

// use context to write the trace statement
_context.Trace.Write("in TraceEmit.SomeMethod");
}
}
}

How Can I Access a Session State Value From My Business Class?

Answer: Easy! Use HttpContext.Current to get the Context object, then access Context.Session.

[Visual Basic]

Imports System
Imports System.Web

Namespace Context

' Demonstrates accessing the ASP.NET Session intrinsic
' from a business class.

Public Class UseSession

Public Sub SomeMethod()

' get the request context
Dim _context As HttpContext = HttpContext.Current

' access the Session intrinsic
Dim _value As Object = _context.Session("TheValue")

End Sub

End Class

End Namespace

[C#]

using System;
using System.Web;

namespace Context
{
// Demonstrates accessing the ASP.NET Session intrinsic
// from a business class.

public class UseSession
{

public void SomeMethod() {

// get the request context
HttpContext _context = HttpContext.Current;

// access the Session intrinsic
object _value = _context.Session["TheValue"];
}
}
}

How Can I Add a Standard Header and Footer to Every Page in My Application?

Answer: Handle the application's BeginRequest and EndRequest events, and use Context.Response.Write to emit the HTML for the header and footer.

Technically, you can handle the application events like BeginRequest in either an HttpModule or by using Global.asax. HttpModules are a bit harder to write, and aren't typically used for functionality that is used by a single application, as in this example. So, we'll use the application-scoped Global.asax file instead.

As with an ASP page, several of the ASP.NET context intrinsics are promoted to be properties of the HttpApplication class, from which the class representing Global.asax inherits. We won't need to use HttpContext.Current to get a reference to the Context object; it's already available in Global.asax.

In this example, I'm putting the and tags, plus a horizontal rule into the header section, and another horizontal rule plus the end tags for these into the footer section. The footer also contains a copyright message. The result looks like the figure below:

Figure 1. Example of standard header and footer as rendered in the browser

This is a trivial example, but you can easily extend this to include your standard header and navigation, or simply output the

statements for these. One caveat—if you want the header or footer to include interactive content, you should consider using ASP.NET user controls instead.

[SomePage.aspx source—sample content]

Normal Page Content

[Visual Basic Global.asax]

<%@ Application Language="VB" %>

[C# Global.asax]

<%@ Application Language="C#" %>

How Can I Show A Welcome Message When The User Is Authenticated?

The Answer: Test the User context object to see if the user is authenticated. If so, get the user's name from the User object too. This is, of course, the example from the beginning of the article.

[Visual Basic]

[C#]

And Now for Something Really Wonderful: Context.Items

I hope the examples above show how much easier it is to write your Web application with a little context information at hand. Wouldn't it be great to be able to access some context that is unique to your application in the same way?

That's the purpose of the Context.Items collection. It holds your application's request-specific values in a way that is available to every part of your code that participates in the processing of a request. For example, the same piece of information can be used in Global.asax, in your ASPX page, in the user controls within the page, and by the business logic the page calls.

Consider the IBuySpy Portal sample application. It uses a single main page—DesktopDefault.aspx—to display portal content. Which content is displayed depends on which tab is selected, as well as the roles of the user, if authenticated.

Figure 2. IbuySpy home page

The querystring includes the TabIndex and TabId parameters for the tab being requested. This information is used throughout the processing of the request to filter which data is displayed to the user. http://www.ibuyspy.com/portal/DesktopDefault.aspx?tabindex=1&tabid=2

To use a querystring value, you need to first make sure it's a valid value and, if not, do a little error handling. It's not a lot of code, but do you really want to duplicate it in every page and component that uses the value? Of course not! In the Portal sample it is even more involved since there is other information that can be preloaded once we know the TabId.

The Portal uses the querystring values as parameters to construct a new "PortalSettings" object and add it to Context.Items in the BeginRequest event in Global.asax. Since the begin request is executed at the beginning of each request, this makes the tab-related values available to all of the pages and components in the application. When the request is complete, the object is automatically discarded—very tidy!

[Visual Basic Global.asax]

Sub Application_BeginRequest(sender As [Object], e As EventArgs)

Dim tabIndex As Integer = 0
Dim tabId As Integer = 0

' Get TabIndex from querystring
If Not (Request.Params("tabindex") Is Nothing) Then
tabIndex = Int32.Parse(Request.Params("tabindex"))
End If

' Get TabID from querystring
If Not (Request.Params("tabid") Is Nothing) Then
tabId = Int32.Parse(Request.Params("tabid"))
End If

Context.Items.Add("PortalSettings", _
New PortalSettings(tabIndex, tabId))

End Sub

[C# Global.asax]

void Application_BeginRequest(Object sender, EventArgs e) {

int tabIndex = 0;
int tabId = 0;

// Get TabIndex from querystring

if (Request.Params["tabindex"] != null) {
tabIndex = Int32.Parse(Request.Params["tabindex"]);
}

// Get TabID from querystring

if (Request.Params["tabid"] != null) {
tabId = Int32.Parse(Request.Params["tabid"]);
}

Context.Items.Add("PortalSettings",
new PortalSettings(tabIndex, tabId));
}

The DesktopPortalBanner.ascx user control pulls the PortalSetting's object from Context to access the Portal's name and security settings. In fact, this one module is a great all-around example of Context in action. To illustrate the point, I've simplified the code a little, and marked all of the places either HTTP or application-specific Context is accessed in bold.

[C# DesktopPortalBanner.ascx]

<%@ Import Namespace="ASPNetPortal" %>
<%@ Import Namespace="System.Data.SqlClient" %>

Request.ApplicationPath %>">Portal Home
|
Request.ApplicationPath %>/Docs/Docs.asp">
Portal Documentation
<%= LogoffLink %>

Request.ApplicationPath %>
/DesktopDefault.aspx?tabindex=<%# Container.ItemIndex %>&tabid=
<%# ((TabStripDetails) Container.DataItem).TabId %>'>
<%# ((TabStripDetails) Container.DataItem).TabName %>
 

<%# ((TabStripDetails) Container.DataItem).TabName %>

You can browse and run the complete source for the IBuySpy portal online in both Visual Basic and C# at http://www.ibuyspy.com, or download it and run it yourself.

Summary

Context is another one of those "good things get even better in ASP.NET" features. It extends the already great context support of ASP to add both hooks into the new runtime features of ASP.NET. Plus it adds Context.Items as a new state mechanism for very short-lived values. But the ultimate benefit to you as a developer is more compact, easier to maintain code, and that's a context we can all get behind.

A low-level Look at the ASP.NET Architecture

2006-02-26T18:07:00.000+07:00

By Rick Strahl

www.west-wind.com

rstrahl@west-wind.com

Last Update:

August 29, 2005

Other Links:

Download Examples for this article

Leave a Comment or Question

ASP.NET is a powerful platform for building Web applications, that provides a tremendous amount of flexibility and power for building just about any kind of Web application. Most people are familiar only with the high level frameworks like WebForms and WebServices which sit at the very top level of the ASP.NET hierarchy. In this article I’ll describe the lower level aspects of ASP.NET and explain how requests move from Web Server to the ASP.NET runtime and then through the ASP.NET Http Pipeline to process requests.

To me understanding the innards of a platform always provides certain satisfaction and level of comfort, as well as insight that helps to write better applications. Knowing what tools are available and how they fit together as part of the whole complex framework makes it easier to find the best solution to a problem and more importantly helps in troubleshooting and debugging of problems when they occur. The goal of this article is to look at ASP.NET from the System level and help understand how requests flow into the ASP.NET processing pipeline. As such we’ll look at the core engine and how Web requests end up there. Much of this information is not something that you need to know in your daily work, but it’s good to understand how the ASP.NET architecture routes request into your application code that usually sits at a much higher level.

Most people using ASP.NET are familiar with WebForms and WebServices. These high level implementations are abstractions that make it easy to build Web based application logic and ASP.NET is the driving engine that provides the underlying interface to the Web Server and routing mechanics to provide the base for these high level front end services typically used for your applications. WebForms and WebServices are merely two very sophisticated implementations of HTTP Handlers built on top of the core ASP.NET framework.

However, ASP.NET provides much more flexibility from a lower level. The HTTP Runtime and the request pipeline provide all the same power that went into building the WebForms and WebService implementations – these implementations were actually built with .NET managed code. And all of that same functionality is available to you, should you decide you need to build a custom platform that sits at a level a little lower than WebForms.

WebForms are definitely the easiest way to build most Web interfaces, but if you’re building custom content handlers, or have special needs for processing the incoming or outgoing content, or you need to build a custom application server interface to another application, using these lower level handlers or modules can provide better performance and more control over the actual request process. With all the power that the high level implementations of WebForms and WebServices provide they also add quite a bit of overhead to requests that you can bypass by working at a lower level.

What is ASP.NET

Let’s start with a simple definition: What is ASP.NET? I like to define ASP.NET as follows:

ASP.NET is a sophisticated engine using Managed Code for front to back processing of Web Requests.

It's much more than just WebForms and Web Services…

ASP.NET is a request processing engine. It takes an incoming request and passes it through its internal pipeline to an end point where you as a developer can attach code to process that request. This engine is actually completely separated from HTTP or the Web Server. In fact, the HTTP Runtime is a component that you can host in your own applications outside of IIS or any server side application altogether. For example, you can host the ASP.NET runtime in a Windows form (check out http://www.west-wind.com/presentations/aspnetruntime/aspnetruntime.asp for more detailed information on runtime hosting in Windows Forms apps).

The runtime provides a complex yet very elegant mechanism for routing requests through this pipeline. There are a number of interrelated objects, most of which are extensible either via subclassing or through event interfaces at almost every level of the process, so the framework is highly extensible. Through this mechanism it’s possible to hook into very low level interfaces such as the caching, authentication and authorization. You can even filter content by pre or post processing requests or simply route incoming requests that match a specific signature directly to your code or another URL. There are a lot of different ways to accomplish the same thing, but all of the approaches are straightforward to implement, yet provide flexibility in finding the best match for performance and ease of development.

The entire ASP.NET engine was completely built in managed code and all extensibility is provided via managed code extensions.

The entire ASP.NET engine was completely built in managed code and all of the extensibility functionality is provided via managed code extensions. This is a testament to the power of the .NET framework in its ability to build sophisticated and very performance oriented architectures. Above all though, the most impressive part of ASP.NET is the thoughtful design that makes the architecture easy to work with, yet provides hooks into just about any part of the request processing.

With ASP.NET you can perform tasks that previously were the domain of ISAPI extensions and filters on IIS – with some limitations, but it’s a lot closer than say ASP was. ISAPI is a low level Win32 style API that had a very meager interface and was very difficult to work for sophisticated applications. Since ISAPI is very low level it also is very fast, but fairly unmanageable for application level development. So, ISAPI has been mainly relegated for some time to providing bridge interfaces to other application or platforms. But ISAPI isn’t dead by any means. In fact, ASP.NET on Microsoft platforms interfaces with IIS through an ISAPI extension that hosts .NET and through it the ASP.NET runtime. ISAPI provides the core interface from the Web Server and ASP.NET uses the unmanaged ISAPI code to retrieve input and send output back to the client. The content that ISAPI provides is available via common objects like HttpRequest and HttpResponse that expose the unmanaged data as managed objects with a nice and accessible interface.

From Browser to ASP.NET

Let’s start at the beginning of the lifetime of a typical ASP.NET Web Request. A request starts on the browser where the user types in a URL, clicks on a hyperlink or submits an HTML form (a POST request). Or a client application might make call against an ASP.NET based Web Service, which is also serviced by ASP.NET. On the server side the Web Server – Internet Information Server 5 or 6 – picks up the request. At the lowest level ASP.NET interfaces with IIS through an ISAPI extension. With ASP.NET this request usually is routed to a page with an .aspx extension, but how the process works depends entirely on the implementation of the HTTP Handler that is set up to handle the specified extension. In IIS .aspx is mapped through an ‘Application Extension’ (aka. as a script map) that is mapped to the ASP.NET ISAPI dll - aspnet_isapi.dll. Every request that fires ASP.NET must go through an extension that is registered and points at aspnet_isapi.dll.

Depending on the extension ASP.NET routes the request to an appropriate handler that is responsible for picking up requests. For example, the .asmx extension for Web Services routes requests not to a page on disk but a specially attributed class that identifies it as a Web Service implementation. Many other handlers are installed with ASP.NET and you can also define your own. All of these HttpHandlers are mapped to point at the ASP.NET ISAPI extension in IIS, and configured in web.config to get routed to a specific HTTP Handler implementation. Each handler, is a .NET class that handles a specific extension which can range from simple Hello World behavior with a couple of lines of code, to very complex handlers like the ASP.NET Page or Web Service implementations. For now, just understand that an extension is the basic mapping mechanism that ASP.NET uses to receive a request from ISAPI and then route it to a specific handler that processes the request.

ISAPI is the first and highest performance entry point into IIS for custom Web Request handling.

The ISAPI Connection

ISAPI is a low level unmanged Win32 API. The interfaces defined by the ISAPI spec are very simplistic and optimized for performance. They are very low level – dealing with raw pointers and function pointer tables for callbacks - but they provide he lowest and most performance oriented interface that developers and tool vendors can use to hook into IIS. Because ISAPI is very low level it’s not well suited for building application level code, and ISAPI tends to be used primarily as a bridge interface to provide Application Server type functionality to higher level tools. For example, ASP and ASP.NET both are layered on top of ISAPI as is Cold Fusion, most Perl, PHP and JSP implementations running on IIS as well as many third party solutions such as my own Web Connection framework for Visual FoxPro. ISAPI is an excellent tool to provide the high performance plumbing interface to higher level applications, which can then abstract the information that ISAPI provides. In ASP and ASP.NET, the engines abstract the information provided by the ISAPI interface in the form of objects like Request and Response that read their content out of the ISAPI request information. Think of ISAPI as the plumbing. For ASP.NET the ISAPI dll is very lean and acts merely as a routing mechanism to pipe the inbound request into the ASP.NET runtime. All the heavy lifting and processing, and even the request thread management happens inside of the ASP.NET engine and your code.

As a protocol ISAPI supports both ISAPI extensions and ISAPI Filters. Extensions are a request handling interface and provide the logic to handle input and output with the Web Server – it’s essentially a transaction interface. ASP and ASP.NET are implemented as ISAPI extensions. ISAPI filters are hook interfaces that allow the ability to look at EVERY request that comes into IIS and to modify the content or change the behavior of functionalities like Authentication. Incidentally ASP.NET maps ISAPI-like functionality via two concepts: Http Handlers (extensions) and Http Modules (filters). We’ll look at these later in more detail.

ISAPI is the initial code point that marks the beginning of an ASP.NET request. ASP.NET maps various extensions to its ISAPI extension which lives in the .NET Framework directory:

<.NET FrameworkDir>\aspnet_isapi.dll

You can interactively see these mapping in the IIS Service manager as shown in Figure 1. Look at the root of the Web Site and the Home Directory tab, then Configuration | Mappings.

Figure 1: IIS maps various extensions like .ASPX to the ASP.NET ISAPI extension. Through this mechanism requests are routed into ASP.NET's processing pipeline at the Web Server level.

You shouldn’t set these extensions manually as .NET requires a number of them. Instead use the aspnet_regiis.exe utility to make sure that all the various scriptmaps get registered properly:

cd <.NetFrameworkDirectory>

aspnet_regiis - i

This will register the particular version of the ASP.NET runtime for the entire Web site by registering the scriptmaps and setting up the client side scripting libraries used by the various controls for uplevel browsers. Note that it registers the particular version of the CLR that is installed in the above directory. Options on aspnet_regiis let you configure virtual directories individually. Each version of the .NET framework has its own version of aspnet_regiis and you need to run the appropriate one to register a site or virtual directory for a specific version of the .NET framework. Starting with ASP.NET 2.0, an IIS ASP.NET configuration page lets you pick the .NET version interactively in the IIS management console.

IIS 5 and 6 work differently

When a request comes in, IIS checks for the script map and routes the request to the aspnet_isapi.dll. The operation of the DLL and how it gets to the ASP.NET runtime varies significantly between IIS 5 and 6. Figure 2 shows a rough overview of the flow.

In IIS 5 hosts aspnet_isapi.dll directly in the inetinfo.exe process or one of its isolated worker processes if you have isolation set to medium or high for the Web or virtual directory. When the first ASP.NET request comes in the DLL will spawn a new process in another EXE – aspnet_wp.exe – and route processing to this spawned process. This process in turn loads and hosts the .NET runtime. Every request that comes into the ISAPI DLL then routes to this worker process via Named Pipe calls.

Figure 2 – Request flow from IIS to the ASP.NET Runtime and through the request processing pipeline from a high level. IIS 5 and IIS 6 interface with ASP.NET in different ways but the overall process once it reaches the ASP.NET Pipeline is the same.

IIS6, unlike previous servers, is fully optimized for ASP.NET

IIS 6 – Viva the Application Pool

IIS 6 changes the processing model significantly in that IIS no longer hosts any foreign executable code like ISAPI extensions directly. Instead IIS 6 always creates a separate worker process – an Application Pool – and all processing occurs inside of this process, including execution of the ISAPI dll. Application Pools are a big improvement for IIS 6, as they allow very granular control over what executes in a given process. Application Pools can be configured for every virtual directory or the entire Web site, so you can isolate every Web application easily into its own process that will be completely isolated from any other Web application running on the same machine. If one process dies it will not affect any others at least from the Web processing perspective.

In addition, Application Pools are highly configurable. You can configure their execution security environment by setting an execution impersonation level for the pool which allows you to customize the rights given to a Web application in that same granular fashion. One big improvement for ASP.NET is that the Application Pool replaces most of the ProcessModel entry in machine.config. This entry was difficult to manage in IIS 5, because the settings were global and could not be overridden in an application specific web.config file. When running IIS 6, the ProcessModel setting is mostly ignored and settings are instead read from the Application Pool. I say mostly – some settings, like the size of the ThreadPool and IO threads still are configured through this key since they have no equivalent in the Application Pool settings of the server.

Because Application Pools are external executables these executables can also be easily monitored and managed. IIS 6 provides a number of health checking, restarting and timeout options that can detect and in many cases correct problems with an application. Finally IIS 6’s Application Pools don’t rely on COM+ as IIS 5 isolation processes did which has improved performance and stability especially for applications that need to use COM objects internally.

Although IIS 6 application pools are separate EXEs, they are highly optimized for HTTP operations by directly communicating with a kernel mode HTTP.SYS driver. Incoming requests are directly routed to the appropriate application pool. InetInfo acts merely as an Administration and configuration service – most interaction actually occurs directly between HTTP.SYS and the Application Pools, all of which translates into a more stable and higher performance environment over IIS 5. This is especially true for static content and ASP.NET applications.

An IIS 6 application pool also has intrinsic knowledge of ASP.NET and ASP.NET can communicate with new low level APIs that allow direct access to the HTTP Cache APIs which can offload caching from the ASP.NET level directly into the Web Server’s cache.

In IIS 6, ISAPI extensions run in the Application Pool worker process. The .NET Runtime also runs in this same process, so communication between the ISAPI extension and the .NET runtime happens in-process which is inherently more efficient than the named pipe interface that IIS 5 must use. Although the IIS hosting models are very different the actual interfaces into managed code are very similar – only the process in getting the request routed varies a bit.

The ISAPIRuntime.ProcessRequest() method is the first entry point into ASP.NET

Getting into the .NET runtime

The actual entry points into the .NET Runtime occur through a number of undocumented classes and interfaces. Little is known about these interfaces outside of Microsoft, and Microsoft folks are not eager to talk about the details, as they deem this an implementation detail that has little effect on developers building applications with ASP.NET.

The worker processes ASPNET_WP.EXE (IIS5) and W3WP.EXE (IIS6) host the .NET runtime and the ISAPI DLL calls into small set of unmanged interfaces via low level COM that eventually forward calls to an instance subclass of the ISAPIRuntime class. The first entry point to the runtime is the undocumented ISAPIRuntime class which exposes the IISAPIRuntime interface via COM to a caller. These COM interfaces low level IUnknown based interfaces that are meant for internal calls from the ISAPI extension into ASP.NET. Figure 3 shows the interface and call signatures for the IISAPIRuntime interface as shown in Lutz Roeder’s excellent .NET Reflector tool (http://www.aisto.com/roeder/dotnet/). Reflector an assembly viewer and disassembler that makes it very easy to look at medadata and disassembled code (in IL, C#, VB) as shown in Figure 3. It’s a great way to explore the bootstrapping process.

Figure 3 – If you want to dig into the low level interfaces open up Reflector, and point at the System.Web.Hosting namespace. The entry point to ASP.NET occurs through a managed COM Interface called from the ISAPI dll, that receives an unmanaged pointer to the ISAPI ECB. The ECB contains has access to the full ISAPI interface to allow retrieving request data and sending back to IIS.

The IISAPIRuntime interface acts as the interface point between the unmanaged code coming from the ISAPI extension (directly in IIS 6 and indirectly via the Named Pipe handler in IIS 5). If you take a look at this class you’ll find a ProcessRequest method with a signature like this:

[return: MarshalAs(UnmanagedType.I4)]

int ProcessRequest([In] IntPtr ecb,

[In, MarshalAs(UnmanagedType.I4)] int useProcessModel);

The ecb parameter is the ISAPI Extension Control Block (ECB) which is passed as an unmanaged resource to ProcessRequest. The method then takes the ECB and uses it as the base input and output interface used with the Request and Response objects. An ISAPI ECB contains all low level request information including server variables, an input stream for form variables as well as an output stream that is used to write data back to the client. The single ecb reference basically provides access to all of the functionality an ISAPI request has access to and ProcessRequest is the entry and exit point where this resource initially makes contact with managed code.

The ISAPI extension runs requests asynchronously. In this mode the ISAPI extension immediately returns on the calling worker process or IIS thread, but keeps the ECB for the current request alive. The ECB then includes a mechanism for letting ISAPI know when the request is complete (via ecb.ServerSupportFunction) which then releases the ECB. This asynchronous processing releases the ISAPI worker thread immediately, and offloads processing to a separate thread that is managed by ASP.NET.

ASP.NET receives this ecb reference and uses it internally to retrieve information about the current request such as server variables, POST data as well as returning output back to the server. The ecb stays alive until the request finishes or times out in IIS and ASP.NET continues to communicate with it until the request is done. Output is written into the ISAPI output stream (ecb.WriteClient()) and when the request is done, the ISAPI extension is notified of request completion to let it know that the ECB can be freed. This implementation is very efficient as the .NET classes essentially act as a fairly thin wrapper around the high performance, unmanaged ISAPI ECB.

Loading .NET – somewhat of a mystery

Let’s back up one step here: I skipped over how the .NET runtime gets loaded. Here’s where things get a bit fuzzy. I haven’t found any documentation on this process and since we’re talking about native code there’s no easy way to disassemble the ISAPI DLL and figure it out.

My best guess is that the worker process bootstraps the .NET runtime from within the ISAPI extension on the first hit against an ASP.NET mapped extension. Once the runtime exists, the unmanaged code can request an instance of an ISAPIRuntime object for a given virtual path if one doesn’t exist yet. Each virtual directory gets its own AppDomain and within that AppDomain the ISAPIRuntime exists from which the bootstrapping process for an individual application starts. Instantiation appears to occur over COM as the interface methods are exposed as COM callable methods.

To create the ISAPIRuntime instance the System.Web.Hosting.AppDomainFactory.Create() method is called when the first request for a specific virtual directory is requested. This starts the ‘Application’ bootstrapping process. The call receives parameters for type and module name and virtual path information for the application which is used by ASP.NET to create an AppDomain and launch the ASP.NET application for the given virtual directory. This HttpRuntime derived object is created in a new AppDomain. Each virtual directory or ASP.NET application is hosted in a separate AppDomain and they get loaded only as requests hit the particular ASP.NET Application. The ISAPI extension manages these instances of the HttpRuntime objects, and routes inbound requests to the right one based on the virtual path of the request.

Figure 4 – The transfer of the ISAPI request into the HTTP Pipeline of ASP.NET uses a number of undocumented classes and interfaces and requires several factory method calls. Each Web Application/Virtual runs in its own AppDomain with the caller holding a reference to an IISAPIRuntime interface that triggers the ASP.NET request processing.

Back in the runtime

At this point we have an instance of ISAPIRuntime active and callable from the ISAPI extension. Once the runtime is up and running the ISAPI code calls into the ISAPIRuntime.ProcessRequest() method which is the real entry point into the ASP.NET Pipeline. The flow from there is shown in Figure 4.

Remember ISAPI is multi-threaded so requests will come in on multiple threads through the reference that was returned by ApplicationDomainFactory.Create(). Listing 1 shows the disassembled code from the IsapiRuntime.ProcessRequest method that receives an ISAPI ecb object and server type as parameters. The method is thread safe, so multiple ISAPI threads can safely call this single returned object instance simultaneously.

Listing 1: The Process request method receives an ISAPI Ecb and passes it on to the Worker request

public int ProcessRequest(IntPtr ecb, int iWRType)

{

HttpWorkerRequest request1 = ISAPIWorkerRequest.CreateWorkerRequest(ecb, iWRType);

string text1 = request1.GetAppPathTranslated();

string text2 = HttpRuntime.AppDomainAppPathInternal;

if (((text2 == null) || text1.Equals(".")) ||

(string.Compare(text1, text2, true, CultureInfo.InvariantCulture) == 0))

{

HttpRuntime.ProcessRequest(request1);

return 0;

}

HttpRuntime.ShutdownAppDomain("Physical application path changed from " +

text2 + " to " + text1);

return 1;

}

The actual code here is not important, and keep in mind that this is disassembled internal framework code that you’ll never deal with directly and that might change in the future. It’s meant to demonstrate what’s happening behind the scenes. ProcessRequest receives the unmanaged ECB reference and passes it on to the ISAPIWorkerRequest object which is in charge of creating the Request Context for the current request as shown in Listing 2.

The System.Web.Hosting.ISAPIWorkerRequest class is an abstract subclass of HttpWorkerRequest, whose job it is to create an abstracted view of the input and output that serves as the input for the Web application. Notice another factory method here: CreateWorkerRequest, which as a second parameter receives the type of worker request object to create. There are three different versions: ISAPIWorkerRequestInProc, ISAPIWorkerRequestInProcForIIS6, ISAPIWorkerRequestOutOfProc. This object is created on each incoming hit and serves as the basis for the Request and Response objects which will receive their data and streams from the data provided by the WorkerRequest.

The abstract HttpWorkerRequest class is meant to provide a highlevel abstraction around the low level interfaces so that regardless of where the data comes from, whether it’s a CGI Web Server, the Web Browser Control or some custom mechanism you use to feed the data to the HTTP Runtime. The key is that ASP.NET can retrieve the information consistently.

In the case of IIS the abstraction is centered around an ISAPI ECB block. In our request processing, ISAPIWorkerRequest hangs on to the ISAPI ECB and retrieves data from it as needed. Listing 2 shows how the query string value is retrieved for example.

Listing 2: An ISAPIWorkerRequest method that uses the unmanged

// *** Implemented in ISAPIWorkerRequest

public override byte[] GetQueryStringRawBytes()

{

byte[] buffer1 = new byte[this._queryStringLength];

if (this._queryStringLength > 0)

{

int num1 = this.GetQueryStringRawBytesCore(buffer1, this._queryStringLength);

if (num1 != 1)

{

throw new HttpException( "Cannot_get_query_string_bytes");

}

return buffer1;

}

// *** Implemented in a specific implementation class ISAPIWorkerRequestInProcIIS6

internal override int GetQueryStringCore(int encode, StringBuilder buffer, int size)

{

if (this._ecb == IntPtr.Zero)

{

return 0;

}

return UnsafeNativeMethods.EcbGetQueryString(this._ecb, encode, buffer, size);

}

ISAPIWorkerRequest implements a high level wrapper method, that calls into lower level Core methods, which are responsible for performing the actual access to the unmanaged APIs – or the ‘service level implementation’. The Core methods are implemented in the specific ISAPIWorkerRequest instance subclasses and thus provide the specific implementation for the environment that it’s hosted in. This makes for an easily pluggable environment where additional implementation classes can be provided later as newer Web Server interfaces or other platforms are targeted by ASP.NET. There’s also a helper class System.Web.UnsafeNativeMethods. Many of these methods operate on the ISAPI ECB structure performing unmanaged calls into the ISAPI extension.

HttpRuntime, HttpContext, and HttpApplication – Oh my

When a request hits, it is routed to the ISAPIRuntime.ProcessRequest() method. This method in turn calls HttpRuntime.ProcessRequest that does several important things (look at System.Web.HttpRuntime.ProcessRequestInternal with Reflector):

Create a new HttpContext instance for the request

Retrieves an HttpApplication Instance

Calls HttpApplication.Init() to set up Pipeline Events

Init() fires HttpApplication.ResumeProcessing() which starts the ASP.NET pipeline processing

First a new HttpContext object is created and it is passed the ISAPIWorkerRequest that wrappers the ISAPI ECB. The Context is available throughout the lifetime of the request and ALWAYS accessible via the static HttpContext.Current property. As the name implies, the HttpContext object represents the context of the currently active request as it contains references to all of the vital objects you typically access during the request lifetime: Request, Response, Application, Server, Cache. At any time during request processing HttpContext.Current gives you access to all of these object.

The HttpContext object also contains a very useful Items collection that you can use to store data that is request specific. The context object gets created at the begging of the request cycle and released when the request finishes, so data stored there in the Items collection is specific only to the current request. A good example use is a request logging mechanism where you want to track start and end times of a request by hooking the Application_BeginRequest and Application_EndRequest methods in Global.asax as shown in Listing 3. HttpContext is your friend – you’ll use it liberally if you need data in different parts of the request or page processing.

Listing 3 – Using the HttpContext.Items collection lets you save data between pipeline events

protected void Application_BeginRequest(Object sender, EventArgs e)

{

//*** Request Logging

if (App.Configuration.LogWebRequests)

Context.Items.Add("WebLog_StartTime",DateTime.Now);

}

protected void Application_EndRequest(Object sender, EventArgs e)

{

// *** Request Logging

if (App.Configuration.LogWebRequests)

{

try

{

TimeSpan Span = DateTime.Now.Subtract(

(DateTime) Context.Items["WebLog_StartTime"] );

int MiliSecs = Span.TotalMilliseconds;

// do your logging

WebRequestLog.Log(App.Configuration.ConnectionString,

true,MilliSecs);

}

Once the Context has been set up, ASP.NET needs to route your incoming request to the appropriate application/virtual directory by way of an HttpApplication object. Every ASP.NET application must be set up as a Virtual (or Web Root) directory and each of these ‘applications’ are handled independently.

The HttpApplication is like a master of ceremonies – it is where the processing action starts

Master of your domain: HttpApplication

Each request is routed to an HttpApplication object. The HttpApplicationFactory class creates a pool of HttpApplication objects for your ASP.NET application depending on the load on the application and hands out references for each incoming request. The size of the pool is limited to the setting of the MaxWorkerThreads setting in machine.config’s ProcessModel Key, which by default is 20.

The pool starts out with a smaller number though; usually one and it then grows as multiple simulataneous requests need to be processed. The Pool is monitored so under load it may grow to its max number of instances, which is later scaled back to a smaller number as the load drops.

HttpApplication is the outer container for your specific Web application and it maps to the class that is defined in Global.asax. It’s the first entry point into the HTTP Runtime that you actually see on a regular basis in your applications. If you look in Global.asax (or the code behind class) you’ll find that this class derives directly from HttpApplication:

public class Global : System.Web.HttpApplication

HttpApplication’s primary purpose is to act as the event controller of the Http Pipeline and so its interface consists primarily of events. The event hooks are extensive and include:

BeginRequest

AuthenticateRequest

AuthorizeRequest

ResolveRequestCache

AquireRequestState

PreRequestHandlerExecute

…Handler Execution…

PostRequestHandlerExecute

ReleaseRequestState

UpdateRequestCache

EndRequest

Each of these events are also implemented in the Global.asax file via empty methods that start with an Application_ prefix. For example, Application_BeginRequest(), Application_AuthorizeRequest(). These handlers are provided for convenience since they are frequently used in applications and make it so that you don’t have to explicitly create the event handler delegates.

It’s important to understand that each ASP.NET virtual application runs in its own AppDomain and that there inside of the AppDomain multiple HttpApplication instances running simultaneously, fed out of a pool that ASP.NET manages. This is so that multiple requests can process at the same time without interfering with each other.

To see the relationship between the AppDomain, Threads and the HttpApplication check out the code in Listing 4.

Listing 4 – Showing the relation between AppDomain, Threads and HttpApplication instances

private void Page_Load(object sender, System.EventArgs e)

{

// Put user code to initialize the page here

this.ApplicationId = ((HowAspNetWorks.Global)

HttpContext.Current.ApplicationInstance).ApplicationId ;

this.ThreadId = AppDomain.GetCurrentThreadId();

this.DomainId = AppDomain.CurrentDomain.FriendlyName;

this.ThreadInfo = "ThreadPool Thread: " +

System.Threading.Thread.CurrentThread.IsThreadPoolThread.ToString() +

"

Thread Apartment: " +

System.Threading.Thread.CurrentThread.ApartmentState.ToString();

// *** Simulate a slow request so we can see multiple

// requests side by side.

System.Threading.Thread.Sleep(3000);

}

This is part of a demo is provided with your samples and the running form is shown in Figure 5. To check this out run two instances of a browser and hit this sample page and watch the various Ids.

Figure 5 – You can easily check out how AppDomains, Application Pool instances, and Request Threads interact with each other by running a couple of browser instances simultaneously. When multiple requests fire you’ll see the thread and Application ids change, but the AppDomain staying the same.

You’ll notice that the AppDomain ID stays steady while thread and HttpApplication Ids change on most requests, although they likely will repeat. HttpApplications are running out of a collection and are reused for subsequent requests so the ids repeat at times. Note though that Application instance are not tied to a specific thread – rather they are assigned to the active executing thread of the current request.

Threads are served from the .NET ThreadPool and by default are Multithreaded Apartment (MTA) style threads. You can override this apartment state in ASP.NET pages with the ASPCOMPAT="true" attribute in the @Page directive. ASPCOMPAT is meant to provide COM components a safe environment to run in and ASPCOMPAT uses special Single Threaded Apartment (STA) threads to service those requests. STA threads are set aside and pooled separately as they require special handling.

The fact that these HttpApplication objects are all running in the same AppDomain is very important. This is how ASP.NET can guarantee that changes to web.config or individual ASP.NET pages get recognized throughout the AppDomain. Making a change to a value in web.config causes the AppDomain to be shut down and restarted. This makes sure that all instances of HttpApplication see the changes made because when the AppDomain reloads the changes from ASP.NET are re-read at startup. Any static references are also reloaded when the AppDomain so if the application reads values from App Configuration settings these values also get refreshed.

To see this in the sample, hit the ApplicationPoolsAndThreads.aspx page and note the AppDomain Id. Then go in and make a change in web.config (add a space and save). Then reload the page. You’ll l find that a new AppDomain has been created.

In essence the Web Application/Virtual completely ‘restarts’ when this happens. Any requests that are already in the pipeline processing will continue running through the existing pipeline, while any new requests coming in are routed to the new AppDomain. In order to deal with ‘hung requests’ ASP.NET forcefully shuts down the AppDomain after the request timeout period is up even if requests are still pending. So it’s actually possible that two AppDomains exist for the same HttpApplication at a given point in time as the old one’s shutting down and the new one is ramping up. Both AppDomains continue to serve their clients until the old one has run out its pending requests and shuts down leaving just the new AppDomain running.

Flowing through the ASP.NET Pipeline

The HttpApplication is responsible for the request flow by firing events that signal your application that things are happening. This occurs as part of the HttpApplication.Init() method (look at System.Web.HttpApplication.InitInternal and HttpApplication.ResumeSteps() with Reflector) which sets up and starts a series of events in succession including the call to execute any handlers. The event handlers map to the events that are automatically set up in global.asax, and they also map any attached HTTPModules, which are essentially an externalized event sink for the events that HttpApplication publishes.

Both HttpModules and HttpHandlersare loaded dynamically via entries in Web.config and attached to the event chain. HttpModules are actual event handlers that hook specific HttpApplication events, while HttpHandlers are an end point that gets called to handle ‘application level request processing’.

Both Modules and Handlers are loaded and attached to the call chain as part of the HttpApplication.Init() method call. Figure 6 shows the various events and when they happen and which parts of the pipeline they affect.

Figure 6 – Events flowing through the ASP.NET HTTP Pipeline. The HttpApplication object’s events drive requests through the pipeline. Http Modules can intercept these events and override or enhance existing functionality.

HttpContext, HttpModules and HttpHandlers

The HttpApplication itself knows nothing about the data being sent to the application – it is a merely messaging object that communicates via events. It fires events and passes information via the HttpContext object to the called methods. The actual state data for the current request is maintained in the HttpContext object mentioned earlier. It provides all the request specific data and follows each request from beginning to end through the pipeline. Figure 7 shows the flow through ASP.NET pipeline. Notice the Context object which is your compadre from beginning to end of the request and can be used to store information in one event method and retrieve it in a later event method.

Once the pipeline is started, HttpApplication starts firing events one by one as shown in Figure 6. Each of the event handlers is fired and if events are hooked up those handlers execute and perform their tasks. The main purpose of this process is to eventually call the HttpHandler hooked up to a specific request. Handlers are the core processing mechanism for ASP.NET requests and usually the place where any application level code is executed. Remember that the ASP.NET Page and Web Service frameworks are implemented as HTTPHandlers and that’s where all the core processing of the request is handled. Modules tend to be of a more core nature used to prepare or post process the Context that is delivered to the handler. Typical default handlers in ASP.NET are Authentication, Caching for pre-processing and various encoding mechanisms on post processing.

There’s plenty of information available on HttpHandlers and HttpModules so to keep this article a reasonable length I’m going to provide only a brief overview of handlers.

HttpModules

As requests move through the pipeline a number of events fire on the HttpApplication object. We’ve already seen that these events are published as event methods in Global.asax. This approach is application specific though which is not always what you want. If you want to build generic HttpApplication event hooks that can be plugged into any Web applications you can use HttpModules which are reusable and don’t require application specific code except for an entry in web.config.

Modules are in essence filters – similar in functionality to ISAPI filters at the ASP.NET request level. Modules allow hooking events for EVERY request that pass through the ASP.NET HttpApplication object. These modules are stored as classes in external assemblies that are configured in web.config and loaded when the Application starts. By implementing specific interfaces and methods the module then gets hooked up to the HttpApplication event chain. Multiple HttpModules can hook the same event and event ordering is determined by the order they are declared in Web.config. Here’s what a handler definition looks like in Web.config:

<system.web>

<add name= "BasicAuthModule"

type="HttpHandlers.BasicAuth,WebStore" />

httpModules>

system.web>

configuration>

Note that you need to specify a full typename and an assembly name without the DLL extension.

Modules allow you look at each incoming Web request and perform an action based on the events that fire. Modules are great to modify request or response content, to provide custom authentication or otherwise provide pre or post processing to every request that occurs against ASP.NET in a particular application. Many of ASP.NET’s features like the Authentication and Session engines are implemented as HTTP Modules.

While HttpModules feel similar to ISAPI Filters in that they look at every request in that comes through an ASP.NET Application, they are limited to looking at requests mapped to a single specific ASP.NET application or virtual directory and then only against requests that are mapped to ASP.NET. Thus you can look at all ASPX pages or any of the other custom extensions that are mapped to this application. You cannot however look at standard .HTM or image files unless you explicitly map the extension to the ASP.NET ISAPI dll by adding an extension as shown in Figure 1. A common use for a module might be to filter content to JPG images in a special folder and display a ‘SAMPLE’ overlay ontop of every image by drawing ontop of the returned bitmap with GDI+.

Implementing an HTTP Module is very easy: You must implement the IHttpModule interface which contains only two methods Init() and Dispose(). The event parameters passed include a reference to the HTTPApplication object, which in turn gives you access to the HttpContext object. In these methods you hook up to HttpApplication events. For example, if you want to hook the AuthenticateRequest event with a module you would do what’s shown in Listing 5.

Listing 5: The basics of an HTTP Module are very simple to implement

public class BasicAuthCustomModule : IHttpModule

{

public void Init(HttpApplication application)

{

// *** Hook up any HttpApplication events

application.AuthenticateRequest +=

new EventHandler(this.OnAuthenticateRequest);

}

public void Dispose() { }

public void OnAuthenticateRequest(object source, EventArgs eventArgs)

{

HttpApplication app = (HttpApplication) source;

HttpContext Context = HttpContext.Current;

… do what you have to do… }

}

Remember that your Module has access the HttpContext object and from there to all the other intrinsic ASP.NET pipeline objects like Response and Request, so you can retrieve input etc. But keep in mind that certain things may not be available until later in the chain.

You can hook multiple events in the Init() method so your module can manage multiple functionally different operations in one module. However, it’s probably cleaner to separate differing logic out into separate classes to make sure the module is modular. In many cases functionality that you implement may require that you hook multiple events – for example a logging filter might log the start time of a request in Begin Request and then write the request completion into the log in EndRequest.

Watch out for one important gotcha with HttpModules and HttpApplication events: Response.End() or HttpApplication.CompleteRequest() will shortcut the HttpApplication and Module event chain. See the sidebar “Watch out for Response.End() “ for more info.

HttpHandlers

Modules are fairly low level and fire against every inbound request to the ASP.NET application. Http Handlers are more focused and operate on a specific request mapping, usually a page extension that is mapped to the handler.

Http Handler implementations are very basic in their requirements, but through access of the HttpContext object a lot of power is available. Http Handlers are implemented through a very simple IHttpHandler interface (or its asynchronous cousin, IHttpAsyncHandler) which consists of merely a single method – ProcessRequest() – and a single property IsReusable. The key is ProcessRequest() which gets passed an instance of the HttpContext object. This single method is responsible for handling a Web request start to finish.

Single, simple method? Must be too simple, right? Well, simple interface, but not simplistic in what’s possible! Remember that WebForms and WebServices are both implemented as Http Handlers, so there’s a lot of power wrapped up in this seemingly simplistic interface. The key is the fact that by the time an Http Handler is reached all of ASP.NET’s internal objects are set up and configured to start processing of requests. The key is the HttpContext object, which provides all of the relevant request functionality to retireve input and send output back to the Web Server.

For an HTTP Handler all action occurs through this single call to ProcessRequest(). This can be as simple as:

public void ProcessRequest(HttpContext context)

{

context.Response.Write("Hello World");

}

to a full implementation like the WebForms Page engine that can render complex forms from HTML templates. The point is that it’s up to you to decide of what you want to do with this simple, but powerful interface!

Because the Context object is available to you, you get access to the Request, Response, Session and Cache objects, so you have all the key features of an ASP.NET request at your disposal to figure out what users submitted and return content you generate back to the client. Remember the Context object – it’s your friend throughout the lifetime of an ASP.NET request!

The key operation of the handler should be eventually write output into the Respone object or more specifically the Response object’s OutputStream. This output is what actually gets sent back to the client. Behind the scenes the ISAPIWorkerRequest manages sending the OutputStream back into the ISAPI ecb.WriteClient method that actually performs the IIS output generation.

Figure 7 – The ASP.NET Request pipeline flows requests through a set of event interfaces that provide much flexibility. The Application acts as the hosting container that loads up the Web application and fires events as requests come in and pass through the pipeline. Each request follows a common path through the Http Filters and Modules configured. Filters can examine each request going through the pipeline and Handlers allow implementation of application logic or application level interfaces like Web Forms and Web Services. To provide Input and Output for the application the Context object provides request specific information throughout the entire process.

WebForms implements an Http Handler with a much more high level interface on top of this very basic framework, but eventually a WebForm’s Render() method simply ends up using an HtmlTextWriter object to write its final final output to the context.Response.OutputStream. So while very fancy, ultimately even a high level tool like Web forms is just a high level abstraction ontop of the Request and Response object.

You might wonder at this point whether you need to deal with Http Handlers at all. After all WebForms provides an easily accessible Http Handler implementation, so why bother with something a lot more low level and give up that flexibility?

WebForms are great for generating complex HTML pages and business level logic that requires graphical layout tools and template backed pages. But the WebForms engine performs a lot of tasks that are overhead intensive. If all you want to do is read a file from the system and return it back through code it’s much more efficient to bypass the Web Forms Page framework and directly feed the file back. If you do things like Image Serving from a Database there’s no need to go into the Page framework – you don’t need templates and there surely is no Web UI that requires you to capture events off an Image served.

There’s no reason to set up a page object and session and hook up Page level events – all of that stuff requires execution of code that has nothing to do with your task at hand.

So handlers are more efficient. Handlers also can do things that aren’t possible with WebForms such as the ability to process requests without the need to have a physical file on disk, which is known as a virtual Url. To do this make sure you turn off ‘Check that file exists’ checkbox in the Application Extension dialog shown in Figure 1.

This is common for content providers, such as dynamic image processing, XML servers, URL Redirectors providing vanity Urls, download managers and the like, none of which would benefit from the WebForm engine.

Have I stooped low enough for you?

Phew – we’ve come full circle here for the processing cycle of requests. That’s a lot of low level information and I haven’t even gone into great detail about how HTTP Modules and HTTP Handlers work. It took some time to dig up this information and I hope this gives you some of the same satisfaction it gave me in understanding how ASP.NET works under the covers.

Before I’m done let’s do the quick review of the event sequences I’ve discussed in this article from IIS to handler:

IIS gets the request

Looks up a script map extension and maps to aspnet_isapi.dll

Code hits the worker process (aspnet_wp.exe in IIS5 or w3wp.exe in IIS6)

.NET runtime is loaded

IsapiRuntime.ProcessRequest() called by non-managed code

IsapiWorkerRequest created once per request

HttpRuntime.ProcessRequest() called with Worker Request

HttpContext Object created by passing Worker Request as input

HttpApplication.GetApplicationInstance() called with Context to retrieve instance from pool

HttpApplication.Init() called to start pipeline event sequence and hook up modules and handlers

HttpApplicaton.ProcessRequest called to start processing

Pipeline events fire

Handlers are called and ProcessRequest method are fired

Control returns to pipeline and post request events fire

It’s a lot easier to remember how all of the pieces fit together with this simple list handy. I look at it from time to time to remember. So now, get back to work and do something non-abstract…

Although what I discuss here is based on ASP.NET 1.1, it looks that the underlying processes described here haven’t changed in ASP.NET 2.0.

Working with Complex Data Types in an XML Web Service

2006-02-24T14:04:00.000+07:00

XML Web Services enable the exchange of complex data types, serialized as XML. Complex data types, such as ADO.NET DataSets and custom classes can be serialized as XML and either sent to the XML Web Service as an input argument, or returned from the XML Web Service as the result. In this article we will build the beginning of an XML Web Service, which we will finish in next week's article, when we will also build a consumer Web application. The XML Web Service will have WebMethods for returning a DataSet and custom classes serialized as XML. For the DataSet, the .NET Framework will handle the formatting of the XML document that represents the DataSet; for the custom classes we will use classes from the System.Xml.Serialization namespace to define the XML format. The information page for the XML Web Service

Working with DataSets

A DataSet can be used with an XML Web Service, and the .NET Framework will automatically handle serializing it as XML. Listing 4.1 shows a serialized DataSet (DataSetName="Northwind") that has one DataTable (TableName="Categories"). The DataTable has two columns, CategoryID (int) and CategoryName (string). The DataTable lists all of the categories in the Northwind Categories table.

Listing 4.1

1
Beverages

2
Condiments

3
Confections

4
Dairy Products

5
Grains/Cereals

6
Meat/Poultry

7
Produce

8
Seafood

In Listing 4.1 we see how the .NET Framework serializes a DataSet when it is used with an XML Web Service. We can see that the DataSet uses the XML namespace (xmlns) http://www.dotnetjunkies.com. This is used to help uniquely identify this DataSet - the XML namespace is defined in the XML Web Service. When the DataSet is serialized, the XML namespace of the XML Web Service is assigned to the DataSet.

In the XML Schema in Listing 4.1 the DataTable is defined and each column is defined, with its data type. Each record in the DataTable is serialized to a element, with child elements for each of the columns.

To generate the XML document shown in Listing 4.1, we create an XML Web Service that returns the defined DataSet. This is shown in Listing 4.2.

Listing 4.2
[ProductServices.asmx]
<%@ WebService Class="XMLWebServices.C04.ProductServices" %>

[ProductServices.asmx.cs]
using System;
using System.Data;
using System.Web.Services;

namespace XMLWebServices.C04
{
[WebService( Namespace="http://www.dotnetjunkies.com" )]
public class ProductServices : WebService
{
[WebMethod(Description="Returns a DataSet with one DataTable (Categories) of all Categories and CategoryIDs in the Northwind Categories table.")]
public DataSet GetCategories()
{
XMLWebServices.Data.NorthwindDB _nwdDB = new XMLWebServices.Data.NorthwindDB();
return _nwdDB.getCategories();
}
}
}

In Listing 4.2 we create the beginning of the ProductServices XML Web Service. In the [WebService()] attribute we define the namespace for the XML Web Service. This namespace is used as the xmlns attribute of the DataSet when it is serialized as XML.

So far we only have one WebMethod, the GetCategories() method. This method returns the DataSet shown in Listing 4.1. Invoking a method on the NorthwindDB data access class does this. The NorthwindDB class is shown in Listing 4.3.

Listing 4.3
[NorthwindDB.cs]
using System;
using System.Data;
using System.Data.SqlClient;
using System.Text;
using System.Configuration;

namespace XMLWebServices.Data
{
///
/// Data Access Layer for Northwind Database
///
public class NorthwindDB
{
public NorthwindDB(){}

// Create a class-level private connection string object
private String _conString = ConfigurationSettings.AppSettings["NWDConString"];

public DataSet getCategories()
{
// Build the SQL statement
StringBuilder _sql = new StringBuilder();
_sql.Append ( "SELECT CategoryID, CategoryName " );
_sql.Append ( "FROM Categories " );
_sql.Append ( "ORDER BY CategoryName" );
// Create the DataSet to return
DataSet _ds = new DataSet ( "Northwind" );
// Create a Data Adapter to fill the DataSet
SqlDataAdapter _sda = new SqlDataAdapter ( _sql.ToString(), _conString );
// Fill the DataSet and Return it
_sda.Fill ( _ds, "Categories" );
return _ds;
}
}
}

In the NorthwindDB data access class we expose a method, getCategories(), which connects to our SQL Server database and returns a DataSet with one DataTable populated with the CategoryID and CategoryName of all the records in the Categories table in the Northwind database.

The DataSet is passed back to the calling component, the ProductServices class, in the GetCategories() WebMethod, which in turn passes it back to the XML Web Service consumer, serialized as XML. We didn't have to do anything to serialize the DataSet as XML - the .NET Framework does that for us.

Unfortunately, there is no way for us to override how the DataSet is serialized. To create custom XML output we can use the System.Xml.Serialization namespace, and serialize custom classes.

Serializing Custom Classes

We can create a custom class to represent our data, and serialize it as XML to be returned to the XML Web Service consumer. We do this using attribute classes from the System.Xml.Serialization namespace. Some of the XML attribute classes are:

* XmlElementAttribute - Indicates that a public field or property represents an XML element - when the XmlSerializer serializes or deserializes the containing object.
* XmlAttributeAttribute - Specifies that the XmlSerializer should serialize the class member as an XML attribute.
* XmlTextAttribute - Indicates to the XmlSerializer that the member should be treated as XML text when the containing class is serialized or deserialized.
* XmlArrayAttribute - Specifies that the XmlSerializer should serialize a particular class member as an array of XML elements.

Descriptions of all of the XML attribute classes can be found in the .NET Framework SDK documentation at
ms-help://MS.NETFrameworkSDK/cpref/html/frlrfSystemXmlSerialization.htm

The XML attribute classes are used to control how the XmlSerializer serializes and deserializes an object. We can apply these classes to a public field or property, and that informs the XmlSerializer what to do with the object's member.

Using the XML Attribute Classes

We can create a custom class to represent our data - for example a Product class to represent a single product - and use the XML attribute classes to define how a class instance should be serialized as XML.

Lets create a Product class. We will use this in an XML Web Service that will return an instance of the Product class. We will define the XML output to look like the XML document shown in Listing 4.4.

Listing 4.4

xmlns="http://www.dotnetjunkies.com">
ChaiProductName>

39InStock>
0OnOrder>
10ReorderLevel>
ProductStockInformation>
Product>

In Listing 4.4 is the XML representation of a product from the Northwind database. The parent element has two child elements, and .

In the element there is an ID attribute that holds the ProductID value from the Products table - the ProductName value is the text value of the element.

The element has two attributes, Cost and QuantityPerUnit, which are UnitCost and QuantityPerUnit from the Products table, respectively. The element has three child elements, (UnitsInStock), (UnitsOnOrder), and (ReorderLevel).

Listing 4.5 shows the code for the Product, ProductInfo, and StockInfo classes.

Listing 4.5
[Product.cs]
using System;
using System.Data;
using System.Data.SqlClient;
using System.Xml.Serialization;

namespace XMLWebServices.C04
{
///
/// Represents a single product from the Northwind Database
///
public class Product
{
[XmlElement(ElementName="ProductName")]
public ProductInfo ProductInformation = new ProductInfo();
[XmlElement(ElementName="ProductStockInformation")]
public StockInfo ProductStockInformation = new StockInfo();
}

public class ProductInfo
{
[XmlAttribute(DataType="int", AttributeName="ID")]
public Int32 ProductID;
[XmlText()]
public String ProductName;
}

public class StockInfo
{
[XmlElement(DataType="short", ElementName="InStock")]
public Int16 UnitsInStock;
[XmlAttribute(DataType="decimal", AttributeName="Cost")]
public Decimal UnitPrice;
[XmlAttribute(DataType="string", AttributeName="QuantityPerUnit")]
public String QuantityPerUnit;
[XmlElement(DataType="short", ElementName="OnOrder")]
public Int16 UnitsOnOrder;
[XmlElement(DataType="short", ElementName="ReorderLevel")]
public Int16 ReorderLevel;
}
}

In Listing 4.5 we create the Product class. The Product class has two public fields, ProductInformation - which uses the ProductInfo data type - and ProductStockInformation - which uses the StockInfo data type.

Each of the Product public fields have the [XmlElement()] modifier applied to them (the Attribute portion of the XmlWhateverAttribute name is optional). This indicates to the XmlSerializer that these fields should be serialized as XML elements. Since each field uses a custom data type, we can go on to define how the public fields and properties in those classes are serialized. With each [XmlElement()] modifier we define what the name of the element should be when it is serialized. The ProductInformation field will be serialized as ; the ProductStockInformation field will be serialized using the same name as the field. In the case of the ProductStockInformation field, we do not have to define the Name property of the [XmlElement()] modifier, but I did so to be very explicit.

In the ProductInfo class we define two public fields, ProductID (Int32) and ProductName (String). We apply the [XmlAttribute()] class to the ProductID field, indicting this should be serialized as an attribute of the element, using the name "ID" and it should be serialized as an int XML data type. The ProductName field uses the [XmlText()] modifier. This indicates that the value of this field should be serialized as the text of the element.

Based on the settings of the Product.ProductInformation field, and the ProductInfo class, we have defined the following XML elements:

ChaiProductName>
Product>

We use the [XmlElement()] and [XmlAttribute()] modifiers in the same manner to create the XML output for the StockInfo class. The resulting XML is:

39InStock>
0OnOrder>
10ReorderLevel>
ProductStockInformation>
Product>

We can create a WebMethod to return a Product class instance - this is shown in Listing 4.6 (this should be added to the ProductServices XML Web Service).

Listing 4.6
[WebMethod(Description="Returns a custom class, Product, which represents a single Product from the Northwind Products table.")]
public Product GetProductByID ( Int32 ProductID )
{
XMLWebServices.Data.NorthwindDB _nwdDB = new XMLWebServices.Data.NorthwindDB();
return _nwdDB.getProductByID ( ProductID );
}

The GetProductByID() WebMethod takes in a ProductID and returns an instance of the Product class that represents the specified product. The Product class instance is created in the NortwindDB data access component, as shown in Listing 4.7 (this method should be added to the NorthwindDB class).

Listing 4.7
public XMLWebServices.C04.Product getProductByID ( Int32 _productID )
{
// Build the SQL statement
StringBuilder _sql = new StringBuilder();
_sql.Append ( "SELECT ProductName, UnitsInStock, UnitPrice, " );
_sql.Append ( "QuantityPerUnit, UnitsOnOrder, ReorderLevel " );
_sql.Append ( "FROM Products " );
_sql.Append ( "WHERE ProductID=" );
_sql.Append ( _productID.ToString() );
// Create the SqlConnection and SqlCommand objects
SqlConnection _con = new SqlConnection ( _conString );
SqlCommand _cmd = new SqlCommand ( _sql.ToString(), _con );
// Create a Product object
XMLWebServices.C04.Product _product = new XMLWebServices.C04.Product();
// Open the connection and return the IDataReader object
try
{
_con.Open();
// Use a data reader to return the result set
SqlDataReader _rdr = _cmd.ExecuteReader();
// Open the reader to the first record
_rdr.Read();
// Assign the product properties
_product.ProductInformation.ProductID = _productID;
_product.ProductInformation.ProductName = _rdr.GetString(0);
_product.ProductStockInformation.UnitsInStock = (Int16)_rdr.GetValue(1);
_product.ProductStockInformation.UnitPrice = (Decimal)_rdr.GetValue(2);
_product.ProductStockInformation.QuantityPerUnit = _rdr.GetString(3);
_product.ProductStockInformation.UnitsOnOrder = (Int16)_rdr.GetValue(4);
_product.ProductStockInformation.ReorderLevel = (Int16)_rdr.GetValue(5);
// Close the data reader and the connection
_rdr.Close();
_con.Close();
//Return the product object
return _product;
}
catch
{
// If there is an error, return null
return null;
}
}

In Listing 4.7 we create the getProductByID() method of the data access class. This method takes in a ProductID, which it uses to retrieve a record from the Northwind database, Products table. We create an instance of the Product class and set its public field values using the column values in the returned record. The result is a Product class instance that we return to the calling object - the ProductServices XML Web Service's GetProductByID() method - which in turn returns it to the consumer, serialized as XML.

Summary

In next weeks article I will show you how to use the XmlArrayAtribute class by creating a Category class and exposing a property that is an array of Product objects. We will also build a consumer application for the ProductServices XML Web Service and look at how we can consume the XML serialized object that we have created.

Rich Internet Applications - RIA everywhere

2006-01-18T15:49:00.000+07:00

Rich Internet Applications and AJAX - Selecting the best product

There are hundreds of criteria for evaluating RIA and AJAX products. So many that it’s easy to lose focus and misjudge priorities. This essay proposes a decision tree that leads through an evaluation process. It asks for the most distinctive requirements and product features in a top-down sequence, discussing the essential differences between technology options.

Figure 1: Decision Tree

The focus is entirely on questions you should answer when evaluating RIA products. There is no assessment of specific products, because the individual features of products tend to obscure fundamental issues. Notice that RIA and AJAX are sometimes used as synonyms. This is not entirely correct: AJAX is short for “Asynchronous JavaScript And XML”, which essentially limits the term to the set of RIA solutions based on JavaScript. I will adhere to this latter definition, although I fully agree with authors who argue that the concepts proposed by AJAX are by no means limited to JavaScript [1].
Simple User Interface?

The first and most important question to ask is whether the user interface (UI) of your application is simple enough for HTML. If the answer is yes, then HTML is your best option because it enables ubiquitous end user access via browser.

Simple enough for HTML means that the UI has modest interactivity requirements. However, if any of the following features improves your UI, you should consider RIA technology:

* Partial screen updates
* Asynchronous communication
* Server push
* Widgets supporting direct manipulation
* Multiple coordinated windows
* Modal dialogs
* Menus
* Keyboard navigation

RIA technology provides rich client capabilities in a web infrastructure. The goal is to combine the advantages of desktop applications with those of web applications. There are three fundamentally different technology options to achieve this: JavaScript, Java, and Flash. Their respective core advantages lead to the next level of the decision tree.
Ubiquity, Industrial Strength, or Fancy Animations?

Figure 2: RIA Technology Subtree

The question that differentiates RIA technology is whether unconditional end user access via browser (ubiquity), industrial strength, or fancy animations are most important. If ubiquity is the answer, a JavaScript/AJAX based solution will be the favorite, because JavaScript is available “out-of-the-box” in every popular browser. If industrial strength characteristics like maintainability, reliability, availability, scalability, performance and security are the top requirements, a Java solution is likely to be the front-runner. Java is unquestionably superior to the scripting alternatives in these areas. Last but not least, if fancy animations are your primary requirement, the attractiveness of Flash may outweigh the arguments speaking for the alternatives. The following sections look closer at these three technology options and present questions you should ask when assessing corresponding products.
Ubiquity: JavaScript/AJAX

Figure 3: JavaScript/AJAX Branch

AJAX is neither a product nor a new technology but a new branding for RIA technology that is based on JavaScript, XML, and other technologies. Since most JavaScript based products have adopted the AJAX terminology, it’s reasonable to put them in a single category. JavaScript/AJAX products are appropriate in situations where the user expects an application to work unconditionally in browsers. When buying a DVD or booking a flight, for instance, nobody will want to install a plug-in.

Plain HTML remains the best solution for simple UI requirements. Yet, DVD stores or travel booking applications with simple UIs are often vexing or even useless. Anyone who has tried to book an itinerary with multiple flights, hotels, and rental cars will agree that current websites are painful to use. JavaScript/AJAX could help here. Unfortunately, there’s only a handful of “killer” applications that show what’s possible: Google Suggest and Google Maps are among them. Cited as the key examples by Jesse James Garrett in his defining article on AJAX [2], they are arguably the most popular JavaScript based RIAs on the web.

Selecting RIA products based on JavaScript/AJAX is challenging. The issue is that JavaScript is hard to deal with. It’s no coincidence that it took the skills and funds of Google to come up with “killer” applications. Internet giants like Amazon or Ebay are not leveraging the potential of JavaScript/AJAX, although the technology has been around for years and would potentially improve their websites substantially. Just imagine how their sites would profit by “search as you type”, scrolling lists, partial screen updates, or browseable category trees.

The following questions will help in deciding whether an RIA product based on JavaScript/AJAX is suitable for your needs.
Browsers and OS supported?

JavaScript implementations are not standardized across browsers and operating systems. This means that programs must be aware of the browser, the operating system, and their respective versions. Programs must execute different code depending on specific combinations thereof, which is a developer’s nightmare. The most essential characteristic of a JavaScript/AJAX product is, therefore, how well it handles these combinations and shields the developer from the complexity of managing them in application code. The combinations supported by the Google applications may serve as a benchmark for evaluation. Google Maps and Google Suggest support:

* IE 6.0+ on Windows
* Firefox 0.8+ on Windows, Mac, Linux
* Safari 1.2.4+ on Mac (Google Suggest 1.2.2 or newer)
* Netscape 7.1+ on Windows, Mac, Linux
* Mozilla 1.4+ on Windows, Mac, Linux
* Opera 8+ on Windows, Mac, Linux (Google Suggest 7.5.4+)

RIA Functions?

JavaScript was designed for scripting rather than full-fledged rich client development. For this reason, a JavaScript/AJAX product may not necessarily support all RIA functions mentioned. Partial screen updates, asynchronous communication, modal dialogs and menus will probably be available. But other RIA goodies may be severely limited in functionality or missing altogether. Typically, the set of rich UI widgets that supports direct manipulation will be poor compared to those we find in toolkits for desktop applications. Important questions to ask include:

* how does the widget set compare to standard sets like those of Java Swing?
* is the look&feel adaptable?
* is there an API for the development of new widgets?
* is it possible to integrate third-party widgets?
* what is the market for third-party widgets?
* is there support for Accessibility?

How Strong?

Industrial strength characteristics like maintainability, reliability, availability, scalability, performance and security are a challenge for JavaScript/AJAX products because they have to rely on a scripting language and must work in a heterogeneous set of execution environments. It is unrealistic to expect the kind of industrial strength that can be found in a homogeneous Java product. Yet, there is a wide range of options regarding the amount of code that needs to be executed in the shaky JavaScript environment. The complete range of architectures shown in Figure 4 is feasible.

Figure 4: RIA Architecture Options

JavaScript code can be limited to an application-independent presentation engine that comes with the product, as illustrated by the rightmost option in Figure 4. This pure thin-client approach has several advantages: applications can be written entirely in Java, with a single programming environment. Moreover, there is a homogeneous server-side programming model, and no need to distribute the application code between client and server, nor between different programming languages. This simplifies both development and testing substantially. Finally, the application code will execute primarily in the robust server-side environment.

Note that pure thin-client architectures are rare in JavaScript based UIs. The typical approach is to split the functionality between client and server individually for each application as shown by the three left-hand options in Figure 4: either within the presentation logic, within the business logic, or between business logic and data. The leftmost option represents a pure fat-client architecture where the entire application code is written in JavaScript. Fat-client or hybrid approaches may be appropriate for certain scenarios. But for industrial strength characteristics, a pure thin-client architecture is preferable. Crucial questions are, therefore:

* what architecture is supported or enforced: thin, fat, or hybrid?
* is the programming model server-side or distributed?
* is application code written purely in Java or in a heterogeneous mixture of languages including JavaScript, Java, and proprietary XML languages?
* how much of the product’s code is JavaScript?
* how can security be handled?

Industrial Strength: Java

Figure 5: Java Branch

RIA products that are based entirely on Java have the best foundation for industrial strength. Java’s solid standards base, homogeneous technology, broad choice in high-quality tooling, and bright perspective regarding future enhancements ensure that a well-designed product can be maintained cost effectively and for any length of time in the foreseeable future. Reliability, availability, scalability and security are given on a wide array of platforms.

Designed for full-fledged productivity applications, Java’s UI technology leaves nothing to be desired regarding rich-client functions. There are comprehensive, customizable widget toolkits for desktops as well as mobile devices, and numerous third-party libraries.

In contrast to many JavaScript products, Java RIA offerings typically follow a thin-client approach as shown by the rightmost picture in Figure 4: the client is an application independent presentation engine that can be executed either as an applet in a browser, or as a Java program on the desktop, or even on a PDA.

The questions that distinguish Java RIAs ask for the JREs (Java Runtime Environment) supported, the UI libraries employed, and leverage of Java standards.
JRE Versions Supported?

Compared to the execution environments for JavaScript, JRE implementations are more standardized. Programs written with a specific JDK (Java Developer’s Kit) will usually run in the corresponding JRE, irrespective of whether this JRE is running within a browser, on a desktop operating system, or on a device like a PDA. The version issue is primarily raised on the level of the JRE and the target environments that support that JRE.

Most RIA products support multiple JRE versions. Some are based on JDK 1.1 with its AWT widget library and will run on JRE 1.1 or later. Others are using Swing and will run on JRE 1.2 or later. The respective pros and cons of earlier or later versions must be evaluated carefully, because the earliest version supported defines the “least common denominator”. A low “denominator” product based on the low-level AWT will often run on more JREs because later Java versions are fairly downward compatible. A high “denominator” product based on the higher-level Swing will be able to leverage the improvements of new Java versions. The most important questions are:

* what are the target devices/environments for the end users?
* which JREs are available and easily installable on the targets?
* on which JREs does the product run?
* which JRE is the “least common denominator” of the product?
* does the product leverage improvements of later JDK/JRE versions?

UI Libraries Employed?

Java’s client-side UI technology has evolved substantially. The initial low-level library AWT has been complemented with the high-level Swing library, and the latter has substantially improved over the years. The Eclipse project has come up with its own low-level and high-level UI libraries SWT and JFace, respectively. RIA products offer the functionality of high-level libraries. Yet, some of them have chosen to implement the high-level functions themselves and use low-level libraries on the client. This has pros and cons. On the upside, a low-level approach can support multiple UI libraries on the client, under the condition that the functionality is limited to the “common denominator” of these libraries. On the downside, the low-level approach will lead to a layer of proprietary software that needs to be maintained. With a high-level library, the proprietary depth of a product can be reduced to a minimum. Even the API can be taken from the standard library, as described by Bernhard Wagner in his article on “Server-Side Swing” [3]. Such an RIA product will have a minimal proprietary footprint and will profit from enhancements of the standard Swing library as it evolves. Essential drill-down questions on the UI libraries are:

* is the product based on low-level libraries (AWT, SWT) or high-level libraries (Swing, JFace)?
o low: does it support multiple UI libraries on the client?
o high: is the API proprietary or a server-side proxy approach?
* what is the proprietary footprint of the UI technology?
* is there an extension API for the development of new widgets?
* is it possible to integrate third-party widgets?
* can the look&feel be customized?
* is there support for Accessibility?

Standards Leverage?

Java standards are a key driver for reducing the proprietary share in a software stack. We have seen that the proprietary footprint of an RIA product’s client-side UI technology can vary considerably. Similar assessments can be made for the server-side part of RIA products, and the communication between client and server. J2EE defines standards that limit client/server communication, for instance, to request/reply protocols and specific transports such as HTTP(S) or RMI/IIOP. Moreover, EJBs disallow multi-threading that is not managed by the container, thereby ensuring that scaling measures like clustering can be handled by the J2EE infrastructure. RIA products that come with their own servers/proxies or allow bi-directional communication violate these standards and will need proprietary software that replaces the corresponding functions of the standard infrastructure. In general, a product that fully leverages the Java standards should have no function of its own if that same function can be delegated to the standard infrastructure. The following questions help in assessing standards leverage:

* is session handling delegated to the J2EE container or does the product come with a proprietary server/proxy?
* is clustering supported and delegated to the J2EE platform?
* is communication handled by J2EE compliant protocols?
* is deployment configurable and possible both in a servlet container and an EJB container?
* is deployment as a portlet possible, according to the JSR 168 standard?
* can the client part and server part be run in a single process for stand-alone deployment, leveraging the Java VM standards?

Fancy Animations: Flash

Figure 6: Flash Branch

Flash has its main strength in animations. This is not surprising, because it was created to animate websites. The core Flash technology includes an execution environment (the Flash player), a binary file format for movies (SWF), and a scripting language (ActionScript). Comfortable end-user tools that generate SWF and support ActionScript programming are available for designing movies and websites. Flash-based RIA products for application development are comparatively new. As a consequence, both the core technology and RIA tools are evolving. This emphasizes the importance of the version issue, not only for the execution environment, but the programming model and programming tools as well.

From an architectural perspective, the Flash-based approaches are close to those relying on JavaScript/AJAX. There is a scripting language that allows code to be sent to the client at run-time. Products can provide an array of architectural options, ranging from fat client to thin client as shown in Figure 4. The relevant questions are, therefore, close to those you should ask for JavaScript/AJAX.
Flash Versions Supported?

The core Flash technology is proprietary and tightly controlled by the vendor. Programs written for a specific version will run in the browsers and on the operating systems that are supported. The version issue is raised on the level of the Flash player, ActionScript, and the SWF format generated by an RIA product. The questions you need to ask are equivalent to those for the Java products:

* what are the target devices/environments for the end users?
* which Flash players are available and easily installable on the targets?
* on which players does the product run?
* which player is the “least common denominator” of the product?
* does the product leverage improvements of later ActionScript/SWF/player versions?

RIA Functions?

ActionScript, SWF and the Flash player were originally not designed for full-fledged rich client development. For this reason, you should ask the same questions as for JavaScript/AJAX products:

* how does the widget set compare to standard sets like those of Java Swing?
* is the look&feel adaptable?
* is there an API for the development of new widgets?
* is it possible to integrate third-party widgets?
* what is the market for third-party widgets?
* is there support for Accessibility?

How strong?

Relying on a scripting language like the JavaScript/AJAX approaches, Flash-based products have a similar range of options regarding the execution of code in the client-side environment. ActionScript can be limited to an application independent presentation engine that comes with the product, enforcing a pure thin-client architecture as shown in the rightmost picture of Figure 4. Alternatively, the developer can be given the option to write arbitrary application code with ActionScript, allowing for hybrid or fat client architectures. In general, a thin-client architecture will lead to superior industrial strength, as discussed above.

Current Flash-based RIA products don’t support a pure thin-client architecture. They provide a heterogeneous programming model and require a hybrid set of tools for the various programming languages that need to be combined. This can be acceptable for industrial strength if their scope is limited to creating animations rather than transferring substantial parts of the application code to the client. Useful questions for an evaluation are:

* what architecture is supported or enforced: thin, fat, or hybrid?
* is the programming model server-side or distributed?
* is application code written purely in Java or in a heterogeneous mixture of languages including ActionScript, Java, and proprietary XML languages?
* how large, powerful and proprietary is the set of tools required?
* is usage of ActionScript and proprietary tools limited to the presentation layer of applications?
* how can security be handled?

Conclusion

RIA and AJAX are new terms for a technology that has been around for years. Given the hype for these terms, it is difficult to differentiate between products. They vary substantially in their technical foundation and their suitability for specific requirements. Careful product evaluation is essential. The decision tree presented in this essay helps asking the right questions and leads through the most important decisions.
References

[1] Coach K. Wey: AJAX: Asynchronous Java + XML?
http://www.developer.com/design/article.php/3526681
[2] Jesse James Garrett: AJAX: A New Approach to Web Applications,
http://www.adaptivepath.com/publications/essays/archives/000385.php
[3] Bernhard Wagner: Server-Side Swing for Rich Internet Applications,
http://javadesktop.org/articles/canoo/index.html

What is CMM (SEI Capability Maturity Model)?

2005-11-03T06:25:00.000+07:00

According to the Carnegie Mellon University Software Engineering Institute, CMM is a common-sense application of software or business process management and quality improvement concepts to software development and maintenance. Its a community-developed guide for evolving towards a culture of engineering excellence, model for organizational improvement. The underlying structure for reliable and consistent software process assessments and software capability evaluations.

The Capability Maturity Model for Software (CMM) is a framework that describes the key elements of an effective software process. There are CMMs for non software processes as well, such as Business Process Management (BPM). The CMM describes an evolutionary improvement path from an ad hoc, immature process to a mature, disciplined process. The CMM covers practices for planning, engineering, and managing software development and maintenance. When followed, these key practices improve the ability of organizations to meet goals for cost, schedule, functionality, and product quality. The CMM establishes a yardstick against which it is possible to judge, in a repeatable way, the maturity of an organization's software process and compare it to the state of the practice of the industry. The CMM can also be used by an organization to plan improvements to its software process. It also reflects the needs of individuals performing software process, improvement, software process assessments, or software capability evaluations; is documented; and is publicly available.

The Five Maturity Levels described by the Capability Maturity Model can be characterized as per their primary process changes made at each level as follows:

1) Initial The software process is characterized as ad hoc, and occasionally even chaotic. Few processes are defined, and success depends on individual effort and heroics.
2) Repeatable Basic project management processes are established to track cost, schedule, and functionality. The necessary process discipline is in place to repeat earlier successes on projects with similar applications.
3) Defined The software process for both management and engineering activities is documented, standardized, and integrated into a standard software process for the organization. All projects use an approved, tailored version of the organization's standard software process for developing and maintaining software.
4) Managed Detailed measures of the software process and product quality are collected. Both the software process and products are quantitatively understood and controlled.
5) Optimizing Continuous process improvement is enabled by quantitative feedback from the process and from piloting innovative ideas and technologies.

The structure of the Capability Maturity model is as shown below:

At the Optimizing Level or the CMM level5, the entire organization is focused on continuous Quantitative feedback from previous projects is used to improve the project management, usually using pilot projects, using the skills shown in level 4. The focus is on continuous process improvement.

Software project teams in SEI CMM Level 5 organizations analyze defects to determine their causes. Software processes are evaluated to prevent known types of defects from recurring, and lessons learned are disseminated to other projects. The software process capability of Level 5 organizations can be characterized as continuously improving because Level 5 organizations are continuously striving to improve the range of their process capability, thereby improving the process performance of their projects. Improvement occurs both by incremental advancements in the existing process and by innovations using new technologies and methods.

Previously known as Key Process Area (KPA)

A process area (PA) contains the goals that must be reached in order to improve a software process. A PA is said to be satisfied when procedures are in place to reach the corresponding goals. A software organization has achieved a specific maturity level once all the corresponding PAs are satisfied. The process areas (PA's) have the following features:

1) Identify a cluster of related activities that, when performed collectively, achieve a set of goals considered important for enhancing process capability.

2) Defined to reside at a single maturity level.

3) Identify the issues that must be addressed to achieve a maturity level.

The different maturity levels have different process areas pre-defined as shown in the figure above. The SEI CMMI Level 5 has 3 PA's defined:

Process change management: To identify the causes of defects and prevent them from recurring.
Technology change management: To identify beneficial new technology and transfer them in an orderly manner
Defect prevention: To continually improve the process to improve quality, increase productivity, and decrease development time.

The purpose of Process Change Management is to continually improve the software processes used in the organization with the intent of improving software quality, increasing productivity, and decreasing the cycle time for product development. When software process improvements are approved for normal practice, the organization's standard software process and the projects' defined software processes are revised as suited.

The goals sought to achieve are as follows:

Continuous process improvement is planned.
Participation in the organization's software process improvement activities is organization wide.
The organization's standard software process and the projects defined software processes are improved continuously.

These defined standards give the organization a commitment to perform because:

The organization follows a written policy for implementing software process improvements.
Senior management sponsors the organization's activities for software process improvement.

The ability of the organization to perform transpires because:

Adequate resources and funding are provided for software process improvement activities.
Software managers receive required training in software process improvement.
The managers and technical staff of the software engineering group and other software-related groups receive required training in software process improvement.
Senior management receives required training in software process improvement.

Capability Maturity Model (CMM) Process Area (PA) Activities

The Process Area Activities performed include:
* A software process improvement program is established which empowers the members of the organization to improve the processes of the organization.
* The group responsible for the organization's software process activities coordinates the software process improvement activities.
* The organization develops and maintains a plan for software process improvement according to a documented procedure.
* The software process improvement activities are performed in accordance with the software process improvement plan.
* Software process improvement proposals are handled according to a documented procedure.
* Members of the organization actively participate in teams to develop software process improvements for assigned process areas.
* Where appropriate, the software process improvements are installed on a pilot basis to determine their benefits and effectiveness before they are introduced into normal practice.
* When the decision is made to transfer a software process improvement into normal practice, the improvement is implemented according to a documented procedure.
* Records of software process improvement activities are maintained.
* Software managers and technical staff receive feedback on the status and results of the software process improvement activities on an event-driven basis.

The primary purpose of periodic reviews by senior management is to provide awareness of, and insight into, software process activities at an appropriate level of abstraction and in a timely manner. The time between reviews should meet the needs of the organization and may be lengthy, as long as adequate mechanisms for exception reporting are available.

The Capability Maturity Model has been criticized in that, that it does not describe how to create an effective software development organization. The traits it measures are in practice very hard to develop in an organization, even though they are very easy to recognize. However, it cannot be denied that the Capability Maturity Model reliably assesses an organization's sophistication about software development.

The CMM was invented to give military officers a quick way to assess and describe contractors' abilities to provide correct software on time. It has been a roaring success in this role. So much so that it caused panic-stricken salespeople to clamor for their engineering organizations to "implement CMM" with CMM level 5 being the ultimate.

Capability Maturity Model aka (CMM)

2005-11-02T14:32:00.000+07:00

The Capability Maturity Model (CMM) is a method for evaluating and measuring the maturity of the software development process of organizations on a scale of 1 to 5. The CMM was developed by the Software Engineering Institute (SEI) at Carnegie Mellon University in Pittsburgh. It has been used extensively for avionics software and for government projects since it was created in the mid-1980s. The Software Engineering Institute has subsequently released a revised version known as the Capability Maturity Model Integration (CMMI).

The purpose of CMM Integration is to provide guidance for improving your organization’s processes and your ability to manage the development, acquisition, and maintenance of products or services.

Levels of the CMM

(See chapter 2 of (March 2002 edition of CMMI^SM from SEI), page 11.)

There are five levels of the CMM. According to the SEI, "Predictability, effectiveness, and control of an organization's software processes are believed to improve as the organization moves up these five levels. While not rigorous, the empirical evidence to date supports this belief."

1 - Initial

At maturity level 1, processes are usually ad hoc and chaotic. The organization usually does not provide a stable environment. Success in these organizations depends on the competence and heroics of the people in the organization and not on the use of proven processes. In spite of this ad hoc, chaotic environment, maturity level 1 organizations often produce products and services that work; however, they frequently exceed the budget and schedule of their projects.
Maturity level 1 organizations are characterized by a tendency to over commit, abandon processes in the time of crisis, and not be able to repeat their past successes.

2 - Repeatable

At maturity level 2, Software development successes are repeatable. The organization may use some basic project management to track cost and schedule.
Process discipline helps ensure that existing practices are retained during times of stress. When these practices are in place, projects are performed and managed according to their documented plans.
Project status and the delivery of services are visible to management at defined points (for example, at major milestones and at the completion of major tasks).

3 - Defined

At maturity level 3, processes are well characterized and understood, and are described in standards, procedures, tools, and methods.
The organization’s set of standard processes, which is the basis for level 3, is established and improved over time. These standard processes are used to establish consistency across the organization. Projects establish their defined processes by tailoring the organization’s set of standard processes according to tailoring guidelines.
The organization’s management establishes process objectives based on the organization’s set of standard processes and ensures that these objectives are appropriately addressed.
A critical distinction between level 2 and level 3 is the scope of standards, process descriptions, and procedures. At level 2, the standards, process descriptions, and procedures may be quite different in each specific instance of the process (for example, on a particular project). At level 3, the standards, process descriptions, and procedures for a project are tailored from the organization’s set of standard processes to suit a particular project or organizational unit.

4 - Managed

Using precise measurements, management can effectively control the software development effort. In particular, management can identify ways to adjust and adapt the process to particular projects without measurable losses of quality or deviations from specifications.
Subprocesses are selected that significantly contribute to overall process performance. These selected subprocesses are controlled using statistical and other quantitative techniques.
A critical distinction between maturity level 3 and maturity level 4 is the predictability of process performance. At maturity level 4, the performance of processes is controlled using statistical and other quantitative techniques, and is quantitatively predictable. At maturity level 3, processes are only qualitatively predictable.

5 - Optimizing

Maturity level 5 focuses on continually improving process performance through both incremental and innovative technological improvements. Quantitative process-improvement objectives for the organization are established, continually revised to reflect changing business objectives, and used as criteria in managing process improvement. The effects of deployed process improvements are measured and evaluated against the quantitative process-improvement objectives. Both the defined processes and the organization’s set of standard processes are targets of measurable improvement activities.
Process improvements to address common causes of process variation and measurably improve the organization’s processes are identified, evaluated, and deployed.
Optimizing processes that are agile and innovative depends on the participation of an empowered workforce aligned with the business values and objectives of the organization. The organization’s ability to rapidly respond to changes and opportunities is enhanced by finding ways to accelerate and share learning.
A critical distinction between maturity level 4 and maturity level 5 is the type of process variation addressed. At maturity level 4, processes are concerned with addressing special causes of process variation and providing statistical predictability of the results. Though processes may produce predictable results, the results may be insufficient to achieve the established objectives. At maturity level 5, processes are concerned with addressing common causes of process variation and changing the process (that is, shifting the mean of the process performance) to improve process performance (while maintaining statistical probability) to achieve the established quantitative process-improvement objectives.

Recent versions of CMMI from SEI indicate a "level 0", characterized as "Incomplete". Many observers leave this level out as redundant or unimportant, but Pressman and others make note of it. See page 18 of the August 2002 edition of CMMI from SEI (Note: PDF file).

Anthony Finkelstein extrapolated that negative levels, or the Capability Immaturity Model, are necessary to represent environments that are not only indifferent, but actively counterproductive, and this was refined by Tom Schorsch:

CMM level 0 (negligent)

CMM level -1 (obstructive)

CMM level -2 (contemptuous)

CMM level -3 (undermining)

Do not completely agree.

Some praise of the CMM

The CMM was developed to give Defense organizations a yardstick to assess and describe the capability of software contractors to provide software on time, within budget, and to acceptable standards. It has arguably been successful in this role, even reputedly causing some software salespeople to clamour for their organizations' software engineers/developers to "implement CMM."
The CMM is intended to enable an assessment of an organization's maturity for software development. It is an important tool for outsourcing and exporting software development work. Economic development agencies in India, Ireland, Egypt, and elsewhere have praised the CMM for enabling them to be able to compete for US outsourcing contracts on an even footing. Whilst this may have been very positive for the employment of software engineers in emerging or Third World economies - notably in India during the early 2000s - it is regarded as having adversely affected the potential employment opportunities for software engineers in the developed economies - notably the USA and Europe. This outsourcing is a form of labor arbitrage which is similar to the movement of manufacturing of (for example) fashion clothing or Nike shoes to Third World economies with relatively cheap labor rates.
It is reputed that IBM, Motorola, Logica, BT, and others have discovered the following:
- It takes 18 months on average to move up one SEI level, but it has been done in 8 (refer XXX? to substantiate this).
- Defect rates can be lowered from 1 per 1,000 lines of code (said to be typical of Microsoft and its peers) down to 1 per 1,000,000 lines - this is roughly Six Sigma quality; refer also relevant Deming references (NB: Deming was not a proponent of Six Sigma per se, and advised against the use of any slogans for process quality improvement).(refer XXX? to substantiate this)
- There is no specific evidence for shortening "time to market", but there is (refer XXX? to substantiate this) for increasing the accuracy of estimating completion date from 75% overruns on average at level 1, to plus or minus 2% at level 5.
- Data on productivity increases is more variable, but it is at least (refer XXX? to substantiate this) a doubling of productivity.

Some criticism of the CMM

CMM has failed to take off the world over. It's hard to tell exactly how wide spread it is as the SEI only publishes the names and achieved levels of compliance of companies that have requested this information to be listed (http://www.sei.cmu.edu/cmmi/faq/ares-faq.html#RATE03). The most current Maturity Profile for CMMI is available at http://www.sei.cmu.edu/appraisal-program/profile/pdf/CMMI/2005marCMMI.pdf)
No external body actually certifies a software development center as being CMM compliant. It is supposed to be an honest self-assessment. http://www.sei.cmu.edu/cmmi/faq/ares-faq.html#APR04 and http://www.sei.cmu.edu/cmmi/faq/ares-faq.html#APR05
The CMM does not describe how to create an effective software development organization. The CMM contains behaviors or best practices that successful projects have demonstrated. Being CMM compliant is not a guarantee that a project will be successful, however being compliant can increase a project's chances of being successful.
The CMM can seem to be overly bureaucratic, promoting process over substance. For example, for emphasizing predictability over service provided to end users. More commercially successful methodologies (for example, the Rational Unified Process) have focused not on the capability of the organization to produce software to satisfy some other organization or a collectively-produced specification, but on the capability of organizations to satisfy specific end user "use cases" as per the Object Management Group's UML (Unified Modeling Language) approach. [1].cccc

The most beneficial elements of CMM Level 2 and Level 3

Creation of Software Specifications, stating what it is that is going to be developed, combined with formal sign off, an executive sponsor and approval mechanism. This is NOT a living document, but additions are placed in a deferred or out of scope section for later incorporation into the next cycle of software development.
A Technical Specification, stating how precisely the thing specified in the Software Specifications is to be developed will be used. This is a living document.
Peer Review of Code (Code Review) with metrics that allow developers to walk through an implementation, and to suggest improvements or changes. Note - This is problematic because the code has already been developed and a bad design can not be fixed by "tweaking", the Code Review gives complete code a formal approval mechanism.
Version Control - a very large number of organizations have no formal revision control mechanism or release mechanism in place.
The idea that there is a "right way" to build software, that it is a scientific process involving engineering design and that groups of developers are not there to simply work on the problem du jour.

References:

CMMI Official Web Site
A Whitepaper on CMMi Implementation
CMMi Automation Software
A critical look at implementing CMM Level 2
http://www.cio.com/archive/030104/cmm.html [CIO Article, Beware the CMM Hype]

The Case for Nullable Types

2005-09-04T19:36:00.000+07:00

Nullable Types

by Kamran Qamar

Imagine some part of your code stores an amount of money. Now software is driven by strict rules of mathematics, and there's normally a need to quantify each variable we use. That is why, in .NET, each type has default value. Value-type variables always have predefined values e.g. int has default value of 0 and bool has default value of false . So far so good, but a problem arises when we try to apply our strict mathematical rules to the real world, in which you might not have all the data available. For example, how does a value of 0 distinguish that a user has no money from the case where they have not entered any value for your software to store. This problem becomes clearer when you start working with relational databases. Relational databases have a concept of a NULL value, which indicates that no data is present, and, for value types, does not yield an appropriate value in .NET programming languages.

Achieving Nullable Values - Current Techniques

Over the years developers have adopted a number of workarounds for this problem, some of which I'll briefly discuss:

Special Values

One of the most notable solutions is to use some special unused value of the given data type as the null value e.g. if you are dealing with an age, you define it as int and use -1 as the null value. This solution is workable only when you can identify value which is definitely unused and won't therefore be mistaken for a real value, e.g. if you are working with dollar ($) amount, -1 may indicate a loss of 1$ and is not therefore suitable to represent a null value if this value might occur.

This solution is simple to implement, but generally frowned upon because of the potential for subtle runtime bugs, especially if the application requirements subsequently change so that the value previously used as the null value becomes a legitimate value.

Using a Flag

One solution that I have used frequently is to define a flag in the object that contains the possibly null field, which is used to determine if the value is null or not. Incidentally system classes also use this approach e.g. SqlDataReader has a property IsDbNull that identifies if value returned from a database record is null or not.

Here's an example of this approach:

public class Person
{
  private string _fullName;
  private int _age;
  private bool _hasAge;
  public string FullName
  {
     get{return _fullName;}
     set{_fullName = value;}
  }
  public int Age
  {
     get
     {
        if(!_hasAge)
           throw new InvalidOperationException("Value not set");
        return _age;
     }
     set
     {
        _age = value;
        _hasAge = true;
     }
  }
  public bool HasAge
  {
     get{return _hasAge;}
  }
  public void Load(SqlDataReader dr)
  {
     if(dr["FullName"].IsDbNull)
        _fullName = "";
     else
        _fullName = dr["FullName"].ToString();
     _hasAge = dr["Age"].IsDbNull;
     if(!_hasAge)
        _age = (int)dr["Age"];
  }
}

The above code is an example of using flag to identify a value as null. Here the code is using the HasAge property to determine if you have an Age value or not. The class sets this flag to true , whenever the Age property is set. In client code, you always need to check HasAge value before using the Age property.

Defining a Nullable Type

Some developers go even a step further and define their own value types e.g. consider the following code:

public struct NullableInt
{
  bool isNull;
  public bool IsNull { get { return isNull;}}
  bool value;
  public int Value {
     get { return value;}
     set { @value=value;isNull=false;}
  }
  public NullableInt()
  {
     isNull = true;
  }
// etc.
}

By using this NullableInt type instead of standard system int type, we obtain more control over our data.

Editors Note: ASP Today readers might be interested to know that the ASP Today codebase makes extensive use of this solution for dates. Most of the dates you see displayed on the site come to you courtesy of a custom-coded Date struct, whose main raison d'etre is to wrap System.DateTime to make it nullable!

One advantage of this approach is you can if you wish also define a DefaultValue field, something that is available in value types, though lost in the above code. If you do define such a field, you'll need to decide what the most reasonable behaviour is for example constructing instances - eg. you might want a value to default to its default value, but be settable to null if the client code requires it.

This approach can work really well, and has the advantage of encapsulating the nullable-ness so client code doesn't have to worry as much about null values. But what happens when you add - say - two NullableInt s? Yes you can do that, all you have to do is upgrade the NullableInt struct with some operator overloads. And what about casting a double to NullableInt , that's less trivial, but you can achieve the same result by providing a constructor at the cost of some easy syntax. (You could instead use an operator overload, but I'll illustrate the principle here with a constructor):

public NullableInt(double d)
{
  Value = d;
}
…

The client code that uses this might look as follows:

NullableInt i;
double d = 2.5;
i = new NullableInt(d);
Console.WriteLine(i.ToString()); //will print 2

Where traditionally you would have simply written code like this:

int i;
double d = 2.5;
i = d;
Console.WriteLine(i.ToString()); //will print 2

This approach is extremely powerful but in some cases can be over kill, because not only that all the values do not require to be null able at all the time, it also requires custom data types and special coding discipline on part of all the developers involved.

Introducing .NET2.0 Nullable Types

Now that I have scared you with the limitation of the third approach, let me give you the good news. The C# 2.0 language solves this long standing problem by proving complete and integrated support for a nullable version of all value types, based on a clever application of generics along with some neat supporting syntax.

A variable of a nullable type can represent all the values of its underlying type, plus an additional null value. The way to define any valuetype as a nullable type is to define the generic System.Nullable type. Therefore the syntax to define a nullable integer is:

System.Nullable i;

This is kind of tedious, that is why C# team did a favor to us by providing a clean syntax for the nullable type in the form of [Value-Type]? hence System.Nullable and T? are interchangeable, and the statements below are equivalent and will both result in declaring a nullable int variable.

System.Nullable i;

int? i;

Using this syntax you can define any value-type as nullable type. You'll see more examples in the rest of this column piece.

Understanding the System.Nullable type

Before I dig deeper into the different features and benefits of the nullable types, it will be beneficial to understand how the nullable behavior is achieved in .NET. As I mentioned, at the heart of this behavior is generics. For more information about generics see the excellent article from Juval Lowy at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnvs05/html/csharp_generics.asp .

The System.Nullable is the generic type that makes the magic of nullable types possible. You can define any value type as nullable. Examples of nullable types are:

int? i = -1;
bool? flag = null;
char? chr = 'z';
int?[] MyArray = new int?[10];

Remember, T in System.Nullable is always a value-type. Hence the following lines of codes are incorrect and will result in compile time warnings (you might expect errors but in practice I found it was simply giving warnings):

string? Message = "Hello, World!";       // Compiler warning
SomeClass? someClass { };          // Compiler warning

System.Nullable has two public properties: HasValue of type bool and Value , which is of the same type as the nullable type's underlying type. The HasValue property is always true for any non-null instance and false for null instances. When HasValue is true, the Value property returns the contained value. When the HasValue is false, an attempt to access the Value property will result in InvalidOperationException exception. This follows what you'd probably regard as expected behaviour for a nullable type.

Using a Nullable Type

The following code shows how you would use a nullable integer

int? i;    // declaration
Console.WriteLine(i.HasValue);     // will print false
Console.WriteLine(i.Value);        // through InvalidOperationException
i = 18;                            //initialization
Console.WriteLine(i.HasValue);     //true
Console.WriteLine(i.Value);        //print 18

Checking for Null Values

As you can see in above code, when the new instance of a nullable type is created, its HasValue property return false, which means, as indicated before, that any attempt to access its value will result in InvalidOperationException . Does this mean, we always have to use the HasValue property to determine if a nullable type variable is null or not? Absolutely not, C# allows you to can check if a nullable object is null or not by comparing it against null keyword just as we do for reference-type objects. Hence, following syntax is valid:

if (i == null)
   Console.WriteLine("i is null");

In client code, you would most commonly check if the object is null or not, if it is not null you would use the value, otherwise you'll need to make some decision based on the situation. (Assuming of course that you are not in a situation where the algorithm you have been coding guarantees that a non-null value has been placed in the variable concerned). For example in the following code, I have defined i as a nullable int and x as a normal int. Next I check if i is null or not, if it is, I get the default value of i. This by the way demonstrates the new default keyword in C#2.0. Next, if i is not null, I set x with the value of i.

int? i = 18;
int x;
if(i == null)
  x = default(int);
else
  x = i.Value;

This is quite a lengthy code isn't it? The C#2.0 introduce new operator ?? that does exactly this for us. See the code below:

int? i = 18;
int x;
x = i?? default(int); //getting default value

x = i ?? 18; //defining my own value

Conclusion

Remember how difficult it was to initialize or compare an enum or struct type for a null value with .NET 1.0 - you typically had to write special code to check if the value of is valid or not. Now with C# 2.0, you can employ the power and elegance of the nullable type and the associated special syntax to check for null just like when using a reference-type.

Nullable types equip us with a powerful mechanism to handle special situations including database null values or other data types that contain fields that may not be assigned a value. Since this functionality is built right into the infrastructure using generics, it does not introduce any drastic performance penalty. However, converting an ordinary value type to nullable value type does introduce additional work for the infrastructure; hence nullable types should be used cautiously. Their use also dictates special consideration while assigning to ordinary value types and operating with them

That was a quick overview of null able types in .NET 2.0. for more information you can consult the .NET2.0 documentation.

Happy Programming!

On the Way to Mastering ASP.NET: Introducing Custom Entity Classes

2005-09-04T02:06:00.000+07:00

Karl Seguin

March 2005

Summary: There are situations for which untyped DataSets may not be the best solution for data manipulation. The goal of this guide is to explore an alternative to DataSets: custom entities and collections. (32 printed pages)

Introduction
Problems with Datasets
Custom Entity Classes
Object-Relational Mapping
Custom Collections
Managing Relationships
Beyond the Basics
Conclusion

Introduction

The days of ADODB.RecordSet and the oft-forgotten MoveNext are gone, replaced by the powerful and flexible capabilities of Microsoft ADO.NET. Our new arsenal is the System.Data namespace, featuring lightning-fast DataReaders, feature-rich DataSets, and packaged in a capable object-oriented model. It's no surprise that we have such tools at our disposal. Any 3-tier architecture relies on a solid Data Access Layer (DAL) to elegantly connect the data layer to the business layer. A quality DAL helps promote code reuse, is key to good performance, and is totally transparent.

As our tools have evolved, so too have our development patterns. Saying goodbye to MoveNext was more than ridding ourselves of cumbersome syntax; it opened our minds to disconnected data, which in turn had a profound impact on how we built applications.

While DataReaders might have been familiar (they behave a lot like RecordSets), it didn't take long for us to venture forward and explore DataAdapters, DataSets, DataTables, and DataViews. It was our growing skills at exploiting these new objects that changed how we developed. Disconnected data allowed us to utilize new caching techniques, and thus greatly improve the performance of our applications. The capability of these classes allowed us to write smarter and more powerful functions, while at the same time reducing, sometimes considerably, the amount of code required for common activities.

There are a number of situations for which DataSets are particularly well suited, such as prototyping, small systems, and support utilities. However, using them in enterprise systems, where ease of maintenance is more important than time to market, may not be the best solution. The goal of this guide is to explore an alternative to DataSets that is geared towards this type of work: custom entities and collections. Other alternatives exist, but none provide the same capability or have more backing. Our first task will be to look at the shortcomings of DataSets in order to understand the problem we are trying to solve.

Keep in mind that every solution has its own advantages and disadvantages, so it's possible that the drawbacks of DataSets are going to be more palatable to you than those of custom entities (which we'll also discuss). You and your team must decide which solution is better suited to your project. Remember to consider the total cost of a solution, including the nature of requirements to change and the likelihood that you'll spend more time in post-production than actually developing code. Finally, note that when I refer to DataSets, I don't mean typed-DataSets, which indeed solve some of the shortcomings associated with untyped-DataSets.

Problems with Datasets

Lack of Abstraction

The first and most obvious reason to consider alternatives is the DataSet's inability to decouple your code from the database structure. DataAdapters do a great job of making your code blind to the underlying database vendor (Microsoft, Oracle, IBM, …) but fail to abstract away the core component of a database: tables, columns, and relationships. These core database components are also the core components of the DataSet. DataSets and databases share more than just common components; unfortunately, they also share their schema. Given the following select statement:

SELECT UserId, FirstName, LastName
  FROM Users

We know that values will be available from the UserId, FirstName, and LastName DataColumns within our DataSet.

Why is this so bad? Let's look at a basic everyday example. First we have a simple DAL function:

'Visual Basic .NET
Public Function GetAllUsers() As DataSet
Dim connection As New SqlConnection(CONNECTION_STRING)
Dim command As SqlCommand = New SqlCommand("GetUsers", connection)
command.CommandType = CommandType.StoredProcedure
Dim da As SqlDataAdapter = New SqlDataAdapter(command)
Try
 Dim ds As DataSet = New DataSet
 da.Fill(ds)
 Return ds
Finally
 connection.Dispose()
 command.Dispose()
 da.Dispose()
End Try
End Function

//C#
public DataSet GetAllUsers() {
SqlConnection connection = new SqlConnection(CONNECTION_STRING);
SqlCommand command = new SqlCommand("GetUsers", connection);
command.CommandType = CommandType.StoredProcedure;
SqlDataAdapter da = new SqlDataAdapter(command);
try {
 DataSet ds = new DataSet();
 da.Fill(ds);
 return ds;
}finally {
 connection.Dispose();
 command.Dispose();
 da.Dispose();
}           
}

Next we have a page with a repeater displaying all the users:



  
    
       
          <%# DataBinder.Eval(Container.DataItem, "FirstName") %>

As we can see, our ASPX page makes use of the DAL function GetAllUsers for the DataSource of the repeater. If the database schema changes, for whatever reason (demoralization for performance, normalization for clarity, change in requirements), the change trickles all the way to the ASPX, namely the Databinder.Eval line, which makes use of the "FirstName" column name. This should immediately raise a red flag in your mind: a database schema change trickling all the way to ASPX code? Doesn't sound very N-Tier, does it?

If all we are doing is a simple column renaming, making changes in this example isn't complicated. But what if GetAllUsers is used in numerous places or, worse yet, exposed as a Web service, feeding countless consumers? How easily or safely can the change be propagated? For this basic example, the stored procedure itself serves as a layer of abstraction, which would probably suffice; but relying on stored procedures for anything but the most basic protection against this will likely cause greater problems down the road. Think of this as a form of hard-coding; in essence, when using DataSets, you are likely going to create a rigid connection between your database schema (regardless of whether you use column names or ordinal positions) and your application/business layers. Hopefully past experience (or logic) has taught you the impact hard-coding has on maintenance and future development.

Another way DataSets fail to provide proper abstraction is by requiring developers to know the underlying schema. We aren't talking about basic knowledge, either, but full knowledge of column names, types, and relationships. Not only does removing this requirement make your code less likely to break as we just saw, but it also makes it easier to write and maintain. Simply put:

Convert.ToInt32(ds.Tables[0].Rows[i]["userId"]);

is both hard to read and requires intimate knowledge of column names and their type. Ideally, your business layer knows nothing about the underlying database, the database schema, or SQL. If you are using DataSets as it is expressed in the previous code string (using CodeBehind doesn't make anything better), you likely have a very thin business layer.

Weakly-Typed

DataSets are weakly-typed, which makes them error prone and likely to impact your development effort. What this means is that whenever you retrieve a value from a DataSet, it comes in the form of a System.Object, which you have to convert. The risk you run is that a conversion will fail. Unfortunately, this failure won't happen at compile time, but rather at runtime. Additionally, tools such as Microsoft Visual Studio.NET (VS.NET) aren't very good at assisting your developers when it comes to weakly-typed objects. When we previously talked about requiring an in-depth knowledge of the schema, this is what was meant. Again, we'll look at a very common example:

'Visual Basic.NET
Dim userId As Integer =
?      Convert.ToInt32(ds.Tables(0).Rows(0)("UserId"))
Dim userId As Integer = CInt(ds.Tables(0).Rows(0)("UserId"))
Dim userId As Integer = CInt(ds.Tables(0).Rows(0)(0))

//C#
int userId = Convert.ToInt32(ds.Tables[0].Rows[0]("UserId"));

This code represents possible ways of retrieving a value from a DataSet—chances are your code has this all over the place (if it doesn't do a conversion and you are using Visual Basic .NET, you might have Option Strict off, in which case you're in even deeper trouble).

Unfortunately, each of those lines has the potential to generate numerous runtime errors:

The conversion can fail because:
- The value could be null.
- The developer might be wrong about the underlying data type (again, intimate knowledge of the database schema is required).
- If you are using ordinal values, who knows what column is actually at position X.
ds.Tables(0) might return a null reference (if any part of the DAL method or the stored procedure failed).
"UserId" might not be a valid column name because:
- It might have changed names.
- It might not be returned by the stored procedure.
- It might have a typo.

We could modify the code and write it more defensively, namely by adding checks for null/nothing and adding try/catch around the conversion, but this does nothing to help the developer.

Worst of all is that, as we've discussed, this isn't abstracted. The significance of this is that, every time you want to get the userId out of the DataSet, you'll run the risks listed previously, or you'll need to reprogram the same defensive steps (granted, a utility function would help alleviate this). Weakly-typed objects moves errors from design time or compile time, where they are always automatically detected and easily fixed, to the run time, where they risk being exposed in production and are harder to pinpoint.

Not Object-Oriented

Just because DataSets are objects and C# and Visual Basic .NET are object-oriented (OO) languages doesn't automatically make your usage of them object-oriented. The "hello world" of OO programming is typically a Person class that is sub-classed by an Employee class. DataSets, however, don't make this type of inheritance, or most other OO techniques, possible (or at least natural/intuitive). Scott Hanselman, an outspoken proponent of class entities, explains it best:

"A DataSet is an object, right? But it's not a Domain Object, it's not an 'Apple' or 'Orange'—it's an object of type 'DataSet.' A DataSet is a bowl (one that knows about the backing Data Store). A DataSet is an object that knows how to HOLD Rows and Columns. It's an object that knows a LOT about the Database. But I don't want to return bowls. I want to return Domain Objects, like 'Apples.'"¹

DataSets keep your data in a relational format, making them powerful and easy to use with relational databases. Unfortunately, this means you lose out on all the benefits of OO.

Since DataSets can't act as domain objects, it's impossible to add functionality to them. Typically, objects have fields, properties, and methods, which behave against an instance of the class. For example, you might have Promote or CalcuateOvertimePay functions associated with a User object, which can be cleanly called through someUser.Promote() or someUser.CalculateOverTimePay(). Since methods can't be added to a DataSet you'll need to use utility functions, deal with weakly-typed objects, and have more instances of hard-coded values spread throughout your code. You basically end up with procedural code where you either continuously keep getting data out of the DataSet, or cumbersomely store them in local variables and pass them around. Both methods have their disadvantages and neither has advantages.

The Case Against DataSet

If your idea of a Data Access Layer is to return a DataSet, you are likely missing out on some significant benefits. One reason for this is that you may be using a thin or non-existent business layer that, amongst other things, limits your ability to abstract. Additionally, it's difficult to take advantage of OO techniques, since you are using a generic pre-built solution. Finally, tools such as Visual Studio.NET can't easily empower developers with weakly-typed objects, such as DataSets, which reduces productivity while increasing the likelihood of bugs.

All of these factors have a direct impact on the maintainability of your code in one way or another. The lack of abstraction makes feature changes and bug fixes more involved and risky. You aren't able to take full advantage of the code reuse or the boost in readability offered by OO. And, of course, your developers, whether they work on the business logic or the presentation logic, must have intimate knowledge of your underlying data structure.

Custom Entity Classes

Most of the problems associated with DataSets can be solved by taking advantage of the rich capabilities of OO programming within a well-defined business layer. In essence we want to take relationally organized data (database) and have it available as objects (code). The idea being that instead of having a DataTable that holds information about cars, you actually have car objects (called custom entities or domain objects).

Before we look at custom entities we'll first look at the challenges that we'll face. The most obvious is the amount of code required. Instead of simply getting the data and automatically filling a DataSet, we get the data and manually map it to the custom entities that must first be created. Seeing as this is a repetitive task, we can mitigate it using code generation tools or O/R mappers (more on this later). The bigger problem is the actual process of mapping data from the relational to the object world. For simple systems the mapping is mostly straightforward, but as the complexity grows the differences between the two worlds can become problematic. For example, a key technique to help code-reuse, as well as maintainability, in the object world is inheritance. Unfortunately, inheritance is a foreign concept to relational databases. Another example is the difference in dealing with relations, with the object world maintaining a reference to a separate object and the relational world making use of foreign keys.

It might sound as if this approach isn't well suited for more complex systems as the amount of code along with the disparity between the relational data and objects grows, but the opposite is true. Complex systems gain from this approach by having their difficulties isolated in a single layer—the mapping process (which, again, can be automated). Additionally, this approach is already quite popular, which means a number of design patterns exist to cleanly deal with added complexity. Magnify the shortcomings of DataSets previously discussed with that of a complex system and you'll end up with a system whose difficulty to build is only surpassed with its inability to change.

What Are Custom Entities?

Custom entities are objects that represent your business domain; as such, they are the foundation of a business layer. If you have a user authentication component (the example we'll be using throughout this guide), you'll likely have User and Role objects. An e-commerce system would likely have Supplier and Merchandise objects and a real estate company might have Houses, Rooms, and Addresses. Within your code, custom entities are simply classes (there's a fairly tight correlation between an entity and a class, as it's used in OO programming). A typical User class might look like:

'Visual Basic .NET
Public Class User
#Region "Fields and Properties"
Private _userId As Integer
Private _userName As String
Private _password As String
Public Property UserId() As Integer
 Get
  Return _userId
 End Get
 Set(ByVal Value As Integer)
   _userId = Value
 End Set
End Property
Public Property UserName() As String
 Get
  Return _userName
 End Get
 Set(ByVal Value As String)
  _userName = Value
 End Set
End Property
Public Property Password() As String
 Get
  Return _password
 End Get
 Set(ByVal Value As String)
  _password = Value
 End Set
End Property
#End Region
#Region "Constructors"
Public Sub New()
End Sub
Public Sub New(id As Integer, name As String, password As String)
 Me.UserId = id
 Me.UserName = name
 Me.Password = password
End Sub
#End Region
End Class

//C#
public class User {
#region "Fields and Properties"
private int userId;
private string userName;
private string password;
public int UserId {
 get { return userId; }
 set { userId = value; }
 }
public string UserName {
 get { return userName; }
 set { userName = value; }
}
public string Password {
 get { return password; }
 set { password = value; }
}
#endregion
#region "Constructors"
public User() {}
public User(int id, string name, string password) {
 this.UserId = id;
 this.UserName = name;
 this.Password = password;
}
#endregion
}

Why Are They Beneficial?

The primary benefit gained from using custom entities comes from the simple fact that they are objects fully in your control. Namely, they allow you to:

Take advantage of OO techniques such as inheritance and encapsulation.
Add custom behavior.

For example, our User class could benefit from having an UpdatePassword function added to it (this is something we could do with datasets using external/utility functions, but at a readability/maintenance cost). Additionally, they are strongly-typed, meaning we get IntelliSense support:

Figure 1. IntelliSense with the User class

Finally, since custom entities are strongly-typed, they require less error-prone casts:

Dim userId As Integer = user.UserId
'versus
Dim userId As Integer =
?         Convert.ToInt32(ds.Tables("users").Rows(0)("UserId"))

Object-Relational Mapping

As discussed earlier, one of the main challenges of this approach is dealing with the differences between relational data and objects. Since our data is persistently stored in a relational database we have no choice but to bridge the two worlds. For the previous User example we could expect to have a user table in our database that looks a lot like:

Figure 2. Data view of the User

Mapping from this relational schema to our custom entity is a simple enough matter:

'Visual Basic .NET
Public Function GetUser(ByVal userId As Integer) As User
Dim connection As New SqlConnection(CONNECTION_STRING)
Dim command As New SqlCommand("GetUserById", connection)
command.Parameters.Add("@UserId", SqlDbType.Int).Value = userId
Dim dr As SqlDataReader = Nothing
Try
 connection.Open()
 dr = command.ExecuteReader(CommandBehavior.SingleRow)
 If dr.Read Then
  Dim user As New User
  user.UserId = Convert.ToInt32(dr("UserId"))
  user.UserName = Convert.ToString(dr("UserName"))
  user.Password = Convert.ToString(dr("Password"))
  Return user
 End If
 Return Nothing
Finally
 If Not dr is Nothing AndAlso Not dr.IsClosed Then
  dr.Close()
 End If
 connection.Dispose()
 command.Dispose()
 End Try
End Function

//C#
public User GetUser(int userId) {
SqlConnection connection = new SqlConnection(CONNECTION_STRING);
SqlCommand command = new SqlCommand("GetUserById", connection);
command.Parameters.Add("@UserId", SqlDbType.Int).Value = userId;
SqlDataReader dr = null;
try{
 connection.Open();
 dr = command.ExecuteReader(CommandBehavior.SingleRow);
 if (dr.Read()){
  User user = new User();
  user.UserId = Convert.ToInt32(dr["UserId"]);
  user.UserName = Convert.ToString(dr["UserName"]);
  user.Password = Convert.ToString(dr["Password"]);
  return user;           
 }
 return null;
}finally{
 if (dr != null && !dr.IsClosed){
  dr.Close();
 }
 connection.Dispose();
 command.Dispose();
}
}

We still set up our connection and command objects like we normally would, but then we create a new instance of our User class and populate it from our DataReader. You could still use a DataSet within this function and map it to your custom entity, but the primary benefit of DataSets over DataReader is that they provide a disconnected view of the data. In this case the User instance provides that disconnected view, letting us take advantage of the DataReader's speed.

Wait a Minute! You Didn't Solve Anything!

Observant readers might notice that one of the problems I pointed out with DataSets is that they aren't strongly-typed, which leads to a loss of productivity and an increase in the potential for runtime errors. They also require developers to have an in-depth knowledge of the underlying data structure. Looking at the previous code you might notice the exact same pitfalls lurking. Consider, however, that we have encapsulated these problems within a very isolated area of the code; meaning consumers of your class entities (web interface, web service consumer, windows form) remain totally unaware of these problems. Conversely, using DataSets spreads these problems throughout the code.

Enhancement

The previous code was useful to show the basic idea behind mapping, but two key enhancements can be done to improve it. First we want to extract the populate code into its own function, as it'll likely be reused:

'Visual Basic .NET
Public Function PopulateUser(ByVal dr As IDataRecord) As User
Dim user As New User
user.UserId = Convert.ToInt32(dr("UserId"))
'example of checking for NULL
If Not dr("UserName") Is DBNull.Value Then
 user.UserName = Convert.ToString(dr("UserName"))
End If
user.Password = Convert.ToString(dr("Password"))
Return user
End Function

//C#
public User PopulateUser(IDataRecord dr) {
User user = new User();
user.UserId = Convert.ToInt32(dr["UserId"]);
//example of checking for NULL
if (dr["UserName"] != DBNull.Value){
 user.UserName = Convert.ToString(dr["UserName"]);  
}
user.Password = Convert.ToString(dr["Password"]);
return user;
}

The second thing to notice is that instead of using a SqlDataReader for our mapping function, we use an IDataRecord. This is an interface that all DataReaders implement. Using IDataRecord makes our mapping process vendor-independent. In other words, we can use the previous function to map a User from an Access database, even if it uses an OleDbDataReader. If you combine this specific approach with the Provider Model Design Pattern (link 1, link 2), you'll have code that can be easily used for different database vendors.

Finally, the above code demonstrates how powerful encapsulation is. Dealing with NULLs in DataSets isn't the easiest thing—that's because every time you pull a value you need to check if it's NULL. With the above population method we've conveniently taken care of this in a single place, and spared our consumers from having to deal with it.

Where to Map?

There is some debate about where such data access and mapping function belongs—as part of a separate class or as part of the appropriate custom entity. There's certainly a nice elegance to having all user-related tasks (fetching data, updating, and mapping) as part of the User custom entity. This tends to work well when the database schema closely resembles the custom entity (as in this example). As your system grows in complexity and the differences between the two worlds start to appear, having a clear separation between your data layer and business layer can greatly help simplify maintenance (I like to call this the Data Access Layer). A side-effect from having the access and mapping code inside its own layer, the DAL, is that it provides us with a nice rule for ensuring a clear separation of our layers:

"Never return a class from the System.Data or child namespace from the DAL"

Custom Collections

So far we've only looked at dealing with individual entities; however, you'll often need to deal with more than a single object. A simple solution would be to store multiple values inside a generic collection, such as an Arraylist. This is a less than ideal solution, as it reintroduces some of the problems we had with DataSets, namely:

They aren't strongly-typed, and
Custom behavior can't be added.

The solution that best fits our needs is to create our own custom collection. Thankfully the Microsoft .NET Framework provides a class specifically meant to be inherited for this purpose: CollectionBase. CollectionBase works by storing any type of object inside private Arraylists, but exposing access to these private collections through methods that only take a specific type, such as a User object. In other words, weakly-typed code is encapsulated within a strongly-typed API.

While custom collections might seem like a lot of code, most of it is code generation or cut and paste friendly, oftentimes requiring only one search and replace. Let's take a look at the different parts that make up a custom collection for our User class:

'Visual Basic .NET
Public Class UserCollection
  Inherits CollectionBase
Default Public Property Item(ByVal index As Integer) As User
 Get
  Return CType(List(index), User)
 End Get
 Set
  List(index) = value
 End Set
End Property
Public Function Add(ByVal value As User) As Integer
 Return (List.Add(value))
End Function
Public Function IndexOf(ByVal value As User) As Integer
 Return (List.IndexOf(value))
End Function
Public Sub Insert(ByVal index As Integer, ByVal value As User)
 List.Insert(index, value)
End Sub
Public Sub Remove(ByVal value As User)
 List.Remove(value)
End Sub
Public Function Contains(ByVal value As User) As Boolean
 Return (List.Contains(value))
End Function
End Class

//C#
public class UserCollection : CollectionBase {
public User this[int index] {
 get {return (User)List[index];}
 set {List[index] = value;}
}
public int Add(User value) {
 return (List.Add(value));
}
public int IndexOf(User value) {
 return (List.IndexOf(value));
}
public void Insert(int index, User value) {
 List.Insert(index, value);
}
public void Remove(User value) {
 List.Remove(value);
}
public bool Contains(User value) {
 return (List.Contains(value));
}
}

More can be done by implementing CollectionBase, but the previous code represents the core functionality that is necessary for a custom collection. Looking at the Add function, we can see how we are simply wrapping the call to List.Add (which is an Arraylist) in a function that only allows a User object.

Mapping Custom Collections

The process of mapping our relational data to custom collections is very similar to the one we examined for custom entities. Instead of creating a single entity and returning it, we add the entity to the collection and loop to the next one:

'Visual Basic .NET
Public Function GetAllUsers() As UserCollection
Dim connection As New SqlConnection(CONNECTION_STRING)
Dim command As New SqlCommand("GetAllUsers", connection)
Dim dr As SqlDataReader = Nothing
Try
 connection.Open()
 dr = command.ExecuteReader(CommandBehavior.SingleResult)
 Dim users As New UserCollection
 While dr.Read()
  users.Add(PopulateUser(dr))
 End While
 Return users
Finally
 If Not dr Is Nothing AndAlso Not dr.IsClosed Then
  dr.Close()
 End If
 connection.Dispose()
 command.Dispose()
End Try
End Function

//C#
public UserCollection GetAllUsers() {
SqlConnection connection = new SqlConnection(CONNECTION_STRING);
SqlCommand command =new SqlCommand("GetAllUsers", connection);
SqlDataReader dr = null;
try{
 connection.Open();
 dr = command.ExecuteReader(CommandBehavior.SingleResult);
 UserCollection users = new UserCollection();
 while (dr.Read()){
  users.Add(PopulateUser(dr));
 }
 return users;
}finally{
 if (dr != null && !dr.IsClosed){
  dr.Close();
 }
 connection.Dispose();
 command.Dispose();
}
}

We get the data from the database, create our custom collection, and loop through the results to create each User object and add it into the collection. Notice also how the PopulateUser mapping function is reused.

Adding Custom Behavior

When talking about custom entities we only peripherally mentioned the ability to add custom behavior to our classes. The type of functionality that you'll be adding to your entities will mostly depend on the type of business logic you are implementing, but there is probably some common functionality you'll want to implement in your custom collections. One such example would be to return a single entity based on some key, for example a user based on a userId:

'Visual Basic .NET
Public Function FindUserById(ByVal userId As Integer) As User
For Each user As User In List
 If user.UserId = userId Then
  Return user
 End If
Next
Return Nothing
End Function

//C#
public User FindUserById(int userId) {
foreach (User user in List) {
 if (user.UserId == userId){
  return user;
 }
}
return null;
}

Another one might be to return a subset of users based on certain criteria, such as a partial user name:

'Visual Basic .NET
Public Function FindMatchingUsers(ByVal search As String) As UserCollection
If search Is Nothing Then
 Throw New ArgumentNullException("search cannot be null")
End If
Dim matchingUsers As New UserCollection
For Each user As User In List
 Dim userName As String = user.UserName
 If Not userName Is Nothing And userName.StartsWith(search) Then
  matchingUsers.Add(user)
 End If
Next
Return matchingUsers
End Function

//C#
public UserCollection FindMatchingUsers(string search) {
if (search == null){
 throw new ArgumentNullException("search cannot be null");
}
UserCollection matchingUsers = new UserCollection();
foreach (User user in List) {
 string userName = user.UserName;
 if (userName != null && userName.StartsWith(search)){
  matchingUsers.Add(user);
 }
}
return matchingUsers;
}

Using DataSets the same way can be achieved with DataTable.Select. It is important to note that while creating your own functionality puts you in absolute control of your code, the Select method provides a very convenient and code-free means of doing the same thing. On the flip side, Select requires developers to know the underlying database and isn't strongly-typed.

Binding Custom Collections

The first example we looked at was that of binding a DataSet to an ASP.NET control. Considering how common this is, you'll be glad to know that custom collections bind just as easily (this is because CollectionBase implements Ilist, which is used for binding). Custom collections can serve as the DataSource for any control that exposes it, and DataBinder.Eval can be used just as you would a DataSet:

'Visual Basic .NET
Dim users as UserCollection = DAL.GetallUsers()
repeater.DataSource = users
repeater.DataBind()

//C#
UserCollection users = DAL.GetAllUsers();
repeater.DataSource = users;
repeater.DataBind();




 
  <%# DataBinder.Eval(Container.DataItem, "UserName") %>

Instead of using the column name as the second parameter for DataBinder.Eval, you specify the property name you wish to display, in this case UserName.

For those doing processing in the OnItemDataBound or OnItemCreated exposed by many data bound controls, you are probably casting e.Item.DataItem to DataRowView. When binding to a custom collection, e.Item.DataItem is instead cast to the custom entity; in our example, the User class:

'Visual Basic .NET
Protected Sub r_ItemDataBound (s As Object, e As RepeaterItemEventArgs)
Dim type As ListItemType = e.Item.ItemType
If type = ListItemType.AlternatingItem OrElse
?   type = ListItemType.Item Then
 Dim u As Label = CType(e.Item.FindControl("userName"), Label)
 Dim currentUser As User = CType(e.Item.DataItem, User)
 If Not PasswordUtility.PasswordIsSecure(currentUser.Password) Then
  ul.ForeColor = Drawing.Color.Red
 End If
End If
End Sub

//C#
protected void r_ItemDataBound(object sender, RepeaterItemEventArgs e) {
ListItemType type = e.Item.ItemType;
if (type == ListItemType.AlternatingItem ||
?    type == ListItemType.Item){
 Label ul = (Label)e.Item.FindControl("userName");
 User currentUser = (User)e.Item.DataItem;
 if (!PasswordUtility.PasswordIsSecure(currentUser.Password)){
  ul.ForeColor = Color.Red;
 }
}
}

Managing Relationships

Within even the simplest system, relationships between entities will exist. With relational databases, relationships are maintained by means of foreign keys; using objects, a relationship is simply a reference to another object. For example, building on our previous examples, it's reasonable to expect a User object to have a Role:

'Visual Basic .NET
Public Class User
Private _role As Role
Public Property Role() As Role
 Get
  Return _role
 End Get
 Set(ByVal Value As Role)
  _role = Value
 End Set
End Property
End Class

//C#
public class User {
private Role role;
public Role Role {
 get {return role;}
 set {role = value;}
}
}

Or a collection of Roles:

'Visual Basic .NET
Public Class User
Private _roles As RoleCollection
Public ReadOnly Property Roles() As RoleCollection
 Get
  If _roles Is Nothing Then
   _roles = New RoleCollection
  End If
  Return _roles
 End Get
End Property
End Class

//C#
public class User {
private RoleCollection roles;
public RoleCollection Roles {
 get {
  if (roles == null){
   roles = new RoleCollection();
  }
  return roles;
 }
}
}

In these two examples, we have a fictitious Role class or RoleCollection class, which are just other custom entity or collection classes like the User and UserCollection classes.

Mapping Relationships

The real issue is how to map relationships. Let's look at a simple example; we want to retrieve a user based on userId along with his or her roles. First, we'll look at the relational model:

Figure 3. Relationships between Users and Roles

Here we see a Users table and a Roles table, both of which we can map in a straightforward manner to custom entities. We also have a UserRoleJoin table, which represents the many-to-many relationship between Users and Roles.

Next we use a stored procedure to pull two separate results: the first for the User, and the second for his or her Role(s):

CREATE PROCEDURE GetUserById(
 @UserId INT
)AS
SELECT UserId, UserName, [Password]
 FROM Users
 WHERE UserId = @UserID
SELECT R.RoleId, R.[Name], R.Code
 FROM Roles R INNER JOIN
    UserRoleJoin URJ ON R.RoleId = URJ.RoleId
 WHERE  URJ.UserId = @UserId

Finally, we map from the relational model to the object model:

'Visual Basic .NET
Public Function GetUserById(ByVal userId As Integer) As User
Dim connection As New SqlConnection(CONNECTION_STRING)
Dim command As New SqlCommand("GetUserById", connection)
command.Parameters.Add("@UserId", SqlDbType.Int).Value = userId
Dim dr As SqlDataReader = Nothing
Try
 connection.Open()
 dr = command.ExecuteReader()
 Dim user As User = Nothing
 If dr.Read() Then
  user = PopulateUser(dr)
  dr.NextResult()
  While dr.Read()
   user.Roles.Add(PopulateRole(dr))
  End While
 End If
 Return user
Finally
 If Not dr Is Nothing AndAlso Not dr.IsClosed Then
  dr.Close()
 End If
 connection.Dispose()
 command.Dispose()
End Try
End Function

//C#
public User GetUserById(int userId) {
SqlConnection connection = new SqlConnection(CONNECTION_STRING);
SqlCommand command = new SqlCommand("GetUserById", connection);
command.Parameters.Add("@UserId", SqlDbType.Int).Value = userId;
SqlDataReader dr = null;
try{
 connection.Open();
 dr = command.ExecuteReader();
 User user = null;
 if (dr.Read()){
  user = PopulateUser(dr);
  dr.NextResult();
  while(dr.Read()){
   user.Roles.Add(PopulateRole(dr));
  }           
 }
 return user;
}finally{
 if (dr != null && !dr.IsClosed){
  dr.Close();
 }
 connection.Dispose();
 command.Dispose();
}
}

The User instance is created and populated; we move to the next result/select and loop through, populating Roles and adding them to the RolesCollection property of the User class.

Beyond the Basics

The purpose of this guide was to introduce the concept and usage of custom entities and collections. Using custom entities is a widely used practice in the industry, and as such numerous patterns have been documented to deal with a wide range of scenarios. Design patterns are great for a number of reasons. First, when it comes to addressing specific situations, chances are you aren't the first to face a given problem. Design patterns let you reuse a tried and tested solution to a given problem (design patterns aren't meant to be 100% cut and paste, but they almost always provide a sound foundation to a solution). This in turn provides you with confidence that your system will scale well with complexity, not only because it's a widely used approach but also because it's a well documented one. Design patterns also provide you with a common vocabulary, which can make knowledge transfer and training much easier.

There's nothing to say that design patterns only apply to custom entities, and in fact many don't. However, if you give them a chance you'll likely be pleasantly surprised at how many well documented patterns do apply to custom entities and the mapping process.

This last section is dedicated to pointing out some more advanced scenarios that larger or more complex systems will likely run into. While most of the topics are probably worthy of individual guides, I'll, at the very least, try to provide you with some starting resources.

A great place to start is Martin Fowler's Patterns of Enterprise Application Architecture, which won't only serve as a great reference (with detailed explanations and plenty of sample code) for common design patterns, but the first 100 pages will really get your mind wrapped around the whole concept. Alternatively, Fowler has an online catalog of patterns, which is great for those who are already familiar with the concepts but need a handy reference.

Concurrency

The previous examples all dealt with pulling data from the database and creating objects from that data. For the most part, updating, deleting, and inserting data is just as straightforward. Our business layer creates an object, passes it to our Data Access Layer, and lets it handle the mapping to the relational world. For example:

'Visual Basic .NET
Public sub UpdateUser(ByVal user As User)
Dim connection As New SqlConnection(CONNECTION_STRING)
Dim command As New SqlCommand("UpdateUser", connection)
'could have a reusable function to inversly map this
command.Parameters.Add("@UserId", SqlDbType.Int)
command.Parameters(0).Value = user.UserId
command.Parameters.Add("@Password", SqlDbType.VarChar, 64)
command.Parameters(1).Value = user.Password
command.Parameters.Add("@UserName", SqlDbType.VarChar, 128)
command.Parameters(2).Value = user.UserName
Try
 connection.Open()
 command.ExecuteNonQuery()
Finally
 connection.Dispose()
 command.Dispose()
End Try
End Sub

//C#
public void UpdateUser(User user) {
SqlConnection connection = new SqlConnection(CONNECTION_STRING);
SqlCommand command = new SqlCommand("UpdateUser", connection);
//could have a reusable function to inversly map this
command.Parameters.Add("@UserId", SqlDbType.Int);
command.Parameters[0].Value = user.UserId;
command.Parameters.Add("@Password", SqlDbType.VarChar, 64);
command.Parameters[1].Value = user.Password;
command.Parameters.Add("@UserName", SqlDbType.VarChar, 128);
command.Parameters[2].Value = user.UserName;
try{
 connection.Open();
 command.ExecuteNonQuery();
}finally{
 connection.Dispose();
 command.Dispose();
}
}

However, one area which isn't as straightforward is when dealing with concurrency—that is, what happens when two users try to update the same data at the same time? The default behavior (if you don't do anything) is that the last person to commit the data will overwrite all previous work. This probably isn't ideal, as one user's work will be silently overwritten. One way to totally avoid any conflicts is to use pessimistic concurrency; however, this method requires some type of locking mechanism, which can be difficult to implement in a scalable manner. The alternative is to use optimistic concurrency techniques. Letting the first commit dominate and notifying subsequent users is typically a gentler and more user-friendly approach to take. This is achieved by some type of row versioning, such as timestamps.

Further reading:

Performance

Too often we worry about minute performance differences as opposed to legitimate flexibility and capability concerns. While performance is indeed important, providing generalized guidelines on anything but the simplest situations is often difficult. Take, for example, custom collections versus DataSets: which is faster? With custom collections you can make heavy use of DataReaders, which is a faster way of pulling data from a database. The point, though, is that the answer really depends on how, and with what type of data, you use them, so a blanket statement is pretty useless. What's even more important to realize is that whatever processing time you are able to save probably doesn't amount to much compared to the difference in maintainability.

Of course, no one said you couldn't have a high performance solution that is also maintainable. While I reiterate that it really depends on how you use them, there are some patterns that can help maximize performance. First, though, it's important to know that custom entities and collections cache as well as DataSets and can make use of the same mechanism—likely HttpCache. One nice thing about DataSets is the ability to write a Select statement to just grab the necessary information. With custom entities you often feel obliged to populate the entire entity as well as child entities. For example, if you want to display a list of Organizations, with a DataSet you might just pull the OganizationId, Name, and Address and bind it to a repeater. With custom entities I always feel the need to also get all the other Organization information, which might be a bit flag to say if it's ISO certified, a collection of all employees, additional contact information, and so on. Maybe others don't share this hang-up, but luckily we have, if we want, fine control over our custom entities. The most common approach is to use a type of lazy-load pattern, which only fetches the information when it's first required (which can be nicely encapsulated in a property). This type of control over individual properties provides tremendous flexibility otherwise not easily achieved (imagine trying to do something similar at the DataColumn level).

Further Reading:

Lazy Load design pattern
CSLA.NET lazy load

Sorting and Filtering

The DataView's built-in support for sorting and filtering, although requiring knowledge of both SQL and the underlying data structure, is a convenience that is somewhat lost with custom collections. We can still sort and filter, but to do so requires us to write the functionality. While the techniques aren't necessarily advanced, a full demonstration of code is outside the scope of this section. Most techniques are fairly similar, such as using a filter class to filter a collection and a comparer class for sorting, no patterns, that I'm aware of, really exist. A number of resources do exist, however:

Code Generation

Once you get past any conceptual roadblocks, the main drawback to custom entities and collections is the amount of additional code all this flexibility, abstraction, and low maintenance costs you. In fact, you might think that all my talk about reduced maintenance cost and bugs doesn't equate with extra code. While this is certainly a valid point (again, no solution is perfect), design patterns and frameworks such as CSLA.NET go a long way to alleviating the problem. While totally different from patterns and frameworks, code generation tools can reduce the amount of code you actually need to write by significant amounts. Initially this guide was going to have an entire section detailed to code-generation tools, in specific the popular and free CodeSmith; however, numerous resources exist that likely exceed my own knowledge of the product.

Before I continue, I realize that code generation sounds like something of a dream. But when properly used and understood, it truly is a powerful arsenal in your bag of tools—even if you aren't doing custom entities. While it's true that code generation doesn't only apply to custom entities, many are specifically tailored for this purpose. The reason is simple: custom entities require a lot of repetitive code.

Briefly, how does code generation work? The idea might sound far fetched or even counterproductive, but you basically write code (templates) to generate code. CodeSmith, for example, comes with powerful classes that let you hook into a database and get all the properties: tables, columns (types, sizes, and so on), and relations. Armed with this information, much of what we've talked about so far can be automated. For example, a developer could pick a table and automatically, with the right template, have a custom entity created (with the correct fields, properties, and constructors), a mapping function, a custom collection, and basic select, insert, update, and delete functionality. It could even go a step further and implement sorting, filtering, and the other advanced features we've touched on.

CodeSmith also comes with many ready-to-use templates, which serve as a great learning resource. Finally, CodeSmith has a number of templates to implement the CSLA.NET framework. The couple of hours I initially took to learn the basics and get comfortable with CodeSmith have saved me untold time. Additionally, when all developers are using the same templates, the high level of consistency throughout your code makes it easy to work on somebody else's functions.

Further readings:

O/R Mappers

Even though my lack of experience with O/R mappers makes me cautious of talking about them, their potential value makes them impossible to ignore. Where code generators create code that is based on templates for you to copy and paste into your own source code, O/R mappers dynamically generates the code at runtime from some type of configuration mechanism. For example, within an XML file you could specify that column X of some table maps to property Y of an entity. You still create the custom entity, but collections, mappings, and other data access functions (including stored procedures) are all created dynamically. In theory, O/R mappers almost entirely mitigate the problems with custom entities. As the relational and object worlds diverge and the mapping process grows in complexity, O/R mappers become even more invaluable. Two of the downsides of O/R mappers are that they are perceived, in the .NET community at least, as being less secure and having poor performance. From what I've read, I'm convinced that they aren't any less secure, and while they might have poorer performance in some situations, they are probably superior in others. O/R mappers aren't suited for all situations, but if you are dealing with complex systems, you owe it to yourself to investigate them.

Further Reading

.NET Framework 2.0 Features

The upcoming 2.0 release of the .NET Framework will change some of the implementation details we've looked at throughout this guide. These changes will reduce the amount of code necessary to support custom entities as well as help deal with mapping issues.

Generics

One of the main reasons for the existence of the much talked about generics is to provide developers with strongly-typed collections out of the box. We shied away from the existing collections such as Arraylists because of their weakly-typed nature. Generics provide the same kind of conveniences as current collections, but in a strongly-typed manner. This is achieved by specifying the type at declaration. For example, we could replace our UserCollection with no additional code, and simply create a new instance of the List generic and specify our User class:

'Visual Basic .NET
Dim users as new IList(of User)

//C#
IList users = new IList();

Once declared, our users collection can only deal with objects of type User, which provides us with all the niceties of compile-time checks and optimizations.

Further Reading

Nullable Types

Nullable types are actually generics that are used for different reasons than those listed previously. One of the challenges faced when dealing with databases is the proper and consistent handling of columns that support NULL. When dealing with string and other classes (known as reference types), you can simply assign nothing/null to a variable in your code:

'Visual Basic .NET
if dr("UserName") Is DBNull.Value Then
  user.UserName = nothing
End If

//C#
if (dr["UserName"] == DBNull.Value){
  user.UserName = null;
}

Or you could simply do nothing (by default, reference types are nothing/null). This doesn't work nearly as well for value types such as integers, booleans, decimals, and so on. You can certainly assign nothing/null to such values, but this will assign a default value. If you just declare an integer, or assign nothing/null to it, the variable will actually hold the value 0. This makes it difficult to map back to the database: is the value 0 or null? Nullable types solve this problem by allowing value types to either hold an actual value or null. For example, if we wanted to support a null value in the userId column (not exactly realistic), we'd first declare our userId field and corresponding property as a nullable type:

'Visual Basic .NET
Private _userId As Nullable(Of Integer)
Public Property UserId() As Nullable(Of Integer)
  Get
     Return _userId
  End Get
  Set(ByVal value As Nullable(Of Integer))
     _userId = value
  End Set
End Property


//C#
private Nullable userId;
public Nullable UserId {
  get { return userId; }
  set { userId = value; }
}

And then make use of the HasValue property to determine whether nothing/null was assigned:

'Visual Basic .NET
If UserId.HasValue Then
  Return UserId.Value
Else
  Return DBNull.Value
End If

//C#
if (UserId.HasValue) {
  return UserId.Value;
} else {
  return DBNull.Value;
}

Further Reading:

Iterators

The UserCollection example that we looked at represents only the base functionality that you'll likely need in your custom collection. Something that you won't be able to do with the provided implementation is loop through the collection in a foreach loop. To do so, your custom collection must have an enumerator support-class that implements the IEnumerable interface. This is a fairly straightforward and repetitive process, but nonetheless introduces even more code. C# 2.0 introduces the new yield keyword to handle the implementation detail of this interface for you. There is currently no Visual Basic .NET equivalent to the new yield keyword.

Further Readings:

Conclusion

Making the switch to custom entities and collections shouldn't be a decision you make lightly. Numerous factors exist that need to be taken into consideration. For example, your familiarity with OO concepts, the time you have to play with this new approach, as well as the environment you are thinking of deploying it in. While the advantages are significant in general, they may not be in your particular situation. Even if they are significant in your case, the drawbacks may negate them. Also keep in mind that numerous alternative solutions exist. Jimmy Nilsson has an overview of some of these alternatives in his 5 part series Choosing Data Containers for .NET (part 1, 2, 3, 4, and 5).

Custom entities empower you with the rich capabilities of object-oriented programming, as well as help you set up the Framework for a solid, maintainable N-Tier architecture. One of the goals of this guide is to make you think of your system in terms of the business entities that make it up, instead of generic DataSets and DataTable. We've also touched on some key issues you should be aware of regardless of the route you chose, namely design patterns, differences between the object and relational world (read more), and N-Tier architecture. Remember that time spent upfront has a way of paying for itself a number of times over throughout the life of a system.

Related Books

Implementing Page Controller in ASP.NET

2005-09-01T14:26:00.000+07:00

Implementing Page Controller in ASP.NET

Version 1.0.1

GotDotNet community for collaboration on this pattern

Complete List of patterns & practices

Context

You are building a Web application in ASP.NET and you want to take
advantage of the event-driven nature of ASP.NET by using the built-in
page controller.

Implementation Strategy

The concepts described in the Page Controller
pattern are implemented in ASP.NET by default. The ASP.NET page
framework implements these concepts in such a way that the underlying
mechanism of capturing an event on the client, transmitting it to the
server, and calling the appropriate method is automatic and invisible
to the implementer. The page controller is extensible in that it
exposes various events at specific points in the life cycle (see "Page
Life Cycle," later in this pattern) so that application-specific
actions can be run when they are appropriate.

For example, assume the user is interacting with a Web Forms page
that contains one button server control (see "Simple Page Example,"
later in this pattern). When the user clicks the button control, an
event is transmitted as an HTTP post to the server, where the ASP.NET
page framework interprets the posted information and associates the
raised event with an appropriate event handler. The framework
automatically calls the appropriate event handler for the button as
part of the framework's normal processing. As a result, you no longer
need to implement this functionality. Furthermore, you can use the
built-in controller, or you can replace the built-in controller with
you own customized controller (see Front Controller).

Page Life Cycle

The following list contains the most common stages of the page
life cycle in the order in which they occur. It also includes the
specific events that are raised and some typical actions that could be
performed at the various stages in the processing of the request:

ASP.NET page framework initialization (Event: Init). This is the first step in the life cycle, which initializes the ASP.NET runtime for the request.

User code initialization (Event: Load). You should
perform common tasks specific to your application, such as opening
database connections, when the page controller raises the Load event. You can assume that when the Load
event is raised, server controls are created and initialized, state has
been restored, and form controls reflect client-side changes. [Reilly02]

Application-specific event handling. At this stage, you should perform processing specific to your application in response to the events raised by the controller.

Cleanup (Event: Unload). The page has finished rendering and is ready to be discarded. You should close any database connections that the Load
event opened and discard any objects that are no longer needed. The
Microsoft.NET Framework closes database connections automatically,
after the connection object is garbage collected. However, you do not
have any control over when the garbage collection occurs. Therefore, it
is good practice to close database connections explicitly to make
efficient use of the database connection pool.

Note: There are several more stages of page processing
than are listed here. However, they are not used for most page
processing scenarios.

Simple Page Example

The first example is a simple page that takes input from the
user and then displays the input on the screen. The example illustrates
the event-driven model that ASP.NET uses to implement server controls.

Figure 1: Simple page

When the user types his or her name and then clicks the Click
Here button, the name appears directly below the button, as shown in
Figure 2.

Figure 2: Simple page displaying user input

In ASP.NET pages, the user interface programming is divided into
two distinct pieces: the visual component, or view, and the logic,
which is a combination of the model and the controller. This division
separates the visible portion of the page (the view) from the code
behind the page with which the page interacts (model and controller).

The visual element is called the Web Forms page. The page
consists of a file containing static HTML or ASP.NET server controls,
or both simultaneously. For this example, the Web Forms page is named
SimplePage.aspx and consists of the following code:



<%@ Page language="c#" Codebehind="SimplePage.aspx.cs" AutoEventWireup="false" Inherits="SimplePage" %>
<HTML>
 <body>
    <form id="Form1" runat="server">
       Name:<asp:textbox id="name" runat="server" />
       <p />
       <asp:button id="MyButton" text="Click Here" OnClick="SubmitBtn_Click" runat="server" />
       <p />
       <span id="mySpan" runat="server"></span>
    </form>
 </body>
</HTML>

The
logic for the Web Forms page consists of code that you create to
interact with the form. The programming logic resides in a file that is
separate from the user interface file. This file, referred to as the code-behind file, is named SimplePage.aspx.cs:



using System;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.UI.HtmlControls;

public class SimplePage : System.Web.UI.Page
{
 protected System.Web.UI.WebControls.TextBox name;
 protected System.Web.UI.WebControls.Button MyButton;
 protected System.Web.UI.HtmlControls.HtmlGenericControl mySpan;

 public void SubmitBtn_Click(Object sender, EventArgs e)
 {
    mySpan.InnerHtml = "Hello, " + name.Text + ".";
 }
}

The
purpose of this code is to indicate to the page controller that when
the user clicks the button, a request will be sent back to the server
and the SubmitBtn_Click function will be executed.

This implementation shows how simple it is to connect to the events
that the controller provides. It also illustrates that code written in
this fashion is easier to understand because the application logic is
not combined with the low-level code that manages the event
dispatching.

Common Look and Feel Example

The following example uses a typical implementation strategy of
the page controller to provide a banner that displays dynamic content:
the authenticated user's e-mail address (which is retrieved from the
database) on every page in the application.

The common implementation is contained in a base class from
which all of the page objects in the site inherit. Figure 3 shows one
of the pages in the site.

Figure 3: Banner displaying dynamic content

The individual pages in the site are responsible for rendering
their own content, while the base class is responsible for rendering
the header. Because the individual pages inherit from the base class,
they all have the same functionality.

This implementation uses a design pattern called Template Method. The pattern defines the skeleton of an algorithm in an operation, deferring some steps to subclasses. Template Method lets subclasses redefine certain steps of an algorithm without changing the algorithm's structure. [Gamma95]

Applying Template Method to this problem involves moving
common code from the individual pages into a base class. This ensures
that the common code is contained in one place and is easily
maintainable. In this example, the base class is named BasePage and is responsible for connecting the Page_Load method to the Load event. After the work associated with the BasePage; which is retrieving the user's e-mail address from the database and setting the site name, the Page_Load function calls a method named PageLoadEvent. Subclasses implement PageLoadEvent to perform their own specific Load functionality. Figure 4 shows the structure of this solution.

Figure 4: Structure of the code-behind pages implementation

When a page is requested, the ASP.NET runtime fires the Load event, which in turn calls the Page_Load method on BasePage. The BasePage method retrieves the data it needs and then calls PageLoadEvent
on the specific page that was requested to perform any page-specific
loading that is required. Figure 5 shows the page request sequence.

Figure 5: Page request sequence

Implementing the common functionality in this manner frees the
pages from having to set up the header and also allows for site-wide
changes to be made easily. If the header rendering and initialization
code is not contained in a single file, the changes must be made to all
files that contain code that is related to the header.

BasePage.cs

The code for the base class implements the following functionality:

Connects the Load event to the Page_Load method for request-specific initialization.

Retrieves the authenticated user's name from the request context and using the DatabaseGateway class finds the user's record in the database. The code assigns the eMail label to the user's e-mail address.

Assigns the site name to the siteName label.

Calls the PageLoadEvent method, which derived classes can implement for any page-specific loading.

Note: It would be better to define the BasePage class as abstract, because doing so would force the implementers to provide an implementation for PageLoadEvent.
However, in Microsoft Visual Studio .NET, it is not possible to define
this base class as abstract. Instead, the class provides a default
implementation that can be overridden by derived classes.



using System;
using System.Web.UI;
using System.Web.UI.WebControls;

public class BasePage : Page
{
 protected Label eMail;
 protected Label siteName;

 virtual protected void PageLoadEvent(object sender, System.EventArgs e)
 {}

 protected void Page_Load(object sender, System.EventArgs e)
 {
       if(!IsPostBack)
       {
             string name = Context.User.Identity.Name;

             eMail.Text = DatabaseGateway.RetrieveAddress(name);
             siteName.Text = "Micro-site";

             PageLoadEvent(sender, e);
       }
 }


 #region Web Form Designer generated code
 override protected void OnInit(EventArgs e)
 {
    //
    //
    // CODEGEN: This call is required by the ASP.NET Web Form Designer.
       //
       InitializeComponent();
       base.OnInit(e);
 }
     
 /// <summary>
 /// Required method for Designer support - do not modify
 /// the contents of this method with the code editor.
 /// </summary>
 private void InitializeComponent()
 {  
       this.Load += new System.EventHandler(this.Page_Load);

 }
 #endregion
}

BasePage.inc

Not only do you have to provide a common base class for the
logic code behind the page, but you also have to provide a common file
that holds the view or UI rendering code. The code is included in each
.aspx page. This HTML file is not intended to be displayed on its own.
Using a common file enhances your ability to make changes in one place
and have them propagate to all the pages that include the file. The
following example code shows the common file for this example, named
BasePage.inc:



<table width="100%" cellspacing="0" cellpadding="0">
 <tr>
    <td align="right" bgcolor="#9c0001" cellspacing="0" cellpadding="0" width="100%" height="20">
       <font size="2" color="#ffffff">Welcome:
       <asp:Label id="eMail" runat="server">username</asp:Label>  </font>
    </td>
 </tr>
 <tr>
    <td align="right" width="100%" bgcolor="#d3c9c7" height="70">
       <font size="6" color="#ffffff">
       <asp:Label id="siteName" Runat="server">Micro-site Banner</asp:Label>  </font>
    </td>
 </tr>
</table>

DatabaseGateway.cs

This class encapsulates all access to the database for these pages. This is an example of a Table Data Gateway [Fowler03] which represents the model code for the pages in this application.

using System;
using System.Collections;
using System.Data;
using System.Data.SqlClient;

public class DatabaseGateway
{
 public static string RetrieveAddress(string name)
 {
       String address = null;

       String selectCmd =
             String.Format("select * from webuser where (id = '{0}')",
             name);

       SqlConnection myConnection =
             new SqlConnection("server=(local);database=webusers;Trusted_Connection=yes");
       SqlDataAdapter myCommand = new SqlDataAdapter(selectCmd, myConnection);

       DataSet ds = new DataSet();
       myCommand.Fill(ds,"webuser");
       if(ds.Tables["webuser"].Rows.Count == 1)
       {
             DataRow row = ds.Tables["webuser"].Rows[0];
             address = row["address"].ToString();
       }

       return address;
 }
}

Page1.aspx

The following is an example of how to use the common functionality in a page:



<%@ Page language="c#" Codebehind="Page1.aspx.cs" AutoEventWireup="false" Inherits="Page1" %>
<HTML>
 <HEAD>
    <title>Page-1</title>
 </HEAD>
 <body>
    <!-- #include virtual="BasePage.inc" -->
    <form id="Page1" method="post" runat="server">
       <h1>Page:
          <asp:label id="pageNumber" Runat="server">NN</asp:label></h1>
    </form>
 </body>
</HTML>

The following directive from the file loads the common HTML for the header:



<!-- #include virtual="BasePage.inc" -->

Page1.aspx.cs

The code-behind class must inherit from the BasePage class and then implement the PageLoadEvent method to do any page-specific loading. In this example, the page-specific activity is to assign the number 1 to the pageNumber label.



using System;
using System.Web.UI;
using System.Web.UI.WebControls;

public class Page1 : BasePage
{
 protected System.Web.UI.WebControls.Label pageNumber;

 protected override void PageLoadEvent(object sender, System.EventArgs e)
 {
    pageNumber.Text = "1";
 }
}

Testing Considerations

The dependence on the ASP.NET runtime makes testing of the
implementation difficult. It is not possible to instantiate classes
that inherit from System.Web.UI.Page or the other various classes
contained in the environment. This makes it impossible to unit test the
individual pieces of the application in isolation. The only remaining
way to test this implementation automatically is to generate HTTP
requests and then retrieve the HTTP response and determine if the
response is correct. This approach is prone to error because you are
comparing the text of the response with expected text.

Resulting Context

The built-in ASP.NET page controller functionality results in the following benefits and liabilities:

Benefits

Takes advantage of framework features. The page
controller functionality is built into ASP.NET and can be easily
extended by connecting application-specific actions to the events
exposed by the controller. It is also easy to separate the
controller-specific code from the model and view code by using the
code-behind feature.

Explicit URLs. The URL that the user enters refers to an
actual page in the application. This means that the pages can be
bookmarked and entered later. The URLs also tend to have fewer
parameters making them easier for users to enter.

Increases modularity and reuse. The Common Look and Feel example demonstrated how you could reuse BasePage for many pages without having to modify the BasePage class or HTML file.

Liabilities

Requires code changes. To share common functionality, as
demonstrated in the Common Look and Feel example, the individual pages
have to be modified to inherit from the newly defined base class
instead of System.Web.UI.Page. The Intercepting Filter pattern describes a mechanism for adding common functionality by changing the Web.config file and not the pages themselves.

Uses inheritance. The Common Look and Feel example uses
inheritance to share the implementation across multiple pages. Most
programmers who learn object-oriented programming initially like
inheritance. However, using inheritance to share implementation can
often lead to software that is difficult to change. If the base class
become complicated with conditional logic, it is better to introduce
helper classes or to consider using Front Controller instead.

Difficult to test. Because the page controller is
implemented in ASP.NET, it is difficult to test in isolation. To
improve the testability, you should separate as much functionality out
of the ASP.NET - specific code in classes that do not depend on
ASP.NET. This enables you to test without having to start the ASP.NET
runtime.

Related Patterns

For more information, see the following related patterns:

Template Method [Gamma95]. The BasePage class and the Page_Load method are an example implementation of this pattern.

Intercepting Filter.

Front Controller.

Acknowledgments

[Gamma95] Gamma, Helm, Johnson, and Vlissides. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, 1995.

[Reilly02] Reilly, Douglas J. Designing Microsoft ASP.NET Applications. Microsoft Press, 2002.

[Fowler03] Fowler, Martin. Patterns of Enterprise Application Architecture. Addison-Wesley, 2003.

Implementing Page Controller in ASP.NET

2005-09-01T14:10:00.000+07:00

Implementing Page Controller in ASP.NET

Version 1.0.1

GotDotNet community for collaboration on this pattern

Complete List of patterns & practices

Context

You
are building a Web application in ASP.NET and you want to take
advantage of the event-driven nature of ASP.NET by using the built-in
page controller.

Implementation Strategy

For example,
assume the user is interacting with a Web Forms page that contains one
button server control (see "Simple Page Example," later in this
pattern). When the user clicks the button control, an event is
transmitted as an HTTP post to the server, where the ASP.NET page
framework interprets the posted information and associates the raised
event with an appropriate event handler. The framework automatically
calls the appropriate event handler for the button as part of the
framework's normal processing. As a result, you no longer need to
implement this functionality. Furthermore, you can use the built-in
controller, or you can replace the built-in controller with you own
customized controller (see Front Controller).

Page Life Cycle

The
following list contains the most common stages of the page life cycle
in the order in which they occur. It also includes the specific events
that are raised and some typical actions that could be performed at the
various stages in the processing of the request:

ASP.NET page framework initialization (Event: Init). This is the first step in the life cycle, which initializes the ASP.NET runtime for the request.

User code initialization (Event: Load).
You should perform common tasks specific to your application, such as
opening database connections, when the page controller raises the Load event. You can assume that when the Load
event is raised, server controls are created and initialized, state has
been restored, and form controls reflect client-side changes. [Reilly02]

Application-specific event handling. At this stage, you should perform processing specific to your application in response to the events raised by the controller.

Note:
There are several more stages of page processing than are listed here.
However, they are not used for most page processing scenarios.

Simple Page Example

The
first example is a simple page that takes input from the user and then
displays the input on the screen. The example illustrates the
event-driven model that ASP.NET uses to implement server controls.

Figure 1: Simple page

When
the user types his or her name and then clicks the Click Here button,
the name appears directly below the button, as shown in Figure 2.

Figure 2: Simple page displaying user input

In
ASP.NET pages, the user interface programming is divided into two
distinct pieces: the visual component, or view, and the logic, which is
a combination of the model and the controller. This division separates
the visible portion of the page (the view) from the code behind the
page with which the page interacts (model and controller).

The visual element is called the Web Forms page.
The page consists of a file containing static HTML or ASP.NET server
controls, or both simultaneously. For this example, the Web Forms page
is named SimplePage.aspx and consists of the following code:


<%@ Page language="c#" Codebehind="SimplePage.aspx.cs" AutoEventWireup="false" Inherits="SimplePage" %>

  
     
        Name:


using System;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.UI.HtmlControls;

public class SimplePage : System.Web.UI.Page
{
  protected System.Web.UI.WebControls.TextBox name;
  protected System.Web.UI.WebControls.Button MyButton;
  protected System.Web.UI.HtmlControls.HtmlGenericControl mySpan;

  public void SubmitBtn_Click(Object sender, EventArgs e)
  {
     mySpan.InnerHtml = "Hello, " + name.Text + ".";
  }
}

The purpose of this code is to indicate to the page controller
that when the user clicks the button, a request will be sent back to
the server and the SubmitBtn_Click function will be executed.

This
implementation shows how simple it is to connect to the events that the
controller provides. It also illustrates that code written in this
fashion is easier to understand because the application logic is not
combined with the low-level code that manages the event dispatching.

Common Look and Feel Example

The
following example uses a typical implementation strategy of the page
controller to provide a banner that displays dynamic content: the
authenticated user's e-mail address (which is retrieved from the
database) on every page in the application.

The
common implementation is contained in a base class from which all of
the page objects in the site inherit. Figure 3 shows one of the pages
in the site.

Figure 3: Banner displaying dynamic content

The
individual pages in the site are responsible for rendering their own
content, while the base class is responsible for rendering the header.
Because the individual pages inherit from the base class, they all have
the same functionality.

Applying Template Method
to this problem involves moving common code from the individual pages
into a base class. This ensures that the common code is contained in
one place and is easily maintainable. In this example, the base class
is named BasePage and is responsible for connecting the Page_Load method to the Load event. After the work associated with the BasePage; which is retrieving the user's e-mail address from the database and setting the site name, the Page_Load function calls a method named PageLoadEvent. Subclasses implement PageLoadEvent to perform their own specific Load functionality. Figure 4 shows the structure of this solution.

Figure 4: Structure of the code-behind pages implementation

Figure 5: Page request sequence

Implementing
the common functionality in this manner frees the pages from having to
set up the header and also allows for site-wide changes to be made
easily. If the header rendering and initialization code is not
contained in a single file, the changes must be made to all files that
contain code that is related to the header.

BasePage.cs

The code for the base class implements the following functionality:

Connects the Load event to the Page_Load method for request-specific initialization.

Assigns the site name to the siteName label.

Calls the PageLoadEvent method, which derived classes can implement for any page-specific loading.

Note: It would be better to define the BasePage class as abstract, because doing so would force the implementers to provide an implementation for PageLoadEvent.
However, in Microsoft Visual Studio .NET, it is not possible to define
this base class as abstract. Instead, the class provides a default
implementation that can be overridden by derived classes.


using System;
using System.Web.UI;
using System.Web.UI.WebControls;

public class BasePage : Page
{
  protected Label eMail;
  protected Label siteName;

  virtual protected void PageLoadEvent(object sender, System.EventArgs e)
  {}
 
  protected void Page_Load(object sender, System.EventArgs e)
  {
        if(!IsPostBack)
        {
              string name = Context.User.Identity.Name;

              eMail.Text = DatabaseGateway.RetrieveAddress(name);
              siteName.Text = "Micro-site";

              PageLoadEvent(sender, e);
        }
  }


  #region Web Form Designer generated code
  override protected void OnInit(EventArgs e)
  {
     //
     //
     // CODEGEN: This call is required by the ASP.NET Web Form Designer.
        //
        InitializeComponent();
        base.OnInit(e);
  }
       
  /// 
  /// Required method for Designer support - do not modify
  /// the contents of this method with the code editor.
  /// 

  private void InitializeComponent()
  {   
        this.Load += new System.EventHandler(this.Page_Load);

  }
  #endregion
}

BasePage.inc

Not
only do you have to provide a common base class for the logic code
behind the page, but you also have to provide a common file that holds
the view or UI rendering code. The code is included in each .aspx page.
This HTML file is not intended to be displayed on its own. Using a
common file enhances your ability to make changes in one place and have
them propagate to all the pages that include the file. The following
example code shows the common file for this example, named BasePage.inc:









       
        Welcome:
        username  
       
       
        
        Micro-site Banner

DatabaseGateway.cs

This class encapsulates all access to the database for these pages. This is an example of a Table Data Gateway [Fowler03] which represents the model code for the pages in this application.

using System;
using System.Collections;
using System.Data;
using System.Data.SqlClient;

public class DatabaseGateway
{
  public static string RetrieveAddress(string name)
  {
        String address = null;

        String selectCmd =
              String.Format("select * from webuser where (id = '{0}')",
              name);

        SqlConnection myConnection =
              new SqlConnection("server=(local);database=webusers;Trusted_Connection=yes");
        SqlDataAdapter myCommand = new SqlDataAdapter(selectCmd, myConnection);

        DataSet ds = new DataSet();
        myCommand.Fill(ds,"webuser");
        if(ds.Tables["webuser"].Rows.Count == 1)
        {
              DataRow row = ds.Tables["webuser"].Rows[0];
              address = row["address"].ToString();
        }

        return address;
  }
}

Page1.aspx

The following is an example of how to use the common functionality in a page:

Page-1
<%@ Page language="c#" Codebehind="Page1.aspx.cs" AutoEventWireup="false" Inherits="Page1" %>

  
     
  
     
     
        Page:
           NN

The following directive from the file loads the common HTML for the header:

Page1.aspx.cs


using System;
using System.Web.UI;
using System.Web.UI.WebControls;

public class Page1 : BasePage
{
  protected System.Web.UI.WebControls.Label pageNumber;

  protected override void PageLoadEvent(object sender, System.EventArgs e)
  {
     pageNumber.Text = "1";
  }
}

Testing Considerations

The
dependence on the ASP.NET runtime makes testing of the implementation
difficult. It is not possible to instantiate classes that inherit from
System.Web.UI.Page or the other various classes contained in the
environment. This makes it impossible to unit test the individual
pieces of the application in isolation. The only remaining way to test
this implementation automatically is to generate HTTP requests and then
retrieve the HTTP response and determine if the response is correct.
This approach is prone to error because you are comparing the text of
the response with expected text.

Resulting Context

The built-in ASP.NET page controller functionality results in the following benefits and liabilities:

Benefits

Takes advantage of framework features.
The page controller functionality is built into ASP.NET and can be
easily extended by connecting application-specific actions to the
events exposed by the controller. It is also easy to separate the
controller-specific code from the model and view code by using the
code-behind feature.

Explicit URLs. The URL
that the user enters refers to an actual page in the application. This
means that the pages can be bookmarked and entered later. The URLs also
tend to have fewer parameters making them easier for users to enter.

Increases modularity and reuse. The Common Look and Feel example demonstrated how you could reuse BasePage for many pages without having to modify the BasePage class or HTML file.

Liabilities

Requires code changes.
To share common functionality, as demonstrated in the Common Look and
Feel example, the individual pages have to be modified to inherit from
the newly defined base class instead of System.Web.UI.Page. The Intercepting Filter pattern describes a mechanism for adding common functionality by changing the Web.config file and not the pages themselves.

Uses inheritance.
The Common Look and Feel example uses inheritance to share the
implementation across multiple pages. Most programmers who learn
object-oriented programming initially like inheritance. However, using
inheritance to share implementation can often lead to software that is
difficult to change. If the base class become complicated with
conditional logic, it is better to introduce helper classes or to
consider using Front Controller instead.

Difficult to test.
Because the page controller is implemented in ASP.NET, it is difficult
to test in isolation. To improve the testability, you should separate
as much functionality out of the ASP.NET - specific code in classes
that do not depend on ASP.NET. This enables you to test without having
to start the ASP.NET runtime.

Related Patterns

For more information, see the following related patterns:

Template Method [Gamma95]. The BasePage class and the Page_Load method are an example implementation of this pattern.

Intercepting Filter.

Front Controller.

Acknowledgments

[Gamma95] Gamma, Helm, Johnson, and Vlissides. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, 1995.

[Reilly02] Reilly, Douglas J. Designing Microsoft ASP.NET Applications. Microsoft Press, 2002.

[Fowler03] Fowler, Martin. Patterns of Enterprise Application Architecture. Addison-Wesley, 2003.

Page Controller

2005-09-01T14:09:00.000+07:00

Page Controller

Version 1.0.1

GotDotNet community for collaboration on this pattern

Complete List of patterns & practices

Context

You decided to use the Model-View-Controller (MVC) pattern to separate the user interface components of your dynamic Web application from the business logic. The application you are building constructs the Web pages dynamically, but the navigation between the pages is mostly static.

Problem

How do you best structure the controller for moderately complex Web applications so that you can achieve reuse and flexibility while avoiding code duplication?

Forces

The following forces act on a system within this context and must be reconciled as you consider a solution to the problem:

The MVC pattern often focuses primarily on the separation between the model and the view, while paying less attention to the controller. In many rich-client scenarios, the separation between controller and view is less critical and is often omitted [Fowler03]. In a thin-client application, however, view and controller are inherently separated because the presentation occurs in the client browser, whereas the controller is part of the server-side application. The controller, therefore, warrants a closer examination.

In dynamic Web applications, multiple user actions can lead to different controller logic, followed by the same page presentation. For example, in a simple Web-based e-mail application, both sending a message and deleting a message from the inbox is likely to return the user to the (refreshed) inbox page. Although the same page is rendered after either activity, the application must perform a different action, based on the previous page and the button the user clicked.

The code that renders most dynamic Web pages involves very similar steps: verifying user authentication, extracting the page parameters from the query string or the form fields, gathering session information, retrieving data from a data source, rendering the dynamic portion of the page, and adding applicable headers and footers. This can lead to a significant amount of code duplication.

Scripted server pages (such as ASP.NET) may be easy to create, but can introduce a number of disadvantages as the application grows. Scripted pages provide poor separation between controller and the view, which reduces the opportunities for reuse. For example, if multiple actions lead to the same page, it is difficult to reuse the display code across multiple controllers, because it is intertwined with the controller code. Scripted server pages that intersperse business logic and presentation logic are also more difficult to test and debug. Finally, developing scripted server pages requires expertise both in developing business logic and in rendering visually appealing and efficient HTML pages; these two skill sets are rarely possessed by a single person. Due to these considerations, it makes sense to minimize the scripted server-page code and develop business logic in actual classes.

As described in the MVC pattern, testing user interface code tends to be time-consuming and tedious. If you can separate user-interface-specific code from the actual business logic, testing the business logic becomes simpler and more repeatable. This is true not only for the presentation portion, but also for the controller part of an application.

Common appearance and navigation tend to improve usability and brand recognition of a Web application. However, common appearance can lead to repetitive presentation code, especially if the code is embedded inside scripted server pages. Therefore, you need a mechanism for improving reuse of presentation logic across pages.

Solution

Use the Page Controller pattern to accept input from the page request, invoke the requested actions on the model, and determine the correct view to use for the resulting page. Separate the dispatching logic from any view-related code. Where appropriate, create a common base class for all page controllers to avoid code duplication and increase consistency and testability. Figure 1 shows how the page controller relates to the model and view.

Figure 1: Page Controller structure

The page controller receives a page request, extracts any relevant data, invokes any updates to the model, and forwards the request to the view. The view in turn depends on the model for retrieval of data to be displayed. Defining a separate page controller isolates the model from the specifics of the Web request - for example, session management, or the use of query strings or hidden form fields to pass parameters to the page. In this basic form, you create a controller for every link in the Web application. This keeps the controllers simple, because you only have to concern yourself with one action at a time.

Creating a separate controller for each Web page (or action) can lead to significant code duplication. Therefore, you should create a BaseController class to incorporate common functions such as validating parameters (see Figure 2). Each individual page controller can inherit this common functionality from BaseController. In addition to inheriting from a common base class, you can also define a set of helper classes that the controllers can call to perform common functions.

Figure 2: Using BaseController to eliminate code duplication

This approach works well if most pages are similar and you can pull the common functions into a single base class. The more page variations you have, the more levels you may have to inject into the inheritance tree. Let's say that all pages parse parameters, but only pages that display lists retrieve data from the database, while pages that require data entry update the model, rather than retrieve data. You could now introduce two new base classes, ListController and DataEntryController, that both inherit from BaseController. The list pages could then inherit from ListController, and the data entry pages could inherit from DataEntryController. Although this approach may work well in this simple example, the inheritance tree can get rather deep and complicated if you are dealing with a real-life businesses application. You may be tempted to add conditional logic into the base classes to accommodate some of the variants, but doing so violates the principles of encapsulation and makes the base classes a notorious bottleneck for any changes to the system. Therefore, you should consider using helper classes or the Front Controller pattern as your application becomes more complex.

Using a page controller for a Web application is such a common need that most Web application frameworks provide a default implementation of the page controller. Most frameworks incorporate the page controller in the form of a server page (for example, ASP, JSP, and PHP). Server pages actually combine the functions of view and controller and do not provide the desired separation between the presentation code and the controller code. Unfortunately, some of the frameworks make it very easy to blend together view-related code with controller-related code and make it difficult to properly separate the controller logic. As a result, the Page Controller approach has developed a bad reputation with many developers. Now, many developers associate Page Controller with bad design and Front Controller with good design. This perception, in fact, resulted from a specific (faulty) implementation choice; both Page Controller and the Front Controller are perfectly viable architectural choices.

Therefore, it is preferable to separate the controller logic into separate classes that can be called from the server page. The ASP.NET page framework provides an excellent mechanism for achieving this separation, called code-behind classes. (See Implementing Page Controller in ASP.NET).

Variants

In most cases, the page controller is dependent on the specifics of an HTTP-based Web request. As a result, the page controllercode usually contains references to HTTP headers, query strings, form fields, multipart form requests, and so forth. This makes it very hard to test the controller code outside the Web application framework. The only option is to test the controller by simulating HTTP requests and parsing the results. This type of testing is both time-consuming and error prone. Therefore, to improve testability you could separate the Web-dependent and the Web-independent code into two separate classes (see Figure 3).

Figure 3: Separating the Web-dependent and Web-independent code

In this example, AspNetController encapsulates all dependencies on the application framework (ASP.NET). For example, it can extract all incoming parameters from the Web request and pass it to BaseController in a way that is independent from the Web interface (for example, in a collection). This approach not only improves testability, but enables you to reuse the controller code with other user interfaces, for example a rich-client interface or a custom scripting language.

The downside of this approach is the additional overhead. You now have an additional class, and each request has to go through a translation before it can be serviced. Therefore, you should keep the environment-specific part of the controller as thin as possible and consider the tradeoffs between reduced dependencies and more efficient development and execution.

Example

See Implementing Page Controller in ASP.NET.

Resulting Context

Using the Page Controller pattern results in a number of benefits and liabilities.

Benefits

Simplicity. Because each dynamic Web page is handled by a specific controller, the controllers have to deal with only a limited scope and can remain simple. Because each page controller deals with only a single page, Page Controller is particularly well-suited for Web applications with simple navigation.

Built-in framework features. In its most basic form, the controller is already built into most Web application platforms. For example, if the user clicks a link in a Web page that leads to a dynamic page generated by an ASP.NET script, the Web server analyzes the URL associated with the link and executes the associated ASP.NET page. In effect, the ASP.NET page is the controller for the action taken by the user. The ASP.NET page framework also provides code-behind classes to execute controller code. Code-behind classes provide better separation between the controller and the view and also allow you to create a controller base class that incorporates common functionality across all controllers. For an example, see Implementing Page Controller in ASP.NET.

Increased reuse. Creating a controller base class reduces code duplication and enables you to reuse common code across page controllers. You can reuse code by implementing recurring logic in the base class. This logic is then automatically inherited by all concrete Page Controller objects. If the implementation of the logic varies from page to page, you can still use Template Method and implement the basic execution structure in the base class; the implementation of specific substeps may vary from page to page.

Expandability. You can expand a page controller quite easily by using helper classes. If the logic inside the controller becomes too complex, you can delegate some of the logic to helper classes. Helper classes also provide another mechanism for reuse, besides inheritance.

Separation of developer responsibilities. Using a Page Controller class helps separate responsibilities among members of the development team. The developer of the controller must be familiar with the domain model and the business logic implemented by the application. The designer of the view, on the other hand, can focus on the presentation style of the results.

Liabilities

Due to its simplicity, Page Controller is the default implementation for most dynamic Web applications. However, you should be aware of the following limitations:

One controller per page. The key constraint of Page Controller is that you create one controller for each Web page. This works well for applications with a static set of pages and a simple navigation path. Some more complex applications require dynamic configuration of pages and navigation maps between them. Spreading this logic across many page controllers would make the application hard to maintain, even if some of the logic could be pulled into the base controller. In addition, the built-in features of the Web framework may reduce the amount of flexibility you have in naming URLs and resource paths (even though you can compensate for some of this with low-level mechanisms like ISAPI filters). In these scenarios, consider using Front Controller that intercepts all Web requests and forwards the request to the appropriate handler, based on configurable rules.

Deep inheritance trees. Inheritance seems to be one of the most loved and most hated features of object-oriented programming. Using inheritance alone to reuse common functionality may lead to inflexible inheritance hierarchies. For more detail, see Implementing Page Controller in ASP.NET.

Dependency on the Web framework. In the basic form, the page controller still depends on the Web application environment and cannot be tested independently. You can use a wrapper mechanism to decouple the Web-dependent part, but doing so requires an additional level of indirection.

Testing Considerations

Because Page Controller is dependent on specifics of the Web application framework (for example, query strings and HTTP headers), you cannot instantiate and test the controller classes outside the Web framework. If you want to run a suite of automated unit tests on the controller class, you would have to start the Web application server for each test case. You would then have to submit HTTP requests in a format that executes the desired function. This configuration introduces many dependencies and side effects into the test. To improve testability, consider separating the business logic (including controller logic as it becomes more complex) from the Web-dependent code.

Related Patterns

For more information, see the following related patterns:

Intercepting Filter. This pattern is another construct to implement recurring functionality inside a Web application. The Web server framework can pass each request through a configurable chain of filters before passing it to the controller. Filters tend to deal with lower-level functions such as decoding, authentication, and session management, whereas Page Controller deals with application functionality. Filters also are not usually page-specific.

Front Controller. This pattern is a more complex, but also more powerful alternative to Page Controller. Front Controller defines a single controller for all page requests, which enables it to make navigational decisions that span multiple pages.

Model-View-Controller. Page Controller is an implementation variant of the controller portion of MVC.

Acknowledgments

[Fowler03] Fowler, Martin. Patterns of Enterprise Application Architecture. Addison-Wesley, 2003.

[Gamma95] Gamma, Helm, Johnson, and Vlissides. Design Patterns: Element of Reusable Object-Oriented Software. Addison-Wesley, 1995.

Recent reading

Continuous Integration

Building a Feature with Continuous Integration

Practices of Continuous Integration

Maintain a Single Source Repository.

Automate the Build

Make Your Build Self-Testing

Everyone Commits Every Day

Every Commit Should Build the Mainline on an Integration Machine

Keep the Build Fast

Test in a Clone of the Production Environment

Make it Easy for Anyone to Get the Latest Executable

Everyone can see what's happening

Automate Deployment

Benefits of Continuous Integration

Introducing Continuous Integration

Final Thoughts

Acknowledgments

Significant Revisions

URL Rewriting in ASP.NET

Contents

Introduction

Common Uses of URL Rewriting

What Happens When a Request Reaches IIS

Examining Requests with ISAPI Filters

What Happens When a Request Enters the ASP.NET Engine

Creating and Registering Custom HTTP Modules and HTTP Handlers

Implementing URL Rewriting

URL Rewriting with HTTP Modules

URL Rewriting in HTTP Handlers

Building a URL Rewriting Engine

Specifying Configuration Information for the URL Rewriting Engine

URL Rewriting with an HTTP Module

Performing Simple URL Rewriting with the URL Rewriting Engine

Handling Postbacks

Creating Truly "Hackable" URLs

Building the Requisite Directory Structure

Conclusion

Related Books

Works consulted

A Matter of Context

A low-level Look at the ASP.NET Architecture

What is ASP.NET

From Browser to ASP.NET

The ISAPI Connection

IIS 5 and 6 work differently

Getting into the .NET runtime

HttpRuntime, HttpContext, and HttpApplication – Oh my

Flowing through the ASP.NET Pipeline

HttpContext, HttpModules and HttpHandlers

Have I stooped low enough for you?

Working with Complex Data Types in an XML Web Service

Rich Internet Applications - RIA everywhere

What is CMM (SEI Capability Maturity Model)?

Previously known as Key Process Area (KPA)

Capability Maturity Model (CMM) Process Area (PA) Activities

Capability Maturity Model aka (CMM)

Levels of the CMM

Some praise of the CMM

Some criticism of the CMM

The most beneficial elements of CMM Level 2 and Level 3

The Case for Nullable Types

Achieving Nullable Values - Current Techniques

Special Values

Using a Flag

Defining a Nullable Type

Introducing .NET2.0 Nullable Types

Understanding the System.Nullable type

Using a Nullable Type

Checking for Null Values

Conclusion

On the Way to Mastering ASP.NET: Introducing Custom Entity Classes

Contents

Introduction

Problems with Datasets

Lack of Abstraction

Weakly-Typed

Not Object-Oriented

The Case Against DataSet

Custom Entity Classes

Page:
NN