Projects focus

I'm seeing first hand why when working on a big project you must have a plan. It has got me thinking about project management (and project non-management ;)

I am liking the idea of outlining a core set of project artifacts, whose creation are beneficial to a projects focus. With each of these artifacts you should install a quality measurement. First I am going to talk about some of these artifacts, then I will examine some measures of quality to go with them..

Goals

Goals are the first artifact I want to talk about. They are usually the driving force in getting a project up and going.

Set goals for the project. What is it you are accomplishing by completing the project? Goals and business objectives are the same thing. The worst thing that could happen when goal setting would be to name a single goal like "finish X". A goal like this says that the project is not being taken seriously and that nobody is really analyzing it's purpose. Since goals are really the first artifact of a project they should be done well, right? Many parties may be effected by a project. Shouldn't this be considered in your set of goals? (NOTE: These parties are often the users of a project's products, i.e. customers)

Sadly many projects are built on poor goal setting. If you don't clearly expose the project's goals you can very easily lose sight of the purpose in your project.

There are at least two default goals for every project. Number one is get the project done as quickly as possible and the second is quality shall not compromised. Goals can help you to know when you are going off the radar for a project. If an aspect of the project can not be reasoned as building towards one or more project goals then it should not be part of the project. That simple. Goals can help cut out a majority of feature creep. The other big help from goals is to measure the success of a project after completion. Everyone should do this. It rarely happens. I will go into that more later in this post.

Here is a (long) simple example...

You have a project for creating an application. It will be used for signing up to play on your companies inter mural sport teams. The big goal is: "make an application that is accessible by every employee across your company intra net". Another goal is: "get rid of paper sign ups, and centralize the data in a database for documentation and reuse".

Lets look at a good system requirement: "the application shall be web based". Using a web application for this makes sense. It will make the application very accessible. That helps to meet the big goal. Great. Now let's keep going with these requirements. You create the system requirement: "Use Ruby on Rails for web application platform". Your team is very familiar with Rails and therefore it makes sense to use it for creating the web platform. Both these system requirements also work with the second goal, mainly to have things backed up in a database.

Now lets look at going off of the radar. One of the new guys on the project is really into learning AJAX. He is like "Hey this will make the application so much cooler!". You and your team do not have any experience with AJAX, therefore using it would require battling a learning curve. This guy wants the system requirement: "Use AJAX to make application more responsive". Which goal does it help to accomplish. None. And in fact it would be violating goal numero uno, which is to get the project done as quickly as possible. This is something that would be nice to have but does not meet any goals.

Now if a goal of the project was: "garner new experience in the Web 2.0 domain" (yes this is generic), then you could reason using AJAX techniques in the project. Unfortunately it is rare that these type of things are included into a project's goals. This is because the person(s) setting the goals are very focused on goal #1, getting things done. Often it is hard to rationalize time spent on learning new things.

Now going back to the section above on good requirements for our goals. What if your team was not familiar with Rails, or any other web application platform (LAMP, J2EE, etc.) for that matter? Maybe you and your team have some Java Swing GUI experience. You figure "hey making a GUI based application won't be too hard". But then you think "oh yeah but getting the applications to connect to a central data store across the whole company...". If doing a web application is the best way to accomplish our goals, then you need a web platform (such as Rails). Taking the time to learn the new platform is key to the project and therefore it is not a violation of rule #1.

In this case there better be one or more goals concerned with learning and using Rails in the project. The business owners in this project would not care so much about these goals. These goals are specific to the engineering domain. The business owner would have to trust that they are necessary. When the project is done you should be able to look at the goals related to working with Rails and know if you met them and/or went outside of them. A goal would be "able to demonstrate good MVC principles in application design using Rails". These goals should narrow the focus of what is being learned to accomplish the project. Remember no AJAX ;)

Goal creation and documentation

You must create the goals and document them. Concise technical writing will almost always beat out spoken word for clarity. Let's say you come up with some goals in a project kickoff meeting. Thats great, but if you don't have a process to document them why bother with the meeting. In a week or two when you and other project members are busy on other things you either won't remember these goals, or worse interpret what they where incorrectly. All you will likely remember is "Get project X done"!

By documenting the goals you should be able to translate and connect them with every other project artifacts. When you look at the business requirements gathered for a project you should be able to easily superimpose the goals over them, and see the interconnections. This principle applies to the finished product to. The product for a project should map very easily to the goals that fostered it.

Goals should be written as clearly and concisely as possible. Most goals should be understandable by persons of varying technical background. Technical aspects related to a goal should be boiled down, with the meat found in the requirements which correspond to the goal. Sometimes goals will be more technical because of the audience. They should still be boiled down to make their purpose clear to as many persons as is possible.

Goal artifact quality

So for the 'goals' artifact here are some quality markers off the top of my head:

1. Each goal clearly identifies its target area such as generate revenue or business, or create new technology. It may help to group like goals when you document them.
   
2. The goals are written concisely, with a general, non-technical audience in mind. This does not mean a goal cannot be technical, just that it should be as non-technical as is possible.
   
3. The goals should be clear enough so when laying out requirements for a project, you are able to validate each requirement versus one or more of the goals without confusion.
4. When the project is finished (or maybe at a milestone) you should be able to draw clear comparisons between the goals and the finished product/milestone.

Goal Assessment (the next artifact)

Why even have goals? We already talked about how they can help keep focus up, and cut down on feature creep. They obviously play a huge part in the requirements gathering process. I would say goals also allow us to gauge our success. This payoff comes mostly after a project is completed. When a product is delivered one can critique it, drawing conclusions about success from how well each goal was accomplished. This critique is part of the documentation. It is an artifact I would call "The goal assessment". It should be done in a way that draws comparisons between the goals and the product's features. On each goal there should be information on what helped or hurt the accomplishing of the goal. If a product feature does not meet the goal(s) then there should be information on how it does not meet the goal, and why that is.

Now going back to the bad case, where you have one goal "Finish project X!". How the heck can you critique this? You can't. All you can do is rubber stamp the project as 'done'. With no clear documentation on the project's goals you can't do a thorough examination after completion. You are done because you have met the basic idea of what the project was supposed to be. You may have nailed the project's product, but you do not have a clearly documented picture of what has been accomplished.

Documenting the goals, and how well those goals where met, has future implications to all projects. When working on later projects you can use these goals to know what has been done and how successful it was. You can examine what made things successful. You can look for tough areas by examining the goals that could not be completed, or that where very hard to complete. If you are working on increasing quality in a particular area you should start by finding related goals on projects past. You can use the data collected about these goals to help in understanding how to increase quality.

Let us imagine a company that builds radios. They finish a project for a brand new radio machine. One of the goals of this project was to develop and implement better radio reception. They test the newly built product against older products they have built. When people listen to the new radio it is a an obvious to them that the sound is more clear and crisp. The company is happy. Now at the end of the project they have accomplished one of their goals. They should take the opportunity to examine what was done correctly to make this a success.

For the goal given above; they may document that a new electronic part came into production that made the job possible. In general to all goals, maybe they worked 50 hours a week, rather than the 40 they normally would have.

Documenting information like the above in a goal assessment allows you to see the good and bad. When going back and reviewing this a year later you would want to be re-reminded of some of the things you had forgotten.

Goal assessment quality

Here is another quick top of the head list for quality in this artifact:

1. Collected information is easy to understand.
2. Each piece of information is connected to one or more goals, or possibly is general enough to connect with all goals (such as the working extra hours example above).
3. Nothing should be barred from this assessment. If you think a project members input is silly still put it in. If someone says the broken coffee pot caused work to slow down, they may be serious.
   

Start a community

It's becoming harder for me to get in here and blog! I have plenty to write, but I'm just so involved in all sorts of little projects out side of work. One of them is looking into getting a web site going. I have one right now, but it is pretty weak. I have started writing my own community site from scratch using Ruby. If you don't know the idea of a community site, it is one where you offer membership. Each member gets privileges to applications such as forums, blogs, email, etc. The number of apps availiable depends on the site. Yahoo is a community site offering a lot of apps to it's members. VWVortex is an example of a site that mostly offers forums for its users. Both sites could be considered community based sites.

So back to my work... I started writing the site using Rails. I have made a lot of headway into creating the articles application. I have a great javascript text editor embeded, which gives the ability to jump back and forth between editing html and rich text. Writing stuff from scratch is always a good learning experience. Writing a web app like this lets you learn to solve lots of little problems, which for me is fun.

Now I'm not sold on having my site run 'On Rails'. Although I really like Ruby, and find the Rails framework very cool, I don't need to have my personal site project be based off of it. I decided to look at some other options. I looked at Zope, which is pretty cool. You can run Plone on Zope, which was my main attraction to Zope. Plone is a feature rich CMS. Plone uses the Kupu project for text editing. Kupu is a free, advanced article editing javascript application. I like it a lot, and tried to tie it into my Rails site. If you watch the Plone screencasts, a lot of the functions being shown are implemented by Kupu. Plone brings together lots of features for editing and managing your content. It uses lots of DHTML features, which is really nice. My main problem with Zope/Plone is needing that specialized hosting to run a Zope site. It is more costly, and also there is not that great a demand for Zope/Plone experts out there.

Now I kept looking. I used to be all into writing my web apps using PHP. Oh yeah, good old get 'r' done PHP.... Well it sure has been awhile. In looking for something that already has built in community features, and is written in PHP I found Drupal! In like one hour I had it installed, set up some users, gave them each a blog, created forums and I'm just getting started. Drupal seems to have a simple, modularized approach to extending its base functions. This means if you don't like the blog app that ships with it you can write a new on and drop it in, or find a better one some else already wrote. That is powerful stuff. Drupal takes care of managing all the security and templating.

I'm going to have to look a little further into how one goes about writing a new module. I am first going to install and configure some of the 3rd party modules that are out there, just to get a feel of how it all works.

Rolling a long

Well it's been a long time since I've posted anything. Lately there has just been soo many projects, work and pet.

I've spent time playing with Portlets, using the Spring 2.0 MVC and Hibernate. I've also been hacking at an interesting application called 'Democracy Player'. The software has versions for Windows, Mac OS X and Linux. Each one uses different means of GUI rendering. On Mac Cocoa is used and on Linux Gtk. It is mostly written in Python, with some C/C++/ObjC used for the native GUI. I'm liking Python, much like I have enjoyed the little work I have got to do with Ruby.

At work it's been lots of integration work. I've been tasked with reimplementing some old integration processes with our new ERP solution. One big integration point in our legacy system is Amazon. I've just finished integrating my companies warehouse fulfillment data with Amazon. Next thing is to integrate our returns/cancellation/refund data with Amazon. I've been doing a bit of tweaking here and there for the order entryways. I created a simple rest like web service to take orders, returning an immediate sales order. This was needed for a couple of satallite sites that we deal with. They need to know right away if a sales order was successfully created. The other entry point which we will soon use for all web order entry is the messaging system. The messaging system makes submitting orders from just about any app, written in any language trivial. It also allows our companies order data to flow smoothly through from web to ERP.

Not a lot new here.... Hopefully I will have something more interesting to post here soon!

Integration across the board

So I've come to a point where I have Windows services reading from ActiveMQ = AMQ (it's MOM software) and feeding a system via a COM connector. The COM connector seems to be holding up really well. Last night I was tossing messages at AMQ at a rate of 10 per second. These messages are about 150-200 characters in size. I tested 360010 message going through. The messages finally ended up in a database. This morning I checked and 360010 messages where in the table! Also the software I use to track the status of the AMQ showed that 360010 messages had been put on a queue, and the same amount read off. So things are looking up.

I used Ruby to send the messages onto AMQ. I think I have mentioned before that there is a simple messaging protocol called STOMP, which I used to connect to AMQ from Ruby. AMQ has a snappy transport connector which allows you to do this. AMQ also allows you to write your own transport connector, so if you don't like a protocol write your own :)

I'm testing changes I made to the STOMP Ruby client. I made it much easier to use the client to send messages that get marshalled in the AMQ STOMP transport connector. The client as it came did not make it possible to send what would be a JMS TextMessage in AMQ. Messages that came from the basic client came into AMQ and got marshalled into the JMS BinaryMessage type. The AMQ transport allows for marshalling to BinaryMessage or TextMessage, depending on STOMP header settings. With some simple Ruby coding I have a modified client that now has an extra method 'sendTextMessage'. When using this method the message in AMQ will be marshalled as a TextMessage! TextMessage fits our current purposes better than BinaryMessage.

So back to the testing. I have 4 producers spitting 20 messages a second onto a queue, and one consumer reading messages off of the queue as fast as it can. I'm up near 200k messages sent/received and counting. I'll see if things can break at some point :) I also have the Windows service coming on and off to take messages from this queue. This tests how well things are handled with consumers going down and coming online on the fly.

So the big picture is to be able to easily integrate a M$ product into a ESB using the AMQ messaging system. Rather than have to handle the problem of the M$ product dying and data being dropped we will have a scenario where messages are kept and fed over when the product comes back online. We should also have an easy time reusing bizz logic components in the ESB.

More messaging fun!

In my last post I talked about using a STOMP client for Ruby. STOMP is a very light weight protocol for doing messaging. I have implemented an ESB (Servicemix) and messaging system (ActiveMQ) for my company. Our business depends on many programming languages, databases, software and such to keep things running. STOMP as I explained before allows the web guys at my work to message our system using Perl, Ruby, Python, C# and more :). This will allow for easy routing of web data through our ESB.

The goal for now is to set up orders to flow through the ESB for our web department. We are using this new piece of software to do all our CRM, product maintenance and other business functions. I have refered to this software as 'Magic' and will continue to here. Orders will need to flow into Magic somehow. Magic is M$ built, and I figured why not put the ActiveMQ NMS implementation to use. NMS is like a .Net version of JMS. The guys over on the ActiveMQ project created an implementation of NMS for ActiveMQ. This is very nice of them. I can now use either a STOMP client or NMS client to message with ActiveMQ from my C# code!

So at first I tried to create .dll libraries of the AMQ NMS implementation, then load those into Magic. Magic has CLR functionality in it's newest release. This means that the language you code with inside Magic can use objects written in C#, VB.net, Java, and on... The problem is the CLR implementation in Magic is lame! It does not support language features like events, delegates, virtual functions etc. So I cannot use the NMS code inside of Magic. Bummer. I mulled for a day about this one while working on other stuff. Then I found an idea.

Magic has a COM connector, which allows you to call pretty much any code written in Magic. This is nice for executing remote procedure calls to Magic. So my new solution is to write Windows services which do the messaging, then pass/pull data from Magic using the COM connector! I have already written my first service. It writes a simple XML doc to a queue on a separate machine every 30 seconds or so.

Now that I have this written I will be focusing on writing the COM connector code to push and pull messages from Magic. Pushing is the easy part as Asynchronous communication is easy to keep. As a message comes in, the service will grab it and push it over to Magic. Pulling data out may be harder. Right now I use a timer object in my service, to execute some functions every 30 seconds. I figure I may have to do this polling type thing to pull data. It sucks, but it is better than the alternative. In the long run it allows finer tuning of how the data will flow across our system.

Right now we are getting data out of Magic using web services. I'm not impressed with web services. The only time people ask me to write them is for improper purposes. When we need to get data out of Magic, we run batch processes which send the data to an external web service. Magic spams out large chunks of data to the web service, causing it to stress badly. But for the most part the web service sits there doing nutin :) It's not a very pretty picture. With messaging we can take the flow and streamline it. Messages flow out as they are created, not at batch time. The message consumer can pull the messages as it likes, rather than having to deal with a large volume being spamed to it. This relieves stress accross the system. very nice. I think my boss is convinced that this is the way we should go. I just need to get the many pieces of this together, so that we can do some testing.