Thoughts in the Wilderness

I’m into my second month at my new job.  I remain amazed at the amount of documentation that is being generated.  Being a Microsoft shop, almost everything finds its way into Sharepoint.  I’m not against Sharepoint per-se, but I do find it very opaque, i.e. hard to find stuff.

Its like having a lot of interesting papers on my desk, but they are all inside sealed envelopes.  I can open the envelopes, but when I’m done with it, it goes back in the envelope.  Sometimes there are other envelopes inside the envelopes.  As I said, very opaque, and very non-agile.  Contrast with what I am used to, which is a Wiki.  Totally searchable, easy to find what people have been editing.  More like a nice folder with happy little index tabs.

Sharepoint rant over, moving on…

The Business Analysts (BAs) had a slow start, with the result that we started developing our product before they had really written any requirement documents about it.  The result is that they are playing catch-up, and documenting what we have provided, and asking all of the questions that we should have asked while we developed it.

We do not have a single customer, or product champion.  This means that the questions are difficult for anyone to answer effectively.  It also means that the team is pulled in several directions:

• software architect tries to get us to do multi-tier and service-based and ultra secure at the expense of usability
• managers care mostly that it is done on time
• business analysts care about “checking off” features
• there is very little optimization of features, i.e. everything is equal priority

The result is that we have a fairly well architected, feature-full product that is hastily put together.  Features are surface deep, i.e. we are not “done” to the level that an Agile team would normally consider “done”.  Usability is low, and the product lacks perceived integrity.

Even given all of that, I remain optimistic.  We have a solid team, and every day we work more closely together.  As I understand more about the environment, I am able to make small adjustments to improve it.  We have a lot of time left to deliver, and upper management expects that it will be very much a v1.0 product.

Well today we had our first retrospective, a day before the end of our first sprint.  I used the tried & true format of “What went well” followed by “what went not as well”.  We scheduled an hour for the meeting.

As usual, people had a lot of trouble coming up with positive things to say, and not much trouble coming up with negative :)  One person refused to write anything positive!

It came together pretty easily that the topic that concerned the most people was “requirements”, or rather the lack of them.  There was a fair bit of conversation, which could easily have got out of hand.  In fact, it did for a little bit when one person tried to hijack the meeting for a topic other than the one we agreed to talk about.  I redirected that to another 1-on-1 meeting between the 2 of us later.

I tried to keep it in front of everyone that we wanted to come up with concrete actions that could be used to address the problems.  This worked ok to keep the conversation on topic, but no-one really suggested any actions except for our manager.  Being that she is the manager, no-one was seemed ready to contradict her suggestions.  They were good enough action items though, so I went with the flow.

The organization is very document-centric, which bothers me.  Rather than talk about ways that the developers could *pull* requirements through 1-on-1 conversations, there were a lot of suggestions on how we could better use email and documents on SharePoint.  Ugh.  I’ll have to live with the document-centric viewpoint for a while I think.  The best I can probably hope for, short term, is to find ways to make the documents useful to the team.

In the end, we finished on time and came out with some actions that we are taking to address our requirements-concerns.  Nobody was insulted or put off, and the team is fairly positive.  I’m pretty happy with those results.

As mentioned a few weeks ago, I have started a new job. My role is still unclear, but that doesn’t bother me at all - this way I can define my own role.

I’m working in a small team of web developers, trying to start on the next generation of the web portion of our core business product. The team was supposed to be Agile, but was decidedly *not*. I’m working on that, i.e. so far I’m defining myself as the Agile-guru.

We had our first planning meeting that I was allowed to run, and I went with my favorite planning tool (FreeMind). The team is very tool and framework oriented, so I think introducing the tool as part of the planning was less intimidating than if I had just gone at it with story cards.

My purpose in using FreeMind is to enable open discussion, and also to focus that discussion. It allows me to acknowledge everyone’s input, by reflecting it on the mind-map. The tree-like structure naturally allows everyone to be focused on the current area of discussion without being too distracted by other concerns. Once we have everything noted, then we can more effectively make estimates.

We had a very short estimating session. This was difficult, because individuals on the team seemed information-starved, i.e. they wanted to gather all of the requirements then and there. I tried with limited success to focus on just getting rough estimates. To keep it ultra-simple, I allowed only 4 time estimates: 1 day, 2 days, 5 days, or more (needs to be broken down further to estimate). As expected for a new Agile team, estimates were *very* ambitious, and we did not complete.

My final note is that it is interesting to be able to observe the team forming. I was the last member of the team to be hired, but before my joining, the others have barely interacted as far as I can tell. This means that we started in the “forming” stage of team evolution. This is characterised by the our manager being very hands-on, and directive. Basically, we are not a team so much as a set of individuals working under the co-ordination of a manager.

The next team-stage after forming is “storming”. This is a little stressful, and I hope we don’t lose any team members, or the confidence of our manager. I can already see some people coming out strongly on opposite sides of issues, and coming into conflict because of it.

Anyway, that’s my update. Wish me luck.

Sorry if the feeds are all messed up - I just moved my blog to my own hosting provider (it was hosted by blogger.com). I was fairly content with the blogger.com service, but I decided to scrap the website at www.perfectapi.com, and felt I needed something to put in its place. (So the blog will be the default website on my domain now).

Its been a while since my last post! Some of the reason is that I went on an intense 3 week job search, followed by a new job that I started on Monday. Its weird that I really enjoy interviews - but they did drain my energy levels substantially.

I’m still working as a senior software developer in a team environment (pretty much my goal for this job). I’ve been picking up a lot of new stuff, as the job involves getting back into the ASP.NET world that I had left behind a few years ago. My first task is to design and implement a new claims based (federated) security system that works across a multiple tiered application. Its fun, and I’m making good progress. Yay.

It is also interesting to be in a new environment that tries to be Agile, but is much more limited by environment than it was in my previous position. I hope I will be able to improve what they have started, but for now I have been just observing.

My biggest shock on the new job so far? Seeing VSS again, in active use. Ouch. Its a very Microsoft-centric shop, so at least they are looking at TFS.

Its not my usual thing to post code on my blog, but so many people keep asking me for this, so here it is:

Public Class DetectActivity
   Implements IDisposable
   Implements System.Windows.Forms.IMessageFilter
 
   Private _lastActivity As Date
 
   Public Sub New()
       _lastActivity = Date.Now
       Windows.Forms.Application.AddMessageFilter(Me)
   End Sub
 
   Public ReadOnly Property LastActivity() As Date
       Get
           Return _lastActivity
       End Get
   End Property
 
   Public Function PreFilterMessage(ByRef m AsSystem.Windows.Forms.Message) As Boolean ImplementsSystem.Windows.Forms.IMessageFilter.PreFilterMessage
       Select Case m.Msg
           Case Win32Message.WM_KEYUP, Win32Message.WM_KEYDOWN
               _lastActivity = Date.Now
           Case Win32Message.WM_LBUTTONDOWN, Win32Message.WM_LBUTTONUP, _  Win32Message.WM_MBUTTONDOWN, Win32Message.WM_MBUTTONUP, _ Win32Message.WM_RBUTTONDOWN, Win32Message.WM_RBUTTONUP, _  Win32Message.WM_MOUSEMOVE, Win32Message.WM_MOUSEWHEEL
 
               _lastActivity = Date.Now
       End Select
 
       Return False
   End Function
 
   Public Sub Dispose() Implements System.IDisposable.Dispose
       Windows.Forms.Application.RemoveMessageFilter(Me)
   End Sub
 
   Enum Win32Message
       WM_KEYFIRST = &H100
       WM_KEYDOWN = &H100
       WM_KEYUP = &H101
       WM_CHAR = &H102
       WM_DEADCHAR = &H103
       WM_SYSKEYDOWN = &H104
       WM_SYSKEYUP = &H105
       WM_SYSCHAR = &H106
       WM_SYSDEADCHAR = &H107
 
       WM_MOUSEFIRST = &H200
       WM_MOUSEMOVE = &H200
       WM_LBUTTONDOWN = &H201
       WM_LBUTTONUP = &H202
       WM_LBUTTONDBLCLK = &H203
       WM_RBUTTONDOWN = &H204
       WM_RBUTTONUP = &H205
       WM_RBUTTONDBLCLK = &H206
       WM_MBUTTONDOWN = &H207
       WM_MBUTTONUP = &H208
       WM_MBUTTONDBLCLK = &H209
       WM_MOUSEWHEEL = &H20A
   End Enum
End Class

This simple class can be instantiated when a Windows Forms application starts up, and then you can ask it at any time when the last user-activity was.

This class works better than most other techniques, because:

• it only detects activity in your own application
• it is simple
• it is 100% managed code
• it works

Much has been written on the inability of the need for future applications to take advantage of dual-core and multi-core systems. I think most of it is hogwash.

Intel is a big proponent of the argument for more parallel-processing tools for programmers. This is no big surprise - Moore’s law has flattened out for the current generation of processors. They are expanding outwards to multiple cores now, no longer upwards to higher CPU speeds. Intel is understandably terrified that their PC CPU market demand is going to dry up. Its all marketing.

I prefer to look at reality.

Looking at whats running on my system, these are the some threading stats:

• Firefox - 18 threads
• Outlook - 31 threads
• Perforce P4V - 8 threads
• Custom .NET application - 13 threads

Seems like plenty of threading goodness is already going on.

Taking the example of the custom .NET application, there is a very noticeable performance hit when I run it within a virtual machine. I attribute this to the virtual environment only making use of a single processing core. Those 13 threads have to fight over the single processor. In addition, the .NET runtime (garbage collection etc) is highly concurrent - when running on a single-processor environment, it is crippled.

I am not arguing that all application threads are particularly busy. Truth be told, of the 13 threads, only one, sometimes 2-3 are ever particularly busy. I’m just saying that there are as many threads as are necessary to get the job done. As programmers, we need not care at all about multi-core, except where there is some advantage to be had in using it.

Applications are written do as many simultaneous things as the user expects. It has nothing to do with multi-core as to whether there are multiple threads - it has to do with what the user expects and what the application needs to do to meet those expectations.

Multiple CPUs are like RAM - you will never have enough for tomorrow’s needs. Heck, you can barely run Vista without a dual-core machine - the OS itself has too much going on to leave enough over for decent performance. Multi-core PCs are here to stay, but as programmers it is simply a case of “nothing to see here - move along”.

When I was a junior-level programmer, I felt like I had one responsibility - be good at programming.

Later in my career, once I became a lead developer, I learned that I now had another responsibility - deliver a project on time.

Still later, as a software architect I picked up another responsibility - to make it easier for the team of developers to be successful.

Today, I recognize another responsibility - deliver value to the business.

In the above chart, blue represents focus on Self, red is focus on the Team, and yellow is focus on the Business.

What does your chart look like?

This is not the simplistic analogy you may be expecting.

Where I live in Minneapolis MN, we have a high profile bridge project going on right now, due to the tragic collapse of the previous structure. If you look at the project page on the state’s website, you can see the following “features of the new bridge”:

• 100-year life span
• 10 lanes of traffic, five in each direction—two lanes wider than the former bridge
• 189 feet wide—the previous bridge was 113 feet wide
• 13 foot wide right shoulders and 14 foot wide left shoulders, the previous bridge had no shoulders
• Light Rail Transport-ready which may help accommodate future transportation needs
• Design-build project complete in 437 days.
• Designed to be aesthetically pleasing and fit in with its environment

These are the most high-level (public) stakeholder values for the new bridge. From an engineer’s perspective, they are the constraints under which the bridge must be delivered. In addition, we see an architect’s rendition (picture) of what the new bridge will look like. This is also a constraint - an engineer cannot add things that will substantially change the appearance of the bridge.

Now, we can be sure that the design of the bridge was a collaborative effort of a team of people. Engineers, Marketing, Architects and the client. Is the design ongoing as they build the structure? Yes - they are using a growing technique known as Design-Build - they purposely start construction before the design is complete.

At first glance, design-build might sounds like a simple case of parallel work - one team is working on designing just-in-time, and another is working on the construction. In practice though, there is a collaborative environment that reportedly results in avoidance of disputes, faster project delivery, and less need for project management oversight.

The Analogy
So how is this similar to building software?

Firstly, the process of programming it is like the design of a bridge - it is the bringing together of people in different roles to creatively find ways to build the end result. Ideally, development involves a lot of Thinking, Talking, and Tweaking, just like a bridge design. In design, we often find that two heads are better than one. Pair-programming has been suggested as one way to do this in software development. Of course, we have many other collaborative techniques to communicate and discuss design.

Like a bridge design, the output of building software can be represented by piles of paper. The bridge has drawings, engineering specifications and requirements. A program has something better though - its code. (No, I’m not arguing that “the code is the design”. I’m just saying that the code “is a representation of the design”). The code accurately describes the parts of the design that it touches.

This “programming code = bridge design” point is key to what I’m trying to convey - the process of programming produces a design output, not a product. The final product is the result of implementing that design (just as the bridge itself is the result of implementing its design).

Specifically, the “building” is the deployment of software in its final environment. Deployments are where the “tires meet the road” - they are the intersection of the design with reality (just like construction). Mostly, the design holds up and does not need tweaking after deployment. Sometimes though, the harsh lights of reality expose the hidden flaws in the design. (In light of that, it is best to expose an application to its first deployment as soon as possible).

Some software groups have QA (quality) departments. Historically, these departments have taken the role of performing trial deployments - they will take the software, and expose it to a simulation of the real environment. Large construction projects also have this role - an independent group audits the designs, with the hope of spotting problems that would cause a problem when the construction occurs.

Finally, we find that the best way of constructing a large bridge project is to simultaneously design and build. The analogy for software is small frequent releases. Research and experience has shown this to be a good way deliver quality software that meets the requirements.

Conclusion
If we accept that building a bridge and building software are similar (they contain the same basic steps), then we can use that information to produce some interesting insights:

• That thing we need to do before developing is “architecture” - There is a fine distinction between architecture and design. The way I like to define it is that architecture describes the parts are visible from the outside, and design describes the inside. A bridge architect is able to construct a working model and rendition of the outside of a bridge without the full engineering specs. To do this, he needs to take into account all of the stakeholder values. Similarly, we need to be able to draw the edges of a software application before we start - we need to understand how the software will interact with the outside world, and how the outside world will interact with the software.
• QA is a misnomer - the primary purpose of a separate QA department should not be to assure quality. We can get quality in better ways than that. The purpose of the QA department should be to validate the design of the software, by simulating real environments. Many QA professionals already know this, of course.

This blog post is inspired by a set of three essays by Jack W. Reeves.

Overheard on the XP list, John A De Goes describing his attitude towards teams and processes.

There’s a growing realization that companies, organizations, and teams, they’re all comprised of humans, and being a human is radically different than being a machine. You can’t engineer something comprised of humans — you have to grow it, nourish it, cultivate it. You can fix a broken machine by following a recipe, but you can’t fix a group of people that way. Prescriptions for success amount to nothing, because they ignore the human component, which, when all is said and done, is the most significant factor determining success — development methodology be damned.