Day three of SQL Server 2008 for Administrators.  I dig training classes regardless of topic.  I just enjoy the vibe of a class with people intent on learning a product, service, ideal, or theory.  The motivated learner is by far superior than the unmotivated learner, which makes such training very useful.

Over the last couple days of SQL Server training I've managed to pick up a few bits of information that I didn't know much about.  For simple context, I've been primarily involved in SQL Server from a developer's point of view.  This has included building queries, using OR/M's to simplify data access and abstractions, and from a high level software architectural point of view.  Some of the new bits however can are useful in these realms, but also just from the vantage point of better understanding.

The functionality that was very useful was the performance data collection and getting the associated management data warehouse (see previous entry for more on this) built to monitor the data.  This collection of information for disk usage, query statistics, and server activity is useful from a developer's point of view, the administrators point of view, and helps in architectural design of systems such as high availability and creating horizontally scalable applications.

I also learned a bit about policies and the needed conditions and facets.  This is a particular bit of functionality that I won't have much use for, but understanding policies for troubleshooting and other needs is pivotal for functional implement of Webtrends or just knowing when behaviors are seen within an environment.

Overall the training was helpful.  It's always nice to step out of the day to day and just focus on something else too.  I'm refreshed and ready to move on to the next job now.

One of the new features in SQL Server 2k8 is the Management Data Warehouse.  This feature is made up of a data collector, storage, and execution that collects data about the particular databases within the server into a reporting warehouse.  This data is stored in the msdb and also in management data warehouse databases that are created when the Data Collection is configured.  The warehouse uses various pieces of SQL Server such as SSIS and other components also.  One of these components is the dcexec.exe file that is responsible for loading and executing SSIS packages that are part of this collection set.

Keep in mind, that there is an API for this warehouse also for extension of the functionality.

The following are the steps for setting up a basic management data warehouse.

Right click on Data Collection and you'll see the menu below.

Select Configure Management Data Warehouse.

On this screen I had to actually come back and use the New button to create a new database to use to host the warehouse.  My suggestion is to NOT use an existing database and instead create a new database.

Just leave these unchecked unless you know that one of these accounts need access.

Now that this is setup for the warehouse host we'll setup the data collection.

The cache directory will default if a path is not entered, but I've entered one just because I felt the inclination to do so.

The screen shot below, which you might want to click on just to review with a full size shot, shows what is now available in SQL Management Studio.  You'll see the collection sets, and also many new jobs setup to perform the collection of data for the warehouse.

After setting up the collection click on the Disk Usage option, right click, then select properties.

On this screen select the T-SQL that is in the input parameters text box.  The code is shown below for ease of copy n' pasting.  :)

DECLARE @dbsize bigint 

DECLARE @logsize bigint 

DECLARE @ftsize bigint 

DECLARE @reservedpages bigint 

DECLARE @pages bigint 

DECLARE @usedpages bigint

SELECT @dbsize = SUM(convert(bigint,case when type = 0 then size else 0 end)) 

      ,@logsize = SUM(convert(bigint,case when type = 1 then size else 0 end)) 

      ,@ftsize = SUM(convert(bigint,case when type = 4 then size else 0 end)) 

FROM sys.database_files

SELECT @reservedpages = SUM(a.total_pages) 

       ,@usedpages = SUM(a.used_pages) 

       ,@pages = SUM(CASE 

                        WHEN it.internal_type IN (202,204) THEN 0 

                        WHEN a.type != 1 THEN a.used_pages 

                        WHEN p.index_id < 2 THEN a.data_pages 

                        ELSE 0 

                     END) 

FROM sys.partitions p  

JOIN sys.allocation_units a ON p.partition_id = a.container_id 

LEFT JOIN sys.internal_tables it ON p.object_id = it.object_id 

SELECT 

        @dbsize as 'dbsize',

        @logsize as 'logsize',

        @ftsize as 'ftsize',

        @reservedpages as 'reservedpages',

        @usedpages as 'usedpages',

        @pages as 'pages'

When you run this the query should give you the results as shown.

One of the sample reports, which won't have the trend graphs until data is processed multiple times.

In summary.  This feature of SQL Server is really helpful for development to keep track of growth, performance, and other characteristics.  I strongly suggest checking out this reporting capability of SQL Server 2008.

I finally got all my virtual image loads setup and my Ubuntu Install is running smoothly again.  I also got a Windows 2008 install running and hope to use it to try out some WAS, Velocity, and other WCF related features.

In addition to that I've got my work in progress up and running at http://mystuff.adronbhall.com if you want to check out my story list application.  This sub domain is literally the work in progress, so don't expect much of anything here.

So far that's been my weekend.  I've also had an onslaught of allergies kick me in the face.  This is never fun, even when I'm trying not to do anything constructive, and I was working on the exact opposite of that.  Argh!

Next step, Mono, getting Silverlight to run on Mono, and a few other tasks.  Then back to Cockatoo, HighBall, and MyStoryList (which has no link, as I'm not sure what I'm going to do with this project yet).

I'm pondering doing an updated tools, trips, and tricks list of my current environment, but right now I'm still putting the finishing touches on it all.  So maybe that is a Monday entry.

NOTE:  This is not forgoing my other project:  Cockatoo & HighBall.

The story starts like this.  I've been using some, what will go unnamed, Agile Project Management Software that I just wasn't digging.  It was too restrictive and focused on the tool over the developers.  Several times the tool literally got in the way of development and forward progress.  Whenever this happens one should see a MASSIVE RED FLAG that the core manifesto concepts are broken.  When those are broken in such a brash and obvious way then the project is in threat of serious risks.  So I set out to fix this with simpler, more versatile, easy to use software that could replace the need for this tool.

I started out writing this software with the standard user story concept, to keep track of user stories.  That sounds kind of odd that I'm creating user stories to describe an app that will track user stories as a story.

:o  wowzerz!

So here we go.  Here are the initial user stories for minimally functional and working software.

1. A user can write a story.
2. A user can read the stories that have been written.
3. A user can change a story.
4. A user can throw away a story.

Those simple stories make sense to me.  With that I?ve started and ensuing entries will be forth coming.

I'm writing this blog entry at various intersections during the day to try out keeping a real 'log' of the things I accomplish.  I've always had trouble keeping up with the past, and when it comes time for reviews and such I usually draw a blank of what has been done.  It appears this is the crux of an Agile mindset always being geared toward the future and the next stories coming off of the back log.  So we'll see how this goes.

9:20am After a morning coffee, reading e-mail, catching up on communications, and all that regular morning activity I'm diving into the first code bits of the day.  I had done a search earlier about ASP.NET MVC task list applications and pulled up a stackoverflow entry.  This caused me to log in and actually see if there was anything I could throw an answer down for.  So far, nothing in particular, so onward with the code bits.

9:28am Had to drop code to step up to bat for a deployment.  One of the cool things about Agile projects is that at any moment ' WHAM ' off to production the code goes.  We don't have all the bits in exactly the right place so we have to manually manipulate a few bits before it moves to production.  (bits being a metaphorical descriptor)

9:49am Back to code?

9:50am Needed some things from the MSDN subscription.  Well lo and behold it is HeldAwaitingReactivation which is a bummer.  I'll have to get with IT and get that renewed or maybe I can do it?  Then I clicked on what appears to be another MSDN subscription and it is active.  Why even default to an inactive subscription?  I'm not really following the logic (if it exists) there.  Hmpf, as I started to download stuff, I realized MS has broken Chrome compatibility or something because it isn't launching right.  I suppose I'll open IE.  :(

MS please fix the fact that there is still active X stuff on your site(s).  K THX.

10:08am Finally got stuff I needed downloading after installing all the active X crap.  Now to try and get some focus on this code'  arrgh.

10:30am Stand up'

10:52am Stand up done, getting back to code in a few minutes.

11:08am Team took a break to watch a video & a bit o' camaraderie.  Hey, one can't code nonstop.  :)  Got about 15 minutes of code review finally done.

11:28am Going for lunch!

12:26pm Returned from lunch, diving back into code and returning e-mails.

1:27pm Actually got to start working on code again.  Hopefully will be able to maintain this for a while without interruptions.

2:10pm Got to a point and realized I didn't have the latest 'manually copied' assemblies and had to move them back into the project.  Wasted 20+ minutes.  Beginning the process of getting the files and fixing the problem.

2:38pm Got the files, project building and executing again properly.  Finally got to adding the feature I've been trying to work on since this morning.

3:43pm Actually taking a short break after making good progress.  Almost done with adding attribute(not the kind you might think, proprietary in nature) support for some reporting.  Things are looking better after a morning of not getting anything done.

4:18pm Finally completed the feature addition, now submitting info for tracking the feature addition and then will pack for dinner time.  Will then code some more tonight, I hope.

5:02pm Out of office for the day.

Now that I've written this, I realize the day has been a slow one, which is by no means the average.  I'm hoping to be in the code and at task (or in the story) a much more significant part of the day tomorrow.

Maybe I'll be able to keep up with my daily summaries, or maybe not.  It was a little bit of effort beyond the normal to keep this one going.  Overall I did get the feature request added and managed to help out a number of individuals with their efforts throughout the day.  I'd say, even with it being a slow to get going day, it has been a good day.  However, even with the feature request, no code to post or nothing particularly new learned.  I'll have to make a point tonight to learn something new.

OLAP?

I got to thinking about my HighBall Project and thought, wouldn?t it be awesome to use some of my know how to wire up some serious cubes and analyze the results from site usage all the way to scheduling analysis?  It?s an idea, but I?m not really sure how I would do it yet.  I do however have some ideas in regards to scheduling and how a nice cube could be used to analyze effective scheduling usage.  Sounds like something TriMet could even use ? if they don?t already do these types of analysis.  ;)

Here?s my thought.

Take a schedule as the time dimension.  That?s simple enough.  Now take X number of routes as opposing dimensions and use ridership counts, peak load, etc as the fact table sums and such.  The data should align accordingly to the apex of routes and concentration of route needs by riders.  This type of data could be used to find out where ridership peaks and drops and how to fill gaps or even where to increase service.

?not really sure, got a ways to go before I try it, but it is an interesting thought for analysis.  Eventually I?ll give it a shot.

Agile is the popular process du jour these days.  Every time I hear "Yeah, we do Agile" or "I know Agile, been doing it for years" I cringe.  These statements are spoken with the idea that Agile is this steadfast, unchanging, unalterable, single process that everyone is following.  It is often spoken without any knowledge of the history of Agile, the people who signed the manifesto, or what the true ideal of Agile is.

Agile is always changing, will always be changing, and we'll most likely all be new to it all the time.  Agile is something that flows differently for each group that tries it, but there are core tenants that must be met.  These tenants aren't part of a "process", but could be, they aren't part of a "method", but might be a result.  The point is, nobody really knows Agile, they know their team.

...or maybe they don't know their team if they think they know Agile?

That is something to seriously ponder.  So how does one know when they're talking to someone who really does adhere to the Agile Manifesto.  This is actually the simple part.  Generally they won't even mention Agile at first, but instead they discuss processes almost immediately.  Things they do to enable the developers, things they do to maintain velocity and keep things upbeat and positive.

Some of the first signs that a shop is truly Agile can be summarized by the answers to a few questions.

1. Does the team always have working software?
2. Can you regress and refactor quickly?
3. Do iterations ending or burn down charts stop velocity?
4. How many hours a week is the average?
5. How often does the team have to work excess hours?

...and one of the most important questions of all...

6. How often does your team get to speak with customers?

Does the team have working software?  If it doesn't, it's probable that they are not Agile, or not quit Agile yet.  A team needs to have working software.  The only lgeitimate excuse I've ever heard is, "we started yesterday and are a few days away from working software".  But even then, there should at least be some paper prototypes, or discussion notes with the customer about the proposed working software.

Once there is working software, how fast can the team regress and make changes?  How fast can the team refactor?  How often is the technical debt paid down?  This is a major sticking point that I hear a lot.  Teams must maintain testing, regression, and the ability to refactor.  The team must refactor regularly and keep the debt paid down, if not, there isn't a reserve entity out there to pay down the technical debt or provide a stimulus!  Without these things a team is running rough of Agile.

At the end of iterations does the team stop completely, take a breath of fresh air, and prepare for the hike up the mound of progress again?  If this happens, something is definitely wrong.  Iterations aren't supposed to break progress, merely reflect and maintain veloctiy.  This is where I've become more and more a fan of lean style or kanban style.  The hard stop that iterations often incur is very detrimental to velocity and sometimes even to morale.

Speaking of iterations and mounds of progress, how many hours are worked for that progress?  Are the hours consistent and maintainable?  If not, they should be.  Beware the other inconsistency of trying to apply 40hr work weeks to everyone.  Some people can do 40 and some can do 50, the key is to maintain velocity and maintain morale.  If one person does 50 hrs per week, great, make sure they're not burning themselves out.  Make it easy for them to maintain that.

The customer?  "What, we have to pay attention to the customer?", is a statement I heard recently.  I can't help but just look down whenever I hear this, at least when it is spoken seriously.  The development process and software process should NEVER get disconnected from the customer.  The industry as a whole learns this lesson over and over and over again each year.  We should all now know, we must stay in communication with the customer.

With all those thoughts, I leave this entry with a question for everyone, "Is your team really Agile?"

In my humble opinion, as long as a team is working on acheiving and getting these key points in place, all is good.  If not, the team should probably reflect a bit on what the end goal is.

With the recent upgrade the ReSharper 4.5 I've noticed two MAJOR things.  The first is that the speed is much improved.  This is a big plus when I'm on machines that can always use a little bit of a boost.  :)

However the other thing that has been added, which I very much enjoy, is the fact that I can use the ReSharper Unit Testing Interface now with mstest.  This has been one of my biggest frustrations with mstest (aside form the fact that they're slow, awkward, aren't as feature packed, etc).  I still prefer NUnit, xUnit, mbUnit, or some other framework more than mstest but at least mstest is tolerable now.  Mstest is used in many shops, so now it won't be as painful and slow to step into development in shops that use mstest - I'll be rolling ahead full speed now.

On completely different topics...   code, code, code, code, code...

One of the things I try to do is to find parallels in my personal software development, professional work related development, what people are looking into currently, and then I try to wrap them all together.  Sometimes that means writing blog entries to increase knowledge about something or just hacking out a quick example or implementing a full application to implement for a client.

One of the applications I've used recently, HighBall, is a combination of personal motive, work related effort, and a combination of other things rolled into one.  As I work through its build out I'll probably shift gears and priority a number of times.  Needless to say, it'll be a bumpy ride for sure.  However, I do intend to keep it up and maintain a flow with the effort.

Some of the other work that has come up is parallel threading and processing.  This isn't a high priority yet, but it isn't a low priority either.  Work demands have encouraged me to look into processing, scaling, and wide horizontal scalability.  My interests definitely lead me to this type of knowledge expansion, so expanding it, it will be.

Portland Code Camp is coming up on May 30th (with a prospective Friday afternoon get together and a possible Sunday event).  What exactly is code camp?  Per the Portland Code Camp Site, "Code Camp is a place for developers to come and learn from their peers. This FREE community driven event has become an international trend where peer groups of all platforms, programming languages and disciplines band together to bring content to the community."

We already have some great sessions coming up too including;  Dependency Injection: A Beginners Guide to Why Ninjas Are Awesome, Document-Based RESTful Databases (CouchDB), SOLID Principles in Practice and TDD/BDD, and Artistic Expression Through Code (Sure it sounds kinda weird, but I'll bet it'll rock!).  If you have a session you'd like to present check out the Sessions Page.

I hope to see many of you at the event, stay tuned for more information and make sure to check out the Portland Code Camp Site!