Where is the business intelligence?

This blog entry may instigate just a bit.  It will also be a little long for a blog entry. You've been warned.  I suppose though, if you know me & the work I do, that is not really something new.  I see something wrong, broken, or otherwise and I am likely to point it out and describe it in detail.

As I roll into 2010 coding, implementing, and rocking with Webtrends, I have noticed something lacking in the analytics industry.  I will add the clause that obviously Webtrends has people thinking about these things and actively working on this topic, but what I want to point out is a general issue.  Where is the other data, where is the existing data?

It seems, even though some company's kind of get to a certain point in connecting data points, not many really do.  The biggest reason is that most companies are just a few steps away from actually being able to do so.  The other even larger reason is, many do not realize what data should or should not be connected.

When someone starts pulling CRM (Customer Relationship Manager/Management), Analytics, POS (Point of Sale), ERP (Enterprise Resource Planning) data, and other sources into a single reporting repository we finally have real business intelligence.  Otherwise so many entities stumble through the land mines of data confusion.  I see this so much it really drives me crazy sometimes.

So how can a company or entity identify and connect these points of data?  It often starts with a ridiculously simple step.  At risk of oversimplifying things, let me just state the first step in getting out of the data confusion land mines is to first figure out your data.  Ask these things:

• What data does the business have?
• What data is currently used and available?

Do NOT ask what data you want, do NOT ask what may not be.  What you want to know first, and so many companies make this mistake, is to know what you know.  Do not, at the early stage of business intelligence information gathering start asking too many hypotheticals.  I promise the risk of failure increases exponentially for every hypothetical data point added.

Once you have identified what data is available, start figuring out how the data is related.  Once you understand the data you can then, and only then, make the huge leap to determining what data you want and how to get it to where you want.

Let me draw this out in a real world example.  Beware; I am using my creative mind now!

What we have so far, for Awe Widgets Incorporated, is several data points.

• Point of Sale/POS Systems in 300+ stores.
• Web Analytics (by Webtrends of course) tracking all sorts of great data points on the Awe Widgets Incorporated Website.
• Internal Accounting Software (Almost ERP, not really)
• In-house Built Customer Lists for Sales.

So there we go, four key pieces of tracking.  So how would they work together?  With a little further analysis (my key creative side now analyzes Awe Widgets Incorporated internal structure) and we find a few connections.

Correlation, POS to Webtrends Analytics

The POS System has a tracking identifier for customers which we can use to sync up with logged in users tracked via Webtrends Analytics.  This data can be used to derive who is and is not in stores purchasing.  In addition trending could follow the user flow to derive some actionable decisions on how to encourage online or store front shopping.  Just these two data points being connected add a lot of value.

Correlation, Internal Account Software ties to POS

Another data point tie in with the aforementioned POS & Webtrends data is the Internal Accounting Software (IAS).  The IAS holds information related to each sale, and other correlated information about how sales are going for the quarter, year, and other performance indicators.

Correlation, In-house Customer Lists for Sales

The sales department, in aggressive technical fashion has built a number of customer lists in Excel & Access.  The Access Application has a partially updated data store with a server based Excel file holding the updated piece of data about each of the sales person's current sales.  I know, I hear it now, every developer that is familiar with this scenario screaming, "OMG, you have your data in Excel AND Access, and it is supposed to have integrity, and be aaaaaaaaaaaaaaaaggggggggggggggggghhhhhhhhhhh noooooo!"  But you know, this @#$% happens.  : )  When things are like this, solutions get creative.

Tying Together the Pieces

Alright, this is when the awesome nerd bits start to happen.  But I have covered enough for this entry.  In the following entries on this topic I will step through this first data finding mission and start discussions on how to connect these sources and get that data mart, warehouse, or other middle tier piece into action.   I will continue on and lead into how the data can finally start telling a real story.  Because in the end, the real story is, somebody needs actionable data to act upon.  Does it really matter where it is?

Check out Part II of this series. 

The SQL Users? Group met at the KOIN tower downtown in the Robert Half Consulting meeting space.  The material covered was based on the presentation titled ?Crossing the BI Chasm?.

Some of the key points in the presentation:

• You must become knowledgeable about the specific business.
• You must be able to speak at a 30k altitude all the way down to the technical nitty gritty.
• Maturity of reporting;  infancy (excel chaos, multiple truths, ad-hoc workarounds), adolescence (dynamic querying tools, etc), mature (scorcards, etc, KPIs)
• ClickTek (anti- data warehouse people because they can get right to the data other ways), DataMart, DataStore, Cubes...
• Maturity levels of culture - infancy (don't understand data, IT overloaded with unrelated work), adolescence (learning what is available, IT starts to know business), maturity (data savvy).

After the presentation there was 5 BI Professionals answering questions from the audience.  Questions ranged from how many people are in or would be in a BI project to who is the key person to manage a BI project.

The multiple roles answer depended highly on the project size, which is obvious.  However the simple idea of people being generalists, and stepping into the communication hat, the guru hat, and then the learning hat all within a short period of time.

The answers where thorough and informative, with audience and panel members participating.

One answer that came later in the panel discussion was something that I?ll just parallel with props for Agile.  One of the main ideas behind Agile is lots of communication, effective communication, based on learning.  Always learning, eating, breathing, and living the learning, never stop.  To learn, one must communicate and successful BI is not possible with effective and steady unending communication and learning.

Again, part of the rocking Portland technology event!  A great night.

OLAP?

I got to thinking about my HighBall Project and thought, wouldn?t it be awesome to use some of my know how to wire up some serious cubes and analyze the results from site usage all the way to scheduling analysis?  It?s an idea, but I?m not really sure how I would do it yet.  I do however have some ideas in regards to scheduling and how a nice cube could be used to analyze effective scheduling usage.  Sounds like something TriMet could even use ? if they don?t already do these types of analysis.  ;)

Here?s my thought.

Take a schedule as the time dimension.  That?s simple enough.  Now take X number of routes as opposing dimensions and use ridership counts, peak load, etc as the fact table sums and such.  The data should align accordingly to the apex of routes and concentration of route needs by riders.  This type of data could be used to find out where ridership peaks and drops and how to fill gaps or even where to increase service.

?not really sure, got a ways to go before I try it, but it is an interesting thought for analysis.  Eventually I?ll give it a shot.

If you happen to be swinging down to Las Vegas during April 7th to April 9th be sure to say hello to Manoj Jasra of web Analytics world and of course, all my fellow WebTrendsians!  Manoj has a good blog so if you're an analytics professional (or just interested in the field), it's definitely worth a subscribe.

On the note of the event, Engage 09?, in Las Vegas I have some great work coming up at WebTrends which I'm looking forward to elaborating on.  Only problem, it?s top secret right now so you'll just have to keep reading to see what's up.  :)

Some of the entries I'll be writing in the next couple of months will be SEO, analytics, and data storage related in nature.  I've got some architectural ideas brewing for further integrating Enterprise Business Intelligence, very high horizontal scale, and analytics.  Of course, I'll be going on about how you can use WebTrends to do these integrations.  So if you're in the business, you'll want to click on the feed and get subscribed.