Category Archives: Presentations

On outcomes and impact

There are many ways to find out about things. Research is obviously part of that. And research likes to use quantitative measures in order to maximise objectivity, even if these measures don’t give you much meaning.

Let’s look at hit rates on a website – a metric commonly used for “statistical purposes”. What does it mean? Well, it means that a website or page view has been looked at a certain amount of times. The inference is that the more hits you have the better must be the result. But what is the result?

If the intended result is to have as many hits as possible since one assumes hit rates equate with “eyeballs”, then surely high hit rate numbers are great. But is this the result an organisation really wants from it’s website? What happens as a consequence of the “eyeballs” is the question I really want to get an answer to. In reality, high hit rates could indicate a bad website. Your website visitors and customers are clicking away, frustrated by their inability to reach an outcome they want to achieve. Just get those click rates up and everything will be fine….hmmmmm.

Let’s do a survey then. A survey is actually pretty limiting.  A questionnaire is bounded by the construction of the questions and limited answer options. In nearly all cases, one could answer a question yes or no, depending on the particular circumstances at some point in time. Surveys also don’t do a great job in measuring continuous change over time. And conducting surveys or focus groups with large numbers of people are often difficult and time consuming, certinly if a continuous process is required.

Yet these methods are still held to be superior to more qualitative approaches to research. However, if you actually asked your website users what they thought of the website, perhaps they might tell you that it takes a lot of clicking to get through to complete the task at hand. They might tell you that your website is poorly organised, with lousy navigation and confusing labels. They might tell you that the photos on the home page add nothing to their customer experience. They might tell you that your website could better… for them. And if you have a continuous dialogue with them, they will be even more insightful as to how to improve or validate what you are trying to do. Observation at point of impact is a good way of thinking about this.

I can see some meaning from getting those kind of responses! Click rate numbers – forget it. Now I have real information that can make an impact to the people I say (and the organisation) I am trying to serve.

So what we are interested in finding out is impact. What is the  impact that occurs from what we are doing? This is different to outcome. Click rates are an outcome. Obtaining continuous feedback to ensure satisfied customers buy from you, recommend you, and stay loyal, is another.

Now what if we could get this feedback quickly, continually over time, on a large scale, context-sensitive, and in a way where the person giving us the information gives it in terms of how it affects them, and not through some intermediary or stilted survey method?

I set the scene this way to introduce some thoughts from a presentation at the ANU yesterday by Dave Snowden,  special guest speaker at the ACT-KM forum. Dave talked about a number of current projects he was working on. The common element from his talk was the importance of  determining impact and how then to take relevant action as a consequence.

I will use the example of the Liverpool Slavery Museum in the UK from Dave’s talk yesterday; albeit the Children of the world project was for me the most fascinating.

One could count the number of people going through the museum each month and year. The numbers might indicate level of popularity but one can’t be sure. At best, they show that “x” number of people came and paid “y” number of UK pounds to do so.  One could do a simple accounting calculation at this point and perhaps leave it at that.

But what if you wanted to know what effect the museum had on people? What if you wanted to know how successful the museum was in educating visitors about slavery, or in providing a unique experience? What really was the impact of the museum visit?

[It is of course true that if you don’t want to know about your customers’ experiences and are happy with just throughput figures – akin to an assembly line – then impact will have very little interest for you. The process will be sufficient].

Dave told us how there are computer screens and keyboards at the museum where people can record their experiences and feelings about the museum exhibits. People can nominate any of the individual exhibits to make a comment or express a feeling. The people making these comments are then able to “index” or tag their comments using terms chosen freely that signify meaning to them. Nobody  is interpretating what they say and adding any bias. At the same time, this information capture is continuous and provides for scale, something a static survey couldn’t do. The museum now has thousands of narrative fragments “indexed” by the individuals themselves. This information is aggregated and patterns observed. These patterns may suggest a change to a particular exhibit, or perhaps some alteration to how a museum education officer conducts a group tour.

In the Liverpool Museum case, they have both quantitative information (number of visitors and monies received) AND what impact the museum had on the visitors.

Yet still there are detractors:

  • stories (narrative fragments is the preferred method used by Dave Snowden) are not real facts
  • some people may just write junk and not tell the truth
  • it’s all so subjective

All three statements might be true. The point is, if we want to measure impact, then we need to know what people think and what effect something had on them. And we need to know what they think, not what what we might guess at. The capture and aggregation of narrative fragments is a good way of doing this. “Junk” can be easily discarded but sometimes “junk” may be of interest as a weak signal, something we should pay attention to. Where you want to establish an impact on people, of course there is subjectivity. However, how the narrative fragments are captured, aggregated and used is quite a rigorous and objective method in itself.

Lastly, no matter what the method, unless people use the tools correctly and respond appropriately, no research activity will have any validity.

Advertisements

On conferences

Apart from just moving house (again) and waiting to get the utilities connected (again), I have been thinking about conferences.  My thinking was instigated by an approach I received from a conference organiser to present at an upcoming conference in September on collective intelligence. Almost at the same time, another conference organiser contacted me asking about case studies in government that could be used to demonstrate effective collaboration. And, of course, there is the plethora of conference invitations and conference pamphlets that come across my email and my desk each week advertising future conferences with discounted early bird rates. The message is clear:  Get in quick, folks!

As I thought about all these conferences I was conscious of the fact that essentially they were all the same. The conference organisers invite speakers to present under a particular conference theme. People attend the conference to listen to these presenters, network with professional peers, and hopefully find some useful information and learnings that will be of personal or workplace relevance. It is pretty standard conference fare.

Now that’s all very well and I am happy to participate in such events. But I am thinking there could be other ways to provide conferences with something different. I know there are un-conferences and the like but I am thinking of something else.

Firstly, I’d be interested in a conference where the theme was not so tightly regimented. I am thinking of a conference at which there are presenters speaking on different and unrelated topics but from which the audience could develop particular personal or collective themes themselves. The audience would therefore become an active participant by discussing these emergent themes rather than having the themes imposed upon them. I see strengths and weaknesses in this approach – my interdisciplinary preferences are also at work here. But at least there would be some active thinking, rather than what often happens at conferences is passive and sleepy acceptance.

Secondly, I’d like the keynote to be in the form of an interview. There would be an interviewer but I’d like the audience to be able to take part as well – perhaps providing some questions in advance from which the interviewer and conference organisers could put into some form of meaningful order (randomness would also work for me but I think effective interviewing relies on a logical progression). The interview lends itself more to a storytelling approach rather than a lecture. The Q&A format stimulates quetioning in the minds of the audience throughout the keynote – something that could be followed up after the conference as well.

Thirdly, I’d like conferences to have some follow-up. We go to a conference, hear some stuff, maybe feel pretty good about things, and then go home or back to the office. Why can’t we tap into the collective experience of people after the conference officially finishes? If the conference is interesting and participatory, then there is the opportunity to extend the discussion outside the formal conference environment.

And talking of follow-up, I’d really be interested in any game that could be developed to reinforce or stimulate further thought about the conference presentations. I am thinking simple card or board games, but more technical games on  a website would be equally useful (if a tad expensive!).  A “snakes and ladders” for effective knowledge management would be absolutely fantastic! Games are great information reinforcements and something worthy of considered thought.

And lastly, I’d like conference organisers to think more creatively about conference “notes”. A few Powerpoint slides from presenters in a drab folder doesn’t cut if for me these days, I’m afraid. It also makes it difficult for presenters since Powerpoint slides often become the presentation (the defacto content) rather than acting as a supporting element to the actual presentation. Powerpoint slides are not conference notes! I really like the idea of podcasts and I am a big fan of the podcasts that come out of the SXSW Conference each year. Great stuff!

Now if I can work out how to introduce African drumming into a conference I would be really satisfied…

On breakfast, lunch, and tea

There are a number of ways in which information and knowledge can be disseminated and exchanged, including breakfast, lunchtime or evening meetings. These meetings can be internally or externally based. Organisations, like the Society for Organisational Learning Australia (SOLA) for example, have run morning information sessions for members featuring a special guest speaker. I first heard Dave Snowden at such a breakfast in Sydney a few years ago. There are plenty of examples across a range of businesses and professional associations.

At Rabobank I organised a number of lunchtime meetings featuring a guest speaker to discuss a topic of interest. The last one I organised dealt with water infrastructure and the speaker was a friend of mine from a leading law firm. The lunch and discussion afterwards were a great success. I hope to arrange similar sessions in a different organisational context in the future.

Having been both an attendee and organiser of such functions, it is often difficult to gauge how successful these meetings will be and what size of audience will actually turn up. Breakfast and lunchtimes are often busy times for people too, even if outside the official working day. And worst of all, sometimes the quality of the presentations are marginal, causing a major rethink as to the value of these types of sessions.

So what is it that makes these events a success?

From my experience, and from the example of the Dow Jones Factiva breakfast session I attended this morning (and the previous session last year), there are five key dimensions:

1) the speaker/s MUST be of good quality with something interesting to say (they don’t have to be famous but they need to be able to speak to a group of people in a relaxed style and have something relevant to say)

2) the speaker MUST be able to present in a professional but informal manner

3) the session should not be too long (1-1.5 hours is sufficient) and timing must be tightly controlled to avoid wasting time

4) a good mix of people in the audience widens appeal and scope for questions and post-session conversation

5) the venue MUST be comfortable and able to fit the number of people attending

The Dow Jones Factiva session this morning had three speakers: Chris Pash from Dow Jones Factiva and his self-advertised blog, Hugh Martin from APN Online, and Martin Quadroy from Telstra). All three speakers in their individual ways provided interesting, entertaining, and relatively short presentations covering content, online marketing, and competitive intelligence.

Well done (again) Dow Jones Factiva!

On SXSW 2008

I am really disappointed that I wasn’t able to get to Austin, Texas, this year to attend SXSW Interactive. I will get there one day (I hope) but in the meantime we are fortunate that the organisers provide podcasts of the presentations and discussions.

Check out this interview with Mark Zuckerberg of Facebook fame – some interesting discussion on how people in different countries are using Facebook and what Facebook holds for the future.

Warning: the interviewer is really annoying, thinks she’s clever, and she probably thinks she is real cute.  Not impressed. Note to potential interviewers and facilitators – “it’s not about you, ok?”

On tagging and the enterprise (and RSS)

I want to conclude my blog summary from the presentation I gave last week on tagging and the enterprise. The previous three entries should be read in conjunction with this instalment, if you haven’t followed the story so far…

I used IBM’s dogear as an example of an enterprise using tagging within the firm. However, instead of me explaining all about it, I have listed here three sources that explain the way in which social bookmarking and tagging may be used within the enterprise, including dogear at IBM:

Am I being lazy? Well, the web is all about links so I may as well use them!

Finally, as an aside, I discovered today a way of using RSS feeds to populate a newsletter. Yes, it is an interesting combination of web 2.0 (RSS) and the old way of communication (newsletters) but it may well work as a valuable bridge for people still not accustomed to the full array of web 2.0 communication channels. The product is Nouri.sh and it’s relatively new. It is definitiely worth a look if you want to mesh RSS content within a newsletter format.

And if anyone knows about other services like this, please advise with a comment!

On tagging, the grey side

My last two posts have been about tagging based on my presentation last week at the conference in Sydney, “Enhancing search and retrieval capabilities and performance”.

I want to look at some of the perceived disadvantages of tagging that I briefly mentioned in my presentation:

  1. Lack of specificity – refers to the fact that an item can have innumerable headings (tags) and there is no fixed agreement as to the most suitable term. A formal taxonomy and classification system have been the traditional ways of asserting specific terms to items.
  2. Ambiguity and inconsistency – because anyone can apply a tag to an item, there will be a multitude of tags that do not clearly and consistently apply to a specific item. Some people may tag something as “locomotive” and another “train”. The same person may use “locomotive” now but three weeks previously used the term “train”. And train may in fact not refer to a locomotive at all (with or without carriages or wagons) but to a wedding dress, a series of thoughts, or to an adult education class.
  3. Lack of structure – The traditional relationship between broad and specific terms (the parent-child tree structure that historically organised information into “like things”) is not there in a tagging system. Weinberger refers to a tagging system as one that looks at the leaves on a tree rather than just the branches.
  4. Problems with stemming or truncation – words like plurals, or words with a s or z in them.
  5. Ceding control of search terminology to the “inexperienced” – using the correct terms is an important exercise not to be trifled with by amateurs and the inexperienced professional.

It is true that there will be imprecision in tag terms and inconsistency in the application of tags to items that look to be the same things. It is also true that the same individual may use different tags over time to describe essentially the same thing. And tagging might thus be perceived as a mess, needing an experienced taxonomist and library professional to make sense for us. People in the information business who like order and structure have a long historical paradigm to work from.

Yet all is not lost. Tagging will become self-refining, gradually highlighting preferred terms (perhaps through a tag cloud) or via suggested or similar headings. Collaborative tagging and folksonomies will help shape a form of group consensus leading to a meaningful sense of order. And technologies will improve to cater for some of the weaknesses of current tagging systems. One example is Raw Sugar.

Overall, tagging will continue to grow simply because digital information will grow at time-warp-like speed. The sheer scale of the digital world, and the cost of ordering that digital information, will not easily permit formal and timely classification. Just imagine trying to keep up with all the blogs in the world, let alone the individual blog posts from each of them. 

Tagging will become more important and self-fulfilling due to both the technology and the demographic changes in society, responsive to the digital world and the need to make sense in it for individuals and their peers. The changing nature of information, and the new consumers and producers of that information, means that change is inevitable.

Interestingly, a recent article highlighted the changing nature of reading – the development of an information browsing culture among the digital natives. The impact of the digital world should not be underestimated.

In looking at tagging so far, perhaps one could say we are in a period of transition from the structure and hierarchy of giving order to physical information (like books, journal articles and celluloid film) to one where digital information allows for innumerable access points, innumerable tags and descriptors, and seemingly available from anywhere.

[Of interest, check out this podcast from Beth Jefferson on transforming public libraries’ online catalogues into environments for social discovery of resources that are catalogued not only by librarians, but also by patrons. A salient quote on social cataloguing – collaborative tagging if you like: “the metadata people create by cataloguing content is what enables social search and discovery”. Beth Jefferson wants to enhance social search and discovery across North American public libraries through collaborative cataloguing, whether by evaluative comment or by description. Tagging and thesauri may indeed coexist.]

So the question remains – is the traditional way of ordering information and establishing a single authority for fixed terms appropriate in the modern digital world? And practically speaking, what is the right balance between order and miscellany in any given context?

I will feature one more blog post on the tagging issue looking at how the enterprise (the firm, not the fictional space ship), might take to the tagging phenomenon. Stay tuned…

On the positive side of tagging

In the light of what I discussed yesterday with respect to my conference presentation on Tuesday, I want to move on to tagging. Tagging is essentially unstructured metadata that is assigned by the content creator and the readers/users of the content, the latter called collaborative tagging. The user-generated classification that emerges is called a folksonomy.

Examples of digital content using tags include de.licio.us, Flickr, LibraryThing, Technorati, and Youtube. Even the web-based news services are using tags, like the ABC in Australia.

In addition to the tags themselves and the act of tagging content, a collection of tags into a group showing relative emphasis or popularity is called a tag cloud.

There are a number of benefits from using tagging and they can be broadly summarised as the following:

  1. terms meaningful to the content creator and/or readers (and not just those terms allowed by a single classification authority)
  2. establishes relationships between content and the people connected to the content (both content creators and readers)
  3. is inexpensive to undertake, especially in relation to traditional cataloguing and thesauris construction
  4. scales exceptionally well, thereby suiting the miscellany of digital space
  5. aggregates especially well, thereby harnessing the so-called wisdom of crowds
  6. permits multiple access points to information instead of just bibliographic data
  7. permits discovery of a range of other items tagged by other content creators and readers
  8. overcomes the lack of currency when using traditional fixed forms of metadata (like the established classification systems)
  9. is highly participatory in that people freely choose the relevant tags they regard as appropriate to their own content and to the content of others
  10. as more applications make tagging available, and as the new digital generations increasingly enter the workforce, tagging will become the established norm in the digital information environment (we can see how blogs may offer such an opportunity)

Point 10 is especially important. There is already some evidence of tagging popularity from a Pew Internet Report showing that nearly one-third of US internet users tagged content. As tagging becomes more familiar and mainstream, new opportunities will open up to enhance the popularity of tagging – what I have called “the tagging locomotive”.

I’ll stop here (but with another post to come) with some recommended readings:

Everything is miscellaneous by David Weinberger

Ontology is overrated by Clay Shirky

Folksonomies: power to the people by Emanuele Quintarelli