Factiva Social Media Roundtable helps to answer “What should we measure”

Categories: Events, Social Media, Social Media Measurement, Voice of the Customer, Web Analytics, Web TheoryPosted on December 6th, 2006

Summary
Last night, Factiva hosted a group of Social Media Practitioners, Bloggers, Corporate Program Managers, PR consultants, room in Palo Alto to try to make sense of Social Media Measurement. Most agreed measurement is important, After last night, we’re a little bit closer, the details of the findings are below.

Background Information

I’m a former customer of Factiva, I suggested they progress their offerings to better measure CGM/Social/New Media, this event is a response to that discussion.
Those that attended already are thought and practice leaders the audience was not the uninitiated.
Metrics of last era to don’t apply to Social Media, I believe that a new way of measuring will come about.
Social Media is complex, complicated, and filled with nuances, as it mirrors or extends ‘real’ life, we will never be able to fully measure and categorize, it doesn’t mean we shouldn’t try.
It’s important to note the following:

1) Social Media is about people. People connecting to other people to build better relationships, fostering communities and increasing collective knowledge. (I enjoy Brian’s blast at SMO tactics losing focus)
2) Measurement and Metrics are one way to help to tell the story of Social Media

Observations

It was assumed the folks in the room all ‘get’ social media, we were not here to answer “Why” or “What” but extending the conversation to how.
A healthy cross section of our ecosystem was present: Social Media practitioners, Folks at Corporations that actually deal/deploy/monitor Social Media, PR companies that ‘get it’, Bloggers, Social Media Consultants (like myself), and Factiva who is a vendor that measures media. We separated this into three distinctions: Companies, Consultants and Vendor
Most people thought measurement was important to some degree, although one noble corporate blogger is just focused on reaching and connecting to customers to build better products.
I quoted the meme by Steve Rubel (I incorrectly thought it was Winer) that Page Views are dead.
Although I suspect we were all thinking the same thing, It was very difficult for us to verbally agree to terms that would encompass the attributes we wanted to measure.
Measurement depends on what the business goals are. Some folks tied measurement back to hard ROI: (Does Social Media shorten the sales cycle/reduce cost, and does it increase revenues). For other companies, business blogging is about thought leadership, or building better products, how could you apply these models to the Wells Fargo blogs, that are designed at reaching out to communities?
Zibibo in Palo Alto was a fine venue, the food was amazing, the atmosphere perfect for conversation.
Some may questions should we measure at all? Corporations need to measure.
People place different values on attributes. We allowed guests to prioritize their measurement needs, we separated the voting by Business Corporation vs Consultants/Bloggers vs Vendor. The results are here
Glenn conducted some breakout sessions, a variety of findings from groups emerged, I’m hoping they, or scriber Jeremy Pepper will publish their findings.
Criticisms: One PR consultant privately pulled me aside and had just concerns that if a company does create such an index it should not be hoarded for raw corporate profit, it should be a community resource, perhaps open source.

Exercise: What attributes should be measured, and which are the most important

Left: The group exercise yielded this prioritized list of Social Media Attributes to be measured.

I leveraged this exercise from former colleague Mary Eileen, it’s also an Information Architecture exercise. I’ve performed similar card sorting exercises before to help determine content priorities for websites.

Attendees were encouraged to voice attributes that were important to measure, they were then grouped and refined.

Roles were broken down into three segments Corporation, (blue dots) Consultant (red dots) and Vendor (yellow dots) and then asked to vote by placing dots on attributes. (see this voting on this page and this page)

There were more Consultants than any other group, followed by Corporations, and Vendor category only had two voters

Each attendee was given three dots of the same color and then voted.

Findings
Although not fully scientific, the thought leaders present voted on which attributes should be measured and then prioritized, in priority it yielded the following list:

1) Participation and Engagement (Voted 14 times)

Corporate folks voted for this as the most important

They wanted to know how are marketers interacting with them, is this quality?

2) Influential Ideas: Memes, and their intensity over time (Voted 11 times)

Consultants voted this as the most important to them

This could make sense as PR is traditionally hired because of ability to spread message, is this mass?

3) Relevance (Voted 8 times)

Does this mean the long tail matters?

4) Sentiment/Tone/Opinion/Favorability/Emotion (Voted 8 times)

Factiva product team this the most important to them

5) Content (Voted 6 times)

6) Relationships and Connections (Voted 5 times)

7) Analytics and Activity (Voted 4 times)

Whoa, this is way down on the list, thereby showing old measurements are not relevant to Social Media

Community Activation or Call to Action (Voted 3 times)

9) Reach (Voted 2 times)

Does this mean a-listers are less relevant?

10) Tied: Conversation Index/Engagement (Voted 1 times)

10) Tied: Demographic/Who (Voted 1 times)

Some were surprised the “Who” was not voted as high. Does this mean that if anyone is engaged in the conversation then they are influential and part of the market? Cluetrain supports this as anyone conversing is in your market. Thereby, those that participate matter.

More Background info of Social Media Measurement
Factiva is a company owned by Dow Jones, and does a great job at measuring EGM/Traditional/Old/Oneway/Broadcast media. Like most measurement companies, they’re making advances to learn about Social Media. Clients are asking more and more about what is it, (although it varies in understanding) measuring the impacts of blogs, wikis, social sites, podcasts and whatever comes next will be needed. At the time when this discussion started, I was a customer of their product. It didn’t meet my full needs in measuring CGM. Since leaving my previous company and joining Podtech as a Social Media consultant this quest to understand Social Media Measurement is perhaps even more important

Factiva listens
Daniela Barbosa, a Factiva employee has really been reaching out to this new medium at a pretty traditional company. She really deserves the praise of her colleagues, the social media community and most importantly, customers. As a former customer of Factiva, Daniela would read my blog daily, and responded in comments, her blog and even using YouTube. Additional Kudos should be delivered to the product team, Glenn, Saurabh for reaching back, listening and trying to understand this new market to build a better product. For those within Factiva and Dow Jones, please learn from what this team has done. Factiva, is a company that listens to the market and to it’s customers.

Additional Resources

Other Event Bloggers

Disclosure of some sorts
As a Web Strategist this is a passion topic to me, my participation was voluntary and unpaid/ They did treat me to a few wonderful meals, and presented me and other guests with an iPod shuffle (which ended up going to the wife anyways). It’s likely I’ll be sharing what I’ve learned with Podtech customers, but I also blog my findings to better the community.

This is just the start
By no means is this the definite answer, and the methodology and process was not scientific. This is just one data point in a fast changing, amorphous, barely defined medium, more on this soon.

Share This Post:

This entry was posted on Wednesday, December 6th, 2006 at 9:09 am and is filed under Events, Social Media, Social Media Measurement, Voice of the Customer, Web Analytics, Web Theory. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

What a post. Clearly Jeremiah has some passion for this topic. We (the Factiva folks in attendance) are still digesting the enormity of the inputs from last night and continued the discussion into the wee hours last night. But we know that this event was more than worth it because we got smart people together and had an open discussion about a very important complex concept. Thanks again to Jeremiah. More to come as we work on the analysis.

Posted by Glenn Fannick on December 6th, 2006 at 10:51 am

Yes, this could be separated into many many posts, but I wanted to capture the highlights, as a springboard.

I agree, this is just the start, and it’s just one data point of many.

Posted by jeremiah_owyang on December 6th, 2006 at 11:08 am

Thanks for detailing the take aways, there were some ideas I forget that you captured. I’m really interested in pushing on the participation/engagement metric. What does that really deliver to the marketer? I think it’s important and an intuitive metric, but what does it really get to, ultimately? Does it tell us about influence? Does it tell us about reach? Or does it tell us about the vibrancy of dialog? What’s interesting about that last bit, is that when a dialog is really in play, participants whose minds are not made up about a topic are most impressionable to influential ideas. If you’re trying to increase brand awareness, for example, getting your message into a dialog that’s really active may have much greater bang-for-the-buck than in a dialog that’s flat.

Posted by Chris Kenton on December 6th, 2006 at 5:09 pm

[...] Factiva Social Media Roundtable helps to answer “What should we measure” (tags: analytics social media) [...]

Posted by Elroy Jetson » Blog Archive » links for 2006-12-07 on December 6th, 2006 at 10:31 pm

I read the suggested list and voting results. It is still very vague.

I was hoping to see more concrete definition.

I post a detail comment few days ago but it didn’t make it to the blog. I lost my writing.

I like to extend some Web 1.0 metric to cover social media. Use more familar metric terms to describe the measurement instead of re-inventing terms cause more confusion.

Again the pageview is dead because of the increasing popularity of RIA (Rich Interactive Application), AJAX in Web 2.0, emerging of WebOS or Webapp, high level of Audio and Video consumption in social media space. Many issues are around user behavior, conversion rate, resulting in ROI.

Metrics are processed via log file and/or database. Community has to agree on how to define active user, valuable user, life time value of a customer/user on various type of social media like blog, wiki, etc.

Posted by Bess on December 7th, 2006 at 1:10 am

[...] Web Strategy by Jeremiah » Factiva Social Media Roundtable helps to answer “What should we measure” (tags: socialmedia to_read) [...]

Posted by Ketcheson.net :: links for 2006-12-07 on December 7th, 2006 at 1:27 am

Bess

Thanks for this, yes, it indeed is still very cloudy, we’re looking at new ways of measuring new mediums and communications.

Please note that it’s not just about ‘metrics’ but really about ‘measurement’. It’s not easy to put metrics around any given conversation two people have offline as well as online.

Posted by jeremiah_owyang on December 7th, 2006 at 5:24 am

Chris

Amazing points. Participation and Engagement could be one attribute that helps to gauge ‘interest level’.

But as we know in real life, just because someone interacts with something, it doesn’t mean they’ll care about it 10 minutes later.

Posted by jeremiah_owyang on December 7th, 2006 at 5:26 am

[...] Jeremiah Owyang posted today on a Round Table he attended with the folks of Factiva and some others (bloggers, corporate program managers, PR consultants and social media practitioners) to discuss what should be measured in the Social Web. They came up with a nice list which they voted against and ranked according to the most votes. [...]

Posted by Instant Cognition » Blog Archive » Thoughts on Social Media Measurement on December 7th, 2006 at 8:17 am

[...] Interesting listing of the important metrics in the blogging / social media space (text and comments entirely Jeremiah's):"1) Participation and Engagement (Voted 14 times) Corporate folks voted for this as the most important They wanted to know how are marketers interacting with them, is this quality? 2) Influential Ideas: Memes, and their intensity over time (Voted 11 times) Consultants voted this as the most important to them This could make sense as PR is traditionally hired because of ability to spread message, is this mass? 3) Relevance (Voted 8 times) Does this mean the long tail matters? 4) Sentiment/Tone/Opinion/Favorability/Emotion (Voted 8 times) Factiva product team this the most important to them 5) Content (Voted 6 times) 6) Relationships and Connections (Voted 5 times) 7) Analytics and Activity (Voted 4 times) Whoa, this is way down on the list, thereby showing old measurements are not relevant to Social Media Community Activation or Call to Action (Voted 3 times) 9) Reach (Voted 2 times) Does this mean a-listers are less relevant? 10) Tied: Conversation Index/Engagement (Voted 1 times) 10) Tied: Demographic/Who (Voted 1 times) Some were surprised the “Who” was not voted as high. Does this mean that if anyone is engaged in the conversation then they are influential and part of the market? Cluetrain supports this as anyone conversing is in your market. Thereby, those that participate matter. " Tags: Analytics , Industry , Marketer , Metrics , ROI [...]

Posted by Kalivo.com - Important social media metrics, from Web Strategy by Jeremiah on December 7th, 2006 at 10:22 am

Understand the measurement part. Our web group will cover analystics. Unfortunately we won’t be able to cover it after 1st quarter of year 2007.

I am familar with the standard and measurement in Web 1.0 on online advertising, e-commerce conversion, web analytics. I hardly mention that I taught Marketing Research with Quantitative Statistics along with industrial experience in internet space.

I have many useful past research materials on the subject. My limitation is I don’t have the spare time to condense these information and transform them to relate to Web 2.0. Keep posting on your research. I’ll see what I can do. I can draw up diagram to illustrate possible measurement and investigate the backend infrastructure on blog and wiki.

Posted by bess on December 7th, 2006 at 7:51 pm

[...] The people here are all fast movers, self starters, it’s really a talent shop, I’m working hard to catch up with this crew. PodTech is hitting a milestone this week, we’ve moved to a new office on Page Mill, (everyone gets their own desk) a much larger office than the incubator startup at Sand Hill road. (about the size of two garage stalls) Keep running This last few days has been so busy, as I’ve changed my communication mix, ran a roundtable exercise, attended and supported an event held by my previous CEO, moved to a new office, and today, I’ll be meeting my first client that I’ve brought in. We’ve some big news to share soon, stay tuned. [...]

Posted by Web Strategy by Jeremiah » Week 2 at PodTech and Community Feedback Encouraged! on December 8th, 2006 at 6:22 am

[...] Not everyone thinks Social Media Measurement is great idea, or it can be done, mixed reactions here. I agree, but we’re going to have to do it anyways. Clients demand it, they pay me. [...]

Posted by Web Strategy by Jeremiah » The Problem with Social Media Measurement on December 8th, 2006 at 7:09 am

Eric Peterson (Author of Web Analytics Demystified) offers some fairly specific tactics on measuring engagement - you might want to check it out.

Posted by Clint Ivy on December 8th, 2006 at 9:58 am

[...] Our research indicates this is the top needed measurement. This is cool, someone has already done the ground work for some of the high level attributes that were discovered in this week’s roundtable on Social Media Measurement, you can see the findings. [...]

Posted by Web Strategy by Jeremiah » Web Analytics Community starts the Engagement Attribute, adding some thoughts. on December 8th, 2006 at 1:07 pm

[...] In fact, this issue has been talked about a lot lately, particularly in the context of social media measurement. It was one topic of a recent Factiva-sponsored roundtable, which was very nicely written up by Jeremiah Owyang and Jeremy Pepper, among others. These folks are diving nicely into the measurement question, and I highly recommend a detour their way. [...]

Posted by Ketcheson.net :: VerizonMath and how to answer the “Why should I care about this blog?” question on December 9th, 2006 at 2:53 pm

[...] Now, score one for Eric for actually thinking through a framework for measuring engagement, and putting it into practice, but it’s late and I’m feeling nit-picky. Because Engagement has been talked about so much lately in terms of Web 2.0 and ‘Social Media‘ I’m putting Eric’s listed activities through that lense. Of the 16 activities, only 5 measure social media engagement: [...]

Posted by Instant Cognition » Blog Archive » E.T.E. Part 2 on December 21st, 2006 at 1:54 am

[...] More on Social Media Measurement We learned that Engagement is the key metrics in this roundtable hosted with Factiva. I’ve catalogued my other thoughts on Social Media Measurement with this category tag. Segment your audience based on how much they participate -Creators (smallest group) -Critics -Collectors -Couch Potatoes (largest group) [...]

Posted by Web Strategy by Jeremiah » Web 2.0 How to Measure Social Engagement: Blogs Podcasts and RIAs on January 19th, 2007 at 7:31 am

[...] Of course what you end up measuring is dependant on your business goals are. As Jeremiah Owyang put it: Some folks tied measurement back to hard ROI: (Does Social Media shorten the sales cycle/reduce cost, and does it increase revenues). For other companies, business blogging is about thought leadership, or building better products, how could you apply these models to the Wells Fargo blogs, that are designed at reaching out to communities? [...]

Posted by Social Media Optimization » Measuring Social Media Campaigns on January 22nd, 2007 at 7:35 am

[...] Also at this meeting was Jeremiah Owyang who took this shot of the flip chart they produed and has blogged about the meeting here. Number 1 Engagement. Social media buyers are interested because online there is a lot of engaged brand evangelizers, Marketing Vox report a Yahoo study that says 40% are brand advocates. For bloggers this finding is interesting to say the least: 32 percent either evangelized their favorite brands via blog posts or message board entries, compared with just 8 percent of the non-advocates. [...]

Posted by Grippr Social Media Metrics » Blog Archive » Measuring Social Media on January 22nd, 2007 at 12:44 pm

[...] Matt Toll of Factiva presents today at Frost and Sullivan’s Sales and Marketing conference, his session, entitled “What have you done for me lately, measuring Marketing’s unmeasurables” is right up my alley. As a former customer of Factiva’s I’m pretty familiar with some of their offerings and products. A few months ago, I helped them with running a Social Media roundtable on Social Media Measurement, do check out the findings. [...]

Posted by Web Strategy by Jeremiah » Session Notes: Measuring the Unmeasurable with Matt Toll of Factiva on January 22nd, 2007 at 2:10 pm

the problem with all this measurement talk is that no one is being clear about what it is that they’re measuring. Factiva, Biz 360 and Cymfony are all measuring blog content — what OTHER people are saying about you in blogs. This has nothing to do with measuring the effectiveness of your own blog. It has everything to do with measuring your reputation, understanding the marketplace and lots of other good stuff, but it is a totally different discussion from Web 2.0 metrics and replacing page views and click thrus. The measures of success for your own blog are one discussion. The measures of your reputation among bloggers is a different discussion. lets be clear about what we’re measuring.

Posted by Katie Delahaye Paine on January 23rd, 2007 at 4:59 pm

[...] Focus on Social Media Measurement PodTech’s clients are asking for this, so it’s toward the top of my interest levels. I’ve been very interested in this topic, as I was trying to measure while deploying in my previous corporate role. In my current role as a consultant, many of our clients are asking “what does success look like” and “how do we show this to management”. I’ve also been involved with Factiva and thought leaders from our industry for a roundtable. View all my posts tagged Social Media Measurement. [...]

Posted by Web Strategy by Jeremiah » ROI of Business Blogging is not a Myth. Reviewing Forrester’s “Calculating The ROI Of Blogging” on January 31st, 2007 at 6:04 am

[...] Jeremiah Owyang: Factiva Social Media Roundtable helps to answer “What should we measure” [...]

Posted by Measuring Engagement Metrics for Social Media Marketing - Part 2 at Emergence Media on February 1st, 2007 at 1:40 am

[...] There has been a great deal of discussion recently about methodologies for measuring and analyzing within the open media space. Jeremiah Owyang has a great article recapping some of the main takeaways from a recent Factiva roundtable on what to measure in the open media space. Lately, I have been doing a lot of thinking about how we might be able to categorize blogs. The question is of prime importance because it directly impacts the manner in which you can engage with a particular blogger. It also can inform the way we can measure the effectiveness of outreach to the blogosphere. Starting with the premise that blogs are extensions of an individual, I initially thought about using some sort of personality-profile model like Myers-Briggs, and attempting to tweak it so it can be applied to bloggers. After a bit of experimentation I felt like I was trying to fit a square peg into a circular hole, so I instead am going to list out all of the different profile attributes I have come up with, by presenting them as binary archetypes. These categories are not mutually exclusive, nor am I going to claim that this list is complete. I’m sure there is a great degree of overlap between them, and if anyone thinks there are two categories that are redundant I’d love to hear about it in the comments section. Without further ado, here goes: [...]

Posted by Categorizing blogs and bloggers at Ether Breather on February 5th, 2007 at 2:03 pm

[...] A few months ago, Dow Jones’s Factiva (They also stopped by at PodTech yesterday as well) and a dozen Social Media, PR experts, and Corporate clients got together in Palo Alto to discuss what are the requirements for new measurement, I encourage you to read the whole post, as well as look at the trackbacks to find out what else was said. Here’s a summary of what was uncovered: [...]

Posted by Web Strategy by Jeremiah » Growing up and Replacing the Antiquated Page View Metric on February 15th, 2007 at 8:48 am

[...] Anyway, I’m going to have beers with the good Mr. Ivy next week and I didn’t want that whole “social media” thing hanging over my head. And while I recognize that this metric (which I still have yet to share the calculation) doesn’t capture fully the elaborate needs of the really smart folks working to pound out Social Media Measurement, I heartily agree with Clint’s friend Jeremiah Owyang when he says that “Social Media is about people. People connecting to other people to build better relationships, fostering communities and increasing collective knowledge” and “Measurement and Metrics are one way to help to tell the story of Social Media.” [...]

Posted by Web Analytics Demystified » Blog Archive » Calculating engagement, part III … social engagement and relative content grouping on March 3rd, 2007 at 11:20 pm

[...] Social Media Measurement has been a focus area of mine for some time. Factiva’s Dow Jones ran a roundtable last winter where we grafted out the requirements for what Social Media Measurement should look like. Engagement, Interaction, Attention and new terms that we’ve not measured will be in addition to Page Views and Web Activity. We want to answer why did they come, and how did they feel are key. [...]

Posted by Web Strategy by Jeremiah » The challenge of measuring the Distributed on May 14th, 2007 at 6:57 am

[...] Social Media Measurement Roundtable with Factiva [...]

Posted by Web Strategy by Jeremiah » Web Strategy: How to Measure your Social Media Program on June 7th, 2007 at 8:02 am

[...] Head’s Still Spinning • Jeremiah Owyang: Factiva Social Media Roundtable helps to answer “What should we measure” • Images on Flickr of event (over [...]

Posted by Social Media White Paper “Tracking the Influence” (Factiva of Dow Jones) on August 20th, 2007 at 5:57 am

[...] trovo le opere di Jeremiah Owyang sempre stimolanti, ad esempio il suo articolo What should we measure ed il documento Tracking the Influence of Conversations: A Roundtable Discussion on Social Media [...]

Posted by Come misurare i media sociali ed alcune trappole da evitare e un caso, questo SEO Blog | Antezeta Web Marketing on November 3rd, 2008 at 8:29 am

[...] Jeremiah Owyang has a lot of interesting things to say on this topic, as exemplified by has article What should we measure and the document Tracking the Influence of Conversations: A Roundtable Discussion on Social Media [...]

Posted by How to measure social media, avoiding some traps in the process | Antezeta Web Marketing on November 7th, 2008 at 7:50 am

[...] post isn’t intended to serve as a summary of the discussion. Jeremy Pepper, Jeremiah, Christopher Kenton, and Daniela Barbosa, among others, have already captured it [...]

Posted by Engage or Die! ROI vs. ROP in Social Media | PR2.0 on June 15th, 2009 at 11:05 am