Social Media Analytics: Interview with TapHunter from User First on Vimeo.

We recently interviewed Mel Gordon from TapHunter, and she had a few comments on analyzing social media and acting on the intelligence gleaned from online conversations. She also mentioned two KPI’s (key performance indicators) that our team hadn’t even thought of!

Social media monitoring can help you identify influencers and brand evangelists and gain insight into customer feedback, powerful information that can be used to strengthen your brand. However, measuring the impact your  social media campaigns requires work. In our previous blog post “Social Media Analytics Measuring the Intangible” we reviewed a few social media tools and KPI’s (key performance indicator) and hope we inspired you to tackle measuring your social media effectiveness.

Still have questions about social media analytics or what beers are on tap?  Didn’t catch the last two seconds? Below you will find the link to Mel Gordon’s and Taphunter.com twitter page.

@MelGordon

@Taphunter

Well there are different reasons as to why you would use one method over the other. Examples below:

Remote Testing

• Cost and time saving
• Global distribution
• Natural/native environments

Lab Testing

• Highly secure client/server
• Handheld products (requires physical interaction)
• Observation of the user
• Long test sessions

In this post I wanted to discuss how both research methods were used in conjunction, and how they both supported the overall goal of leveraging valuable information about search engine users. Google recently released a usability study that demonstrated how a lab study and an online study can complement one another. The usability lab test provided the hypothesis and the online study quantitatively supported the hypothesis using a more generalizable data set.

We are a huge believers in integrated methods, such as utilizing analytics with remote testing. One method isn’t necessarily better than the other. Designing a research project has to be thought out strategically. Below we will analyze Google’s Study “How Does Search Behavior Change as Search becomes More Difficult?” and showcase how both methods supported the overall results.

Overall Summary & Results

Search engines make it easy to check facts online, but finding specific kinds of information sometimes proves to be difficult. Google studied the behavioral signals that suggest that a user is having trouble in a search task, first by running a lab study to gain a preliminary understanding on how users’ behavior changes when they struggle finding the information they’re looking for.

Experiment #1: Usability lab study
23 participants
Two or three difficult search tasks (closed informational tasks)

Task: “I was recently watching footage of one of Prada’s fashion shows from a few months ago, where two models fell (and several others stumbled) due to the footwear. Find the names of the models who fell.”

Example of how users changed their strategy from keywords to direct questions. User A:

1. Prada fashion show models falling
2. Prada fashion show models name falling
3. Prada fashion model name
4. Prada fashion model name fall
5. Prada fashion model name falling
6. Prada fashion models names that fell
7. Prada models fall
8. what were the names of the models that fell at Prada

The analysis of the laboratory study generated qualitative findings that were used as hypotheses to be tested with the larger-scale online study. The data suggested that as users’ keyword queries failed, they also tended to:

• enter direct questions as queries (question queries may start with who, what, when, how, where, why or end with a question mark)
• enter additional keywords, increasing the total length of their search string
• enter advanced operators (such as “+”)

Experiment #2: Online study
179 participants
22 avg search tasks (closed informational tasks)

To test if these hypotheses hold within the general population for a diverse set of tasks, Google analyzed the number of question queries for the successful and failed search tasks within a larger data set online.

Harder tasks (those in which fewer participants reported success) tended to have longer queries. In the tasks where more than 80% of the participants are successful, the queries tend to be between 2-5 terms long.

Google found five behaviors that were prevalent among users who were struggling in their search task. As we can see, the data suggests that all three behaviors hypothesized as tending to occur were accurate.

Those signals were:

• use of question queries
• use of advanced operators
• spending more time on the search results page
• formulating the longest query in the middle of the session
• spending a larger proportion of the time on the search results page

Based on these experiments, Google was able to conclude that the signals related to successful and less successful search strategies could be used to build a model that would predict the user satisfaction in a search session. These insights could be used to gain a better understanding of how often users leave search engines unhappy - or how often they are frustrated and in need of help, and perhaps an intervention, at some point during the search session.

Final Thoughts

This study exemplifies how a qualitative lab test can be used to uncover possible causes for certain behaviors and how a quantitative method such as remote testing can be used to test the accuracy of such cause and effect relationships. Google might just know the next time your frustrated based on these signals but the real value creating a better user experience.

What are you thoughts? Do you like one method over the other, or has integrating both methods been more of a pain than joy of awesome data. I would love to know!

How do you measure love when the overall goal of any company is to generate more ROI? Internet market research firms like eMarketer and Marketing Sherpa published what the primary objective of Social Media Marketing was to B2C and B2B companies in 2010. Interestingly enough, “Brand Awareness” was on top of the list in both reports. (Full list below)

1) 28% Brand Awaareness   (improve brand reputation)
2) 25% Customer Growth/Loyalty
3) 19% Customer Acquisition
4) 27% Other

That’s right; customer acquisition was on the bottom and online conversations about the brand was on top of the list. Now you may ask: how do you measure “Brand Awareness” on a day to day basis without going insane? Well, unfortunately there is no great answer because web analytics tools don’t quite cut it and social media monitoring tools are not exactly fully developed to handle the numbers and love.

Below is a list that I have compiled on tools and possible metrics. I have to state that the list is not the end all be all and I welcome any reader to let me know what clever metric they have up their sleeve. I will, however, state that I agree with Avinash Kaushik’s 10/90 rule. Ok so here we go!

Web Analytics Tools

I want to assume that you have a tool implemented on your website already, considering there are hundreds out there.

If you don’t, I would recommend reviewing the tools below:

• Google Analytics
• Piwik
• Coremetrices

Now, what metrics can you pull? They may be the everyday stuff but they are just as important to compiling an awesome dashboard:

1) Unique visitors
2) Page views per visitor
3) Time spent on site
4) Total time spent per user
5) Frequency of visits
6) Depth of visit
7) Conversions

Social Media Monitoring Tools

These tools help you track quite a lot of signals and get insights into your brand’s performance on various social media channels. They enable you to act on intelligence gleaned from those online conversations.

Below are a few tools:

• Sentiment Metrics
• Klout
• Viral Heat
• Topsy

You don’t have to follow all of the metrics. Just consider them if they are important to your overall goal.

1) Content Consumption
2) Content Contribution
3) Social Bookmarking
4) Subscribing to a RSS feed
5) Who is talking about you (Identify advocates, threats)
6) Profile Engagement

We recently spoke with Mel Gordon from TapHunter, and she had a few of her own metrics that our team never even thought of! Look out for that interview in the next couple of weeks.

I would love to hear about your experiences. Is pulling the data still a manual process or have you become a social media analytics ninja?

The study was conducted by Mari Carmen Marcos and Cristina González Caro at the Pompeu Fabra University in Barcelona (Spanish PDF  here) and summarized in English by Ani López.  Our notes are based on this post by Ani López.

The Study

The general hypothesis tested was whether or not a user’s intention behind a given search affects that user’s behavior on the results page received from that search.

Participants were asked to perform the following types of searches:

• Informational (find specific information)
• Navigational (locate a particular website)
• Transactional (execute a transaction such as make a purchase, download software, or book a reservation)
• Multimedia (find a photo or video)

For each type of search, the researchers measured both the number of fixations and the time spent fixating on each of the following areas of interest within the search engine’s results page:

• Title
• Snippet
• URL
• Image

Our Take

We found multimedia search intents a bit incongruent with the other types.  For this reason, we will disregard differences in that area.  As the authors note in their conclusion, a multimedia search includes inherent differences in both how results are presented and the type of data (image vs. text) the user expects to receive, both of which affect the measurable results.

Looking only at the other three types of searches (informational, navigational, and transactional), we found the metrics showing only marginal differences in attention given among the title, snippet, and URL.  However, the number of fixations and time spent fixating on the organic vs. sponsored search results varied with significance, showing the user giving most attention to sponsored search results during a transactional search, followed by navigational search, and then informational searches had the lowest amount of attention given to sponsored search results.

We’ll leave any specific SEO/SEM conclusions to the reader, but this study is compelling because it demonstrates that user behavior, specifically visual attention, may vary when reviewing SERPs based on the search goal of the user.  And eye tracking is a valuable tool in measuring these differences, an example of eye tracking’s unique value in measuring the near passive behavior of visual data gathering.

Eye tracking will not tell us why a user behaved the way they did, but eye tracking provides a clearer picture of how they behaved including non-interactive behavior not traceable by analytics.  And this gets us closer to that goal of knowing why.

What would you like to learn from eye tracking?  Let us know and we just may study it!

Count

The count is defined as the sum of the number of fixations from all the selected recordings for each section of the image. This plot shows the accumulated number of fixations from all the selected test persons. This means that this plot only shows where people have looked and not for how long. That means that a 2000 ms fixation will look the same as a 100 ms fixation.

Absolute Duration

This plot shows how long the selected test persons have looked at the different areas in the image. This means a 2000 ms fixation will be 20 times higher (in color value) than a 100 ms fixation. It also means that a person that has spent a long time looking at a stimulus will have a greater impact on the heat map than someone who only glanced at it shortly.

Relative Duration

The sum of the individual fixation lengths relative to the total fixation time on the image for each recording, if in recording A, the subject looked at the image for 10 seconds and at a specific object for 3 seconds and in recording B, the subject looked at the image for 10 minutes and at the specific object for 3 minutes. In this kind of Heat Map the data from recording A and recording B will have the same weight. Naturally, what is considered a fixation is dependent on the settings chosen in Fixation Filters.

All three fixation styles are valuable and are dependent on the design of the research study,  but if you have any questions on which one you should just let me know!

UPDATE: Jon Ward from Acuity added a very useful comment on when you should use the different styles of heatmaps. Check the comment below!

Here are three questions in the order I feel you would ask me:

1). Who is Bazaarvoice ?

Bazaarvoice help e-commerce sites become social by providing a platform for visitors to leave comments and reviews about products. Shoppers can find their own Facebook friends while on the brand’s website and discover their top-rated products. In my opinion user-generated ratings and reviews have gone from a bonus feature to a must-have for e-commerce sites.

Did you know that about 52 % of Americans purchase online as of May 2010, it’s true you could read more at The Pew Internet Project.

2). Why should I care about the integration?

If you’re using Bazaarvoice with this Google Analytics integration you can now see the metrics on things like:

• the number of visitors paginating through reviews
• sharing user-generated content with social networks
• clicking on related products found in reviews

Advanced Segments can then be used to compare the behavior of visits that interact with Bazaarvoice generated content vs. those that don’t. I love segmentation, it makes it easy to see the influence of user-generated content on your conversion goals. The team over at Bazaarvoice did a study on measuring their software and on average a customers who interacted with Bazaarvoice solutions had more than a 50% higher conversion rate  and over a 70% increase in revenue per visit.

3). Is the integration hard?

Integration managed by Bazaarvoice is at no cost and requires less than five hours of client management time. I think that’s pretty cool of the team!

I hope that covers it for you, but if you had any questions or comments please let me know. Now go enjoy the rest of the weekend!

1. Relevant- is your analytics solution collecting information you need.

2. Reliable- is it providing accurate data.

3. Usable- is it presenting data in actionable summaries.

Big questions now is “how do you check a new account and make sure that it is reliable and collecting the appropriate data”. Well it’s not super simple but it’s not super hard either, below are the five ways in which I quickly audit a Google Analytics account.

1). KPI Awareness

I know I know at times this could be like pulling teeth because the answer always is “to generate more leads and increase revenue.”  Don’t roll your eyes, instead ask your client what is important for a user to do on the website, and also how are they supporting their business goals. If you still don’t get a good answer then usually what I do is go through the website and experience it like a user. I buy and return something, fill out a form (soft joins too), download software, double very my email, whatever it takes to understand what the website and how a user experiences it.This gives me a high-level perspective of what data I would want to see if I was the analyst, such as a commerce report that displayed the following:

• Purchase order (SKU number, sales amount, color, etc)
• Returns
• Cart Abandonment

Doing this exercise helps me understand what data I need to see within Google Analytics to create valuable KPI’s. The next step is checking the code, and making sure all the variables have there I’s dotted and T’s crossed.

(Tip: I usually open a excel file and snag-it every page that I go through to make comment, and collect the URLs of the pages.)

2). JavaScript Scan

If you simply just want to find out if the website has properly place the Google Analytics tracking code on the website just go to SiteScan put your URL in and wait for the report to be delivered. (It once found that a client had both the legacy Urchin code and the new Google Analytics on their website!) For more custom data, let’s take the ecommerce example above checking to see if this code fires up correctly gets a bit more extricate. For this I usually view and copy the source data when I am going the website and if the numbers don’t match up I take it to my awesome developer to find out why.

(TIP: You can’t do this for every single thing just the important information you need for those KPI’s.)

3). Profile Configuration

When you log into a account you will see that you can edit a profile, open i

t up and see what you’re working with. Check to see if:

• Site search is turned on (if applicable)
• Ecommerce is turned on (if applicable)
• Goals/Funnels have been implemented
• Filters includes/excludes have been properly implemented

While you’re under the knife check if there are any other profiles set-up within the account. Again this would only be necessary if it was a KPI concern. For instance traffic patterns from Google organic search is extremely important to create revenue for your client. A separate profile would be something your client should have then because sometime segments just don’t cut it.

(Tip: In order to check profile set-ups you need admin access.)

4). Reporting Overview

This is the fun part; you get go inside the reporting suite and check out what the situation is (no pun intended). I review the following items:

• Dashboard (is it custom or the generic profile setting)
• Goals & Funnels
• Segments
• Intelligence Reports
• Google Adwords integration/PPC tracking and reporting
• Site Search
  

  

5). Customization

I like seeing if any customizations have been activated, most people don’t know of the cool stuff that Google Analytics can do. Some of the top set-ups I look for are:

• Advanced Segments
• Custom Reports
• Events
• Pivot tables
• Website Optimizer integration
• Kampyle integration
• Mailchimp integration
• Saleforce integration

I also check and see how clients are receiving their data. If they are collecting data one report at time I highly recommend that they invest $30 and get ShufflePoint (or any API tool).

Well those are my top five ways to do a Google Analytics audit. Of course they are not the end all be all and I would love to hear your experiences or what tools are you using.

Eyetracking can be defined as a technique that is used to record and measure eye movements. Definition is simple enough, but I always get a follow-up of “how does it record” and “will it hurt”? First let me say no it will not hurt, second you will not go blind, and third you will not become a mutant-sorry. At User First we use the Tobii T60 model, so I will discuss how this equipment works specifically.

Fantastic Machinery, The Eye

Imagine if you will that you are looking out a window from your home or office onto a city street.  As you look outside, your eyes are constantly moving.  Some of these movements are conscious.  For example, you notice the movement of a dog and glance above it for a glimpse of its owner.  But more so your eyes are moving involuntarily, focusing only on certain areas of the visual field in order to form a picture of the scene for your brain. The human eye is a fantastic piece of machinery; it is not capable of absorbing 100% of the visual field in an instant with clarity. We call the area of the eye capable of this focus the foveal area and the brief pauses of our gaze the fixations. The foveal area accounts for only 8% of the visual field at any one time but supplies 50% of the visual data received by our brain.  And the movement of the eye controls which regions of the visual field we fixate on and which regions are ignored and left to the poor acuity of our peripheral vision which is only useful for picking up movement and strong contrast.

So how are Eye Movements Tracked?

Just as the human eye relies on the focus and detection of light to see, so does the most common technique used to track eye movements called Pupil Centre Corneal Reflection (PCCR).  This technique is non-intrusive and the technology making it possible comes in two forms: either a specially equipped computer monitor or a head-mounted device.  Both options use a light source to illuminate the eye causing highly visible reflections.  The illumination is near infrared and therefore unnoticeable to the user but creates reflection patterns on the cornea and pupil of the eye and two image sensors on either the computer monitor or the head-mounted device are used to capture images of the eyes and the reflection patterns.  A computer then uses advanced image processing algorithms and a physiological 3D model of the eye to estimate the position of the eye in space and the point of gaze with high accuracy.

The location of these gaze points during each fixation, the time spent on each fixation, and the pattern in movement from one gaze point to another are the key pieces of data collected during an eye tracking study.  These data can then be visualized using a gaze plot or a heatmap.

CAPTION: The Gaze Plot visualization shows the movement sequence and position of fixations (dots) and saccades (lines) on the observed image or visual scene.

CAPTION: The Heatmap visualization highlights the areas of the image where the participants fixated. Warm colors indicate areas where the participants either fixated for a long time or at many occasions.

Not only can we determine what and how visual information is consumed but patterns in eye movement tell us more.  Emotional responses are evident in eye movement patterns and thus allow us to connect physical behavior of the eye to cognitive behavior in the brain.  This is why eyetracking is a strong supplement to traditional qualitative studies.  They allow a scientific measure beyond the subjective responses provided by a participant in an interview.

Eyetracking is especially important in the age of mass media.  The amount of content, the speed at which it is delivered, and the speed at which a user consumes it, means users make less and less time fixated on each image.

How will your message not get lost?  How will your brand be recognized? Question please don’t hesitate to ask.

Here is a brief overview of three approaches to analyzing mobile eye tracking output.

1. Watch the video output and manually tally the ‘hits’. Slowly playback the video and count how often the point of gaze lands on the area of interest. This gives you a general sense of what is looked at and what is ignored, but only limited conclusions can be drawn with respect to the amount of time spent viewing each AOI. One could slow the video down further (frame by frame) and take note of the time stamp for each time the participant starts looking at one particular region. This is enormously tedious, especially for video segments longer than 2 minutes and with more than just a handful of AOI. It is also imprecise; the time stamps are obtained at 30 fps, with up to 5 or more data points lost with each frame, before and after the time stamp. There is also substantial opportunity for human error in tallying hits or recording time (I speak from experience).

2.Identify fixations, then tag AOI based on fixation. In this approach the analysis software identifies fixations following a particular algorithm developed by the manufacturer. Then, thumbnails are generated for the video segment during which the fixation occurred. The research analyst tags each thumbnail with an identification of which AOI is being looked at during that fixation. The output sums the tags and can compute dwell time based on the length of the fixation. This is all very promising and most certainly not as time consuming as option 1. Nonetheless, each data file needs to be addressed individually and to a certain degree, manually. A greater concern, in my opinion, is the reliance on the identification of fixations to then in turn make conclusions about viewing attention and dwell time. Fixations, by definition, assume a moment of movement cessation of the pupil as the eye fixates on an object. What if the object is moving and the pupil is following in smooth pursuit? How is the fixation captured? Experience with this approach has left me wondering why the dwell times on AOIs total up to only a fraction of the total testing session. What did the participant look at the rest of the time? Did he keep moving his eyes so quickly that he never truly looked at anything? Hard to imagine.

3. Draw regions ‘on’ the output videos and process the data against the defined AOI. This is essentially the same idea as most analysis approaches with desktop systems. Identify the AOI in the scene and the software will tally when the x,y coordinates of the point of gaze fall within that region. There are two approaches:

a. Draw each AOI ‘by hand’ for each frame. This is can be reasonable for a small number of AOI and a fairly stable scene. The markers for the AOI can be dragged and re-sized in order to consistently overlay the actual regions in the scene as the video plays. It is potentially more time consuming than option 1, but is about as precise as mobile eye data analysis can get.

b. Use a form of image recognition software and have the software identify AOI. This approach works well if there is some contrast within the scene and you have primarily only one scene to deal. Here you create a video of the scene and identify the AOI for the software. It, in turn, uses this as a key to automatically identify the AOI in the recorded data files. One the AOI are identified, the process is the same as in (a) above. This potentially is a very quick process and results in data that is potentially very precise. It is almost too good to be true.

So that is where we stand at User First with our understanding of mobile tracking. We welcome corrections, other perspectives, elaboration, and the sharing of experiences as it pertains to this topic as we are ourselves learning each day about what is available, what is in the works, and how we might meet client needs in the future.

As we move forward with mobile tracking and expand on our experiences with different analysis methods in particular, we will keep you posted

]]> http://www.userfirst.com/our-blog/2010/07/06/mobile-eye-tracking-part-3/feed/ http://www.userfirst.com/our-blog/2010/06/28/unmoderated-usability-study/ http://www.userfirst.com/our-blog/2010/06/28/unmoderated-usability-study/#comments Mon, 28 Jun 2010 18:27:38 +0000 admin http://www.userfirst.com/our-blog/?p=280 If you have ever ran an un-moderated usability study you know that most solutions don’t provide advanced features for research panels or the functionality for custom analysis. We personally use Loop 11 for un-moderated usability studies. Although we love the tool and the great insights we get we always sigh when it’s time to crunch the numbers. But not anymore!! Loop 11 just released new features that will make setting up and analyzing un-moderated usability studies a breeze. Within this post we will review the changes and why they are important.

1). Tracking participants using unique IDs
When using a research panel for a usability test tracking individual participants is important not only for segmentation but also to know exactly what participant completed the study and should be paid their incentive, terminated, or was over-quota. Previously, I had to ask the participant to fill in their Ids and as you can imagine some participants didn’t answer correctly. I would also download all of the IP address Loop11 gave me and try to match it back just to double check, which took hours!

Why this rocks: Now all you have to do is customize the end of the URL with a unique ID for each participant. You can now easily pay your participants and segment your users without having to invest a lot of time.

2).Re-Categorise multiple URLs at a time
Instead of re-categorising one URL at a time you can now re-categorise multiple URLs at the same time.

Why this rocks: Some websites have dynamic URLs, and you can’t set-up a goal for every possible combination within Loop 11. Now instead of re-categorising one URL at a time you can select what pages you want to re-categorise. This saves time during the set-up process of the study and analysis because once you re-categorised the URLs the data re-configures!

3). Pop-up invitation controls
If you are recruiting your participants by intercepting them via your website; you now have a feature allowing you to control the percentage of visitors that you ‘invite’.

Why this rocks: Selecting what percentage of your visitors is super valuable because too high a sample rate might mean you are surveying more visitors than you really need to in order to get valid results. Remember, too small a sample could produce results that lack statistical validity.

4). Individual participant path analysis in exports
Full path analysis of individual participants can now be exported into reports. This allows you to segment the conversion funnel by your top segments, whether you segment participants by female only or participants that clicked abandoned when they actually succeed the task.

Why this rocks: In the past, the way we collected this data was by using the interface within Loop 11 and simply copying and pasting each link for each participant for each task. Needless to say that it took awhile and there was lots of room for human error. Now all you have to do is download the report. Once you have this data you can segment your funnel analysis to view how different users completed or failed the task based either on their demographics and/or geographic.

5). Individual responses for Rating Scale Matrix questions
The results for rating scale matrix questions, while always available at an aggregated level were never available so you could see the individual participant responses.

Why this rocks: Let’s say you asked a gender question because you wanted to know how many females or males participated in your study. Loop would just tell you the percentage but not who was female or male. Now you can download the report and know who was who. This type of information is important if you had follow-up question after the task, you can find what type of user rated the task easy or difficult. Or if you didn’t use a research panel you can start your test with some demographic questions so your analysis can get a little juicer with segmentation.

What is your experience with Loop 11, are you excited about these changes? As Toby Biddle would say “Happy Testing”!

]]> http://www.userfirst.com/our-blog/2010/06/28/unmoderated-usability-study/feed/