Art, Technology & People

Comscore and Quantcast – How they work and why they are the gold standard of guess-work.

November 12, 2012 Dan Richard Comments 22 comments

Greetings sports fans, welcome to another post that I hope will either spark some discussion, open some eyes or perhaps help me understand this situation in a different light, granting it some legitimacy a bit heavier than the feather-weight I feel it has. It is often said that comScore is the gold standard of measured site metrics used by advertising agencies when they make their digital media buy decisions, shortly followed by Quantcast. These companies have the power to influence multi-million dollar decisions that can potentially be made based on their demographic reports.

I deal with comScore and recently Quantcast on a regular basis, and quite honestly I’ve come to the conclusion that they are really nothing more than peddlers of educated guesses. I also feel that the possible error-range in their evaluations has the potential to me massive, hence unfairly denying someone possible revenue dollars due to their metrics net being too small. I often scratch my head wondering, how in god’s green earth did these guys get so influential in the world of advertising? Really good PR I guess. I also often wonder just how far is the line pushed when polling the demographic data they gather in the efforts for online accuracy supremacy.

I would say one of the biggest errors the digital ad industry makes is confusing or blurring the lines between actual site metrics, vs site demographics beyond what is provided in the browsers header information. Let me be very clear here, the ONLY way a site can truly know the age, race, income or favorite beer of any given user is if that user enters it in a form and clicks enter. Anything beyond that, is quite honestly guesswork.

Site Metrics

Real site metrics easily defined is the collection and reporting of REAL visitor data to your website by a piece of analytical software, the most popular in the world being Google Analytics. GA uses the REAL data from your REAL visitors, records and collects that anonymous data and provides you easy-to-read formats of that traffic information in a vast palette of reporting options.

The site demographic metrics reports you get from comScore are really just educated guesses based on a pool of poll users that are tracked via plugins, web beacons and/or tracking cookies. They then average out that information with the number of uniques and page views you have and their advanced algorithm provides you with Gold Seal demographics… or what I call educated guesswork.

This is also why you can’t marry the data comScore spits out vs actual Google Analytics information. So if your comScore reporting is telling you the median age of your site is 28 and 35% of your audience is gamers, you can’t ACTUALLY track who those people are and what parts of your site they are visiting. You can’t track them because they aren’t real visitors, it’s just a guess.

What is comScore?

comScore is an Internet marketing research company providing marketing data and services to many of the Internet’s largest businesses. comScore tracks all internet data on its surveyed computers in order to study online behavior. You can visit their site at http://www.comscore.com.

How does comScore work?

One of the most common questions asked is how does comScore work? Where do they get the demographics information that almost seems to come out of thin air? Well, they in fact have a couple of primary methods.

comScore is a paid measuring tool, meaning you must subscribe to their services in order to track and report your comScore metrics reports. The website to be tracked must have the comScore tag propagated throughout the entire website in the same way you would a GA code and this will allow comScore to accurately measure your traffic, pageviews and other information that you a standard analytics program would collect. So typically pageviews and unique visitors are quite accurate and should line up with GA numbers, although UVs will likely be a bit lower due to the way comScore measures uniques vs Google Analytics. GA uses actual uniques, comScore magically tracks you and guesses when you are using your work PC, phone and Home PC and then numbers you as a single unique.

When I asked a comScore rep how exactly age is calculated by comScore for the demographics reporting, they confirmed the above:

“Demographic information is gathered from our panel. When someone opts into the comScore panel, they are required to fill out a short questionnaire where we gather demographic information for themselves as well as other people in the HHLD who will be using the metered computer. We then use census populations estimates to project out to the total internet population.”

So how does this part of the magic happen? How does it know what color your poop is, how old your car is and how much you make a year? A couple of ways.

The Research Panel

comScore maintains a group of around 2 million monitored research panelists that run a background monitoring software package that tracks everything they do online. comScore partners up with several technology brands to create and maintain this software, such as Permission Research, Opinion Square and VoiceFive Networks. The users in this group are given benefits for being a panelist, including free software, online storage data, and chances to win cash and prizes from comScore. comScore then uses a series of weights to adjust the statistics so that they have a better reach that reflects US and global browsing than just the two million panel members bring to the table. This is where the guess-work comes in, as well as comScore panelists…

comScore Panelists

comScore regularly recruits panelists through random digital dialing, as well as additional online and offline methods such as polls, quick questionnaires and much more. comScore then uses that data to determine total people online, geographic location, income, age, and other factors and then apply that information to “massage” their research panel stats and generate an accurate global or US demographic.

Additional information about comScore

I sent comScore a request with some of my questions and got back some great information directly from comScore:

comScore measures people and not cookies or IP addresses as your internal data does. The way we do this is as follows:

When a company wants to become unified they sign up and place a beacon (or a tag) on every page of their website. Whenever anyone goes to one of those pages (regardless of whether they’re in our comScore panel or not) our beacon call will place a comScore cookie on the machine. We count these cookies and then we use the people in our panel to understand the following:

Users deleting or blocking cookies

Users using multiple browsers per machine

Users using multiple machines across home and work

Users using multiple machines at home

Multiple users on the same machine

Computer Overlap: An adjustment is applied to the cookie counts to account for usage across multiple machines within the same household.

Based on our findings from the above criteria, we then assign an average Cookie Per Person ratio for that site. This CPP is updated every single month for every single entity we report on.

We then take the following steps to calculate the Unique Visitors:

We sum up the number of cookies for the particular country we’re measuring and just for Home and Work (so we filter out International traffic, shared computers and mobile devices)

We use the Cookie Per Person ratio based on the criteria above to calculate Census Only Unique Visitors from cookies

We use the panel to understand the number of UV’s NOT seen from cookies.  Since the site may not have tagged every single page, some UV’s will go unnoticed by cookies.

We sum the Panel only Unique Visitors and Census Only Unique Visitors to reach Unified Home and Work Unique Visitors

We use the panel to understand the overlap between people using BOTH home and work computers

We report the final UNIFIED UNIQUE VISITORS

Page View numbers are calculated by the following:

This measure comes 100% from the tags on your site. We take the raw tag number and filter it from:

International traffic (if you are only purchasing US data)

Shared environment/mobile traffic

Auto refreshes and don’t comply

Forced viewing (pop-ups….)

Nedoms (non-essential domains) comScore maintains a list of pages.

Non human traffic from bots and spiders are also removed

Why the comScore numbers don’t match my Web Analytics data:

comScore measures unique persons

Web Analytics “unique” numbers are a measure of unique cookies

Differences in the numbers come from:

Users deleting or blocking cookies

Users using multiple browsers per machine

Users using multiple machines for home/work

Users using multiple machines at home

Multiple users on the same machine

Web Analytics data totals include approximations of visitors using a  combination of IP addresses and user agent when a cookie cannot be dropped.

comScore filters out the following:

International data (for US subscribers)

Non-human traffic (bots and spiders)

Nedoms (non-user initiated traffic) pop-ups, & partial page loads

Shared usage environments (internet cafes, libraries, airport kiosks…..)

What is Quantcast?

Quantcast is a media measurement, web analytics service that allows users to view audience statistics for millions of websites. Quantcast Corporation’s prime focus is to analyze the Internet’s web sites in order to obtain accurate usage statistics by surfers from the USA. It is primarily used by online advertisers looking to target specific demographics such as age, income or other traits. You can visit their site at http://www.quantcast.com.

How does Quantcast Work?

Quantcast has a large network with millions of sites running its data collection feeds, web beacons and anonymous cookies, so it can track a person as he/she visits any of the websites in its network, and can build a profile of that person’s browsing habits, and then extrapolate demographics. Quantcast tends to associate themselves to the way search engines examine how webpages are interlinked, and thereby determine relevancy within it’s network and the demographics they collect. You can read more about what they do from a company perspective at http://www.quantcast.com/how-we-do-it.

Is it Legal?

Both comScore and Quantcast use proprietary algorithms they use to try to make educated guesses about the age of the user based on their internal measures and then displaying that person the ad. This is again not real data, it’s based on educated guesses based on their own measures and not on the real data of the actual visitor, which is what true analytics use. They can only track anonymous browsing habits… AGAIN, the only real way for a site to know 100% the age of the user is if the user enters their age on the site and submits that information… anything above and beyond that is quite honestly educated guesswork on an internal formula of indicators and measures gathered by web beacons and cookies that track behavioral browsing patterns.

This is actually why Quantcast and other demographic tracking companies have faced numerous lawsuits and privacy violations… they all have to walk a very fine line of how intrusive they can be in their data gathering methods until they start to break the law. It is illegal to gather personal individual data without authorization via cookies, beacons and other tracking means unless the user agrees to it, which is why contest pages etc all have terms and conditions the user has to agree to (one of the reasons anyhow). All general traffic behavior research gathering has to be anonymous and is generally mentioned in website privacy policies, which you can see on just about every company website you can think of. This is why the demographics information from comScore can’t actually be applied to the real analytical data.

Both companies have an extensive history of lawsuits, accusations and general privacy violations as they try to push the boundaries of what they collect and the collection processes. A quick Google Search shows dozens and dozens of privacy violation lawsuits against comScore.

There is no question that online privacy is a farce, and hopefully as individuals continue to see how easily their privacy is exploited online, we learn to keep things closer to the vest. Check out this little blog post by Robert Dempsey about comScore, a bit of an eye opener in case your head is in the sand.

Why do ad agencies LOVE Comscore?

Is it a lack of options? Awesome marketing on comScore’s part? Dominance in the industry? Well, it’s a bit of everything to be honest. We know that it’s nothing but educated guesswork, yet multi-million dollar deals are lost and won because of what comScore has to say, more so than REAL numbers such as the metrics provided by Google Analytics for example. comScore has positioned itself to be the leader in demographic research and through aggressive business propagation and smart marketing have made it as common and standard to digital ad planning as TVs are to a consumer’s home. You may not realize it, but you are likely touched by comScore on a daily basis… when you surf, when you read magazines, when your boss is making decisions that affect you and your company and so much more. It’s used daily in media kits, presentations, infographs, marketing plans and many other mediums for everything from entertainment to purchase proposals.

comScore is everywhere and has become masters of educated guesswork, ninjas of not-so-naked truth and warriors of what’s up online. There is no question that if you plan to work with ad agencies on premium ad buys for your website, you have no choice but to tag with comScore if you plan on playing with the big boys. BUT if you’re concerned about your online privacy and how you are tracked, fact-finding on the practises of comScore, Quantcast and other demographic measuring companies is a scary eye-opener.

How do you feel about your online privacy or about comScore and their metrics? Are you an actual comScore research panel member? Please chime in with your comments by using the form at the bottom of the page and please share this article socially with your friends and connections.

Thanks for reading!
Dan

Please help me win the “Ugliest Bathroom in Quebec” contest – I need your votes!

November 9, 2012 Dan Richard Comments 1 comment

Hey all… ok so this isn’t exactly a tutorial, an article, or really anything but a blantant call for roughly 2 minutes of you rinternet viewing time. My hideos, disgusting blue bathroom from hell that I am DYING to renovate has been entered in an “Ugliest Bathroom in Quebec” contest with Bain Depot. The winner of this contest wins a $5000 gift card from the store to buy whatever you need for your bathroom. Look at this 1981 horror show:

Blue walls, blue floor, blue ceiling, blue floral decor, blue toilet, blue sink (that I had to change out because it was completely rusted through), blue counter and even a blue toilet dispenser! It also had a blue towel rack that I ripped off and a DISGUSTING blue toilet seat lid. Not only that, the bath area is surrounded with this shower liner that is loosly glued to the wall and when you pull it away from the window, you get a great look at all the mold and get a facefull of the stench that seems to be a mix of week old dead skunk and something that I am sure is on par with shoving your face up a whale’s arse.

PLEASE VOTE HERE! Just click here, and scroll to the orange vote button right under the photo of my bathroom and you’re done… you don’t have to enter your email or anything. ALSO PLEASE SHARE! Please share this post or the contest link or whatever on Twitter, Facebook, whatever you got! All your friends or family members that spam 5 year old jokes? Send them this! And please vote as often as you can… you can vote once per day per IP so please vote! I have to keep this going until the end of January!

Need more reason to help me wipe the bathroom from the face of the planet? Behold! Check out the drywall rotting away and the ceramic tiles literally falling off the walls… I dare you to put your hands in there. Reminds me of that scene in Indaiana Jones and Temple of Doom where the girl has to shove her hand in the hole full of bugs and nastiness to save Indy and Shortround.

So again, please vote and share this link like your life depended on it:

http://bit.ly/UzzZSo

Thank you for humoring me! Make sure you follow me on twitter for more pics of the famous bathroom and the fun I will have tearing it down.

Dan

Problem with WP-Stats-Dashboard Plugin “wp-content/plugins/wp-stats-dashboard/view/admin/graph.php” solved!

November 5, 2012 Dan Richard Comments 0 Comment

If you’re looking for a pretty slick WordPress stats package, you have to check out WP-Stats-Dashboard and the awesome functionality this brings to your admin dashboard. As I toy around with this package, I am having a few issues I am slowly working on as I continue to re-import my tutorials back in to the site and look for missing images and errors. One of the very first issues I had was when the package was first installed and instead of getting my trending graph reports, I would get one of the following two lines of text instead of a graph:

../wp-content/plugins/wp-stats-dashboard/view/admin/graph_trend.php
../wp-content/plugins/wp-stats-dashboard/view/admin/graph.php

I fiddled around with this for about 30 minutes, deleted and re-installed the plugin and tried a few other things to no avail… I ended up throwing my hands in the air in disgust and moved on to other things instead of wasting more time. Well, turns out that was the right decision as it looks like this is an issue many folks are seeing when they first install the plugin. How do you fix it? Wait! Yep, just wait for the graph to cache and it will start working all on its own… I came back to the WordPress dashboard a couple of hours later and VOILA, it started working.

So with that solved, I only have a couple of other issues:

The WP-Stats-Dashboard iOS app looks awesome, but apparently Apple Store Canada does not carry the app… what the hell is that all about!? I know there is a long tedious way to bypass this issue, but I’m too lazy to mess with it. I’ve contacted Dave, the owner of WPSD so hopefully he can look in to this and start going international!
When I go to the main WPSD stats dashboard, there is a box with all your social media metrics labeled “Stats – Social Media Metrics”. When I click the reload button, it generates the following error: “Warning: number_format() expects parameter 1 to be double, string given in ../wp-content/plugins/wp-stats-dashboard/classes/util/metrics/WPSDTechnoratiRank.php on line 59″. Now I have my WPSD set not to track Technorati so I have no idea why it is calling this page and why it’s generating an error. I fiddled around with this as well and couldn’t find a solution.
I’m also having a bit of an issue getting my social stats to track properly beyond Twitter and Facebook (Stumble, LinkedIn and G+ refuse to track) but I haven’t had a chance to mess with any of this yet. I’m wondering if you need a paid account with LinkedIn for it to work.

Anyhow, hope this helps relax your nerves if you’re dealing with the graph.php issue and if you know the reason why my other 3 issues are happening, please feel free to chime in.

Thanks!
Dan

How to search for specific text within your website’s PHP files using SSH

October 29, 2012 Dan Richard Comments 4 comments

Here’s a quick little tip for anyone that needs to plow through a zillion PHP files to find a specific line of code or a specific keyword. I recently needed to scan the entire P2L folder for a specific word within any PHP files in the main public_html and subfolders. The alternative would be to download everything to your local PC and then run a search locally on your PC. That’s fine and dandy if you have a small site that would only take seconds to transfer, but it’s a different story when you have dozens of folders and subfolders and hundreds of pages.

So, let’s put the magic of Linux to work shall we? Open up Putty or whatever SSH program you use and log in and make your way to the public_html folder of the account you want to search.

Here’s the command you want to run:

grep -r -i -l "keyword" *

Hit enter and the quary will run… be patient as it will take some time to complete. It will list any files found that contain your keyword as the query runs. Here’s a breakdown of the command flags used:

-r recursive (will search folder and subfolders)

-i case insensitive (ignores UPPERCASE or lowercase)

-l list only (displays names of files found only)

Hope that helps!
Dan

The SEO Marketing Traps – There is no secret SEO recipe you don’t know about!

October 21, 2012 Dan Richard Comments 3 comments

After a recent experience of mine with some SEO related service questions, I wanted to put together a few thoughts on SEO services, tactics and maybe put some of your minds at ease… both for people who provide SEO, people who do SEO package reviews like me, and users of SEO. If you’re trying to find the secret ingredient to SEO success, you might like to read this. Please note this is an updated posting, the tutorial has been moved from the old host.

You’ve probably seen the emails… a company in India is a leading SEO specialist, and for a monthly fee ranging from $100 a month to over $1000 will perform some SEO magic on your website, making you more visible, higher ranked on keywords, increased PR and even possibly a better lover! You’ve been busting your head trying to figure out what these guys are doing that you’re not…? your site has been stagnant and after a couple of months of work, you’re not really getting anywhere. So what the HECK are these guys doing that’s you’re not? What secret recipe are they using that will yield the amazing results you’ve been after but have failed to produce? What do they know that I don’t?!

The answer is likely… nothing. And here lies the trap, and likely leads to the most common SEO mistakes most of us have fallen prey to at some point.

SEO is real, powerful and necessary for the success of your website even at a basic level. Internet marketers and specialists focus (or least should focus) very strongly on SEO practises, and SEO companies like YEAH! Local are trying very hard to let all webmasters know that SEO is important and that you need to use them and their elite SEO knowledge to get results and be the top of the world quickly and effectively. The truth of the matter is many of us think that SEO is a bag of tricks that changes month to month and that only a few folks that have dedicated hundreds of hours to the study of SEO truly know what they’re doing.

Good news… that’s completely untrue, and most SEO tactics that result in overnight increases in traffic often do more harm than good as they are probably against search engine guidelines and their TOS. You have to change your mindset completely… there is no secret recipe or overnight tricks. You need patience and time, plain and simple.

01. There is no SEO smoking gun!

The first mistake to avoid is looking for the secret that you believe SEO experts know and you don’t. Every single SEO concept and tactic you need has been repeated over and over in thousands of free tutorials and articles all over the web. Deep down you probably even know that, but you’re on a mission for more organic search engine traffic so you throw caution to the wind and decide to pay an expensive SEO service that will employ exactly the same tactics you already know about, they will simply dedicate more time to it than you probably realized was required, and even the best of SEO companies can’t deliver overnight results unless they are performing shady blackhat tactics. SEO is a lot like investing money… you need to think long-term, as in 6 months down the road, not 6 days or 6 weeks. Forget about spam campaigns or link directories with a million other SEO “gurus” have posted links. You need to focus on quality backlinks and real keyword research, and of course the king of web marketing: CONTENT!

02. Search Engine Guidelines

Another mistake most webmasters make is to completely disregard user guidelines of the major search engines. I think it’s safe to say that most of us know that Google has a TOS with very specific guidelines of what you can and can’t do to optimize your site for it’s index, but how many of us have actually read it? Oh, you have? Good stuff, now how many of you check it for changes on a regular basis? That’s what I thought… and yes it changes quite often. Following these terms is critical if you care about search engine traffic because you can score some very effective tips on optimizing your site, plus find out what could end up getting your site penalized.

03. Internal Linking Strategy

The final mistake I see quite a bit, that even I am working on correcting, is internal linking. You can bust your butt on backlinks, but if you ignore the topology of your internal link, you are missing a VERY important ingredient to your SEO success and your ability to earn what is called “link juice”. First time you hear this term? Link juice is the currency used by Google to determine why your site should outrank others. For a great explanation on link juice and how it works, check out this article.

The fact is, it’s extremely difficult, if not damn near impossible to control how other sites link to you, but you can do whatever the heck you want with your own site. Internal linking is an art-form on it’s own and can yield significant long-term gains when properly applied. By creating effective internal linking, you’re more or less promoting other areas of your site that are less popular, but still possibly relevant to your visitors. This increased visibility on an internal level can introduce your users to features, areas and content they may have never realized you had, and these users are all potential backlinks, social media buffs or word-of-mouth links to untapped traffic. Just check out large sites like Microsoft, Amazon, eBay, Digg and many more… you’re presented with the content you are looking for, as well as internal links to other relevant areas in an effort to engage you and explore further. This is effective internal linking.

Again, you’ve read SEO articles before and most say the same thing… you need to focus on the following:

Onsite SEO

Establish and optimize your website pages for specific keywords. This includes metatags, URLs and content.
Investigate and research your competitors… see where they rank for keywords and how they are optimizing their pages. Are they ranked higher than you? You can find out why by checking meta tags, backlinks and their content pages.
Optimize your title, meta and header tags.
Optimize your anchor text, your robots.txt and your image Alt tags.
Write relevant content as often as possible!
Fix broken links on your site.
Create an XML sitemap and submit it.
Ensure your site is included in major search engines and see how you rank for your top 10 – 20 keywords. Keep track on a monthly basis to check your progress.
Install and use Google Analytics.
Validate your code for easy crawler inclusion.

Offsite SEO

Verify search engine guidelines and make sure you are compliant
Research and build up backlinks to your site… there are dozens of strategies.
Create a separate blog and keep it updated with content regarding your site, updates, research and other projects. Use it to generate interest!
Create articles for open distribution
Take advantage of social media and social bookmarking sites
RSS Feed submission
Forum posting
Search for link directories and link exchanges relevant to your site’s content and join them! Don’t just submit to everything under the sun, most link directories are completely useless.

I’m sure I’m missing a couple of items on here, but this is everything you need to know, and this is everything an SEO company will do for you no matter how much you spend per month. There are no magic bullets, smoking gun or any other catch-phrase worthy techniques you don’t know about, it’s all out in the open. The key is to dedicate time to your SEO strategies and set long goals for results, but short-term goals to get there. For example, rather than say “OK this month I want to increase my traffic an extra 5000 visitors a day” look for a more tangible short term goal such as “This month I will create 50 new one way links to my site”. With small monthly goals, the long-term traffic goals will follow and will likely exceed your initial estimates.

Of course, if you have absolutely no time to optimize your site for search engine traffic, that’s an whole different issue and at that point you may wish to employ the services of a reputable SEO consultant, but don’t spend a cent if you’re thinking they know something you don’t. Patience and hard work always trumps looking for a quick fix and instant results.

Now for some handy links you should know about…

Google Webmaster Guidelines
Yahoo Terms Center
Bing’s Webmaster Center
W3C Standards
SEO Centro Keyword Rank Checker

That’s for me on this one guys, thanks for reading and see you on the next tutorial!

Take care,
Dan