Does Amazon Vine Bias Reviews?
What is Amazon Vine?
For those not in the know, Amazon describes their Vine customer review program as:
Amazon Vine™ is a program that enables a select group of Amazon customers to post opinions about new and pre-release items to help their fellow customers make educated purchase decisions. Customers are invited to become Amazon Vine™ Voices based on the trust they have earned in the Amazon community for writing accurate and insightful reviews. Amazon provides Amazon Vine™ members with free copies of products that have been submitted to the program by vendors. Amazon does not influence the opinions of Amazon Vine™ members, nor do we modify or edit their reviews.
I disagree. I believe that Amazon Vine fundamentally biases reviewers. Sending customers free merchandise and require them to write about it should naturally produce positive bias. After all, if the book sucked, it was free, right?
Before we start, I would like to reference a number of existing article about Amazon Vine which may be relevant. First, in this Hacker News post, a member comments: “It is awfully tempting to start reviewing products with glowing praise in hopes that you will soon start getting free stuff in the mail.” Jon Bischke complains in Why Amazon Vine is a Threat Worth Talking About that “We’re being shown reviews for people who didn’t pay a dime for a product adjacent to people who shelled out their hard earned cash to pay for the product.” AvidBookReader promises that he’ll “just have to ignore these reviews, too.”
What’s the Data?
Note: between starting the idea for this research, and executing the idea, Amazon removed support from their API for reviews, making it many times more annoying to gather the necessary data. Instead of using a sanctioned means, I have resorted to scraping, which is unfortunate.
The dataset I am analyzing consists of 134,881 reviews gathered from the top 20 items of each category in the Amazon Best Sellers list. Each review consists of a star rating (out of five) and a boolean indicating if it is an Amazon vine review or not. Items which lack any Vine reviews at all were excluded, which includes the entire Industrial & Scientific section. A first pass through the data turns up 130,077 regular reviews, and 4,804 amazon vine reviews. Across all items, they fall into the following distribution:
Some categories do not have many Vine samples, but almost 16% of Software reviews are from the Vine program, with the following distribution:
Here we can visually observe that the Vine reviews appear to be skewed away from the 1/2-star reviews towards the 4/5-star area. To prove this, we will perform Chi-square significance tests on each category to try to prove that vine reviews significantly differ from regular reviews. We will use the alpha-cutoff of .5 (95%), as a standard measure. If you like, you can download the full results: Amazon Vine – Chi Squared Results (PDF).
Sample Results
In the Software category, we see some significant differences between Vine and regular Amazon reviews:
Here we get twice as many 4-star ratings as expected, and only about a quarter as many 1-star ratings, showing statistically significant Vine-inflation. The Chi Square statistic of 77 exceeds the critical area of 9.5, so we know that there’s a 0% chance of having a bad sample.
In the Office Products category, we see the opposite:
There is no statistically significant variance between the vine and regular reviews. It’s a coin toss if the vine reviews here are different or not from the regular Amazon reviews.
Full Results
The following categories, totaling 58% of reviews (77,762), showed significant differences between Vine and Regular reviews:
Amazon Video On Demand (5313), Automotive (2107), Beauty (4106), Books (5556), Camera & Photo (2655), Cell Phones & Accessories (5678), Grocery & Gourmet Food (5998), Health & Personal Care (8205), Home & Garden (11382), Home Improvement (2364), Magazines (2600), Movies & TV (10490), Music (3652), Software (1761), Toys & Games (1240), Video (4655)
These categories, totaling 42% of reviews (57,136) did not:
Baby (4879), Clothing (1222), Computer & Accessories (651), Electronics (24205), Industrial & Scientific (127), Jewelry (924), Kitchen & Dining (11382), Musical Instruments (1701), Office Products (877), Patio, Lawn & Garden (2588), Shoes (1001), Sports & Outdoors (3208), Video Games (3729), Watches (642)
This roughly corresponds to the high-low density partition of categories by percentage of Amazon Vine reviews. The significant categories enjoy an average Vine review rate of 5.6% of the total reviews, while the insignificant categories average just 1.9% vine reviews. So, I can attribute the significance either to insufficient Vine data in the categories I sample, or a network effect where more Vine reviews produce the desired bias. The following graph, of significance in distribution of reviews, to the percentage of vine data in the reviews, supports this theory:
Conclusion
When we look at the categories which display bias, we arrive at the following distribution of reviews:
We see a significant skew. Overall, we see that the Amazon Vine reviews average a 4.31, while the non-Vine reviews average 4.26. Standard deviations are 0.99 for Vine, compared to 1.22 for non-Vine. So the overall averages are not affected by the bias present in Amazon vine reviews. Rather, it is the distribution itself that is affected. Vine reviewers give out fewer 5s than regular reviewers, while giving more 4s, and fewer 1s. In some sense, we can say that the Vine program functions as a range compression in the review-space.
Given these facts, does the Amazon Vine program bias reviews? No. In fact, it can be thought of as a statistical reviews moderator. While the Vine reviews are differently distributed (compressed), they still hover about the same averages. And it’s the averages that inform the next consumer on the shopping site.
Note: I am not a Statistician, so if I really screwed up some concepts here, please leave a comment. I have the raw data, and can redo analysis if warranted.
| This entry was posted on Friday, December 10th, 2010 at 12:30 am and is tagged with amazon community, amazon customers, hacker news, fellow customers, bischke, insightful reviews, purchase decisions, mail, free merchandise, free copies, necessary data, news post, member comments, free stuff, select group, biases, dataset, reviewers, bias, best sellers. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback. |
Fantastic analysis! Really glad that you did this. It brings data to bear on the question of Vine reviewer bias.
Your analysis is very interesting, but is does have a few methodological flaws, like choosing the wrong statistic. Not every matrix implies a chi-square is the appropriate test. You’re taking an ordinal or possible interval scale- mean score- and artificially constructing categorical matrices. In doing so you’re actually losing information.
Doing multiple chi squares over subsets of the data set is another no-no, since you can’t assume independence. The fact that you find half of your samples show “significance” and half don’t should have clued you that you’re looking at a random distribution of differences.
The first test you might have done is a simple difference of means T-test. It’s very robust, and would show you right away if there was a difference in the mean scores between Vine reviewers and reviewers at large. You could also check to see if there’s a difference between verified purchasers and others.
After a year in the vine program, I have some observations.
I have noticed that my reviews are much more middle of the road than before. I feel obligated to finish the vine book and post a review; on my own, I put down books I’m not that excited by, and mostly write reviews for books I either loved or hated. It’s hard and not fun to write middle of the road reviews. The charts seems to agree with this.
I don’t think the free-ness of the books matters for me (I’m a heavy library user), but for non-book items, free does mean the reviews are lower quality. I got some dog food, which the dog thought was average. Unlike books, any serious assessment of dog food would include cost, and I couldn’t do that when I have no idea what it would cost in the store.
I have seem a small feelings of nudge toward postitive reviews, since I looks like positive reviews get more helpful votes and my ratings are more visible to me from vine. I think this would be for anyone who gets to thinking about helpful votes, any serious reviewer, not particular to vine people.
Also, since these are new books and at least one by a new author, I sometimes feel like the poor author needs a boost or at least me to be gentle; I don’t want to hurt their feelings.
I’m a Vines Reviewer, and Vines does not require you to review everything you get. You need only review 2 out of 3 items. I don’t get paid. I was reviewing things on Amazon long before I was in Vines, and I don’t write the reviews I do for Vine any differently then the ones I do on Amazon for items I buy.
With that said, Amazon doesn’t force things on you, so no one is going to review things they think they are sure to hate. They may end up hating them anyway, but they certainly aren’t going to pick things they are sure they would hate.
On the other hand, when I am buying things on Amazon I don’t buy things I think I’m going to hate either, so there is really no difference in that respect.
I also don’t pick things every time I get a Vines newsletter, because if they are out of everything of interest to me, and the quantities they have are limited, I don’t get something I don’t want.
So, all-in-all, I know for a fact that my Vines reviews are not prejudiced.
I must take issue here, as an Amazon Vine reviewer. I am not paid by Amazon. An invitation to join the Vine scheme was sent to me because I was already a frequent reviewer and other customers had given some of my reviews the ‘thumbs-up’ (thanks for that!). My reviews are genuine and honest. There is no ‘deal’ involved.
I dismiss Amazon Vine reviewers as infomercials. The are paid by the manufacture through it agent.
This is why the INDEPENDENT Consumer Union exists.
The significant question is how are top reviewers selected? Answer: to sell more products. Dah?
I an engineer I crave for factual observations from knowledgeable persons. The low rated reviews are more important than the cheer-leading ones.
The Amazon reviewers must worry about staying top-rated to continue to receive more free products.
The trend in occurring because consumers like to buy top-rated products. Good products don’t need advertising as they sell themselves.
Amazon Vine is all about manipulation and lobbying influence.
Thanks for posting an interesting article and adding to the debate. The comments from vine reviewers and statisticians are also fascinating.
I wonder if the ‘perceived’ bias in vine reviews was equivalent to that in free kindle books? Many of these (and I’m not thinking of acknowledged classics) are given 4 or 5 star reviews. I appreciate that these could be by a secret vine reviewer, but are probably by an ordinary reader taking account of the cost of purchase. I may be guilty of this myself, although one book was so poor, I probably should have given it 1 star, buy gave it 2 as it was free.
I’m an invited Vine reviewer. I suspect because my hundreds of book reviews are RELATIVELY unbiased. Note the deliberate emphasis.
All reviews/reviewers are biased. You can’t get away from it. The question ought to be, are the reviews rational, balanced and are opinions backed up, where possible, by facts.
The rest is just smoke in the wind.
It is crazy out there! On the Amazon website it reports in a green font something like “this review is from a Vine member”. HOWEVER, and this is a big effing however, when viewing literature on the actual Amazon Kindle device, from within the “Shop in Kindle Store”, Amazon DOES NOT report if a review is vine or not. Why is that? Because it’s a decoy to get you to *think* you might enjoy a book since somebody took the time to write a review when in actuality, they were employed to do so.
I was just invited to join vine, and was doing a little research. Early on you say that there might be a positive bias since non-vine folks pay for the products… but actually, the bulk of my reviews I’ve submitted are for books I’ve got from the library.
Well done. As a Vine reviewer who has done a fair amount of work in statistics, I have one observation: Vine reviewers, compared to non-Vine reviewers, have a higher probability of having actually tried the product under review. Perhaps it would be useful to do yet another analysis, making a distinction between reviews who are verified Amazon purchasers, non-verified reviewers, and Vine reviewers.
Elliott, I am a member of Amazon Vine, and I actually specifically asked my husband (a computer programmer) to write a tool to analyze a reviewer’s reviewing patterns, in order to show whether they skew more positively on their “free” Vine items rather than on their “regular” reviews. This tool – Amazon Reviewer Analysis Tool – can run an analysis against the profile of ANY Amazon reviewer, and you can see, for instance, how much of a percentage their 1-star reviews constitute their overall reviews vs. their vine reviews. You’re welcome to download the tool and use it, for instance on my account.
>From my personal experience, my Vine reviews are actually more negative than my “regular” reviews. This is partly due to selection – I shop for my regular items more selectively than for my Vine items, which are narrowed by definition. But it’s also partly because I’ve gotten some really crummy books via Vine and there’s a HUGE difference between being able to set aside a book forever…
That’s brilliant! Kudos to him. Comparing Vine reviewers’ reviews of Vine products vs. non-Vine is the key analysis!
To put it another way– if data showed a bias in Vine reviews, it could be due to a selection effect. In other words, the Vine reviewers may be more favorable reviewers in general!
As to the comparisons here and someone’s suggested stat test, I’m at a loss that the proper stat test would be a T-test. There is no random selection here of reviews–there is no defined population from which the reviews were randomly taken. In this case, what the reviews are, is what they are.
What’s of most importance is the practical significance of the difference. Even if you found a statistical difference, it could be meaningless in practice.
Now, as to the graphs… it’d be helpful if the y-axis were labeled. What do the gridlines correspond to?
Even on a superficial look at them, it seems that the non-Vine reviews are the ones that are incredibly positively skewed. If you look closely at the first graph, e.g., you’ll see that the non-Vine reviewers pick a 5 almost 3x as often as a 4. For the Vine reviewers, it’s only 2x as often.
Or, are the graphs mis-labeled?!
My own view is that Vine reviews tend to be more objective and critical since there is no investment on the part of the reviewer. Viners also tend to avoid the short, 1-star review temper tantrums if they never receive a particular item.
I am in UK Vine. I’m not wildly enthusiastic about it. In all the time I’ve been in Vine (15 monthe), I’ve accepted 5 books of which I’ve reviewed three so far. Two got three stars and one got two stars. I read about half the fourth book and got side-tracked so it will likely get four starts when I review it, unless it gets markedly better or worse when I get round to reviewing it. I haven’t started the fifth book but it is by one of my favorite authors. Let’s suppose it will get 5 stars. That makes 5 books with beteen them 17 stars out of 25. Yet I am one of those reviewers who are known for posting loads of 5-star reviews, so 17/25 actually lowers my average star rating.
I have not studied whether Amazon select reviewers for Vine based on the star ratings they give, though it might be worth doing. I’ve long been a high profile reviewer and there are any number of reasons why I might have qualified for Vine, so my entry into Vine proves nothing either way.
However, the basic premise that people rate Vine items highly is a false one, and not just based on my own misleading case. Once in Vine, people are given a list of stuff from which to choose, and they are limited to how much they can choose each month. Sometimes, people order stuff that they wouldn’t normally consider ordering – stuff that they may or may not like. As such, those reviewers who value their integrity and/or credibility (as I do) give low star ratings for stuff they don’t like.
It may be that some reviewers are only concerned with free stuff and fear that they’ll get kicked out of Vine if they don’t give glowing reviews. They are wrong.
People have been kicked out of Vine, but not for giving low star ratings. One person was apparently kicked out for just copying the product description and posting it as their review for each Vine item they ordered. I think others may have been kicked out for re-selling Vine items (which is against the rules).
Sorry, but your misleading statistics prove nothing. As somebody else said, you’d have to compare each reviewer’s performance, looking at the average rating of their Vine reviews and comparing it to their non-Bine reviews. Asoftware tool is available that will allow you to do this, providing you have a Windows PC that you can download the software to. I didn’t write the software, but I wrote this guide about it
Analyse your reviews and track votes guide.
Vine work by seeding the reviews, had you thought of that? Look for Vine items where there are no or few non-Vine reviews and you will see this happening. In fact we in Vine notice it a lot: we are often the first to review items.
Have you considered that the non-Vine reviews might be influenced by the Vine ones? Perhaps something to pursue further.
My experience of other Vine members, certainly the vociferous ones on the forum, is that they are often very unknowledgeable about what they write reviews on. Thus in that sense a bias is introduced.
This is an interesting topic; thanks for doing this analysis. I am a Vine reviewer and have given the sources of bias some thought.
When investigating the possibility of significant bias in Vine reviews, it is crucial to take into account the fact that there are significant biases in NON-Vine reviews. These biases are well known; if I simply ask the question, you will immediately recognize the bias:
Who is more likely to submit a review, somebody who is reasonably happy with their purchase or someone who is upset with their purchase?
The answer, as I’m sure you know, is that people who are upset with their purchases are much more likely to write reviews. This in part reflects a genuine desire to warn others; it also reflects a desire to exact vengeance. (interesting reading in evolutionary psychology, if you want to go there)
Next most likely to submit a review are those who are absolutely delighted with their purchase. People like to tell others of their good fortune.
Buyers who are OK with what they got but not delighted are less likely to write a review at all, not only because neither of the above emotions come into play, but also because they may feel that they have little to say that would be of any interest. “Yes, I got one of these things, and it’s OK, does what you think” is not a sentiment that particularly motivates people to take the time to write a review. Moreover, it is certain to score you “not helpful” votes. We simply should not expect many reviews of this type.
So, I’m making these numbers up, but let’s imagine that 80% of people who are upset are motivated to write a 1-star review, and 30% of people who are delighted are motivated to write a 5-star review, and 10% of the rest write a review. The real numbers, while surely different from these, will nonetheless reflect the biases discussed.
OK, now think about the Vine program. Generally people write reviews regardless of how disappointed or excited they are about the product. That clearly leads one to expect a lower proportion of 5-star reviews, and a much lower proportion of 1-star reviews, among Vine reviews. That is, in fact, what we see. There is no need to assume anything nefarious to explain what we see.
I’m not claiming that Vine reviews are *not* biased; I’m just suggesting that we exercise care in trying to demonstrate that they *are* by comparing them to non-Vine reviews.
Yes, you screwed up the analysis. All that you have proven is that Vine reviews are statistically deviant from non-Vine reviews. There are two problems with your claim.
First, you have no proof of causation. It is perfectly possible that this is merely a correlation, and an underlying phenomenon is to blame.
Second, you have no evidence that neither (or both) of the following two confounds exist. However, I gave advice on how to address them, for your gratification.
A) The ratings given may or may not reflect a bias as a result of free items. To analyze this, you would need to check their average ratings in a category for things chosen from Vine against things that were NOT free. More accurately yet, it would be nice to see an analysis of the difference between the Vine member’s review and the average for each type of item, and then average by category and by Vine member.
This would allow you to discern whether a particular member (and members in general) unconsciously skew their ratings for free products as compared with those that they pay for. Moreover, it would allow you to see if there is a correlation between price and amount of skew, which IS what a cynic would expect, a priori.
B) Another interesting phenomenon, and one much harder to track, is whether Pollyannas are selected for the program. I.e., are the people selected representative of the Amazon user community with respect to their ratings. In order to track this, you’d have to look at a lot of reviews from each group and see if there is a deviation from the other group’s average for a significant number of items, preferably from a mix of categories.
I won’t dismiss your results out of hand, because I have done similar sorts of analyses and noticed that reviews from Vine members are skewed toward more favorable reviews. I, however, didn’t come to the conclusion that they were biased. Merely that they are bad a normal distributions and following Amazon’s advice about how to rate products (a third possibility, perhaps).
When I pointed this out, I was told that they are “better at choosing things that they will like” and that they “take time to find something they will enjoy”. Both sound like smug self-justifications to me, and avoidance of attempting to analyze one’s own behavior. (At the time, I also analyzed my own reviews and found that I had a normal, if flattened, distribution, in the sake of complete disclosure.)