Elliott C. Back: Internet & Technology

Does Amazon Vine Bias Reviews?

Posted in Amazon by Elliott Back on December 10th, 2010.

What is Amazon Vine?

For those not in the know, Amazon describes their Vine customer review program as:

Amazon Vine™ is a program that enables a select group of Amazon customers to post opinions about new and pre-release items to help their fellow customers make educated purchase decisions. Customers are invited to become Amazon Vine™ Voices based on the trust they have earned in the Amazon community for writing accurate and insightful reviews. Amazon provides Amazon Vine™ members with free copies of products that have been submitted to the program by vendors. Amazon does not influence the opinions of Amazon Vine™ members, nor do we modify or edit their reviews.

I disagree. I believe that Amazon Vine fundamentally biases reviewers. Sending customers free merchandise and require them to write about it should naturally produce positive bias. After all, if the book sucked, it was free, right?

Before we start, I would like to reference a number of existing article about Amazon Vine which may be relevant. First, in this Hacker News post, a member comments: “It is awfully tempting to start reviewing products with glowing praise in hopes that you will soon start getting free stuff in the mail.” Jon Bischke complains in Why Amazon Vine is a Threat Worth Talking About that “We’re being shown reviews for people who didn’t pay a dime for a product adjacent to people who shelled out their hard earned cash to pay for the product.” AvidBookReader promises that he’ll “just have to ignore these reviews, too.”

What’s the Data?

Note: between starting the idea for this research, and executing the idea, Amazon removed support from their API for reviews, making it many times more annoying to gather the necessary data. Instead of using a sanctioned means, I have resorted to scraping, which is unfortunate.

The dataset I am analyzing consists of 134,881 reviews gathered from the top 20 items of each category in the Amazon Best Sellers list. Each review consists of a star rating (out of five) and a boolean indicating if it is an Amazon vine review or not. Items which lack any Vine reviews at all were excluded, which includes the entire Industrial & Scientific section. A first pass through the data turns up 130,077 regular reviews, and 4,804 amazon vine reviews. Across all items, they fall into the following distribution:

Some categories do not have many Vine samples, but almost 16% of Software reviews are from the Vine program, with the following distribution:

Here we can visually observe that the Vine reviews appear to be skewed away from the 1/2-star reviews towards the 4/5-star area. To prove this, we will perform Chi-square significance tests on each category to try to prove that vine reviews significantly differ from regular reviews. We will use the alpha-cutoff of .5 (95%), as a standard measure. If you like, you can download the full results: Amazon Vine – Chi Squared Results (PDF).

Sample Results

In the Software category, we see some significant differences between Vine and regular Amazon reviews:

Here we get twice as many 4-star ratings as expected, and only about a quarter as many 1-star ratings, showing statistically significant Vine-inflation. The Chi Square statistic of 77 exceeds the critical area of 9.5, so we know that there’s a 0% chance of having a bad sample.

In the Office Products category, we see the opposite:

There is no statistically significant variance between the vine and regular reviews. It’s a coin toss if the vine reviews here are different or not from the regular Amazon reviews.

Full Results

The following categories, totaling 58% of reviews (77,762), showed significant differences between Vine and Regular reviews:

Amazon Video On Demand (5313), Automotive (2107), Beauty (4106), Books (5556), Camera & Photo (2655), Cell Phones & Accessories (5678), Grocery & Gourmet Food (5998), Health & Personal Care (8205), Home & Garden (11382), Home Improvement (2364), Magazines (2600), Movies & TV (10490), Music (3652), Software (1761), Toys & Games (1240), Video (4655)

These categories, totaling 42% of reviews (57,136) did not:

Baby (4879), Clothing (1222), Computer & Accessories (651), Electronics (24205), Industrial & Scientific (127), Jewelry (924), Kitchen & Dining (11382), Musical Instruments (1701), Office Products (877), Patio, Lawn & Garden (2588), Shoes (1001), Sports & Outdoors (3208), Video Games (3729), Watches (642)

This roughly corresponds to the high-low density partition of categories by percentage of Amazon Vine reviews. The significant categories enjoy an average Vine review rate of 5.6% of the total reviews, while the insignificant categories average just 1.9% vine reviews. So, I can attribute the significance either to insufficient Vine data in the categories I sample, or a network effect where more Vine reviews produce the desired bias. The following graph, of significance in distribution of reviews, to the percentage of vine data in the reviews, supports this theory:


When we look at the categories which display bias, we arrive at the following distribution of reviews:

We see a significant skew. Overall, we see that the Amazon Vine reviews average a 4.31, while the non-Vine reviews average 4.26. Standard deviations are 0.99 for Vine, compared to 1.22 for non-Vine. So the overall averages are not affected by the bias present in Amazon vine reviews. Rather, it is the distribution itself that is affected. Vine reviewers give out fewer 5s than regular reviewers, while giving more 4s, and fewer 1s. In some sense, we can say that the Vine program functions as a range compression in the review-space.

Given these facts, does the Amazon Vine program bias reviews? No. In fact, it can be thought of as a statistical reviews moderator. While the Vine reviews are differently distributed (compressed), they still hover about the same averages. And it’s the averages that inform the next consumer on the shopping site.

Note: I am not a Statistician, so if I really screwed up some concepts here, please leave a comment. I have the raw data, and can redo analysis if warranted.

MPAA Needs No Evidence To Sue You

Posted in bit torrent,bittorrent,Copyright,DMCA,iPhone,iPod,P2P by Elliott Back on June 30th, 2008.

The following remark was made by Marie. L. van Uitert, MPAA attorney in the Jammie Thomas trial. She wrote in a brief:

It is often very difficult, and in some cases, impossible, to provide such direct proof when confronting modern forms of copyright infringement, whether over P2P networks or otherwise; understandably, copyright infringers typically do not keep records of infringement. Mandating that proof could thus have the pernicious effect of depriving copyright owners of a practical remedy against massive copyright infringement in many cases.

The rest of the brief goes on to list the reasons why the MPAA feels it should not have to meet the full burden of proof in its case (i.e. proving actual distribution). For them, the existence of a location where the copyright material could be copied is sufficient grounds for prosecution. When you take this off the internet, this is equivalent to suing some for 12 * $150,00 for loaning someone a CD they later copied.

For more coverage, see Wired and TorrentFreak.

Mark Cuban’s P2P Ideas Suck

Posted in bit torrent,bittorrent,Celebrities,P2P,Performance,Scalability by Elliott Back on November 25th, 2007.

In a three-part rant about peer-to-peer technologies (1, 2, 3), Mark Cuban demands that peer-to-peer technologies “die a quick death” in order to”speed up [his own] internet connection.” He suggests that “Google Video is a far better solution for audio and video distribution than any P2P solution” and that cable companies “charge for upstream bandwidth usage.”

Guess what–I already get charged for all the bandwidth I use, either up or down. When Verizon strings a fiberoptic cable to my home, I’m getting a certain amount of fixed capacity into the greater internet at large. If I want to trade a little upstream capacity for greater downstream capacity, that’s my call! Have you ever noticed that downloading over http is typically slow because there are 100s of clients and 1 host? If I download the same information over bittorrent, I can sustain 12Mbs because everyone is a server–including me. Distributed protocols, such as the ones powering Amazon Dynamo or bittorrent, are more efficient, cost effective, and fault tolerant than single-server models.

Reactions around the blogosphere indicate that Mark Cuban’s thoughts on P2P are nonsensical rubbish. Mashable calls him “a guy who does not understand how P2P works, and yet he wants it shut down.” Ars Technica notes that “if users who are currently saturating their connections with BitTorrent start saturating their connections with Google Video content, the end result is more or less the same.” And a slashdotter comments, “Just imagine how fast the internet would be if there were no content to view. After P2Ps gone, get rid of all these freeloading websites, emails, etc. and it will be blisteringly fast.”

My guess is that billionaire Mark Cuban has a slow, shared cable internet connection at home, the modern equivalent of a party line. This might lead him to confuse his own slow internet connection with a greater systemic problem. What he should be complaining about is why Verizon hasn’t strung fiber in his area yet.

Next Page »