Elliott C. Back: Internet & Technology

I tried leaving this as a reply to Jesse Anderson’s A Few More Million Amazonian Monkeys but the blog comments are broken:

This is fundamentally pointless–there are only 27^9 (a-z and a space) combinations to run through. If you can do maybe 1M/s you can do them all in ~62 days. A better question is why, unless you hate the universe and want to spend our computational power producing entropy?

If you have a Best Buy giftcard, you might be wondering what to do with it. You could spend it at Best Buy, where they frisk you like a thief as you exit with your purchase, sell you overpriced goods, or force their “optimization” service on you when you try to buy a laptop. Or, you could try to convert your near-useless Best Buy giftcard into another currency. While cash is ideal, Amazon.com giftcards are almost as good. One way to do this is to sell your card to Cardpool, which take a 10% cut for the service:

Yeah, don’t do that, you’re literally throwing away your money. (OK–if there’s no Best Buy near you, maybe you have no choice…)

Did you know Best Buy sells Amazon Kindle gift cards? They do! Look for them in the accessories for e-readers section, near the Nooks, Kindles, Sony readers, and Kobos. At the Best Buy I tried out, they had both $25 and $50 denominations, making converting my $150 gift card a cinch. Even better, at least in NYC, there’s no sales tax charged on gift card sales–they’re treated as cash equivalents, I suppose.

So don’t trade your BestBuy.com giftcards in online for a 10% discount or sell them on eBay, etc. You can convert them to Amazon credit by simply going into the store and buying Kindle giftcards. The credit doesn’t appear to be specific to Kindle books–you can use it for anything sold on Amazon.

What is Amazon Vine?

For those not in the know, Amazon describes their Vine customer review program as:

Amazon Vine™ is a program that enables a select group of Amazon customers to post opinions about new and pre-release items to help their fellow customers make educated purchase decisions. Customers are invited to become Amazon Vine™ Voices based on the trust they have earned in the Amazon community for writing accurate and insightful reviews. Amazon provides Amazon Vine™ members with free copies of products that have been submitted to the program by vendors. Amazon does not influence the opinions of Amazon Vine™ members, nor do we modify or edit their reviews.

I disagree. I believe that Amazon Vine fundamentally biases reviewers. Sending customers free merchandise and require them to write about it should naturally produce positive bias. After all, if the book sucked, it was free, right?

Before we start, I would like to reference a number of existing article about Amazon Vine which may be relevant. First, in this Hacker News post, a member comments: “It is awfully tempting to start reviewing products with glowing praise in hopes that you will soon start getting free stuff in the mail.” Jon Bischke complains in Why Amazon Vine is a Threat Worth Talking About that “We’re being shown reviews for people who didn’t pay a dime for a product adjacent to people who shelled out their hard earned cash to pay for the product.” AvidBookReader promises that he’ll “just have to ignore these reviews, too.”

What’s the Data?

Note: between starting the idea for this research, and executing the idea, Amazon removed support from their API for reviews, making it many times more annoying to gather the necessary data. Instead of using a sanctioned means, I have resorted to scraping, which is unfortunate.

The dataset I am analyzing consists of 134,881 reviews gathered from the top 20 items of each category in the Amazon Best Sellers list. Each review consists of a star rating (out of five) and a boolean indicating if it is an Amazon vine review or not. Items which lack any Vine reviews at all were excluded, which includes the entire Industrial & Scientific section. A first pass through the data turns up 130,077 regular reviews, and 4,804 amazon vine reviews. Across all items, they fall into the following distribution:

Some categories do not have many Vine samples, but almost 16% of Software reviews are from the Vine program, with the following distribution:

Here we can visually observe that the Vine reviews appear to be skewed away from the 1/2-star reviews towards the 4/5-star area. To prove this, we will perform Chi-square significance tests on each category to try to prove that vine reviews significantly differ from regular reviews. We will use the alpha-cutoff of .5 (95%), as a standard measure. If you like, you can download the full results: Amazon Vine – Chi Squared Results (PDF).

Sample Results

In the Software category, we see some significant differences between Vine and regular Amazon reviews:

Here we get twice as many 4-star ratings as expected, and only about a quarter as many 1-star ratings, showing statistically significant Vine-inflation. The Chi Square statistic of 77 exceeds the critical area of 9.5, so we know that there’s a 0% chance of having a bad sample.

In the Office Products category, we see the opposite:

There is no statistically significant variance between the vine and regular reviews. It’s a coin toss if the vine reviews here are different or not from the regular Amazon reviews.

Full Results

The following categories, totaling 58% of reviews (77,762), showed significant differences between Vine and Regular reviews:

Amazon Video On Demand (5313), Automotive (2107), Beauty (4106), Books (5556), Camera & Photo (2655), Cell Phones & Accessories (5678), Grocery & Gourmet Food (5998), Health & Personal Care (8205), Home & Garden (11382), Home Improvement (2364), Magazines (2600), Movies & TV (10490), Music (3652), Software (1761), Toys & Games (1240), Video (4655)

These categories, totaling 42% of reviews (57,136) did not:

Baby (4879), Clothing (1222), Computer & Accessories (651), Electronics (24205), Industrial & Scientific (127), Jewelry (924), Kitchen & Dining (11382), Musical Instruments (1701), Office Products (877), Patio, Lawn & Garden (2588), Shoes (1001), Sports & Outdoors (3208), Video Games (3729), Watches (642)

This roughly corresponds to the high-low density partition of categories by percentage of Amazon Vine reviews. The significant categories enjoy an average Vine review rate of 5.6% of the total reviews, while the insignificant categories average just 1.9% vine reviews. So, I can attribute the significance either to insufficient Vine data in the categories I sample, or a network effect where more Vine reviews produce the desired bias. The following graph, of significance in distribution of reviews, to the percentage of vine data in the reviews, supports this theory:

Conclusion

When we look at the categories which display bias, we arrive at the following distribution of reviews:

We see a significant skew. Overall, we see that the Amazon Vine reviews average a 4.31, while the non-Vine reviews average 4.26. Standard deviations are 0.99 for Vine, compared to 1.22 for non-Vine. So the overall averages are not affected by the bias present in Amazon vine reviews. Rather, it is the distribution itself that is affected. Vine reviewers give out fewer 5s than regular reviewers, while giving more 4s, and fewer 1s. In some sense, we can say that the Vine program functions as a range compression in the review-space.

Given these facts, does the Amazon Vine program bias reviews? No. In fact, it can be thought of as a statistical reviews moderator. While the Vine reviews are differently distributed (compressed), they still hover about the same averages. And it’s the averages that inform the next consumer on the shopping site.

Note: I am not a Statistician, so if I really screwed up some concepts here, please leave a comment. I have the raw data, and can redo analysis if warranted.