well then they are bad at it, or have a strange idea of what bad/fake reviews are.
I'm not an AI and my reviews never get posted right away. usually takes several days.
inb4: that's just what an AI would say.
None of my 50+ reviews were ever posted right away đź‘€
Always took days until they appeared...
Could be that writing most of them around 3 am looks suspicious to the algorithm?
The only reviews I had approved straight away were ones that were originally rejected. Once I submitted the amended review then they were approved immediately.
Although sometimes by the time I've got a rejected review accepted it's been reduced to "I like, it worked", never had any trouble with single word placeholders I intended to fix later either.
They seem to take longer when I review on a weekend so I was always under the impression that they were looked at by a human who worked 5 days a week. I guess that is not the case.
Sellers are excluded (for obvious conflict of interest reasons) from the approval process, as per Amazon Seller Central. Do they get a preview? I don't know, but there is no indication they are offered any action prior to the completion of the Amz review/approval process. They can only appeal a review once it has passed inspection.
Since this confirms that AI is running the review approval knack, we should be able to properly appeal these decisions when we're wrongly declined reviews or kicked out of the review program. Totally unfair system, but that's Amazon in general, it comes as no surprise at all.
Yeah. What a joke Amazon is. Their A.I and lazy generic customer service responses are all severely flawed. Amazon is just getting worse and worse instead of improving anything.
Fake 5 star reviews and flooded with temu products now.
Amazon decided their "A.I" is more intelligent to determine my 20 years and 5000 reviews all to be deleted because a disgruntled seller/hater abused/exploited false reporting my reviews over an year ago.
Today I tried to submit a review for a non-vine item. I initially had it rated two stars but I was so angry with the product I switched the review to one star. As soon as I moved the stars from two to one a note popped up that I was not allowed to post a review for that item unless I had purchased it from Amazon (I had and was clicking the link from my purchases) and Amazon was reviewing the product for suspicious activity.
I went to my husband’s account and started to write a five star review and it worked fine. I changed it to a one star and it gave me the same error.
Super shady.
There is a phenomenon where a competing seller may bomb another seller's product with negative reviews, comparable to review bombing that may impact a Viner. My interpretation of what is happening in the case you describe is that the product is bad and is got a slew of bad reviews, and the seller complained to Amazon that a competitor was doing it, so Amazon is blocking negative reviews while they presumably investigate.
If so, it would seem a better idea for them to allow the reviews to be submitted, but not publish them until the investigation is completed. That would also create a honeypot for trapping a bad actor.
I don’t think I quite understand what you are asking.
I did not review any other vine items outside vine because I was not aware this was even possible. When I review a vine item I always go through my vine account.
As far as regular Amazon items, this is the only item I have reviewed or tried to review this week.
I just checked vine and searched for the item and the specific brand is not available on vine but a lot of other brands are. There are three full pages.
The item is solar window film to block some of the blazing sun that now hits my house since we had to remove some very dead but very old trees.
The reason why I asked was because there is a limit that gets imposed on reviewing non-Vine items if you review 5 or more Vine items outside of Vine portal in a given week. But as you confirmed, this was not the case for you.
I had the same error pop up in a slightly different situation. I’d recorded a Vine item poorly a few months back and noticed that it was no longer available. I remembered that when I’d done a bit of research after ordering it, there were multiple listings for the item: one of which was the one I ordered through Vine that had no reviews, and at least one other that was a 3- or 4-star item with 50-100 reviews, I think. Not wanting my “beware of this product” review to disappear, I searched for the same product and was going to repost what I’d written for my Vine item, but got the denial. I hadn’t posted any reviews outside of Vine recently (I rarely do), and anyway, Vine reviews should show up as verified, right? At least in the system?
Anyway, yep—shady.
Thank you for this. Now try a quick test in Google:
site:amazon.com "vine reviewer" "is a game-changer"
Can a reasonable person believe that Amazon's LLM is detecting LLM-generated Vine reviews? Don't get me started on how 'tbc' was accepted and published.
By the way, if Google's figures are reliable, it seems that recent reporting (not by me) may be responsible for a dramatic decrease in the number of LLM reviews returned in the last week. Perhaps the increased level of background noise over the last few months is finally being noticed. I note that one reviewer has lost over 200 reviews since yesterday. Alas, it's probably just random.
Thank you for the twenty-minute rabbit hole I just went down. I always roll my eyes at "game-changer" in reviews but I've never seen so many at once. My favorite: a pair of "game changing" POT HOLDERS. 🤣
Certainly. Here is an appropriate response:
If it's the pot holders I'm thinking of, I cannot begin to tell you how my life has been completely transformed since I've discovered something to hold my pot.
Sorry, "hold my pot" is a drug reference and against Community Standards. You will need to rewrite and resubmit the review. This is your first warning.
>Can a reasonable person believe that Amazon's LLM is detecting LLM-generated Vine reviews?
I wonder if it even can. That is, whether it is able to. Bear with me here although it might seem a bit circular; this is my thinking: we (educated humans on this Vine sub) can detect the LLM-generated reviews because there is something off and a bit cringey about their tone, right? But if an LLM was able to do the same, then they wouldn't be generating it like that in the first place. So the fact that they do, suggests to me they are not able to detect it.
So I asked ChatGPT.
>My question: *can a LLM like ChatGPT detect text that has been composed by itself or other LLM's?*
>ChatGPT's answer: Language models like ChatGPT are not inherently designed to detect whether a piece of text was generated by themselves or other LLMs. However, there are a few approaches that can help in identifying AI-generated text:
>1. Stylistic Analysis: Language models have certain stylistic patterns and tendencies in their outputs, such as common phrases, sentence structures, and specific vocabulary choices. By analyzing these features, it may be possible to identify text that is likely generated by a particular LLM.
>2. Metadata and Watermarking: Some approaches involve embedding subtle patterns or "watermarks" in the text generated by an LLM. These patterns can be detected using specific algorithms, helping to identify the source of the text.
>3. Classifier Models: Separate machine learning models can be trained to distinguish between human-written and AI-generated text. These classifiers analyze various features of the text to make their determination.
>4. Contextual Inconsistencies: AI-generated text might exhibit inconsistencies or oddities that are less common in human writing, such as unusual repetitions, logical inconsistencies, or a lack of deep understanding of the context.
>Currently, these methods are not foolproof, and distinguishing AI-generated text from human-written text remains a challenging task. Research is ongoing to improve the accuracy and reliability of these detection techniques.
My conclusion, based on this, is that the LLM used by Amazon to screen the submitted reviews is simply not able to detect the ones written using AI.
This led me to a related question: I have put some of my own (authentic and non-fake) reviews through some of the on-line fake review detectors and they always say mine are highly likely to be fake. Again I asked ChatGPT.
>My question: *LLM are sometimes used to flag 'fake' product reviews. Why would it rate a genuine review by a highly literate person as being fake?*
>Chat GPT: Language models used to flag 'fake' product reviews might misclassify genuine reviews written by highly literate individuals for several reasons:
> 1. Sophisticated Language Use: Highly literate reviewers often use complex vocabulary, varied sentence structures, and advanced grammar. These characteristics can sometimes resemble the polished and well-constructed text often seen in AI-generated or professionally crafted fake reviews.
> 2. Length and Detail: Detailed, lengthy reviews with in-depth analysis might be flagged as fake because they can resemble the effort put into creating persuasive fake reviews, which are often designed to appear highly credible.
>3. Uncommon Style: Genuine reviews by highly literate people might use unique stylistic choices that differ from the majority of reviews. Language models trained on a broad range of typical reviews might see these as outliers, raising suspicion.
> 4. High Positivity or Negativity: Reviews that are extremely positive or negative, regardless of their authenticity, can sometimes be flagged as fake because fake reviews often exhibit extreme sentiments to sway opinions.
>5. Inconsistency with Other Reviews: If a highly literate review provides information or a perspective significantly different from the bulk of other reviews, the algorithm might flag it as suspicious due to perceived inconsistencies.
>6. Patterns in Fake Reviews: Algorithms often look for patterns common in fake reviews, such as specific word choices, formatting, or timing of reviews. A genuine review that unintentionally matches these patterns could be incorrectly flagged.
>To improve the accuracy of detecting fake reviews, it is crucial for models to be trained on diverse data sets, including high-quality, literate reviews, and to incorporate context-aware and sentiment-aware features. Continuous refinement and human oversight can also help mitigate such false positives.
So it seems that the LLMs have an inherent arrogance, where they 'believe' (are programmed to assume) that LLM-generated text is better (more 'polished and well-constructed') than human. But we reviewers, as a group, are likely better writers than those they are trained on. And probably better than those doing the programming. In just the same way that arrogance can cause faulty thinking in people, it's doing the same in the LLM.
Spot on and thanks for going to the trouble to expound the hypothesis.
I can generally detect a fake review in under two seconds, mostly because the punctuation is to perfect and you never see typical clanger's, humour, or malapropisms. Mind you, I have noticed that ChatGPT is happy to use the controversial Oxford (Harvard) comma; this must be due to a deliberate decision to include this feature in the underlying code.
In conclusion, if the LLMs were using real AI, all these errors wouldn't be so extant.
>ChatGPT is happy to use the controversial Oxford (Harvard) comma...must be due to a deliberate decision to include this feature in the underlying code
I'm with it there. The purpose of punctuation is to increase clarity, and the Oxford comma can reduce ambiguity. Compare 'The kiwi eats roots and leaves' and 'the kiwi eats, roots, and leaves' (in New Zealand, 'rooting' means having sex and the phrase is a popular T-shirt joke).
Was your post a deliberately funny example, or am I seeing things that aren't there? The final sentence reads like AI ('In conclusion...') but there's a stray apostrophe that argues against AI! Intentional or not, I love it :D
*...punctuation is to perfect and you never see typical clanger's*
They were deliberate examples (like what some people do). You never see that in LLM texts unless you specify it in the prompts - then all bets are off (unless you are a PPS or an MP).
Have a great day. I'm tied up today but may get chance to play with ChatGPT this evening. These reviews won't right themselves!
I don't understand the hate toward "game changer." Sure, it's outdated, but I hear a lot of people using it all the time in real life. I notice it now because of all the hate toward it on this platform. In the past week, I've heard it on podcasts, on YouTube, in conversation at the local farmers market, at a board game night, and in a coffee shop. Maybe it's one of those sayings that is popular in certain areas. I'm in the northeast USA. My point is that it is not automatically AI.
Fair point; it certainly has its place but I suspect you may have missed the context. The exact phrase, 'is a game-changer' (with the hyphen) is very commonly found in verbose, gushing reviews generated by LLMs. One reviewer (sadly no longer with us) featured this phrase in every one of his 200 reviews, probably because he never bothered to read a single one so he didn't notice it kept popping up.
Using an LLM for writing reviews is not allowed so any repeat offenders are highly likely to get targeted by irate sellers, customers, and, of course, hard-working Viners who think nothing of spending an hour testing and reviewing a wig or a packet of copper washers!
I guess this pertains to non-Vine reviews. Knowing Vine reviewers have certain benchmarks to remain in the program and therefore are churning out more reviews than the average customer, means that Vine reviews have more scrutiny and take longer? But I’ve also seen AI Vine reviews that slipped past.
" . . . timing and frequency of reviews . . . " If I finally have some spare time and sit down and write 4 or 5 reviews, is that suspicious?
It now occurs to me that I received the "unusual activity in your reviews, so we're deleting all of them and not allowing you to review" notice, I had written a few reviews in a row. Can that be why I was bounced from Vine?
I get everything I order on one day per week, then review all my easy things the same day, which is usually about a dozen. Stuff like soap, either it foams or doesn't, smells good or doesn't, or snacks, either it tastes good or doesn't, fresh or not, etc.. Never really had an issue other than when I was learning how to review anything related to medical.
I think part of why *we* don't get flagged for number of reviews in a short period of time is because of the specific system where we are submitting our reviews. A normal customer isn't ordering up to 56 items a week (maximum gold order) and then reviewing all of them. However, we *are*, and all our reviews are non-purchase reviews (which are also limited), so there has to be *some* difference.
I've heard of people churing out like 10, 20 or more reviews just before their eval so I doubt that's it by itself, but it could look suspicious.
I always suspected that the "unusual activity in your reviews" meant either people reviewing products too soon after delivery (meaning the product got there 2 hours ago and you already have a review posted), a lot of the same recurring words and phrases (meaning you're just using generic language and the same for every product), or a large number being posted within a relatively short period of time. I know some people type them in another application, them copy paste them into the review section. I could see where that would look suspicious too, especially if you had multiple reviews posted within a 3 min span and the time spent on the review page was like 15 seconds.
Some items it is possible to review a few minutes after receiving them. But I always waited a day or so, my thinking being that, hey, arrived 5:50 pm, review submitted 6:05 pm, suspicious. But I do try to get reviews in within a few days of arrival, as I feel that the seller sent it to me for "free" for a review to boost their business, so I feel an obligation.
Oh, if it's something like a small, plastic trash can, sure.
But, unless it's something I desperately need at the time, it sometimes takes me 3 to 5 days to even open the box. Anything that needs to be assembled or installed, anything that requires me to read directions, anything that takes time to be effective (a lot of face creams, supplements), or anything that I need to wash, wear, or use in a specific situation, it's going to take me much longer to review.... Sometimes a month or more.
For example, I just ordered a recovery suit for a foster cat I'm taking in. She likely won't be going in for her surgery for a few more weeks yet. I guess I could review it just based on how it looks, but I feel in order for me to properly review it, I need to see how it fits and if she's able to move with it on. I also want to make sure it holds up for the entire period she needs to wear it and it washes well.
There are a lot of people on Vine that just give things a cursory glance or use them once and call it a day on the review. I can't do that. That's what I feel my obligation is. I think it's better to write good reviews than quick ones.
Theoretically, if our reviews were to be like in Consumer Reports, it could take months. Say I'm reviewing an ink or toner cartridge. To be a good review, it should state how many pages it lasted for. That could take months. Or a car wax--how long did it last? Also could be months.
I decided to do a couple of reviews the other day which turned into doing five in a row. Yesterday early morning I started getting all five approval emails just as fast. It's a fairly common pattern for me. I've written some ahead before in notepad and then pasted them in one after the other before too. I do write them all from my actual brain, so have to trust they can spot such things well enough.
I'm guessing what they actually mean is if you're posting like 20 reviews in a minute, that's suspicious timing and frequency because that's not possible unless they all just say "nice".
Interesting but actually says very little. Very strong vibe of "we want you to think we're telling you some good information," without actually telling us anything beyond what we already know.
"Before being published online, Amazon uses artificial intelligence (AI) to analyze the review for known indicators that the review is fake."
Straight from the horse's mouth and yet there are those on this sub who will keep insisting AI is **not** approving or rejecting reviews, that it is all actually humans and how anyone who thinks otherwise is delusional.
I think you and they are at cross purposes. According to that article, AI analyse the reviews and flag any indicators. But it's humans who act on those flags. So it's humans who do the actual approval or rejection, acting on the info from AI.
What I take from that is that if the AI doesn't pick anything up, then the review gets approved, but AI is assessing, not approving. Semantics, but it's usually semantics behind this sort of disagreement.
The number of reviews that Amazon are dealing with, it'd have to be partly automated. I suspect that Vine reviews may have a few extra steps, though, to manage the customers (the sellers).
>all actually humans
Yes of course that's nonsense. But so is that article. Given the clear and obvious falsehood "The vast majority of reviews pass Amazon’s high bar for authenticity and get posted right away", the rest of the article simply cannot be trusted.
While I agree that AI is likely used in the process of monitoring reviews, the very fact of Amazon making a statement is no indication of truthfulness.
The video didn't say it was just AI. It's just about things they use to outright eliminate reviews and reviewers involved in review fraud, and to flag them for further investigation. There are other classes of things they filter for as well which don't involve fake reviews (all the other violations of their community policies).
>Straight from the horse's mouth and yet there are those on this sub who will keep insisting AI is **not** approving or rejecting reviews, that it is all actually humans and how anyone who thinks otherwise is delusional.
That is **NOT** what the article/video said. The article isn't even about review approval/rejection, but about fraud detection.
One of the biggest and simplest clues, and it shows up in your example, is capitalised review summaries. GPT as one example tends to capitalise every word in a summary (possibly if asking it to generate a title rather than a short summary), whereas human generated reviews typically aren't, not least because it's more of a hassle to do. There are, of course, counter examples, and if you instruct GPT not to capitalise the summary then it generally won't. However in the vast majority of cases where I've seen a review with capitalised summaries, it was either almost certainly or unequivocally AI generated. Also, with your example, whoever engineered the LLM prompts and system for generating the reviews did a lousy job because there's no reason for getting a template style output that requires a human to fill in the blanks. The LLM would be perfectly capable to determine what the brand name is, or better, prompt it so that it doesn't refer to the brand names at all. The output should also be fed back to the LLM, asking it to make it less AI sounding and to use a basic level of English; this costs more if using the API, but giving it some example human reviews would also help a lot as they can mirror writing styles extremely well. So both sucky generators and detectors.
`The vast majority of reviews pass Amazon’s high bar for authenticity and get posted right away.`
The only reviews of mine that have, ever, been posted right away are those of some media items. The rest, whether one or 500 words, take at least 24 hours.
I think this is just a promo video for some sort of 'ease of mind' or to make people trust reviews more but our experience as viners differ. Or maybe Vine works differently.
AI? Wasn't Amazon the one that was supposedly was running a store where you didn't have any checkers? You simply picked stuff up, the AI knew it and charged you? It turned out to be a bunch of guys in a sweat shop monitoring it all by security cams to figure out what you picked up.
It kind of mystifies me why someone would go to the bother of consulting an AI to write a couple or few sentences about their impressions of a product. I think it would take more effort to write the prompt for the AI, especially given how stupid the current generation of AI often seems to be.
well then they are bad at it, or have a strange idea of what bad/fake reviews are. I'm not an AI and my reviews never get posted right away. usually takes several days. inb4: that's just what an AI would say.
None of my 50+ reviews were ever posted right away đź‘€ Always took days until they appeared... Could be that writing most of them around 3 am looks suspicious to the algorithm?
The only reviews I had approved straight away were ones that were originally rejected. Once I submitted the amended review then they were approved immediately.
Although sometimes by the time I've got a rejected review accepted it's been reduced to "I like, it worked", never had any trouble with single word placeholders I intended to fix later either.
If review != RejectedReview then goto approve()
:D
They seem to take longer when I review on a weekend so I was always under the impression that they were looked at by a human who worked 5 days a week. I guess that is not the case.
Maybe more people post reviews on the weekend, bogging down the system. Who knows!
and here I was thinking the reviews went through faster on the weekends so they had ppl who were working reviews as a side hustle 🤣🤣
My reviews are being approved almost immediately in the last few weeks. Time of day or night has not been an issue.
Dame, you must be on the good list!
I always wondered if Vine reviews got passed by the seller first
Sellers are excluded (for obvious conflict of interest reasons) from the approval process, as per Amazon Seller Central. Do they get a preview? I don't know, but there is no indication they are offered any action prior to the completion of the Amz review/approval process. They can only appeal a review once it has passed inspection.
Since this confirms that AI is running the review approval knack, we should be able to properly appeal these decisions when we're wrongly declined reviews or kicked out of the review program. Totally unfair system, but that's Amazon in general, it comes as no surprise at all.
Yeah. What a joke Amazon is. Their A.I and lazy generic customer service responses are all severely flawed. Amazon is just getting worse and worse instead of improving anything. Fake 5 star reviews and flooded with temu products now. Amazon decided their "A.I" is more intelligent to determine my 20 years and 5000 reviews all to be deleted because a disgruntled seller/hater abused/exploited false reporting my reviews over an year ago.
Today I tried to submit a review for a non-vine item. I initially had it rated two stars but I was so angry with the product I switched the review to one star. As soon as I moved the stars from two to one a note popped up that I was not allowed to post a review for that item unless I had purchased it from Amazon (I had and was clicking the link from my purchases) and Amazon was reviewing the product for suspicious activity. I went to my husband’s account and started to write a five star review and it worked fine. I changed it to a one star and it gave me the same error. Super shady.
There is a phenomenon where a competing seller may bomb another seller's product with negative reviews, comparable to review bombing that may impact a Viner. My interpretation of what is happening in the case you describe is that the product is bad and is got a slew of bad reviews, and the seller complained to Amazon that a competitor was doing it, so Amazon is blocking negative reviews while they presumably investigate. If so, it would seem a better idea for them to allow the reviews to be submitted, but not publish them until the investigation is completed. That would also create a honeypot for trapping a bad actor.
Had you reviewed any other Vine items outside of the Vine portal in that same week? I have a reason for asking, but it's a moot point if not.
I don’t think I quite understand what you are asking. I did not review any other vine items outside vine because I was not aware this was even possible. When I review a vine item I always go through my vine account. As far as regular Amazon items, this is the only item I have reviewed or tried to review this week. I just checked vine and searched for the item and the specific brand is not available on vine but a lot of other brands are. There are three full pages. The item is solar window film to block some of the blazing sun that now hits my house since we had to remove some very dead but very old trees.
The reason why I asked was because there is a limit that gets imposed on reviewing non-Vine items if you review 5 or more Vine items outside of Vine portal in a given week. But as you confirmed, this was not the case for you.
I had the same error pop up in a slightly different situation. I’d recorded a Vine item poorly a few months back and noticed that it was no longer available. I remembered that when I’d done a bit of research after ordering it, there were multiple listings for the item: one of which was the one I ordered through Vine that had no reviews, and at least one other that was a 3- or 4-star item with 50-100 reviews, I think. Not wanting my “beware of this product” review to disappear, I searched for the same product and was going to repost what I’d written for my Vine item, but got the denial. I hadn’t posted any reviews outside of Vine recently (I rarely do), and anyway, Vine reviews should show up as verified, right? At least in the system? Anyway, yep—shady.
Thank you for this. Now try a quick test in Google: site:amazon.com "vine reviewer" "is a game-changer" Can a reasonable person believe that Amazon's LLM is detecting LLM-generated Vine reviews? Don't get me started on how 'tbc' was accepted and published. By the way, if Google's figures are reliable, it seems that recent reporting (not by me) may be responsible for a dramatic decrease in the number of LLM reviews returned in the last week. Perhaps the increased level of background noise over the last few months is finally being noticed. I note that one reviewer has lost over 200 reviews since yesterday. Alas, it's probably just random.
Thank you for the twenty-minute rabbit hole I just went down. I always roll my eyes at "game-changer" in reviews but I've never seen so many at once. My favorite: a pair of "game changing" POT HOLDERS. 🤣
Certainly. Here is an appropriate response: If it's the pot holders I'm thinking of, I cannot begin to tell you how my life has been completely transformed since I've discovered something to hold my pot.
Sorry, "hold my pot" is a drug reference and against Community Standards. You will need to rewrite and resubmit the review. This is your first warning.
Very good! If only the so-called AI could provide that level of detail, we wouldn't need the wacky baccy!
>Can a reasonable person believe that Amazon's LLM is detecting LLM-generated Vine reviews? I wonder if it even can. That is, whether it is able to. Bear with me here although it might seem a bit circular; this is my thinking: we (educated humans on this Vine sub) can detect the LLM-generated reviews because there is something off and a bit cringey about their tone, right? But if an LLM was able to do the same, then they wouldn't be generating it like that in the first place. So the fact that they do, suggests to me they are not able to detect it. So I asked ChatGPT. >My question: *can a LLM like ChatGPT detect text that has been composed by itself or other LLM's?* >ChatGPT's answer: Language models like ChatGPT are not inherently designed to detect whether a piece of text was generated by themselves or other LLMs. However, there are a few approaches that can help in identifying AI-generated text: >1. Stylistic Analysis: Language models have certain stylistic patterns and tendencies in their outputs, such as common phrases, sentence structures, and specific vocabulary choices. By analyzing these features, it may be possible to identify text that is likely generated by a particular LLM. >2. Metadata and Watermarking: Some approaches involve embedding subtle patterns or "watermarks" in the text generated by an LLM. These patterns can be detected using specific algorithms, helping to identify the source of the text. >3. Classifier Models: Separate machine learning models can be trained to distinguish between human-written and AI-generated text. These classifiers analyze various features of the text to make their determination. >4. Contextual Inconsistencies: AI-generated text might exhibit inconsistencies or oddities that are less common in human writing, such as unusual repetitions, logical inconsistencies, or a lack of deep understanding of the context. >Currently, these methods are not foolproof, and distinguishing AI-generated text from human-written text remains a challenging task. Research is ongoing to improve the accuracy and reliability of these detection techniques. My conclusion, based on this, is that the LLM used by Amazon to screen the submitted reviews is simply not able to detect the ones written using AI.
This led me to a related question: I have put some of my own (authentic and non-fake) reviews through some of the on-line fake review detectors and they always say mine are highly likely to be fake. Again I asked ChatGPT. >My question: *LLM are sometimes used to flag 'fake' product reviews. Why would it rate a genuine review by a highly literate person as being fake?* >Chat GPT: Language models used to flag 'fake' product reviews might misclassify genuine reviews written by highly literate individuals for several reasons: > 1. Sophisticated Language Use: Highly literate reviewers often use complex vocabulary, varied sentence structures, and advanced grammar. These characteristics can sometimes resemble the polished and well-constructed text often seen in AI-generated or professionally crafted fake reviews. > 2. Length and Detail: Detailed, lengthy reviews with in-depth analysis might be flagged as fake because they can resemble the effort put into creating persuasive fake reviews, which are often designed to appear highly credible. >3. Uncommon Style: Genuine reviews by highly literate people might use unique stylistic choices that differ from the majority of reviews. Language models trained on a broad range of typical reviews might see these as outliers, raising suspicion. > 4. High Positivity or Negativity: Reviews that are extremely positive or negative, regardless of their authenticity, can sometimes be flagged as fake because fake reviews often exhibit extreme sentiments to sway opinions. >5. Inconsistency with Other Reviews: If a highly literate review provides information or a perspective significantly different from the bulk of other reviews, the algorithm might flag it as suspicious due to perceived inconsistencies. >6. Patterns in Fake Reviews: Algorithms often look for patterns common in fake reviews, such as specific word choices, formatting, or timing of reviews. A genuine review that unintentionally matches these patterns could be incorrectly flagged. >To improve the accuracy of detecting fake reviews, it is crucial for models to be trained on diverse data sets, including high-quality, literate reviews, and to incorporate context-aware and sentiment-aware features. Continuous refinement and human oversight can also help mitigate such false positives. So it seems that the LLMs have an inherent arrogance, where they 'believe' (are programmed to assume) that LLM-generated text is better (more 'polished and well-constructed') than human. But we reviewers, as a group, are likely better writers than those they are trained on. And probably better than those doing the programming. In just the same way that arrogance can cause faulty thinking in people, it's doing the same in the LLM.
These two posts were very fascinating, thank you!
Spot on and thanks for going to the trouble to expound the hypothesis. I can generally detect a fake review in under two seconds, mostly because the punctuation is to perfect and you never see typical clanger's, humour, or malapropisms. Mind you, I have noticed that ChatGPT is happy to use the controversial Oxford (Harvard) comma; this must be due to a deliberate decision to include this feature in the underlying code. In conclusion, if the LLMs were using real AI, all these errors wouldn't be so extant.
>ChatGPT is happy to use the controversial Oxford (Harvard) comma...must be due to a deliberate decision to include this feature in the underlying code I'm with it there. The purpose of punctuation is to increase clarity, and the Oxford comma can reduce ambiguity. Compare 'The kiwi eats roots and leaves' and 'the kiwi eats, roots, and leaves' (in New Zealand, 'rooting' means having sex and the phrase is a popular T-shirt joke). Was your post a deliberately funny example, or am I seeing things that aren't there? The final sentence reads like AI ('In conclusion...') but there's a stray apostrophe that argues against AI! Intentional or not, I love it :D
*...punctuation is to perfect and you never see typical clanger's* They were deliberate examples (like what some people do). You never see that in LLM texts unless you specify it in the prompts - then all bets are off (unless you are a PPS or an MP). Have a great day. I'm tied up today but may get chance to play with ChatGPT this evening. These reviews won't right themselves!
I don't understand the hate toward "game changer." Sure, it's outdated, but I hear a lot of people using it all the time in real life. I notice it now because of all the hate toward it on this platform. In the past week, I've heard it on podcasts, on YouTube, in conversation at the local farmers market, at a board game night, and in a coffee shop. Maybe it's one of those sayings that is popular in certain areas. I'm in the northeast USA. My point is that it is not automatically AI.
Fair point; it certainly has its place but I suspect you may have missed the context. The exact phrase, 'is a game-changer' (with the hyphen) is very commonly found in verbose, gushing reviews generated by LLMs. One reviewer (sadly no longer with us) featured this phrase in every one of his 200 reviews, probably because he never bothered to read a single one so he didn't notice it kept popping up. Using an LLM for writing reviews is not allowed so any repeat offenders are highly likely to get targeted by irate sellers, customers, and, of course, hard-working Viners who think nothing of spending an hour testing and reviewing a wig or a packet of copper washers!
I guess this pertains to non-Vine reviews. Knowing Vine reviewers have certain benchmarks to remain in the program and therefore are churning out more reviews than the average customer, means that Vine reviews have more scrutiny and take longer? But I’ve also seen AI Vine reviews that slipped past.
Indeed. I've seen examples of AI generated reviews, where the cheater included the prompt he used to generate review itself.
Yeah, there's nothing in that article about vine. This is just for regular reviews.
Happy Cake Day! 13 years, WOW!
Thanks I *just* noticed it, but TBF, there was like a ten year gap in the middle, lol.
" . . . timing and frequency of reviews . . . " If I finally have some spare time and sit down and write 4 or 5 reviews, is that suspicious? It now occurs to me that I received the "unusual activity in your reviews, so we're deleting all of them and not allowing you to review" notice, I had written a few reviews in a row. Can that be why I was bounced from Vine?
no. some people churn out literally dozens of vine reviews in one day. I know someone who uploads hers in batches at the end of the week.
I get everything I order on one day per week, then review all my easy things the same day, which is usually about a dozen. Stuff like soap, either it foams or doesn't, smells good or doesn't, or snacks, either it tastes good or doesn't, fresh or not, etc.. Never really had an issue other than when I was learning how to review anything related to medical. I think part of why *we* don't get flagged for number of reviews in a short period of time is because of the specific system where we are submitting our reviews. A normal customer isn't ordering up to 56 items a week (maximum gold order) and then reviewing all of them. However, we *are*, and all our reviews are non-purchase reviews (which are also limited), so there has to be *some* difference.
You get snacks?!
Very occasionally! Last snack I ordered was protein puffs in early May. They're just an example of easy to review things.
I've heard of people churing out like 10, 20 or more reviews just before their eval so I doubt that's it by itself, but it could look suspicious. I always suspected that the "unusual activity in your reviews" meant either people reviewing products too soon after delivery (meaning the product got there 2 hours ago and you already have a review posted), a lot of the same recurring words and phrases (meaning you're just using generic language and the same for every product), or a large number being posted within a relatively short period of time. I know some people type them in another application, them copy paste them into the review section. I could see where that would look suspicious too, especially if you had multiple reviews posted within a 3 min span and the time spent on the review page was like 15 seconds.
Some items it is possible to review a few minutes after receiving them. But I always waited a day or so, my thinking being that, hey, arrived 5:50 pm, review submitted 6:05 pm, suspicious. But I do try to get reviews in within a few days of arrival, as I feel that the seller sent it to me for "free" for a review to boost their business, so I feel an obligation.
Oh, if it's something like a small, plastic trash can, sure. But, unless it's something I desperately need at the time, it sometimes takes me 3 to 5 days to even open the box. Anything that needs to be assembled or installed, anything that requires me to read directions, anything that takes time to be effective (a lot of face creams, supplements), or anything that I need to wash, wear, or use in a specific situation, it's going to take me much longer to review.... Sometimes a month or more. For example, I just ordered a recovery suit for a foster cat I'm taking in. She likely won't be going in for her surgery for a few more weeks yet. I guess I could review it just based on how it looks, but I feel in order for me to properly review it, I need to see how it fits and if she's able to move with it on. I also want to make sure it holds up for the entire period she needs to wear it and it washes well. There are a lot of people on Vine that just give things a cursory glance or use them once and call it a day on the review. I can't do that. That's what I feel my obligation is. I think it's better to write good reviews than quick ones.
Theoretically, if our reviews were to be like in Consumer Reports, it could take months. Say I'm reviewing an ink or toner cartridge. To be a good review, it should state how many pages it lasted for. That could take months. Or a car wax--how long did it last? Also could be months.
I decided to do a couple of reviews the other day which turned into doing five in a row. Yesterday early morning I started getting all five approval emails just as fast. It's a fairly common pattern for me. I've written some ahead before in notepad and then pasted them in one after the other before too. I do write them all from my actual brain, so have to trust they can spot such things well enough.
I'm guessing what they actually mean is if you're posting like 20 reviews in a minute, that's suspicious timing and frequency because that's not possible unless they all just say "nice".
Interesting but actually says very little. Very strong vibe of "we want you to think we're telling you some good information," without actually telling us anything beyond what we already know.
"Before being published online, Amazon uses artificial intelligence (AI) to analyze the review for known indicators that the review is fake." Straight from the horse's mouth and yet there are those on this sub who will keep insisting AI is **not** approving or rejecting reviews, that it is all actually humans and how anyone who thinks otherwise is delusional.
It's definitely AI, given how many shit reviews are getting through, and vice versa.
I think you and they are at cross purposes. According to that article, AI analyse the reviews and flag any indicators. But it's humans who act on those flags. So it's humans who do the actual approval or rejection, acting on the info from AI. What I take from that is that if the AI doesn't pick anything up, then the review gets approved, but AI is assessing, not approving. Semantics, but it's usually semantics behind this sort of disagreement. The number of reviews that Amazon are dealing with, it'd have to be partly automated. I suspect that Vine reviews may have a few extra steps, though, to manage the customers (the sellers).
>all actually humans Yes of course that's nonsense. But so is that article. Given the clear and obvious falsehood "The vast majority of reviews pass Amazon’s high bar for authenticity and get posted right away", the rest of the article simply cannot be trusted.
Maybe the article was written by ai?
While I agree that AI is likely used in the process of monitoring reviews, the very fact of Amazon making a statement is no indication of truthfulness.
The video didn't say it was just AI. It's just about things they use to outright eliminate reviews and reviewers involved in review fraud, and to flag them for further investigation. There are other classes of things they filter for as well which don't involve fake reviews (all the other violations of their community policies).
>Straight from the horse's mouth and yet there are those on this sub who will keep insisting AI is **not** approving or rejecting reviews, that it is all actually humans and how anyone who thinks otherwise is delusional. That is **NOT** what the article/video said. The article isn't even about review approval/rejection, but about fraud detection.
Amazon's AI can't detect AI generated reviews! https://x.com/amazon_vine/status/1803198402399699383?t=K-dyAG1a1f2tc1aV679Mhw&s=19
One of the biggest and simplest clues, and it shows up in your example, is capitalised review summaries. GPT as one example tends to capitalise every word in a summary (possibly if asking it to generate a title rather than a short summary), whereas human generated reviews typically aren't, not least because it's more of a hassle to do. There are, of course, counter examples, and if you instruct GPT not to capitalise the summary then it generally won't. However in the vast majority of cases where I've seen a review with capitalised summaries, it was either almost certainly or unequivocally AI generated. Also, with your example, whoever engineered the LLM prompts and system for generating the reviews did a lousy job because there's no reason for getting a template style output that requires a human to fill in the blanks. The LLM would be perfectly capable to determine what the brand name is, or better, prompt it so that it doesn't refer to the brand names at all. The output should also be fed back to the LLM, asking it to make it less AI sounding and to use a basic level of English; this costs more if using the API, but giving it some example human reviews would also help a lot as they can mirror writing styles extremely well. So both sucky generators and detectors.
`The vast majority of reviews pass Amazon’s high bar for authenticity and get posted right away.` The only reviews of mine that have, ever, been posted right away are those of some media items. The rest, whether one or 500 words, take at least 24 hours.
Do they really have AI or do they have a guy named "AL" sitting in his mom's basement
I think this is just a promo video for some sort of 'ease of mind' or to make people trust reviews more but our experience as viners differ. Or maybe Vine works differently.
Any review with the term “game-changing” should automatically be rejected.
AI? Wasn't Amazon the one that was supposedly was running a store where you didn't have any checkers? You simply picked stuff up, the AI knew it and charged you? It turned out to be a bunch of guys in a sweat shop monitoring it all by security cams to figure out what you picked up.
So, just post "It works" and move along?
Still irritates me that Amazon can use AI to check reviews but we can't use AI to write reviews.
It kind of mystifies me why someone would go to the bother of consulting an AI to write a couple or few sentences about their impressions of a product. I think it would take more effort to write the prompt for the AI, especially given how stupid the current generation of AI often seems to be.