How we read comments

Which movies we score

Only full movies. We judge each upload by its length and its title: one to four hours long with “full movie” in the title counts strongly in favour; shorts, livestreams, skits, reviews and reaction videos are left out. Uploads we aren’t sure about wait for a person to check them before they appear.

Which comments we read

• Each time we check a movie we fetch its most relevant comments and its newest ones. We keep them, so the pile grows each visit.
• To score a movie we read up to 300 of its most-liked comments plus its 50 newest, so a new release’s first comments count before they collect likes.
• We check again roughly a day, a week and a month after a movie is published.
• We read top-level comments only. Replies aren’t read, though how many replies a comment gets counts towards engagement.
• A movie needs at least 30 of those comments before we score it.

How WatchScore is worked out

WatchScore runs from 0 to 100. It blends up to six parts. These are the default weights:

• Member ratings count once a movie has at least 5; thumbs once it has at least 10. Parts a movie doesn’t have yet are left out and the rest are re-weighted.
• If our editors rate a movie themselves, their rating replaces the worked-out score.

How we tell if a comment is positive

A sentiment model reads each comment and labels it loved it, mixed or didn’t like it. Every comment counts the same, however many likes it has. Comment mood in the score is the share that loved it minus the share that didn’t.

If the model fails, we use an estimate from the AI model described below. If that isn’t available either, we fall back to looking for positive and negative words and emoji. That word list is mostly English, so it often reads Pidgin, Yoruba, Igbo and Hausa comments as mixed.

We can also show that split on a movie as “% loved it”, once at least 30 of its comments have been labelled. For now it’s switched off. When we checked the model against comments labelled by hand, it agreed less than 80% of the time, and it was weakest on Pidgin. We won’t show the figure until it passes. The score uses the split either way.

What people liked and complained about

We group comments that say nearly the same thing. An AI model reads the biggest groups and names what people liked and complained about, with a short summary of each. Each theme shows how many of the comments we read are behind it, where we can tell. Complaints count against the score a little.

Text in speech marks is a real comment from one of those groups, never one the AI wrote. We remove links and @handles, and we never quote a comment the sentiment model read the other way, so a complaint is never shown as praise, or praise as a complaint.

Spam and bought engagement

We look for sudden bursts of comments, the same text posted many times, comments coming from only a few accounts, and links or “subscribe” spam. These set how far to trust a score, shown on each movie page. They don’t change the score itself, but movies with almost no trust are kept out of top-rated lists.