Jev: first steps

Disclaimer: With this article I want to try out and show what TypeSafe Jev can do. It is not a scientific evaluation and not a statement about public opinion. The comments are not representative of the voting population.

As a basis, I collected 8640 reader comments written under articles about the food initiative on Swiss news portals. I used them to test Jev's three basic building blocks (primitives) one after the other. Each step builds on the previous one:

1. Noul: Is the comment relevant at all?
2. Choice: Is the comment for or against the initiative?
3. Score: How well-reasoned is the comment, and how strong is its stance?

Step 1: Is the comment relevant?

Noul is the simplest primitive: a yes/no question. But Jev doesn't just answer yes or no. It answers with a probability between 0 (definitely not) and 1 (definitely yes).

This first filter alone already removes quite a lot of comments that are no longer of interest for further analysis, e.g. comments that only agree with the parent comment.

QUESTION = Noul(
    instructions=(
        "Does the comment address the Swiss food initiative ('Ernährungsinitiative' / "
        "'Initiative für sichere Ernährung') that is voted on in Switzerland on 27 September?"
    ),
    criteria=NoulCriteria(
        true=(
            "The comment refers to the initiative, the vote on it or its content: fewer animal "
            "products, more plant-based food, food self-sufficiency, agriculture, 'forced veganism', "
            "being patronised about what to eat. Also when the initiative is not named explicitly. "
            "Also statements like 'reject both proposals'."
        ),
        false=(
            "The comment is only about the neutrality initiative, about parties or politicians "
            "without any link to food, is a quarrel between commenters, or is about another topic."
        ),
    ),
)

In my own code, I split the probability that Jev returns into three groups: from 0.8 yes, up to 0.2 no, in between unsure. You set these thresholds yourself; Jev only provides the probability. This of course already lets you build interesting branches in the code: whether something can keep being handled automatically, or whether someone needs to be notified.

p = response.answers["about_initiative"].noul
label = "yes" if p >= 0.8 else "no" if p <= 0.2 else "unsure"

The result for all 8640 comments:

8640

medium_A    yes:  703  unsure: 1353  no: 1512
medium_B    yes: 1323  unsure: 1223  no:  415
medium_C    yes: 1047  unsure:  710  no:  354

condition         calls   time/call   cost/call  tokens/call
typesafe_noul      8640       262ms   $0.000024          574

Total:    4,963,670 tokens, $0.2085, 142.7s (60.6 calls/s at concurrency 16)
Latency:  median 258 ms, p95 304 ms, max 822 ms

What stands out first: how fast and cheap the whole thing is. 8640 comments for 21 US cents. So news portals could pre-sort their comments for practically no money.

For this analysis, I only continued working with the 3073 yes comments.

In a second run, I also wanted to know which arguments appear in the comments. For this, I defined seven more Noul questions, one per argument. All eight questions go to Jev in the same request, so it's still one call per comment.

ARGUMENTS = {
    "paternalism": "the state dictating what people eat, 'forced veganism', loss of freedom of choice",
    "prices": "food prices, cost of living or what people can afford",
    "self_sufficiency": "food security, the self-sufficiency rate, imports of food or animal feed, "
                        "resilience in a crisis",
    "farmers": "farmers' incomes or livelihoods, livestock farming, grassland that is only usable for animals",
    "environment": "climate, biodiversity, soil, water, pesticides or land use",
    "animal_welfare": "how animals are kept or treated",
    "health": "whether a diet with less meat or more plants is healthy",
}

ARGUMENT_QUESTIONS = {
    f"arg_{name}": Noul(
        instructions=(
            f"Does the comment use or discuss this argument about the Swiss food initiative: {topic}? "
            "It does not matter whether the commenter is for or against the initiative."
        ),
        criteria=NoulCriteria(
            true=f"The comment itself makes a point about {topic}, also when rebutting someone else's point.",
            false="The comment does not touch this topic, or only mentions it in passing without making a point.",
        ),
    )
    for name, topic in ARGUMENTS.items()
}

8640 comments -> results/relevance_all.csv

medium_A   yes:  711  unsure: 1362  no: 1495
medium_B   yes: 1340  unsure: 1206  no:  415
medium_C   yes: 1035  unsure:  714  no:  362

condition           calls   time/call   cost/call  tokens/call
typesafe_noul        8640       272ms   $0.000059         1401

Total:    12,108,950 tokens, $0.5086, 147.3s (58.6 calls/s at concurrency 16)
Latency:  median 262 ms, p95 344 ms, max 1078 ms

Arguments in relevant comments (share and count per portal, p >= 0.5):
                            medium_A        medium_B        medium_C
  paternalism               43%  308        28%  370        26%  265
  prices                    10%   74        14%  190        13%  135
  self_sufficiency          38%  273        30%  397        38%  397
  farmers                   16%  117        28%  381        23%  235
  environment               25%  178        37%  491        31%  321
  animal_welfare             6%   44         9%  118         8%   80
  health                     7%   53        12%  156         6%   61
  no argument: 687 of 3086 relevant comments

This raises the cost from 21 to 51 US cents, because more tokens are processed per call (1401 instead of 574). The duration stays practically the same.

An excerpt from the result. jev_probability is the probability that Jev returns for the relevance question:

Step 2: For or against?

With the 3073 relevant comments, I moved on to the question: Is the comment for or against the initiative? That's what Choice is for. Instead of a yes/no question, you specify several options and Jev picks one of them. Here there are three: in_favour, against and neutral.

Each option is described using the pattern what, not_for and examples, which TypeSafe recommends as best practice in the docs:

• what: when the option applies
• not_for: what looks similar but doesn't belong. This is where the difficult cases are, e.g. someone thinks animal welfare is important but still rejects the initiative.
• examples: real comments as examples

QUESTION = Choice(
    instructions=(
        "What is the commenter's own stance on the Swiss food initiative ('Ernährungsinitiative' / "
        "'Initiative für sichere Ernährung') voted on in Switzerland on 27 September? "
        "Judge the commenter's position, not the article's. If the comment is a reply, use the "
        "replied-to comment only to understand what the commenter means: agreeing with an opponent "
        "of the initiative means being against it."
    ),
    criteria={
        "in_favour": {
            "what": "Supports the initiative, will vote yes, or defends it against criticism",
            "not_for": "Sympathy for its goals (animal welfare, less meat) while rejecting the initiative itself",
            "examples": [
                "Böden schützen, weniger mit Gift arbeiten, Biodiversität schützen. Ja bitte!",
                "Gerade deswegen ein pflanzlich-fettes JA!",
            ],
        },
        "against": {
            "what": "Rejects the initiative, will vote no, or attacks it (paternalism, costs, 'forced veganism', "
                    "state intervention, unrealistic)",
            "not_for": "Criticism of the no-campaign or of opponents' arguments",
            "examples": [
                "Ich lasse mich nicht gerne beim Essen auch noch bevormunden.",
                "Wer weniger Fleisch essen möchte, kann das ohne diese Initiative tun.",
            ],
        },
        "neutral": {
            "what": "No recognisable position: asks a question, shares facts or polls, weighs both sides "
                    "without deciding, or comments on the campaign itself",
            "not_for": "Sarcasm or irony that clearly implies a position",
            "examples": [
                "Was genau würde sich für die Bauern ändern?",
                "Laut Umfrage liegt das Nein-Lager vorne.",
            ],
        },
    },
)

What Jev returns is interesting. Not just the chosen option, but three things:

1. choice: the option with the highest probability
2. confidence: how clear this answer is
3. probabilities: a probability for each option, including the ones not chosen

You can keep working with the third point in the code. For example, you could take into account every option above 0.25. Or you could detect comments where in_favour and against are both high and have a human look at them.

This is what the result looks like per comment:

As with Noul, I decide in my own code when I trust the answer. If the confidence is below the threshold, the comment is marked as unsure.

stance = a.choice if a.confidence >= MIN_CONFIDENCE else "unsure"

TypeSafe writes about this in the docs:

One of TypeSafe's most powerful features is confidence. By being intentional with the way you gate decisions on confidence, you can build systems that are both reliable and safe.

Step 3: Rating the arguments

The third primitive is Score. Instead of options, you specify a scale, here from 0 to 4, and describe each level in words. I used it to rate two things, both in the same call:

• argument_quality: How well-reasoned is the comment? From "just an insult or a slogan" to "several concrete arguments, numbers or sources".
• stance_intensity: How strong is the stance? From "clearly against" through "no position" to "clearly in favour".

This would be interesting for news portals, for example: sorting comments not just by likes or date, but by how well-reasoned they are.

QUESTIONS = {
    "argument_quality": Score(
        instructions=(
            "How well-argued is the comment's contribution to the debate about the Swiss food initiative "
            "('Ernährungsinitiative')? Judge the reasoning, not whether you agree with it."
        ),
        criteria=[
            "Only an insult, slogan, emoji or joke without any content",
            "States an opinion without giving any reason",
            "Gives one general reason, e.g. paternalism, costs or health, without going into detail",
            "Makes a concrete argument about what the initiative would change, e.g. self-sufficiency, "
            "feed imports, food prices or farmers' incomes",
            "Makes several concrete arguments, cites numbers or sources, or engages with counter-arguments",
        ],
    ),
    "stance_intensity": Score(
        instructions=(
            "Where does the commenter stand on the Swiss food initiative ('Ernährungsinitiative')? "
            "Judge the commenter's own position, not the article's. If the comment is a reply, use the "
            "replied-to comment only to understand what the commenter means."
        ),
        criteria=[
            "Rejects the initiative outright, e.g. calls it extreme or dangerous or will definitely vote no",
            "Has doubts or criticises parts of the initiative, but is not firmly against it",
            "Takes no position, weighs both sides without deciding, or only asks or informs",
            "Is sympathetic to the initiative or defends parts of it, but has reservations",
            "Clearly supports the initiative, will definitely vote yes or defends it strongly",
        ],
    ),
}

For each question, Jev again returns a probability for each level, plus a confidence and a score. The score is the average of the levels, weighted by their probabilities. That's why there are also values like 2.92, and you can sort the comments finely instead of just putting them into five buckets.

The result for the 3073 relevant comments. The first table shows how many comments end up at which quality level (0 to 4):

3073 comments

medium_A     0:   23  1:   94  2:  251  3:  228  4:  107
medium_B     0:   17  1:  146  2:  453  3:  324  4:  383
medium_C     0:   22  1:  146  2:  292  3:  317  4:  270

condition           calls   time/call   cost/call  tokens/call
typesafe_2x_score    3073       266ms   $0.000032          768

Total:    2,360,239 tokens, $0.0991, 51.5s (59.7 calls/s at concurrency 16)
Latency:  median 260 ms, p95 313 ms, max 805 ms

Argument quality vs. upvotes (Spearman, per portal):
  medium_A     rho -0.07   mean quality 2.37   n=703
  medium_B     rho -0.12   mean quality 2.65   n=1323
  medium_C     rho -0.09   mean quality 2.59   n=1047

Stance intensity (0 = against, 4 = in favour) by stance.py label:
  against    mean 0.67   n=1340
  neutral    mean 1.91   n=24
  in_favour  mean 3.56   n=774
  unsure     mean 2.29   n=935
  agreement Choice vs. Score: 98% of in_favour/against comments are on the matching side of 2

Again fast and cheap: 3073 comments with two questions for 10 US cents.

An excerpt from the result:

The columns in detail:

Conclusion

My first impression: Jev is very easy to use. One question, a few criteria, and you get a clean, typed answer back. That makes getting started easy. All the runs in this article together cost around one US dollar.

What I'm taking away:

• Filter first, then ask. Use Noul to sort out the irrelevant comments and only keep working with the rest. This saves costs and makes the further results cleaner.
• The decision lives in your own code. Jev provides probabilities and confidence; you set the thresholds yourself. That's probably where the real work lies once it comes to a production use case: finding the right threshold.
• Not just an answer, but how certain it is. With a classic LLM call and a predefined JSON schema you do get a label, but no reliable indication of how certain it is. You can tell it to include a confidence, but I don't know how good that is. With Jev you can react to exactly that: process automatically when the answer is clear, and otherwise have a human take a look.
• Intelligence per Dollar is increasing. I haven’t measured it precisely, but classifying the same amount of data using a standard AI API call would cost about 100 times as much.

For developers this is a big step, and I'm curious to see what other use cases will come up.

If you'd like to see the code I used for this: Jev-playground