Jev: Revolutionary or Just Hype?

(This article was entirely written by hand, by a human, without any AI assistance.)

Just a couple of days ago, a company called Typesafe announced a new kind of model: Jev. They claim it's "a new class of frontier models built to make fast, structured decisions that software can use directly". And it's getting a lot of attention on social media.

Yesterday I managed to get access to the model and decided to play with it a little. Here are my thoughts.

By the way, if you want to jump straight to the example code, click here.

Jev is a general-purpose classifier. It takes natural language as input, but returns a discrete output: either one of a list of predefined categories, a boolean yes/no value, or a "score": a number on a scale. The big difference with LLMs here is the return value. Instead of text, it returns strongly typed results which are really easy to integrate with software. We'll go over a code example below.

So you can give it your emails and have it decide whether they are spam. Or give it some natural language input, like chat input from a customer, and categorize it to see which action to take. It's an easy drop-in for any project where you need to make a decision based on some text.

But it's still a general-purpose model, that is not fine-tuned to your specific problem, and it's very slow (compared to a local custom classifier) and quite wasteful. In those regards it's just like LLMs. Let me explain.

There is one real advantage that Jev has over custom classifiers: you don't have to train it, it's already there. So for quick demos, or one-off tasks where you don't care too much about accuracy, Jev is really cool. It's super easy to integrate as well. So I think we will see Jev used a lot for vibe-coded apps, because it's great if you don't want to create your own classifier or don't have the skills.

Let's say you have a relatively simple problem, like spam-detection for your email. This is a yes/no classification problem. You decide to use Jev.

This means you now have to send your sensitive emails, over the network, to a huge general-purpose neural network with billions of parameters. You will use thousands of tokens (which you will pay for) per email, wait a bit for the answer to return, and finally get a yes/no answer.

Instead you could run something like SpamAssassin locally. That software is fine-tuned for the problem at hand and will run thousands of times faster. You will not have to hand your data to a company you hardly know. You will use millions, if not hundreds of millions of times less compute (not an exaggeration). And it's free.

Jev response times I measured for my simple example task.

And that is true for just about any classification problem: a fine-tuned model will generally be better, faster, cheaper and safer than some general-purpose, third-party neural network. And even for complex problems, creating your own classifier (e.g. BERT) is not that hard anymore. So for any business that needs to run a classifier at scale, I don't think Jev will be that attractive.

Here's another example: I see people online talking about how to use Jev in gaming. Maybe you can use it to make decisions for the AI that you are playing against. But each call to Jev costs money and time. Who is going to pay for that? If you have a game that has lots of users you'd be much better off sticking to well-established AI algorithms that run inside the game.

That brings me to another thing that Jev has in common with LLMs. I think there's an inverse relation between how competent someone is at something and how impressed they are by LLMs doing that thing. I think the same is true for Jev. Yes, it's cool, and its fun to use. But is it revolutionary?

In my view, when you use Jev - or an LLM - you waste compute, energy, time, and money for something that will always be second-best (or worse) at whatever task you give it.

There's also no guarantee that the model performance will not change over time (except the "trust me bro" style claims from Typesafe). It's a black box - there's no way of knowing when they will change, retrain or even downgrade the model.

Let me go over my little test project. In the past I have used beancount for text-mode accounting, and I wrote my own classifier to decide how to file each transaction in the CSV files that I download from my bank.

This is a relatively simple task and I just used regular expressions for it, but I decided to try Jev for this. Of course, if you try this kind of thing for yourself, make sure to properly anonymize your data before sending it to a third party.

The cool thing about Jev is that the code to use it is really simple:

  async def classify_transaction(row: dict[str, str]) -> str:
        response = await client.system_one(
            state={"transaction": json.dumps(row)},
            questions={
                "expense_type": Choice(
                    instructions="Which kind of expense does this transaction represent?",
                    criteria={
                    "Groceries": "Supermarkets: Albert Heijn, (and here I fill in the name of other shops that I shop at)",
                    "Energy": "Energy bills from Vattenfall",
                    "Insurance": "(name of my insurance company here)",
                    ...
                    },
                ),
            },
        )
        return response.answers["expense_type"]

We send the data to classify as the state argument, and the questions to answer about that data as questions. In this case I send the categories as criteria, with some extra context data to improve matching. We tell Jev that the type of question is Choice, which means that it will have to choose one of the categories Groceries, Energy, ... as a return value.

I loop over a CSV file and run this classifier for each line asynchronously:

with csv_file.open(newline="", encoding="utf-8") as in_file:
    reader = csv.DictReader(in_file)
    async with AsyncTypeSafeClient() as client:
        results = await asyncio.gather(
            *(
                classify_transaction(row) for row in reader
            )
        )

We can then store the results and compare them with the correct answers from my custom classifier. This way I got pretty good results for a quick test, with minimal effort:

Accuracy: 0.8023
F1 (macro): 0.8531
F1 (weighted): 0.8238

You could probably boost these results by giving Jev even better inputs. So for these tasks, when you are able to give clear instructions about your domain, Jev can perform quite well. Here are some other statistics from my script:

jev_time stats: mean=0.3518s min=0.2848s max=1.4200s stddev=0.0855s
jev_tokens_in stats: total=4401735.0000 mean=1843.2726 min=1835.0000 max=1857.0000 stddev=4.6193
jev_tokens_out stats: total=855218.0000 mean=358.1315 min=356.0000 max=363.0000 stddev=1.9230
jev_tokens_total stats: total=5256953.0000 mean=2201.4041 min=2192.0000 max=2219.0000 stddev=5.2502

We are using millions of tokens and we have to wait at least 280 ms for every call. To compare: my own custom classifier runs so fast I cannot really measure how long it takes per row, and it costs nothing. And it is almost 100% accurate and deterministic, because it's just regular expressions and if statements.

You can have Jev answer yes/no questions as well. In that case, the question type is Noul, and it returns a value between 0 and 1. The documentation reads: "The number is the answer and the certainty in one. A value near 1 is a strong yes. A value near 0 is a strong no. A value near 0.5 means the model gives yes and no similar probability."

We can use this, for example, to determine whether we should water our plants in the garden based on a weather forecast:

mport requests
from dotenv import load_dotenv
from typesafe_sdk import TypeSafeClient, Noul

load_dotenv() # JEV Api key in .env

def jev_sprinklers() -> None:
    params = {
        "latitude": 52.3,
        "longitude": 4.9,
        # Let's get the forecasst for temp and rain in the next 3 days
        "daily": ["precipitation_sum", "temperature_2m_max", "temperature_2m_min"],
        "forecast_days": 3,
    }
    weather_resp = requests.get("https://api.open-meteo.com/v1/forecast", params=params)
    weather_resp.raise_for_status()
    weather_forecast = weather_resp.json()

    client = TypeSafeClient()
    response = client.system_one(
        state={"forecast": f"""Forecast for next 3 days: 
                Total precipitation in mm": {weather_forecast["daily"]["precipitation_sum"]},
                Max temp (C)": {weather_forecast["daily"]["temperature_2m_max"]},
                Min temp (C)": {weather_forecast["daily"]["temperature_2m_min"]},
                """},
        questions={
            "sprinklers": Noul(
                instructions="Should I turn the garder sprinklers on?",
            ),
        },
    )

    value = response.answers["sprinklers"].noul
    if value > 0.5:
        print("Sprinklers on")
    else:
        print("Sprinklers off")

Again, this code is really simple. We probably call this once a day so latency and price are not really an issue here. But we don't really know how well Jev performs on this specific task. So if you really care about your plants, I'd still go for a proven solution like pyfao56.

So yeah, I think that supports my view on Jev: it's cool to play with, but for many real-world business scenarios I think it doesn't make much sense. And if you take ethics, privacy and the environment into account, it makes no sense at all.

Just create your own classifier, it's fun and you might learn something!

Reindert-Jan Ekker