Ant Spirit announced the data from the launch of version 2.0 half a month ago: managed 3 million items, executed 11 million tasks (when it was launched for a week, these two numbers were 1.16 million and 3.6 million)
The metric of managed items is probably the most reasonable Personal Agent North Star metric at present.
But! This article does not talk about Ant Spirit (a diversion), but rather follows the metric of "how many items were managed" to discuss with everyone: What metrics should Personal Agents use?
As a background, over the past twenty years, we have always evaluated the quality of an internet product project by using a reference metric, such as: • In the portal website era, we looked at PV: each page opened means the advertisement is sold again, thus the screen was full of pop-ups (like the one next to this) • In the era of social media or mobile internet, we looked at daily active users • For video websites, we looked at total duration.
These metrics are referred to as North Star metrics.
But in the age of agents, or the current age of personal agents, what should the North Star metrics be? There is actually still no conclusion.
Where North Star Metrics Come From
The so-called North Star metrics are the most representative figures that reflect the progress of the project, typical North Star metrics include DAU, GMV, ARR, etc.
Our internet waves have come several rounds, and each new wave of product forms brings new metrics. For example, in the early days, how did we evaluate the greatness of a website? We used hit count, which was the number of interactions users had with the website.
For example, if a website has dozens of images, each time a user opens it, the server will receive dozens of requests. Therefore, in the first version of internet websites, it looked somewhat like this:
emmm... back then, someone even came up with a full name for HITS: How Idiots Track Success.
Later, everyone changed it to count pages, with each opening counting as one (PV), which lasted for several years until Ajax became popular (Gmail, Google Maps used it): the page did not need to refresh, new content would load automatically. If you stayed in Gmail for half an hour, PV would only be counted once.
In 2007, Nielsen simply changed to rank websites based on time spent. This company started with ratings, and the American advertisers invested money based on its rankings. The result? AOL, which ranked only sixth based on PV, suddenly became number one in America. After all, many Americans at that time received emails and chatted using AOL.
From hits to PV to duration, each change in metrics occurred because the old metric had grown, but advertisers’ money could not buy more user attention.
Metrics Determine Product Appearance
No product operates as an entire machine; it is composed of individuals and departments. Therefore, setting evaluation goals from the top down means you must optimize a certain value to a certain extent.
Thus, the project becomes something that the whole company strives to optimize for one or several numbers and further designs monetization around these numbers. These two forces influence each other, leading everyone to work hard to optimize it.
Taking portal websites as an example, they rely on selling ad placements to make money (how much for every thousand impressions), but considering that the number of ads on each page is limited, everyone at that time worked hard to increase the exposure of the same information, letting the same information cross platforms and appear on more pages.
Remembering websites from around 2005, was it not a bizarre scene like the one below?
But clearly, this method of harming user experience led more people to choose competing products as a footnote. Around ten years ago, an American gossip blog, Gawker, found that one of its posts achieved 14,308 PV, but only attracted 756 new readers. Thus, Gawker changed the editor's bonuses from being counted by page views to being calculated based on unique visitors (UV).
Now let's talk about social networks.
If we look back now, that generation of products was all about competing for people's attention, and the best representation of this was DAU.
When Facebook went public in 2012, its daily active users were 483 million. Focusing on this number for over a decade, by the end of 2023, daily active users had risen to 2.11 billion.
Video sites, on the other hand, focused on duration. In its early days, YouTube recommended videos based on the number of views, which led to a prevalence of clickbait titles. In 2012, YouTube changed to recommend videos based on watch time, setting a goal to increase daily watch time from 100 million hours to 1 billion hours, achieving this goal four years later.
At this point, you will find that the main revenue stream of this internet wave is advertising, which needs to sell user usage time to advertisers. Therefore, we have seen metrics like PV, DAU, duration, etc., to capture user attention, which also mutually determines the form of products: What metrics you choose will eventually determine what kind of product you get.
Personal Agent
In the era of Personal Agents, what metrics should we look at? First of all, what should be denied is user duration, because Personal Agents do not generate revenue by selling attention.
The money for Personal Agents comes from users paying to "get things done": either there is a personal version subscription fee (ChatGPT, Claude's personal membership starts at $20 per month), or enterprises have already started to charge based on results.
As for interactions and duration, those are just burning through tokens, so keep it minimal. The better Personal Agents handle tasks, the fewer times users need to open them.
Such data and intuition were previously encountered.
During World War II, American statisticians helped the military analyze the bullet holes on returning bombers: there were dense holes on the wings and fuselage, but very few on the engines. Intuitively, armor should be added where bullet holes are most prevalent. Statistician Abraham Wald, however, suggested adding armor in places with few bullet holes since planes that got shot in the engine would not make it back.
There’s also a well-known video website, commonly called P station (not directly stated here). There has been a saying in the community: in its early days, it judged the quality of videos based on how many people closed them halfway...
emmm... because back then, the users were already "satisfied."
If daily active users were still assessed at that time, the team would push more notifications, set more reminders, and come up with various ways to ensure you visited more often, which goes against "saving you the trouble."
On the flip side, a user who completed their tasks and never opened the app again is called a silent user in the daily active users report, but that user might have used it the best.
To start with a joke, here I want to introduce a well-known personal agent: WeChat.
Supports multi-modal, response speed depends on mood
Inside, there are many types of agents; some agents are very humorous, while others can solve various problems in your life, and all of these agents can recognize images and even send you voice messages.
Regarding Xiao Long from WeChat’s saying of "use it and leave," applied to Personal Agents, it should mean you don’t need to use it up; the person just leaves.
What Should Personal Agents Focus On
According to the reasoning above, people do not need to always stare at the personal agent's screen. If we really need to establish a North Star metric for Personal Agents, what should it be related to?
How many tasks did users hand over to it?
How many of those tasks were completed?
These two numbers also need to be linked to revenue. For example, with customer service software Intercom, its customer service agents directly charge based on results, solving a problem for a customer and charging 0.99 dollars (actually, I haven't found a good example, just used it for illustration).
The beginning of this article mentioned Ant Spirit, which is probably also heading in this direction, to compete for the custody rights users have, like letting it monitor whether the recently booked flight has dropped in price or bringing up those forgotten auto-renewal subscriptions... When conditions are appropriate, it will remind you, and in some cases, it will directly take action to handle it. The number of tasks you pass on is the custody volume.
Of course, in the future, we may see other metrics like these:
Custody Rate: how many users are in custody.
Completion Rate: how many submitted tasks were completed.
Retention Based on Tasks: after 30 days of custody of a task, is it still under the AI’s control?
Of course, what the future metrics will specifically look like can’t be predicted right now, even custody volume itself still has many standards waiting to be aligned, such as... does setting an alarm count as having managed a task? (Xiao Ai is ecstatic).
Similar to these Personal Agents, they are quite akin to autonomous driving. I once read material stating that companies conducting road tests in California need to report to the DMV the average distance driven before a human needs to take over. This could also be translated to Personal Agents, though it might not be called a business metric; it may be a product performance optimization metric.
Finally
There may be bold claims to come: The classic user profile for Personal Agents is high-frequency openings, but each usage lasts very short, with the backend tasked machines working around the clock (packed with various tasks).
免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到[email protected]，本平台相关工作人员将会进行核查。
