Article by: Adrian Chow
Contributions by Jonathan Yuen and Wintersoldier
Abstract:
Oracles are indispensable in safeguarding the locked value of DeFi protocols. Among the total locked value of 500 billion USD in DeFi, 330 billion USD is secured by oracles.
However, the inherent time delay in oracle price feed updates leads to the extraction of value in a specific type known as Maximal Extractable Value (MEV), which includes frontrunning, arbitrage, and inefficient liquidations. This is referred to as Oracle Extractable Value (OEV).
There are increasingly more design implementation solutions to prevent or mitigate the negative impact of OEV, each with its unique trade-offs. This article discusses the choices and trade-offs of existing designs, as well as presents two new concepts, their value propositions, pending issues, and development bottlenecks.
Introduction
Oracles can be said to be one of the most critical infrastructures in today's DeFi. They are an essential part of most DeFi protocols, relying on price feeds for settling derivative contracts, undercollateralized liquidations, and more. Currently, oracles secure a value of 330 billion USD, accounting for at least two-thirds of the total on-chain locked value of 500 billion USD. However, for application developers, integrating oracles brings significant design trade-offs and issues due to value extraction such as frontrunning, arbitrage, and inefficient liquidations. This article categorizes this value extraction as Oracle Extractable Value (OEV), outlines its key issues from the perspective of applications, and attempts to explain the key considerations for safely and reliably integrating oracles into DeFi protocols based on industry research.
Oracle Extractable Value (OEV)
This section assumes that readers have a basic understanding of oracle functionality and the differences between push-based and pull-based oracles. Price feeds for individual oracles may vary. For an overview, classification, and definitions, please refer to the appendix.
Most applications that use oracle price feeds only need to read prices: decentralized exchanges running their own pricing models use oracle price feeds as reference prices; for overcollateralized loan positions, collateral is deposited based on oracle price feeds to determine initial parameters such as the borrowing value compared to the liquidation price. Excluding extreme cases where pricing updates are infrequent, the delay in oracle price feed updates is not crucial when considering system design. Therefore, the most important consideration for oracles is evaluating the accuracy of price contributors and the decentralization performance of oracle providers.
However, if the delay in price feed updates is an important consideration, more attention should be paid to how oracles interact with applications. In such cases, this delay can lead to value extraction opportunities, namely frontrunning, arbitrage, and liquidations. This sub-type of MEV is referred to as OEV. Before discussing various implementation solutions and their trade-offs, we will outline the various forms of OEV.
Arbitrage
Frontrunning and arbitrage in oracle price feeds in derivative protocols are commonly referred to as "toxic flow" because these trades are conducted with asymmetric information, often at the expense of liquidity providers' costs to obtain risk-free profits. OG DeFi protocols like Synthetix have been addressing this issue since 2018 and have attempted various solutions over time to mitigate these negative externalities.
Let's illustrate with a simple example: a decentralized perpetual contract exchange XYZ uses Chainlink oracle for the ETH/USD market, with the example based on the ETH/USD price feed:
Figure 1: Example of arbitrage using Chainlink oracle
While the above is an oversimplified example that does not consider slippage, fees, or capital, it illustrates the role of deviation thresholds leading to insufficient price granularity and the opportunities it presents. Searchers can monitor the delay in price updates in the on-chain storage of Chainlink and extract risk-free value from liquidity providers (LP).
Frontrunning
Frontrunning is another form of value extraction where searchers monitor oracle updates in the memory pool and run actual market trades before submitting them on-chain, allowing them to execute trades at favorable prices before the oracle updates.
Perpetual contract decentralized exchanges like GMX have been victims of toxic frontrunning; approximately 10% of protocol profits were lost to frontrunning before all oracles were coordinated through the KeeperDAO protocol.
What if we only adopt a pull-based model?
One of Pyth's value propositions is that with the Pythnet built on the Solana architecture, publishers can push price updates to the network every 300 milliseconds, maintaining low-latency price feeds. Therefore, when applications query prices through Pyth's application programming interface (API), they can retrieve the latest prices, update them to on-chain storage, and execute any downstream operations in a single transaction.
As mentioned above, applications can directly query the latest price updates from Pythnet, update on-chain storage, and complete all relevant logic in a single transaction, effectively addressing frontrunning and arbitrage issues, right? Not quite - Pyth's updates give users the ability to choose which prices to use in transactions, which may lead to adversarial selection (another form of toxic flow). While on-chain stored prices must be recent, users can still choose any price that meets these constraints - meaning arbitrage still exists as it allows searchers to see future prices before using past prices. Pyth's documentation suggests that a simple way to mitigate this attack is to include a staleness check, ensuring that prices are recent enough - however, the timing threshold for updating transaction data in the next block must be determined.
Let's analyze using the example of perpetual contract decentralized exchange XYZ, where they use Pyth's ETH/USD price feed with a staleness check time of 20 seconds, meaning the timestamp of Pyth's price must be within 20 seconds of the block timestamp for executing downstream transactions:
Figure 2: Process of frontrunning using Pyth
An intuitive idea is to simply reduce the staleness check threshold, but a lower threshold may lead to unpredictable network responses affecting user experience. Due to Pyth's price feed dependency on bridging, sufficient buffering is required to a) provide proof of price to Wormhole guardians, and b) allow the target chain to process and include transactions in a block. The next section will detail these trade-offs.
Liquidations
Liquidations are a core part of any protocol involving leverage, and the granularity of price feed updates plays a crucial role in determining liquidation efficiency.
For push-based oracles based on threshold, when the spot price reaches the threshold but does not match the preset parameters of the oracle price feed, the granularity (or lack thereof) of price updates can lead to missed liquidation opportunities, resulting in negative externalities in market inefficiency.
When liquidations occur, applications typically pay out a portion of the liquidated collateral and sometimes provide rewards to the user initiating the liquidation. For example, in 2002, Aave paid out $37.9 million in liquidation rewards on the mainnet, which clearly overcompensated third parties and resulted in poor user experience. Additionally, when extractable value exists, the resulting Gas Wars can cause value to flow from the application to the MEV supply chain.
Design Space and Considerations
Given the above issues, the following will discuss various implementation solutions based on push-based, pull-based, and alternative designs, each with their effectiveness in addressing the aforementioned issues and the trade-offs involved. The trade-offs can take the form of added centralization and trust assumptions, or poor user experience.
Oracle-specific Order Flow Auctions (OFA)
Order Flow Auctions (OFA) have become a solution to eliminate the negative externalities generated by MEV. Broadly, OFA is a generic third-party auction service where users can send orders (trades or intents), and MEV extractors can bid for exclusive rights to run their strategies on these orders. A significant portion of the auction proceeds is returned to users to compensate for the value they create in these orders. The adoption of OFA has surged recently, with over 10% of Ethereum transactions being conducted through private channels (private RPC/OFAs) (Figure 3), and it is expected to further catalyze growth.
Figure 3: Combined daily private Ethereum transaction volume. Source: Blocknative
The challenge in implementing a generic OFA for oracles is that oracles cannot know if rule-based updates will generate any OEV, and if not, it will introduce additional delays when the oracle sends transactions to the auction. On the other hand, the simplest way to streamline OEV and minimize delays is to provide all oracle order flows to a single dominant searcher. However, this would obviously pose a significant centralization risk, potentially fostering rent-seeking behavior and censorship, and leading to a poor user experience.
Figure 4: General OFA vs. Oracle-specific OFA
Price updates for oracle-specific OFA without existing rule-based updates still occur in the public memory pool. This allows oracle price updates and any resulting extractable value to be retained at the application layer. As a byproduct, it also allows searchers to request data source updates without the additional cost of more frequent updates borne by the oracle nodes, thereby increasing data granularity.
Oracle-specific OFA is an ideal choice for liquidations as it brings finer-grained price updates, maximizes capital return to the liquidated borrower, reduces protocol rewards paid to liquidators, and retains the value extracted from bidders in the protocol for redistribution to users. They also partially - though not entirely - address frontrunning and arbitrage issues. Under conditions of perfect competition and a first-price sealed bid auction process, the auction results should be close to the block space cost of execution opportunities, the value extracted from frontrunning OEV data feeds, and the reduced arbitrage opportunities due to increased price granularity from price feed updates.
Currently, to implement oracle-specific OFA, it either requires joining a third-party auction service (such as OEV-Share) or building an auction service as part of the application. Inspired by Flashbots, API3 utilizes an OEV relay as a DoS protection service in design to conduct auctions. The relay is responsible for collecting meta-transactions from oracles, organizing and aggregating bids from searchers, and redistributing profits in a trustless manner without controlling bids. When a searcher wins the bid, updating the data source relies on transferring the bid amount to a proxy contract owned by the protocol, and then the proxy contract updates the price source with the signature data provided by the relay.
Figure 5: API3's OEV relay
Alternatively, protocols can forego intermediaries and establish their own auction service to capture all extractable value from OEV. BBOX is an upcoming protocol that aims to embed auctions into its liquidation mechanism to capture OEV and redistribute it to the application and its users.
Running Centralized Nodes or Keepers
Originating from the early idea of combating OEV in the first wave of decentralized perpetual contract exchanges, running a centralized Keeper network, aggregating prices received from third-party sources (such as centralized exchanges), and using a data feed similar to Chainlink as a contingency or circuit breaker has been promoted in GMX v110 and many subsequent forks. The main value proposition is that a Keeper network operated by a single operator can absolutely prevent frontrunning.
While this solves many of the above issues, there are clear centralization concerns. A centralized Keeper system can decide to execute prices without proper validation of pricing sources and aggregation methods. In the case of GMX v1, the Keeper was not a transparent mechanism on-chain but a program run on a centralized server address signed by the team. The core function of the Keeper is not only to execute orders but also to "decide" on the trading price based on its predefined definition, without the ability to verify the authenticity or source of the execution price used.
Automated Keeper Network and Chainlink Data Streams
To address the centralization risk posed by a single operator Keeper network, a more decentralized automated network can be established using third-party service providers. Chainlink Automation is one such product that, together with Chainlink Data Streams - a new pull-based, low-latency oracle - provides this service. The product has recently been released and is currently in closed beta testing, but GMX v211 is already using it and can serve as a reference for systems adopting this design.
At a high level, Chainlink Data Streams consists of three main components: Data DON (decentralized oracle network), Automation DON, and on-chain verification contracts. Data DON is an off-chain data network with a structure similar to Pythnet for maintaining and aggregating data. Automation DON is a guardian network operated by the same node operators as Data DON, used to extract prices from on-chain Data DON. Finally, the verification contract is used to verify off-chain signatures.
Figure 6: Chainlink Data Streams architecture
The diagram above shows the transaction flow for invoking the open trading function, where Automation DON is responsible for fetching prices from Data DON and updating on-chain storage. Currently, direct querying of Data DON endpoints is limited to whitelisted users, so the protocol can choose to offload Keeper maintenance to Automation DON or run its own Keeper. However, as the product development lifecycle progresses, this is expected to gradually transition to a permissionless structure.
On the security front, relying on the trust assumptions of Automation DON is equivalent to using the trust assumptions of Data DON alone, which is a significant improvement over the single Keeper design. However, if the right to update the price feed is handed over to Automation DON, the opportunity for value extraction will be left only to the nodes in the Keeper network. This, in turn, implies that the protocol will trust Chainlink node operators (primarily institutions) to maintain their social reputation and not front-run user operations, similar to how Lido Node operators are trusted to maintain their reputation and not monopolize block space due to their large market share.
Pull-based: Delayed Settlement
One of the major changes in Synthetix perps v2 is the introduction of Pyth price feeds for perpetual contract settlements. This allows orders to settle based on Chainlink or Pyth prices, provided their deviation does not exceed a predefined threshold and the timestamps pass the expiry check. However, as mentioned above, simply switching to a pull-based oracle does not solve all protocol-related OEV issues. To address frontrunning, a "last look" pricing mechanism can be introduced in the form of delayed orders. In practice, this splits a user's market order into two parts:
- Trade #1: Submitting the "intent" to open a market order on-chain, providing standard order parameters such as size, leverage, collateral, and slippage tolerance. Additional Keeper fees are also paid to reward the Keeper for executing Trade #2.
- Trade #2: The Keeper receives the order submitted in Trade #1, requests the latest Pyth price feed, and calls the Synthetix execution contract in a single transaction. The contract checks predefined parameters such as expiry and slippage, and if they pass, the order is executed, the on-chain price storage is updated, and the position is established. The Keeper charges a fee to compensate for the gas used in the execution and maintenance of the network.
This implementation prevents users from having the opportunity to front-run the prices they submit on-chain, effectively addressing the protocol's frontrunning and arbitrage opportunities. However, the trade-off of this design is user experience: executing this market order requires a two-step process, with users compensating for the Keeper's operations with gas fees and sharing the cost of updating the on-chain price feed. Previously, there was a fixed fee of 2 sUSD, but it has recently been changed to a dynamic fee based on the Optimism gas oracle + premium, with the premium varying based on Layer 2 network activity. Regardless, this can be seen as a solution that sacrifices the user experience of traders to improve LP profitability.
Pull-based: Optimistic Settlement
As delayed orders incur additional network fees for users (proportional to Layer 2 DA fees), another settlement model called "optimistic settlement" has been proposed to potentially reduce user costs while maintaining decentralization and protocol security. As the name suggests, this mechanism allows traders to execute market trades atomically, with the system actively accepting all prices and providing a window for searchers to submit evidence proving malicious orders. This section outlines different versions of this concept, our thought process, and the unresolved issues.
The initial idea was to establish a mechanism for users to submit prices through parsePriceFeedUpdates when opening market orders, and then allow users or any third party to submit settlement transactions using price feed data and complete the trade at that price upon transaction confirmation. During settlement, any negative difference between the two prices would be accounted for as slippage in the user's P&L. The advantages of this approach include reducing the cost burden on users and mitigating the risk of frontrunning. Users no longer need to bear the premium for rewarding Keepers, and since the settlement price is not known when the order is submitted, the risk of frontrunning remains manageable. However, this still introduces a two-step settlement process, which was found to be a drawback in Synthetix's delayed settlement model. In most cases, the additional settlement transaction may be unnecessary if the volatility between order placement and settlement does not exceed the system-defined profitable frontrunning threshold.
Another solution to avoid the above issues is to allow the system to actively accept orders and then open an unpermissioned challenge period during which evidence can be submitted to prove price deviations between the price timestamp and block timestamp that allow profitable frontrunning.
The specific operations are as follows:
- Users create orders based on the current market price. They then transmit the price along with embedded Pyth price feed byte data as part of the order creation transaction.
- The smart contract actively verifies and stores this information.
- After the on-chain confirmation of the order, there is a challenge period during which searchers can submit evidence of frontrunning. This evidence will confirm that the trader used outdated price feed data with the intention to arbitrage the system. If the system accepts the evidence, the difference will be applied as slippage to the trader's execution price, and the excess value will be rewarded to the Keeper.
- After the challenge period ends, the system considers all prices to be valid.
This model has two advantages: it reduces the cost burden on users, as they only need to pay gas fees for order creation and oracle updates in the same transaction, without the need for additional settlement transactions. It also prevents frontrunning, protects the integrity of liquidity pools, ensures a healthy Keeper network, and provides economic incentives for submitting evidence of frontrunning.
However, before putting this idea into practice, there are still some unresolved issues:
- Defining "front-running": How does the system differentiate between users who submit outdated prices due to network latency and those intentionally arbitraging the system? A preliminary idea could be to measure the volatility within the expiry check period (e.g., 15 seconds). If the volatility exceeds the net execution fee, the order would be flagged as a potential exploit.
- Setting an appropriate challenge period: Considering that toxic order flows may only be open for a short time, what is an appropriate time window for Keepers to challenge prices? Batch proofs may be more cost-effective, but given the unpredictability of order flows over a period, it is difficult to determine the timing of batch proofs to ensure that all price information is proven or there is sufficient time for challenge.
- Economic incentives for Keepers: To make submitting evidence reasonable for incentivized Keepers, the rewards for submitting winning evidence must exceed the gas costs associated with submitting evidence. This assumption may not be guaranteed due to different order sizes.
- Whether a similar mechanism needs to be established for closing orders, and if so, how it would affect user experience.
- Ensuring that "unreasonable" slippage does not fall on users: In the event of a flash crash, there may be significant price differences between order creation and on-chain confirmation. Some form of backup or circuit breaker may be needed, possibly using Pyth's EMA price, to ensure the stability of the price feed before use.
ZK Co-processors - Another Form of Data Consumption
Another direction worth exploring is the use of ZK co-processors, which are designed to obtain on-chain state for complex off-chain computations and provide proofs of computation execution in a permissionless manner. Projects like Axiom enable contracts to query historical blockchain data, perform computations off-chain, and submit ZK proofs to demonstrate that the computation result is correctly derived from valid on-chain data. Co-processors open up the possibility of using historical price data from multiple DeFi-native liquidity sources (such as Uniswap + Curve) to construct custom TWAP oracles with manipulation resilience.
Compared to traditional oracles that can only provide the latest asset price data, ZK co-processors will expand the range of data securely provided to dApps (Pyth does provide EMA prices for developers to use as a reference check for the latest price). This allows applications to introduce more business logic working with historical blockchain data to enhance protocol security or improve user experience.
However, ZK co-processors are still in the early stages of development, and there are some bottlenecks, such as:
- Obtaining and computing a large amount of blockchain data in the co-processor environment may require a longer proof time.
- Only providing blockchain data does not address the need for secure communication with non-Web3 applications.
Oracle-Free Solutions - The Future of DeFi?
Another approach to addressing this issue is to design a primitive from scratch, eliminating the need for external price feeds and thus addressing DeFi's reliance on oracles. The latest development in this area involves using various AMM LP tokens as a pricing mechanism, with the core idea being that LP positions of constant function market makers represent tokens with preset weights of two assets and have an automatic pricing formula for these two tokens (i.e., xy=k). By utilizing LP tokens (as collateral, the basis for loans, or, in recent use cases, moving v3 LP positions to different scales), the protocol can obtain information that would typically require a price feed from an oracle. As a result, a new wave of oracle-free solutions that are free from the challenges mentioned above has emerged. Examples of applications based on this approach include:
Panoptic is building a perpetual, oracle-free options protocol that utilizes Uniswap v3 concentrated liquidity positions. Since concentrated liquidity positions convert 100% to the base asset when the spot price exceeds the upper range of the LP position, the returns for liquidity providers are very similar to those of put option sellers. Therefore, the operation of the options market involves liquidity providers depositing LP assets or positions, option buyers and sellers borrowing liquidity and moving it in or out of range, resulting in dynamic option returns. Since loans are priced in LP positions, no oracle is required at settlement.
Infinity Pools is leveraging Uniswap v3 concentrated liquidity positions to create a margin trading platform without liquidation and without oracles. Liquidity providers on Uniswap v3 can lend out their LP tokens, and traders can deposit collateral, borrow LP tokens, and redeem the assets related to their directional trades. The loan at redemption is priced in the base or quote asset, depending on the price at redemption, and can be calculated directly by checking the LP composition on Uniswap, eliminating the need for an oracle.
Timeswap is building a fixed-term, non-liquidation, oracle-free lending platform. It is a three-party market composed of lenders, borrowers, and liquidity providers. Unlike traditional lending markets, it uses time-based settlement instead of price-based liquidation. In decentralized exchanges, liquidity providers are automatically set to always buy from sellers and sell to buyers; in Timeswap, liquidity providers always lend to borrowers and borrow from lenders, playing a similar role in the market. They are also responsible for bearing the default risk of the loan and have priority in receiving confiscated collateral as compensation.
Conclusion
Pricing data remains an essential part of many decentralized applications, and over time, the total value obtained from oracles is increasing, further affirming their product-market fit. This article aims to inform and outline the OEV-related challenges we currently face, as well as the design space in implementation solutions based on push-based, pull-based, and the use of AMM liquidity providers or off-chain co-processors.
We are excited to see vibrant developers eager to solve these challenging design problems. If you are also working on disruptive projects in this field, we are eager to hear your feedback!
References and Acknowledgments
Thanks to Jonathan Yuen and Wintersoldier for their contributions and discussions, which greatly contributed to this article.
Thanks to Erik Lie, Richard Yuen (Hailstone), Marc, Mario Bernardi, Anirudh Suresh (Pyth), Ugur Mersin (API3 DAO), and Mimi (Timeswap) for their valuable insights, feedback, and reviews.
https://defillama.com/oracles (14 Nov) ↩︎
OEV Litepaper https://drive.google.com/file/d/1wuSWSI8WY9ChChu2hvRgByJSyQlv_8SO/edit↩︎
Frontrunning on Synthetix: A History by Kain Warwick https://blog.synthetix.io/frontrunning-synthetix-a-history/
https://snapshot.org/#/rook.eth/proposal/0x523ea386c3e42c71e18e1f4a143533201083655dc04e6f1a99f1f0b340523c58 ↩︎
https://docs.pyth.network/documentation/pythnet-price-feeds/on-demand ↩︎
https://docs.pyth.network/documentation/solana-price-feeds/best-practices#latency ↩︎
Aave liquidation figures https://dune.com/queries/3247324
https://drive.google.com/file/d/1wuSWSI8WY9ChChu2hvRgByJSyQlv_8SO/edit
https://twitter.com/bboexchange/status/1726801832784318563
https://gmx-io.notion.site/gmx-io/GMX-Technical-Overview-47fc5ed832e243afb9e97e8a4a036353
https://gmxio.substack.com/p/gmx-v2-powered-by-chainlink-data ↩︎
https://docs.chain.link/data-streams ↩︎
https://sips.synthetix.io/sips/sip-281/
↩︎
︎︎
Appendix
Definition: Push-based and Pull-based Oracles
Push-based oracles maintain off-chain prices in a P2P network and maintain on-chain price updates based on predefined on-chain nodes. Using Chainlink as an example, price updates are based on two trigger parameters: deviation threshold and heartbeat. As long as the off-chain price deviates 0.5% from the latest on-chain price, or the heartbeat timer reaches zero after one hour, the Ethereum ETH/USD price feed below will update.
In this scenario, oracle operators must pay transaction fees for each price update, which is a trade-off between cost and scalability. Increasing the number of price feeds, supporting additional blockchains, or increasing the frequency of updates all incur additional transaction costs. Therefore, long-tail assets with higher trigger parameters inevitably have less reliable price feeds. This can be illustrated using CRV/USD as an example - to allow new prices to update on-chain, a 1% deviation threshold and a 24-hour heartbeat are required, meaning that if the price does not deviate by more than 1% within 24 hours, there will only be one new price update every 24 hours. Intuitively, the lack of precision in price feeds for long-tail assets will inevitably lead to additional risk factors that applications need to consider when creating markets for these assets, explaining why the majority of DeFi activity still revolves around tokens with the strongest liquidity and largest market capitalization.
In contrast, pull-based oracles allow prices to be pulled onto the chain as needed. Pyth is the most prominent example today, transmitting price updates off-chain, signing each update so that anyone can verify its authenticity, and maintaining aggregated prices on Pythnet, a private blockchain based on Solana code. When updates are needed, data is transmitted through Wormhole, verified on Pythnet, and can be pulled onto the chain without permission.
The diagram above describes the architecture of the Pyth price feed: when an on-chain price update is needed, users can request an update via the Pyth API, and the verified price on Pythnet will be sent to the Wormhole contract. The Wormhole contract will observe, create, and send a unique VAA, which can be verified on any blockchain where the Pyth contract is deployed.
免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到support@aicoin.com，本平台相关工作人员将会进行核查。
