In 2022, GMX, the leading perpetual decentralized exchange with a TVL (Total Value Locked) exceeding $500 million, awarded a million-dollar bug bounty to Collider's research arm. The vulnerability, initially discovered by Collider, was reported to GMX some time ago and has been successfully addressed in recent months. However, the specific details of this vulnerability are only now being made public.
Aftermath of GLP price skew bug
When evaluating investments for Collider’s Defi Fund we follow a strict methodology of risk analysis. As part of that analysis, we launch an internal audit of the project in question, reviewing both the technical and operational aspects.
GMX is well-regarded for its remarkable growth. Having tracked its trajectory closely both from afar and as users, we approached the audit with confidence in the security of their contracts. Instead, we went beyond the usual and delved into potential operational vulnerabilities, often generally overlooked. These diverge from security weaknesses as they can emanate from the system's very design rather than explicit flaws. Detecting them mandates more than code examination – a comprehensive grasp of the financial dynamics at play is required.
For those who are not familiar, GMX is a unique DEX on Arbitrum and Avalanche that allows users to swap or leverage-trade different blue-chip assets like BTC and ETH.
The liquidity for trades comes from LPs who deposit these and other tokens into a single Vault contract. In return, they are given the GLP token which can be redeemed at any time.
One of the paths we went out to explore was the price fluctuation of GLP. The underlying assets represented in the vault contract are roughly stablecoins, ETH, BTC and also LINK, and UNI (this is true for Arbitrum, which will be the focus of this text going forward).
Since traders are using these tokens for trading, the vault is also exposed to their PnL performance, that is - when traders make money it comes out of the LPs pocket and vice versa.
This is the flow for minting\burning GLP: 1. LP chooses a token they would like to supply.
2. The system calculates the current fair value of GLP (this is the important part).
3. Tokens are deposited and in return, new GLP is minted.
4. For burning, the flow is the same. (but in reverse)
What we set out to find
Our goal was to determine what affects the price of GLP and see if it can be manipulated. For this task, we used the following approach:
Examine the variables that affect the final value of GLP.
Determine if any of them can be manipulated.
Go over each variable, examine the variables that compose it and determine if they can be manipulated.
Continue until all layers and angles are covered.
How GLP price is being calculated
Starting with the topmost layer - GLP’s price is the outcome of this straightforward logic:
AUM \ totalSupply = GLP
We have 2 variables that comprise the final price: total supply changes whenever GLP is being minted or burned, so nothing much is there. Let’s dig further and see what comprises the value of the AUM aspect.
This value is derived from the method `getAum` within the GlpManager contract:
Fetch the list of participating tokens from the vault, and iterate.
If a token is a stablecoin, simply add its value to the final AUM.
Else, calculate the number of tokens, multiplied by current price.
Apply the unrealized PnL of current traders and add that sum to the AUM total.
How PnL is calculated
The current state of traders’ positions is always embedded within the GLP vault, in quite a sophisticated way. It led us to these two important facts: 1. GLP price is not affected by minting or burning
2. GLP price is not affected by a trader opening or closing a position.
Meaning, that the price can’t be manipulated just by opening a position, rather only by an actual change in PnL.
For long positions
(This part is not necessary for understanding the bug so you can skip if you are impatient)
When a trader opens a long position, the system will reserve a certain amount of tokens to make sure it will be able to payout.
For example, suppose Alice opens a 10 ETH long position. The system needs to reserve 10 ETH and with that, no matter how high ETH’s price might go, there should always be a way to pay her out. This value is called `reservedAmount` and it shows up on line 37 when it is subtracted from the AUM calculation.
The second parameter is `guaranteedUsd`, a concept that warrants a more nuanced explanation which is out of the current scope of this writeup. Let’s just say that it is denominated in dollars and represents the minimum value of USD in the pool.
Suppose Bob pledges 1 Ether as collateral and opens a 10 ETH long position, then guaranteed is (10-9) * ether_price, effectively representing the USD value of the delta between Bob’s collateral and Bob’s position at the moment of initiation.
Notice that the reserved amount is denominated in tokens and guaranteedUsd in dollars, and so the divergence between the two will represent the PnL of all long positions.
The final equation for pool value, considering long positions is (lines 35-38):
(poolAmount - reservedAmount) * price + guaranteedUsd
We acknowledge that this might be a bit daunting to fully understand, so let’s move on as these specific intricacies bear less relevance going forward.
For short positions
After adding the pool value and the PnL of long positions we need to calculate the PnL of short positions and subtract or add that to the AUM, which is slightly more complicated.
For tracking the PnL of short positions the system saves two values: the total short size and the average short price. When a trader opens a short position, the vault updates those two values here.
Suppose the value of short positions is $1,000 and the average price is $1,500.
Now Charlie opens another short position valued at $1,000 with an average price of $2,000.
The global positions’ value is now $2,000 (1,000+1,000) and the average price is $1,750
(1,500*1,000+2,000*1,000) / (1,000+1,000).
We can see how GlpManager handles that on lines 23-33:
Take the average price of shorts and the current price of the token.
Calculate the delta and multiply it by global shorts’ size.
If shorters are in profit, subtract from AUM. Else, add to AUM.
Same as with longs - these values should only be affected by market change and not by traders simply opening or closing a position. Only that it doesn’t.
Backtesting the results
After we learned how GLP is truly priced, we went out to test these dynamics in the wild. We chose a sample of Arbitrum blocks, and used the data to create a shadow GLP using our own getAum implementation, to then compare the results of our calculation with the ones from the historical node.
We were expecting to see both values zero themselves out but instead we saw this strange divergence:
As you can see in the right column, the addition of Shadow PnL and GLP which are inverted should be 0. Instead we noticed a fluctuation in value. Here we can see it is a small amount, but keep in mind we only used a sample of blocks. Had we allowed the backtester to run further back, the divergence would have grown in value, substantially.
We assumed there was an obvious mistake in our calculation since all values are solid and shouldn’t have affected GLP’s price, as we discussed earlier.
This thought led to a long and frustrating research, as we tried to pinpoint exactly what is causing this divergence.
We can’t put our frustration at the time into words. Things didn’t add up.
At one point we’ve decided to examine every block, dissecting and comparing it to the next, to see what we might be missing.
We’ve created a spreadsheet with each block’s values relevant to the getAum calculation, and even implemented the formula and tried to manipulate the numbers by hand.
Nothing worked. But, we had a lead: We were able to detect that the rogue value is related to short positions, so that became our main focus.
What we did was to locate two consecutive blocks where the only difference was a trader closing a short position, all other values used by getAum were isolated and unchanged between the two blocks. Circling back, we derived that the closing of a short or long position should not affect the result of getAum, but in reality it was the opposite: the two blocks yielded different values for getAum!
We knew then something weird was happening, but what exactly?
Finding the solution
As mentioned above, when a trader opens a short position its value is added to the global short size and the average short price is adjusted. (The Charlie example)
When a short position is being closed, those two values need to be adjusted as well: If Charlie later closed their $1,000 short position at a loss when the market price was $2,500 the calculation for short average price needs to be:
(2,000*1,750 - 1,000*2,500) / (2,000-1,000) = $1000
This way the average price of the shorts is being adjusted to maintain the same PnL even when the position is closed and so no change to getAum is caused.
When closing a position we see this directive:
if (!_isLong) {
_decreaseGlobalShortSize(_indexToken, _sizeDelta);
}
Meaning, when closing a short position, the global short size is being adjusted, but average short price is left unchanged.
Put simply
Given:
globalShortSize is $1,000.
average price is $200 (5 units).
market price is $125.
getAum will calculate shortProfits as 1,000*(200-100)/200 = $500 (loss)
A trader opens another 5 unit short position when the market price is at 100.
Now the short average price is (5*200 + 5*100)/10 = $150.
And the global short size is: 1,000+(5*100) = $1500
Looking at the shorts’ PnL, nothing had changed: 1500*(150-100)/150 = still $500.
Now, the same trader immediately closes their position.
Global short size is now 1,500- (5*100) = $1,000 (as before)
Average short price is still 150 since it didn’t change (remember, they closed immediately)
Now let's calculate the PnL: 1000*(150-100)/150 = 333~
Yeah but what does it *really* mean?
What we just saw is that by opening and closing a position, we are affecting the way GLP calculates short profits, and since this calculation increases or decreases the AUM, it essentially means that the price of GLP is affected by other factors that are not token values and PnL. It means that just by opening and closing a short position, a trader will cause GLP’s price itself to fluctuate.
So, if you are holding GLP looking to enjoy long profits from the market along with your juicy yield you will actually net *less* money than you should’ve.
And if you are a smart LP that provides liquidity to GLP but also hedges the directional PnL to be close to delta-neutral as one can be (Umami, et al) - you will incur significant losses since GLP will not increase in value as much as your shorts will lose.
Fixing the error
After this discovery, we felt quite uneasy. Results from backtest samples showed that not only did this bug constantly affect users, it proved to be a very complex endeavor to attempt to assess both the damage already done to existing LPs and future potential losses. The reason is simple - it’s dependent on market conditions and traders’ activity.
We monitored and kept pulling data from past positions and states. At times we saw the financial damage amounting to 5-6% of each position, whereas on occasions LPs were actually net positive. All depending on which side of the trade you were, in light of how GLP is truly priced.
Fortunately, the market was trading sideways for quite a while. It was the most preferred timing to deploy a fix without altering the current price of GLP, as to not hurt live positions. (this is untrue for all the positions that were settled though).
Once we had all the evidence, we decided to reach out anonymously and contact the GMX team through Immunefi. We pride ourselves in our professionalism and morals, and thinking about potential ways to abuse, game or exploit the system was out of the question. We decided to notify anonymously to be as direct and neutral as possible. We notified them of the bug, provided a test case for recreating it and suggested multiple ways to potentially fix it.
The GMX team responded immediately and, with our help, began working on the chosen fix, which, due to its complex nature, required a great deal of work. We offered to tail their process and agreed to review the result, to make sure it all runs smoothly.
We have only good things to say about the GMX team, how they carried themselves, and how they put users’ funds and safety first, and as DeFi users we wish and hope every disclosure will fall on the ears of like-minded people.