## 1.Introduction

During week `40` of the Glow protocol, an issue arose in the submission process for the weekly report. It's important to note that the week `40` report contains all the data from week `39`, as per the protocol's reporting structure on the weekly reports page.

A bug in the submission script led to the publication of incorrect rewards weights on-chain. This error was promptly identified by the Veto Council, who took immediate action by delaying the finalization of the affected bucket for `90 days`. As a consequence, user rewards are expected to be delayed for `100 days`.

The impact of this incident is significant but temporary. With the upcoming launch of GlowV2 on the horizon, there is a strong expectation that this error will be rectified as part of the relaunch process. This incident underscores the importance of rigorous checking and the value of having oversight mechanisms like the Veto Council in place to catch and address such issues promptly.

### 1.1 Key Details

| Event | Date | Transaction |
| --- | --- | --- |
| Faulty Submission | August 31, 2024 | [View on Etherscan](https://etherscan.io/tx/0xa382dba8e0b69488f137c6903f8128f0530e3ec6e4554e276c9651af62602450) |
| Issue Detected | September 6, 2024 | N/A |
| Bucket Delay by Veto Council | September 6, 2024 | [View on Etherscan](https://etherscan.io/tx/0xef5110153497f2a2d867ccf6c0535933149087e1c08ed96470328b5f9d2052bb) |

### 1.2 Affected Systems/Components

The rewards for bucket `40` (containing the week `39` weekly report) are expected to be delayed for `100 days`.

## 2.IncidentDescription

An automated script used by Glow Certification Agents (GCAs) to generate weekly on-chain reports contained an error. This error incorrectly converted a farm's carbon credit production into an Ethereum `uint256`, significantly overstating its output. While this didn't affect the total carbon credits reported on-chain, it skewed the reward distribution weights. As USDG rewards are allocated based on relative carbon production, this farm could have claimed a disproportionately large share of USDG compared to other farms.

### 2.1 Discovery

Simon from the Veto Council was reviewing the weekly report and noticed the inconsistency. He confirmed the inconsistency, found the root source of the bug in the automated script, and then delayed the bucket.

### 2.2 Impact

The impact is that the rewards for bucket `40` (containing the week `39` weekly report) are expected to be delayed for `100 days`.

## 3.RootCauseAnalysis

The code used to create the report for week `40` contained a bug in the conversion of float values to BigNumbers. Specifically, the issue occurred in the following code block:

```
1const finalLeaves: FinalLeaf[] = merkleLeaves.map(
2  ({ wallet, glowWeight, usdgWeight }) => ({
3    wallet,
4    glowWeight: parseUnits(
5      glowWeight.toString(),
6      GLOW_WEIGHT_DECIMAL_PRECISION
7    ).toString(),
8    usdgWeight: parseUnits(
9      usdgWeight.toString(),
10      USDG_WEIGHT_DECIMAL_PRECISION
11    ).toString(),
12  })
13);
```

One of the devices had a very small amount of carbon credits produced that caused the USDG weight to be represented in scientific notation. The exact number was: `9.955189695275401e-7`. This number is then stringified and parsed into a BigInt using viem's `parseUnits` function. The error came from viem not recognizing scientific notation. This caused viem to parse `9.9551...` with `USDG_WEIGHT_DECIMALS` rather than parsing `.000000995....` into a BigNumber which caused the reported weight to be `10x` bigger than intended.

## 4.Resolution

### 4.1 The Fix

To properly reconcile the rewards for week `40`, the veto council delayed the finalization of the bucket for `90 days`. It is expected that GlowV2 will launch before the bucket finalizes, and the rewards will be corrected upon relaunch. In the case that GlowV2 is no longer expected to launch before the new bucket finalization timestamp, Governance is expected to slash the GCA which will invalidate the report and give new GCAs time to submit the correct report.

## 5.PreventativeMeasures

### 5.1 Patch

1. As soon as the bug was discovered, Simon reached out to the viem team to address the issue.
2. The viem team acknowledged the severity of the issue and is planning to implement a patch to throw on scientific notation strings. [The planned patch can be found here](https://github.com/wevm/viem/commit/19a66282e6e1d2a7702cb82f1f12b69141389b76)

The automated script has also now been adjusted to include several more checks:

1. Convert numbers using `customToFixed` which ensures proper decimals as opposed to using `toString`
   - [The function can be found here](https://github.com/glowlabs-org/glow-reporting-data/blob/master/src/utils/custom-to-fixed.ts)
   - [The tests for the function can be found here](https://github.com/glowlabs-org/glow-reporting-data/blob/master/src/utils/custom-to-fixed.test.ts)
2. Added invariant checks to ensure that carbon credits produced and weekly payments match up with the usdg and glow weight reported respectively [in this commit](https://github.com/glowlabs-org/glow-reporting-data/commit/779426a4e9c0e668dd31532628e9d70f84f28eaa) with [tests for the function here](https://github.com/glowlabs-org/glow-reporting-data/blob/master/src/utils/greater-than-max-deviation.test.ts)

### Author:Glow Team