# Triple and Presignature Asset Generation

Some multi-party computation (MPC) protocols require pre-computed
cryptographic material in the form of **triples** and **presignatures**
(herein simply called **assets**) as input. This document describes how
our MPC nodes manage asset generation, storage and consumption. It
applies only to **OT-based ECDSA** (**Cait-Sith**) at this point.

The system supports an arbitrary number of secret keys. Each key is
uniquely identified by a `DomainId`, which maps to a combination of
signature scheme (ECDSA/EdDSA), protocol (OT-Based ECDSA, Robust ECDSA,
FROST EdDSA) and application (signatures, CKD, foreign transactions).
See [#1649](https://github.com/near/mpc/issues/1649) for more information on
domains. Triples are not domain-specific; a single shared triple store
feeds presignature generation for all OT-based ECDSA domains.
Presignatures are per-domain because they depend on the domain's key
share.

```
Triples  ──►  Presignatures  ──►  Signatures
(slow)         (fast)              (fastest)
```

Each MPC signature requires one **presignature**, and each presignature
requires a pair of **triples** (called a `PairedTriple`). Triple
generation is the bottleneck: it involves heavy OT-based cryptographic
computation. Presignature generation is significantly faster, and
signature generation is fairly cheap.

## Queue design

The asset queue manages asset usage with three goals:

- Efficiently provide an asset compatible with the current set of online
  participants when one is needed.
- Ensure each asset is consumed at most once.
- Prevent the store from growing indefinitely by evicting unusable assets
  over time.

### Owned vs unowned

Every asset is generated by a set of participants. The participant who
initiates the computation is the **leader**, while the others are
**followers**. The generic protocol runner (`crates/node/src/protocol.rs`)
handles incoming generation requests from leaders over the P2P network.
When a leader spawns a triple or presignature protocol, each follower
receives the protocol messages, runs its side of the MPC computation,
and stores the result. All participants hold a different share of the
same asset, identified by a common unique ID.

The leader of the computation becomes the **owner** of the resulting
asset, while the followers become **borrowers**. Owned assets sit in a
queue and are consumed in roughly FIFO order. Unowned assets are only
consumed by borrowers when the owner requests it.

This asymmetry means only the owner's queue management matters for
throughput. Borrowers are passive.

### Online and offline

A participant is **online** from the perspective of another participant if
it has an active, bidirectional P2P connection with it and is within a
few blocks of its indexer height (see
`all_alive_participant_ids()` on `MeshNetworkClient`). An asset is
**online** if all of its borrowers are online from the perspective of
the asset owner, and **offline** otherwise.

### Public interface

The asset store (`DistributedAssetStorage`) exposes the following
operations:

| Method | Description |
|--------|-------------|
| `add_owned(id, value)` | Stores a newly generated asset as owned. Persists to `RocksDB` and pushes to the in-memory queue. |
| `take_owned()` | Blocks until an online asset is available, removes it from storage, and returns it. |
| `maybe_discard_owned(n)` | Examines up to `n` assets. Discards those with offline borrowers; keeps online ones aside as ready. |
| `add_unowned(id, value)` | Stores another owner's asset share in `RocksDB`. |
| `take_unowned(id)` | Looks up an unowned asset by ID, removes it from `RocksDB`, and returns it. Fails if not found. |
| `generate_and_reserve_id()` | Returns a fresh unique ID for a new asset. |
| `generate_and_reserve_id_range(n)` | Returns the start of a contiguous range of `n` unique IDs. |
| `num_owned()` | Count of owned assets excluding confirmed-offline (online + unknown). |
| `num_owned_ready()` | Count of owned online assets. |
| `num_owned_offline()` | Count of owned offline assets. |

### Properties of an asset taken from the queue

When `take_owned()` returns an asset, the following holds:

1. **All borrowers were online at check time.** The queue verifies
   that the asset is online by the time it is taken from the queue
   (see [Cold queue](#cold-queue) for how this classification works).

2. **The asset has not been used before.** It is removed from storage
   atomically on retrieval. Each asset is consumed exactly once.

`take_unowned(id)` does not guarantee either property — it performs a
plain database lookup with no liveness check. Borrowers trust the
owner's choice.

**Remark: No real-time liveness guarantee.** A participant can go offline
between the liveness check and the start of the protocol. If that
happens, the protocol will fail and the asset is lost. There is no
reservation or retry mechanism for the asset itself — the node simply
uses the next available one. The queue provides a **best-effort liveness
filter**, not a guarantee that the protocol will succeed.

### Asset Cleanup

The queue discards assets to prevent the store from filling up with
unusable entries:

- **Offline assets** are the only ones subject to discarding. They are
  evicted from the back of the queue when the store reaches its target
  buffer size.
- **Online assets are never discarded.** If an asset is online, it is
  moved to the front of the queue, even during a discard pass.

Additional bulk cleanup happens on startup — see
[Asset cleanup on epoch transition](#asset-cleanup-on-epoch-transition).

## Asset storage

All assets are persisted to `RocksDB`, keyed by an optional domain ID
prefix followed by the asset's `UniqueId`. Owned assets are additionally
loaded into the in-memory `DoubleQueue` (described below) on startup by
iterating the `RocksDB` key range for the local participant ID. After
startup, every write and delete goes to both `RocksDB` and the in-memory
queue, keeping them in sync.

## Hot/cold queue architecture

The `DoubleQueue` that holds owned assets has two layers:

```
  ┌──────────────┐       ┌────────────────────────────────────────────────┐
  │  Hot queue   │──────►│                Cold queue                      │
  │ (MPMC chan)  │       │          [ready | unknown | offline]           │
  └──────────────┘       └────────────────────────────────────────────────┘
```

### Hot queue

An unbounded multi-producer multi-consumer (MPMC) channel. Newly generated assets are pushed here by
`add_owned()`. The hot queue is drained into the cold queue the first
time an asset is needed.

### Cold queue

A `VecDeque` divided into three logical regions by two barriers:

```
0                          cold_ready        cold_available                len
 ──────────────────────────── ──────────────────── ─────────────────────────
│ Condition-satisfying       │   Unknown          │ Non-satisfying          │
 ───────────────────────────────────────────────────────────────────────────
```

- **`cold_ready`** (index 0..cold_ready): online assets. Returned
  immediately by `take()`.
- **Unknown** (cold_ready..cold_available): not yet checked against
  current condition. Checked lazily during `take()`.
- **Non-satisfying** (cold_available..len): offline assets. Skipped by
  `take()`, targeted by `discard()`.

The **condition** is whether the asset is online. The set of corresponding
online participants is cached for up to 1 second. When it changes, the barriers
reset — the entire queue becomes "unknown" — and assets are re-classified lazily
on next access.

Offline assets are kept rather than immediately removed, because they may
become usable again when participants reconnect. However, `take_owned()`
will never return an offline asset — if all owned assets are offline, it
blocks and re-checks the set of online participants every second until
an asset comes back online.

### take_owned() flow

1. Force-refresh the condition value.
2. Try to pop from the cold queue's ready/unknown region.
3. If the cold queue is exhausted, wait for an item from the hot queue
   (with a 1-second timeout so condition changes are noticed).
4. A freshly received hot-queue item that doesn't satisfy the condition
   is pushed to the cold queue's non-satisfying region.

### maybe_discard_owned() flow

Called when the store is full (`num_owned == desired_to_buffer`). It
processes a fixed number of elements:

1. Pops from the back of the cold queue. If an asset doesn't satisfy
   the condition, it is permanently removed (deleted from DB too).
2. If the cold queue is exhausted, drains available hot-queue items and
   classifies them.

This prevents the store from filling up with unusable assets when
participants go offline.

**Note on orphaned unowned assets:** When an owned asset is discarded, only the
local copy is deleted. There is no mechanism to notify borrower nodes to delete
their unowned copies of the same asset. Since `take_unowned(id)` is never called
for a discarded asset, those copies remain in borrowers' `RocksDB` indefinitely.
`clean_db()` cannot find them either — it only iterates keys namespaced by
`my_participant_id`, while unowned assets are keyed by the original owner's
participant ID. This also means the same-epoch TLS-key-change cleanup
(`KeepOnly` branch in `delete_stale_triples_and_presignatures()`) misses
unowned assets, since it delegates to `clean_db()`. The only event that
clears them is a full asset wipe on epoch change (resharing). In normal
operation this constitutes a slow disk storage leak.

## Background asset generation

Assets are continuously generated in the background. The node
configuration specifies:

- The number of triples and presignatures each node should own
  (`desired_triples_to_buffer`, `desired_presignatures_to_buffer`).
- The number of asset generation computations a node can run
  concurrently (`concurrency`).

If a node owns fewer assets than configured, it initiates new generation
computations (see
[`providers/ecdsa/triple.rs`](https://github.com/near/mpc/blob/main/crates/node/src/providers/ecdsa/triple.rs)
and
[`providers/ecdsa/presign.rs`](https://github.com/near/mpc/blob/main/crates/node/src/providers/ecdsa/presign.rs)).
Triples are generated in batches of 64, while presignatures are generated one at
a time. There is one triple generation loop shared across all OT-based ECDSA
domains, and one presignature generation loop per domain.

The followers for a presignature computation must match the borrowers of
the triple pair consumed in that computation. Since `take_owned()` on
the triple store **blocks** if no online triples are available, this is
the most common reason presignature generation stalls.

See [Appendix](#appendix)
for a more detailed explanation.

## Signature consumption

Source: `crates/node/src/providers/ecdsa/sign.rs`

Unlike triples and presignatures, signatures are not pre-generated.
When a signature request arrives:

1. **Leader** calls `presignature_store.take_owned()` for the relevant
   domain, consuming one presignature.
2. Leader opens a network channel with the presignature's borrowers
   and broadcasts the presignature ID along with the signature request.
3. **Followers** call `presignature_store.take_unowned(id)` to retrieve
   their share, then run the protocol.
4. The leader does **not** wait for all followers to confirm success
   (`leader_waits_for_success` returns `false`).

## Asset cleanup on epoch transition

Source: `crates/node/src/assets/cleanup.rs`

Every time the node enters the Running state,
`delete_stale_triples_and_presignatures()` compares the current epoch
data with what was stored in `RocksDB`:

| Scenario | Action |
|----------|--------|
| First run (no epoch in DB) | Keep all assets |
| **Epoch ID changed** (resharing occurred) | **Delete all** triples and presignatures |
| Same epoch, no participant TLS key changes | Keep all assets |
| Same epoch, some participant changed TLS key | Delete only assets involving changed participants |

Deleting all assets on epoch change is a conservative choice.
Presignatures incorporate key shares and are definitely invalidated by
resharing, but triples are independent of key shares and could
theoretically be preserved if the participant set remains compatible.

A participant's P2P public key changes when they regenerate their
`secrets.json` (e.g. the node is redeployed from scratch). The P2P key
itself is unrelated to the cryptographic shares, but changing it signals
that the participant likely lost their local state (`RocksDB`), meaning
they no longer have their copies of asset shares. Any asset involving
that participant is therefore unrecoverable. These are removed via
`clean_db()`, which iterates the `RocksDB` key range for owned assets and
checks whether all the asset's participants are in the "unchanged"
set.

## Prometheus metrics

### Asset counts (gauges, updated each loop iteration)

| Metric | Description |
|--------|-------------|
| `mpc_owned_num_triples_available` | Owned triples excluding confirmed-offline (online + unknown). Maps to `num_owned()`. |
| `mpc_owned_num_triples_online` | Owned online triples. Maps to `num_owned_ready()`. |
| `mpc_owned_num_triples_with_offline_participant` | Owned offline triples. Maps to `num_owned_offline()`. |
| `mpc_owned_num_presignatures_available` | Owned presignatures excluding confirmed-offline. Same semantics as triples. |
| `mpc_owned_num_presignatures_online` | Owned online presignatures. |
| `mpc_owned_num_presignatures_with_offline_participant` | Owned offline presignatures. |

## Configuration

Defined in `crates/node/src/config.rs` and set in `config.yaml`.

### `TripleConfig`

| Field | Type | Description |
|-------|------|-------------|
| `concurrency` | `usize` | Max concurrent triple generation protocol executions (semaphore permits). |
| `desired_triples_to_buffer` | `usize` | Target number of owned triples to maintain. Generation pauses when `num_owned + in_flight >= desired`. |
| `timeout_sec` | `u64` | Timeout for a single triple generation protocol execution. |
| `parallel_triple_generation_stagger_time_sec` | `u64` | Delay between spawning successive triple generation tasks, to prevent all tasks from starting at the same time. |

### `PresignatureConfig`

| Field | Type | Description |
|-------|------|-------------|
| `concurrency` | `usize` | Max concurrent presignature generation protocol executions. |
| `desired_presignatures_to_buffer` | `usize` | Target number of owned presignatures per domain. |
| `timeout_sec` | `u64` | Timeout for a single presignature generation protocol execution. |

### `SignatureConfig`

| Field | Type | Description |
|-------|------|-------------|
| `timeout_sec` | `u64` | Timeout for a single signature computation. |

## Troubleshooting: observed production anomaly

The following was observed simultaneously in production (#2123):

1. The presignature generation task is not reported as running.
2. The node is computing signatures as leader.
3. The number of available presignatures does not decrease.

**Explanation:** The asset count gauges are only updated inside the
presignature generation loop. The signature path consumes presignatures
via `take_owned()` but never updates these metrics. If the generation
loop dies, the gauges freeze at their last value. The node can continue
signing (consuming presignatures from the store) while the metrics show
a stale, unchanging count.

Eventually the node will exhaust its owned presignatures. At that point
`take_owned()` blocks indefinitely (waiting for a presignature that will
never arrive), and the node stops being able to lead signature
computations. It can still participate as a **follower**, since
`take_unowned(id)` does not depend on the local generation loop.

## Appendix

### Triple generation loop

Source: `crates/node/src/providers/ecdsa/triple.rs`

```
loop {
    update metrics
    if num_owned + in_flight < desired_triples_to_buffer
       AND in_flight < concurrency * 2 * 64:
        pick threshold random online participants
        reserve batch of 64 IDs
        spawn async task (semaphore-limited to `concurrency`):
            run MPC triple protocol → 64 triples
            pair and store each as owned
        sleep(parallel_triple_generation_stagger_time_sec)
        continue

    if store is full (num_owned == desired):
        maybe_discard_owned(32)   // clean out unusable assets

    sleep 100ms
}
```

- Triples are generated in **batches of 64** (`SUPPORTED_TRIPLE_GENERATION_BATCH_SIZE`).
  Each batch produces 64 raw triples which are paired into 32 `PairedTriple` values.
- The leader selects `threshold` random online participants.
- `InFlightGenerationTracker` counts how many triples are "in flight"
  (spawned but not yet completed) using an atomic counter with a drop guard.
- A tokio semaphore limits actual concurrent protocol executions to
  `config.concurrency`.
- A stagger delay (`parallel_triple_generation_stagger_time_sec`) between
  successive spawns prevents all tasks from starting at the same time
  and competing for resources.

### Presignature generation loop

Source: `crates/node/src/providers/ecdsa/presign.rs`

```
loop {
    update metrics + progress tracker
    if num_owned + in_flight < desired_presignatures_to_buffer
       AND in_flight < concurrency * 2:
        reserve 1 ID
        take_owned triple (BLOCKS until one is available)
        participant set = participants from that triple
        spawn async task (semaphore-limited to `concurrency`):
            run MPC presign protocol → 1 presignature
            store as owned presignature
        continue

    if store is full (num_owned == desired):
        maybe_discard_owned(1)

    sleep 100ms
}
```

- Presignatures are generated **one at a time**.
- `take_owned()` on the triple store will **block** if no online triples
  are available.
- The progress tracker reports whether the loop is "waiting for triples".