# MPC Node Deployment in TDX - External Guide ## Introduction Chain signatures is a Multi-Party Computation (MPC) service that lets you sign arbitrary payloads by calling a smart contract and receiving a signature. The returned signature can be used to derive public keys on external chains (for example, Ethereum or Bitcoin). This guide walks you through deploying a self-hosted MPC node on a bare-metal server using Intel Trusted Domain Extensions (TDX). The node runs inside a Confidential VM (CVM) for isolation and attestation. We use Dstack (from Phala) to orchestrate the environment and run the MPC container inside the CVM. ## Limitations and Restrictions **Important:** The CVM filesystem is encrypted with a hardware-bound key derived from SGX sealing, so copying the CVM or disk data to another machine will not work and may result in data loss, including loss of key shares and P2P identity keys. Platform-bound sealed data may also become unrecoverable if TDX-related hardware changes (for example, a CPU replacement). To move a node between hosts, follow the supported procedure described in the [Node Migration](./node-migration-guide.md) section, which uses the backup-cli tool to securely transfer key shares. ## Main difference between TEE and non TEE MPC nodes From an operator’s perspective, the key differences between a **TEE-based** MPC node and a **non-TEE** node are: For a full architecture review of the TEE-based MPC, see: [design doc](securing-mpc-with-tee-design-doc.md) | Category/feature | Non TEE | With TEE | | :---- | :---- | :---- | | | | | | Hardware Setup | Any hardware that meets the bandwidth, compute and memory requirements. | Additional requirements to support Intel TDX Hardware. | | Deployment | Deploy an MPC Docker image directly. | Use Dstack interface to deploy a Launcher docker image that will deploy the MPC node. | | Node Account key and P2P key | Generated by the operator, and provided to the node. | Generated by the node. Private keys are never exposed outside of the CVM. Public keys are exported by the node. And added to the operator's account. | | Debug, Recovery| Operator has complete control over the environment | Operator has no access into the CVM (except for defined entry points) | ## Prerequisites and Requirements ### Hardware Requirements * Have a TDX enabled, bare metal, server. Note - we currently only support bare metal and do not support virtualized TDX solutions (such as GCP). * Intel Xeon 5th/6th Generation CPU (TDX Support) and 8 RAM slots filled See [Intel TDX HW requirements](https://cc-enabling.trustedservices.intel.com/intel-tdx-enabling-guide/03/hardware_selection/) The memory, cores, and disk below are the resources consumed by a single MPC CVM. Because you may need to run two CVMs concurrently while migrating to a new launcher version, size your TDX host for at least **2x** these values, plus some margin. * Memory - 64GB per CVM * (v)Cores - 8 per CVM * Disk space - 1TB (1000 GB) per CVM, SSD NVMe or similar performance For a list of supported cloud providers offering bare metal servers with Intel TDX, see [Cloud Providers Supporting Bare Metal Servers with Intel TDX](./cloud-providers-tdx.md). > **Sharing one host between mainnet and testnet?** See [Running multiple MPC nodes on one host](./running-multiple-mpc-nodes-on-one-host.md) for the additional setup (one `dstack-vmm` hosting both CVMs, with each CVM bound to a distinct host IP at port-forward time). Note: this setup is discouraged as it couples mainnet and testnet availability — a single failure takes both nodes offline. ### Software Requirements * [`near-cli-rs`](https://github.com/near/near-cli-rs) — install per the upstream README; the `near` binary must be on your `$PATH`. ### General * Firewall:allow ingress port 80 (MPC), 24567 (near) and port 8080 (web) * Assign a static public IP for access towards machine from outside ### Create DNS A record (optional) Although a node can be accessed using a public IP address, it is recommended to use a domain name instead. Using a domain name allows some flexibility in case of public IP address change/repurpose or failover scenarios. To use a domain name, one must register a DNS A record. Some recommended providers: * [Namecheap](https://www.namecheap.com/support/knowledgebase/article.aspx/319/2237/how-can-i-set-up-an-a-address-record-for-my-domain/) * [Cloudflare](https://developers.cloudflare.com/dns/manage-dns-records/how-to/create-dns-records/) ### TDX and Dstack Setup This section describes how to enable TDX on your machine (BIOS, operating system, and software configurations), and how to install and configure Dstack. Follow the three steps below to ensure you have a working TDX machine with Dstack configured: 1. [Set up a bare-metal TDX server](#1-tdx-bare-metal-server-setup) 2. [Dstack Setup and Configuration](#2-dstack-setup-and-configuration) 3. [Set up a Local Gramine-Sealing-Key-Provider](#3-local-gramine-sealing-key-provider-setup) --- #### 1. TDX Bare-Metal Server Setup To create a bare-metal TDX server, follow the [canonical/tdx guide](https://github.com/canonical/tdx/blob/9023cb2d952f5fe9d72004092b93a155482ba18a/README.md). Make sure to complete: * Steps 1–4 (basic TDX setup configuration) * Steps 9.1–9.2 (Remote Attestation setup) This ensures that your TDX setup is correctly configured. --- #### 2. Dstack Setup and Configuration This section will guide you through installing and configuring `dstack-vmm`, which is the only dstack component needed for running MPC nodes in TDX environments. The instructions are based on the [dstack deployment guide](https://github.com/Dstack-TEE/dstack/blob/eab86e8a3fd934656946a39bf07849bd75cf20fb/docs/deployment.md). **Prerequisites:** * Follow the [TDX setup guide](https://github.com/canonical/tdx) to setup the TDX host (completed in step 1 above) * Install system dependencies ```bash sudo apt update sudo apt install build-essential qemu-system-x86=1:8.2.2* docker.io docker-compose-v2 docker-buildx ``` > **Note:** `docker-compose-v2` and `docker-buildx` are required by the > key-provider build (`key-provider-build/run.sh` runs `docker compose up > --build`); the `docker.io` package alone does not include them. > **Note:** The QEMU version is pinned to **8.2.2** because TDX attestation measurements > (MRTD/RTMR0) depend on the QEMU version. Using a different version will produce different > measurements and attestation might fail. * Create `mpc` user and installation folder ```bash # allow the MPC user to access docker and KVM for running CVMs. sudo useradd -m -G docker,kvm -s /usr/bin/bash mpc sudo passwd mpc # create installation folder sudo mkdir /opt/mpc sudo chown mpc:mpc /opt/mpc # change user sudo -u mpc -s cd /opt/mpc ``` * Install `cargo` and `rustc`. This can be done with the following command. ```bash curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` Note that after running this command you might need to restart the shell. **Installation Steps:** All steps below assume the current user is `mpc` and the current directory is `/opt/mpc`. > **dstack versions.** `dstack-vmm` and the **guest OS image** are both > **v0.5.8** (their `MRTD`/`RTMR0-2` are pinned on-chain, so use exactly that > version). Only the **`key-provider-build/`** recipe is taken from dstack > **v0.5.11** (see the [key-provider step](#3-local-gramine-sealing-key-provider-setup)), > which builds a reproducible `mr_enclave` from build-args. ```bash git clone https://github.com/Dstack-TEE/dstack ``` 2. **Compile dstack-vmm:** ```bash cd dstack git checkout 368c62e7de5d4016bd75332824aa7f2ef1d7d19e # dstack v0.5.8 cargo build --release -p dstack-vmm -p supervisor mkdir -p vmm-data cp target/release/dstack-vmm vmm-data/ cp target/release/supervisor vmm-data/ cd vmm-data/ ``` 3. **Create the VMM configuration file:** ```bash cat < vmm.toml # Communication endpoint for the VMM: # - Use an IP and port when you want to access the VMM web UI from a browser (recommended) address = "127.0.0.1" port = 10000 # - Or use a UNIX socket if you don’t need the web UI and prefer not to expose an IP/port # address = "unix:./vmm.sock" reuse = true image_path = "./images" run_path = "./run/vm" [cvm] kms_urls = ["https://kms.test2.dstack.phala.network:9201"] gateway_urls = [] cid_start = 30000 cid_pool_size = 1000 max_disk_size = 1000 [cvm.port_mapping] enabled = true address = "127.0.0.1" range = [ { protocol = "tcp", from = 1, to = 30000 }, { protocol = "udp", from = 1, to = 30000 }, ] [host_api] address = "vsock:2" port = 9300 EOF ``` 4. **Download Guest OS images:** ```bash DSTACK_VERSION=0.5.8 wget "https://github.com/Dstack-TEE/meta-dstack/releases/download/v${DSTACK_VERSION}/dstack-${DSTACK_VERSION}.tar.gz" mkdir -p images/ tar -xvf dstack-${DSTACK_VERSION}.tar.gz -C images/ rm -f dstack-${DSTACK_VERSION}.tar.gz ``` **Configuration Notes:** * KMS and Gateway are not used in this MPC setup * The configuration includes port **24567** which is required for MPC nodes * The `max_disk_size = 1000` setting is specifically required for MPC operations ##### VMM service persistence To run dstack-vmm, you can start it manually with: ```bash ./dstack-vmm -c vmm.toml ``` When `dstack-vmm` is running, you should be able to access its web interface at port `10000`. However, for persistent operation, we recommend using the following systemd service: ```text [Unit] Description=Daemon for dstack-vmm [Service] Type=simple WorkingDirectory=/opt/mpc/dstack/vmm-data/ ExecStart=/opt/mpc/dstack/vmm-data/dstack-vmm -c vmm.toml Restart=on-failure RestartSec=5 User=mpc Group=mpc [Install] WantedBy=multi-user.target ``` The file should be located in the user systemd config folder, for example `/etc/systemd/system/dstack-vmm.service`. After the file is created or modified you must run: ```bash # to reload the service files sudo systemctl daemon-reload # to start/stop/restart the service sudo systemctl start/stop/restart dstack-vmm # to check the status the service systemctl status dstack-vmm ``` Notice that some of the commands require `sudo`, so they cannot be run using the `mpc` user which has no such permissions by default. --- ##### Guest OS Image (optional) > **Important:** The guest OS image that runs inside the CVM must be **identical across all nodes**. > The image is **measured**, and those measurements are **hardcoded in the contract**. The guest OS image was downloaded automatically during **Step 4** of the installation process using version **0.5.8**. This version ensures **compatibility** and **reproducibility** across all MPC nodes. If you need to **verify**, **re-download**, or **rebuild** the image, follow one of the methods below. --- ###### Option 1 — Re-download the pre-built image Use this method to retrieve the official pre-built image provided by the Dstack project. ```bash DSTACK_VERSION=0.5.8 wget "https://github.com/Dstack-TEE/meta-dstack/releases/download/v${DSTACK_VERSION}/dstack-${DSTACK_VERSION}.tar.gz" mkdir -p images/ tar -xvf dstack-${DSTACK_VERSION}.tar.gz -C images/ rm -f dstack-${DSTACK_VERSION}.tar.gz ``` This ensures you are using the verified release image corresponding to version **0.5.8**. --- ###### Option 2 — Build the image from source This method is intended for advanced users who wish to inspect, rebuild, or reproduce the image for verification purposes. 1. **Clone and check out the exact release:** ```bash git clone https://github.com/Dstack-TEE/meta-dstack.git cd meta-dstack/ git checkout 48dd3df6f443bfe25a65701d4453fb7cf9c3dbb9 git submodule update --init --recursive ``` 2. **Choose one of the following actions:** - **Download the pre-built image (recommended, faster):** ```bash ./build.sh dl 0.5.8 ``` - **Build a reproducible image from source (slower, ~1–2 hours):** ```bash cd repro-build && ./repro-build.sh -n ``` ###### Verification Steps Run these commands from inside your image folder (e.g., `dstack-0.5.8`). **1. Verify file hashes against expected values:** ```bash #!/usr/bin/env bash set -euo pipefail # Hard-coded expected hashes declare -A EXPECTED EXPECTED["ovmf.fd"]="76888ce69c91aed86c43f840b913899b40b981964b7ce6018667f91ad06301f0" EXPECTED["bzImage"]="2afe5b0571363fe2278a3438e337630bfeffc74bafba3d116630e2a1ef1805f3" EXPECTED["initramfs.cpio.gz"]="1272ab4b10db1933d02a80059fbb94b4be9eb4af8c4f79e739dfc0b0101acc40" EXPECTED["metadata.json"]="20fde70b9e4f31ab6ef55d8a5bf33b1734593a9e605982c510c0963d69af075b" ALL_OK=1 for FILE in "${!EXPECTED[@]}"; do if [[ ! -f "$FILE" ]]; then echo "$FILE not found" ALL_OK=0 continue fi CALC_HASH=$(sha256sum "$FILE" | awk '{print $1}') if [[ "$CALC_HASH" == "${EXPECTED[$FILE]}" ]]; then echo "$FILE: OK" else echo "$FILE: FAILED" echo " Expected: ${EXPECTED[$FILE]}" echo " Found: $CALC_HASH" ALL_OK=0 fi done if [[ $ALL_OK -eq 1 ]]; then echo "All files verified successfully" else echo "One or more files did not match" fi ``` **2. Verify the `rootfs.img.verity`:** ```bash #!/usr/bin/env bash set -euo pipefail META="metadata.json" # Extract values from metadata.json ROOTFS=$(jq -r '.rootfs' "$META") ROOTFS_SIZE=$(jq -r '.cmdline' "$META" | sed -n 's/.*dstack.rootfs_size=\([0-9]*\).*/\1/p') ROOTFS_HASH=$(jq -r '.cmdline' "$META" | sed -n 's/.*dstack.rootfs_hash=\([a-f0-9]*\).*/\1/p') # Compute parameters BLOCK_SIZE=4096 DATA_BLOCKS=$(( ROOTFS_SIZE / BLOCK_SIZE )) HASH_OFFSET=$ROOTFS_SIZE echo "Verifying $ROOTFS" echo " Rootfs size: $ROOTFS_SIZE bytes" echo " Data blocks: $DATA_BLOCKS" echo " Hash offset: $HASH_OFFSET" echo " Rootfs hash: $ROOTFS_HASH" # Run verification veritysetup verify \ --data-blocks=$DATA_BLOCKS \ --hash-offset=$HASH_OFFSET \ --data-block-size=$BLOCK_SIZE \ --hash-block-size=$BLOCK_SIZE \ "$ROOTFS" "$ROOTFS" "$ROOTFS_HASH" echo "✅ Verification succeeded" ``` **3. Check actual vs. expected RTMR values:** 1. Build `dstack-mr` tool (using Docker). 2. Calculate the actual RTMRs of the image. 3. Compare against expected RTMRs from the contract (shown below). For more details, see the [Dstack attestation guide](https://github.com/Dstack-TEE/dstack/blob/master/attestation.md). Build `dstack-mr` docker image: ```bash cd /opt/mpc/dstack/vmm-data/images/dstack-0.5.8 ``` Create a Dockerfile file with the following contents: ```shell # Dockerfile FROM rust:1.86.0@sha256:300ec56abce8cc9448ddea2172747d048ed902a3090e6b57babb2bf19f754081 AS kms-builder # Pinned to a commit (immutable) rather than a tag (which can be moved). This is # the dstack v0.5.8 commit — matches the guest OS image this guide uses (the # `dstack-0.5.8/` dir you cd into below). For a different OS-image version, # override with the matching commit: --build-arg DSTACK_REV= ARG DSTACK_REV=368c62e7de5d4016bd75332824aa7f2ef1d7d19e WORKDIR /build RUN apt-get update && \ apt-get install -y --no-install-recommends \ git \ build-essential \ musl-tools \ libssl-dev \ protobuf-compiler \ libprotobuf-dev \ clang \ libclang-dev RUN git clone https://github.com/Dstack-TEE/dstack.git RUN rustup target add x86_64-unknown-linux-musl # Build dstack-mr from the SAME dstack version as your guest OS image (DSTACK_REV). # Building from master is not reproducible: the measurement logic changes between # releases and may compute different MRTD/RTMR values than the on-chain set was # generated with. RUN cd dstack && git checkout "${DSTACK_REV}" && \ cargo build --release -p dstack-mr-cli --target x86_64-unknown-linux-musl # kvin/kms supplies the `dstack-acpi-tables` helper (feeds RTMR0). Pinned by digest # (not the moving `:latest`) for reproducible measurements: this digest reproduces # the MRTD/RTMR0-2 published on-chain in `v1.signer`'s `allowed_os_measurements` # (the same values shown in the example output below), so an operator can re-check # it. It is a third-party image; a NEAR/Dstack-owned published image would be a # better long-term trust root. FROM kvin/kms@sha256:ad6a8c5c43aed7278e665cd0960ae5be95060847f7d517633be685cabda95a3d COPY --from=kms-builder /build/dstack/target/x86_64-unknown-linux-musl/release/dstack-mr /usr/local/bin/ ENTRYPOINT ["dstack-mr"] CMD [] ``` Build. Pass `DSTACK_REV` = the **commit** matching your guest OS image's dstack version (here the v0.5.8 commit), so `dstack-mr`'s measurement logic matches the image and the on-chain values: ```bash docker build --build-arg DSTACK_REV=368c62e7de5d4016bd75332824aa7f2ef1d7d19e -t dstack-mr . ``` Run: > **Important:** MRTD and RTMR0 depend on the QEMU version installed on the TDX host. > The `dstack-mr` tool must be told this version via `--qemu-version`, otherwise it > produces incorrect MRTD/RTMR0 values. > > The MPC contract is tested and verified with **QEMU 8.2.2**. > Verify your version: > ```bash > qemu-system-x86_64 --version > # Expected: QEMU emulator version 8.2.2 > ``` > If your version is not **8.2.2**, attestation might fail. ```bash docker run --rm \ -v "$(pwd)":/dstack-0.5.8 \ dstack-mr \ measure -c 8 -m 64G --qemu-version 8.2.2 /dstack-0.5.8/metadata.json ``` Example output: ```text Machine measurements: MRTD: f06dfda6dce1cf904d4e2bab1dc370634cf95cefa2ceb2de2eee127c9382698090d7a4a13e14c536ec6c9c3c8fa87077 RTMR0: e673be2f70beefb70b48a6109eed4715d7270d4683b3bf356fa25fafbf1aa76e39e9127e6e688ccda98bdab1d4d47f46 RTMR1: b598fde9491427341bc4683b75d10d3e36770af3a36a6954d8b6b7b22aa66358f13e1f172e51b7d6e6710d99a8d8532f RTMR2: 9284cde236231d5ddace01104a440fd504df5182a2ad1ac3d2138b80c6a7864bd2c30f69041d8264217f3d24541580cf ``` --- ##### MPC-Specific Configurations The `vmm.toml` configuration provided in the installation steps above already includes a few necessary MPC-specific settings: * Port **24567** is included in the `cvm.port_mapping.range` (1-30000) * port **80** is used by default for MPC node to node communication. To allow binding to port 80, run the following command: ```bash sudo setcap 'cap_net_bind_service=+ep' $(which qemu-system-x86_64) ``` * The `max_disk_size = 1000` setting is configured in the `[cvm]` section * KMS and Gateway are not used in this MPC setup If you need to modify the configuration later, ensure these settings remain in your `vmm.toml`: ```toml [cvm] max_disk_size = 1000 [cvm.port_mapping] enabled = true address = "127.0.0.1" range = [ { protocol = "tcp", from = 1, to = 30000 }, { protocol = "udp", from = 1, to = 30000 }, ] ``` #### 3. Local Gramine-Sealing-Key-Provider Setup In this solution, we use the `gramine-sealing-key-provider`, which runs inside an SGX enclave, to generate a key. This key is derived from TDX measurements and the SGX enclave’s hardware sealing key, and it is used to encrypt the CVM’s file system. > **Note:** This key is tied to the platform. Losing it will prevent the CVM from decrypting the drive on subsequent VM boots. For more information, see [local-key-provider-from-phala](https://github.com/Dstack-TEE/dstack/tree/master/kms#local-key-provider-mode-1). ##### Setup Instructions 1. Follow the [canonical/tdx setup](#1-tdx-bare-metal-server-setup) if not already completed — especially step 9.1–2 (establishing an SGX PCCS: Provisioning Certification Caching Service). 2. Deploy an instance of `gramine-sealing-key-provider` on the host machine. * On the TDX server, run the script [run.sh](https://github.com/Dstack-TEE/dstack/blob/master/key-provider-build/run.sh) > **Prerequisite:** Docker must be installed. Build the key provider with the canonical apt snapshot date so the resulting `mr_enclave` is reproducible and matches the value attested by the contract. Pass the date via the `APT_SNAPSHOT` build-arg (the Rust toolchain version and `rustup-init` are pinned inside the recipe). The `APT_SNAPSHOT` build-arg is available in `key-provider-build/` as of dstack **v0.5.11**. Since `dstack-vmm` and the OS image stay on v0.5.8, add a v0.5.11 worktree just for this build (it does not change your vmm or image): ```bash cd /opt/mpc/dstack git worktree add ../dstack-v0.5.11 v0.5.11 cd /opt/mpc/dstack-v0.5.11/key-provider-build APT_SNAPSHOT=20260423T000000Z ./run.sh ``` > The snapshot date `20260423T000000Z` is the specific value that reproduces > the canonical `mr_enclave` below; the build host must be able to reach > `snapshot.ubuntu.com`. Don't change it — it changes the key provider's > `mr_enclave`, which is measured and attested on-chain. **Quote collateral / PCCS.** The key provider's `aesmd` fetches this platform's PCK certificate via `key-provider-build/sgx_default_qcnl.conf`, which defaults to **Phala's public PCCS** (`pccs.phala.network`). **Recommended for production: point this at your own local PCCS instead.** Relying on a third-party public PCCS to boot the CVM puts an external service on the critical attestation-collateral path — a single point of failure for a step that gates the CVM's disk-encryption key. Running your own PCCS (step 9.1–2 / [Appendix: Self-hosting a local PCCS](#appendix-self-hosting-a-local-pccs)) removes that dependency. Save the following into `key-provider-build/sgx_default_qcnl.conf` (replacing its contents) **before** running `./run.sh`: ```json { "pccs_url": "https://localhost:8081/sgx/certification/v4/", "use_secure_cert": false, "retry_times": 6, "retry_delay": 10, "pck_cache_expire_hours": 168, "verify_collateral_cache_expire_hours": 168, "local_cache_only": false } ``` 3. To find the `mr_enclave` value of the SGX key provider, run: ```bash docker logs gramine-sealing-key-provider 2>&1 | grep mr_enclave | head -n 1 ``` Ensure that the `mr_enclave` matches the expected value: ```console 6b5ed02e549a1c30aaa8e3171a045f1f449b0017353ef595e78e39c348c98d01 ``` **Note**: As part of the mutual attestation between the CVM and the key provider, the CVM will check that the key provider’s `mr_enclave` matches the above hash. 4. Verify both key-provider containers are running: ```bash docker ps --filter name=aesmd --filter name=gramine-sealing-key-provider \ --format 'table {{.Names}}\t{{.Status}}' ``` Both should show `Up` (`gramine-sealing-key-provider` listens on `127.0.0.1:3443`). The bundled `sgx_default_qcnl.conf` works for any platform **registered with Intel** (step 9.2); if `gramine-sealing-key-provider` instead crash-loops (`Restarting`) with `AESM service returned error 44`, the platform is almost always not registered — see [Troubleshooting](#gramine-sealing-key-provider-crash-loops-with-aesm-service-returned-error-44). ## MPC Node Setup and Deployment This section will describe how to configure and deploy your MPC node inside a CVM. Including * Creating a NEAR account for your node * Preparing a configuration file based on [user-config.toml](https://github.com/near/mpc/blob/main/deployment/cvm-deployment/user-config.toml) * Rendering the launcher docker compose template [launcher_docker_compose.yaml.template](https://github.com/near/mpc/blob/main/crates/contract/assets/launcher_docker_compose.yaml.template) with the launcher and MPC node manifest digests approved by the contract. * Configuring and starting your CVM with the MPC node. * Accessing mpc docker logs. * Retrieve keys from the CVM. * Verify the node's attestation before trusting the keys. * Add the node key to your NEAR account. ### Create a NEAR Account for Your Node > **Important** – In the following examples, the account keys are auto-generated as part of the command. But it is also possible to create the keys separately and add them to the account creation command. > > In either case, it is the operator's full responsibility to manage and protect these keys. > > See the [NEAR CLI](https://github.com/near/near-cli-rs/blob/main/docs/GUIDE.en.md) documentation for more options and details. #### **Mainnet** A named mainnet account is required. To create one, you can use a known wallet like [https://www.mynearwallet.com](https://www.mynearwallet.com) or [https://wallet.meteorwallet.app](https://wallet.meteorwallet.app), or fund it yourself: ```bash near account create-account fund-myself ' NEAR' autogenerate-new-keypair save-to-keychain sign-as network-config mainnet ``` #### **Testnet** If you're using testnet, the easiest way to get started is to create an account sponsored by the faucet — the [NEAR command line interface](https://github.com/near/near-cli-rs) can set this up for you. Public and private keys are generated during this process. You can also use an existing account if you have one. Using auto-generated keypair: ```bash near account create-account sponsor-by-faucet-service autogenerate-new-keypair save-to-keychain network-config testnet create ``` Using a manually provided public key: ```bash near account create-account sponsor-by-faucet-service use-manually-provided-public-key network-config testnet create ``` For more details, please refer to the NEAR account documentation. ### Prepare MPC Node Configuration Create a `user-config.toml` file based on the [user-config.toml](https://github.com/near/mpc/blob/main/deployment/cvm-deployment/user-config.toml) template. The configuration has two sections: `[launcher_config]` for the launcher and `[mpc_node_config]` for the MPC node. ```toml [launcher_config] image_reference = "nearone/mpc-node" port_mappings = [ { host = 80, container = 80 }, { host = 8080, container = 8080 }, { host = 8079, container = 8079 }, { host = 3030, container = 3030 }, { host = 24567, container = 24567 }, ] [mpc_node_config] home_dir = "/data" # PCCS endpoints for TDX attestation collateral. Tried in order until # one succeeds. To customize (e.g. add a self-hosted local PCCS), see # [Customizing PCCS endpoints](#customizing-pccs-endpoints-optional) below. [[mpc_node_config.pccs_endpoints]] url = "https://pccs.phala.network/" [[mpc_node_config.pccs_endpoints]] url = "https://api.trustedservices.intel.com/" # NEAR node bootstrap. Applied on first init only — see the field notes below. [mpc_node_config.near_init] chain_id = "$CHAIN_ID" # "testnet" or "mainnet" boot_nodes = "$BOOT_NODES" # comma-separated; see the curl snippet below download_genesis = true download_config = "rpc" tier3_public_addr = "$IP:24567" # required — your node's public IP; state sync needs a reachable advertised address external_storage_fallback_threshold = 100 # required for node versions before 3.12.0; ignored on 3.12.0+ [mpc_node_config.secrets] secret_store_key_hex = "$SECRET_STORE_KEY" backup_encryption_key_hex = "$BACKUP_ENCRYPTION_KEY" [mpc_node_config.node] my_near_account_id = "$MY_MPC_NEAR_ACCOUNT_ID" near_responder_account_id = "$MY_MPC_NEAR_ACCOUNT_ID" migration_web_ui = "0.0.0.0:8079" # required; matches the port-forward above [mpc_node_config.node.indexer] mpc_contract_id = "$CONTRACT_ID" # v1.signer-prod.testnet for testnet, v1.signer for mainnet validate_genesis = false sync_mode = "Latest" finality = "optimistic" concurrency = 1 port_override = 80 # MPC P2P convention [mpc_node_config.log] format = "plain" filter = "mpc=debug,info" ``` The snippet above shows only the fields you are likely to change. Required fields not shown (e.g. `number_of_responder_keys`, `web_ui`, and the `triple` / `presignature` / `signature` / `ckd` / `foreign_chains` blocks) and inline `# mainnet: …` swap hints are inherited from the [`user-config.toml`](https://github.com/near/mpc/blob/main/deployment/cvm-deployment/user-config.toml) template — always start from that file and edit the highlighted fields rather than building a config from this snippet alone. > **⚠️ Set `tier3_public_addr` before first start.** State sync is decentralized (peer-to-peer) and requires the node to advertise a **publicly reachable** `IP:24567`. The template ships `tier3_public_addr` as a `REPLACE_WITH_…` placeholder and the node **fails to start if it's left unset or left as the placeholder** — replace it with the IP your dstack port-forward exposes for `:24567`. This matters most on hosts with more than one external IP or running multiple nodes, where auto-discovery would advertise an unreachable address and state sync would stall. It is applied at first init only, so getting it right up front avoids a CVM redeploy later. Adjust the variables as per your environment. * `image_reference` — the Docker image reference. The actual image version is determined by the manifest digest from the contract (stored in the approved hashes file), not by a tag. A tag may be appended for readability (e.g., `"nearone/mpc-node:3.8.1"`) but is ignored during pull. * `my_near_account_id` — use the NEAR account ID created in the previous step * `mpc_contract_id` — **v1.signer-prod.testnet** for testnet, **v1.signer** for mainnet * `migration_web_ui` — bind address for the migration HTTP endpoint, used by the [Node Migration](./node-migration-guide.md) flow. Required. Keep at `0.0.0.0:8079` to match the port-forward and the `--mpc-node-address …:8079` form the migration guide uses. * `port_mappings` — port forwarding rules for the MPC container. These should be a subset of the port forwarding for the CVM defined in the [Using the Web Interface](#using-the-web-interface) section. * `tier3_public_addr` *(under `[mpc_node_config.near_init]`)* — `IP:24567` the node advertises for decentralized (Tier3) state-sync responses. **Required — the template ships this as a `REPLACE_WITH_…` placeholder and the node fails to start if it's left unset or left as the placeholder** (intentionally, so state sync never silently runs with an unreachable advertised address). It is especially critical on any host with more than one external IP, or when running [multiple nodes on one host](./running-multiple-mpc-nodes-on-one-host.md): otherwise the node auto-discovers its advertised address as the host's default-route outbound IP, which peers may not be able to reach, and state sync stalls. Applied at first init only; changing later requires a CVM redeploy via the [Node Migration](./node-migration-guide.md) flow. * `external_storage_fallback_threshold` *(under `[mpc_node_config.near_init]`)* — **required for node versions before 3.12.0; leave at `100`.** Number of decentralized (peer-to-peer) state-sync attempts per state part before falling back to the external storage bucket. Released 3.11.x nodes default this to `0` (bucket-only), which no longer works now that nearcore has stopped serving the centralized buckets — leaving it unset makes the node crash in a state-sync restart loop (last log line `running state sync shard_id=5`). Ignored on 3.12.0+, which always uses peer-to-peer state sync, so it is safe to keep set across the upgrade. Applied at first init only, like `tier3_public_addr`. * `near_init.boot_nodes` — comma-separated NEAR boot-node list. The testnet template at `deployment/cvm-deployment/user-config.toml` already ships with a working testnet boot-node list, so testnet operators usually don't need to fetch a fresh one. For **mainnet** (or to refresh testnet), select boot nodes from the Testnet/Mainnet RPC endpoints and copy at least 4-5 of them into this field. **Important:** Boot nodes must not contain duplicate addresses or peer IDs. Duplicates will cause the node to crash on startup. The command below deduplicates automatically: ```bash curl -s -X POST https://rpc.[testnet|mainnet].near.org \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "method": "network_info", "params": [], "id": "dontcare" }'| \ jq -r '.result.active_peers | unique_by(.addr) | unique_by(.id) | map("\(.id)@\(.addr)") | .[]' |\ paste -sd',' - ``` #### Customizing PCCS endpoints (optional) The MPC node fetches Intel-signed attestation collateral from a PCCS. The example above lists Phala and Intel as `pccs_endpoints` — the node tries each in order until one succeeds. This works out of the box for most operators. To customize (e.g. add a self-hosted local PCCS as the primary), modify the `pccs_endpoints` array. Whatever entries you list become the entire fallback chain, in order — no defaults are auto-inserted. For a self-hosted local PCCS, see [Appendix: Self-hosting a local PCCS](#appendix-self-hosting-a-local-pccs). ### Preparing a Docker Compose File The launcher Docker Compose file is **rendered** at deploy time from the template at [`crates/contract/assets/launcher_docker_compose.yaml.template`](https://github.com/near/mpc/blob/main/crates/contract/assets/launcher_docker_compose.yaml.template) by substituting two manifest digests: 1. The **launcher** image digest (`{{LAUNCHER_IMAGE_HASH}}`) 2. The **MPC node** image digest (`{{DEFAULT_IMAGE_DIGEST_HASH}}`) Both must be approved by the contract — otherwise the SHA256 of the rendered file will not match an entry in `allowed_launcher_compose_hashes` and the node's attestation will be rejected. #### Step 1 — discover the currently-allowed digests ```bash # MPC node digest — LATEST = FIRST element of the returned vector # (goes into DEFAULT_IMAGE_DIGEST in the rendered compose) near contract call-function as-read-only \ v1.signer-prod.testnet allowed_docker_image_hashes \ json-args '{}' network-config testnet now # Launcher digest — LATEST = LAST element (entries are returned in # insertion order, oldest first) # (goes into the launcher `image` line in the rendered compose) near contract call-function as-read-only \ v1.signer-prod.testnet allowed_launcher_image_hashes \ json-args '{}' network-config testnet now ``` For details on how to verify each digest before trusting it, see [MPC Node Image Upgrade](#mpc-node-image-upgrade) and [Launcher image voting](#launcher-image-voting). #### Step 2 — render the template > **If you plan to deploy via `deploy-launcher.sh`** (the script path > below): skip this step. Instead, uncomment and set the > `LAUNCHER_MANIFEST_DIGEST` and `MPC_MANIFEST_DIGEST` lines in your > `.env` file. The script renders the template internally; you do not > need to produce a `launcher_docker_compose.yaml` here. The manual render (needed if you deploy via the Dstack Web interface): ```bash # Paste either form — with or without 'sha256:' prefix. The substitutions # below strip the prefix if present, so the rendered file always has # exactly one 'sha256:' per digest line. export LAUNCHER_IMAGE_HASH= export MPC_IMAGE_HASH= sed \ -e "s|{{LAUNCHER_IMAGE_HASH}}|${LAUNCHER_IMAGE_HASH#sha256:}|g" \ -e "s|{{DEFAULT_IMAGE_DIGEST_HASH}}|${MPC_IMAGE_HASH#sha256:}|g" \ crates/contract/assets/launcher_docker_compose.yaml.template \ > launcher_docker_compose.yaml ``` > **Note:** The rendered file is measured, and its SHA256 is part of the > remote attestation. Do not modify the rendered file further (including > whitespace) — only the two digest substitutions should differ from the > template. ### Required Ports and IP Allocation MPC nodes bind a fixed set of ports (listed below). Two supported deployment shapes: * **One node per host**: the host's primary public IP carries all port forwards. * **Multiple nodes on one host** (mainnet + testnet): each CVM is bound to its own public IP on the same host via dstack's per-port-mapping `host_address`. See [Running multiple MPC nodes on one host](./running-multiple-mpc-nodes-on-one-host.md). --- #### Required Ports | Port | Purpose | |--------|-------------------------------------------------------------------------| | **80** | Node-to-node communication (port override convention) | | **24567** | Decentralized state sync | | **8080** | Debug and telemetry collection, plus the `/public_data` endpoint | | **3030** | Debug and telemetry collection | | **8079** | Migration port | ### Configuring and Starting the MPC Binary in a CVM There are 2 ways to manage the VM that will run the MPC node. 1. Using the Web interface 2. Using a script. Note - both methods provide the same functionality. The Web interface provides a more manual approach and control. While the script is useful for automating processes. #### **Using the Web interface** Follow the [Dstack guide](https://github.com/Dstack-TEE/dstack?tab=readme-ov-file#deploy-an-app) (deploy an App): Use the following custom settings for MPC: 1. Launcher docker compose file - provided above. 2. VM HW setting (use exactly those settings, since vCPU/Memory are measured): vCPU number=8, Memory = 64GB, disk = 1000 GB 3. Pre-launch Script and Init Script - both must be empty (a non-empty script fails attestation). Caution: the Pre-launch Script may not be empty by default - clear it before deploying. 4. user-config - provided above 5. Toggles: - KMS = disable - Local Keyprovider = enabled - Tproxy = disable - public logs = enabled - public sysinfo = enabled - pin NUMA = disabled 6. Port mapping (format: `:` → ``): Public 0.0.0.0:80 → 80 (main node to node communication port) Public 0.0.0.0:24567 → 24567 (required for decentralized state sync) Public 0.0.0.0:8080 → 8080 (required for collecting debug and telemetry information) Public 0.0.0.0:8079 → 8079 (required for the node-migration HTTP endpoint) Local 127.0.0.1:3030 → 3030 (use a public host address if you want the debug metrics available on the internet) Local 127.0.0.1: → 8090 (required for access CVM information and container logs) The **host address** is the IP qemu binds each forward to. Single-node deployments use `0.0.0.0` to bind on every host interface. **Multi-node deployments** (mainnet + testnet on one host) use a specific public IP per CVM — see [Running multiple MPC nodes on one host](./running-multiple-mpc-nodes-on-one-host.md). 7. Key Provider ID: (The MrEnclave for the sgx local key provider) 6b5ed02e549a1c30aaa8e3171a045f1f449b0017353ef595e78e39c348c98d01 ![VMM Web Page (1/2)](./attachments/VMM_web_page_deploy_1.png) ![VMM Web Page (2/2)](./attachments/VMM_web_page_deploy_2.png) #### Using the script The [`deploy-launcher.sh`](https://github.com/near/mpc/blob/main/deployment/cvm-deployment/deploy-launcher.sh) helper in `deployment/cvm-deployment/` handles both the template render and the dstack-vmm deploy. End-to-end flow: 1. **Pick or create an `.env` file** for the deploy. Either copy one of the examples under `deployment/cvm-deployment/configs/` (e.g. [`configs/sgx.env`](https://github.com/near/mpc/blob/main/deployment/cvm-deployment/configs/sgx.env)) or use the default [`default.env`](https://github.com/near/mpc/blob/main/deployment/cvm-deployment/default.env). 2. **Uncomment and set the two digest lines** in the `.env` file, using the values discovered in Step 1 above: ```env LAUNCHER_MANIFEST_DIGEST=sha256: MPC_MANIFEST_DIGEST=sha256: ``` These are shipped commented out by default. If left commented, the script refuses to run and prints the on-chain discovery commands — you can't accidentally deploy without setting them. 3. **Place your `user-config.toml`** at the path `USER_CONFIG_FILE_PATH=` points to (default: `user-config.toml` in the same directory as `deploy-launcher.sh`). 4. **Run the script**: ```bash cd deployment/cvm-deployment ./deploy-launcher.sh \ --env-file \ --base-path /path/to/meta-dstack/dstack ``` The script will: - Source the env file and validate that both digests are set + well-formed (`sha256:<64 hex>`). - Render the contract template inline with the two digests. The result lives in a tempfile — there is **no persistent** `launcher_docker_compose.yaml` on disk afterwards. (If you need one for `attestation-cli` later, see the [manual render snippet](#step-2--render-the-template) above.) - Generate `app-compose.json`. - Prompt for confirmation, then deploy the CVM via `vmm-cli`. Full flag reference and `.env` field-by-field documentation: [deploy-launcher-guide.md](https://github.com/near/mpc/blob/main/deployment/cvm-deployment/deploy-launcher-guide.md). ### Accessing MPC (or Launcher) Docker Logs #### Overview Dstack provides a dedicated web page to view CVM information, including links to the Docker logs. More details can be found in [Phala's guide](https://github.com/Dstack-TEE/dstack?tab=readme-ov-file#deploy-an-app). --- #### Local Access The web page is available on the **TDX server** at **`dstack_agent_port`** configured earlier in [Using the Web Interface](#using-the-web-interface). Open in your browser: ```txt http://localhost: ``` --- #### Remote Access If you need to access the web page from another machine, set up SSH port forwarding. For example, if `dstack_agent_port = 8090`: ```bash ssh -NL 17190:localhost:8090 USER_NAME@TDX_SERVER ``` Then open: ```txt http://localhost:17190 ``` --- #### Example / Screenshot ![CVM Web Page](./attachments/CVM_web_page.png) ### Retrieve Public Keys from the MPC Node There are 2 keys that should be retrieved from node. * P2P (`near_p2p_public_key`) - this key is used by the nodes to authenticate with one another. This key needs to be registered on the contract (see details below). * Node Account Key (`near_signer_public_key`) - this key is used by the node to issue operations such as `vote_reshared`. This key needs to be added to the NEAR account that was created in the step above. ### Retrieve the Node Account Key and P2P Key To retrieve the node account key and the P2P key, call the HTTP endpoint [http://localhost:8080/public_data](http://localhost:8080/public_data) from your server and search for `near_signer_public_key` and `near_p2p_public_key`. ```json {"near_signer_public_key":"ed25519:B2JvaYmgzfXsvCxrqd4nBrBt8jo9ReqUZatG3dAZEBv5","near_p2p_public_key":"ed25519:5SiS1SJiABiM79Yt6uEjMabAT9UguQY9hSyF7xfHLGYt"} ``` Sample curl command to extract the keys: ```bash $ curl -s http://:8080/public_data | jq -r '.near_signer_public_key' ed25519:B2JvaYmgzfXsvCxrqd4nBrBt8jo9ReqUZatG3dAZEBv5 $ curl -s http://:8080/public_data | jq -r '.near_p2p_public_key' ed25519:5SiS1SJiABiM79Yt6uEjMabAT9UguQY9hSyF7xfHLGYt ``` ### Verify Node Attestation > **Important:** Before using the node's keys (P2P key or account key), you must verify the node's attestation to confirm that the keys were generated inside a genuine TEE. Without this step, you are susceptible to a man-in-the-middle attack — an adversary could substitute their own keys for the node's real keys. The `attestation-cli` tool performs the same Intel TDX (DCAP) attestation verification that the NEAR contract and MPC nodes use, allowing you to independently validate that the node is running trusted code inside genuine hardware. > **Note:** Run the `attestation-cli` on a trusted machine. The verification should be performed from an environment you control and trust. The CLI supports two modes: - **Online mode** (`--url`) — Fetches attestation data directly from the node's `/public_data` endpoint. - **Offline mode** (`--file`) — Reads attestation data from a previously saved JSON file. This is useful if you want to save the data first, inspect it, or verify on an air-gapped machine. For full documentation, see the [attestation-cli README](../crates/attestation-cli/README.md). #### Install the attestation-cli From the [NEAR MPC repository](https://github.com/near/mpc) root: ```bash cargo install --path crates/attestation-cli ``` #### Gather the required inputs 1. **Allowed MPC Docker image manifest digest** — The SHA256 manifest digest of the approved MPC Docker image. You can query it from the contract: ```bash near contract call-function as-read-only \ v1.signer-prod.testnet allowed_docker_image_hashes \ json-args '{}' network-config testnet now ``` The latest allowed manifest digest will appear first in the returned vector. 2. **Launcher docker-compose file** — A `launcher_docker_compose.yaml` rendered from the contract template. The CLI computes its SHA256 hash and compares it against the hash attested by the node. If you deployed via `deploy-launcher.sh` (script path), the rendered compose was used in a tempfile and removed on exit — you don't have a persistent copy. Materialize one now by running the manual render block from [Preparing a Docker Compose File → Step 2](#step-2--render-the-template) (the `sed …` snippet) with the same `LAUNCHER_MANIFEST_DIGEST` and `MPC_MANIFEST_DIGEST` you deployed with. The output `launcher_docker_compose.yaml` is what the CLI needs below. #### Run the verification **Online mode** — fetch and verify directly from the node: ```bash attestation-cli \ --url http://:8080/public_data \ --allowed-image-hash \ --launcher-compose-file launcher_docker_compose.yaml ``` **Offline mode** — save the data first, then verify locally: ```bash # Save the attestation data curl -o public_data.json http://:8080/public_data # Verify from the saved file attestation-cli \ --file public_data.json \ --allowed-image-hash \ --launcher-compose-file launcher_docker_compose.yaml ``` Replace `` with your node's IP address, and `` with the hash from the contract. #### Verify the output On success, the output will show the node's keys and end with `Verdict: PASS`: ``` === MPC Node Attestation Verification === TLS Public Key (P2P): ed25519: Account Public Key: ed25519: Attestation Type: Dstack (TDX) --- Extracted Values --- MPC Image Hash: <64-char hex> Launcher Compose Hash: <64-char hex> Expiry Timestamp: 2025-07-15 12:00:00 UTC (unix: 1752577200) Verdict: PASS ``` Confirm that the **TLS Public Key (P2P)** and **Account Public Key** shown in the output match the keys you retrieved in the previous step. If they match and the verdict is PASS, the keys are authenticated — you can proceed to register them. If the verdict is FAIL, **do not use the keys**. See the [attestation-cli troubleshooting](../crates/attestation-cli/README.md#troubleshooting) section for guidance. ### Add the Node Account Key to Your Account This section shows how to add the MPC node's public key (from the previous section) as a **restricted function-call access key** to your NEAR account using the previously mentioned NEAR-CLI, allowing the MPC node to interact with the **MPC signer contract**. --- #### Parameters * **`ACCOUNT_ID`** → The NEAR account that will own the new key. Example: `your-node-account.testnet` * **`MPC_CONTRACT_ID`** → The MPC signer contract ID: Testnet: v1.signer-prod.testnet Mainnet: v1.signer * **`MPC_NODE_PUBLIC_KEY`** → The public key of the MPC node you want to add. Example: `ed25519:ABCDEFG...` * **Allowed methods** → The key is granted access to **all** methods on the MPC contract (an empty `--function-names` list). The key is still a function-call key scoped to the MPC contract (`--contract-account-id`) with an allowance, so it cannot transfer funds or call other contracts. > **Why allow all methods instead of an explicit list?** The set of methods an > MPC node must call changes across releases (e.g. `register_foreign_chain_config` > was added for foreign-chain support). A hand-maintained method list silently > drifts out of date, and the node then fails — with no obvious error — on any > new method the key was never granted. Allowing all methods on the contract > avoids this class of breakage while keeping the key scoped to the MPC contract. * **`ALLOWANCE`** → Use `unlimited`. A finite allowance just means the node will eventually stop being able to submit `respond*` transactions once the budget runs out, with no error visible in node logs. --- #### Example Command ```bash near account add-key $ACCOUNT_ID \ grant-function-call-access \ --allowance unlimited \ --contract-account-id $MPC_CONTRACT_ID \ --function-names '' \ use-manually-provided-public-key $MPC_NODE_PUBLIC_KEY \ network-config testnet \ sign-with-keychain \ send ``` --- #### Sample Bash Script ```bash #!/bin/bash # === Configuration === ACCOUNT_ID="your-node-account.testnet" MPC_CONTRACT_ID="v1.signer-prod.testnet" # use "v1.signer" for mainnet MPC_NODE_PUBLIC_KEY="ed25519:YOUR_PUBLIC_KEY_HERE" ALLOWANCE="unlimited" NETWORK="testnet" # or "mainnet" # The key is granted access to all methods on the MPC contract (empty list), # while staying scoped to the contract via --contract-account-id. # === Add Access Key === near account add-key $ACCOUNT_ID \ grant-function-call-access \ --allowance "$ALLOWANCE" \ --contract-account-id $MPC_CONTRACT_ID \ --function-names '' \ use-manually-provided-public-key $MPC_NODE_PUBLIC_KEY \ network-config $NETWORK \ sign-with-keychain \ send ``` --- #### Transaction Verification After sending the transaction, check that the new key was added: ```bash near account list-keys $ACCOUNT_ID \ network-config $NETWORK \ now ``` The key you just added should appear with `permission` listing the MPC contract as the receiver and an **empty** `method_names` list — an empty list means the key may call **all** methods on that contract. --- #### Updating an Existing Key to Allow All Methods If your node's key was previously added with a restricted `method_names` list (e.g. an older guide granted an explicit list), the node will fail — with no obvious error — on any contract method that was not in that list. A symptom of this is the node being unable to call `register_foreign_chain_config`, so the contract reports no foreign chains supported by your node. Access-key permissions are **immutable** in NEAR: you cannot edit an existing key's allowed methods. Instead, delete the restricted key and re-add the same public key with an empty `--function-names` list. ```bash # 1. Delete the existing (restricted) key. Use the SAME public key the node uses. near account delete-keys $ACCOUNT_ID \ public-keys $MPC_NODE_PUBLIC_KEY \ network-config $NETWORK \ sign-with-keychain \ send # 2. Re-add it granting access to all methods on the MPC contract. near account add-key $ACCOUNT_ID \ grant-function-call-access \ --allowance unlimited \ --contract-account-id $MPC_CONTRACT_ID \ --function-names '' \ use-manually-provided-public-key $MPC_NODE_PUBLIC_KEY \ network-config $NETWORK \ sign-with-keychain \ send ``` > **Note:** Delete and re-add each affected key (e.g. both the node key and the > responder key). Verify with `near account list-keys $ACCOUNT_ID` that each key > now shows an empty `method_names` list. After the key is fixed, **restart the node** so foreign-chain registration runs again. ## Joining the MPC Cluster After the MPC node has been deployed and its NEAR account key successfully added to the operator's account, the node will attempt to sync and then submit its attestation information to the contract. Once these steps are complete, the operator should request all other operators to vote for adding the new MPC node by calling the **vote_new_parameters** method. ### Wait for NEAR Indexer to Sync Wait until the NEAR Indexer has completed state sync. This process can take several hours. You can check the progress in the Docker container logs or via the metrics endpoint: ```bash $ curl http://127.0.0.1:3030/metrics | grep near_sync_status # Sync is completed when status is 0: near_sync_status 0 ``` ### Submitting Participant Info > **Note:** During the [transition phase](#transition-phase), this step is optional. The contract will accept nodes that do not submit an attestation. Once the MPC node is fully synced, it will call `submit_participant_info` to submit its attestation information to the contract. If the node’s key has not been added to the account, this operation will fail. In that case, the node will retry the operation in a loop. > **Note:** Calling this method will incur a cost (TBD, XXX NEAR). Ensure this amount is available in your account. > _(TBD [#903](https://github.com/near/mpc/issues/903) – confirm exact cost)_ #### Verifying the attestation was accepted The MPC node emits no explicit "submitted" log line. Once the verification poll observes the attestation on-chain it logs `INFO mpc_node::indexer::tx_sender: node found dstack attestation on chain attestation_age=... attestation_is_fresh=true`, but the authoritative check is on-chain. First, list every TEE-attested node by calling `get_tee_accounts` (no args) and confirm your `account_id` and `tls_public_key` are present: ```bash near contract call-function as-read-only \ v1.signer-prod.testnet get_tee_accounts \ json-args '{}' network-config testnet now ``` Example response (truncated): ```jsonc [ { "account_id": "n1-multichain.testnet", "account_public_key": "ed25519:3DWAWiStjQsaHwqVR91u8viuRPf2qtYpQzQHw16y5V9s", "tls_public_key": "ed25519:14q1d9LYaVUWRuyJieGGiAjAMj3KxNJeNuLS6CjUojp9" } // ... ] ``` To inspect the stored `VerifiedAttestation` for your node, call `get_attestation` with your node's TLS public key (the P2P key retrieved in [Retrieve the Node Account Key and P2P Key](#retrieve-the-node-account-key-and-p2p-key)): ```bash near contract call-function as-read-only \ v1.signer-prod.testnet get_attestation \ json-args '{"tls_public_key":"ed25519:"}' \ network-config testnet now ``` The response shape tells you what the contract accepted: - `{ "Dstack": { ... } }` — a real TEE attestation. This is what a production operator should see. - `{ "Mock": "Valid" }` — a mock attestation. Acceptable on testnet during the [transition phase](#transition-phase), but means the node is **not** running in a TEE. Many existing testnet entries are in this state. - `null` — the node has not yet submitted, or the submission is failing — see [`submit_participant_info` failures](#submit_participant_info-failures) below. Example `Dstack` response from `v1.signer-prod.testnet` for an existing participant: ```json { "Dstack": { "expiry_timestamp_seconds": 1779618069, "launcher_compose_hash": "efb095f3e9adfeb04d637813a838fa666778b9915d752cfd796ae2a254fe705f", "measurements": { "key_provider_event_digest": "61ce56b6be756a9e45af7715b13c15040a4e6090cc740be24e2cc02e33b4fb53ae4e3c945c9af83e2a26c6d5efa414a8", "mrtd": "f06dfda6dce1cf904d4e2bab1dc370634cf95cefa2ceb2de2eee127c9382698090d7a4a13e14c536ec6c9c3c8fa87077", "rtmr0": "e673be2f70beefb70b48a6109eed4715d7270d4683b3bf356fa25fafbf1aa76e39e9127e6e688ccda98bdab1d4d47f46", "rtmr1": "b598fde9491427341bc4683b75d10d3e36770af3a36a6954d8b6b7b22aa66358f13e1f172e51b7d6e6710d99a8d8532f", "rtmr2": "c812d42bfff1c75382e91a37c867ab117b97eb5e8d6797488928ea38e5fd38b5ed2f87d9613d392507f1c3af94657c93" }, "mpc_image_hash": "51ed33bb2d62c7aa8ba1a56d37550e415cf29d6a2c656ef35fa89c1ab9c0604d" } } ``` ### Voting: (vote_new_parameters) Ask other members to vote for your candidacy via the `vote_new_parameters` contract method. To generate a voting command, follow these steps: 1. **Get the current state** ```bash near contract call-function as-read-only v1.signer-prod.testnet state json-args {} network-config testnet now ``` Example output (truncated for clarity): ```json { "Running": { "keyset": { "domains": [ { "attempt": 0, "domain_id": 0, "key": { "Secp256k1": { "near_public_key": "secp256k1:4NfTiv3UsGahebgTaHyD9vF8KYKMBnfd6kh94mK6xv8fGBiJB8TBtFMP5WWXz6B89Ac1fbpzPwAvoyQebemHFwx3" } } }, { "attempt": 0, "domain_id": 1, "key": { "Ed25519": { "near_public_key_compressed": "ed25519:6vSEtQxrQj6txUMh33WC4ERyCWmNMRTdufDWAaDY3Un2" } } } ], "epoch_id": 10 }, "parameters": { "participants": { "next_id": 12, "participants": [ [ "aurora-multichain.testnet", 0, { "tls_public_key": "ed25519:BSgizrs...", "url": "http://34.49.211.4" } ], [ "bst-near.testnet", 1, { "tls_public_key": "ed25519:AadQTC...", "url": "http://34.98.94.79" } ] ... ... ... ] }, "threshold": 5 } } } ``` From this output, extract: * `participants` * `epoch_id` (10 in this example) * `next_id` (12 in this example) 2. **Update the state** * Add your new participant to the `participants` array. * Set `prospective_epoch_id = epoch_id + 1` (11 in this example). * Set `next_id = next_id + 1` (13 in this example). #### Before ```json "participants": { "next_id": 12, "participants": [ [ "aurora-multichain.testnet", 0, { "tls_public_key": "ed25519:BSgizrs...", "url": "http://34.49.211.4" } ], [ "bst-near.testnet", 1, { "tls_public_key": "ed25519:AadQTC...", "url": "http://34.98.94.79" } ] ... ... ... ] } ``` #### After (new participant `new-node.testnet` added) ```json "participants": { "next_id": 13, "participants": [ [ "aurora-multichain.testnet", 0, { "tls_public_key": "ed25519:BSgizrs...", "url": "http://34.49.211.4" } ], [ "bst-near.testnet", 1, { "tls_public_key": "ed25519:AadQTC...", "url": "http://34.98.94.79" } ], [ "new-node.testnet", 12, { "tls_public_key": "ed25519:NEWKEY...", "url": "http://example.org" } ] ... ... ... ] } ``` Example request payload: ```json REQUEST='{ "prospective_epoch_id": 11, "proposal": { "participants": { "next_id": 13, "participants": } } }' ``` #### **Create the vote command** ```bash near contract call-function as-transaction v1.signer vote_new_parameters json-args "$REQUEST" prepaid-gas '100.0 Tgas' attached-deposit '0 NEAR' sign-as $YOUR_MPC_NEAR_ACCOUNT network-config mainnet sign-with-keychain send ``` After all participants have voted, the contract will move to a resharing phase. **When resharing starts.** Each node detects the contract state change and logs: ``` INFO mpc_node::coordinator: A resharing started. INFO mpc_node::coordinator: Starting key resharing. INFO mpc_node::coordinator: Resharing process is: Some(ResharingContractState { ... }) ``` **During resharing.** Each domain runs its own attempt. Per-attempt progress lines (in order): ``` INFO mpc_node::key_events: Key resharing attempt KeyEventId { epoch_id: EpochId(1), domain_id: DomainId(0), attempt_id: AttemptId(0) }: starting key resharing. INFO mpc_node::providers::::key_resharing: Key resharing completed INFO mpc_node::key_events: Key resharing attempt KeyEventId { ... }: committing keyshare. INFO mpc_node::key_events: Key resharing attempt KeyEventId { ... }: sending vote_reshared transaction. INFO mpc_node::key_events: Key resharing attempt KeyEventId { ... } completed successfully. ``` Where `` is `ecdsa`, `eddsa`, or `ckd`. The `epoch_id` is the resharing's new epoch; `domain_id` identifies which keyspace (each gets its own attempt). **When resharing finishes.** Once the contract leaves the Resharing state, each node logs: ``` INFO mpc_node::coordinator: Concluded resharing state. ``` You can also confirm on-chain (replace `v1.signer-prod.testnet` / `testnet` with `v1.signer` / `mainnet` on mainnet): ```bash near contract call-function as-read-only \ v1.signer-prod.testnet state \ json-args {} network-config testnet now ``` The response should show `Running` (not `Resharing`) and a new `keyset.epoch_id`. **If resharing fails.** An attempt that times out or errors logs: ``` ERROR mpc_node::key_events: Key resharing attempt KeyEventId { ... } failed: ; sending vote_abort_key_event_instance ``` The contract will retry with a fresh attempt; repeated failures need investigation. ## Upgrades There are two types of upgrades, with different frequencies and operator effort: | Upgrade Type | Frequency | Operator Effort | Sealing Key Changes | | :--- | :--- | :--- | :--- | | MPC node image | High (~monthly) | Vote + restart CVM | No | | Launcher / CVM | Low | Vote + deploy new CVM + migrate key shares | Yes | When either type of hash is voted in, the contract automatically derives the expected launcher docker compose hash from an on-chain template. Operators do not need to vote on compose hashes separately. ## MPC Node Image Upgrade This is the most common upgrade. When a new MPC node version is released, operators vote for the new image manifest digest and restart their CVM. The MPC node image does **not** affect the sealing key, so existing key shares remain accessible. **Steps:** 1. Verify the Docker image (see [Image/Code Inspection](#imagecode-inspection)). 2. Vote for the new manifest digest in the contract. 3. Restart the CVM. The launcher will pull the new image by manifest digest automatically. ### Image/Code Inspection The following steps allow you to inspect the code that was used to build the docker image. Let's assume you want to vote for a docker image with tag [mpc-node:main-828f816](https://hub.docker.com/layers/nearone/mpc-node/main-828f816/), corresponding to the commit hash `828f816be36aed6f0d2438e0131b3e9d7d0931ad`. Notice that the suffix of the image tag is the short version of the git hash. * The manifest digest is shown on DockerHub and in the reproducible build script output. To verify it, build the image yourself from the same commit and compare the manifest digest. * Download the MPC code from this repository: ```bash git clone https://github.com/near/mpc cd mpc/ git checkout 828f816be36aed6f0d2438e0131b3e9d7d0931ad ``` * Compile it using the reproduce build script. For this you need to install `repro-env`, `docker-buildx`, and `skopeo`, and have the `docker` daemon running. ```bash $ ./deployment/build-images.sh --node ... commit hash: 828f816be36aed6f0d2438e0131b3e9d7d0931ad SOURCE_DATE_EPOCH used: 0 node binary hash: 86c8f7d8913d6fe37a6992bba165d15a3a1d88fbf6cdff605e4827d5183721bc node manifest digest: sha256:331cfec941671ac343c52847e255eb36a280da65535d2a1e4d002c4c64686e19 ``` The `node manifest digest` is what you vote for. When submitting the `code_hash` value in the voting command, strip the `sha256:` prefix and provide only the hex digest. The launcher pulls the image directly by this digest — Docker verifies the content matches during the pull. * Do your own due diligence on the code/binary > **Important:** Each operator is responsible for verifying that the image hashes being voted for correspond to the intended Git commit, and for performing their own due diligence on the code. ### Voting for the MPC Image Hash Each participant submits a vote for the new MPC Docker image **manifest digest**. A **threshold** number of participant votes is required for the vote to pass. Set `MANIFEST_DIGEST` to the SHA-256 hex digest (without the `sha256:` prefix), then send the vote: ```bash MANIFEST_DIGEST=331cfec941671ac343c52847e255eb36a280da65535d2a1e4d002c4c64686e19 near contract call-function as-transaction \ v1.signer-prod.testnet \ vote_code_hash \ json-args "{\"code_hash\": \"$MANIFEST_DIGEST\"}" \ prepaid-gas '100.0 Tgas' \ attached-deposit '0 NEAR' \ sign-as \ network-config testnet \ sign-with-keychain \ send ``` > **Note:** There is no `vote_remove_code_hash`. Once a successor hash is voted in, the previous hash remains valid for a 7-day grace period and then auto-expires — so unlike launcher and OS-measurement voting there is no explicit remove command. #### Query allowed MPC image hashes ```bash near contract call-function as-read-only \ v1.signer-prod.testnet \ allowed_docker_image_hashes \ json-args '{}' \ network-config testnet \ now ``` Returns the list of currently-accepted image hashes, most recent first. #### Query MPC image hash votes ```bash near contract call-function as-read-only \ v1.signer-prod.testnet \ code_hash_votes \ json-args '{}' \ network-config testnet \ now ``` Shows per-participant votes so you can see how many more are needed to reach threshold. ### Update the MPC Node After voting has finished, the MPC node will detect the new approved manifest digest on the contract and save it to a secure location inside the CVM. Restart the CVM (see [CVM management](#cvm-management)). The launcher will pull the new image by manifest digest automatically, verify it, and re-attest to the contract. > You can see the image update and re-sync in the node logs — TBD [#910](https://github.com/near/mpc/issues/910). ## Launcher / CVM Upgrade Launcher or CVM upgrades are less frequent than MPC node upgrades. Unlike MPC node upgrades, changing the launcher image or OS measurements affects the sealing key derivation, which means existing encrypted key shares **cannot** be decrypted by the new CVM. This requires deploying a new CVM and migrating key shares from the old one. For full design details, see the [CVM Upgrades section in the TEE design doc](securing-mpc-with-tee-design-doc.md#cvm-upgrades). **Steps:** 1. Verify the new launcher manifest digest and/or OS measurements. 2. Participants vote to approve the new launcher manifest digest and/or OS measurements. 3. Operator deploys a new CVM with the new launcher image and/or OS. 4. Operator migrates key shares from the old CVM to the new one using the [migration service](node-migration-guide.md). 5. After all operators have migrated, participants vote to remove the old launcher manifest digest and/or OS measurements. ### Launcher Image Voting #### Launcher image/code inspection The following steps allow you to inspect the code used to build the launcher image and verify its manifest digest before voting. * The launcher manifest digest is shown on DockerHub and in the reproducible build script output. To verify it, build the launcher yourself from the same commit and compare the manifest digest. * Download the MPC code from this repository: ```bash git clone https://github.com/near/mpc cd mpc/ git checkout ``` * Compile it using the reproducible build script. For this you need to install `repro-env`, `docker-buildx`, and `skopeo`, and have the `docker` daemon running. ```bash $ ./deployment/build-images.sh --rust-launcher ... rust launcher binary hash: rust launcher manifest digest: sha256: ``` The `rust launcher manifest digest` is what you vote for. When submitting the `launcher_hash` value in the voting command, strip the `sha256:` prefix and provide only the hex digest. * Do your own due diligence on the code/binary. > **Important:** Each operator is responsible for verifying that the manifest digest being voted for corresponds to the intended Git commit, and for performing their own due diligence on the code. #### Adding a launcher manifest digest Requires a threshold of participants to vote. The `launcher_hash` argument must be an SHA-256 hex digest (without the `sha256:` prefix). ```bash near contract call-function as-transaction \ v1.signer-prod.testnet \ vote_add_launcher_hash \ json-args '{"launcher_hash": ""}' \ prepaid-gas '100.0 Tgas' \ attached-deposit '0 NEAR' \ sign-as \ network-config testnet \ sign-with-keychain \ send ``` #### Removing a launcher manifest digest Requires **all** participants to vote. The last launcher manifest digest cannot be removed. ```bash near contract call-function as-transaction \ v1.signer-prod.testnet \ vote_remove_launcher_hash \ json-args '{"launcher_hash": ""}' \ prepaid-gas '100.0 Tgas' \ attached-deposit '0 NEAR' \ sign-as \ network-config testnet \ sign-with-keychain \ send ``` #### Query allowed launcher manifest digests The contract method is named `allowed_launcher_image_hashes` for historical reasons, but the values returned are manifest digests. ```bash near contract call-function as-read-only \ v1.signer-prod.testnet \ allowed_launcher_image_hashes \ json-args '{}' \ network-config testnet \ now ``` #### Query launcher hash votes ```bash near contract call-function as-read-only \ v1.signer-prod.testnet \ launcher_hash_votes \ json-args '{}' \ network-config testnet \ now ``` ### OS Measurement Voting OS measurements (MRTD, RTMR0-2, key-provider event digest) identify the CVM environment. Participants can vote to approve new measurement sets, enabling OS/Dstack upgrades without contract redeployment. #### Adding an OS measurement Requires a threshold of participants to vote. ```bash near contract call-function as-transaction \ v1.signer-prod.testnet \ vote_add_os_measurement \ json-args '{"measurement": {"mrtd": "", "rtmr0": "", "rtmr1": "", "rtmr2": "", "key_provider_event_digest": ""}}' \ prepaid-gas '100.0 Tgas' \ attached-deposit '0 NEAR' \ sign-as \ network-config testnet \ sign-with-keychain \ send ``` #### Removing an OS measurement Requires **all** participants to vote. The last measurement cannot be removed. ```bash near contract call-function as-transaction \ v1.signer-prod.testnet \ vote_remove_os_measurement \ json-args '{"measurement": {"mrtd": "", "rtmr0": "", "rtmr1": "", "rtmr2": "", "key_provider_event_digest": ""}}' \ prepaid-gas '100.0 Tgas' \ attached-deposit '0 NEAR' \ sign-as \ network-config testnet \ sign-with-keychain \ send ``` #### Query allowed OS measurements ```bash near contract call-function as-read-only \ v1.signer-prod.testnet \ allowed_os_measurements \ json-args '{}' \ network-config testnet \ now ``` #### Query OS measurement votes ```bash near contract call-function as-read-only \ v1.signer-prod.testnet \ os_measurement_votes \ json-args '{}' \ network-config testnet \ now ``` ### Deploy New CVM and Migrate Key Shares After the new launcher manifest digest and/or OS measurements are approved, deploy a new CVM with the updated configuration and migrate key shares from the old node. Both old and new configurations are accepted by the contract during the migration period. For the migration procedure, see the [node migration guide](node-migration-guide.md) and [migration service design](migration-service.md). ### Remove Old Launcher Manifest Digest / OS Measurements After all operators have migrated to the new CVM, participants should vote to remove the old launcher manifest digest using `vote_remove_launcher_hash` and/or old OS measurements using `vote_remove_os_measurement`. This requires **all** participants to vote, ensuring no node is still running with the old configuration. ## CVM management Common operations on a deployed CVM: **stop**, **start**, and **update `user-config.toml`**. You usually don't need to edit `user-config.toml` after initial deployment. However, a future MPC release may introduce new `user-config.toml` fields; if so, the release notes will call it out. ### Via Web UI | Operation | Action | |-----------|--------| | Stop the CVM | Press **Stop** | | Start the CVM | Press **Start** | | Update `user-config.toml` | Stop the CVM → press **update** → upload the new file → press **Upgrade** → start the CVM | ![](./attachments/cvm_options.png) ![](./attachments/config_upgrade.png) ### Via Command Line See the [VMM CLI user guide](https://github.com/Dstack-TEE/dstack/blob/master/docs/vmm-cli-user-guide.md). The CLI script is at `meta-dstack/dstack/vmm/src/vmm-cli.py`. First, define environment variables (once per shell session): ```bash export VMM_URL=http://127.0.0.1:11100 # change to your port export VMM_CLI_PATH="meta-dstack/dstack/vmm/src/vmm-cli.py" # change to your meta-dstack location ``` Then: ```bash # Enumerate and find your VM ID python $VMM_CLI_PATH --url $VMM_URL lsvm # Stop the CVM python $VMM_CLI_PATH --url $VMM_URL stop # Update user-config (only if needed) python $VMM_CLI_PATH --url $VMM_URL update-user-config ./new-user-config.toml # Start the CVM python $VMM_CLI_PATH --url $VMM_URL start ``` ## Troubleshooting TBD [#912](https://github.com/near/mpc/issues/912) Reviewers — please add here more scenarios (with or without solutions) * do we have logs that indicate the node version/hash? * How to see what MPC node hash is expected by the launcher (docker-compose v.s file on disk) * Recovery — how to erase the indexer state (e.g data folder) * ….. ### `gramine-sealing-key-provider` crash-loops with `AESM service returned error 44` Symptom: the `gramine-sealing-key-provider` container keeps `Restarting`, and `docker logs gramine-sealing-key-provider` shows: ``` error: AESM service returned error 44; this may indicate that infrastructure for the DCAP attestation requested by Gramine is missing on this machine error: load_enclave() failed with error: Operation not permitted (EPERM) ``` This means the key provider can't get an SGX quote because its PCK certificate collateral is unavailable — **the usual cause is that the host platform is not registered with Intel's provisioning service.** PCK certs are only issued for registered platforms, so the quote (and thus the key provider) fails. **Fix — register the platform:** enable **SGX Auto MP Registration** (and **SGX Factory Reset** to force a fresh registration) in BIOS, reboot, and confirm `/var/log/mpa_registration.log` shows a fresh `PLATFORM_ESTABLISHMENT … passed successfully` (canonical TDX guide §9.2 step 10). After that the QE's `GET /pckcert…` succeeds against whichever PCCS it uses (Phala by default, or your local one). To pinpoint, check the PCCS the `aesmd` queries (its `sgx_default_qcnl.conf`). If it's a local PCCS, its log (`sudo journalctl -u pccs` or `/opt/intel/sgx-dcap-pccs/logs/pccs_server.log`) shows the failing request: - **`Intel PCS server returns error(404) … No cache data for this platform`** — platform not registered (404, not 401, so the API key is fine). Register as above. - **`401` / "API key"** — the local PCCS Intel PCS API key is missing/invalid; set it via `sudo /usr/bin/pccs-configure` and `sudo systemctl restart pccs`. ### `submit_participant_info` failures When the contract rejects a submission the node retries on a backoff and logs a generic line every attempt: ``` ERROR periodic_attestation_submission: mpc_node::tee::remote_attestation: failed to submit attestation cause=attestation submission was not executed backoff_duration=60s ``` This line **doesn't tell you why** — `cause` is one of `attestation submission was not executed` (most common), `attestation submission has unknown response`, or `failed to submit transaction` (lower-level RPC send failure). None of these carry the underlying reason. The actual failure shows up in one of three places. #### 1. Client-side pre-flight WARN — in the node logs Before submitting, the node runs a partial pre-flight check against the contract's allowed-image and allowed-launcher-compose lists (boot measurements are checked against a compiled-in default, so measurement-related rejections only appear on-chain — see section 3). Failures are logged as a non-blocking warning (the node still submits, and the contract will reject for the same reason): ``` WARN periodic_attestation_submission: mpc_node::tee::remote_attestation: Attestation is not valid: custom error: `the allowed mpc image hashes list is empty` ``` Common messages: - ``custom error: `the allowed mpc image hashes list is empty` `` — the contract has no allowed image hashes voted in yet. Vote yours in (see [Voting for the MPC image hash](#voting-for-the-mpc-image-hash)). - ``custom error: `MPC image hash 0x... is not in the allowed hashes list` `` — your image hash isn't voted in. Same fix. - ``custom error: `the allowed mpc launcher compose hashes list is empty` `` / ``custom error: `MPC launcher compose hash 0x... is not in the allowed hashes list` `` — same, for the launcher compose hash (see [Launcher image voting](#launcher-image-voting)). - **`the attestation certificate with timestap ... has expired since ...`** — the quote's certificate chain has expired. The node regenerates on the next tick; if it keeps failing, your PCCS endpoints are stale (see [Customizing PCCS endpoints](#customizing-pccs-endpoints-optional)). #### 2. NEAR runtime / pre-execution errors — in the node logs For errors caught before the transaction executes (account balance, nonce, access key, etc.) the node logs them separately from `tx_sender`: ``` ERROR mpc_node::indexer::tx_sender: Failed to forward transaction SubmitParticipantInfo(...) err=unexpected ProcessTxResponse: InvalidTx(LackBalanceForState { signer_id: ..., amount: ... }) ``` The error after `err=` is the NEAR runtime error. Common ones: - **`InvalidTx(LackBalanceForState { signer_id, amount })`** — the node's account doesn't have enough NEAR to cover the storage-staking threshold the contract is about to write. Top up the account by at least `amount` yoctoNEAR. - **`InvalidTx(InvalidAccessKey(...))`** — the node's access key was not added to the account, was removed, or doesn't grant the `submit_participant_info` method. - **`InvalidTx(InvalidNonce { .. })` / `InvalidTx(Expired)`** — clock or block-hash drift; usually transient. #### 3. Contract-rejection errors — only visible on the explorer / RPC If the transaction reaches execution and the contract panics, the node logs only the generic retry line above; the actual message lives in the transaction receipt. Find the tx on `https://testnet.nearblocks.io/address/` and open the failed `submit_participant_info` call — the error appears under the action's status / logs. The contract wraps the attestation-side error like this: ``` Invalid TEE Remote Attestation: TeeQuoteStatus is invalid: the submitted attestation failed verification, reason: Custom("...") ``` The `reason` is the same `VerificationError` the client-side WARN reports (see section 1) — for example `Custom("the allowed mpc image hashes list is empty")`. Errors that **only** surface on-chain (because they're checked against the contract's allowed-measurements list, the contract's deposit logic, or the contract's caller assertion): - **`MeasurementsNotAllowed`** — your boot measurements (MRTD / RTMR0–2) are not in the contract's allowed set. Vote them in (see [OS measurement voting](#os-measurement-voting)). - **`EmptyMeasurementsList`** — the contract has no allowed measurements yet; the first set must be voted in before any node can attest. - **`Attached deposit is lower than required. Attached: X, required: Y`** — first-time joiners must attach enough yoctoNEAR for storage; the node attaches `0`, so call `submit_participant_info` manually with `--deposit` once. Exact amount tracked in [#903](https://github.com/near/mpc/issues/903). - **`Caller is not the signer account.`** — the access key used to sign does not match the node's `my_near_account_id`. If you see no error logs at all but `get_attestation` still returns `null`, the node has not yet generated a quote. Check `mpc_tee_attestation_attempts_total` on the `/metrics` endpoint. ## Transition phase {#transition-phase} During the transition phase, both MPC nodes with TEE and without TEE will be allowed. After the transition phase is completed, only MPC nodes with valid remote attestation of a valid TEE configuration will be allowed to join the MPC cluster, and any node without a valid TEE configuration will be kicked out of the cluster. ## Appendix: Self-hosting a local PCCS Run your own PCCS instead of (or alongside) Phala / Intel's public services. Install PCCS following step 9.1–2 of the canonical TDX setup (see [1. TDX Bare-Metal Server Setup](#1-tdx-bare-metal-server-setup) above). That gets you a Node.js service listening on `127.0.0.1:8081`, config under `/opt/intel/sgx-dcap-pccs/config/default.json`, and self-signed TLS material under `/opt/intel/sgx-dcap-pccs/ssl_key/`. The two MPC-specific steps below cover what's needed beyond the stock install: replacing the unusable shipped TLS cert, and pointing the node at the local PCCS with cert pinning. ### Replace the Stock TLS Certificate The cert that ships with `sgx-dcap-pccs` is X.509 v1, has no Subject Alternative Name, and is short-dated. Modern TLS clients — including the rustls-based verifier the MPC node uses — will reject it. Regenerate it as a proper PKI before pinning. The recipe below produces a 2-cert chain: a long-lived **root CA** used only as a trust anchor, plus a **leaf cert** (signed by the root) that the PCCS server actually serves. This shape is required: rustls rejects a single self-signed CA cert that's also presented as the leaf. ```bash mkdir -p ~/pccs-pki && cd ~/pccs-pki # 1. Root CA (CA:TRUE; valid 10y; only used as trust anchor) openssl req -x509 -newkey rsa:2048 -nodes \ -keyout pccs-rootCA.key -out pccs-rootCA.crt \ -days 3650 \ -subj "/CN=PCCS Local Root CA" \ -addext "basicConstraints=critical,CA:TRUE,pathlen:0" \ -addext "keyUsage=critical,keyCertSign,cRLSign" # 2. Leaf CSR openssl req -newkey rsa:2048 -nodes \ -keyout pccs-leaf.key -out pccs-leaf.csr \ -subj "/CN=pccs.local" # 3. Sign leaf with root CA — CA:FALSE; SAN must match the URL the # MPC node uses to reach the PCCS. Common SANs: # IP:10.0.2.2 — when the MPC node runs in a QEMU/dstack CVM # on the same host (slirp gateway address) # IP: — when the node is on a separate machine cat > pccs-leaf.ext </dev/null \ | grep -E "subject=|issuer=" # Expected: # subject=CN = pccs.local # issuer=CN = PCCS Local Root CA ``` ### Configure the MPC Node In `user-config.toml`, add the **root CA** as a trust anchor (default public-CA roots remain active). The server presents the leaf, which chains to this root: ```toml [[mpc_node_config.pccs_endpoints]] url = "https://10.0.2.2:8081/" tls = { override = "ca_cert_pem", ca_cert_pem = """ -----BEGIN CERTIFICATE----- ... contents of pccs-rootCA.crt ... -----END CERTIFICATE----- """ } # Phala fallback for resilience (Intel direct could also be added as a third entry) [[mpc_node_config.pccs_endpoints]] url = "https://pccs.phala.network/" ``` ### About `tls.override = "insecure"` Disables all TLS certificate validation (cert chain *and* hostname). The startup log emits a clearly-labeled WARN when this mode is active. Acceptable for local-host bring-up before you've provisioned a proper cert; not recommended for any persistent setup. The code does **not** enforce loopback-only — `insecure` disables TLS for any URL configured. It is the operator's responsibility to use this value correctly. The startup WARN is the only guardrail.