Storage

The Synapse SDK automatically handles all the complexity of storage setup for you - selecting providers, managing data sets, and coordinating with the blockchain.

Storage Context Creation

You have two options:

1. Simple mode: Just use synapse.storage.upload() directly - the SDK auto-manages contexts for you.
2. Explicit mode: Create a context with synapse.storage.createContext() for more control. Contexts can be used directly or passed in the options to synapse.storage.upload() and synapse.storage.download().

Behind the scenes, the process may be:

• Fast (< 1 second): When reusing existing data sets that match your requirements (including all metadata)
• Slower (2-5 minutes): When setting up new blockchain infrastructure (i.e. creating a brand new data set)

Basic Usage

// Option 1: Auto-managed context (simplest)
await synapse.storage.upload(data)  // Context created/reused automatically

// Option 2: Explicit context creation
const context = await synapse.storage.createContext()
await context.upload(data)  // Upload to this specific context

// Option 3: Context with metadata requirements
const context = await synapse.storage.createContext({
  metadata: {
    withIPFSIndexing: '',
    category: 'videos'
  }
})
// This will reuse any existing data set that has both of these metadata entries,
// or create a new one if none match

Metadata Limits

Metadata is subject to the following contract-enforced limits:

Data Set Metadata:

• Maximum of 10 key-value pairs per data set
• Keys: Maximum 32 characters
• Values: Maximum 128 characters

Piece Metadata:

• Maximum of 5 key-value pairs per piece
• Keys: Maximum 32 characters
• Values: Maximum 128 characters

These limits are enforced by the blockchain contracts. The SDK will validate metadata before submission and throw descriptive errors if limits are exceeded.

Advanced Usage with Callbacks

Monitor the creation process with detailed callbacks:

const context = await synapse.storage.createContext({
  providerAddress: '0x...', // Optional: use specific provider address
  withCDN: true,            // Optional: enable CDN for faster downloads
  callbacks: {
    // Called when a provider is selected
    onProviderSelected: (provider) => {
      console.log(`Selected provider: ${provider.owner}`)
      console.log(`  PDP URL: ${provider.pdpUrl}`)
    },

    // Called when data set is found or created
    onDataSetResolved: (info) => {
      if (info.isExisting) {
        console.log(`Using existing data set: ${info.dataSetId}`)
      } else {
        console.log(`Created new data set: ${info.dataSetId}`)
      }
    },

    // Only called when creating a new data set
    onDataSetCreationStarted: (transaction, statusUrl) => {
      console.log(`Creation transaction: ${transaction.hash}`)
      if (statusUrl) {
        console.log(`Monitor status at: ${statusUrl}`)
      }
    },

    // Progress updates during data set creation
    onDataSetCreationProgress: (status) => {
      const elapsed = Math.round(status.elapsedMs / 1000)
      console.log(`[${elapsed}s] Mining: ${status.transactionMined}, Live: ${status.dataSetLive}`)
    }
  }
})

Creation Options

interface StorageServiceOptions {
  providerId?: number                      // Specific provider ID to use
  providerAddress?: string                 // Specific provider address to use
  dataSetId?: number                       // Specific data set ID to use
  withCDN?: boolean                        // Enable CDN services (alias for metadata: { withCDN: '' })
  metadata?: Record<string, string>        // Metadata requirements for data set selection/creation
  callbacks?: StorageCreationCallbacks     // Progress callbacks
  uploadBatchSize?: number                 // Max uploads per batch (default: 32, min: 1)
}

Data Set Selection and Matching

The SDK intelligently manages data sets to minimize on-chain transactions. The selection behavior depends on the parameters you provide:

Selection Scenarios:

1. Explicit data set ID: If you specify dataSetId, that exact data set is used (must exist and be accessible)
2. Specific provider: If you specify providerId or providerAddress, the SDK searches for matching data sets only within that provider’s existing data sets
3. Automatic selection: Without specific parameters, the SDK searches across all your data sets with any approved provider

Exact Metadata Matching: In scenarios 2 and 3, the SDK will reuse an existing data set only if it has exactly the same metadata keys and values as requested. This ensures data sets remain organized according to your specific requirements.

Selection Priority: When multiple data sets match your criteria:

• Data sets with existing pieces are preferred over empty ones
• Within each group (with pieces vs. empty), the oldest data set (lowest ID) is selected

Provider Selection (when no matching data sets exist):

• If you specify a provider (via providerId or providerAddress), that provider is used
• Otherwise, the SDK currently uses random selection from all approved providers
• Before finalizing selection, the SDK verifies the provider is reachable via a ping test
• If a provider fails the ping test, the SDK tries the next candidate

// Scenario 1: Explicit data set (no matching required)
const context1 = await synapse.storage.createContext({
  dataSetId: 42  // Uses data set 42 directly
})

// Scenario 2: Provider-specific search
const context2 = await synapse.storage.createContext({
  providerId: 3,
  metadata: { app: 'myapp', env: 'prod' }
})
// Searches ONLY within provider 3's data sets for exact metadata match

// Scenario 3: Automatic selection across all providers
const context3 = await synapse.storage.createContext({
  metadata: { app: 'myapp', env: 'prod' }
})
// Searches ALL your data sets across any approved provider

// Metadata matching examples (exact match required):
// These will use the SAME data set (if it exists)
const contextA = await synapse.storage.createContext({
  metadata: { app: 'myapp', env: 'prod' }
})
const contextB = await synapse.storage.createContext({
  metadata: { env: 'prod', app: 'myapp' }  // Order doesn't matter
})

// These will use DIFFERENT data sets
const contextC = await synapse.storage.createContext({
  metadata: { app: 'myapp' }  // Missing 'env' key
})
const contextD = await synapse.storage.createContext({
  metadata: { app: 'myapp', env: 'prod', extra: 'data' }  // Has extra key
})

// Provider selection when no data sets match:
const newContext = await synapse.storage.createContext({
  metadata: { app: 'newapp', version: 'v1' }
})
// If no existing data sets have this exact metadata:
// 1. SDK randomly selects from approved providers
// 2. Pings the selected provider to verify availability
// 3. Creates a new data set with that provider

The withCDN Option: This is a convenience alias for adding { withCDN: '' } to metadata:

// These are equivalent:
const context1 = await synapse.storage.createContext({ withCDN: true })
const context2 = await synapse.storage.createContext({
  metadata: { withCDN: '' }
})

Storage Context Properties

Once created, the storage context provides access to:

// The data set ID being used
console.log(`Data set ID: ${context.dataSetId}`)

// The service provider address
console.log(`Service provider: ${context.serviceProvider}`)

Storage Context Methods

Preflight Upload

Check if an upload is possible before attempting it:

const preflight = await context.preflightUpload(dataSize)
console.log('Estimated costs:', preflight.estimatedCost)
console.log('Allowance sufficient:', preflight.allowanceCheck.sufficient)

Upload and Download

Upload and download data with the storage context:

// Upload with optional progress callbacks and piece-specific metadata
const result = await context.upload(data, {
  // Optional: Add metadata specific to this piece (key-value pairs)
  metadata: {
    snapshotVersion: 'v2.1.0',
    generator: 'backup-system'
  },
  onUploadComplete: (pieceCid) => {
    console.log(`Upload complete! PieceCID: ${pieceCid}`)
  },
  onPieceAdded: (transaction) => {
    // For new servers: transaction object with details
    // For old servers: undefined (backward compatible)
    if (transaction) {
      console.log(`Transaction confirmed: ${transaction.hash}`)
    } else {
      console.log('Data added to data set (legacy server)')
    }
  },
  onPieceConfirmed: (pieceIds) => {
    // Only called for new servers with transaction tracking
    console.log(`Piece IDs assigned: ${pieceIds.join(', ')}`)
  }
})

// Download data from this context's specific provider
const downloaded = await context.download(result.pieceCid)

// Get the list of piece CIDs in the current data set by querying the provider
const pieceCids = await context.getDataSetPieces()
console.log(`Piece CIDs: ${pieceCids.map(cid => cid.toString()).join(', ')}`)

// Check the status of a piece on the service provider
const status = await context.pieceStatus(result.pieceCid)
console.log(`Piece exists: ${status.exists}`)
console.log(`Data set last proven: ${status.dataSetLastProven}`)
console.log(`Data set next proof due: ${status.dataSetNextProofDue}`)

Size Constraints

The storage service enforces the following size limits for uploads:

• Minimum: 65 bytes
• Maximum: 200 MiB (209,715,200 bytes)

Attempting to upload data outside these limits will result in an error.

Note: these limits are temporary during this current pre-v1 period and will eventually be extended. You can read more in this issue thread

Efficient Batch Uploads

When uploading multiple files, the SDK automatically batches operations for efficiency. Due to blockchain transaction ordering requirements, uploads are processed sequentially. To maximize efficiency:

// Efficient: Start all uploads without await - they'll be batched automatically
const uploads = []
for (const data of dataArray) {
  uploads.push(context.upload(data))  // No await here
}
const results = await Promise.all(uploads)

// Less efficient: Awaiting each upload forces sequential processing
for (const data of dataArray) {
  await context.upload(data)  // Each waits for the previous to complete
}

The SDK batches up to 32 uploads by default (configurable via uploadBatchSize). If you have more than 32 files, they’ll be processed in multiple batches automatically.

Removing Data

To delete an entire data set and discontinue payments for the service, call context.terminate(). This method submits an on-chain transaction to initiate the termination process. Following a defined termination period, payments will cease, and the service provider will be able to delete the data set.

You can also terminate a data set using synapse.storage.terminateDataSet(dataSetId), in a case that creation of the context is not possible or dataSetId is known and creation of the context is not necessary.

Important: Data set termination is irreversible and cannot be canceled once initiated.

Deletion of individual pieces is not supported at this time but is on the roadmap.

Storage Information

Get comprehensive information about the storage service:

// Get storage service info including pricing and providers
const info = await synapse.getStorageInfo()
console.log('Price per TiB/month:', info.pricing.noCDN.perTiBPerMonth)
console.log('Available providers:', info.providers.length)
console.log('Network:', info.serviceParameters.network)

// Get details about a specific provider
const providerInfo = await synapse.getProviderInfo('0x...')
console.log('Provider PDP URL:', providerInfo.pdpUrl)

Download Options

The SDK provides flexible download options with clear semantics:

SP-Agnostic Download (from anywhere)

Download pieces from any available provider using the StorageManager:

// Download from any provider that has the piece
const data = await synapse.storage.download(pieceCid)

// Download with CDN optimization (if available)
const dataWithCDN = await synapse.storage.download(pieceCid, { withCDN: true })

// Prefer a specific provider (falls back to others if unavailable)
const dataFromProvider = await synapse.storage.download(pieceCid, {
  providerAddress: '0x...'
})

Context-Specific Download (from this provider)

When using a StorageContext, downloads are automatically restricted to that specific provider:

// Downloads from the provider associated with this context
const context = await synapse.storage.createContext({ providerAddress: '0x...' })
const data = await context.download(pieceCid)

// The context passes its withCDN setting to the download
const contextWithCDN = await synapse.storage.createContext({ withCDN: true })
const dataWithCDN = await contextWithCDN.download(pieceCid) // Uses CDN if available

CDN Option Inheritance

The withCDN option (which is an alias for metadata: { withCDN: '' }) follows a clear inheritance hierarchy:

1. Synapse level: Default setting for all operations
2. StorageContext level: Can override Synapse’s default
3. Method level: Can override instance settings

// Example of inheritance
const synapse = await Synapse.create({ withCDN: true })                  // Global default: CDN enabled
const context = await synapse.storage.createContext({ withCDN: false })  // Context override: CDN disabled
await synapse.storage.download(pieceCid)                                 // Uses Synapse's withCDN: true
await context.download(pieceCid)                                         // Uses context's withCDN: false
await synapse.storage.download(pieceCid, { withCDN: false })             // Method override: CDN disabled

Note: When withCDN: true is set, it adds { withCDN: '' } to the data set’s metadata, ensuring CDN-enabled and non-CDN data sets remain separate.