Guile-ERIS Manual

Table of Contents

Guile-ERIS

This document describes Guile-ERIS version 1.0.0.

The Encoding for Robust Immutable Storage (ERIS) is an encoding of arbitrary content into a set of uniformly sized, encrypted and content-addressed blocks as well as a read capability. The content can be reassembled from the encrypted blocks only with this read capability. The read capability can be encoded as an URN allowing encoded content to be referenced from existing applications.

For more information on ERIS see also the specification document.

Guile-ERIS is a Guile Scheme implementation of ERIS.


1 Introduction

A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable.

Leslie Lamport

Unavailability of content is a major cause for reduced reliability of distributed systems.

Availability can be increased by caching content on multiple peers. However most content on the Internet is identified by its location. Caching location-addressed content is complicated as the content receives a new location when cached.

Caching content-addressed content and making it available redundantly is much easier as the content is completely decoupled from any physical location. Integrity of content is automatically ensured with content-addressing (when using a cryptographic hash) as the identifier of the content can be recomputed to check that the content matches the requested identifier. However, naive content-addressing has certain drawbacks:

ERIS addresses these issues by splitting content into small uniformly sized and encrypted blocks. These blocks can be reassembled to the original content only with access to a short read capability, which can be encoded as an URN.

Encodings similar to ERIS are already widely-used in applications and protocols such as GNUNet, BitTorrent, Freenet, Gnutella, Direct Connect, IPFS and others. However, they all use slightly different encodings that are tied to the respective protocols and applications. ERIS defines an encoding independent of any specific protocol or application and decouples content from transport and storage layers.

Blocks of encoded content can be transported over various protocols and stored on various mediums. Implementation and bindings to transport layers is (currently) beyond scope of this library. See Transport and Storage of Blocks for some ideas and pointers.

See the specification document for an in-depth description of the encoding. A short overview is provided in this manual (see Overview and Concepts).


1.1 Contact

The source code of Guile-ERIS is avilable on Codeberg.

A mailing list for general discussion on ERIS is available at ~pukkamustard/eris@lists.sr.ht. Ephemeral discussions take place in the #eris channel on the Libera IRC network. See also the project page for more information.

Please feel free to direct any questions or comments to the mailing list or to the IRC channel. You are also invited to share your applications and use-cases.


2 Overview and Concepts

ERIS splits content into uniformly sized and encrypted blocks (leaf nodes). References and keys to blocks are collected in intermediary nodes. Nodes are encrypted and references are recursively collected in higher level nodes until a single root node remains.

To decode content we need:

This information is encoded in the read capability. The read capability can be encoded as a binary string or as a URN (see Working with Read Capabilities).


2.1 Block Size

ERIS allows two fixed block sizes:


2.2 Convergent Encryption

Deterministic convergent encryption allows deterministic identifiers of content. However, it also suffers from two known attacks that might allow adversaries confirm that certain files have been decoded (the Confirmation of a File Attack) and in some cases even learn the encoded content (The Learn-The-Remaining-Information Attack).

The solution to both attacks is to use a convergence secret. A convergence secret adds some entropy to the encoding process. Using the same convergence secret results in the same encoding and the same identifiers for the same content. Using different convergence secrets results in different encodings and different identifiers for the same content.

The convergence secret allows a flexible level of protection against the known attacks against deterministc convergent encryption:

The convergence secret is not required when decoding content.

For more information please consult the ERIS specification.


2.3 Security Considerations

ERIS is designed to increase the availability of content. ERIS is NOT designed for secure communication.

Please read the specification for detailed security considerations.


3 Encoding and Decoding Content

(use-modules (eris))

Functionality to encode and decode content using ERIS is provided in the (eris) module.

Note that the version of the ERIS specification implemented is different than the version of this library. The variable %eris-spec-version holds the version of the ERIS specification implemented by this library:

Scheme Variable: %eris-spec-version

The version of the ERIS specification implemented.


3.1 Encoding

Some content that is available in a string, a bytevector or in an input port can be encoded with ERIS. The output is a set of content-addressable blocks and a read capability.

Guile-ERIS does not make any assumptions about how to store or transport blocks (see Transport and Storage of Blocks). The encoding procedures take a SRFI-171 reducer (see SRFI-171 Reducers in GNU Guile Reference Manual) that stores blocks. The reducing procedure is called with no arguments to return an initial state, with two arguments (the current state, and the input block) when blocks should be stored and with a single argument (the final state) when encoding is completed.

For conveniences, the null convergence secret is provided as a variable:

Scheme Variable: %null-convergence-secret

A bytevector of size 32 bytes with all bytes set to zero. Using the %null-convergence-secret as convergence secret enables deterministc convergent encryption. Users should be aware of the security implications of using deterministc convergent encryption (see Convergent Encryption).

The most direct encoding procedure is eris-encode, which can be used to encode some content in one go:

Scheme Procedure: eris-encode readable #:block-size block-size #:block-reducer block-reducer #:convergence-secret convergence-secret

Encode content from the port, string or bytevector readable and reduce blocks into block-reducer. Returns two values: the read capability as URI string and the reduced blocks.

readable is either an input port, a string or a bytevector holding the content to be encoded.

block-size specifies the block size that should be used to encode the content (see Block Size). Valid values are 'small and 'large.

block-reducer specifies a SRFI-171 reducer that is used to store the blocks . The values input to the reducer are pairs consisting of the reference to the block (as bytevector) and the block itself (as bytevector). The reference to the block is always the Blake2b hash of the block. By default the reverse-rcons is used, which stores blocks in a list.

convergence-secret specifies the convergence secret as bytevector of size 32 bytes (see Convergent Encryption).

Example:

(use-modules (eris)
	     (sodium generichash)
	     (rnrs base)
	     (rnrs bytevectors))

(define (blake2b-256 bv)
  (crypto-generichash bv #:out-len 32))

(define my-block-reducer
  (case-lambda
    ;; Initialization
    (() '()) ; just return an empty list

    ;; Completion
    ((result) result) ; nothing to do

    ;; Input
    ((result ref-block)
     (let ((ref (car ref-block)) ; the reference to the block
	   (block (cdr ref-block))) ; the block itself

       (assert (bytevector=? ref (blake2b-256 block)))
       (assert (equal? (bytevector-length block) 1024))

       ;; add the ref-block to the result
       (cons ref-block result)))))

(eris-encode "Hello world!"
	     #:block-size 'small
	     #:block-reducer my-block-reducer
	     #:convergence-secret %null-convergence-secret)

;; => "urn:urn:eris:BIAD77QDJMFAKZYH2DXBUZYAP3MXZ3DJZVFYQ5DFWC6T65WSFCU5S2IT4YZGJ7AC4SYQMP2DM2ANS2ZTCP3DJJIRV733CRAAHOSWIYZM3M"
;; => ( ... ) ; list of blocks

Note that my-block-reducer is exactly reverse-rcons from SRFI-171 (without the asserts). The initialization and completion arity could be used for stateful setup and tear-down of a block storage (e.g. a database connection).

Alternatively, a binary output port can be opened that will encode content that is written to it. This is useful for encoding content that is larger than the available memory or is obtained as a stream.

Scheme Procedure: open-eris-output-port #:block-size block-size #:block-reducer block-reducer #:convergence-secret convergence-secret

Returns two values: a binary output port and a procedure. The latter should be called with zero arguments to obtain the read capability of all the data accumulated by the port and the the reduced blocks.

Encoded blocks will be emited eagerly. The memory requirement is logarithmic (with a large base) to the size of the input.

block-size specifies the block size that should be used to encode the content (see Block Size). Valid values are 'small and 'large.

block-reducer specifies a SRFI-171 reducer that is used to store the blocks . The values input to the reducer are pairs consisting of the reference to the block (as bytevector) and the block itself (as bytevector). The reference to the block is always the Blake2b hash of the block. By default the reverse-rcons is used, which stores blocks in a list.

convergence-secret specifies the convergence secret as bytevector of size 32 bytes (see Convergent Encryption).

Example:

(use-modules (eris)
	     (srfi srfi-71))

(let ((port finalize
	    (open-eris-output-port
	     #:block-size 'small
	     #:convergence-secret %null-convergence-secret)))

  (display "Hello " port)
  (display "world!" port)

  (finalize))

;; => "urn:urn:eris:BIAD77QDJMFAKZYH2DXBUZYAP3MXZ3DJZVFYQ5DFWC6T65WSFCU5S2IT4YZGJ7AC4SYQMP2DM2ANS2ZTCP3DJJIRV733CRAAHOSWIYZM3M"
;; => ( ... ) ; list of blocks

3.1.1 Encoder

(use-modules (eris encoder))

The (eris encoder) module provides an encoder object. This is a more low-level interface for encoding content. For example it allows the block reducers state to be inspected and modified while encoding. The procedures eris-encode and open-eris-output-port are implemented on top of this interface.

Scheme Procedure: eris-encoder-init #:block-size block-size #:convergence-secret convergence-secret #:block-reducer block-reducer #:identity identity

Returns a new encoder object (a <eris-encoder> object).

block-size specifies the block size in bytes. Valid values are 1024 (for small block size) or 32768 (for large block size).

block-reducer specifies a SRFI-171 reducer that is used to store the blocks . The values input to the reducer are pairs consisting of the reference to the block (as bytevector) and the block itself (as bytevector). The reference to the block is always the Blake2b hash of the block. By default the reverse-rcons is used, which stores blocks in a list.

identity specifies the initial reducer state. If identity is not given, the block-reducer is run without arguments to return the initial reducer state.

convergence-secret specifies the convergence secret as bytevector of size 32 bytes (see Convergent Encryption).

Scheme Procedure: eris-encoder obj

Returns true if obj is an encoder object.

Scheme Procedure: eris-encoder-update encoder bv

Add bv to the content being encoded and return the updated encoder.

Content that is smaller than block-size is buffered by the encoder. For best performance add pieces of size exactly block-size to the encoder.

Scheme Procedure: eris-encoder-finalize finalize #:finalize-reducer finalize-reducer?

Finalizes the encoder with all content previously added with eris-encoder-update and returns two values: the read capability (as <read-capability> object, see Working with Read Capabilities) and the reduced blocks.

If finalize-reducer? is true (default value is true) then the reducer state is finalized by running the procedure block-reducer on the final reducer state.

Scheme Procedure: eris-encoder-result encoder

Returns the current state of the block reducer.

Scheme Procedure: set-eris-encoder-result! encoder vaue

Sets the state of the block reducer.


3.2 Decoding

Two procedures are provided for decoding content: eris-decode and open-eris-input-port.

eris-decode can be used to decode the entire content at once:

Scheme Procedure: eris-decode read-capability #:block-ref block-ref

Returns the decoded content as bytevector.

read-capability is the ERIS read capability as uri, string, bytevector (binary encoded read capability) or as a read-capability record (see Working with Read Capabilities).

block-ref is a procedure which is called to de-reference blocks. A single argument is passed with the reference of the block as bytevector (the Blake2b hash of the block).

Example:

(use-modules (eris)
	     (srfi srfi-171)
	     (rnrs bytevectors))

(let ((read-capability blocks
       (eris-encode "Hello world!"
		    #:block-size 'small
		    #:convergence-secret %null-convergence-secret)))
  (utf8->string
   (eris-decode read-capability
		#:block-ref (lambda (ref) (assoc-ref blocks ref)))))

;; => "Hello world!"

open-eris-input-port allows more fine-grained access. The encoded content is exposed on a binary input port. The binary port allows random-access to specific positions of the content, while only de-referencing the necessary blocks:

Scheme Procedure: open-eris-input-port read-capability #:block-ref block-ref

Returns a binary input port that can be used to read decoded content.

read-capability is the ERIS read capability as uri, string, bytevector (binary encoded read capability) or as a read-capability record (see Working with Read Capabilities).

block-ref is a procedure which is called to de-reference blocks. A single argument is passed with the reference of the block as bytevector (the Blake2b hash of the block).

Example:

(use-modules (eris)
	     (srfi srfi-171)
	     (rnrs io ports))

(let* ((read-capability
	blocks
	(eris-encode "Hello world!"
		     #:block-size 'small
		     #:convergence-secret %null-convergence-secret))
       (port (open-eris-input-port read-capability
				   #:block-ref
				   (lambda (ref) (assoc-ref blocks ref)))))

  (get-string-n port 2)
  ;; => "He"
 
  (set-port-position! port 6)
  (get-string-all port)
  ;; => "world!"

  (set-port-position! port 0)
  (get-string-all port)
  ;; => "Hello world!"

  (port-position port))

;; => 12

3.2.1 Decoder

(use-modules (eris decoder))

The (eris decoder) module provides a decoder object. This is a more low-level interface for decoding content. For example it allows efficient computation of the length of some encoded content while de-referencing a minimal amount of blocks. It can also be used to implement parallel decoders or look-ahead decoders (when using a caching block-ref procedure).

Decoders are purely functional. The implementation is based on Zippers (Functional Pearl: The Zipper, GĂ©rard Huet, 1997).

Scheme Procedure: eris-decoder-init read-capability #:block-ref block-ref

Returns an initialized <eris-decoder> object. Initial position is set to the beginning of the content. No blocks are de-referenced during initialization.

read-capability is the ERIS read capability as uri, string, bytevector (binary encoded read capability) or as a read-capability record (see Working with Read Capabilities).

block-ref is a procedure which is called to de-reference blocks. A single argument is passed with the reference of the block as bytevector (the Blake2b hash of the block).

Scheme Procedure: eris-decoder? obj

Returns true if obj is an ERIS decoder (a <eris-decoder> object).

Scheme Procedure: eris-decoder-position decoder

Returns current position of decoder as offset from the beginning of the encoded content.

Scheme Procedure: eris-decoder-seek decoder pos

Returns a new decoder with position set to pos.

Scheme Procedure: eris-decoder-length decoder

Returns the length of the encoded content in bytes.

The length is computed by descending the tree by the right most nodes at every level.

Scheme Procedure: eris-decoder-read decoder target target-start len

Reads at most len bytes from decoder and copies them to the bytevector target starting at offset target-start. Returns two values: The number of bytes read and a new decoder with position set to after the bytes read.


3.3 Working with Read Capabilities

(use-modules (eris read-capability))

Read capabilities hold the necessary information to decode ERIS encoded content from blocks.

Read capabilities can be encoded as binary sequences (i.e. as bytevectors in Scheme) or as URIs (URNs to be precise). The (eris read-capability) module provides functions for working with read capabailities.

Scheme Procedure: make-eris-read-capability block-size level root-reference root-key

Returns a new read capability object.

Scheme Procedure: eris-read-capability-block-size read-capability

Returns the block size of read-capability.

Scheme Procedure: eris-read-capability-level read-capability

Returns the level of read-capability.

Scheme Procedure: eris-read-capability-root-reference read-capability

Returns the root reference of read-capability.

Scheme Procedure: eris-read-capability-root-key read-capability

Returns the root key of read-capability.

Scheme Procedure: eris-read-capability->bytevector read-capability

Returns the binary encoding of read-capability as bytevector.

Scheme Procedure: eris-read-capability->string read-capability

Returns the URN encoding of read-capability as string.

Scheme Procedure: ->eris-read-capability obj

Returns a new read capability object that is parsed from obj. obj is either a bytevector containing the binary encoding of a read capability, a string containing the URN encoding, a URI. If no read capability can be parsed false is returned.


4 Transport and Storage of Blocks

Currently this library does not provide any block reducers or block getters (see Encoding and Decoding Content) for transporting and storing blocks.

Possible transport and storage layers include:

Many other block transports and storages should be possible. Get in contact and get hacking!


Index

Jump to:   C   R  
Index Entry  Section

C
convergence secret: Convergent Encryption

R
read capability: Overview and Concepts

Jump to:   C   R  
Jump to:   %   -  
E   M   O   S  
Index Entry  Section

%
%eris-spec-version: Encoding and Decoding Content
%null-convergence-secret: Encoding

-
->eris-read-capability: Working with Read Capabilities

E
eris-decode: Decoding
eris-decoder-init: Decoder
eris-decoder-length: Decoder
eris-decoder-position: Decoder
eris-decoder-read: Decoder
eris-decoder-seek: Decoder
eris-decoder?: Decoder
eris-encode: Encoding
eris-encoder: Encoder
eris-encoder-finalize: Encoder
eris-encoder-init: Encoder
eris-encoder-result: Encoder
eris-encoder-update: Encoder
eris-read-capability->bytevector: Working with Read Capabilities
eris-read-capability->string: Working with Read Capabilities
eris-read-capability-block-size: Working with Read Capabilities
eris-read-capability-level: Working with Read Capabilities
eris-read-capability-root-key: Working with Read Capabilities
eris-read-capability-root-reference: Working with Read Capabilities

M
make-eris-read-capability: Working with Read Capabilities

O
open-eris-input-port: Decoding
open-eris-output-port: Encoding

S
set-eris-encoder-result!: Encoder

Jump to:   %   -  
E   M   O   S