LBRY

DATABASE.md

LBRY Database

Prefix Row	Prefix	Key	Key Parameters	Value	Value parameters	Conditions
ClaimToSupportPrefixRow	K	ClaimToSupportKey	claim_hash,tx_num,position	ClaimToSupportValue	amount	PUT: every output with support claim and matching normalized name
SupportToClaimPrefixRow	L	SupportToClaimKey	tx_num,position	SupportToClaimValue	claim_hash	PUT: every output with support claim and matching normalized name
ClaimToTXOPrefixRow	E	ClaimToTXOKey	claim_hash	ClaimToTXOValue	tx_num,position,root_tx_num,root_position,amount,activation,channel_signature_is_valid,name
TXOToClaimPrefixRow	G	TXOToClaimKey	tx_num,position	TXOToClaimValue	claim_hash,name
ClaimToChannelPrefixRow	I	ClaimToChannelKey	claim_hash,tx_num,position	ClaimToChannelValue	signing_hash
ChannelToClaimPrefixRow	J	ChannelToClaimKey	signing_hash,name,tx_num,position	ChannelToClaimValue	claim_hash
ClaimShortIDPrefixRow	F	ClaimShortIDKey	normalized_name,partial_claim_id,root_tx_num,root_position	ClaimShortIDValue	tx_num,position
BidOrderPrefixRow	D	BidOrderKey	normalized_name,effective_amount,tx_num,position	BidOrderValue	claim_hash
ClaimExpirationPrefixRow	O	ClaimExpirationKey	expiration,tx_num,position	ClaimExpirationValue	claim_hash,normalized_name
ClaimTakeoverPrefixRow	P	ClaimTakeoverKey	normalized_name	ClaimTakeoverValue	claim_hash,height
PendingActivationPrefixRow	Q	PendingActivationKey	height,txo_type,tx_num,position	PendingActivationValue	claim_hash,normalized_name
ActivatedPrefixRow	R	AcitvationKey	txo_type,tx_num,position	ActivationValue	height,claim_hash,normalized_name
ActiveAmountPrefixRow	S	ActiveAmountKey	claim_hash,txo_type,activation_height,tx_num,position	ActiveAmountValue	amount
RepostPrefixRow	V	RepostKey	claim_hash	RepostValue	reposted_claim_hash
RepostedPrefixRow	W	RepostedKey	reposted_claim_hash,tx_num,position	RepostedValue	claim_hash
UndoPrefixRow	M	UndoKey	height,block_hash	-	Raw data
TouchedOrDeletedPrefixRow	Y	TouchedOrDeletedClaimKey	height	TouchedOrDeletedClaimValue	touched_claims,deleted_claims
TXPrefixRow	B	TxKey	tx_hash	TxValue	raw_tx	PUT: every transaction
BlockHashPrefixRow	C	BlockHashKey	height	BlockHashValue	block_hash	PUT: every block
BlockHeaderPrefixRow	H	BlockHeaderKey	height	BlockHeaderValue	header	PUT: every block
TXNumPrefixRow	N	TxNumKey	tx_hash	TxNumValue	tx_num	PUT: every transaction
TxCountPrefixRow	T	TxCountKey	height	TxCountValue	tx_count	PUT: every block
TXHashPrefixRow	X	TxHashKey	tx_num	TxHashValue	tx_hash	PUT: every transaction
UTXOPrefixRow	u	UTXOKey	hashX,tx_num,nout	UTXOValue	amount	DELETE: every non-generation input PUT: every output
HashXUTXOPrefixRow	h	HashXUTXOKey	short_tx_hash,tx_num,nout	HashXUTXOValue	hashX	DELETE: every non-generation input PUT: every output
HashXHistoryPrefixRow	x	HashXHistoryKey	hashX,height	-	Array of tx_num (unsigned int32)
DBStatePrefixRow	s	-	-	DBState	genesis,height,tx_count,tip,utxo_flush_count,wall_time,bit_fields,db_version,hist_flush_count,comp_flush_count,comp_cursor,es_sync_height,hashX_status_last_indexed_height
ChannelCountPrefixRow	Z	ChannelCountKey	channel_hash	ChannelCountValue	count
SupportAmountPrefixRow	a	SupportAmountKey	claim_hash	SupportAmountValue	amount
BlockTxsPrefixRow	b	BlockTxsKey	height	BlockTxsValue	tx_hashes	PUT: every block
TrendingNotificationPrefixRow	c	TrendingNotificationKey	height,claim_hash	TrendingNotificationValue	previous_amount,new_amount
MempoolTXPrefixRow	d	MempoolTxKey	tx_hash	MempoolTxValue	raw_tx
TouchedHashXPrefixRow	e	TouchedHashXKey	height	TouchedHashXValue	touched_hashXs	PUT: every block
HashXStatusPrefixRow	f	HashXStatusKey	hashX	HashXStatusValue	status
HashXMempoolStatusPrefixRow	g	HashXStatusKey	hashX	HashXStatusValue	status
RepostedCountPrefixRow	j	RepostedCountKey	claim_hash	RepostedCountValue	reposted_count
EffectiveAmountPrefixRow	i	EffectiveAmountKey	claim_hash	EffectiveAmountValue	activated_sum,activated_support_sum
FutureEffectiveAmountPrefixRow	k	FutureEffectiveAmountKey	claim_hash	FutureEffectiveAmountValue	future_effective_amount
HashXHistoryHasherPrefixRow	l	HashXHistoryHasherKey	hashX	HashXHistoryHasherValue	hasher

PROTOCOLS.md

LBRY Protocols

These are protocols used in the LBRY network:

Name	Default Port	LBRY-specific Port	Description
Peer Protocol	3333/TCP 4444/TCP (distributed) 5567/TCP (fixed)	Yes	Protocol to pull blobs from the network.
DHT	4444/UDP	Yes	Protocol to discover distributed nodes and if they have a specific blob.
SDK RPC	5279	Yes	Localhost RPC to talk to the SDK. Used by LBRY Desktop to get all claim info.
SDK Media	5280	Yes	Localhost media/streaming server. Used by LBRY Desktop to get all media files.
Reflector Protocol	5566	Yes	Protocol to push blobs to the network.
BitTorrent Local Discovery	6771	No
Blockchain RPC	9245	Yes	RPC to talk to the blockchain. Used by the hub to synchronize the database.
Blockchain P2P	9246	Yes	Protocol between the blockchain nodes.
Tracker	9252	Yes
Reflector Cluster	17946/TCP	Yes	-
Hub Ping	50001/UDP	No
Hub	50001/TCP	No	BIP 40 compatible SPV server with LBRY extensions. Used by the SDK to get all claim info.
Hub SSL	50002/TCP	No	Same as Hub, but over SSL/TLS.

DHT Protocol

The DHT protocol is used to discover and connect with other nodes. It follows the Kademlia specification, with some slight modiciations. In the Kademlia spec there are 4 message types: PING, FIND_NODE, FIND_VALUE and STORE. The protocol uses Bencoding, but is different from the BitTorrent specification (BEP 5).

• Port: 4444/UDP

The root dictionary can hold 5 properties: 0, 1, 2, 3 and 4. Note: Bencoding requires those keys to be strings, but some old software could send those as integers. This is wrong. The request must have all those properties, the response only requires 0, 1, 2 and 3. The error, like the request, also requires all properties.

• The 0 property is the message type, where the value is an integer with 0 for request, 1 for response and 2 for error.
• The 1 property is the message ID, which contains a string of 20 bytes and is unique for every request-response or request-error pair.
• The 2 property is the node ID, which contains a string of 48 bytes and is unique for every node.
• The 3 property is the method (string; one of the 4) in case of a request message type, the response (mixed) in case of a response message type, and the error type (string) in case of a error message type.
• The 4 property is the arguments (list) in case of a request message type, absent in case of a response message type, and the response (string) in case of a error message type.

When a node wants to send an error back (it doesn't need to in every case), then the error message looks like the response message. However, the 3 property now contains the error type and optionally, 4 contains more information.

Ping

The request has a property 3 with value ping.

In protocol version 0, there are no request arguments.

In protocol version 1, there is 1 request argument:

• A dictionary containing protocolVersion with integer value 1.

In both versions, the response has a property 3 with value pong.

Find Node

The request has a property 3 with value findNode.

In protocol version 0, there is 1 request argument:

• A 48-byte string containing a key.

In protocol version 1, there are 2 request arguments:

• A 48-byte string containing a key.
• A dictionary containing protocolVersion with integer value 1.

In both versions, the response has a property 3 with a list as value. Every item in the list is a tuple (another list), containing the node ID as 48-byte string, the IP as string and the port as integer.

Find Value

The request has a property 3 with value findValue.

In protocol version 0, there is 1 request argument:

• A 48-byte string containing a key.

In protocol version 1, there are 2 request arguments:

• A 48-byte string containing a key.
• A dictionary containing p (optionally) with an integer value to indicate the page, and protocolVersion with integer value 1.

In both versions, the response has a property 3 with a dictionary as value:

• The directory at least contains a property token, which is needed when storing values on the connected node. If the node supports protocol version 1, it should have protocolVersion set to integer 1. If it only supports version 0, protocolVersion can be integer 0 or absent.
• Based on if the node has the key, it will return the contacts property or the property where the name is the same as the key argument in the request. If the key is not known by the node, the contacts property is present. Like the findNode function response, the contacts property will contain a list of tuples, containing the node ID, IP address and DHT port number.
• If the key is known by the node, contacts isn't required, but allowed to be present. The response now at least has a property with the same 48 byte long name as the key in the request. This property is a list, where every item in the list is a compact address. This compact address contains information on where the blob can be downloaded.
• The optional p property is an integer with the amount of pages of download locations. If there are no download locations, the p value is integer 0. If the p property is set in the request, the contacts property is absent.

Compact Address

The compact address is a value of 54 bytes. The first 4 bytes are the binary format of the IPv4 address. The next 2 bytes are the TCP port where the blob can be downloaded. The remanining 48 bytes are the node id of the associated DHT peer.

Store

The request has a property 3 with value store.

In protocol version 0, there are 4 request arguments:

• A 48-byte string containing a key (e.g. blob hash).
• A value.
• An original publisher ID.
• An age.

The value is a dictory with the properties token, lbryid and port. The token property holds the token value of the connected node, where the token value is received from an earlier findValue request. The lbryid property holds the node ID of the sending node. The port property holds the port number of where the blob can be downloaded.

In protocol version 1, there are 6 request arguments:

• A blob hash.
• A token.
• A port.
• An original publisher ID.
• An age.
• A dictionary containing protocolVersion with integer value 1.

In both versions, the response has a property 3 with value OK.

Peer Protocol

This protocol provides a way to pull a blob from a specific node. The protocol uses JSON blocks. The end of a JSON block can be detected if the JSON is valid after receiving a }, thereby closing the root object. Another JSON block directly follows it, without any whitespace. This protocol supports composite requests, so multiple requests can be merged into one, as also for the response. For example, one request can both contain requested_blobs and blob_data_payment_rate, and the server will answer with one response containing both available_blobs and blob_data_payment_rate.

• Port: 3333/TCP (distributed peer, previously)
• Port: 4444/TCP (distributed peer)
• Port: 5567/TCP (fixed peer)

Availability

This request will check what blobs are available on the peer. It takes the sent list of blob hashes from requested_blobs, checks everyone of them, and returns a list of available blobs in available_blobs. Also, the request contains lbrycrd_address, which is a boolean property. In the response the lbrycrd_address property is a string containing an address. However, the use of lbrycrd_address for paying for blob data is not implemented yet.

//REQUEST
{"lbrycrd_address":false,"requested_blobs":["aabbcc","ddeeff"]}
  //or
{"lbrycrd_address":true,"requested_blobs":["aabbcc","ddeeff"]}

//RESPONSE
{"lbrycrd_address":"bJxKvpD96kaJLriqVajZ7SaQTsWWyrGQct","requested_blobs":["aabbcc","ddeeff"]}
  //or
{"lbrycrd_address":"bJxKvpD96kaJLriqVajZ7SaQTsWWyrGQct","requested_blobs":["aabbcc"]}
  //or
{"lbrycrd_address":"bJxKvpD96kaJLriqVajZ7SaQTsWWyrGQct","requested_blobs":["ddeeff"]}
  //or
{"lbrycrd_address":"bJxKvpD96kaJLriqVajZ7SaQTsWWyrGQct","requested_blobs":[]}

Payment Rate

The reason behind this request is unknown at this moment. The request contains the blob_data_payment_rate property with a float. The result is always RATE_ACCEPTED, unless the float is below zero. Then the server will respond with RATE_TOO_LOW. At this moment, RATE_UNSET is not used.

//REQUEST
{"blob_data_payment_rate":123.456}

//RESPONSE
{"blob_data_payment_rate":"RATE_ACCEPTED"}
  //or
{"blob_data_payment_rate":"RATE_TOO_LOW"}
  //or
{"blob_data_payment_rate":"RATE_UNSET"}

Blob

This request will get the blob itself. The hash will be sent using requested_blob. The server then reacts with an incoming_blob. If everything is correct, the incoming_blob contains a blob_hash property and a length property. The blob data directly follows after the JSON block response. If an error occurs, the incoming_blob contains an error property (and blob_hash is empty and length is zero).

//REQUEST
{"requested_blob":"aabbcc"}

//RESPONSE
{"incoming_blob":{"blob_hash":"aabbcc","length":123}}/*[Raw Blob Data]*/
  //or
{"incoming_blob":{"blob_hash":"","length":0,"error":"Blob not found"}}

Reflector Protocol

This protocol provides a way to actively push blobs to the network, instead of waiting for them to get pulled. The protocol uses JSON blocks. The end of a JSON block can be detected if the JSON is valid after receiving a }, thereby closing the root object. Another JSON block directly follows it, without any whitespace. At this moment, there will be no error block sent when an error occurs. Version 0 of the protocol supports blobs, version 1 of the protocol also supports SD blobs. Unknown properties are ignored and will not cause an error.

• Port: 5566/TCP

Handshake

Before doing anything with blobs, the handshake should be sent. The version property is required and should be an integer valued 0 or 1. If the server supports that version, it will send back an JSON block where the version property has the same value. If the value property contains an invalid protocol number or a protocol number that isn't supported, the server throws an error and closes the connection.

//REQUEST
{"version":0}
  //or
{"version":1}

//RESPONSE
{"version":0}
  //or
{"version":1}

Blob Hash and Blob Size

After the handshake, blobs can be received. First, the server wants to know more about the blob itself, before it wants to receive the blob data itself. The clients needs to send a JSON block with the blob hash and the blob size. It is important to note that this request is different for normal blobs and SD (Stream Descriptor) blobs. The properties blob_hash and blob_size are required OR the properties sd_blob_hash and sd_blob_size are required. The hash cannot be empty and the size cannot be zero or exceed the maximum blob size.

Then, the server will check if it wants to receive the blob. It can do several checks, e.g. checking if it already has the blob or if the blob is blacklisted. If the server wants the blob, it will send a JSON block with the send_blob property for normal blobs and a JSON block with send_sd_blob for SD blobs. If the server notices that it has the SD blob, but misses some of the blobs defined in the SD blob, it will at those hashes to needed_blobs. If the server doesn't want the blob, the client can send information about another blob.

//REQUEST
{"blob_hash":"aabbcc","blob_size":123} // if version>=0
  //or
{"sd_blob_hash":"ddeeff","sd_blob_size":456} // if version>=1

//RESPONSE
{"send_blob":false} // if version>=0
  //or
{"send_blob":true} // if version>=0
  //or
{"send_sd_blob":false} // if version>=1
  //or
{"send_sd_blob":true} // if version>=1
  //or
{"send_sd_blob":false,"needed_blobs":[]} // if version>=1
  //or
{"send_sd_blob":true,"needed_blobs":[]} // if version>=1
  //or
{"send_sd_blob":false,"needed_blobs":["aabbcc","ddeeff"]} // if version>=1
  //or
{"send_sd_blob":true,"needed_blobs":["aabbcc","ddeeff"]} // if version>=1

Blob Data

If the server wants the blob, it will read exactly the amount of bytes that were stated in the blob information. If there goes something wrong with sending the blob data (e.g. there was a socket timeout or the blob hash calculation didn't match the blob hash from the blob information), the server will send a message that it didn't receive the blob. This will be a JSON block with the received_blob property for normal blobs and a JSON block with received_sd_blob for SD blobs. After this message, the client can send information about another blob.

//REQUEST
  /*[Raw Blob Data]*/

//RESPONSE
{"received_blob":false} // if version>=0
  //or
{"received_blob":true} // if version>=0
  //or
{"received_sd_blob":false} // if version>=1
  //or
{"received_sd_blob":true} // if version>=1