Skip to content

Instantly share code, notes, and snippets.

@frsyuki

frsyuki/presto-connector-dev-1.md

Last active December 10, 2019 22:34
Download ZIP

Revisions

  1. frsyuki revised this gist Nov 19, 2013. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion presto-connector-dev-1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -2,7 +2,7 @@

    One of the very good design decisions Presto designers made is that it's loosely coupled from storages.

    Presto is a distributed SQL executor engine, and doesn't manager schema or metadata of tables by itself. It doesn't manage read data from storage by itself. Those businesses are done by plugins called *Connector*. Presto comes with Hive connector built-in.
    Presto is a distributed SQL executor engine, and doesn't manager schema or metadata of tables by itself. It doesn't manage read data from storage by itself. Those businesses are done by plugins called *Connector*. Presto comes with Hive connector built-in, which connects Hive's metastore and HDFS to Presto.

    We can connect any storages into Presto by writing connector plugins.

  2. frsyuki revised this gist Nov 19, 2013. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions presto-connector-dev-1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,6 +1,7 @@
    # Presto connector development 1

    One of the very good design decisions Presto designers made is that it's loosely coupled from storages.

    Presto is a distributed SQL executor engine, and doesn't manager schema or metadata of tables by itself. It doesn't manage read data from storage by itself. Those businesses are done by plugins called *Connector*. Presto comes with Hive connector built-in.

    We can connect any storages into Presto by writing connector plugins.
  3. frsyuki revised this gist Nov 19, 2013. 1 changed file with 4 additions and 2 deletions.
    6 changes: 4 additions & 2 deletions presto-connector-dev-1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,7 +1,9 @@
    # Presto connector development 1

    Presto has a concept named "connector", which provides data and metadata of the data source.
    We can plug-in our own connectors Presto so that it can read data from the data source.
    One of the very good design decisions Presto designers made is that it's loosely coupled from storages.
    Presto is a distributed SQL executor engine, and doesn't manager schema or metadata of tables by itself. It doesn't manage read data from storage by itself. Those businesses are done by plugins called *Connector*. Presto comes with Hive connector built-in.

    We can connect any storages into Presto by writing connector plugins.

    ## Plugin Architecture

  4. frsyuki revised this gist Nov 19, 2013. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion presto-connector-dev-1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -11,7 +11,9 @@ Plugin needs to implement **List<T> getServices(Class<T> type)**.

    ### PluginManager

    `presto.server.PluginManager` calls `Plugin.getServices(ConnectorFactory.class)` to get **presto.spi.ConnectorFactory** implementations. Plugin needs to instantiate ConnectorFactory and return it.
    `presto.server.PluginManager` calls `Plugin.getServices(ConnectorFactory.class)` to get **presto.spi.ConnectorFactory** implementations.
    Plugin needs to instantiate ConnectorFactory and return it.

    PluginManager registers them to `presto.connector.ConnectorManager` (in presto-main directory).
    ConnectorFactory needs to implement **String getName()** and **Connector create(connectorId, config)** methods.

  5. frsyuki revised this gist Nov 19, 2013. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion presto-connector-dev-1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -11,7 +11,7 @@ Plugin needs to implement **List<T> getServices(Class<T> type)**.

    ### PluginManager

    `presto.server.PluginManager` calls `Plugin.getServices(ConnectorFactory.class)` to get **presto.spi.ConnectorFactory** implementations.
    `presto.server.PluginManager` calls `Plugin.getServices(ConnectorFactory.class)` to get **presto.spi.ConnectorFactory** implementations. Plugin needs to instantiate ConnectorFactory and return it.
    PluginManager registers them to `presto.connector.ConnectorManager` (in presto-main directory).
    ConnectorFactory needs to implement **String getName()** and **Connector create(connectorId, config)** methods.

  6. frsyuki revised this gist Nov 19, 2013. 1 changed file with 12 additions and 12 deletions.
    24 changes: 12 additions & 12 deletions presto-connector-dev-1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -53,22 +53,22 @@ ConnectorManager registers those instances into other managers:

    Presto has 4 ConnectorFactory implementations:

    * presto.hive.HiveConnectorFactory
    * **presto.hive.HiveConnectorFactory**
    * Reads metadata from Hive metastore
    * Reads records from HDFS using Hive record readers
    * presto.connector.jmx.JmxConnectorFactory
    * Reads metadata from a remote javax.management.MBeanServer server
    * Reads records from a remote javax.management.MBeanServer server
    * **presto.connector.jmx.JmxConnectorFactory**
    * Reads metadata from a remote `javax.management.MBeanServer` server
    * Reads records from a remote `javax.management.MBeanServer` server
    * It gets mbeanServer.getAttributes and convert the attributes to records
    * presto.connector.NativeConnectorFactor
    * **presto.connector.NativeConnectorFactor**
    * Reads metadata from RDBMS
    * Reads records from interface presto.metadata.LocalStorageManager
    * Reads records from interface `presto.metadata.LocalStorageManager`
    * actual instance of LocalStorageManager is injected by Dependency Injection
    * Presto has one built-in implementation presto.metadata.DatabaseLocalStorageManager
    * Presto has one built-in implementation `presto.metadata.DatabaseLocalStorageManager`
    * DatabaseLocalStorageManager reads data from local files; list of files are managed by RDBMS again
    * presto.tpch.TpchConnectorFactory
    * Metadata is hard coded in TpchMetadata class
    * Read records from TpchBlocksProvider interface; Presto has 2 implementations:
    * DataFileTpchBlocksProvider to read data from local files
    * InMemoryTpchBlocksProvider to read data in memory
    * **presto.tpch.TpchConnectorFactory**
    * Metadata is hard coded in `TpchMetadata` class
    * Read records from `TpchBlocksProvider` interface; Presto has 2 implementations:
    * `DataFileTpchBlocksProvider` to read data from local files
    * `InMemoryTpchBlocksProvider` to read data in memory
    * Tpch is mainly used for benchmark and test
  7. frsyuki revised this gist Nov 19, 2013. 1 changed file with 4 additions and 4 deletions.
    8 changes: 4 additions & 4 deletions presto-connector-dev-1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -44,10 +44,10 @@ Connector needs to implement **T getService(Class<T> type)** method. ConnectorMa
    In other words, to create our own Connector, we need to implement above interfaces.
    ConnectorManager registers those instances into other managers:

    * ConnectorMetadata to presto.metadata.MetadataManager
    * ConnectorSplitManager to presto.split.SplitManager
    * ConnectorDataStreamProvider to presto.split.DataStreamManager
    * ConnectorHandleResolver to presto.metadata.HandleResolver
    * ConnectorMetadata to `presto.metadata.MetadataManager`
    * ConnectorSplitManager to `presto.split.SplitManager`
    * ConnectorDataStreamProvider to `presto.split.DataStreamManager`
    * ConnectorHandleResolver to `presto.metadata.HandleResolver`

    ## Examples

  8. frsyuki revised this gist Nov 19, 2013. 1 changed file with 5 additions and 5 deletions.
    10 changes: 5 additions & 5 deletions presto-connector-dev-1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -34,12 +34,12 @@ CatalogManager does almost nothing excepting loading configuration files for Con
    Actually connector has similar design with Plugin.
    Connector needs to implement **T getService(Class<T> type)** method. ConnectorManager calls the method for each following classes:

    * connector.getService(ConnectorMetadata.class)
    * connector.getService(ConnectorSplitManager.class)
    * connector.getService(ConnectorDataStreamProvider.class)
    * connector.getService(**ConnectorMetadata.class**)
    * connector.getService(**ConnectorSplitManager.class**)
    * connector.getService(**ConnectorDataStreamProvider.class**)
    * DataStreamProvider can be null. If it's null, it calls:
    * connector.getService(ConnectorRecordSetProvider.class)
    * connector.getService(ConnectorHandleResolver.class)
    * connector.getService(**ConnectorRecordSetProvider.class**)
    * connector.getService(**ConnectorHandleResolver.class**)

    In other words, to create our own Connector, we need to implement above interfaces.
    ConnectorManager registers those instances into other managers:
  9. frsyuki revised this gist Nov 19, 2013. 1 changed file with 9 additions and 7 deletions.
    16 changes: 9 additions & 7 deletions presto-connector-dev-1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,19 +1,21 @@
    # Presto connector development 1

    Presto has a concept named "connector", which provides data and metadata of the data source.
    We can plug-in our own connectors Presto so that it can read data from the data source.

    # Plugin Architecture
    ## Plugin Architecture

    ## Plugin interface
    ### Plugin interface
    **interface presto.spi.Plugin** is the entry point of any kinds of plugins of Presto. It uses `presto.server.PluginManager` loads them using Java's standard [java.util.ServiceLoader](https://www.google.com/search?q=java+serviceloader).
    Plugin needs to implement **List<T> getServices(Class<T> type)**.

    ## PluginManager
    ### PluginManager

    `presto.server.PluginManager` calls `Plugin.getServices(ConnectorFactory.class)` to get **presto.spi.ConnectorFactory** implementations.
    PluginManager registers them to `presto.connector.ConnectorManager` (in presto-main directory).
    ConnectorFactory needs to implement **String getName()** and **Connector create(connectorId, config)** methods.

    ## ConnectorManager
    ### ConnectorManager

    ConnectorManager calls `ConnectorFactory.create` when it gets `createConnection()` request from `presto.metadata.CatalogManager`.
    CatalogManager instance is owned by `presto.server.PrestoServer`. So, the relationship is:
    @@ -22,12 +24,12 @@ CatalogManager instance is owned by `presto.server.PrestoServer`. So, the relati
    * CatalogManager has many ConnectorManager(s)
    * ConnectorManager has many Connector(s) (managed in Map<String, Connector> name => connector)

    ## CatalogManager
    ### CatalogManager

    `CatalogManager.loadCatalogs()` reads files in a directory. Each file contains configuration of a connector. Its file format is Java's property file.
    CatalogManager does almost nothing excepting loading configuration files for Connector.

    ## Connector
    ### Connector

    Actually connector has similar design with Plugin.
    Connector needs to implement **T getService(Class<T> type)** method. ConnectorManager calls the method for each following classes:
    @@ -47,7 +49,7 @@ ConnectorManager registers those instances into other managers:
    * ConnectorDataStreamProvider to presto.split.DataStreamManager
    * ConnectorHandleResolver to presto.metadata.HandleResolver

    # Examples
    ## Examples

    Presto has 4 ConnectorFactory implementations:

  10. frsyuki revised this gist Nov 19, 2013. 1 changed file with 29 additions and 24 deletions.
    53 changes: 29 additions & 24 deletions presto-connector-dev-1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -38,30 +38,35 @@ Connector needs to implement **T getService(Class<T> type)** method. ConnectorMa
    * DataStreamProvider can be null. If it's null, it calls:
    * connector.getService(ConnectorRecordSetProvider.class)
    * connector.getService(ConnectorHandleResolver.class)

    In other words, to create our own Connector, we need to implement above interfaces.
    ConnectorManager registers those instances into other managers:
    ConnectorMetadata to presto.metadata.MetadataManager
    ConnectorSplitManager to presto.split.SplitManager
    ConnectorDataStreamProvider to presto.split.DataStreamManager
    ConnectorHandleResolver to presto.metadata.HandleResolver
    Examples

    * ConnectorMetadata to presto.metadata.MetadataManager
    * ConnectorSplitManager to presto.split.SplitManager
    * ConnectorDataStreamProvider to presto.split.DataStreamManager
    * ConnectorHandleResolver to presto.metadata.HandleResolver

    # Examples

    Presto has 4 ConnectorFactory implementations:
    presto.hive.HiveConnectorFactory
    Reads metadata from Hive metastore
    Reads records from HDFS using Hive record readers
    presto.connector.jmx.JmxConnectorFactory
    Reads metadata from a remote javax.management.MBeanServer server
    Reads records from a remote javax.management.MBeanServer server
    It gets mbeanServer.getAttributes and convert the attributes to records
    presto.connector.NativeConnectorFactor
    Reads metadata from RDBMS
    Reads records from interface presto.metadata.LocalStorageManager
    actual instance of LocalStorageManager is injected by Dependency Injection
    Presto has one built-in implementation presto.metadata.DatabaseLocalStorageManager
    DatabaseLocalStorageManager reads data from local files; list of files are managed by RDBMS again
    presto.tpch.TpchConnectorFactory
    Metadata is hard coded in TpchMetadata class
    Read records from TpchBlocksProvider interface; Presto has 2 implementations:
    DataFileTpchBlocksProvider to read data from local files
    InMemoryTpchBlocksProvider to read data in memory
    Tpch is mainly used for benchmark and test

    * presto.hive.HiveConnectorFactory
    * Reads metadata from Hive metastore
    * Reads records from HDFS using Hive record readers
    * presto.connector.jmx.JmxConnectorFactory
    * Reads metadata from a remote javax.management.MBeanServer server
    * Reads records from a remote javax.management.MBeanServer server
    * It gets mbeanServer.getAttributes and convert the attributes to records
    * presto.connector.NativeConnectorFactor
    * Reads metadata from RDBMS
    * Reads records from interface presto.metadata.LocalStorageManager
    * actual instance of LocalStorageManager is injected by Dependency Injection
    * Presto has one built-in implementation presto.metadata.DatabaseLocalStorageManager
    * DatabaseLocalStorageManager reads data from local files; list of files are managed by RDBMS again
    * presto.tpch.TpchConnectorFactory
    * Metadata is hard coded in TpchMetadata class
    * Read records from TpchBlocksProvider interface; Presto has 2 implementations:
    * DataFileTpchBlocksProvider to read data from local files
    * InMemoryTpchBlocksProvider to read data in memory
    * Tpch is mainly used for benchmark and test
  11. frsyuki revised this gist Nov 19, 2013. 1 changed file with 17 additions and 14 deletions.
    31 changes: 17 additions & 14 deletions presto-connector-dev-1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -4,14 +4,14 @@ We can plug-in our own connectors Presto so that it can read data from the data
    # Plugin Architecture

    ## Plugin interface
    `interface presto.spi.Plugin` is the entry point of any kinds of plugins of Presto. It uses `presto.server.PluginManager` loads them using Java's standard [java.util.ServiceLoader](https://www.google.com/search?q=java+serviceloader).
    Plugin needs to implement `List<T> getServices(Class<T> type)`.
    **interface presto.spi.Plugin** is the entry point of any kinds of plugins of Presto. It uses `presto.server.PluginManager` loads them using Java's standard [java.util.ServiceLoader](https://www.google.com/search?q=java+serviceloader).
    Plugin needs to implement **List<T> getServices(Class<T> type)**.

    ## PluginManager

    `presto.server.PluginManager` calls `Plugin.getServices(ConnectorFactory.class)` to get `presto.spi.ConnectorFactory` implementations.
    `presto.server.PluginManager` calls `Plugin.getServices(ConnectorFactory.class)` to get **presto.spi.ConnectorFactory** implementations.
    PluginManager registers them to `presto.connector.ConnectorManager` (in presto-main directory).
    ConnectorFactory needs to implement `String getName()` and `Connector create(connectorId, config)` methods.
    ConnectorFactory needs to implement **String getName()** and **Connector create(connectorId, config)** methods.

    ## ConnectorManager

    @@ -24,17 +24,20 @@ CatalogManager instance is owned by `presto.server.PrestoServer`. So, the relati

    ## CatalogManager

    **CatalogManager.loadCatalogs()** reads files in a directory. Each file contains configuration of a connector. Its file format is Java's property file.
    `CatalogManager.loadCatalogs()` reads files in a directory. Each file contains configuration of a connector. Its file format is Java's property file.
    CatalogManager does almost nothing excepting loading configuration files for Connector.
    Connector
    Actually connector has similar design with Plugin (to be extensible, probably).
    Connector needs to implement T getService(Class<T> type) method. ConnectorManager calls the method for each following classes:
    connector.getService(ConnectorMetadata.class)
    connector.getService(ConnectorSplitManager.class)
    connector.getService(ConnectorDataStreamProvider.class)
    DataStreamProvider can be null. If it's null, it calls:
    connector.getService(ConnectorRecordSetProvider.class)
    connector.getService(ConnectorHandleResolver.class)

    ## Connector

    Actually connector has similar design with Plugin.
    Connector needs to implement **T getService(Class<T> type)** method. ConnectorManager calls the method for each following classes:

    * connector.getService(ConnectorMetadata.class)
    * connector.getService(ConnectorSplitManager.class)
    * connector.getService(ConnectorDataStreamProvider.class)
    * DataStreamProvider can be null. If it's null, it calls:
    * connector.getService(ConnectorRecordSetProvider.class)
    * connector.getService(ConnectorHandleResolver.class)
    In other words, to create our own Connector, we need to implement above interfaces.
    ConnectorManager registers those instances into other managers:
    ConnectorMetadata to presto.metadata.MetadataManager
  12. frsyuki revised this gist Nov 19, 2013. 1 changed file with 19 additions and 12 deletions.
    31 changes: 19 additions & 12 deletions presto-connector-dev-1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -6,18 +6,25 @@ We can plug-in our own connectors Presto so that it can read data from the data
    ## Plugin interface
    `interface presto.spi.Plugin` is the entry point of any kinds of plugins of Presto. It uses `presto.server.PluginManager` loads them using Java's standard [java.util.ServiceLoader](https://www.google.com/search?q=java+serviceloader).
    Plugin needs to implement `List<T> getServices(Class<T> type)`.
    PluginManager
    presto.server.PluginManager calls Plugin.getServices(ConnectorFactory.class) to get presto.spi.ConnectorFactory implementations.
    PluginManager registers them to presto.connector.ConnectorManager (in presto-main directory).
    ConnectorFactory needs to implement String getName() and Connector create(connectorId, config).
    ConnectorManager
    ConnectorManager calls ConnectorFactory.create when it gets createConnection() request from presto.metadata.CatalogManager.
    presto.server.PrestoServer has CatalogManager.
    PrestoServer has a CatalogManager
    CatalogManager has many ConnectorManager(s)
    ConnectorManager has many Connector(s) (managed in Map<String, Connector> name => connector)
    CatalogManager
    CatalogManager.loadCatalogs() reads files in a directory. Each file contains configuration of a connector. Its file format is Java's property file.

    ## PluginManager

    `presto.server.PluginManager` calls `Plugin.getServices(ConnectorFactory.class)` to get `presto.spi.ConnectorFactory` implementations.
    PluginManager registers them to `presto.connector.ConnectorManager` (in presto-main directory).
    ConnectorFactory needs to implement `String getName()` and `Connector create(connectorId, config)` methods.

    ## ConnectorManager

    ConnectorManager calls `ConnectorFactory.create` when it gets `createConnection()` request from `presto.metadata.CatalogManager`.
    CatalogManager instance is owned by `presto.server.PrestoServer`. So, the relationship is:

    * PrestoServer has a CatalogManager
    * CatalogManager has many ConnectorManager(s)
    * ConnectorManager has many Connector(s) (managed in Map<String, Connector> name => connector)

    ## CatalogManager

    **CatalogManager.loadCatalogs()** reads files in a directory. Each file contains configuration of a connector. Its file format is Java's property file.
    CatalogManager does almost nothing excepting loading configuration files for Connector.
    Connector
    Actually connector has similar design with Plugin (to be extensible, probably).
  13. frsyuki renamed this gist Nov 19, 2013. 1 changed file with 0 additions and 0 deletions.
    0 presto-connector-dev-1.txt → presto-connector-dev-1.md
    View file
    File renamed without changes.
  14. frsyuki created this gist Nov 19, 2013.
    57 changes: 57 additions & 0 deletions presto-connector-dev-1.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,57 @@
    Presto has a concept named "connector", which provides data and metadata of the data source.
    We can plug-in our own connectors Presto so that it can read data from the data source.

    # Plugin Architecture

    ## Plugin interface
    `interface presto.spi.Plugin` is the entry point of any kinds of plugins of Presto. It uses `presto.server.PluginManager` loads them using Java's standard [java.util.ServiceLoader](https://www.google.com/search?q=java+serviceloader).
    Plugin needs to implement `List<T> getServices(Class<T> type)`.
    PluginManager
    presto.server.PluginManager calls Plugin.getServices(ConnectorFactory.class) to get presto.spi.ConnectorFactory implementations.
    PluginManager registers them to presto.connector.ConnectorManager (in presto-main directory).
    ConnectorFactory needs to implement String getName() and Connector create(connectorId, config).
    ConnectorManager
    ConnectorManager calls ConnectorFactory.create when it gets createConnection() request from presto.metadata.CatalogManager.
    presto.server.PrestoServer has CatalogManager.
    PrestoServer has a CatalogManager
    CatalogManager has many ConnectorManager(s)
    ConnectorManager has many Connector(s) (managed in Map<String, Connector> name => connector)
    CatalogManager
    CatalogManager.loadCatalogs() reads files in a directory. Each file contains configuration of a connector. Its file format is Java's property file.
    CatalogManager does almost nothing excepting loading configuration files for Connector.
    Connector
    Actually connector has similar design with Plugin (to be extensible, probably).
    Connector needs to implement T getService(Class<T> type) method. ConnectorManager calls the method for each following classes:
    connector.getService(ConnectorMetadata.class)
    connector.getService(ConnectorSplitManager.class)
    connector.getService(ConnectorDataStreamProvider.class)
    DataStreamProvider can be null. If it's null, it calls:
    connector.getService(ConnectorRecordSetProvider.class)
    connector.getService(ConnectorHandleResolver.class)
    In other words, to create our own Connector, we need to implement above interfaces.
    ConnectorManager registers those instances into other managers:
    ConnectorMetadata to presto.metadata.MetadataManager
    ConnectorSplitManager to presto.split.SplitManager
    ConnectorDataStreamProvider to presto.split.DataStreamManager
    ConnectorHandleResolver to presto.metadata.HandleResolver
    Examples
    Presto has 4 ConnectorFactory implementations:
    presto.hive.HiveConnectorFactory
    Reads metadata from Hive metastore
    Reads records from HDFS using Hive record readers
    presto.connector.jmx.JmxConnectorFactory
    Reads metadata from a remote javax.management.MBeanServer server
    Reads records from a remote javax.management.MBeanServer server
    It gets mbeanServer.getAttributes and convert the attributes to records
    presto.connector.NativeConnectorFactor
    Reads metadata from RDBMS
    Reads records from interface presto.metadata.LocalStorageManager
    actual instance of LocalStorageManager is injected by Dependency Injection
    Presto has one built-in implementation presto.metadata.DatabaseLocalStorageManager
    DatabaseLocalStorageManager reads data from local files; list of files are managed by RDBMS again
    presto.tpch.TpchConnectorFactory
    Metadata is hard coded in TpchMetadata class
    Read records from TpchBlocksProvider interface; Presto has 2 implementations:
    DataFileTpchBlocksProvider to read data from local files
    InMemoryTpchBlocksProvider to read data in memory
    Tpch is mainly used for benchmark and test