Pierre Villard pvillard31
pvillard31
/ MinIO.json
Created
July 21, 2023 09:23
Flow to interact with MinIO using Apache NiFi S3 processors
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {"flowContents":{"identifier":"cbeea532-a4db-35cb-8040-717e36e7dbf4","instanceIdentifier":"01871049-ede0-1bf6-2a90-c0a8f25c44ec","name":"MinIO","comments":"","position":{"x":1600.0,"y":-232.0},"processGroups":[],"remoteProcessGroups":[],"processors":[{"identifier":"e23ddc8b-757a-3822-88ef-e5ff26444c60","instanceIdentifier":"681634d1-d0ae-1e7c-3438-d000d6c67f56","name":"FetchS3Object","comments":"","position":{"x":472.0,"y":176.0},"type":"org.apache.nifi.processors.aws.s3.FetchS3Object","bundle":{"group":"org.apache.nifi","artifact":"nifi-aws-nar","version":"2.0.0-SNAPSHOT"},"properties":{"Proxy Host":null,"range-length":null,"Access Key":"#{Access Key ID}","range-start":null,"proxy-configuration-service":null,"Endpoint Override URL":"#{Endpoint Override URL}","custom-signer-class-name":null,"requester-pays":"false","Object Key":"${filename}","AWS Credentials Provider service":null,"custom-signer-module-location":null,"SSL Context Service":null,"Signer Override":"Default Signature","Bucket":"#{Bucket}","Versio |
pvillard31
/ deploy.sh
Created
March 22, 2023 11:00
deploy.sh for automated deployment of flows via NiFi Registry scripted hook
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| LOG="/hadoopfs/fs1/automation/nifi-cli.log" | |
| echo $@ >> $LOG | |
| EVENT=$1 | |
| BUCKET=$2 | |
| FLOW=$3 | |
| VERSION=$4 |
pvillard31
/ template.json
Created
October 21, 2022 02:14
Simple template for DataFlow Functions deployment in Azure Functions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#", | |
| "contentVersion": "1.0.0.0", | |
| "parameters": { | |
| "subscriptionId": { | |
| "type": "string" | |
| }, | |
| "name": { | |
| "type": "string" | |
| }, |
pvillard31
/ my-function-definition.json
Created
October 21, 2022 02:03
Simple template for DataFlow Functions deployment in AWS Lambda
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "FunctionName": "FUNCTIONNAME", | |
| "Timeout": 300, | |
| "MemorySize": 1024, | |
| "Environment": { | |
| "Variables": { | |
| "DF_ACCESS_KEY": "DFACCESSKEY", | |
| "DF_PRIVATE_KEY": "DFPRIVATEKEY", | |
| "FLOW_CRN": "FLOWCRN", | |
| "EXTENSIONS_DIR_1": "/opt", |
pvillard31
/ Resize-Image-Function.json
Created
October 21, 2022 02:02
NiFi flow to be executed as a function for resizing images with an HTTP trigger
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {"flowContents":{"identifier":"5c277775-7108-338e-8fde-78c8863a62fc","instanceIdentifier":"fdc668c2-0182-1000-62e2-28dd424f41d2","name":"ImageResizeFunction","comments":"","position":{"x":560.0,"y":88.0},"processGroups":[{"identifier":"9ac7f940-5a58-3aff-b5c0-c6431737a210","instanceIdentifier":"c352162d-37d7-1087-17c5-8a26eaafdb42","name":"AZURE","comments":"","position":{"x":784.0,"y":560.0},"processGroups":[],"remoteProcessGroups":[],"processors":[{"identifier":"ae298fa6-4043-3412-be7c-d2d3f1d9d049","instanceIdentifier":"c3521631-37d7-1087-250e-ce6fc61f3142","name":"UpdateAttribute","comments":"","position":{"x":648.0,"y":144.0},"type":"org.apache.nifi.processors.attributes.UpdateAttribute","bundle":{"group":"org.apache.nifi","artifact":"nifi-update-attribute-nar","version":"1.18.0"},"properties":{"Delete Attributes Expression":null,"Store State":"Do not store state","canonical-value-lookup-cache-size":"100","resizeWidth":"${azure.http.header.resize-width:isEmpty():ifElse(#{defaultWidth},${azure.http.header |
pvillard31
/ NiFi_Function_Quickstart-definition.json
Created
September 22, 2022 08:26
NiFi_Function_Quickstart-definition.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "FunctionName": "NiFi_Function_Quickstart", | |
| "Description": "NiFi Function that uses the AWS S3 Lambda trigger to move telemetry data from landing zone S3 folder into processed S3 folder", | |
| "Timeout": 300, | |
| "MemorySize": 4096, | |
| "Environment": { | |
| "Variables": { | |
| "DF_ACCESS_KEY": "<<REPLACE>>", | |
| "DF_PRIVATE_KEY": "<<REPLACE>>", | |
| "FLOW_CRN": "<<REPLACE>>", |
pvillard31
/ DataFlowFunctionsQuickstart.json
Last active
September 22, 2022 08:16
DataFlowFunctionsQuickstart.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "flowContents": { | |
| "identifier": "f7ec47e6-9057-30b0-abb7-0b38eb709ef6", | |
| "instanceIdentifier": "a4c2cb58-0182-1000-565f-6ad654e140b8", | |
| "name": "DataFlowFunctionsQuickstart", | |
| "comments": "", | |
| "position": { | |
| "x": 536.0, | |
| "y": 768.0 | |
| }, |
pvillard31
/ template.json
Created
September 8, 2022 15:54
Template for deploying a Function App in Azure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#", | |
| "contentVersion": "1.0.0.0", | |
| "parameters": { | |
| "subscriptionId": { | |
| "type": "string" | |
| }, | |
| "name": { | |
| "type": "string" | |
| }, |
pvillard31
/ sidecar-example.yml
Created
February 1, 2022 14:05
sidecar-example.yml for MiNiFi and NiFi
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| apiVersion: v1 | |
| data: | |
| # TODO: replace with own minifi properties | |
| minifi.properties: | | |
| # Core Properties # | |
| nifi.version=0.11.0 | |
| nifi.flow.configuration.file=./conf/config.yml | |
| nifi.administrative.yield.duration=30 sec | |
| # If a component has no work to do (is "bored"), how long should we wait before checking again for work? | |
| nifi.bored.yield.duration=100 millis |
pvillard31
/ syncDataFlow.sh
Created
December 24, 2021 19:29
Script to be used with the NiFi Registry Event Script Hook for automatic sync with the Cloudera DataFlow Catalog
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/sh | |
| set -e | |
| # This script is used to automatically sync new flows into the DataFlow Catalog | |
| # It assumes that the CDP CLI has been installed first. See: | |
| # https://docs.cloudera.com/dataflow/cloud/cli/index.html | |
| # It also assumes that the NiFi CLI has been properly configured to interact with the NiFi Registry instance. See: | |
| # https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#nifi_CLI | |
| NIFICLI="/opt/cloudera/parcels/CFM/TOOLKIT/bin/cli.sh" |
NewerOlder