Introduction:

Introduction: The Power of Plugins

Imagine you're a master chef. You don't just have one knife - you have specialized tools for every task: a paring knife for delicate work, a chef's knife for chopping, a boning knife for meat, and a bread knife for, well, bread. That's what Kestra's plugin ecosystem gives you: specialized tools for every data engineering task.

In this article, you'll learn to:

• Navigate Kestra's vast plugin library (150+ plugins and counting)

• Create your own custom plugins for proprietary systems

• Build plugin-based architectures for enterprise integration

• Debug and optimize plugin performance

• Contribute to the open-source plugin ecosystem

Quick Stats:

• 150+ official plugins available

• 1,000+ companies using Kestra plugins

• 10,000+ plugin downloads per week

• 0 lines of code needed for most integrations

Part 1: The Plugin Architecture

Understanding Plugin Layers

Kestra's plugin architecture is like a LEGO system:

Application Layer (Your Flows)
    ↓
Plugin Interface (Java Interfaces)
    ↓
Plugin Implementation (Built-in/Custom)
    ↓
Target System (Databases, APIs, Cloud Services)

Core Plugin Types

# Every plugin belongs to one of these categories
plugins:
  - Input/Output:
    - Filesystems: S3, GCS, Azure Blob, Local, FTP
    - Databases: PostgreSQL, MySQL, Snowflake, BigQuery
    - APIs: REST, GraphQL, SOAP

  - Processing:
    - Scripting: Python, R, Node.js, Bash
    - Data Processing: Spark, Flink, Pandas
    - ML/AI: TensorFlow, PyTorch, Hugging Face

  - Messaging:
    - Queues: Kafka, RabbitMQ, AWS SQS
    - Streaming: Kafka Streams, Apache Pulsar

  - Cloud Services:
    - AWS: S3, Lambda, Glue, Redshift
    - GCP: BigQuery, Cloud Functions, Dataflow
    - Azure: Blob Storage, Data Factory, Synapse

  - Monitoring:
    - Metrics: Prometheus, Datadog, New Relic
    - Logging: ELK, Splunk, CloudWatch
    - Alerting: Slack, PagerDuty, Email

  - Custom:
    - Enterprise Systems: SAP, Salesforce, Oracle
    - Internal APIs: Your company's systems
    - Legacy Systems: Mainframes, AS/400

Plugin Anatomy: What Makes a Good Plugin?

// The anatomy of a Kestra plugin (simplified)
public class MyPlugin extends Task implements FlowableTask {

    // 1. Configuration (YAML properties)
    @PluginProperty private String connectionString;
    @PluginProperty private Integer timeout;

    // 2. Execution logic
    @Override
    public List<TaskRun> run(RunContext runContext) throws Exception {
        // Your plugin logic here
        return List.of(TaskRun.of(this, Outputs.of("result", "success")));
    }

    // 3. Input/Output handling
    public interface Output {
        String result();
    }
}

Part 2: Using Built-in Plugins

Plugin Discovery and Installation

# List available plugins
kestra plugins list

# Search for specific plugins
kestra plugins search --query "postgres"

# Install a plugin
kestra plugins install kestra-plugin-jdbc-postgresql

# View installed plugins
kestra plugins installed

# Update all plugins
kestra plugins update-all

Database Plugins Deep Dive

# flows/database-demo.yml
id: database-plugin-showcase
namespace: demo.plugins
description: Showcase of database plugins

tasks:
  # PostgreSQL - Full-featured SQL database
  - id: postgres_example
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: jdbc:postgresql://localhost:5432/mydb
    username: "{{ secret('PG_USER') }}"
    password: "{{ secret('PG_PASS') }}"
    sql: |
      -- Parameterized query
      SELECT * FROM users 
      WHERE created_at > '{{ execution.startDate | dateAdd(-7, 'DAYS') }}'
        AND status = '{{ inputs.user_status }}'
    fetch: true
    fetchSize: 1000
    store: true

  # Snowflake - Cloud data warehouse  
  - id: snowflake_example
    type: io.kestra.plugin.jdbc.snowflake.Query
    url: jdbc:snowflake://account.snowflakecomputing.com
    username: "{{ secret('SNOWFLAKE_USER') }}"
    password: "{{ secret('SNOWFLAKE_PASS') }}"
    role: SYSADMIN
    warehouse: COMPUTE_WH
    database: PRODUCTION
    schema: ANALYTICS
    sql: |
      COPY INTO @my_stage/users.parquet
      FROM (
        SELECT * FROM users 
        WHERE DAY = '{{ execution.startDate | date('yyyy-MM-dd') }}'
      )
      FILE_FORMAT = (TYPE = PARQUET)
    store: true

  # BigQuery - Serverless data warehouse
  - id: bigquery_example
    type: io.kestra.plugin.gcp.bigquery.Query
    sql: |
      SELECT 
        user_id,
        COUNT(*) as session_count,
        SUM(duration) as total_duration
      FROM `project.dataset.sessions`
      WHERE DATE(timestamp) = '{{ execution.startDate | date('yyyy-MM-dd') }}'
      GROUP BY user_id
    destinationTable:
      projectId: my-project
      datasetId: results
      tableId: daily_sessions
    writeDisposition: WRITE_TRUNCATE
    createDisposition: CREATE_IF_NEEDED

  # ClickHouse - OLAP database
  - id: clickhouse_example
    type: io.kestra.plugin.jdbc.clickhouse.Query
    url: jdbc:clickhouse://localhost:8123/analytics
    sql: |
      SELECT 
        toStartOfDay(timestamp) as day,
        uniq(user_id) as daily_active_users,
        count(*) as events
      FROM events
      WHERE timestamp >= '{{ execution.startDate | dateAdd(-30, 'DAYS') }}'
      GROUP BY day
      ORDER BY day
    fetch: true

  # Multi-database Join (Virtual Data Warehouse)
  - id: cross_database_join
    type: io.kestra.plugin.scripts.python.Script
    description: "Join data from multiple databases"
    inputFiles:
      postgres_data.json: "{{ outputs.postgres_example.uri }}"
      snowflake_data.json: "{{ outputs.snowflake_example.uri }}"
      bigquery_data.json: "{{ outputs.bigquery_example.uri }}"
    script: |
      import pandas as pd
      import json

      # Load from different sources
      pg_df = pd.read_json('postgres_data.json')
      sf_df = pd.read_json('snowflake_data.json')
      bq_df = pd.read_json('bigquery_data.json')

      # Perform cross-database join
      merged = pd.merge(
          pg_df, 
          sf_df, 
          on='user_id',
          how='left'
      ).merge(
          bq_df,
          on='user_id',
          how='left'
      )

      # Save result
      merged.to_parquet('cross_database_result.parquet', index=False)
      print(f"Merged {len(merged)} records from 3 databases")

Cloud Storage Plugins

# flows/cloud-storage.yml
id: cloud-storage-operations
namespace: demo.plugins

tasks:
  # AWS S3 Operations
  - id: s3_operations
    type: io.kestra.plugin.core.flow.Sequential
    tasks:
      - id: upload_to_s3
        type: io.kestra.plugin.aws.s3.Upload
        from: "data/output.parquet"
        bucket: my-data-lake
        key: "raw/{{ execution.startDate | date('yyyy/MM/dd') }}/data.parquet"
        region: us-east-1
        accessKeyId: "{{ secret('AWS_ACCESS_KEY') }}"
        secretKeyId: "{{ secret('AWS_SECRET_KEY') }}"

      - id: list_s3_files
        type: io.kestra.plugin.aws.s3.List
        bucket: my-data-lake
        region: us-east-1
        prefix: "raw/{{ execution.startDate | date('yyyy/MM/dd') }}/"

      - id: sync_s3_to_local
        type: io.kestra.plugin.aws.s3.Download
        bucket: my-data-lake
        key: "{{ outputs.list_s3_files.uris[0] }}"
        region: us-east-1

  # Google Cloud Storage
  - id: gcs_operations
    type: io.kestra.plugin.gcp.gcs.Upload
    from: "data/processed.parquet"
    bucket: my-gcp-bucket
    name: "processed/{{ execution.startDate | date('yyyy-MM-dd') }}.parquet"
    projectId: my-gcp-project

  # Azure Blob Storage
  - id: azure_operations
    type: io.kestra.plugin.azure.blob.Upload
    from: "data/archive.parquet"
    connectionString: "{{ secret('AZURE_CONNECTION_STRING') }}"
    container: my-container
    name: "archive/{{ execution.startDate | date('yyyy/MM') }}/data.parquet"

  # Multi-cloud synchronization
  - id: multi_cloud_sync
    type: io.kestra.plugin.core.flow.Parallel
    tasks:
      - id: sync_to_aws
        type: io.kestra.plugin.aws.s3.Copy
        from: "{{ outputs.gcs_operations.uri }}"
        to: "s3://my-data-lake/gcs-mirror/{{ execution.startDate | date('yyyy-MM-dd') }}.parquet"

      - id: sync_to_azure
        type: io.kestra.plugin.azure.blob.Copy
        from: "{{ outputs.gcs_operations.uri }}"
        to: "https://myaccount.blob.core.windows.net/mycontainer/gcs-mirror/data.parquet"

Messaging and Streaming Plugins

# flows/messaging-demo.yml
id: messaging-patterns
namespace: demo.plugins

tasks:
  # Kafka Producer
  - id: produce_to_kafka
    type: io.kestra.plugin.kafka.Produce
    topic: user-events
    key: "{{ inputs.user_id }}"
    value: |
      {
        "event_type": "{{ inputs.event_type }}",
        "user_id": "{{ inputs.user_id }}",
        "timestamp": "{{ now() }}",
        "properties": {{ inputs.properties | tojson }}
      }
    properties:
      bootstrap.servers: "kafka-broker:9092"
      acks: "all"
      compression.type: "snappy"
    serializer: STRING

  # Kafka Consumer (Stream Processing)
  - id: consume_from_kafka
    type: io.kestra.plugin.kafka.Consume
    topic: user-events
    groupId: "etl-processor-{{ execution.startDate | date('yyyy-MM-dd') }}"
    maxRecords: 1000
    timeout: PT1M
    properties:
      bootstrap.servers: "kafka-broker:9092"
      auto.offset.reset: "earliest"

  # RabbitMQ
  - id: rabbitmq_example
    type: io.kestra.plugin.amqp.Write
    uri: "amqp://user:pass@rabbitmq:5672"
    exchange: events
    routingKey: "user.{{ inputs.user_type }}"
    content: |
      {{ inputs.event_data | tojson }}
    contentType: "application/json"

  # AWS SQS
  - id: sqs_example
    type: io.kestra.plugin.aws.sqs.Send
    queueUrl: "https://sqs.us-east-1.amazonaws.com/123456789012/my-queue"
    messageBody: |
      {{ inputs.message | tojson }}
    region: us-east-1

  # Real-time Data Pipeline
  - id: real_time_pipeline
    type: io.kestra.plugin.core.flow.Sequential
    tasks:
      - id: consume_stream
        type: io.kestra.plugin.kafka.Consume
        topic: clickstream
        groupId: real-time-processor
        maxRecords: 100
        timeout: PT10S

      - id: process_in_realtime
        type: io.kestra.plugin.scripts.python.Script
        inputFiles:
          messages.json: "{{ outputs.consume_stream.uri }}"
        script: |
          import json
          import pandas as pd
          from datetime import datetime

          with open('messages.json', 'r') as f:
              messages = json.load(f)

          # Real-time processing
          events = []
          for msg in messages:
              event = json.loads(msg['value'])

              # Enrich in real-time
              event['processed_at'] = datetime.now().isoformat()
              event['hour_of_day'] = datetime.fromisoformat(event['timestamp']).hour
              event['is_peak'] = 9 <= event['hour_of_day'] <= 17

              events.append(event)

          # Batch write for efficiency
          if events:
              df = pd.DataFrame(events)
              df.to_parquet(f'processed_{datetime.now().strftime("%H%M%S")}.parquet')
              print(f"Processed {len(events)} events in real-time")

      - id: produce_results
        type: io.kestra.plugin.kafka.Produce
        topic: processed-events
        key: "{{ execution.startDate | date('yyyy-MM-dd HH') }}"
        value: "{{ outputs.process_in_realtime.outputFiles | tojson }}"
        conditions:
          - type: io.kestra.plugin.core.condition.Expression
            expression: "{{ outputs.process_in_realtime.state.current == 'SUCCESS' }}"

Scripting and Data Processing Plugins

# flows/scripting-demo.yml
id: multi-language-scripts
namespace: demo.plugins

tasks:
  # Python with dependencies
  - id: python_with_deps
    type: io.kestra.plugin.scripts.python.Script
    description: "Python script with custom dependencies"
    script: |
      import pandas as pd
      import numpy as np
      from scipy import stats
      import requests

      # Your data science code here
      data = pd.DataFrame({
          'x': np.random.normal(0, 1, 1000),
          'y': np.random.normal(0, 1, 1000)
      })

      correlation = data['x'].corr(data['y'])
      print(f"Correlation: {correlation:.3f}")

      # Save results
      data.to_parquet('analysis_result.parquet', index=False)
    docker:
      image: "kestra/kestra-python:latest"
      volumes:
        - /tmp:/tmp
    inputFiles:
      config.yaml: |
        dataset:
          name: "sample_data"
          version: "1.0"
    outputFiles:
      - "*.parquet"
      - "*.csv"

  # R for statistical analysis
  - id: r_analysis
    type: io.kestra.plugin.scripts.r.Script
    script: |
      library(dplyr)
      library(ggplot2)
      library(jsonlite)

      # Load data
      data <- read.csv("input_data.csv")

      # Statistical analysis
      result <- data %>%
        group_by(category) %>%
        summarise(
          mean = mean(value),
          sd = sd(value),
          n = n()
        )

      # Create visualization
      plot <- ggplot(data, aes(x=category, y=value)) +
        geom_boxplot() +
        theme_minimal()

      ggsave("boxplot.png", plot, width=8, height=6)

      # Output results
      write_json(result, "analysis_result.json")
      cat("Analysis complete\n")
    docker:
      image: rocker/tidyverse:latest

  # Node.js for API interactions
  - id: nodejs_api_client
    type: io.kestra.plugin.scripts.node.Script
    script: |
      const axios = require('axios');
      const fs = require('fs').promises;

      async function fetchData() {
          const response = await axios.get('https://api.example.com/data');
          const data = response.data;

          // Transform data
          const transformed = data.map(item => ({
              id: item.id,
              name: item.name.toUpperCase(),
              timestamp: new Date().toISOString()
          }));

          // Save to file
          await fs.writeFile(
              'api_data.json',
              JSON.stringify(transformed, null, 2)
          );

          console.log(`Fetched ${transformed.length} records`);
      }

      fetchData().catch(console.error);
    docker:
      image: node:18-alpine

  # Bash for system operations
  - id: bash_operations
    type: io.kestra.plugin.scripts.shell.Commands
    commands:
      - find /data -name "*.parquet" -type f -mtime -7 | wc -l
      - du -sh /data/* | sort -hr
      - echo "Processing started at $(date)"
      - tar -czf archive_$(date +%Y%m%d_%H%M%S).tar.gz /data/input
    exitOnFailed: false

  # SQL-based transformation
  - id: sql_transformation
    type: io.kestra.plugin.jdbc.duckdb.Query
    sql: |
      -- Load data from multiple formats
      CREATE TABLE users AS 
      SELECT * FROM read_parquet('users.parquet');

      CREATE TABLE orders AS
      SELECT * FROM read_csv('orders.csv');

      -- Complex transformation
      WITH user_stats AS (
        SELECT 
          u.user_id,
          u.join_date,
          COUNT(o.order_id) as order_count,
          SUM(o.amount) as total_spent,
          AVG(o.amount) as avg_order_value
        FROM users u
        LEFT JOIN orders o ON u.user_id = o.user_id
        GROUP BY u.user_id, u.join_date
      )
      SELECT * FROM user_stats
      ORDER BY total_spent DESC;
    fetch: true
    store: true

Machine Learning Plugins

# flows/ml-pipeline.yml
id: ml-training-pipeline
namespace: demo.ml

tasks:
  # Data Preparation
  - id: prepare_training_data
    type: io.kestra.plugin.scripts.python.Script
    script: |
      import pandas as pd
      from sklearn.model_selection import train_test_split
      import joblib

      # Load and prepare data
      df = pd.read_parquet('raw_data.parquet')

      # Feature engineering
      df['feature1'] = df['value1'] * df['value2']
      df['feature2'] = df['value3'].apply(lambda x: 1 if x > 0 else 0)

      # Split data
      X = df[['feature1', 'feature2', 'value4']]
      y = df['target']

      X_train, X_test, y_train, y_test = train_test_split(
          X, y, test_size=0.2, random_state=42
      )

      # Save prepared data
      joblib.dump({
          'X_train': X_train,
          'X_test': X_test,
          'y_train': y_train,
          'y_test': y_test,
          'feature_names': X.columns.tolist()
      }, 'prepared_data.pkl')

      print(f"Prepared data: {len(X_train)} training, {len(X_test)} test samples")

  # Model Training with TensorFlow
  - id: train_tensorflow_model
    type: io.kestra.plugin.scripts.python.Script
    inputFiles:
      data.pkl: "{{ outputs.prepare_training_data.outputFiles['prepared_data.pkl'] }}"
    script: |
      import tensorflow as tf
      from tensorflow import keras
      import joblib
      import numpy as np

      # Load prepared data
      data = joblib.load('data.pkl')
      X_train, X_test, y_train, y_test = (
          data['X_train'], data['X_test'], 
          data['y_train'], data['y_test']
      )

      # Build model
      model = keras.Sequential([
          keras.layers.Dense(64, activation='relu', input_shape=(X_train.shape[1],)),
          keras.layers.Dropout(0.2),
          keras.layers.Dense(32, activation='relu'),
          keras.layers.Dense(1, activation='sigmoid')
      ])

      model.compile(
          optimizer='adam',
          loss='binary_crossentropy',
          metrics=['accuracy']
      )

      # Train model
      history = model.fit(
          X_train, y_train,
          validation_split=0.2,
          epochs=10,
          batch_size=32,
          verbose=1
      )

      # Evaluate
      test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
      print(f"Test accuracy: {test_acc:.4f}")

      # Save model
      model.save('trained_model.h5')

      # Save metrics
      metrics = {
          'test_accuracy': float(test_acc),
          'test_loss': float(test_loss),
          'training_history': history.history
      }

      import json
      with open('model_metrics.json', 'w') as f:
          json.dump(metrics, f)

    docker:
      image: tensorflow/tensorflow:2.13.0
      gpu: true  # Enable GPU if available

  # Model Serving with Triton
  - id: deploy_model_triton
    type: io.kestra.plugin.triton.DeployModel
    modelRepository: "/models"
    modelName: "customer-churn"
    modelVersion: "1"
    modelFile: "{{ outputs.train_tensorflow_model.outputFiles['trained_model.h5'] }}"
    config: |
      platform: "tensorflow_savedmodel"
      max_batch_size: 64
      input [
        {
          name: "input"
          data_type: TYPE_FP32
          dims: [ -1, 3 ]
        }
      ]
      output [
        {
          name: "output"
          data_type: TYPE_FP32
          dims: [ -1, 1 ]
        }
      ]

  # ML Monitoring with Evidently
  - id: monitor_model_drift
    type: io.kestra.plugin.scripts.python.Script
    script: |
      import pandas as pd
      import numpy as np
      from evidently.report import Report
      from evidently.metric_preset import DataDriftPreset
      import joblib

      # Load reference data (training data)
      ref_data = pd.read_parquet('reference_data.parquet')

      # Load current data (production data)
      curr_data = pd.read_parquet('current_data.parquet')

      # Generate drift report
      drift_report = Report(metrics=[DataDriftPreset()])
      drift_report.run(
          reference_data=ref_data,
          current_data=curr_data
      )

      # Save report
      drift_report.save_html('data_drift_report.html')

      # Extract metrics
      drift_metrics = drift_report.as_dict()
      drift_score = drift_metrics['metrics'][0]['result']['dataset_drift']

      print(f"Data drift detected: {drift_score}")

      # Alert if drift exceeds threshold
      if drift_score:
          print("ALERT: Significant data drift detected!")

Part 3: Creating Custom Plugins

Plugin Development Environment Setup

# Create plugin project
mvn archetype:generate \
  -DarchetypeGroupId=io.kestra \
  -DarchetypeArtifactId=kestra-plugin-archetype \
  -DarchetypeVersion=0.15.0 \
  -DgroupId=com.company.kestra \
  -DartifactId=kestra-plugin-custom \
  -Dversion=1.0.0 \
  -Dpackage=com.company.kestra.plugin.custom

# Build plugin
cd kestra-plugin-custom
mvn clean package

# Install locally
mvn install

# Test plugin
mvn test

# Deploy to Kestra
cp target/kestra-plugin-custom-1.0.0.jar /path/to/kestra/plugins/

Simple Custom Plugin Example

// src/main/java/com/company/kestra/plugin/custom/HelloWorld.java
package com.company.kestra.plugin.custom;

import io.kestra.core.models.annotations.Example;
import io.kestra.core.models.annotations.Plugin;
import io.kestra.core.models.annotations.PluginProperty;
import io.kestra.core.models.tasks.RunnableTask;
import io.kestra.core.models.tasks.Task;
import io.kestra.core.runners.RunContext;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.*;
import lombok.experimental.SuperBuilder;

import javax.validation.constraints.NotNull;

@SuperBuilder
@ToString
@EqualsAndHashCode
@Getter
@NoArgsConstructor
@Plugin(
    examples = {
        @Example(
            title = "Say hello to someone",
            code = {
                "id: hello",
                "type: com.company.kestra.plugin.custom.HelloWorld",
                "name: John"
            }
        )
    }
)
@Schema(
    title = "Hello World Task",
    description = "A simple custom plugin that greets someone"
)
public class HelloWorld extends Task implements RunnableTask<HelloWorld.Output> {

    @PluginProperty
    @Schema(title = "Name to greet")
    @NotNull
    private String name;

    @PluginProperty
    @Schema(title = "Greeting message")
    private String message;

    @Override
    public Output run(RunContext runContext) throws Exception {
        String greeting = message != null ? message : "Hello";
        String fullMessage = greeting + ", " + name + "!";

        // Log the greeting
        runContext.logger().info(fullMessage);

        // You can also render variables
        String renderedName = runContext.render(name);

        return Output.builder()
            .greeting(fullMessage)
            .renderedName(renderedName)
            .executionTime(System.currentTimeMillis())
            .build();
    }

    @Builder
    @Getter
    public static class Output implements io.kestra.core.models.tasks.Output {
        @Schema(title = "The greeting message")
        private final String greeting;

        @Schema(title = "Rendered name")
        private final String renderedName;

        @Schema(title = "Execution time in milliseconds")
        private final Long executionTime;
    }
}

Database Query Plugin Example

// src/main/java/com/company/kestra/plugin/custom/CustomDatabaseQuery.java
package com.company.kestra.plugin.custom;

import io.kestra.core.models.annotations.Example;
import io.kestra.core.models.annotations.Plugin;
import io.kestra.core.models.annotations.PluginProperty;
import io.kestra.core.models.tasks.RunnableTask;
import io.kestra.core.models.tasks.Task;
import io.kestra.core.runners.RunContext;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.*;
import lombok.experimental.SuperBuilder;

import javax.validation.constraints.NotNull;
import java.sql.*;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

@SuperBuilder
@ToString
@EqualsAndHashCode
@Getter
@NoArgsConstructor
@Plugin(
    examples = {
        @Example(
            title = "Query custom database",
            code = {
                "id: custom_query",
                "type: com.company.kestra.plugin.custom.CustomDatabaseQuery",
                "connectionString: jdbc:custom://host:port/db",
                "username: admin",
                "password: \"{{ secret('CUSTOM_DB_PASS') }}\"",
                "sql: SELECT * FROM users WHERE active = true"
            }
        )
    }
)
@Schema(title = "Custom Database Query", description = "Query a custom database system")
public class CustomDatabaseQuery extends Task implements RunnableTask<CustomDatabaseQuery.Output> {

    @NotNull
    @PluginProperty(dynamic = true)
    @Schema(title = "JDBC connection string")
    private String connectionString;

    @PluginProperty(dynamic = true)
    @Schema(title = "Database username")
    private String username;

    @PluginProperty(dynamic = true)
    @Schema(title = "Database password", secret = true)
    private String password;

    @NotNull
    @PluginProperty(dynamic = true)
    @Schema(title = "SQL query to execute")
    private String sql;

    @PluginProperty
    @Schema(title = "Fetch results (true) or execute without results (false)")
    @Builder.Default
    private Boolean fetch = true;

    @PluginProperty
    @Schema(title = "Fetch size for large results")
    @Builder.Default
    private Integer fetchSize = 1000;

    @PluginProperty
    @Schema(title = "Query timeout in seconds")
    @Builder.Default
    private Integer queryTimeout = 30;

    @Override
    public Output run(RunContext runContext) throws Exception {
        // Render dynamic properties
        String renderedConnectionString = runContext.render(connectionString);
        String renderedUsername = runContext.render(username);
        String renderedPassword = runContext.render(password);
        String renderedSql = runContext.render(sql);

        Connection connection = null;
        PreparedStatement statement = null;
        ResultSet resultSet = null;

        try {
            // Load custom JDBC driver
            Class.forName("com.custom.Driver");

            // Create connection
            connection = DriverManager.getConnection(
                renderedConnectionString,
                renderedUsername,
                renderedPassword
            );

            // Create statement
            statement = connection.prepareStatement(renderedSql);
            statement.setFetchSize(fetchSize);
            statement.setQueryTimeout(queryTimeout);

            if (fetch) {
                // Execute query and fetch results
                resultSet = statement.executeQuery();

                List<Map<String, Object>> rows = new ArrayList<>();
                ResultSetMetaData metaData = resultSet.getMetaData();
                int columnCount = metaData.getColumnCount();

                while (resultSet.next()) {
                    Map<String, Object> row = new HashMap<>();
                    for (int i = 1; i <= columnCount; i++) {
                        String columnName = metaData.getColumnName(i);
                        Object value = resultSet.getObject(i);
                        row.put(columnName, value);
                    }
                    rows.add(row);
                }

                return Output.builder()
                    .rowCount(rows.size())
                    .rows(rows)
                    .query(renderedSql)
                    .build();

            } else {
                // Execute update
                int affectedRows = statement.executeUpdate();

                return Output.builder()
                    .rowCount(affectedRows)
                    .query(renderedSql)
                    .build();
            }

        } finally {
            // Clean up resources
            if (resultSet != null) {
                try { resultSet.close(); } catch (SQLException ignored) {}
            }
            if (statement != null) {
                try { statement.close(); } catch (SQLException ignored) {}
            }
            if (connection != null) {
                try { connection.close(); } catch (SQLException ignored) {}
            }
        }
    }

    @Builder
    @Getter
    public static class Output implements io.kestra.core.models.tasks.Output {
        @Schema(title = "Number of rows affected or returned")
        private final Integer rowCount;

        @Schema(title = "Query results (only for SELECT queries)")
        private final List<Map<String, Object>> rows;

        @Schema(title = "Executed SQL query")
        private final String query;
    }
}

REST API Plugin Example

// src/main/java/com/company/kestra/plugin/custom/CustomApiClient.java
package com.company.kestra.plugin.custom;

import com.fasterxml.jackson.databind.ObjectMapper;
import io.kestra.core.models.annotations.Example;
import io.kestra.core.models.annotations.Plugin;
import io.kestra.core.models.annotations.PluginProperty;
import io.kestra.core.models.tasks.RunnableTask;
import io.kestra.core.models.tasks.Task;
import io.kestra.core.runners.RunContext;
import io.kestra.core.serializers.JacksonMapper;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.*;
import lombok.experimental.SuperBuilder;
import okhttp3.*;

import javax.validation.constraints.NotNull;
import java.time.Duration;
import java.util.HashMap;
import java.util.Map;

@SuperBuilder
@ToString
@EqualsAndHashCode
@Getter
@NoArgsConstructor
@Plugin(
    examples = {
        @Example(
            title = "Call custom API",
            code = {
                "id: call_custom_api",
                "type: com.company.kestra.plugin.custom.CustomApiClient",
                "baseUrl: https://api.company.com/v1",
                "endpoint: /users",
                "method: GET",
                "headers:",
                "  Authorization: Bearer {{ secret('API_TOKEN') }}",
                "queryParams:",
                "  status: active",
                "  limit: 100"
            }
        )
    }
)
@Schema(title = "Custom API Client", description = "Interact with a custom REST API")
public class CustomApiClient extends Task implements RunnableTask<CustomApiClient.Output> {

    private static final ObjectMapper OBJECT_MAPPER = JacksonMapper.ofJson();
    private static final OkHttpClient HTTP_CLIENT = new OkHttpClient.Builder()
        .connectTimeout(Duration.ofSeconds(30))
        .readTimeout(Duration.ofSeconds(30))
        .writeTimeout(Duration.ofSeconds(30))
        .build();

    @NotNull
    @PluginProperty(dynamic = true)
    @Schema(title = "API base URL")
    private String baseUrl;

    @NotNull
    @PluginProperty(dynamic = true)
    @Schema(title = "API endpoint")
    private String endpoint;

    @PluginProperty
    @Schema(title = "HTTP method", defaultValue = "GET")
    @Builder.Default
    private Method method = Method.GET;

    @PluginProperty(dynamic = true)
    @Schema(title = "Request headers")
    private Map<String, String> headers;

    @PluginProperty(dynamic = true)
    @Schema(title = "Query parameters")
    private Map<String, String> queryParams;

    @PluginProperty(dynamic = true)
    @Schema(title = "Request body")
    private Object body;

    @PluginProperty
    @Schema(title = "Timeout in seconds")
    @Builder.Default
    private Integer timeout = 30;

    @PluginProperty
    @Schema(title = "Store response as file")
    @Builder.Default
    private Boolean store = false;

    public enum Method {
        GET, POST, PUT, DELETE, PATCH
    }

    @Override
    public Output run(RunContext runContext) throws Exception {
        // Render dynamic properties
        String renderedBaseUrl = runContext.render(baseUrl);
        String renderedEndpoint = runContext.render(endpoint);
        Map<String, String> renderedHeaders = renderMap(runContext, headers);
        Map<String, String> renderedQueryParams = renderMap(runContext, queryParams);
        Object renderedBody = renderBody(runContext, body);

        // Build URL with query parameters
        HttpUrl.Builder urlBuilder = HttpUrl.parse(renderedBaseUrl + renderedEndpoint).newBuilder();
        if (renderedQueryParams != null) {
            renderedQueryParams.forEach(urlBuilder::addQueryParameter);
        }

        // Build request
        Request.Builder requestBuilder = new Request.Builder()
            .url(urlBuilder.build());

        // Add headers
        if (renderedHeaders != null) {
            renderedHeaders.forEach(requestBuilder::addHeader);
        }

        // Add body for methods that support it
        if (renderedBody != null && 
            (method == Method.POST || method == Method.PUT || 
             method == Method.PATCH || method == Method.DELETE)) {

            String bodyString;
            if (renderedBody instanceof String) {
                bodyString = (String) renderedBody;
                requestBuilder.addHeader("Content-Type", "text/plain");
            } else {
                bodyString = OBJECT_MAPPER.writeValueAsString(renderedBody);
                requestBuilder.addHeader("Content-Type", "application/json");
            }

            RequestBody requestBody = RequestBody.create(
                bodyString, 
                MediaType.parse(requestBuilder.build().header("Content-Type"))
            );

            requestBuilder.method(method.name(), requestBody);
        } else {
            requestBuilder.method(method.name(), null);
        }

        // Execute request with custom timeout
        OkHttpClient client = HTTP_CLIENT.newBuilder()
            .connectTimeout(Duration.ofSeconds(timeout))
            .readTimeout(Duration.ofSeconds(timeout))
            .writeTimeout(Duration.ofSeconds(timeout))
            .build();

        try (Response response = client.newCall(requestBuilder.build()).execute()) {
            String responseBody = response.body() != null ? 
                response.body().string() : null;

            Map<String, Object> responseMap = new HashMap<>();
            if (responseBody != null && !responseBody.isEmpty()) {
                try {
                    responseMap = OBJECT_MAPPER.readValue(responseBody, Map.class);
                } catch (Exception e) {
                    responseMap.put("raw", responseBody);
                }
            }

            return Output.builder()
                .statusCode(response.code())
                .headers(response.headers().toMultimap())
                .body(responseMap)
                .success(response.isSuccessful())
                .message(response.message())
                .build();
        }
    }

    private Map<String, String> renderMap(RunContext runContext, Map<String, String> map) {
        if (map == null) return null;

        Map<String, String> rendered = new HashMap<>();
        map.forEach((key, value) -> {
            rendered.put(key, runContext.render(value));
        });
        return rendered;
    }

    private Object renderBody(RunContext runContext, Object body) {
        if (body == null) return null;

        if (body instanceof String) {
            return runContext.render((String) body);
        } else if (body instanceof Map) {
            // Deep render map values
            Map<?, ?> map = (Map<?, ?>) body;
            Map<Object, Object> rendered = new HashMap<>();
            map.forEach((key, value) -> {
                Object renderedKey = key instanceof String ? 
                    runContext.render((String) key) : key;
                Object renderedValue = value instanceof String ? 
                    runContext.render((String) value) : value;
                rendered.put(renderedKey, renderedValue);
            });
            return rendered;
        }
        return body;
    }

    @Builder
    @Getter
    public static class Output implements io.kestra.core.models.tasks.Output {
        @Schema(title = "HTTP status code")
        private final Integer statusCode;

        @Schema(title = "Response headers")
        private final Map<String, java.util.List<String>> headers;

        @Schema(title = "Response body")
        private final Map<String, Object> body;

        @Schema(title = "Whether the request was successful")
        private final Boolean success;

        @Schema(title = "Response message")
        private final String message;
    }
}

Plugin Configuration (pom.xml)

<!-- pom.xml for custom plugin -->
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
         http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>com.company.kestra</groupId>
    <artifactId>kestra-plugin-custom</artifactId>
    <version>1.0.0</version>
    <packaging>jar</packaging>

    <name>Kestra Custom Plugin</name>
    <description>Custom plugins for Company's systems</description>

    <properties>
        <kestra.version>0.15.0</kestra.version>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <!-- Kestra Core -->
        <dependency>
            <groupId>io.kestra</groupId>
            <artifactId>core</artifactId>
            <version>${kestra.version}</version>
            <scope>provided</scope>
        </dependency>

        <!-- External Dependencies -->
        <dependency>
            <groupId>com.squareup.okhttp3</groupId>
            <artifactId>okhttp</artifactId>
            <version>4.11.0</version>
        </dependency>

        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.15.2</version>
        </dependency>

        <!-- Test Dependencies -->
        <dependency>
            <groupId>io.kestra</groupId>
            <artifactId>junit5</artifactId>
            <version>${kestra.version}</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.11.0</version>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.4.1</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <createDependencyReducedPom>false</createDependencyReducedPom>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

Testing Custom Plugins

// src/test/java/com/company/kestra/plugin/custom/CustomApiClientTest.java
package com.company.kestra.plugin.custom;

import io.kestra.core.runners.RunContextFactory;
import io.kestra.core.utils.TestsUtils;
import io.micronaut.test.extensions.junit5.annotation.MicronautTest;
import jakarta.inject.Inject;
import org.junit.jupiter.api.Test;

import java.util.HashMap;
import java.util.Map;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.*;

@MicronautTest
class CustomApiClientTest {

    @Inject
    private RunContextFactory runContextFactory;

    @Test
    void testApiCall() throws Exception {
        // Create task instance
        CustomApiClient task = CustomApiClient.builder()
            .id("test-api")
            .type(CustomApiClient.class.getName())
            .baseUrl("https://httpbin.org")
            .endpoint("/get")
            .method(CustomApiClient.Method.GET)
            .headers(Map.of(
                "User-Agent", "Kestra-Plugin-Test"
            ))
            .queryParams(Map.of(
                "test", "value"
            ))
            .build();

        // Execute task
        CustomApiClient.Output output = task.run(
            TestsUtils.mockRunContext(
                runContextFactory, 
                task, 
                new HashMap<>()
            )
        );

        // Verify results
        assertThat(output.getStatusCode(), is(200));
        assertThat(output.getSuccess(), is(true));
        assertThat(output.getBody(), notNullValue());

        // Verify response contains our query param
        Map<String, Object> args = (Map<String, Object>) 
            ((Map<String, Object>) output.getBody().get("args"));
        assertThat(args.get("test"), is("value"));
    }

    @Test
    void testPostRequest() throws Exception {
        Map<String, Object> requestBody = Map.of(
            "name", "John Doe",
            "email", "john@example.com",
            "active", true
        );

        CustomApiClient task = CustomApiClient.builder()
            .id("test-post")
            .type(CustomApiClient.class.getName())
            .baseUrl("https://httpbin.org")
            .endpoint("/post")
            .method(CustomApiClient.Method.POST)
            .body(requestBody)
            .build();

        CustomApiClient.Output output = task.run(
            TestsUtils.mockRunContext(
                runContextFactory, 
                task, 
                new HashMap<>()
            )
        );

        assertThat(output.getStatusCode(), is(200));

        Map<String, Object> json = (Map<String, Object>) 
            output.getBody().get("json");
        assertThat(json.get("name"), is("John Doe"));
        assertThat(json.get("active"), is(true));
    }
}

Plugin Documentation

# Custom Database Plugin

## Overview
This plugin allows querying custom database systems from Kestra workflows.

## Installation
1. Build the plugin:
   ```bash
   mvn clean package

2. Copy the JAR to Kestra's plugins directory:

   bash

    cp target/kestra-plugin-custom-1.0.0.jar /opt/kestra/plugins/
   

3. Restart Kestra.

Usage

Basic Query

id: query_custom_db
type: com.company.kestra.plugin.custom.CustomDatabaseQuery
connectionString: jdbc:custom://host:5432/database
username: admin
password: "{{ secret('DB_PASSWORD') }}"
sql: |
  SELECT * FROM users 
  WHERE created_at > '2024-01-01'
fetch: true

Update Query

id: update_records
type: com.company.kestra.plugin.custom.CustomDatabaseQuery
connectionString: jdbc:custom://host:5432/database
sql: |
  UPDATE users 
  SET status = 'inactive'
  WHERE last_login < '2024-01-01'
fetch: false

Properties

Outputs

For SELECT queries (fetch: true):

{
  "rowCount": 100,
  "rows": [...],
  "query": "SELECT * FROM users"
}

For UPDATE/INSERT/DELETE queries (fetch: false):

{
  "rowCount": 50,
  "query": "UPDATE users SET status = 'active'"
}

Examples

See the examples directory for more usage scenarios.

---

## **Part 4: Plugin Management and Optimization**

### **Plugin Registry Configuration**

```yaml
# kestra.yml - Plugin configuration
kestra:
  plugins:
    # Plugin repositories
    repositories:
      - id: kestra-io
        url: https://repo.kestra.io/repository/maven-public/
        type: maven
        releases: true
        snapshots: true

      - id: company-private
        url: https://nexus.company.com/repository/maven-releases/
        type: maven
        releases: true
        snapshots: false
        authentication:
          username: "{{ secret('NEXUS_USER') }}"
          password: "{{ secret('NEXUS_PASS') }}"

    # Plugin locations
    locations:
      - /opt/kestra/plugins  # System plugins
      - /app/plugins         # User plugins

    # Plugin scanning
    scan:
      enabled: true
      interval: PT5M  # Scan every 5 minutes

    # Plugin isolation
    isolation:
      enabled: true
      mode: PROCESS  # PROCESS or THREAD
      process:
        maxThreads: 10
        maxMemory: 1Gi
        timeout: PT5M

    # Cache configuration
    cache:
      enabled: true
      size: 1000
      expireAfterWrite: PT1H

Plugin Version Management

# flows/plugin-management.yml
id: plugin-version-manager
namespace: system.plugins

tasks:
  - id: check_plugin_updates
    type: io.kestra.plugin.core.http.Request
    uri: "https://repo.kestra.io/repository/maven-public/io/kestra/plugin/maven-metadata.xml"
    store: true

  - id: parse_plugin_versions
    type: io.kestra.plugin.scripts.python.Script
    inputFiles:
      metadata.xml: "{{ outputs.check_plugin_updates.uri }}"
    script: |
      import xml.etree.ElementTree as ET
      import json

      tree = ET.parse('metadata.xml')
      root = tree.getroot()

      versions = []
      for version in root.find('.//versioning/versions'):
          versions.append(version.text)

      latest = root.find('.//versioning/latest').text
      release = root.find('.//versioning/release').text

      result = {
          'total_versions': len(versions),
          'latest_version': latest,
          'release_version': release,
          'all_versions': versions[-10:]  # Last 10 versions
      }

      print(json.dumps(result, indent=2))

  - id: update_plugins_if_needed
    type: io.kestra.plugin.core.flow.EachSequential
    value: ["postgresql", "s3", "snowflake", "bigquery"]
    tasks:
      - id: "check_{{ task.value }}_version"
        type: io.kestra.plugin.core.debug.Return
        format: "{{ task.value }}: 1.2.3"  # Would be actual version check

      - id: "update_{{ task.value }}_if_outdated"
        type: io.kestra.plugin.core.flow.Subflow
        namespace: system.maintenance
        flowId: update-plugin
        wait: true
        transmit:
          - name: plugin_name
            value: "kestra-plugin-jdbc-{{ task.value }}"
          - name: target_version
            value: "latest"
        conditions:
          - type: io.kestra.plugin.core.condition.Expression
            expression: "{{ true }}"  # Actual version comparison

Plugin Performance Optimization

# flows/plugin-performance.yml
id: plugin-performance-optimization
namespace: demo.performance

tasks:
  # Batch operations instead of individual calls
  - id: batch_database_operations
    type: io.kestra.plugin.jdbc.postgresql.Query
    sql: |
      -- Batch insert
      INSERT INTO users (id, name, email) VALUES
      {% for user in inputs.users %}
      ('{{ user.id }}', '{{ user.name }}', '{{ user.email }}'){{ ',' if not loop.last }}
      {% endfor %}

      ON CONFLICT (id) DO UPDATE SET
        name = EXCLUDED.name,
        email = EXCLUDED.email;
    fetch: false

  # Connection pooling
  - id: connection_pool_config
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: jdbc:postgresql://localhost:5432/db
    username: user
    password: pass
    sql: SELECT * FROM large_table
    connectionProperties:
      maximumPoolSize: "10"
      minimumIdle: "5"
      connectionTimeout: "30000"
      idleTimeout: "600000"
      maxLifetime: "1800000"

  # Parallel plugin execution
  - id: parallel_plugin_execution
    type: io.kestra.plugin.core.flow.Parallel
    tasks:
      - id: api_call_1
        type: io.kestra.plugin.core.http.Request
        uri: "https://api.example.com/data1"

      - id: api_call_2
        type: io.kestra.plugin.core.http.Request
        uri: "https://api.example.com/data2"

      - id: api_call_3
        type: io.kestra.plugin.core.http.Request
        uri: "https://api.example.com/data3"

  # Plugin caching
  - id: cached_operation
    type: io.kestra.plugin.core.cache.Cached
    key: "expensive_operation:{{ inputs.user_id }}"
    ttl: PT1H
    tasks:
      - id: expensive_computation
        type: io.kestra.plugin.scripts.python.Script
        script: |
          import time
          import json

          # Simulate expensive computation
          time.sleep(5)

          result = {"computed": True, "value": 42}
          print(json.dumps(result))

  # Memory optimization for large data
  - id: process_large_data
    type: io.kestra.plugin.jdbc.postgresql.Query
    sql: SELECT * FROM very_large_table
    fetch: true
    fetchSize: 10000  # Process in chunks
    chunkSize: 1000   # Smaller chunks for memory efficiency

Plugin Monitoring and Metrics

# flows/plugin-monitoring.yml
id: plugin-monitoring-dashboard
namespace: system.monitoring

tasks:
  - id: collect_plugin_metrics
    type: io.kestra.plugin.jdbc.postgresql.Query
    fetch: true
    sql: |
      WITH plugin_stats AS (
        SELECT 
          SUBSTRING(task_type FROM 'plugin\.([^.]*)') as plugin_name,
          COUNT(*) as execution_count,
          AVG(EXTRACT(EPOCH FROM (end_date - start_date))) * 1000 as avg_duration_ms,
          PERCENTILE_CONT(0.95) WITHIN GROUP (
            ORDER BY EXTRACT(EPOCH FROM (end_date - start_date)) * 1000
          ) as p95_duration_ms,
          SUM(CASE WHEN state = 'FAILED' THEN 1 ELSE 0 END) as failure_count
        FROM executions
        WHERE start_date >= NOW() - INTERVAL '7 days'
          AND task_type LIKE 'io.kestra.plugin.%'
        GROUP BY plugin_name
      ),
      plugin_health AS (
        SELECT 
          plugin_name,
          execution_count,
          avg_duration_ms,
          p95_duration_ms,
          failure_count,
          CASE 
            WHEN failure_count = 0 THEN 'HEALTHY'
            WHEN failure_count::float / execution_count < 0.01 THEN 'WARNING'
            ELSE 'CRITICAL'
          END as health_status
        FROM plugin_stats
      )
      SELECT * FROM plugin_health
      ORDER BY execution_count DESC

  - id: generate_plugin_report
    type: io.kestra.plugin.scripts.python.Script
    inputFiles:
      metrics.json: |
        {{ outputs.collect_plugin_metrics.rows | tojson }}
    script: |
      import json
      import pandas as pd
      from datetime import datetime

      data = json.loads('metrics.json')
      df = pd.DataFrame(data)

      # Create summary report
      report = {
          'generated_at': datetime.now().isoformat(),
          'total_plugins': len(df),
          'total_executions': df['execution_count'].sum(),
          'health_summary': {
              'healthy': len(df[df['health_status'] == 'HEALTHY']),
              'warning': len(df[df['health_status'] == 'WARNING']),
              'critical': len(df[df['health_status'] == 'CRITICAL'])
          },
          'top_plugins_by_usage': df.nlargest(10, 'execution_count').to_dict('records'),
          'slowest_plugins': df.nlargest(5, 'p95_duration_ms').to_dict('records'),
          'most_failing_plugins': df.nlargest(5, 'failure_count').to_dict('records')
      }

      print(json.dumps(report, indent=2))

  - id: alert_on_plugin_issues
    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {
        "blocks": [
          {
            "type": "header",
            "text": {
              "type": "plain_text",
              "text": "⚠️ Plugin Health Alert"
            }
          },
          {
            "type": "section",
            "text": {
              "type": "mrkdwn",
              "text": "*Critical Plugins:*\n{% for plugin in outputs.generate_plugin_report.vars.most_failing_plugins %}- {{ plugin.plugin_name }}: {{ plugin.failure_count }} failures\n{% endfor %}"
            }
          }
        ]
      }
    conditions:
      - type: io.kestra.plugin.core.condition.Expression
        expression: "{{ outputs.generate_plugin_report.vars.health_summary.critical > 0 }}"

Part 5: Enterprise Plugin Patterns

Plugin Factory Pattern

// Plugin factory for creating plugin instances dynamically
public class PluginFactory {

    private final Map<String, PluginCreator> creators = new HashMap<>();

    public PluginFactory() {
        // Register plugin creators
        creators.put("salesforce", new SalesforcePluginCreator());
        creators.put("sap", new SapPluginCreator());
        creators.put("oracle", new OraclePluginCreator());
    }

    public Task createPlugin(String system, Map<String, Object> config) {
        PluginCreator creator = creators.get(system.toLowerCase());
        if (creator == null) {
            throw new IllegalArgumentException("Unknown system: " + system);
        }
        return creator.create(config);
    }

    interface PluginCreator {
        Task create(Map<String, Object> config);
    }

    static class SalesforcePluginCreator implements PluginCreator {
        @Override
        public Task create(Map<String, Object> config) {
            return SalesforceQuery.builder()
                .instanceUrl((String) config.get("instanceUrl"))
                .clientId((String) config.get("clientId"))
                .clientSecret((String) config.get("clientSecret"))
                .soql((String) config.get("query"))
                .build();
        }
    }
}

Plugin Adapter Pattern

# flows/plugin-adapter.yml
id: legacy-system-adapter
namespace: enterprise.integration

tasks:
  # Adapter for legacy SOAP service
  - id: soap_to_rest_adapter
    type: io.kestra.plugin.scripts.python.Script
    script: |
      import zeep
      import requests
      import json

      # Legacy SOAP client
      client = zeep.Client('https://legacy.company.com/wsdl')

      # Convert REST request to SOAP
      request_data = {{ inputs.rest_request | tojson }}

      soap_response = client.service.ProcessRequest(
          method=request_data['method'],
          params=json.dumps(request_data['params'])
      )

      # Convert SOAP response to REST format
      rest_response = {
          'status': 'success' if soap_response.success else 'error',
          'data': json.loads(soap_response.data),
          'timestamp': soap_response.timestamp
      }

      print(json.dumps(rest_response))

  # Adapter for mainframe (via FTP)
  - id: mainframe_adapter
    type: io.kestra.plugin.fs.sftp.Download
    host: mainframe.company.com
    port: 22
    username: "{{ secret('MAINFRAME_USER') }}"
    password: "{{ secret('MAINFRAME_PASS') }}"
    from: "/reports/{{ execution.startDate | date('yyyyMMdd') }}.txt"

  - id: parse_mainframe_data
    type: io.kestra.plugin.scripts.python.Script
    inputFiles:
      mainframe_data.txt: "{{ outputs.mainframe_adapter.uri }}"
    script: |
      # Parse fixed-width mainframe format
      import struct
      import pandas as pd

      with open('mainframe_data.txt', 'r') as f:
          lines = f.readlines()

      # Define fixed-width format (example)
      format_spec = [
          ('customer_id', 10),
          ('name', 30),
          ('balance', 12),
          ('status', 1)
      ]

      records = []
      for line in lines:
          record = {}
          pos = 0
          for field, width in format_spec:
              value = line[pos:pos+width].strip()
              record[field] = value
              pos += width
          records.append(record)

      df = pd.DataFrame(records)
      df.to_parquet('mainframe_parsed.parquet', index=False)

Plugin Security Patterns

# flows/secure-plugin-usage.yml
id: secure-plugin-patterns
namespace: enterprise.security

tasks:
  # Secrets management
  - id: secure_database_connection
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: "jdbc:postgresql://{{ secret('DB_HOST') }}:5432/{{ secret('DB_NAME') }}"
    username: "{{ secret('DB_USER') }}"
    password: "{{ secret('DB_PASSWORD') }}"
    sql: SELECT * FROM secure_table
    connectionProperties:
      ssl: "true"
      sslmode: "verify-full"
      sslrootcert: "/certs/ca.pem"

  # Encrypted data handling
  - id: process_encrypted_data
    type: io.kestra.plugin.scripts.python.Script
    script: |
      import gnupg
      import pandas as pd
      from io import StringIO

      # Initialize GPG
      gpg = gnupg.GPG(gnupghome='/app/.gnupg')

      # Decrypt data
      with open('encrypted_data.pgp', 'rb') as f:
          decrypted = gpg.decrypt_file(f, passphrase='{{ secret('GPG_PASSPHRASE') }}')

      if decrypted.ok:
          # Process decrypted data
          df = pd.read_csv(StringIO(str(decrypted)))
          # ... processing ...

          # Re-encrypt results
          encrypted = gpg.encrypt(
              df.to_csv(index=False),
              recipients=['data-team@company.com'],
              always_trust=True
          )

          with open('results.pgp', 'w') as f:
              f.write(str(encrypted))
      else:
          raise Exception(f"Decryption failed: {decrypted.stderr}")
    docker:
      image: python:3.11-slim
      volumes:
        - /secrets/gnupg:/app/.gnupg:ro

  # Audit logging for sensitive operations
  - id: audited_plugin_operation
    type: io.kestra.plugin.core.flow.Sequential
    tasks:
      - id: log_operation_start
        type: io.kestra.plugin.jdbc.postgresql.Query
        sql: |
          INSERT INTO audit_log 
          (operation, user_id, start_time, parameters, execution_id)
          VALUES (
            'pii_data_extract',
            '{{ execution.user }}',
            NOW(),
            '{{ inputs | tojson }}',
            '{{ execution.id }}'
          )

      - id: perform_sensitive_operation
        type: io.kestra.plugin.jdbc.postgresql.Query
        sql: |
          SELECT 
            user_id,
            email,
            phone_number,
            masked_ssn
          FROM pii_data
          WHERE department = '{{ inputs.department }}'
        fetch: true

      - id: log_operation_complete
        type: io.kestra.plugin.jdbc.postgresql.Query
        sql: |
          UPDATE audit_log 
          SET 
            end_time = NOW(),
            row_count = {{ outputs.perform_sensitive_operation.rowCount }},
            status = 'SUCCESS'
          WHERE execution_id = '{{ execution.id }}'

Plugin Testing Framework

# flows/plugin-testing-framework.yml
id: plugin-testing-pipeline
namespace: quality.assurance

inputs:
  - name: plugin_name
    type: STRING
    required: true
  - name: test_scenarios
    type: JSON
    required: true

tasks:
  - id: setup_test_environment
    type: io.kestra.plugin.core.flow.Parallel
    tasks:
      - id: start_test_database
        type: io.kestra.plugin.docker.Run
        image: postgres:15
        env:
          POSTGRES_PASSWORD: test
          POSTGRES_DB: test_db
        ports:
          - 5432:5432
        wait: true
        maxDuration: PT1M

      - id: start_test_api
        type: io.kestra.plugin.docker.Run
        image: mockserver/mockserver
        command: -serverPort 1080
        ports:
          - 1080:1080
        wait: true
        maxDuration: PT1M

  - id: run_plugin_tests
    type: io.kestra.plugin.core.flow.EachSequential
    value: "{{ inputs.test_scenarios }}"
    tasks:
      - id: "execute_test_{{ taskloop.index }}"
        type: io.kestra.plugin.core.flow.Switch
        value: "{{ task.value.type }}"
        cases:
          database:
            - id: run_database_test
              type: io.kestra.plugin.jdbc.postgresql.Query
              url: jdbc:postgresql://localhost:5432/test_db
              username: postgres
              password: test
              sql: "{{ task.value.query }}"
              fetch: true

          api:
            - id: setup_api_mock
              type: io.kestra.plugin.core.http.Request
              uri: http://localhost:1080/mockserver/expectation
              method: PUT
              body: "{{ task.value.mock_config | tojson }}"

            - id: test_api_call
              type: io.kestra.plugin.core.http.Request
              uri: "http://localhost:1080{{ task.value.endpoint }}"
              method: "{{ task.value.method }}"
              body: "{{ task.value.request_body | tojson }}"

          performance:
            - id: run_performance_test
              type: io.kestra.plugin.core.flow.EachSequential
              value: "{{ range(1, task.value.iterations + 1) }}"
              maxParallel: 10
              tasks:
                - id: "iteration_{{ taskloop.value }}"
                  type: io.kestra.plugin.core.debug.Return
                  format: "{{ task.value }}"

  - id: collect_test_results
    type: io.kestra.plugin.scripts.python.Script
    script: |
      import json
      import statistics

      # Collect results from all tests
      test_results = []

      {% for i in range(1, inputs.test_scenarios|length + 1) %}
      test_output = {{ outputs.run_plugin_tests.outputs['execute_test_' + i|string] | default({}) | tojson }}
      if test_output:
          test_results.append({
              'scenario': {{ i }},
              'success': test_output.get('state') == 'SUCCESS',
              'duration': test_output.get('duration', 0)
          })
      {% endfor %}

      # Calculate statistics
      summary = {
          'total_tests': len(test_results),
          'passed_tests': sum(1 for r in test_results if r['success']),
          'failed_tests': sum(1 for r in test_results if not r['success']),
          'pass_rate': sum(1 for r in test_results if r['success']) / len(test_results) * 100 if test_results else 0,
          'avg_duration': statistics.mean(r['duration'] for r in test_results) if test_results else 0,
          'results': test_results
      }

      print(json.dumps(summary, indent=2))

  - id: generate_test_report
    type: io.kestra.plugin.notifications.email.EmailSend
    to: "qa-team@company.com"
    subject: "Plugin Test Results - {{ inputs.plugin_name }}"
    htmlBody: |
      <h1>Plugin Test Report</h1>
      <p><strong>Plugin:</strong> {{ inputs.plugin_name }}</p>
      <p><strong>Pass Rate:</strong> {{ outputs.collect_test_results.vars.pass_rate | round(2) }}%</p>
      <p><strong>Total Tests:</strong> {{ outputs.collect_test_results.vars.total_tests }}</p>
      <p><strong>Execution:</strong> {{ execution.id }}</p>

Part 6: Contributing to the Plugin Ecosystem

Open Source Contribution Guide

# 1. Fork the repository
https://github.com/kestra-io/kestra

# 2. Clone your fork
git clone https://github.com/YOUR_USERNAME/kestra.git
cd kestra

# 3. Create a feature branch
git checkout -b feature/new-plugin

# 4. Create plugin structure
mkdir -p plugin-<name>/src/main/java/io/kestra/plugin/<category>
mkdir -p plugin-<name>/src/test/java/io/kestra/plugin/<category>

# 5. Implement your plugin
# (Follow existing plugin patterns)

# 6. Write tests
# (Aim for >80% test coverage)

# 7. Update documentation
# - README.md in plugin directory
# - Add examples in @Plugin annotation
# - Update plugin list in docs

# 8. Run tests
mvn test -pl plugin-<name>

# 9. Submit pull request
git push origin feature/new-plugin
# Create PR on GitHub

Plugin Contribution Checklist

## New Plugin Contribution Checklist

### ✅ Code Quality
- [ ] Follows Kestra coding conventions
- [ ] Proper error handling and logging
- [ ] Input validation with @NotNull annotations
- [ ] Comprehensive JavaDoc comments
- [ ] No SonarQube issues

### ✅ Testing
- [ ] Unit tests for all public methods
- [ ] Integration tests with real systems
- [ ] Test edge cases and error conditions
- [ ] Mock external dependencies
- [ ] Test coverage > 80%

### ✅ Documentation
- [ ] README.md with usage examples
- [ ] @Plugin annotation with examples
- [ ] Swagger annotations for all properties
- [ ] Clear property descriptions
- [ ] Example flows in /examples directory

### ✅ Configuration
- [ ] Proper Maven pom.xml
- [ ] Plugin registration in parent pom
- [ ] Version aligned with Kestra core
- [ ] No conflicting dependencies

### ✅ Security
- [ ] Secrets marked with secret = true
- [ ] No hardcoded credentials
- [ ] Input sanitization
- [ ] SSL/TLS support where applicable

### ✅ Performance
- [ ] Connection pooling for databases
- [ ] Batch operations for bulk data
- [ ] Memory-efficient streaming
- [ ] Timeout configurations

Community Plugin Gallery

# Example of a well-documented community plugin
id: community-plugin-showcase
namespace: demo.community

tasks:
  # Example: dbt plugin from community
  - id: run_dbt_models
    type: io.kestra.plugin.dbt.cli.Run
    projectDir: "s3://my-dbt-project/"
    profilesDir: "s3://my-dbt-profiles/"
    target: prod
    select:
      - tag:hourly
      - model_name+
    exclude:
      - tag:disabled

  # Example: Great Expectations plugin
  - id: data_quality_check
    type: io.kestra.plugin.greatexpectations.Validation
    expectationSuite: |
      {
        "expectation_suite_name": "customer_data",
        "expectations": [
          {
            "expectation_type": "expect_column_values_to_not_be_null",
            "kwargs": {"column": "customer_id"}
          }
        ]
      }
    data:
      type: file
      path: "{{ inputs.data_file }}"

  # Example: Airbyte connector
  - id: sync_with_airbyte
    type: io.kestra.plugin.airbyte.Sync
    connectionId: "{{ secret('AIRBYTE_CONNECTION_ID') }}"
    workspaceId: "{{ secret('AIRBYTE_WORKSPACE_ID') }}"
    apiUrl: "https://api.airbyte.com"
    wait: true

  # Example: Dagster integration
  - id: trigger_dagster_pipeline
    type: io.kestra.plugin.dagster.Run
    repository: my_repo
    pipeline: process_data
    mode: default
    runConfig:
      resources:
        io_manager:
          config:
            s3_bucket: "dagster-output"

Conclusion: Becoming a Plugin Master

Key Takeaways

1. Kestra's plugin ecosystem is vast: 150+ plugins covering every data engineering need

2. Custom plugins are straightforward: Java-based, with clear interfaces and patterns

3. Enterprise integration is possible: Create plugins for any system, no matter how legacy

4. Performance matters: Optimize plugins for production workloads

5. Community contribution is valuable: Share your plugins to help others

Next Steps for You

1. Audit your data stack: What systems need plugins?

2. Start simple: Create one custom plugin for an internal API

3. Contribute: Find a missing plugin in the ecosystem and build it

4. Share: Write a blog post about your plugin journey

5. Optimize: Review existing plugins for performance improvements

Resources

• Official Documentation: https://kestra.io/docs/plugins/develop

• Plugin Examples: https://github.com/kestra-io/examples

• Community Forum: https://github.com/kestra-io/kestra/discussions

• Plugin Template: https://github.com/kestra-io/plugin-template

In the next article, we'll dive into Monitoring and Observability in Kestra - how to ensure your workflows are running smoothly and troubleshoot when they're not.

Before the next article, try:

1. Create a simple custom plugin for a system you use

2. Optimize an existing plugin configuration for better performance

3. Contribute documentation or examples to an existing plugin

4. Set up a plugin testing pipeline for your team

5. Share your plugin creation experience in the Kestra community

Remember: Every great data platform starts with great plugins. What will you build?

Happy plugin developing! 🛠️

Introduction: The Orchestrator's Toolkit

Imagine you're conducting a symphony. You don't just wave your baton - you cue sections, adjust tempo, handle surprises, and ensure harmony. That's what advanced workflow patterns give you in Kestra: the ability to conduct complex data orchestrations with precision and grace.

In this article, we'll move beyond simple linear pipelines into sophisticated orchestration patterns that handle real-world complexity. You'll learn patterns used by top data teams to build resilient, scalable, and intelligent workflows.

What makes a workflow "advanced"?

• Intelligence: Making decisions based on data

• Resilience: Graceful handling of the unexpected

• Efficiency: Optimal resource usage and parallelism

• Adaptability: Dynamic behavior based on context

• Maintainability: Clean, reusable patterns

Pattern 1: Dynamic Parallel Processing

Beyond Simple Parallel Tasks

While Kestra's Parallel task is great for fixed concurrency, real-world scenarios often require dynamic parallelism based on data volume or external factors.

id: dynamic-parallel-processing
namespace: acme.advanced
description: Dynamic parallelism based on data volume

tasks:
  - id: discover_files
    type: io.kestra.plugin.core.http.Request
    uri: "https://api.acme.com/files"
    store: true

  - id: calculate_optimal_parallelism
    type: io.kestra.plugin.scripts.python.Script
    inputFiles:
      files.json: "{{ outputs.discover_files.uri }}"
    script: |
      import json
      import math

      with open('files.json', 'r') as f:
          files = json.load(f)

      total_files = len(files['files'])
      avg_file_size = sum(f['size'] for f in files['files']) / total_files if total_files > 0 else 0

      # Dynamic parallelism calculation
      # Rule: 1 worker per 100MB, max 10 workers
      max_workers = min(10, max(1, math.ceil((total_files * avg_file_size) / (100 * 1024 * 1024))))

      # Split files into batches
      batch_size = math.ceil(total_files / max_workers)
      batches = []
      for i in range(0, total_files, batch_size):
          batches.append(files['files'][i:i + batch_size])

      print(json.dumps({
          'max_workers': max_workers,
          'batch_size': batch_size,
          'batches': batches,
          'total_files': total_files
      }))

  - id: process_in_dynamic_parallel
    type: io.kestra.plugin.core.flow.EachParallel
    value: "{{ outputs.calculate_optimal_parallelism.vars.batches }}"
    maxParallel: "{{ outputs.calculate_optimal_parallelism.vars.max_workers }}"
    tasks:
      - id: "process_batch_{{ taskloop.index }}"
        type: io.kestra.plugin.scripts.python.Script
        description: "Processing batch {{ taskloop.index }} with {{ task.value | length }} files"
        inputFiles:
          batch_files.json: |
            {{ task.value | tojson }}
        script: |
          import json
          import time

          with open('batch_files.json', 'r') as f:
              files = json.load(f)

          # Simulate processing
          results = []
          for file in files:
              time.sleep(0.1)  # Simulate work
              results.append({
                  'file': file['name'],
                  'processed': True,
                  'size': file['size']
              })

          print(f"Processed {len(results)} files")
          print(json.dumps({'results': results}))

Fan-Out/Fan-In Pattern

This pattern distributes work across many workers (fan-out) and aggregates results (fan-in).

id: fan-out-fan-in-pattern
namespace: acme.advanced
description: Distribute work, process in parallel, aggregate results

tasks:
  # Fan-Out: Distribute work
  - id: generate_work_items
    type: io.kestra.plugin.core.debug.Return
    format: >
      {{ range(1, 101) | map('toString') | list | tojson }}

  # Parallel Processing
  - id: parallel_processing
    type: io.kestra.plugin.core.flow.EachParallel
    value: "{{ outputs.generate_work_items.value }}"
    maxParallel: 20
    tasks:
      - id: "process_item_{{ task.value }}"
        type: io.kestra.plugin.scripts.python.Script
        script: |
          import time
          import random

          # Simulate variable processing time
          processing_time = random.uniform(0.1, 2.0)
          time.sleep(processing_time)

          # Generate result
          result = {
              'item': {{ task.value }},
              'processed_at': time.time(),
              'processing_time': processing_time,
              'result': {{ task.value }} * 2  # Simple transformation
          }

          print(f"Processed item {{ task.value }} in {processing_time:.2f}s")

          # Return result
          import json
          print(json.dumps(result))

  # Fan-In: Aggregate results
  - id: aggregate_results
    type: io.kestra.plugin.scripts.python.Script
    description: "Aggregate all parallel results"
    script: |
      import json
      import statistics

      # Collect all results from parallel tasks
      all_results = []
      processing_times = []

      {% for i in range(1, 101) %}
      try:
          result_json = '''{{ outputs.parallel_processing.outputs["process_item_" + i|string].vars | default('{}') | tojson }}'''
          if result_json:
              result = json.loads(result_json)
              if 'result' in str(result):  # Check if result exists
                  # Extract the actual result from the output
                  # This depends on how your script outputs data
                  all_results.append(result)
                  if 'processing_time' in result:
                      processing_times.append(result['processing_time'])
      except:
          pass
      {% endfor %}

      # Calculate statistics
      stats = {
          'total_processed': len(all_results),
          'average_processing_time': statistics.mean(processing_times) if processing_times else 0,
          'total_processing_time': sum(processing_times),
          'throughput': len(all_results) / (sum(processing_times) if processing_times else 1),
          'results_summary': {
              'min': min(r['result'] for r in all_results) if all_results else 0,
              'max': max(r['result'] for r in all_results) if all_results else 0,
              'avg': statistics.mean(r['result'] for r in all_results) if all_results else 0
          }
      }

      print(json.dumps(stats, indent=2))

    dependsOn:
      - parallel_processing

Pattern 2: Circuit Breaker Pattern

Preventing Cascading Failures

The circuit breaker pattern prevents systems from being overwhelmed by repeated failures, similar to an electrical circuit breaker.

id: circuit-breaker-pattern
namespace: acme.advanced
description: Implement circuit breaker for unreliable external services

variables:
  circuit_breaker_state_key: "circuit_breaker:api.acme.com"
  failure_threshold: 5
  reset_timeout: PT5M

tasks:
  - id: check_circuit_breaker
    type: io.kestra.plugin.core.cache.Get
    key: "{{ vars.circuit_breaker_state_key }}"

  - id: evaluate_circuit_state
    type: io.kestra.plugin.core.flow.Switch
    value: "{{ outputs.check_circuit_breaker.value.state | default('closed') }}"
    cases:
      open:
        - id: circuit_open_check_timeout
          type: io.kestra.plugin.core.condition.Expression
          expression: >
            {{ now().timestamp() - outputs.check_circuit_breaker.value.timestamp > 300 }}

        - id: attempt_reset
          type: io.kestra.plugin.core.log.Log
          message: "Circuit breaker open for {{ vars.reset_timeout }}. Testing if service recovered..."
          conditions:
            - type: io.kestra.plugin.core.condition.Expression
              expression: "{{ outputs.circuit_open_check_timeout.value }}"

        - id: skip_service_call
          type: io.kestra.plugin.core.log.Log
          message: "Circuit breaker is OPEN - skipping service call"
          conditions:
            - type: io.kestra.plugin.core.condition.Expression
              expression: "{{ not outputs.circuit_open_check_timeout.value }}"

  - id: call_external_service
    type: io.kestra.plugin.core.http.Request
    uri: "https://api.acme.com/unstable-endpoint"
    method: GET
    retry:
      type: exponential
      maxAttempt: 2
      delay: PT5S
    timeout: PT30S
    allowFailure: true
    conditions:
      - type: io.kestra.plugin.core.condition.Expression
        expression: >
          {{ outputs.check_circuit_breaker.value.state != 'open' or 
             outputs.circuit_open_check_timeout.value == true }}

  - id: update_circuit_state
    type: io.kestra.plugin.core.flow.Switch
    value: "{{ outputs.call_external_service.state.current }}"
    cases:
      SUCCESS:
        - id: reset_circuit_on_success
          type: io.kestra.plugin.core.cache.Put
          key: "{{ vars.circuit_breaker_state_key }}"
          value: |
            {
              "state": "closed",
              "failure_count": 0,
              "last_success": "{{ now() }}",
              "timestamp": {{ now().timestamp() }}
            }
          ttl: PT1H

      FAILED:
        - id: get_current_failure_count
          type: io.kestra.plugin.core.cache.Get
          key: "{{ vars.circuit_breaker_state_key }}_failures"
          defaultValue: "0"

        - id: increment_failure_count
          type: io.kestra.plugin.core.cache.Put
          key: "{{ vars.circuit_breaker_state_key }}_failures"
          value: "{{ outputs.get_current_failure_count.value | int + 1 }}"
          ttl: PT1H

        - id: check_if_should_trip
          type: io.kestra.plugin.core.condition.Expression
          expression: >
            {{ (outputs.get_current_failure_count.value | int + 1) >= vars.failure_threshold }}

        - id: trip_circuit_breaker
          type: io.kestra.plugin.core.cache.Put
          key: "{{ vars.circuit_breaker_state_key }}"
          value: |
            {
              "state": "open",
              "tripped_at": "{{ now() }}",
              "failure_count": {{ outputs.get_current_failure_count.value | int + 1 }},
              "timestamp": {{ now().timestamp() }}
            }
          ttl: "{{ vars.reset_timeout }}"
          conditions:
            - type: io.kestra.plugin.core.condition.Expression
              expression: "{{ outputs.check_if_should_trip.value }}"

        - id: log_failure
          type: io.kestra.plugin.core.log.Log
          message: >
            Service call failed. Failure count: {{ outputs.get_current_failure_count.value | int + 1 }}/{{ vars.failure_threshold }}
            {% if outputs.check_if_should_trip.value %}
            Circuit breaker TRIPPED to OPEN state
            {% endif %}

  - id: fallback_operation
    type: io.kestra.plugin.core.debug.Return
    format: "Using cached data or alternative service"
    conditions:
      - type: io.kestra.plugin.core.condition.Expression
        expression: >
          {{ outputs.call_external_service.state.current == 'FAILED' or
             (outputs.check_circuit_breaker.value.state == 'open' and 
              not outputs.circuit_open_check_timeout.value) }}

Advanced Circuit Breaker with Monitoring

id: monitored-circuit-breaker
namespace: acme.advanced

tasks:
  - id: circuit_breaker_monitor
    type: io.kestra.plugin.scripts.python.Script
    script: |
      import redis
      import json
      from datetime import datetime

      # Connect to shared state store
      r = redis.Redis(host='redis', port=6379, decode_responses=True)

      # Get all circuit breaker states
      circuit_keys = r.keys("circuit_breaker:*")
      circuits = []

      for key in circuit_keys:
          state = r.get(key)
          if state:
              circuits.append({
                  'service': key.replace('circuit_breaker:', ''),
                  'state': json.loads(state),
                  'age_seconds': datetime.now().timestamp() - json.loads(state).get('timestamp', 0)
              })

      # Generate health report
      report = {
          'timestamp': datetime.now().isoformat(),
          'total_circuits': len(circuits),
          'open_circuits': [c for c in circuits if c['state'].get('state') == 'open'],
          'half_open_circuits': [c for c in circuits if c['state'].get('state') == 'half_open'],
          'closed_circuits': [c for c in circuits if c['state'].get('state') == 'closed'],
          'health_score': len([c for c in circuits if c['state'].get('state') == 'closed']) / len(circuits) if circuits else 1.0
      }

      # Alert if health score drops below threshold
      if report['health_score'] < 0.8:
          print("ALERT: Circuit breaker health score below 80%")

      print(json.dumps(report, indent=2))

  - id: send_circuit_health_alert
    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {
        "blocks": [
          {
            "type": "header",
            "text": {
              "type": "plain_text",
              "text": "⚡ Circuit Breaker Health Report"
            }
          },
          {
            "type": "section",
            "text": {
              "type": "mrkdwn",
              "text": "*Health Score:* {{ outputs.circuit_breaker_monitor.vars.health_score | round(2) * 100 }}%\n*Open Circuits:* {{ outputs.circuit_breaker_monitor.vars.open_circuits | length }}\n*Total Circuits:* {{ outputs.circuit_breaker_monitor.vars.total_circuits }}"
            }
          }
        ]
      }
    conditions:
      - type: io.kestra.plugin.core.condition.Expression
        expression: "{{ outputs.circuit_breaker_monitor.vars.health_score < 0.8 }}"

Pattern 3: Saga Pattern for Distributed Transactions

Implementing Compensating Transactions

The Saga pattern manages distributed transactions by breaking them into a sequence of local transactions, each with a compensating action.

id: saga-pattern-order-processing
namespace: acme.ecommerce
description: Distributed transaction with compensation logic

inputs:
  - name: order_data
    type: JSON
    required: true

tasks:
  # SAGA: Main transaction sequence
  - id: saga_orchestrator
    type: io.kestra.plugin.core.flow.Sequential
    tasks:
      # Step 1: Reserve inventory (with compensation)
      - id: reserve_inventory
        type: io.kestra.plugin.core.http.Request
        uri: "https://inventory.acme.com/api/reserve"
        method: POST
        body: |
          {
            "items": {{ inputs.order_data.items | tojson }},
            "order_id": "{{ execution.id }}"
          }
        store: true
        allowFailure: false

      # Step 2: Process payment (with compensation)
      - id: process_payment
        type: io.kestra.plugin.core.http.Request
        uri: "https://payments.acme.com/api/charge"
        method: POST
        body: |
          {
            "order_id": "{{ execution.id }}",
            "amount": {{ inputs.order_data.total }},
            "payment_method": {{ inputs.order_data.payment_method | tojson }}
          }
        store: true
        allowFailure: false

      # Step 3: Schedule shipping (with compensation)
      - id: schedule_shipping
        type: io.kestra.plugin.core.http.Request
        uri: "https://shipping.acme.com/api/schedule"
        method: POST
        body: |
          {
            "order_id": "{{ execution.id }}",
            "address": {{ inputs.order_data.shipping_address | tojson }},
            "items": {{ inputs.order_data.items | tojson }}
          }
        store: true
        allowFailure: false

      # Step 4: Send confirmation (no compensation needed - idempotent)
      - id: send_confirmation
        type: io.kestra.plugin.core.http.Request
        uri: "https://notifications.acme.com/api/send"
        method: POST
        body: |
          {
            "order_id": "{{ execution.id }}",
            "customer_email": "{{ inputs.order_data.customer_email }}",
            "template": "order_confirmation"
          }
        store: true
        allowFailure: true  # Non-critical step

  # SAGA: Compensation handlers (run on failure)
  - id: compensation_orchestrator
    type: io.kestra.plugin.core.flow.Sequential
    tasks:
      # Compensate in reverse order
      - id: cancel_shipping_if_scheduled
        type: io.kestra.plugin.core.http.Request
        uri: "https://shipping.acme.com/api/cancel/{{ execution.id }}"
        method: DELETE
        conditions:
          - type: io.kestra.plugin.core.condition.Expression
            expression: >
              {{ outputs.saga_orchestrator.outputs.schedule_shipping != null and
                 outputs.saga_orchestrator.outputs.schedule_shipping.state.current == 'SUCCESS' }}

      - id: refund_payment_if_charged
        type: io.kestra.plugin.core.http.Request
        uri: "https://payments.acme.com/api/refund/{{ execution.id }}"
        method: POST
        conditions:
          - type: io.kestra.plugin.core.condition.Expression
            expression: >
              {{ outputs.saga_orchestrator.outputs.process_payment != null and
                 outputs.saga_orchestrator.outputs.process_payment.state.current == 'SUCCESS' }}

      - id: release_inventory_if_reserved
        type: io.kestra.plugin.core.http.Request
        uri: "https://inventory.acme.com/api/release/{{ execution.id }}"
        method: DELETE
        conditions:
          - type: io.kestra.plugin.core.condition.Expression
            expression: >
              {{ outputs.saga_orchestrator.outputs.reserve_inventory != null and
                 outputs.saga_orchestrator.outputs.reserve_inventory.state.current == 'SUCCESS' }}

    conditions:
      - type: io.kestra.plugin.core.condition.Expression
        expression: "{{ outputs.saga_orchestrator.state.current == 'FAILED' }}"

  # SAGA: State persistence
  - id: persist_saga_state
    type: io.kestra.plugin.jdbc.postgresql.Query
    sql: |
      INSERT INTO saga_states 
      (saga_id, execution_id, step, status, data, created_at, compensated)
      VALUES (
        'order_processing',
        '{{ execution.id }}',
        '{{ taskrun.id }}',
        '{{ taskrun.state.current }}',
        '{{ taskrun.value | tojson }}',
        NOW(),
        {{ outputs.compensation_orchestrator != null }}
      )
    conditions:
      - type: io.kestra.plugin.core.condition.Expression
        expression: "true"  # Always run after each task
    send:
      - saga_orchestrator
      - compensation_orchestrator

Choreographed Saga Pattern

id: choreographed-saga
namespace: acme.advanced
description: Event-driven saga pattern

tasks:
  - id: start_saga
    type: io.kestra.plugin.core.debug.Return
    format: "SAGA_STARTED:{{ execution.id }}"

  - id: publish_order_created
    type: io.kestra.plugin.kafka.Produce
    topic: "order-events"
    key: "{{ execution.id }}"
    value: |
      {
        "event_type": "ORDER_CREATED",
        "order_id": "{{ execution.id }}",
        "timestamp": "{{ now() }}",
        "data": {{ inputs.order_data | tojson }}
      }
    properties:
      bootstrap.servers: "kafka:9092"

  # Each service listens to events and publishes its own
  # This is more decoupled but harder to monitor

  - id: monitor_saga_completion
    type: io.kestra.plugin.kafka.Consume
    topic: "saga-completion-events"
    groupId: "saga-monitor-{{ execution.id }}"
    maxRecords: 1
    timeout: PT5M
    conditions:
      - type: io.kestra.plugin.core.condition.Expression
        expression: "{{ outputs.publish_order_created.state.current == 'SUCCESS' }}"

Pattern 4: Event Sourcing with Kestra

Building Event-Sourced Systems

Event sourcing stores state changes as a sequence of events, providing audit trails and time-travel capabilities.

id: event-sourced-shopping-cart
namespace: acme.ecommerce
description: Event-sourced shopping cart implementation

inputs:
  - name: cart_id
    type: STRING
    required: false
    defaults: "cart_{{ execution.id }}"
  - name: action
    type: STRING
    required: true
    choices: ["add_item", "remove_item", "update_quantity", "clear_cart", "checkout"]
  - name: item_data
    type: JSON
    required: false

tasks:
  # Validate action based on current state
  - id: load_current_state
    type: io.kestra.plugin.jdbc.postgresql.Query
    fetch: true
    sql: |
      WITH current_state AS (
        SELECT 
          cart_id,
          SUM(CASE WHEN event_type = 'ITEM_ADDED' THEN 1 ELSE 0 END) as items_added,
          SUM(CASE WHEN event_type = 'ITEM_REMOVED' THEN 1 ELSE 0 END) as items_removed,
          MAX(occurred_at) as last_event_at
        FROM cart_events
        WHERE cart_id = '{{ inputs.cart_id }}'
        GROUP BY cart_id
      ),
      last_checkout AS (
        SELECT MAX(occurred_at) as checkout_time
        FROM cart_events
        WHERE cart_id = '{{ inputs.cart_id }}'
          AND event_type = 'CHECKOUT_COMPLETED'
      )
      SELECT 
        cs.cart_id,
        cs.items_added - cs.items_removed as current_item_count,
        lc.checkout_time
      FROM current_state cs
      LEFT JOIN last_checkout lc ON 1=1
    ignoreMissing: true

  - id: validate_action
    type: io.kestra.plugin.core.flow.Switch
    value: "{{ inputs.action }}"
    cases:
      checkout:
        - id: check_cart_not_empty
          type: io.kestra.plugin.core.condition.Expression
          expression: >
            {{ outputs.load_current_state.rows[0].current_item_count | default(0) > 0 }}
          errorMessage: "Cannot checkout empty cart"

      add_item:
        - id: validate_item_data
          type: io.kestra.plugin.core.condition.Expression
          expression: >
            {{ inputs.item_data != null and 
               inputs.item_data.item_id and 
               inputs.item_data.quantity > 0 }}
          errorMessage: "Invalid item data"

  # Store the event
  - id: store_event
    type: io.kestra.plugin.jdbc.postgresql.Query
    sql: |
      INSERT INTO cart_events
      (event_id, cart_id, event_type, event_data, occurred_at, version)
      VALUES (
        '{{ execution.id }}',
        '{{ inputs.cart_id }}',
        '{{ inputs.action | upper }}',
        '{{ inputs.item_data | default({}) | tojson }}',
        NOW(),
        COALESCE(
          (SELECT MAX(version) + 1 
           FROM cart_events 
           WHERE cart_id = '{{ inputs.cart_id }}'),
          1
        )
      )

  # Project current state from events
  - id: project_current_state
    type: io.kestra.plugin.scripts.python.Script
    description: "Project current cart state from event stream"
    sql: |
      SELECT * FROM cart_events 
      WHERE cart_id = '{{ inputs.cart_id }}'
      ORDER BY version
    script: |
      import json

      # Replay events to build current state
      cart_state = {
          'cart_id': '{{ inputs.cart_id }}',
          'items': {},
          'total_quantity': 0,
          'total_price': 0.0,
          'event_count': 0
      }

      # In a real implementation, you would fetch events from DB
      # For this example, we'll simulate event replay

      # Process based on action
      action = '{{ inputs.action }}'
      item_data = {{ inputs.item_data | default('{}') | tojson }}

      if action == 'add_item':
          item_id = item_data.get('item_id')
          if item_id:
              if item_id not in cart_state['items']:
                  cart_state['items'][item_id] = {
                      'quantity': 0,
                      'price': item_data.get('price', 0)
                  }
              cart_state['items'][item_id]['quantity'] += item_data.get('quantity', 1)
              cart_state['total_quantity'] += item_data.get('quantity', 1)
              cart_state['total_price'] += item_data.get('price', 0) * item_data.get('quantity', 1)

      elif action == 'remove_item':
          item_id = item_data.get('item_id')
          if item_id in cart_state['items']:
              quantity_to_remove = min(
                  item_data.get('quantity', 1),
                  cart_state['items'][item_id]['quantity']
              )
              cart_state['items'][item_id]['quantity'] -= quantity_to_remove
              cart_state['total_quantity'] -= quantity_to_remove
              cart_state['total_price'] -= cart_state['items'][item_id]['price'] * quantity_to_remove

              if cart_state['items'][item_id]['quantity'] <= 0:
                  del cart_state['items'][item_id]

      cart_state['event_count'] += 1

      print(json.dumps(cart_state, indent=2))

  # Publish event to event bus
  - id: publish_to_event_bus
    type: io.kestra.plugin.kafka.Produce
    topic: "cart-events"
    key: "{{ inputs.cart_id }}"
    value: |
      {
        "cart_id": "{{ inputs.cart_id }}",
        "event_type": "{{ inputs.action | upper }}",
        "event_data": {{ inputs.item_data | default({}) | tojson }},
        "timestamp": "{{ now() }}",
        "event_id": "{{ execution.id }}"
      }
    properties:
      bootstrap.servers: "kafka:9092"

Time-Travel Query Pattern

id: time-travel-query
namespace: acme.advanced
description: Query historical state at a specific point in time

inputs:
  - name: entity_id
    type: STRING
    required: true
  - name: as_of_time
    type: DATETIME
    required: true

tasks:
  - id: replay_events_to_timestamp
    type: io.kestra.plugin.jdbc.postgresql.Query
    fetch: true
    sql: |
      SELECT 
        event_type,
        event_data,
        occurred_at,
        version
      FROM entity_events
      WHERE entity_id = '{{ inputs.entity_id }}'
        AND occurred_at <= '{{ inputs.as_of_time }}'
      ORDER BY version

  - id: reconstruct_historical_state
    type: io.kestra.plugin.scripts.python.Script
    inputFiles:
      events.json: |
        {{ outputs.replay_events_to_timestamp.rows | tojson }}
    script: |
      import json

      with open('events.json', 'r') as f:
          events = json.load(f)

      # Start with empty state
      state = {}

      # Replay events in order
      for event in events:
          event_type = event['event_type']
          event_data = event['event_data']

          if event_type == 'ENTITY_CREATED':
              state = event_data
          elif event_type == 'ENTITY_UPDATED':
              state.update(event_data)
          elif event_type == 'FIELD_SET':
              field_name = event_data['field']
              state[field_name] = event_data['value']
          elif event_type == 'FIELD_INCREMENTED':
              field_name = event_data['field']
              if field_name in state:
                  state[field_name] += event_data['amount']
              else:
                  state[field_name] = event_data['amount']
          # Add more event handlers as needed

      result = {
          'entity_id': '{{ inputs.entity_id }}',
          'as_of_time': '{{ inputs.as_of_time }}',
          'state': state,
          'events_replayed': len(events),
          'last_event_time': events[-1]['occurred_at'] if events else None
      }

      print(json.dumps(result, indent=2))

Pattern 5: CQRS (Command Query Responsibility Segregation)

Separating Read and Write Models

CQRS separates read and write operations for better scalability and optimization.

id: cqrs-user-profile
namespace: acme.social
description: CQRS pattern for user profiles

inputs:
  - name: command
    type: STRING
    required: true
    choices: ["update_profile", "get_profile", "search_profiles"]
  - name: user_id
    type: STRING
    required: false
  - name: profile_data
    type: JSON
    required: false

tasks:
  # Command Side: Handle writes
  - id: handle_command
    type: io.kestra.plugin.core.flow.Switch
    value: "{{ inputs.command }}"
    cases:
      update_profile:
        - id: validate_profile_data
          type: io.kestra.plugin.core.condition.Expression
          expression: >
            {{ inputs.profile_data != null and 
               inputs.user_id != null }}
          errorMessage: "Missing profile data or user_id"

        - id: store_write_event
          type: io.kestra.plugin.jdbc.postgresql.Query
          sql: |
            INSERT INTO user_profile_events
            (event_id, user_id, event_type, event_data, occurred_at)
            VALUES (
              '{{ execution.id }}',
              '{{ inputs.user_id }}',
              'PROFILE_UPDATED',
              '{{ inputs.profile_data | tojson }}',
              NOW()
            )

        - id: update_read_model_async
          type: io.kestra.plugin.core.flow.Subflow
          namespace: acme.background
          flowId: update-profile-read-model
          wait: false  # Fire and forget - eventual consistency
          transmit:
            - name: user_id
              value: "{{ inputs.user_id }}"
            - name: profile_data
              value: "{{ inputs.profile_data | tojson }}"

        - id: invalidate_cache
          type: io.kestra.plugin.redis.Delete
          key: "profile:{{ inputs.user_id }}"
          host: "redis"
          port: 6379

        - id: send_profile_updated_event
          type: io.kestra.plugin.kafka.Produce
          topic: "user-profile-events"
          key: "{{ inputs.user_id }}"
          value: |
            {
              "event_type": "PROFILE_UPDATED",
              "user_id": "{{ inputs.user_id }}",
              "timestamp": "{{ now() }}",
              "event_id": "{{ execution.id }}"
            }

  # Query Side: Handle reads (optimized for reading)
  - id: handle_query
    type: io.kestra.plugin.core.flow.Switch
    value: "{{ inputs.command }}"
    cases:
      get_profile:
        - id: try_cache_first
          type: io.kestra.plugin.redis.Get
          key: "profile:{{ inputs.user_id }}"
          host: "redis"
          port: 6379

        - id: cache_hit
          type: io.kestra.plugin.core.debug.Return
          format: "{{ outputs.try_cache_first.value }}"
          conditions:
            - type: io.kestra.plugin.core.condition.Expression
              expression: "{{ outputs.try_cache_first.value != null }}"

        - id: cache_miss_query
          type: io.kestra.plugin.jdbc.postgresql.Query
          fetch: true
          sql: |
            SELECT 
              user_id,
              display_name,
              avatar_url,
              bio,
              location,
              website,
              follower_count,
              following_count,
              post_count,
              last_updated
            FROM user_profiles_read
            WHERE user_id = '{{ inputs.user_id }}'
          conditions:
            - type: io.kestra.plugin.core.condition.Expression
              expression: "{{ outputs.try_cache_first.value == null }}"

        - id: populate_cache
          type: io.kestra.plugin.redis.Put
          key: "profile:{{ inputs.user_id }}"
          value: |
            {{ outputs.cache_miss_query.rows[0] | tojson }}
          ttl: PT1H
          host: "redis"
          port: 6379
          conditions:
            - type: io.kestra.plugin.core.condition.Expression
              expression: >
                {{ outputs.cache_miss_query.rows | length > 0 }}

        - id: return_profile
          type: io.kestra.plugin.core.debug.Return
          format: >
            {{ outputs.cache_miss_query.rows[0] | tojson if outputs.cache_miss_query.rows else '{}' }}
          conditions:
            - type: io.kestra.plugin.core.condition.Expression
              expression: "{{ outputs.try_cache_first.value == null }}"

      search_profiles:
        - id: search_read_model
          type: io.kestra.plugin.jdbc.elasticsearch.Search
          connection:
            hosts: ["elasticsearch:9200"]
          index: "user_profiles"
          query: |
            {
              "query": {
                "multi_match": {
                  "query": "{{ inputs.profile_data.query }}",
                  "fields": ["display_name^2", "bio", "location"]
                }
              },
              "size": 20
            }

        - id: return_search_results
          type: io.kestra.plugin.core.debug.Return
          format: >
            {{ outputs.search_read_model.hits | tojson }}

Read Model Update Processor

id: update-profile-read-model
namespace: acme.background
description: Background processor to update read models

tasks:
  - id: listen_for_profile_events
    type: io.kestra.plugin.kafka.Consume
    topic: "user-profile-events"
    groupId: "read-model-updater"
    maxRecords: 100
    timeout: PT10S

  - id: process_events_batch
    type: io.kestra.plugin.core.flow.EachSequential
    value: "{{ outputs.listen_for_profile_events.messages }}"
    tasks:
      - id: "process_event_{{ taskloop.index }}"
        type: io.kestra.plugin.core.flow.Switch
        value: "{{ task.value.value.event_type }}"
        cases:
          PROFILE_UPDATED:
            - id: update_read_model
              type: io.kestra.plugin.jdbc.postgresql.Query
              sql: |
                INSERT INTO user_profiles_read
                (user_id, display_name, avatar_url, bio, location, website, last_updated)
                VALUES (
                  '{{ task.value.value.user_id }}',
                  '{{ task.value.value.event_data.display_name | default('') }}',
                  '{{ task.value.value.event_data.avatar_url | default('') }}',
                  '{{ task.value.value.event_data.bio | default('') }}',
                  '{{ task.value.value.event_data.location | default('') }}',
                  '{{ task.value.value.event_data.website | default('') }}',
                  NOW()
                )
                ON CONFLICT (user_id) DO UPDATE SET
                  display_name = EXCLUDED.display_name,
                  avatar_url = EXCLUDED.avatar_url,
                  bio = EXCLUDED.bio,
                  location = EXCLUDED.location,
                  website = EXCLUDED.website,
                  last_updated = NOW()

            - id: update_search_index
              type: io.kestra.plugin.jdbc.elasticsearch.Index
              connection:
                hosts: ["elasticsearch:9200"]
              index: "user_profiles"
              id: "{{ task.value.value.user_id }}"
              document: |
                {
                  "user_id": "{{ task.value.value.user_id }}",
                  "display_name": "{{ task.value.value.event_data.display_name | default('') }}",
                  "bio": "{{ task.value.value.event_data.bio | default('') }}",
                  "location": "{{ task.value.value.event_data.location | default('') }}",
                  "last_updated": "{{ now() }}"
                }

          FOLLOW_ADDED:
            - id: increment_follower_count
              type: io.kestra.plugin.jdbc.postgresql.Query
              sql: |
                UPDATE user_profiles_read
                SET follower_count = follower_count + 1
                WHERE user_id = '{{ task.value.value.target_user_id }}'

            - id: increment_following_count
              type: io.kestra.plugin.jdbc.postgresql.Query
              sql: |
                UPDATE user_profiles_read
                SET following_count = following_count + 1
                WHERE user_id = '{{ task.value.value.source_user_id }}'

  - id: commit_kafka_offsets
    type: io.kestra.plugin.kafka.Consume
    topic: "user-profile-events"
    groupId: "read-model-updater"
    maxRecords: 0  # Just commit offsets
    conditions:
      - type: io.kestra.plugin.core.condition.Expression
        expression: "{{ outputs.listen_for_profile_events.messages | length > 0 }}"

Pattern 6: Strangler Fig Pattern

Incremental Migration Strategy

The Strangler Fig pattern incrementally replaces a legacy system by building new functionality around it, then gradually migrating.

id: strangler-fig-migration
namespace: acme.migration
description: Incrementally migrate from legacy to new system

variables:
  migration_phase: "parallel_run"  # Options: shadow, parallel_run, cutover
  traffic_percentage: 50  # Percentage of traffic to send to new system

tasks:
  # Feature Flag: Route traffic based on migration phase
  - id: route_traffic
    type: io.kestra.plugin.core.flow.Switch
    value: "{{ vars.migration_phase }}"
    cases:
      shadow:
        # Shadow mode: Send to both, compare results
        - id: call_legacy_system
          type: io.kestra.plugin.core.http.Request
          uri: "https://legacy-api.acme.com/process"
          method: POST
          body: "{{ inputs.request_body }}"
          store: true

        - id: call_new_system_shadow
          type: io.kestra.plugin.core.http.Request
          uri: "https://new-api.acme.com/process"
          method: POST
          body: "{{ inputs.request_body }}"
          store: true

        - id: compare_results
          type: io.kestra.plugin.scripts.python.Script
          inputFiles:
            legacy_response.json: "{{ outputs.call_legacy_system.uri }}"
            new_response.json: "{{ outputs.call_new_system_shadow.uri }}"
          script: |
            import json
            import difflib

            with open('legacy_response.json', 'r') as f:
                legacy = json.load(f)
            with open('new_response.json', 'r') as f:
                new = json.load(f)

            # Compare results
            comparison = {
                'match': legacy == new,
                'legacy_response': legacy,
                'new_response': new,
                'differences': []
            }

            if legacy != new:
                # Find specific differences
                legacy_str = json.dumps(legacy, sort_keys=True, indent=2)
                new_str = json.dumps(new, sort_keys=True, indent=2)

                diff = list(difflib.unified_diff(
                    legacy_str.splitlines(),
                    new_str.splitlines(),
                    lineterm=''
                ))

                comparison['differences'] = diff[:10]  # First 10 differences

            print(json.dumps(comparison, indent=2))

        - id: return_legacy_response
          type: io.kestra.plugin.core.debug.Return
          format: "{{ outputs.call_legacy_system.body | tojson }}"

      parallel_run:
        # Parallel run: Split traffic between systems
        - id: decide_routing
          type: io.kestra.plugin.core.debug.Return
          format: >
            {{ random() * 100 <= vars.traffic_percentage }}

        - id: call_new_system_parallel
          type: io.kestra.plugin.core.http.Request
          uri: "https://new-api.acme.com/process"
          method: POST
          body: "{{ inputs.request_body }}"
          store: true
          conditions:
            - type: io.kestra.plugin.core.condition.Expression
              expression: "{{ outputs.decide_routing.value == true }}"

        - id: call_legacy_system_fallback
          type: io.kestra.plugin.core.http.Request
          uri: "https://legacy-api.acme.com/process"
          method: POST
          body: "{{ inputs.request_body }}"
          store: true
          conditions:
            - type: io.kestra.plugin.core.condition.Expression
              expression: "{{ outputs.decide_routing.value == false }}"

        - id: return_response
          type: io.kestra.plugin.core.flow.Switch
          value: "{{ outputs.decide_routing.value }}"
          cases:
            true:
              - id: return_new_response
                type: io.kestra.plugin.core.debug.Return
                format: "{{ outputs.call_new_system_parallel.body | tojson }}"
            false:
              - id: return_legacy_response_fallback
                type: io.kestra.plugin.core.debug.Return
                format: "{{ outputs.call_legacy_system_fallback.body | tojson }}"

      cutover:
        # Cutover: All traffic to new system
        - id: call_new_system_primary
          type: io.kestra.plugin.core.http.Request
          uri: "https://new-api.acme.com/process"
          method: POST
          body: "{{ inputs.request_body }}"
          store: true

        - id: return_new_response_primary
          type: io.kestra.plugin.core.debug.Return
          format: "{{ outputs.call_new_system_primary.body | tojson }}"

        - id: legacy_system_backup
          type: io.kestra.plugin.core.http.Request
          uri: "https://legacy-api.acme.com/process"
          method: POST
          body: "{{ inputs.request_body }}"
          store: true
          allowFailure: true  # Legacy might be turned off

  # Migration Monitoring
  - id: track_migration_metrics
    type: io.kestra.plugin.jdbc.postgresql.Query
    sql: |
      INSERT INTO migration_metrics
      (migration_phase, traffic_percentage, request_count, 
       success_count, error_count, avg_response_time, timestamp)
      SELECT 
        '{{ vars.migration_phase }}',
        {{ vars.traffic_percentage }},
        COUNT(*) as request_count,
        SUM(CASE WHEN status_code = 200 THEN 1 ELSE 0 END) as success_count,
        SUM(CASE WHEN status_code != 200 THEN 1 ELSE 0 END) as error_count,
        AVG(response_time_ms) as avg_response_time,
        NOW()
      FROM api_requests
      WHERE timestamp >= NOW() - INTERVAL '1 minute'
    conditions:
      - type: io.kestra.plugin.core.condition.Expression
        expression: "{{ vars.migration_phase != 'completed' }}"

Data Migration with Dual Write

id: dual-write-migration
namespace: acme.migration
description: Dual write to both old and new databases during migration

tasks:
  - id: write_to_both_systems
    type: io.kestra.plugin.core.flow.Parallel
    tasks:
      - id: write_to_legacy
        type: io.kestra.plugin.jdbc.postgresql.Query
        url: "jdbc:postgresql://legacy-db.acme.com:5432/app"
        username: "{{ secret('LEGACY_DB_USER') }}"
        password: "{{ secret('LEGACY_DB_PASSWORD') }}"
        sql: |
          INSERT INTO orders 
          (order_id, customer_id, amount, status, created_at)
          VALUES (
            '{{ inputs.order_id }}',
            '{{ inputs.customer_id }}',
            {{ inputs.amount }},
            'pending',
            NOW()
          )
        allowFailure: false

      - id: write_to_new
        type: io.kestra.plugin.jdbc.postgresql.Query
        url: "jdbc:postgresql://new-db.acme.com:5432/app"
        username: "{{ secret('NEW_DB_USER') }}"
        password: "{{ secret('NEW_DB_PASSWORD') }}"
        sql: |
          INSERT INTO orders 
          (order_id, customer_id, amount, status, created_at, metadata)
          VALUES (
            '{{ inputs.order_id }}',
            '{{ inputs.customer_id }}',
            {{ inputs.amount }},
            'pending',
            NOW(),
            '{"migration_phase": "dual_write"}'
          )
        allowFailure: true  # New system might have issues

  - id: verify_data_consistency
    type: io.kestra.plugin.scripts.python.Script
    description: "Verify data was written to both systems consistently"
    script: |
      import psycopg2
      import json

      def query_db(host, dbname, user, password, query):
          conn = psycopg2.connect(
              host=host, dbname=dbname, user=user, password=password
          )
          cur = conn.cursor()
          cur.execute(query)
          result = cur.fetchone()
          cur.close()
          conn.close()
          return result

      # Query both systems
      legacy_query = f"""
          SELECT order_id, customer_id, amount, status
          FROM orders
          WHERE order_id = '{ {{ inputs.order_id }} }'
      """

      new_query = f"""
          SELECT order_id, customer_id, amount, status
          FROM orders
          WHERE order_id = '{ {{ inputs.order_id }} }'
      """

      try:
          legacy_result = query_db(
              'legacy-db.acme.com', 'app',
              '{{ secret('LEGACY_DB_USER') }}', '{{ secret('LEGACY_DB_PASSWORD') }}',
              legacy_query
          )
      except Exception as e:
          legacy_result = None
          print(f"Legacy query failed: {e}")

      try:
          new_result = query_db(
              'new-db.acme.com', 'app',
              '{{ secret('NEW_DB_USER') }}', '{{ secret('NEW_DB_PASSWORD') }}',
              new_query
          )
      except Exception as e:
          new_result = None
          print(f"New query failed: {e}")

      # Compare results
      comparison = {
          'order_id': '{{ inputs.order_id }}',
          'legacy_found': legacy_result is not None,
          'new_found': new_result is not None,
          'consistent': legacy_result == new_result,
          'legacy_data': legacy_result,
          'new_data': new_result
      }

      print(json.dumps(comparison, indent=2))

      if not comparison['consistent']:
          # Log inconsistency for manual review
          print("WARNING: Data inconsistency detected!")

  - id: handle_inconsistency
    type: io.kestra.plugin.core.flow.Subflow
    namespace: acme.reconciliation
    flowId: reconcile-dual-write
    wait: false  # Async reconciliation
    conditions:
      - type: io.kestra.plugin.core.condition.Expression
        expression: >
          {{ outputs.verify_data_consistency.vars.consistent == false }}

Pattern 7: Sidecar Pattern for Cross-Cutting Concerns

Implementing Cross-Cutting Logic

The sidecar pattern attaches supporting functionality to the main workflow without modifying its core logic.

id: main-business-workflow
namespace: acme.business
description: Main business logic with sidecar attachments

tasks:
  - id: business_logic
    type: io.kestra.plugin.core.flow.Sequential
    tasks:
      - id: step1
        type: io.kestra.plugin.core.log.Log
        message: "Business step 1"

      - id: step2
        type: io.kestra.plugin.core.debug.Return
        format: "Business step 2 result"

      - id: step3
        type: io.kestra.plugin.core.log.Log
        message: "Business step 3 with result: {{ outputs.step2.value }}"

# Sidecar flow that can be attached to any workflow
id: monitoring-sidecar
namespace: acme.sidecars
description: Attachable monitoring and observability sidecar

tasks:
  - id: start_monitoring
    type: io.kestra.plugin.core.debug.Return
    format: "MONITORING_START:{{ parent.execution.id }}"

  - id: collect_metrics
    type: io.kestra.plugin.core.http.Request
    uri: "http://metrics-collector.acme.com/api/metrics"
    method: POST
    body: |
      {
        "execution_id": "{{ parent.execution.id }}",
        "flow_id": "{{ parent.flow.id }}",
        "namespace": "{{ parent.namespace.id }}",
        "start_time": "{{ parent.execution.startDate }}",
        "metrics": {
          "task_count": {{ parent.tasks | length }},
          "inputs": {{ parent.inputs | default({}) | tojson }}
        }
      }
    store: true

  - id: log_performance
    type: io.kestra.plugin.core.log.Log
    message: |
      Sidecar monitoring for {{ parent.execution.id }}
      Flow: {{ parent.flow.id }}
      Tasks monitored: {{ parent.tasks | length }}

Composed Workflow with Sidecars

id: composed-workflow-with-sidecars
namespace: acme.composed
description: Compose main workflow with multiple sidecars

tasks:
  # Attach monitoring sidecar
  - id: attach_monitoring
    type: io.kestra.plugin.core.flow.Subflow
    namespace: acme.sidecars
    flowId: monitoring-sidecar
    wait: false  # Run alongside

  # Attach auditing sidecar
  - id: attach_auditing
    type: io.kestra.plugin.core.flow.Subflow
    namespace: acme.sidecars
    flowId: auditing-sidecar
    wait: false

  # Main business logic
  - id: main_business_process
    type: io.kestra.plugin.core.flow.Subflow
    namespace: acme.business
    flowId: main-business-workflow
    wait: true

  # Attach cleanup sidecar (runs after main process)
  - id: attach_cleanup
    type: io.kestra.plugin.core.flow.Subflow
    namespace: acme.sidecars
    flowId: cleanup-sidecar
    wait: false
    conditions:
      - type: io.kestra.plugin.core.condition.Expression
        expression: >
          {{ outputs.main_business_process.state.current in ['SUCCESS', 'FAILED'] }}

Pattern 8: Pipeline Pattern with Middleware

Building Processing Pipelines

id: data-processing-pipeline
namespace: acme.pipelines
description: Configurable pipeline with middleware support

variables:
  pipeline_stages: ["validate", "transform", "enrich", "filter", "output"]
  middleware_enabled: ["logging", "metrics", "validation", "caching"]

tasks:
  # Pipeline orchestrator
  - id: pipeline_orchestrator
    type: io.kestra.plugin.core.flow.Sequential
    tasks:
      - id: load_data
        type: io.kestra.plugin.core.http.Download
        uri: "https://api.acme.com/data"
        store: true

      # Dynamic pipeline stages
      - id: execute_pipeline_stages
        type: io.kestra.plugin.core.flow.EachSequential
        value: "{{ vars.pipeline_stages }}"
        tasks:
          - id: "execute_{{ task.value }}"
            type: io.kestra.plugin.core.flow.Switch
            value: "{{ task.value }}"
            cases:
              validate:
                - id: run_validation
                  type: io.kestra.plugin.core.flow.Subflow
                  namespace: acme.pipeline.middleware
                  flowId: validation-middleware
                  wait: true
                  transmit:
                    - name: data
                      value: "{{ outputs.load_data.uri }}"

              transform:
                - id: run_transformation
                  type: io.kestra.plugin.scripts.python.Script
                  inputFiles:
                    data.json: "{{ outputs.execute_validation.outputs.validated_data | default(outputs.load_data.uri) }}"
                  script: |
                    import json
                    import pandas as pd

                    with open('data.json', 'r') as f:
                        data = json.load(f)

                    # Transformation logic
                    df = pd.DataFrame(data)
                    # ... transformation steps

                    df.to_parquet('transformed.parquet', index=False)
                    print("Transformation complete")

              enrich:
                - id: run_enrichment
                  type: io.kestra.plugin.core.flow.Subflow
                  namespace: acme.pipeline.middleware
                  flowId: enrichment-middleware
                  wait: true

              filter:
                - id: run_filtering
                  type: io.kestra.plugin.scripts.python.Script
                  script: |
                    # Filtering logic
                    print("Filtering complete")

              output:
                - id: produce_output
                  type: io.kestra.plugin.core.debug.Return
                  format: "Pipeline execution complete"

          # Apply middleware after each stage
          - id: "apply_middleware_{{ task.value }}"
            type: io.kestra.plugin.core.flow.Parallel
            tasks:
              - id: "logging_middleware_{{ task.value }}"
                type: io.kestra.plugin.core.flow.Subflow
                namespace: acme.pipeline.middleware
                flowId: logging-middleware
                wait: false
                transmit:
                  - name: stage
                    value: "{{ task.value }}"
                  - name: execution_id
                    value: "{{ execution.id }}"
                conditions:
                  - type: io.kestra.plugin.core.condition.Expression
                    expression: "'logging' in vars.middleware_enabled"

              - id: "metrics_middleware_{{ task.value }}"
                type: io.kestra.plugin.core.flow.Subflow
                namespace: acme.pipeline.middleware
                flowId: metrics-middleware
                wait: false
                conditions:
                  - type: io.kestra.plugin.core.condition.Expression
                    expression: "'metrics' in vars.middleware_enabled"

Middleware Implementation

id: logging-middleware
namespace: acme.pipeline.middleware
description: Logging middleware for pipeline stages

tasks:
  - id: log_stage_execution
    type: io.kestra.plugin.jdbc.postgresql.Query
    sql: |
      INSERT INTO pipeline_logs
      (execution_id, stage, status, start_time, end_time, duration_ms)
      VALUES (
        '{{ inputs.execution_id }}',
        '{{ inputs.stage }}',
        '{{ parent.state.current }}',
        '{{ parent.startDate }}',
        '{{ now() }}',
        EXTRACT(EPOCH FROM (NOW() - '{{ parent.startDate }}')) * 1000
      )

Pattern 9: Bulkhead Pattern for Fault Isolation

Isolating Failures with Resource Pools

id: bulkhead-pattern
namespace: acme.resilience
description: Isolate failures using resource pools

variables:
  # Different resource pools for different services
  resource_pools:
    payment_service:
      max_concurrent: 5
      timeout: PT30S
    inventory_service:
      max_concurrent: 3
      timeout: PT20S
    shipping_service:
      max_concurrent: 2
      timeout: PT60S

tasks:
  # Resource pool manager
  - id: acquire_resource_slot
    type: io.kestra.plugin.core.cache.Increment
    key: "resource_pool:{{ inputs.service_name }}:active"
    defaultValue: 0
    ttl: PT5M

  - id: check_pool_capacity
    type: io.kestra.plugin.core.condition.Expression
    expression: >
      {{ outputs.acquire_resource_slot.value <= 
         vars.resource_pools[inputs.service_name].max_concurrent }}

  - id: execute_with_resource_limit
    type: io.kestra.plugin.core.flow.Switch
    value: "{{ inputs.service_name }}"
    cases:
      payment_service:
        - id: call_payment_service
          type: io.kestra.plugin.core.http.Request
          uri: "https://payments.acme.com/api/process"
          method: POST
          body: "{{ inputs.request_data }}"
          timeout: "{{ vars.resource_pools.payment_service.timeout }}"
          conditions:
            - type: io.kestra.plugin.core.condition.Expression
              expression: "{{ outputs.check_pool_capacity.value }}"

      inventory_service:
        - id: call_inventory_service
          type: io.kestra.plugin.core.http.Request
          uri: "https://inventory.acme.com/api/check"
          method: POST
          body: "{{ inputs.request_data }}"
          timeout: "{{ vars.resource_pools.inventory_service.timeout }}"
          conditions:
            - type: io.kestra.plugin.core.condition.Expression
              expression: "{{ outputs.check_pool_capacity.value }}"

  - id: pool_capacity_exceeded
    type: io.kestra.plugin.core.debug.Return
    format: "Resource pool capacity exceeded for {{ inputs.service_name }}"
    conditions:
      - type: io.kestra.plugin.core.condition.Expression
        expression: "{{ not outputs.check_pool_capacity.value }}"

  # Release resource slot
  - id: release_resource_slot
    type: io.kestra.plugin.core.cache.Decrement
    key: "resource_pool:{{ inputs.service_name }}:active"
    conditions:
      - type: io.kestra.plugin.core.condition.Expression
        expression: >
          {{ outputs.execute_with_resource_limit != null or
             outputs.pool_capacity_exceeded != null }}

Pattern 10: Ambassador Pattern for Service Abstraction

Abstracting External Services

id: payment-service-ambassador
namespace: acme.ambassadors
description: Ambassador pattern for payment service abstraction

tasks:
  # Service discovery and health check
  - id: discover_payment_endpoints
    type: io.kestra.plugin.core.http.Request
    uri: "https://service-discovery.acme.com/api/endpoints/payment"
    method: GET
    store: true

  - id: select_healthy_endpoint
    type: io.kestra.plugin.scripts.python.Script
    inputFiles:
      endpoints.json: "{{ outputs.discover_payment_endpoints.uri }}"
    script: |
      import json
      import requests
      from concurrent.futures import ThreadPoolExecutor, as_completed

      with open('endpoints.json', 'r') as f:
          endpoints = json.load(f)

      def check_endpoint_health(endpoint):
          try:
              response = requests.get(
                  f"{endpoint['url']}/health",
                  timeout=2
              )
              return {
                  'endpoint': endpoint,
                  'healthy': response.status_code == 200,
                  'response_time': response.elapsed.total_seconds()
              }
          except:
              return {
                  'endpoint': endpoint,
                  'healthy': False,
                  'response_time': float('inf')
              }

      # Check all endpoints in parallel
      with ThreadPoolExecutor(max_workers=5) as executor:
          futures = [executor.submit(check_endpoint_health, ep) for ep in endpoints]
          results = [f.result() for f in as_completed(futures)]

      # Select the healthiest endpoint
      healthy_endpoints = [r for r in results if r['healthy']]
      if healthy_endpoints:
          # Choose the fastest healthy endpoint
          selected = min(healthy_endpoints, key=lambda x: x['response_time'])
          print(json.dumps(selected['endpoint']))
      else:
          raise Exception("No healthy payment endpoints available")

  # Circuit breaker and retry logic
  - id: call_payment_service_with_retry
    type: io.kestra.plugin.core.http.Request
    uri: "{{ outputs.select_healthy_endpoint.vars.url }}/api/process"
    method: POST
    body: "{{ inputs.payment_data }}"
    retry:
      type: exponential
      maxAttempt: 3
      delay: PT1S
    timeout: PT10S

  # Response transformation
  - id: transform_payment_response
    type: io.kestra.plugin.scripts.python.Script
    inputFiles:
      response.json: "{{ outputs.call_payment_service_with_retry.uri }}"
    script: |
      import json

      with open('response.json', 'r') as f:
          response = json.load(f)

      # Standardize response format
      standardized = {
          'success': response.get('status') == 'success',
          'transaction_id': response.get('id'),
          'amount': response.get('amount'),
          'currency': response.get('currency', 'USD'),
          'timestamp': response.get('created'),
          'provider': 'payment_service',
          'metadata': {
              'original_response': response
          }
      }

      print(json.dumps(standardized, indent=2))

  # Cache successful responses
  - id: cache_payment_response
    type: io.kestra.plugin.redis.Put
    key: "payment:{{ inputs.payment_data.transaction_id }}"
    value: "{{ outputs.transform_payment_response.vars }}"
    ttl: PT5M
    conditions:
      - type: io.kestra.plugin.core.condition.Expression
        expression: "{{ outputs.transform_payment_response.vars.success == true }}"

Putting It All Together: A Complete Advanced Workflow

id: advanced-order-processing
namespace: acme.production
description: Complete advanced workflow combining multiple patterns

tasks:
  # Pattern 1: Circuit Breaker for external services
  - id: check_service_health
    type: io.kestra.plugin.core.flow.Parallel
    tasks:
      - id: check_payment_service
        type: io.kestra.plugin.core.flow.Subflow
        namespace: acme.resilience
        flowId: circuit-breaker-check
        wait: true
        transmit:
          - name: service_name
            value: "payment_service"

      - id: check_inventory_service
        type: io.kestra.plugin.core.flow.Subflow
        namespace: acme.resilience
        flowId: circuit-breaker-check
        wait: true
        transmit:
          - name: service_name
            value: "inventory_service"

  # Pattern 2: Bulkhead for resource isolation
  - id: process_with_bulkheads
    type: io.kestra.plugin.core.flow.Parallel
    tasks:
      - id: validate_order_bulkhead
        type: io.kestra.plugin.core.flow.Subflow
        namespace: acme.resilience
        flowId: bulkhead-processor
        wait: true
        transmit:
          - name: service_name
            value: "validation"
          - name: request_data
            value: "{{ inputs.order_data }}"

      - id: reserve_inventory_bulkhead
        type: io.kestra.plugin.core.flow.Subflow
        namespace: acme.resilience
        flowId: bulkhead-processor
        wait: true
        transmit:
          - name: service_name
            value: "inventory"
          - name: request_data
            value: "{{ inputs.order_data }}"
        conditions:
          - type: io.kestra.plugin.core.condition.Expression
            expression: >
              {{ outputs.check_service_health.outputs.check_inventory_service.outputs.circuit_state != 'open' }}

  # Pattern 3: Saga pattern for distributed transaction
  - id: execute_order_saga
    type: io.kestra.plugin.core.flow.Sequential
    tasks:
      - id: saga_step1_payment
        type: io.kestra.plugin.core.flow.Subflow
        namespace: acme.ambassadors
        flowId: payment-service-ambassador
        wait: true
        transmit:
          - name: payment_data
            value: "{{ inputs.payment_data }}"

      - id: saga_step2_fulfillment
        type: io.kestra.plugin.core.http.Request
        uri: "https://fulfillment.acme.com/api/process"
        method: POST
        body: |
          {
            "order_id": "{{ execution.id }}",
            "items": {{ inputs.order_data.items | tojson }},
            "payment_reference": "{{ outputs.saga_step1_payment.outputs.transaction_id }}"
          }

      - id: saga_step3_notification
        type: io.kestra.plugin.core.flow.Parallel
        tasks:
          - id: notify_customer
            type: io.kestra.plugin.notifications.email.EmailSend
            to: "{{ inputs.customer_email }}"
            subject: "Order Confirmation #{{ execution.id }}"
            htmlBody: |
              <h1>Thank you for your order!</h1>
              <p>Order ID: {{ execution.id }}</p>

          - id: notify_internal
            type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
            url: "{{ secret('SLACK_ORDERS_WEBHOOK') }}"
            payload: |
              {
                "text": "New order processed: {{ execution.id }}"
              }

  # Pattern 4: Event sourcing for audit trail
  - id: record_order_events
    type: io.kestra.plugin.core.flow.Sequential
    tasks:
      - id: record_order_created
        type: io.kestra.plugin.jdbc.postgresql.Query
        sql: |
          INSERT INTO order_events
          (event_id, order_id, event_type, event_data, occurred_at)
          VALUES (
            '{{ execution.id }}_created',
            '{{ execution.id }}',
            'ORDER_CREATED',
            '{{ inputs.order_data | tojson }}',
            NOW()
          )

      - id: record_order_processing
        type: io.kestra.plugin.jdbc.postgresql.Query
        sql: |
          INSERT INTO order_events
          (event_id, order_id, event_type, event_data, occurred_at)
          VALUES (
            '{{ execution.id }}_processing',
            '{{ execution.id }}',
            'ORDER_PROCESSING',
            '{{ outputs.execute_order_saga | tojson }}',
            NOW()
          )

  # Pattern 5: CQRS update for read models
  - id: update_read_models
    type: io.kestra.plugin.core.flow.Parallel
    wait: false  # Async updates
    tasks:
      - id: update_order_summary
        type: io.kestra.plugin.jdbc.postgresql.Query
        sql: |
          INSERT INTO order_summaries
          (order_id, customer_id, total_amount, status, items_count, created_at)
          VALUES (
            '{{ execution.id }}',
            '{{ inputs.customer_id }}',
            {{ inputs.order_data.total }},
            'processed',
            {{ inputs.order_data.items | length }},
            NOW()
          )

      - id: update_customer_metrics
        type: io.kestra.plugin.jdbc.postgresql.Query
        sql: |
          UPDATE customer_metrics
          SET 
            total_orders = total_orders + 1,
            total_spent = total_spent + {{ inputs.order_data.total }},
            last_order_at = NOW()
          WHERE customer_id = '{{ inputs.customer_id }}'

  # Pattern 6: Sidecar for monitoring
  - id: attach_monitoring_sidecar
    type: io.kestra.plugin.core.flow.Subflow
    namespace: acme.sidecars
    flowId: monitoring-sidecar
    wait: false

Best Practices for Advanced Patterns

1. Pattern Selection Guide

• For fault tolerance: Circuit Breaker, Bulkhead, Retry

• For distributed transactions: Saga, Event Sourcing

• For migration: Strangler Fig, Dual Write

• For scalability: CQRS, Sharding, Partitioning

• For observability: Sidecar, Ambassador

2. Testing Advanced Patterns

# Test circuit breaker
id: test-circuit-breaker
namespace: acme.tests

tasks:
  - id: simulate_failures
    type: io.kestra.plugin.core.flow.EachSequential
    value: "{{ range(1, 6) }}"
    tasks:
      - id: call_failing_service
        type: io.kestra.plugin.core.http.Request
        uri: "http://mock-service/fail"
        allowFailure: true

  - id: verify_circuit_tripped
    type: io.kestra.plugin.core.cache.Get
    key: "circuit_breaker:mock-service"

  - id: assert_circuit_open
    type: io.kestra.plugin.core.condition.Expression
    expression: '{{ outputs.verify_circuit_tripped.value.state == "open" }}'

3. Monitoring and Observability

id: pattern-metrics-collector
namespace: acme.monitoring

tasks:
  - id: collect_pattern_metrics
    type: io.kestra.plugin.scripts.python.Script
    script: |
      # Collect metrics for each pattern
      metrics = {
          'circuit_breaker': {
              'open_circuits': count_open_circuits(),
              'tripped_today': count_trips_today()
          },
          'saga': {
              'active_sagas': count_active_sagas(),
              'compensation_rate': calculate_compensation_rate()
          },
          'cqrs': {
              'read_lag_ms': calculate_read_lag(),
              'write_throughput': calculate_write_throughput()
          }
      }

4. Documentation Strategy

# Annotate patterns in flow metadata
labels:
  patterns: "circuit-breaker,saga,cqrs"
  complexity: "advanced"
  team: "platform-engineering"

documentation: |
  ## Applied Patterns

  ### Circuit Breaker
  - Prevents cascading failures from payment service
  - Config: 5 failures trips circuit, 5 minute timeout

  ### Saga Pattern  
  - Manages distributed order processing
  - Compensation: reverse payment, release inventory

  ### CQRS
  - Separate read/write models for orders
  - Eventual consistency: < 1 second lag

Common Anti-Patterns to Avoid

1. Over-Engineering

# BAD: Using saga for simple local transaction
# GOOD: Use simple sequential tasks for local operations

2. Pattern Misapplication

# BAD: Using event sourcing for simple CRUD
# GOOD: Event sourcing for audit-critical domains only

3. Ignoring Consistency Boundaries

# BAD: Distributed transaction across 10 microservices
# GOOD: Saga within bounded context, eventual consistency between

4. Neglecting Monitoring

# BAD: Complex patterns without observability
# GOOD: Comprehensive metrics for each pattern

Conclusion: Becoming a Workflow Architect

You've now explored the most powerful patterns in workflow orchestration. Remember:

1. Patterns are tools, not goals - Use them to solve specific problems

2. Start simple, add complexity gradually - Don't over-engineer from day one

3. Monitor everything - Complex patterns need comprehensive observability

4. Document decisions - Future you (and your team) will thank you

5. Test thoroughly - Edge cases multiply with complexity

Your Advanced Challenge:

Pick one complex business problem and solve it with multiple patterns:

1. E-commerce checkout with inventory, payment, and fulfillment

2. Data pipeline with validation, transformation, and quality checks

3. User onboarding with verification, provisioning, and welcome sequence

4. Report generation with data collection, processing, and distribution

5. System migration from monolith to microservices

Share your architecture in the comments!

In the next article, we'll explore Kestra's Plugin Ecosystem - how to extend Kestra with custom functionality and integrate with any system.

Before the next article, try:

1. Implement one new pattern in an existing workflow

2. Create a reusable pattern template for your team

3. Set up monitoring for pattern-specific metrics

4. Refactor a complex workflow using appropriate patterns

5. Document your pattern decisions and trade-offs

Remember: Mastery comes from practice, not just knowledge. Build, iterate, and learn from each implementation.

Happy orchestrating! 🚀

Introduction: Why ETL Still Matters in the Modern Data Stack

Remember when data engineering was "extract, transform, load"? Some say ETL is dead, replaced by ELT, reverse ETL, and data mesh. But here's the truth: ETL is more alive than ever, just smarter. And Kestra makes it accessible to everyone, not just senior data engineers.

In this article, you'll build a complete production-ready ETL pipeline that:

• Extracts data from multiple sources (API, CSV, database)

• Transforms with proper error handling and validation

• Loads to a data warehouse

• Monitors data quality

• Sends alerts on failures

• Scales to handle millions of records

The Business Problem: ACME Corp needs daily sales reports from multiple sources. Currently, marketing runs manual Excel reports, finance has SQL queries, and sales uses a legacy system. Your mission: automate everything into a single source of truth.

Phase 1: Blueprint Your ETL Pipeline

Understand Your Data Sources

Before writing a single line of YAML, document your sources:

Design Your Target Schema

-- Target: Data warehouse (Snowflake/BigQuery/PostgreSQL)
CREATE TABLE sales_fact (
    sale_id VARCHAR(100) PRIMARY KEY,
    transaction_date DATE,
    customer_id VARCHAR(50),
    product_id VARCHAR(50),
    quantity INTEGER,
    unit_price DECIMAL(10,2),
    total_amount DECIMAL(10,2),
    payment_method VARCHAR(50),
    region VARCHAR(100),
    sales_rep VARCHAR(100),
    source_system VARCHAR(50),
    loaded_at TIMESTAMP,
    batch_id VARCHAR(100)
);

CREATE TABLE customer_dim (
    customer_id VARCHAR(50) PRIMARY KEY,
    customer_name VARCHAR(255),
    email VARCHAR(255),
    join_date DATE,
    customer_tier VARCHAR(50),
    lifetime_value DECIMAL(10,2),
    last_updated TIMESTAMP
);

Define Success Criteria

Your pipeline must:

1. Process all data within 2 hours

2. Maintain 99.9% data accuracy

3. Handle missing data gracefully

4. Send alerts on failures within 15 minutes

5. Support backfills for 90 days

Phase 2: Setting Up Your Development Environment

Project Structure

# Create project structure
mkdir -p etl-pipeline/{flows,scripts,config,tests}
mkdir -p etl-pipeline/flows/{extract,transform,load,monitor}
mkdir -p etl-pipeline/scripts/{python,sql}
mkdir -p etl-pipeline/config/{dev,prod}
mkdir -p etl-pipeline/tests/{unit,integration}

# Initialize git
cd etl-pipeline
git init
echo "# ACME Sales ETL Pipeline" > README.md

Configuration Files

# config/dev/kestra.yml
kestra:
  server:
    basic-auth:
      enabled: true
      username: admin
      password: ${DEV_PASSWORD:?}

  repository:
    type: postgres
    postgres:
      host: localhost
      port: 5432
      database: kestra_dev

  storage:
    type: local
    local:
      base-path: ./storage

variables:
  environment: dev
  api_timeout: PT5M
  batch_size: 10000
  alert_email: dev-team@acme.com

secrets:
  stripe_api_key: ${STRIPE_API_KEY}
  shopify_token: ${SHOPIFY_TOKEN}
  db_password: ${DB_PASSWORD}

# config/prod/kestra.yml  
kestra:
  server:
    basic-auth:
      enabled: true
      username: ${ADMIN_USER}
      password: ${ADMIN_PASSWORD}

  repository:
    type: postgres
    postgres:
      host: prod-db.acme.com
      port: 5432
      database: kestra_prod

  storage:
    type: s3
    s3:
      bucket: acme-kestra-prod
      region: us-east-1

variables:
  environment: production
  api_timeout: PT10M
  batch_size: 50000
  alert_email: data-eng@acme.com
  slack_channel: "#data-alerts"

secrets:
  stripe_api_key: ${STRIPE_API_KEY}
  shopify_token: ${SHOPIFY_TOKEN}
  db_password: ${DB_PASSWORD_PROD}
  snowflake_password: ${SNOWFLAKE_PASSWORD}

Environment Variables

# .env.dev
DEV_PASSWORD=dev_password_123
STRIPE_API_KEY=sk_test_abc123
SHOPIFY_TOKEN=shpat_xyz789
DB_PASSWORD=postgres_pass
SNOWFLAKE_PASSWORD=snowflake_dev_pass

# .env.prod (keep secret!)
ADMIN_USER=admin
ADMIN_PASSWORD=super_secure_prod_pass_456
STRIPE_API_KEY=sk_live_xyz789
SHOPIFY_TOKEN=shpat_live_abc123
DB_PASSWORD_PROD=prod_super_secure_pass
SNOWFLAKE_PASSWORD=snowflake_prod_secure_pass

Docker Compose for Development

# docker-compose.yml
version: '3.8'

services:
  kestra:
    image: kestra/kestra:latest
    ports:
      - "8080:8080"
    environment:
      - KESTRA_CONFIGURATION=file:///app/config/dev/kestra.yml
    volumes:
      - ./flows:/app/flows
      - ./scripts:/app/scripts
      - ./config/dev:/app/config
      - ./storage:/app/storage
      - ./logs:/app/logs
    env_file:
      - .env.dev
    depends_on:
      - postgres
      - minio

  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: kestra_dev
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./init-db:/docker-entrypoint-initdb.d

  minio:
    image: minio/minio
    command: server /data --console-address ":9090"
    ports:
      - "9000:9000"
      - "9090:9090"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    volumes:
      - minio_data:/data

  # Optional: Add test data sources
  mock-stripe:
    image: stoplight/prism:4
    command: mock -h 0.0.0.0 https://raw.githubusercontent.com/stripe/openapi/master/openapi/spec3.json
    ports:
      - "4010:4010"

volumes:
  postgres_data:
  minio_data:

Phase 3: Building the Extract Layer

Source 1: Stripe Payments API

# flows/extract/stripe-payments.yml
id: extract-stripe-payments
namespace: acme.sales.etl
revision: 1
description: |
  Extract payment data from Stripe API.

  ## Data Characteristics
  - Daily volume: ~10,000 transactions
  - Historical data available for 90 days
  - Includes: charges, customers, refunds

  ## Rate Limits
  - 100 requests per second
  - Paginated responses (limit 100 per page)

labels:
  source: stripe
  data-type: transactions
  pii: true
  compliance: pci-dss

variables:
  stripe_base_url: "https://api.stripe.com/v1"
  default_limit: 100
  max_pages: 1000

inputs:
  - name: start_date
    type: DATETIME
    description: "Start date for data extraction"
    defaults: "{{ now() | dateAdd(-1, 'DAYS') | date('yyyy-MM-dd') }}"

  - name: end_date
    type: DATETIME
    description: "End date for data extraction"
    defaults: "{{ now() | date('yyyy-MM-dd') }}"

  - name: data_types
    type: ARRAY
    description: "Types of data to extract"
    defaults: ["charges", "customers", "refunds", "payment_intents"]

tasks:
  # Task 1: Validate inputs
  - id: validate_dates
    type: io.kestra.plugin.core.condition.Expression
    expression: >
      {{ inputs.start_date | date('yyyy-MM-dd') <= 
         inputs.end_date | date('yyyy-MM-dd') }}
    errorMessage: "Start date must be before or equal to end date"

  # Task 2: Calculate date range
  - id: generate_date_range
    type: io.kestra.plugin.core.debug.Return
    format: >
      {{ range(inputs.start_date | date('yyyy-MM-dd'), 
               inputs.end_date | date('yyyy-MM-dd'), 
               'P1D') | tojson }}

  # Task 3: Extract each data type in parallel
  - id: extract_data_types
    type: io.kestra.plugin.core.flow.EachParallel
    value: "{{ inputs.data_types }}"
    tasks:
      - id: "extract_{{ task.value }}"
        type: io.kestra.plugin.core.http.Request
        description: "Extract {{ task.value }} from Stripe"
        uri: "{{ vars.stripe_base_url }}/{{ task.value }}"
        method: GET
        headers:
          Authorization: "Bearer {{ secret('STRIPE_API_KEY') }}"
        queryParams:
          limit: "{{ vars.default_limit }}"
          created[gte]: "{{ inputs.start_date | date('timestamp') }}"
          created[lte]: "{{ inputs.end_date | date('timestamp') }}"

        # Pagination handling
        pagination:
          type: next_page
          results: body.data
          nextPageToken: body.has_more
          nextPageTokenPath: body.has_more
          nextPageCursorPath: body.data[-1].id
          nextPageQueryParams:
            starting_after: "{{ nextPageToken }}"

        # Rate limiting
        rateLimit:
          requestsPerSecond: 10

        # Error handling
        retry:
          type: exponential
          maxAttempt: 5
          delay: PT10S

        # Store results
        store: true
        outputFile: "stripe_{{ task.value }}_{{ execution.startDate | date('yyyyMMddHHmmss') }}.json"

  # Task 4: Merge and validate extracted data
  - id: merge_stripe_data
    type: io.kestra.plugin.scripts.python.Script
    description: "Merge and validate Stripe data"
    inputFiles:
      {% for data_type in inputs.data_types %}
      stripe_{{ data_type }}.json: "{{ outputs.extract_data_types.outputs['extract_' + data_type].uri }}"
      {% endfor %}
    script: |
      import json
      import pandas as pd
      from datetime import datetime

      # Load all extracted data
      all_data = {}
      for data_type in {{ inputs.data_types | tojson }}:
          with open(f'stripe_{data_type}.json', 'r') as f:
              data = json.load(f)
              all_data[data_type] = data['data']

      # Validate data
      validation_results = {
          'extraction_time': datetime.now().isoformat(),
          'total_records': sum(len(data) for data in all_data.values()),
          'record_counts': {k: len(v) for k, v in all_data.items()},
          'date_range': {
              'start': '{{ inputs.start_date }}',
              'end': '{{ inputs.end_date }}'
          }
      }

      # Check for missing required fields
      required_fields = {
          'charges': ['id', 'amount', 'currency', 'customer'],
          'customers': ['id', 'email', 'created'],
          'refunds': ['id', 'charge', 'amount']
      }

      validation_results['missing_fields'] = {}
      for data_type, records in all_data.items():
          if data_type in required_fields:
              for record in records[:10]:  # Sample check
                  missing = [field for field in required_fields[data_type] 
                            if field not in record]
                  if missing:
                      validation_results['missing_fields'].setdefault(data_type, []).append(missing)

      # Save merged data
      merged_data = {}
      for data_type, records in all_data.items():
          df = pd.DataFrame(records)
          df.to_parquet(f'stripe_{data_type}_merged.parquet', index=False)
          merged_data[data_type] = f'stripe_{data_type}_merged.parquet'

      # Save validation results
      with open('validation_report.json', 'w') as f:
          json.dump(validation_results, f, indent=2)

      # Output to Kestra
      print(json.dumps({
          'validation': validation_results,
          'files': merged_data
      }))

    taskRunner:
      type: io.kestra.plugin.core.runner.Process
      memory: 2Gi
      cpu: 1
      image: python:3.11-slim

    dependsOn:
      - extract_data_types

  # Task 5: Send extraction summary
  - id: send_extraction_summary
    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {
        "blocks": [
          {
            "type": "header",
            "text": {
              "type": "plain_text",
              "text": "📥 Stripe Data Extraction Complete"
            }
          },
          {
            "type": "section",
            "fields": [
              {
                "type": "mrkdwn",
                "text": "*Date Range:*\n{{ inputs.start_date }} to {{ inputs.end_date }}"
              },
              {
                "type": "mrkdwn",
                "text": "*Total Records:*\n{{ outputs.merge_stripe_data.vars.validation.total_records }}"
              }
            ]
          },
          {
            "type": "section",
            "text": {
              "type": "mrkdwn",
              "text": "{% for data_type, count in outputs.merge_stripe_data.vars.validation.record_counts.items() %}• *{{ data_type }}*: {{ count }} records\n{% endfor %}"
            }
          }
        ]
      }

    dependsOn:
      - merge_stripe_data

triggers:
  - id: daily_extraction
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 1 * * *"  # 1 AM daily
    timezone: UTC
    inputs:
      start_date: "{{ now() | dateAdd(-1, 'DAYS') | date('yyyy-MM-dd') }}"
      end_date: "{{ now() | date('yyyy-MM-dd') }}"

timeout: PT1H

Source 2: Shopify E-commerce Data

# flows/extract/shopify-orders.yml
id: extract-shopify-orders
namespace: acme.sales.etl
description: Extract order data from Shopify

tasks:
  - id: get_order_ids
    type: io.kestra.plugin.core.http.Request
    uri: "https://{{ secret('SHOPIFY_STORE') }}.myshopify.com/admin/api/2024-01/orders.json"
    method: GET
    headers:
      X-Shopify-Access-Token: "{{ secret('SHOPIFY_TOKEN') }}"
    queryParams:
      status: any
      created_at_min: "{{ execution.startDate | dateAdd(-1, 'DAYS') | date('yyyy-MM-dd') }}T00:00:00Z"
      limit: 250
    store: true
    pagination:
      type: header_link
      maxPage: 100

  - id: enrich_orders
    type: io.kestra.plugin.core.flow.EachParallel
    value: "{{ outputs.get_order_ids.body.orders | map('id') | list }}"
    maxParallel: 5
    tasks:
      - id: "get_order_{{ task.value }}"
        type: io.kestra.plugin.core.http.Request
        uri: "https://{{ secret('SHOPIFY_STORE') }}.myshopify.com/admin/api/2024-01/orders/{{ task.value }}.json"
        method: GET
        headers:
          X-Shopify-Access-Token: "{{ secret('SHOPIFY_TOKEN') }}"
        queryParams:
          fields: "id,email,created_at,financial_status,total_price,line_items,customer"
        store: true
        outputFile: "shopify_order_{{ task.value }}.json"

Source 3: Internal PostgreSQL Database

# flows/extract/internal-sales.yml
id: extract-internal-sales
namespace: acme.sales.etl
description: Extract sales data from internal database

tasks:
  - id: extract_sales_transactions
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: "jdbc:postgresql://{{ secret('DB_HOST') }}:5432/{{ secret('DB_NAME') }}"
    username: "{{ secret('DB_USER') }}"
    password: "{{ secret('DB_PASSWORD') }}"
    sql: |
      WITH daily_sales AS (
        SELECT 
          t.transaction_id,
          t.transaction_date,
          c.customer_id,
          c.customer_name,
          p.product_id,
          p.product_name,
          t.quantity,
          t.unit_price,
          t.total_amount,
          t.sales_rep_id,
          sr.region,
          'internal' as source_system,
          CURRENT_TIMESTAMP as extracted_at
        FROM transactions t
        JOIN customers c ON t.customer_id = c.customer_id
        JOIN products p ON t.product_id = p.product_id
        JOIN sales_reps sr ON t.sales_rep_id = sr.rep_id
        WHERE t.transaction_date >= '{{ execution.startDate | dateAdd(-1, 'DAYS') | date('yyyy-MM-dd') }}'
          AND t.transaction_date < '{{ execution.startDate | date('yyyy-MM-dd') }}'
          AND t.status = 'completed'
      )
      SELECT * FROM daily_sales
    fetch: true
    fetchSize: 10000
    chunkSize: 5000

  - id: save_to_parquet
    type: io.kestra.plugin.scripts.python.Script
    inputFiles:
      sales_data.json: |
        {{ outputs.extract_sales_transactions.rows | tojson }}
    script: |
      import pandas as pd
      import json

      data = json.loads('sales_data.json')
      df = pd.DataFrame(data)

      # Convert date columns
      date_columns = ['transaction_date', 'extracted_at']
      for col in date_columns:
          if col in df.columns:
              df[col] = pd.to_datetime(df[col])

      # Save to parquet
      df.to_parquet('internal_sales.parquet', index=False)
      print(f"Saved {len(df)} records to internal_sales.parquet")

    dependsOn:
      - extract_sales_transactions

Source 4: Google Sheets (Manual Input)

# flows/extract/google-sheets.yml
id: extract-google-sheets
namespace: acme.sales.etl
description: Extract manual inputs from Google Sheets

tasks:
  - id: authenticate_google
    type: io.kestra.plugin.gcp.Oauth
    scopes:
      - https://www.googleapis.com/auth/spreadsheets.readonly
    credentials: "{{ secret('GOOGLE_CREDENTIALS_JSON') }}"

  - id: extract_sheet_data
    type: io.kestra.plugin.gcp.sheets.Read
    spreadsheetId: "{{ secret('GOOGLE_SHEET_ID') }}"
    range: "SalesData!A:Z"
    store: true

  - id: convert_to_structured
    type: io.kestra.plugin.scripts.python.Script
    inputFiles:
      sheet_data.json: "{{ outputs.extract_sheet_data.uri }}"
    script: |
      import pandas as pd
      import json

      with open('sheet_data.json', 'r') as f:
          data = json.load(f)

      # Assuming first row is headers
      rows = data['values']
      if len(rows) < 2:
          raise ValueError("No data in sheet")

      headers = rows[0]
      data_rows = rows[1:]

      df = pd.DataFrame(data_rows, columns=headers)

      # Clean up data
      df = df.dropna(how='all')  # Remove empty rows

      # Standardize column names
      df.columns = df.columns.str.lower().str.replace(' ', '_')

      # Save
      df.to_parquet('manual_sales.parquet', index=False)
      print(f"Extracted {len(df)} manual entries")

Phase 4: Building the Transform Layer

Transformation Pipeline Architecture

# flows/transform/master-transformer.yml
id: master-transformer
namespace: acme.sales.etl
description: Orchestrate all data transformations

tasks:
  # Parallel transformation of all sources
  - id: transform_all_sources
    type: io.kestra.plugin.core.flow.Parallel
    tasks:
      - id: transform_stripe
        type: io.kestra.plugin.core.flow.Subflow
        namespace: acme.sales.etl
        flowId: transform-stripe-payments
        wait: true
        transmit:
          - name: input_files
            value: "{{ parent.outputs.extract_stripe_payments.outputs.merge_stripe_data.vars.files }}"

      - id: transform_shopify
        type: io.kestra.plugin.core.flow.Subflow
        namespace: acme.sales.etl
        flowId: transform-shopify-orders
        wait: true

      - id: transform_internal
        type: io.kestra.plugin.core.flow.Subflow
        namespace: acme.sales.etl  
        flowId: transform-internal-sales
        wait: true

      - id: transform_manual
        type: io.kestra.plugin.core.flow.Subflow
        namespace: acme.sales.etl
        flowId: transform-google-sheets
        wait: true

  # Deduplicate across sources
  - id: deduplicate_records
    type: io.kestra.plugin.core.flow.Subflow
    namespace: acme.sales.etl
    flowId: deduplicate-sales
    wait: true
    transmit:
      - name: stripe_data
        value: "{{ outputs.transform_all_sources.outputs.transform_stripe.outputs.transformed_files }}"
      - name: shopify_data
        value: "{{ outputs.transform_all_sources.outputs.transform_shopify.outputs.transformed_files }}"
      - name: internal_data
        value: "{{ outputs.transform_all_sources.outputs.transform_internal.outputs.transformed_files }}"
      - name: manual_data
        value: "{{ outputs.transform_all_sources.outputs.transform_manual.outputs.transformed_files }}"

  # Enrich with customer data
  - id: enrich_customer_data
    type: io.kestra.plugin.core.flow.Subflow
    namespace: acme.sales.etl
    flowId: enrich-customer-dimension
    wait: true
    transmit:
      - name: sales_data
        value: "{{ outputs.deduplicate_records.outputs.deduplicated_file }}"

  # Calculate metrics
  - id: calculate_metrics
    type: io.kestra.plugin.core.flow.Subflow
    namespace: acme.sales.etl
    flowId: calculate-sales-metrics
    wait: true
    transmit:
      - name: enriched_data
        value: "{{ outputs.enrich_customer_data.outputs.enriched_file }}"

  # Generate data quality report
  - id: data_quality_check
    type: io.kestra.plugin.core.flow.Subflow
    namespace: acme.data.quality
    flowId: run-data-quality-checks
    wait: true
    transmit:
      - name: transformed_data
        value: "{{ outputs.calculate_metrics.outputs.final_data }}"
      - name: validation_rules
        value: |
          {
            "rules": [
              {"field": "total_amount", "type": "not_null"},
              {"field": "customer_id", "type": "not_null"},
              {"field": "transaction_date", "type": "not_null"},
              {"field": "total_amount", "type": "positive"},
              {"field": "quantity", "type": "positive_integer"}
            ]
          }

Stripe Transformation Flow

# flows/transform/transform-stripe-payments.yml
id: transform-stripe-payments
namespace: acme.sales.etl
description: Transform raw Stripe data into standardized format

inputs:
  - name: input_files
    type: JSON
    description: "Map of data type to file path"

  - name: transformation_date
    type: DATETIME
    defaults: "{{ now() }}"

tasks:
  - id: load_and_transform
    type: io.kestra.plugin.scripts.python.Script
    description: "Transform Stripe payments data"
    inputFiles:
      {% for data_type, file_path in inputs.input_files.items() %}
      {{ data_type }}.parquet: "{{ file_path }}"
      {% endfor %}
      transformation_rules.py: |
        # Custom transformation rules
        def transform_stripe_charge(charge):
            """Transform a Stripe charge record"""
            return {
                'sale_id': f"stripe_{charge['id']}",
                'transaction_date': charge['created'],
                'customer_id': charge.get('customer'),
                'product_id': 'stripe_payment',
                'quantity': 1,
                'unit_price': charge['amount'] / 100,  # Convert from cents
                'total_amount': charge['amount'] / 100,
                'payment_method': charge.get('payment_method_details', {}).get('type', 'unknown'),
                'region': charge.get('billing_details', {}).get('address', {}).get('country'),
                'sales_rep': 'stripe',
                'source_system': 'stripe',
                'metadata': {
                    'currency': charge.get('currency'),
                    'description': charge.get('description'),
                    'status': charge.get('status')
                }
            }

        def transform_stripe_customer(customer):
            """Transform a Stripe customer record"""
            return {
                'customer_id': customer['id'],
                'customer_name': customer.get('name'),
                'email': customer.get('email'),
                'join_date': customer.get('created'),
                'customer_tier': 'standard',
                'lifetime_value': customer.get('metadata', {}).get('lifetime_value', 0),
                'metadata': {
                    'description': customer.get('description'),
                    'phone': customer.get('phone'),
                    'address': customer.get('address')
                }
            }
    script: |
      import pandas as pd
      import numpy as np
      from datetime import datetime
      import json
      from pathlib import Path

      # Import transformation rules
      import sys
      sys.path.append('.')
      from transformation_rules import transform_stripe_charge, transform_stripe_customer

      # Load data
      charges_df = pd.read_parquet('charges.parquet')
      customers_df = pd.read_parquet('customers.parquet')
      refunds_df = pd.read_parquet('refunds.parquet')

      # Transform charges
      print(f"Transforming {len(charges_df)} charges...")
      transformed_charges = []
      for _, charge in charges_df.iterrows():
          try:
              transformed = transform_stripe_charge(charge.to_dict())
              transformed_charges.append(transformed)
          except Exception as e:
              print(f"Error transforming charge {charge.get('id')}: {e}")

      # Transform customers
      print(f"Transforming {len(customers_df)} customers...")
      transformed_customers = []
      for _, customer in customers_df.iterrows():
          try:
              transformed = transform_stripe_customer(customer.to_dict())
              transformed_customers.append(transformed)
          except Exception as e:
              print(f"Error transforming customer {customer.get('id')}: {e}")

      # Handle refunds (adjust charges)
      if len(refunds_df) > 0:
          print(f"Processing {len(refunds_df)} refunds...")
          refunds_by_charge = refunds_df.groupby('charge')
          # This would adjust the charges data
          # Implementation depends on business rules

      # Convert to DataFrames
      sales_df = pd.DataFrame(transformed_charges)
      customers_df = pd.DataFrame(transformed_customers)

      # Data validation
      validation_errors = []

      # Check for nulls in required fields
      required_sales_fields = ['sale_id', 'transaction_date', 'total_amount']
      for field in required_sales_fields:
          null_count = sales_df[field].isnull().sum()
          if null_count > 0:
              validation_errors.append(f"Sales {field}: {null_count} null values")

      # Check for negative amounts
      negative_amounts = (sales_df['total_amount'] < 0).sum()
      if negative_amounts > 0:
          validation_errors.append(f"Sales: {negative_amounts} negative amounts")

      # Check for future dates
      today = pd.Timestamp.now()
      future_dates = (pd.to_datetime(sales_df['transaction_date']) > today).sum()
      if future_dates > 0:
          validation_errors.append(f"Sales: {future_dates} future dates")

      # Save validation report
      validation_report = {
          'transformation_date': datetime.now().isoformat(),
          'records_processed': {
              'sales': len(sales_df),
              'customers': len(customers_df)
          },
          'validation_errors': validation_errors,
          'error_rate': len(validation_errors) / (len(sales_df) + len(customers_df)) if (len(sales_df) + len(customers_df)) > 0 else 0
      }

      with open('validation_report.json', 'w') as f:
          json.dump(validation_report, f, indent=2)

      # Save transformed data
      sales_df.to_parquet('stripe_sales_transformed.parquet', index=False)
      customers_df.to_parquet('stripe_customers_transformed.parquet', index=False)

      # Output to Kestra
      output = {
          'transformed_files': {
              'sales': 'stripe_sales_transformed.parquet',
              'customers': 'stripe_customers_transformed.parquet'
          },
          'validation': validation_report,
          'summary': {
              'total_sales': len(sales_df),
              'total_customers': len(customers_df),
              'total_revenue': sales_df['total_amount'].sum() if len(sales_df) > 0 else 0
          }
      }

      print(json.dumps(output))

    taskRunner:
      type: io.kestra.plugin.core.runner.Process
      memory: 4Gi
      cpu: 2
      image: python:3.11-slim

  - id: handle_validation_errors
    type: io.kestra.plugin.core.flow.Switch
    value: "{{ outputs.load_and_transform.vars.validation.error_rate }}"
    cases:
      0:
        - id: log_success
          type: io.kestra.plugin.core.log.Log
          message: "Stripe transformation successful with 0 validation errors"

      0.01:
        - id: log_warning
          type: io.kestra.plugin.core.log.Log
          message: "Stripe transformation completed with {{ (outputs.load_and_transform.vars.validation.error_rate * 100) | round(2) }}% error rate"

      default:
        - id: alert_high_error_rate
          type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
          url: "{{ secret('SLACK_WEBHOOK_URL') }}"
          payload: |
            {
              "blocks": [
                {
                  "type": "header",
                  "text": {
                    "type": "plain_text",
                    "text": "⚠️ High Error Rate in Stripe Transformation"
                  }
                },
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "Error rate: *{{ (outputs.load_and_transform.vars.validation.error_rate * 100) | round(2) }}%*\n\nValidation errors:\n{% for error in outputs.load_and_transform.vars.validation.validation_errors %}- {{ error }}\n{% endfor %}"
                  }
                }
              ]
            }