gh-secondsky-sap-skills-skills-sap-hana-cloud-data-intelligence/references/operators-reference.md

SAP Data Intelligence Operators Reference

Complete reference for built-in operators in SAP Data Intelligence.

Table of Contents

1. Operator Generations
2. ABAP Operators
3. File and Storage Operators
4. Database Operators
5. Messaging Operators
6. Script Operators
7. Data Processing Operators
8. Machine Learning Operators
9. Integration Operators
10. Workflow Operators

---

Operator Generations

Generation 1 (Gen1)

Legacy operators with broad compatibility.

Characteristics:

• Process-based execution
• Manual error handling
• Suitable for simpler workflows
• Compatible with older graphs

Generation 2 (Gen2)

Enhanced operators with advanced features.

Characteristics:

• Improved error recovery
• State management with snapshots
• Native multiplexing support
• Better performance characteristics

Gen2 Exclusive Features:

• Automatic graph recovery from failures
• Periodic state checkpoints
• Structured native message streaming
• Enhanced Python3 operator

Critical Rule: Gen1 and Gen2 operators cannot be mixed in the same graph.

---

ABAP Operators

Operators for integrating ABAP-based SAP systems.

ABAP CDS Reader

Reads data from ABAP CDS views.

Configuration:

• Connection: ABAP system connection
• CDS View: Target view name
• Selection: Filter criteria
• Package Size: Batch size for reading

Ports:

• Output: CDS view data

ABAP Table Reader

Reads data from ABAP tables.

Configuration:

• Connection: ABAP system connection
• Table: Target table name
• Fields: Column selection
• Where Clause: Filter condition

SLT Connector

Connects to SAP Landscape Transformation for real-time replication.

Configuration:

• Mass Transfer ID: SLT configuration
• Table: Source table
• Initial Load: Enable/disable full load
• Delta Load: Enable/disable CDC

ODP Consumer

Consumes Operational Data Provisioning sources.

Configuration:

• Connection: ABAP connection
• ODP Context: Extraction context (SAPI, ABAP CDS, etc.)
• ODP Name: Data provider name
• Extraction Mode: Full or Delta

---

File and Storage Operators

Binary File Consumer

Reads binary files from storage.

Configuration:

• Connection: Storage connection
• Path: File path pattern
• Recursive: Include subdirectories

Output: Binary content

Binary File Producer

Writes binary content to files.

Configuration:

• Connection: Storage connection
• Path: Output file path
• Mode: Overwrite, Append, or Fail if exists

Structured File Consumer

Reads structured data from files (CSV, Parquet, ORC, JSON).

Configuration:

• Connection: Storage connection
• Source: File path or pattern
• Format: CSV, Parquet, ORC, JSON
• Schema: Column definitions

Structured File Producer

Writes structured data to files.

Configuration:

• Connection: Storage connection
• Target: Output path
• Format: CSV, Parquet, ORC, JSON
• Partition: Partitioning strategy

Cloud Storage Operators

Amazon S3:

• S3 Consumer: Read from S3 buckets
• S3 Producer: Write to S3 buckets

Azure Blob/ADLS:

• Azure Blob Consumer/Producer
• ADLS Gen2 Consumer/Producer

Google Cloud Storage:

• GCS Consumer: Read from GCS buckets
• GCS Producer: Write to GCS buckets

HDFS:

• HDFS Consumer: Read from Hadoop clusters
• HDFS Producer: Write to Hadoop clusters

---

Database Operators

SAP HANA Operators

HANA Client:

• Executes SQL statements
• Supports DDL, DML, queries

Table Consumer:

• Reads from HANA tables
• Supports filtering and projection

Table Producer:

• Writes to HANA tables
• Supports INSERT, UPSERT, DELETE

Flowgraph Executor:

• Runs HANA calculation views
• Executes stored procedures

SQL Operators

SQL Consumer:

• Executes SELECT queries
• Supports parameterized queries

SQL Executor:

• Runs DDL/DML statements
• Returns affected row count

Supported Databases

• SAP HANA (Cloud and on-premise)
• SAP BW/4HANA
• Microsoft SQL Server
• Oracle Database
• PostgreSQL
• MySQL
• SAP IQ/Sybase

---

Messaging Operators

Kafka Operators

Kafka Consumer:

• Subscribes to Kafka topics
• Supports consumer groups
• Offset management (earliest, latest, committed)

Kafka Producer:

• Publishes to Kafka topics
• Key/value serialization
• Partitioning strategies

MQTT Operators

MQTT Consumer:

• Subscribes to MQTT topics
• QoS level configuration

MQTT Producer:

• Publishes MQTT messages
• Retain flag support

Additional Messaging

• NATS: Lightweight messaging
• WAMP: Web Application Messaging Protocol
• AWS SNS: Amazon Simple Notification Service
• SAP Event Mesh: SAP cloud messaging

---

Script Operators

Python Operator (Gen2)

Execute Python code within graphs.

Configuration:

• Script: Python code
• Codelanguage: python36 or python39

Example:

def on_input(msg_id, header, body):
    # Process input
    result = transform(body)
    api.send("output", api.Message(result))

api.set_port_callback("input", on_input)

JavaScript Operator

Execute JavaScript/Node.js code.

Configuration:

• Script: JavaScript code

Example:

$.setPortCallback("input", function(ctx, s) {
    var result = process(s);
    $.output(result);
});

R Operator

Execute R scripts for statistical analysis.

Configuration:

• Script: R code
• Libraries: Required R packages

Go Operator

Execute Go code for high-performance processing.

---

Data Processing Operators

Data Transform

Visual SQL-like transformation editor.

Nodes:

• Projection: Column selection/transformation
• Aggregation: GROUP BY with functions
• Join: Combine datasets
• Union: Merge datasets
• Case: Conditional logic
• Filter: Row filtering

Data Quality Operators

Validation Rule:

• Define data quality rules
• Generate validation reports

Anonymization:

• Mask sensitive data
• Hash, shuffle, or generalize

Data Mask:

• Apply masking patterns
• Preserve format while anonymizing

Conversion Operators

Type Converters:

• Binary to Table
• Table to Binary
• Dynamic to Static
• Static to Dynamic

Format Converters:

• JSON Parser/Formatter
• CSV Parser/Formatter
• Avro Encoder/Decoder

---

Machine Learning Operators

TensorFlow Operators

TensorFlow Training:

• Train TensorFlow models
• Distributed training support

TensorFlow Serving:

• Deploy TensorFlow models
• REST API inference

PyTorch Operators

PyTorch Training:

• Train PyTorch models
• GPU acceleration

HANA ML Operators

HANA ML Training:

• Train models in HANA
• Automated ML (AutoML)

HANA ML Inference:

• Score data with HANA ML models

Metrics Operators

Submit Metrics:

• Track training metrics
• Integration with Metrics Explorer

---

Integration Operators

OData Operators

OData Consumer:

• Query OData services
• Supports v2 and v4

OData Producer:

• Expose data as OData
• CRUD operations

REST API Operators

REST Client:

• Call REST APIs
• Configurable HTTP methods
• Header/body templates

OpenAPI Client:

• Generate clients from OpenAPI specs
• Automatic request/response handling

SAP Integration

SAP CPI Operator:

• Trigger SAP Cloud Platform Integration flows

SAP Application Consumer/Producer:

• Connect to SAP applications
• S/4HANA, ECC, SuccessFactors

---

Workflow Operators

Data Workflow Operators

Workflow Trigger:

• Start workflow execution
• Scheduled or event-based

Workflow Terminator:

• End workflow with status

Pipeline Executor:

• Run child pipelines
• Pass parameters

Control Flow

BW Process Chain:

• Execute SAP BW process chains

Data Services Job:

• Run SAP Data Services jobs

HANA Flowgraph:

• Execute HANA calculation views

Notification

Email Notification:

• Send status emails
• Configurable templates

---

Documentation Links

• Operators Reference: https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/repositoryobjects/data-intelligence-operators
• Graphs Reference: https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/repositoryobjects/data-intelligence-graphs

---

Last Updated: 2025-11-22