---
name: project-architecture
description: Detailed architecture, data flow, pipeline execution, dependencies, and system design for the Unify data migration project. Use when you need deep understanding of how components interact.
---

# Project Architecture

Comprehensive architecture documentation for the Unify data migration project.

## Medallion Architecture Deep Dive

### Bronze Layer
**Purpose**: Raw data ingestion from parquet files
**Location**: `python_files/pipeline_operations/bronze_layer_deployment.py`
**Process**:
1. Lists parquet files from Azure ADLS Gen2 or local storage
2. Creates bronze databases: `bronze_cms`, `bronze_fvms`, `bronze_nicherms`
3. Reads parquet files and applies basic transformations
4. Adds versioning, row hashes, and data source columns

### Silver Layer
**Purpose**: Validated, standardized data organized by source
**Location**: `python_files/silver/` (cms, fvms, nicherms subdirectories)
**Process**:
1. Drops and recreates silver databases
2. Recursively finds all Python files in `python_files/silver/`
3. Executes each silver transformation file in sorted order
4. Uses threading for parallel execution (currently commented out)

### Gold Layer
**Purpose**: Business-ready, aggregated analytical datasets
**Location**: `python_files/gold/`
**Process**:
1. Creates business-ready analytical tables in `gold_data_model` database
2. Executes transformations from `python_files/gold/`
3. Aggregates and joins data across multiple silver tables

## Data Sources

### FVMS (Family Violence Management System)
- **Tables**: 32 tables
- **Key tables**: incident, person, address, risk_assessment
- **Purpose**: Family violence incident tracking and management

### CMS (Case Management System)
- **Tables**: 19 tables
- **Key tables**: offence_report, case_file, person, victim
- **Purpose**: Criminal offence investigation and case management

### NicheRMS (Records Management System)
- **Tables**: 39 TBL_* tables
- **Purpose**: Legacy records management system

## Azure Integration

### Storage (ADLS Gen2)
- **Containers**: `bronze-layer`, `code-layer`, `legacy_ingestion`
- **Authentication**: Managed Identity (`AZURE_MANAGED_IDENTITY_CLIENT_ID`)
- **Path Pattern**: `abfss://container@account.dfs.core.windows.net/path`

### Key Services
- **Key Vault**: `AuE-DataMig-Dev-KV` for secret management
- **Synapse Workspace**: `auedatamigdevsynws`
- **Spark Pool**: `dm8c64gb`

## Environment Detection Pattern

All processing scripts auto-detect their runtime environment:

```python
if "/home/trusted-service-user" == env_vars["HOME"]:
    # Azure Synapse Analytics production environment
    import notebookutils.mssparkutils as mssparkutils
    spark = SparkOptimiser.get_optimised_spark_session()
    DATA_PATH_STRING = "abfss://code-layer@auedatamigdevlake.dfs.core.windows.net"
else:
    # Local development environment using Docker Spark container
    from python_files.utilities.local_spark_connection import sparkConnector
    config = UtilityFunctions.get_settings_from_yaml("configuration.yaml")
    connector = sparkConnector(...)
    DATA_PATH_STRING = config["DATA_PATH_STRING"]
```

## Core Utilities Architecture

### SparkOptimiser
- Configured Spark session with optimized settings
- Handles driver memory, encryption, authentication
- Centralized session management

### NotebookLogger
- Rich console logging with fallback to standard print
- Structured logging (info, warning, error, success)
- Graceful degradation when Rich library unavailable

### TableUtilities
- DataFrame operations (deduplication, hashing, timestamp conversion)
- `add_row_hash()`: Change detection
- `save_as_table()`: Standard table save with timestamp conversion
- `clean_date_time_columns()`: Intelligent timestamp parsing
- `drop_duplicates_simple/advanced()`: Deduplication strategies
- `filter_and_drop_column()`: Remove duplicate flags

### DAGMonitor
- Pipeline execution tracking and reporting
- Performance metrics and logging

## Configuration Management

### configuration.yaml
Central YAML configuration includes:
- **Data Sources**: FVMS, CMS, NicheRMS table lists (`*_IN_SCOPE` variables)
- **Azure Settings**: Storage accounts, Key Vault, Synapse workspace, subscription IDs
- **Spark Settings**: Driver, encryption, authentication scheme
- **Data Paths**: Local (`/workspaces/data`) vs Azure (`abfss://`)
- **Logging**: LOG_LEVEL, LOG_ROTATION, LOG_RETENTION
- **Nulls Handling**: STRING_NULL_REPLACEMENT, NUMERIC_NULL_REPLACEMENT, TIMESTAMP_NULL_REPLACEMENT

## Error Handling Strategy

- **Decorator-Based**: `@synapse_error_print_handler` for consistent error handling
- **Loguru Integration**: Structured logging with proper levels
- **Graceful Degradation**: Handle missing dependencies (Rich library fallback)
- **Context Information**: Include table/database names in all log messages

## Local Data Filtering

`TableUtilities.save_as_table()` automatically filters to last N years when `date_created` column exists, controlled by `NUMBER_OF_YEARS` global variable in `session_optimiser.py`. Prevents full dataset processing in local development.

## Testing Architecture

### Test Structure
- `python_files/testing/`: Unit and integration tests
- `medallion_testing.py`: Full pipeline validation
- `bronze_layer_validation.py`: Bronze layer tests
- `ingestion_layer_validation.py`: Ingestion tests

### Testing Strategy
- pytest integration with PySpark environments
- Quality gates: syntax validation and linting before completion
- Integration tests for full medallion flow

## DuckDB Integration

After running pipelines, build local DuckDB database for fast SQL analysis:
- **File**: `/workspaces/data/warehouse.duckdb`
- **Command**: `make build_duckdb`
- **Purpose**: Fast local queries without Azure connection
- **Contains**: All bronze, silver, gold layer tables

## Recent Architectural Changes

### Path Migration
- Standardized all paths to use `unify_2_1_dm_synapse_env_d10`
- Improved portability and environment consistency
- 12 files updated across utilities, notebooks, configurations

### Code Cleanup
- Removed unused utilities: `file_executor.py`, `file_finder.py`
- Reduced codebase complexity
- Regular cleanup pattern for maintainability