# General Scientific Data Formats Reference

This reference covers general-purpose scientific data formats used across multiple disciplines.

## Numerical and Array Data

### .npy - NumPy Array
**Description:** Binary NumPy array format
**Typical Data:** N-dimensional arrays of any data type
**Use Cases:** Fast I/O for numerical data, intermediate results
**Python Libraries:**
- `numpy`: `np.load('file.npy')`, `np.save()`
- Memory-mapped access: `np.load('file.npy', mmap_mode='r')`
**EDA Approach:**
- Array shape and dimensionality
- Data type and precision
- Statistical summary (mean, std, min, max, percentiles)
- Missing or invalid values (NaN, inf)
- Memory footprint
- Value distribution and histogram
- Sparsity analysis
- Correlation structure (if 2D)

### .npz - Compressed NumPy Archive
**Description:** Multiple NumPy arrays in one file
**Typical Data:** Collections of related arrays
**Use Cases:** Saving multiple arrays together, compressed storage
**Python Libraries:**
- `numpy`: `np.load('file.npz')` returns dict-like object
- `np.savez()` or `np.savez_compressed()`
**EDA Approach:**
- List of contained arrays
- Individual array analysis
- Relationships between arrays
- Total file size and compression ratio
- Naming conventions
- Data consistency checks

### .csv - Comma-Separated Values
**Description:** Plain text tabular data
**Typical Data:** Experimental measurements, results tables
**Use Cases:** Universal data exchange, spreadsheet export
**Python Libraries:**
- `pandas`: `pd.read_csv('file.csv')`
- `csv`: Built-in module
- `polars`: High-performance CSV reading
- `numpy`: `np.loadtxt()` or `np.genfromtxt()`
**EDA Approach:**
- Row and column counts
- Data type inference
- Missing value patterns and frequency
- Column statistics (numeric: mean, std; categorical: frequencies)
- Outlier detection
- Correlation matrix
- Duplicate row detection
- Header and index validation
- Encoding issues detection

### .tsv / .tab - Tab-Separated Values
**Description:** Tab-delimited tabular data
**Typical Data:** Similar to CSV but tab-separated
**Use Cases:** Bioinformatics, text processing output
**Python Libraries:**
- `pandas`: `pd.read_csv('file.tsv', sep='\t')`
**EDA Approach:**
- Same as CSV format
- Tab vs space validation
- Quote handling

### .xlsx / .xls - Excel Spreadsheets
**Description:** Microsoft Excel binary/XML formats
**Typical Data:** Tabular data with formatting, formulas
**Use Cases:** Lab notebooks, data entry, reports
**Python Libraries:**
- `pandas`: `pd.read_excel('file.xlsx')`
- `openpyxl`: Full Excel file manipulation
- `xlrd`: Reading .xls (legacy)
**EDA Approach:**
- Sheet enumeration and names
- Per-sheet data analysis
- Formula evaluation
- Merged cells handling
- Hidden rows/columns
- Data validation rules
- Named ranges
- Formatting-only cells detection

### .json - JavaScript Object Notation
**Description:** Hierarchical text data format
**Typical Data:** Nested data structures, metadata
**Use Cases:** API responses, configuration, results
**Python Libraries:**
- `json`: Built-in module
- `pandas`: `pd.read_json()`
- `ujson`: Faster JSON parsing
**EDA Approach:**
- Schema inference
- Nesting depth
- Key-value distribution
- Array lengths
- Data type consistency
- Missing keys
- Duplicate detection
- Size and complexity metrics

### .xml - Extensible Markup Language
**Description:** Hierarchical markup format
**Typical Data:** Structured data with metadata
**Use Cases:** Standards-based data exchange, APIs
**Python Libraries:**
- `lxml`: `lxml.etree.parse()`
- `xml.etree.ElementTree`: Built-in XML
- `xmltodict`: Convert XML to dict
**EDA Approach:**
- Schema/DTD validation
- Element hierarchy and depth
- Namespace handling
- Attribute vs element content
- CDATA sections
- Text content extraction
- Sibling and child counts

### .yaml / .yml - YAML
**Description:** Human-readable data serialization
**Typical Data:** Configuration, metadata, parameters
**Use Cases:** Experiment configurations, pipelines
**Python Libraries:**
- `yaml`: `yaml.safe_load()` or `yaml.load()`
- `ruamel.yaml`: YAML 1.2 support
**EDA Approach:**
- Configuration structure
- Data type handling
- List and dict depth
- Anchor and alias usage
- Multi-document files
- Comments preservation
- Validation against schema

### .toml - TOML Configuration
**Description:** Configuration file format
**Typical Data:** Settings, parameters
**Use Cases:** Python package configuration, settings
**Python Libraries:**
- `tomli` / `tomllib`: TOML reading (tomllib in Python 3.11+)
- `toml`: Reading and writing
**EDA Approach:**
- Section structure
- Key-value pairs
- Data type inference
- Nested table validation
- Required vs optional fields

### .ini - INI Configuration
**Description:** Simple configuration format
**Typical Data:** Application settings
**Use Cases:** Legacy configurations, simple settings
**Python Libraries:**
- `configparser`: Built-in INI parser
**EDA Approach:**
- Section enumeration
- Key-value extraction
- Type conversion
- Comment handling
- Case sensitivity

## Binary and Compressed Data

### .hdf5 / .h5 - Hierarchical Data Format 5
**Description:** Container for large scientific datasets
**Typical Data:** Multi-dimensional arrays, metadata, groups
**Use Cases:** Large datasets, multi-modal data, parallel I/O
**Python Libraries:**
- `h5py`: `h5py.File('file.h5', 'r')`
- `pytables`: Advanced HDF5 interface
- `pandas`: HDF5 storage via HDFStore
**EDA Approach:**
- Group and dataset hierarchy
- Dataset shapes and dtypes
- Attributes and metadata
- Compression and chunking strategy
- Memory-efficient sampling
- Dataset relationships
- File size and efficiency
- Access patterns optimization

### .zarr - Chunked Array Storage
**Description:** Cloud-optimized chunked arrays
**Typical Data:** Large N-dimensional arrays
**Use Cases:** Cloud storage, parallel computing, streaming
**Python Libraries:**
- `zarr`: `zarr.open('file.zarr')`
- `xarray`: Zarr backend support
**EDA Approach:**
- Array metadata and dimensions
- Chunk size optimization
- Compression codec and ratio
- Synchronizer and store type
- Multi-scale hierarchies
- Parallel access performance
- Attribute metadata

### .gz / .gzip - Gzip Compressed
**Description:** Compressed data files
**Typical Data:** Any compressed text or binary
**Use Cases:** Compression for storage/transfer
**Python Libraries:**
- `gzip`: Built-in gzip module
- `pandas`: Automatic gzip handling in read functions
**EDA Approach:**
- Compression ratio
- Original file type detection
- Decompression validation
- Header information
- Multi-member archives

### .bz2 - Bzip2 Compressed
**Description:** Bzip2 compression
**Typical Data:** Highly compressed files
**Use Cases:** Better compression than gzip
**Python Libraries:**
- `bz2`: Built-in bz2 module
- Automatic handling in pandas
**EDA Approach:**
- Compression efficiency
- Decompression time
- Content validation

### .zip - ZIP Archive
**Description:** Archive with multiple files
**Typical Data:** Collections of files
**Use Cases:** File distribution, archiving
**Python Libraries:**
- `zipfile`: Built-in ZIP support
- `pandas`: Can read zipped CSVs
**EDA Approach:**
- Archive member listing
- Compression method per file
- Total vs compressed size
- Directory structure
- File type distribution
- Extraction validation

### .tar / .tar.gz - TAR Archive
**Description:** Unix tape archive
**Typical Data:** Multiple files and directories
**Use Cases:** Software distribution, backups
**Python Libraries:**
- `tarfile`: Built-in TAR support
**EDA Approach:**
- Member file listing
- Compression (if .tar.gz, .tar.bz2)
- Directory structure
- Permissions preservation
- Extraction testing

## Time Series and Waveform Data

### .wav - Waveform Audio
**Description:** Audio waveform data
**Typical Data:** Acoustic signals, audio recordings
**Use Cases:** Acoustic analysis, ultrasound, signal processing
**Python Libraries:**
- `scipy.io.wavfile`: `scipy.io.wavfile.read()`
- `wave`: Built-in module
- `soundfile`: Enhanced audio I/O
**EDA Approach:**
- Sample rate and duration
- Bit depth and channels
- Amplitude distribution
- Spectral analysis (FFT)
- Signal-to-noise ratio
- Clipping detection
- Frequency content

### .mat - MATLAB Data
**Description:** MATLAB workspace variables
**Typical Data:** Arrays, structures, cells
**Use Cases:** MATLAB-Python interoperability
**Python Libraries:**
- `scipy.io`: `scipy.io.loadmat()`
- `h5py`: For MATLAB v7.3 files (HDF5-based)
- `mat73`: Pure Python for v7.3
**EDA Approach:**
- Variable names and types
- Array dimensions
- Structure field exploration
- Cell array handling
- Sparse matrix detection
- MATLAB version compatibility
- Metadata extraction

### .edf - European Data Format
**Description:** Time series data (especially medical)
**Typical Data:** EEG, physiological signals
**Use Cases:** Medical signal storage
**Python Libraries:**
- `pyedflib`: EDF/EDF+ reading and writing
- `mne`: Neurophysiology data (supports EDF)
**EDA Approach:**
- Signal count and names
- Sampling frequencies
- Signal ranges and units
- Recording duration
- Annotation events
- Data quality (saturation, noise)
- Patient/study information

### .csv (Time Series)
**Description:** CSV with timestamp column
**Typical Data:** Time-indexed measurements
**Use Cases:** Sensor data, monitoring, experiments
**Python Libraries:**
- `pandas`: `pd.read_csv()` with `parse_dates`
**EDA Approach:**
- Temporal range and resolution
- Sampling regularity
- Missing time points
- Trend and seasonality
- Stationarity tests
- Autocorrelation
- Anomaly detection

## Geospatial and Environmental Data

### .shp - Shapefile
**Description:** Geospatial vector data
**Typical Data:** Geographic features (points, lines, polygons)
**Use Cases:** GIS analysis, spatial data
**Python Libraries:**
- `geopandas`: `gpd.read_file('file.shp')`
- `fiona`: Lower-level shapefile access
- `pyshp`: Pure Python shapefile reader
**EDA Approach:**
- Geometry type and count
- Coordinate reference system
- Bounding box
- Attribute table analysis
- Geometry validity
- Spatial distribution
- Multi-part features
- Associated files (.shx, .dbf, .prj)

### .geojson - GeoJSON
**Description:** JSON format for geographic data
**Typical Data:** Features with geometry and properties
**Use Cases:** Web mapping, spatial analysis
**Python Libraries:**
- `geopandas`: Native GeoJSON support
- `json`: Parse as JSON then process
**EDA Approach:**
- Feature count and types
- CRS specification
- Bounding box calculation
- Property schema
- Geometry complexity
- Nesting structure

### .tif / .tiff (Geospatial)
**Description:** GeoTIFF with spatial reference
**Typical Data:** Satellite imagery, DEMs, rasters
**Use Cases:** Remote sensing, terrain analysis
**Python Libraries:**
- `rasterio`: `rasterio.open('file.tif')`
- `gdal`: Geospatial Data Abstraction Library
- `xarray` with `rioxarray`: N-D geospatial arrays
**EDA Approach:**
- Raster dimensions and resolution
- Band count and descriptions
- Coordinate reference system
- Geotransform parameters
- NoData value handling
- Pixel value distribution
- Histogram analysis
- Overviews and pyramids

### .nc / .netcdf - Network Common Data Form
**Description:** Self-describing array-based data
**Typical Data:** Climate, atmospheric, oceanographic data
**Use Cases:** Scientific datasets, model output
**Python Libraries:**
- `netCDF4`: `netCDF4.Dataset('file.nc')`
- `xarray`: `xr.open_dataset('file.nc')`
**EDA Approach:**
- Variable enumeration
- Dimension analysis
- Time series properties
- Spatial coverage
- Attribute metadata (CF conventions)
- Coordinate systems
- Chunking and compression
- Data quality flags

### .grib / .grib2 - Gridded Binary
**Description:** Meteorological data format
**Typical Data:** Weather forecasts, climate data
**Use Cases:** Numerical weather prediction
**Python Libraries:**
- `pygrib`: GRIB file reading
- `xarray` with `cfgrib`: GRIB to xarray
**EDA Approach:**
- Message inventory
- Parameter and level types
- Spatial grid specification
- Temporal coverage
- Ensemble members
- Forecast vs analysis
- Data packing and precision

### .hdf4 - HDF4 Format
**Description:** Older HDF format
**Typical Data:** NASA Earth Science data
**Use Cases:** Satellite data (MODIS, etc.)
**Python Libraries:**
- `pyhdf`: HDF4 access
- `gdal`: Can read HDF4
**EDA Approach:**
- Scientific dataset listing
- Vdata and attributes
- Dimension scales
- Metadata extraction
- Quality flags
- Conversion to HDF5 or NetCDF

## Specialized Scientific Formats

### .fits - Flexible Image Transport System
**Description:** Astronomy data format
**Typical Data:** Images, tables, spectra from telescopes
**Use Cases:** Astronomical observations
**Python Libraries:**
- `astropy.io.fits`: `fits.open('file.fits')`
- `fitsio`: Alternative FITS library
**EDA Approach:**
- HDU (Header Data Unit) structure
- Image dimensions and WCS
- Header keyword analysis
- Table column descriptions
- Data type and scaling
- FITS convention compliance
- Checksum validation

### .asdf - Advanced Scientific Data Format
**Description:** Next-gen data format for astronomy
**Typical Data:** Complex hierarchical scientific data
**Use Cases:** James Webb Space Telescope data
**Python Libraries:**
- `asdf`: `asdf.open('file.asdf')`
**EDA Approach:**
- Tree structure exploration
- Schema validation
- Internal vs external arrays
- Compression methods
- YAML metadata
- Version compatibility

### .root - ROOT Data Format
**Description:** CERN ROOT framework format
**Typical Data:** High-energy physics data
**Use Cases:** Particle physics experiments
**Python Libraries:**
- `uproot`: Pure Python ROOT reading
- `ROOT`: Official PyROOT bindings
**EDA Approach:**
- TTree structure
- Branch types and entries
- Histogram inventory
- Event loop statistics
- File compression
- Split level analysis

### .txt - Plain Text Data
**Description:** Generic text-based data
**Typical Data:** Tab/space-delimited, custom formats
**Use Cases:** Simple data exchange, logs
**Python Libraries:**
- `pandas`: `pd.read_csv()` with custom delimiters
- `numpy`: `np.loadtxt()`, `np.genfromtxt()`
- Built-in file reading
**EDA Approach:**
- Format detection (delimiter, header)
- Data type inference
- Comment line handling
- Missing value codes
- Column alignment
- Encoding detection

### .dat - Generic Data File
**Description:** Binary or text data
**Typical Data:** Instrument output, custom formats
**Use Cases:** Various scientific instruments
**Python Libraries:**
- Format-specific: requires knowledge of structure
- `numpy`: `np.fromfile()` for binary
- `struct`: Parse binary structures
**EDA Approach:**
- Binary vs text determination
- Header detection
- Record structure inference
- Endianness
- Data type patterns
- Validation with documentation

### .log - Log Files
**Description:** Text logs from software/instruments
**Typical Data:** Timestamped events, messages
**Use Cases:** Troubleshooting, experiment tracking
**Python Libraries:**
- Built-in file reading
- `pandas`: Structured log parsing
- Regular expressions for parsing
**EDA Approach:**
- Log level distribution
- Timestamp parsing
- Error and warning frequency
- Event sequencing
- Pattern recognition
- Anomaly detection
- Session boundaries