gh-k-dense-ai-claude-scientific-skills-scientific-skills/skills/exploratory-data-analysis/references/general_scientific_formats.md

General Scientific Data Formats Reference

This reference covers general-purpose scientific data formats used across multiple disciplines.

Numerical and Array Data

.npy - NumPy Array

Description: Binary NumPy array format Typical Data: N-dimensional arrays of any data type Use Cases: Fast I/O for numerical data, intermediate results Python Libraries:

• numpy: np.load('file.npy'), np.save()
• Memory-mapped access: np.load('file.npy', mmap_mode='r') EDA Approach:
• Array shape and dimensionality
• Data type and precision
• Statistical summary (mean, std, min, max, percentiles)
• Missing or invalid values (NaN, inf)
• Memory footprint
• Value distribution and histogram
• Sparsity analysis
• Correlation structure (if 2D)

.npz - Compressed NumPy Archive

Description: Multiple NumPy arrays in one file Typical Data: Collections of related arrays Use Cases: Saving multiple arrays together, compressed storage Python Libraries:

• numpy: np.load('file.npz') returns dict-like object
• np.savez() or np.savez_compressed() EDA Approach:
• List of contained arrays
• Individual array analysis
• Relationships between arrays
• Total file size and compression ratio
• Naming conventions
• Data consistency checks

.csv - Comma-Separated Values

Description: Plain text tabular data Typical Data: Experimental measurements, results tables Use Cases: Universal data exchange, spreadsheet export Python Libraries:

• pandas: pd.read_csv('file.csv')
• csv: Built-in module
• polars: High-performance CSV reading
• numpy: np.loadtxt() or np.genfromtxt() EDA Approach:
• Row and column counts
• Data type inference
• Missing value patterns and frequency
• Column statistics (numeric: mean, std; categorical: frequencies)
• Outlier detection
• Correlation matrix
• Duplicate row detection
• Header and index validation
• Encoding issues detection

.tsv / .tab - Tab-Separated Values

Description: Tab-delimited tabular data Typical Data: Similar to CSV but tab-separated Use Cases: Bioinformatics, text processing output Python Libraries:

• pandas: pd.read_csv('file.tsv', sep='\t') EDA Approach:
• Same as CSV format
• Tab vs space validation
• Quote handling

.xlsx / .xls - Excel Spreadsheets

Description: Microsoft Excel binary/XML formats Typical Data: Tabular data with formatting, formulas Use Cases: Lab notebooks, data entry, reports Python Libraries:

• pandas: pd.read_excel('file.xlsx')
• openpyxl: Full Excel file manipulation
• xlrd: Reading .xls (legacy) EDA Approach:
• Sheet enumeration and names
• Per-sheet data analysis
• Formula evaluation
• Merged cells handling
• Hidden rows/columns
• Data validation rules
• Named ranges
• Formatting-only cells detection

.json - JavaScript Object Notation

Description: Hierarchical text data format Typical Data: Nested data structures, metadata Use Cases: API responses, configuration, results Python Libraries:

• json: Built-in module
• pandas: pd.read_json()
• ujson: Faster JSON parsing EDA Approach:
• Schema inference
• Nesting depth
• Key-value distribution
• Array lengths
• Data type consistency
• Missing keys
• Duplicate detection
• Size and complexity metrics

.xml - Extensible Markup Language

Description: Hierarchical markup format Typical Data: Structured data with metadata Use Cases: Standards-based data exchange, APIs Python Libraries:

• lxml: lxml.etree.parse()
• xml.etree.ElementTree: Built-in XML
• xmltodict: Convert XML to dict EDA Approach:
• Schema/DTD validation
• Element hierarchy and depth
• Namespace handling
• Attribute vs element content
• CDATA sections
• Text content extraction
• Sibling and child counts

.yaml / .yml - YAML

Description: Human-readable data serialization Typical Data: Configuration, metadata, parameters Use Cases: Experiment configurations, pipelines Python Libraries:

• yaml: yaml.safe_load() or yaml.load()
• ruamel.yaml: YAML 1.2 support EDA Approach:
• Configuration structure
• Data type handling
• List and dict depth
• Anchor and alias usage
• Multi-document files
• Comments preservation
• Validation against schema

.toml - TOML Configuration

Description: Configuration file format Typical Data: Settings, parameters Use Cases: Python package configuration, settings Python Libraries:

• tomli / tomllib: TOML reading (tomllib in Python 3.11+)
• toml: Reading and writing EDA Approach:
• Section structure
• Key-value pairs
• Data type inference
• Nested table validation
• Required vs optional fields

.ini - INI Configuration

Description: Simple configuration format Typical Data: Application settings Use Cases: Legacy configurations, simple settings Python Libraries:

• configparser: Built-in INI parser EDA Approach:
• Section enumeration
• Key-value extraction
• Type conversion
• Comment handling
• Case sensitivity

Binary and Compressed Data

.hdf5 / .h5 - Hierarchical Data Format 5

Description: Container for large scientific datasets Typical Data: Multi-dimensional arrays, metadata, groups Use Cases: Large datasets, multi-modal data, parallel I/O Python Libraries:

• h5py: h5py.File('file.h5', 'r')
• pytables: Advanced HDF5 interface
• pandas: HDF5 storage via HDFStore EDA Approach:
• Group and dataset hierarchy
• Dataset shapes and dtypes
• Attributes and metadata
• Compression and chunking strategy
• Memory-efficient sampling
• Dataset relationships
• File size and efficiency
• Access patterns optimization

.zarr - Chunked Array Storage

Description: Cloud-optimized chunked arrays Typical Data: Large N-dimensional arrays Use Cases: Cloud storage, parallel computing, streaming Python Libraries:

• zarr: zarr.open('file.zarr')
• xarray: Zarr backend support EDA Approach:
• Array metadata and dimensions
• Chunk size optimization
• Compression codec and ratio
• Synchronizer and store type
• Multi-scale hierarchies
• Parallel access performance
• Attribute metadata

.gz / .gzip - Gzip Compressed

Description: Compressed data files Typical Data: Any compressed text or binary Use Cases: Compression for storage/transfer Python Libraries:

• gzip: Built-in gzip module
• pandas: Automatic gzip handling in read functions EDA Approach:
• Compression ratio
• Original file type detection
• Decompression validation
• Header information
• Multi-member archives

.bz2 - Bzip2 Compressed

Description: Bzip2 compression Typical Data: Highly compressed files Use Cases: Better compression than gzip Python Libraries:

• bz2: Built-in bz2 module
• Automatic handling in pandas EDA Approach:
• Compression efficiency
• Decompression time
• Content validation

.zip - ZIP Archive

Description: Archive with multiple files Typical Data: Collections of files Use Cases: File distribution, archiving Python Libraries:

• zipfile: Built-in ZIP support
• pandas: Can read zipped CSVs EDA Approach:
• Archive member listing
• Compression method per file
• Total vs compressed size
• Directory structure
• File type distribution
• Extraction validation

.tar / .tar.gz - TAR Archive

Description: Unix tape archive Typical Data: Multiple files and directories Use Cases: Software distribution, backups Python Libraries:

• tarfile: Built-in TAR support EDA Approach:
• Member file listing
• Compression (if .tar.gz, .tar.bz2)
• Directory structure
• Permissions preservation
• Extraction testing

Time Series and Waveform Data

.wav - Waveform Audio

Description: Audio waveform data Typical Data: Acoustic signals, audio recordings Use Cases: Acoustic analysis, ultrasound, signal processing Python Libraries:

• scipy.io.wavfile: scipy.io.wavfile.read()
• wave: Built-in module
• soundfile: Enhanced audio I/O EDA Approach:
• Sample rate and duration
• Bit depth and channels
• Amplitude distribution
• Spectral analysis (FFT)
• Signal-to-noise ratio
• Clipping detection
• Frequency content

.mat - MATLAB Data

Description: MATLAB workspace variables Typical Data: Arrays, structures, cells Use Cases: MATLAB-Python interoperability Python Libraries:

• scipy.io: scipy.io.loadmat()
• h5py: For MATLAB v7.3 files (HDF5-based)
• mat73: Pure Python for v7.3 EDA Approach:
• Variable names and types
• Array dimensions
• Structure field exploration
• Cell array handling
• Sparse matrix detection
• MATLAB version compatibility
• Metadata extraction

.edf - European Data Format

Description: Time series data (especially medical) Typical Data: EEG, physiological signals Use Cases: Medical signal storage Python Libraries:

• pyedflib: EDF/EDF+ reading and writing
• mne: Neurophysiology data (supports EDF) EDA Approach:
• Signal count and names
• Sampling frequencies
• Signal ranges and units
• Recording duration
• Annotation events
• Data quality (saturation, noise)
• Patient/study information

.csv (Time Series)

Description: CSV with timestamp column Typical Data: Time-indexed measurements Use Cases: Sensor data, monitoring, experiments Python Libraries:

• pandas: pd.read_csv() with parse_dates EDA Approach:
• Temporal range and resolution
• Sampling regularity
• Missing time points
• Trend and seasonality
• Stationarity tests
• Autocorrelation
• Anomaly detection

Geospatial and Environmental Data

.shp - Shapefile

Description: Geospatial vector data Typical Data: Geographic features (points, lines, polygons) Use Cases: GIS analysis, spatial data Python Libraries:

• geopandas: gpd.read_file('file.shp')
• fiona: Lower-level shapefile access
• pyshp: Pure Python shapefile reader EDA Approach:
• Geometry type and count
• Coordinate reference system
• Bounding box
• Attribute table analysis
• Geometry validity
• Spatial distribution
• Multi-part features
• Associated files (.shx, .dbf, .prj)

.geojson - GeoJSON

Description: JSON format for geographic data Typical Data: Features with geometry and properties Use Cases: Web mapping, spatial analysis Python Libraries:

• geopandas: Native GeoJSON support
• json: Parse as JSON then process EDA Approach:
• Feature count and types
• CRS specification
• Bounding box calculation
• Property schema
• Geometry complexity
• Nesting structure

.tif / .tiff (Geospatial)

Description: GeoTIFF with spatial reference Typical Data: Satellite imagery, DEMs, rasters Use Cases: Remote sensing, terrain analysis Python Libraries:

• rasterio: rasterio.open('file.tif')
• gdal: Geospatial Data Abstraction Library
• xarray with rioxarray: N-D geospatial arrays EDA Approach:
• Raster dimensions and resolution
• Band count and descriptions
• Coordinate reference system
• Geotransform parameters
• NoData value handling
• Pixel value distribution
• Histogram analysis
• Overviews and pyramids

.nc / .netcdf - Network Common Data Form

Description: Self-describing array-based data Typical Data: Climate, atmospheric, oceanographic data Use Cases: Scientific datasets, model output Python Libraries:

• netCDF4: netCDF4.Dataset('file.nc')
• xarray: xr.open_dataset('file.nc') EDA Approach:
• Variable enumeration
• Dimension analysis
• Time series properties
• Spatial coverage
• Attribute metadata (CF conventions)
• Coordinate systems
• Chunking and compression
• Data quality flags

.grib / .grib2 - Gridded Binary

Description: Meteorological data format Typical Data: Weather forecasts, climate data Use Cases: Numerical weather prediction Python Libraries:

• pygrib: GRIB file reading
• xarray with cfgrib: GRIB to xarray EDA Approach:
• Message inventory
• Parameter and level types
• Spatial grid specification
• Temporal coverage
• Ensemble members
• Forecast vs analysis
• Data packing and precision

.hdf4 - HDF4 Format

Description: Older HDF format Typical Data: NASA Earth Science data Use Cases: Satellite data (MODIS, etc.) Python Libraries:

• pyhdf: HDF4 access
• gdal: Can read HDF4 EDA Approach:
• Scientific dataset listing
• Vdata and attributes
• Dimension scales
• Metadata extraction
• Quality flags
• Conversion to HDF5 or NetCDF

Specialized Scientific Formats

.fits - Flexible Image Transport System

Description: Astronomy data format Typical Data: Images, tables, spectra from telescopes Use Cases: Astronomical observations Python Libraries:

• astropy.io.fits: fits.open('file.fits')
• fitsio: Alternative FITS library EDA Approach:
• HDU (Header Data Unit) structure
• Image dimensions and WCS
• Header keyword analysis
• Table column descriptions
• Data type and scaling
• FITS convention compliance
• Checksum validation

.asdf - Advanced Scientific Data Format

Description: Next-gen data format for astronomy Typical Data: Complex hierarchical scientific data Use Cases: James Webb Space Telescope data Python Libraries:

• asdf: asdf.open('file.asdf') EDA Approach:
• Tree structure exploration
• Schema validation
• Internal vs external arrays
• Compression methods
• YAML metadata
• Version compatibility

.root - ROOT Data Format

Description: CERN ROOT framework format Typical Data: High-energy physics data Use Cases: Particle physics experiments Python Libraries:

• uproot: Pure Python ROOT reading
• ROOT: Official PyROOT bindings EDA Approach:
• TTree structure
• Branch types and entries
• Histogram inventory
• Event loop statistics
• File compression
• Split level analysis

.txt - Plain Text Data

Description: Generic text-based data Typical Data: Tab/space-delimited, custom formats Use Cases: Simple data exchange, logs Python Libraries:

• pandas: pd.read_csv() with custom delimiters
• numpy: np.loadtxt(), np.genfromtxt()
• Built-in file reading EDA Approach:
• Format detection (delimiter, header)
• Data type inference
• Comment line handling
• Missing value codes
• Column alignment
• Encoding detection

.dat - Generic Data File

Description: Binary or text data Typical Data: Instrument output, custom formats Use Cases: Various scientific instruments Python Libraries:

• Format-specific: requires knowledge of structure
• numpy: np.fromfile() for binary
• struct: Parse binary structures EDA Approach:
• Binary vs text determination
• Header detection
• Record structure inference
• Endianness
• Data type patterns
• Validation with documentation

.log - Log Files

Description: Text logs from software/instruments Typical Data: Timestamped events, messages Use Cases: Troubleshooting, experiment tracking Python Libraries:

• Built-in file reading
• pandas: Structured log parsing
• Regular expressions for parsing EDA Approach:
• Log level distribution
• Timestamp parsing
• Error and warning frequency
• Event sequencing
• Pattern recognition
• Anomaly detection
• Session boundaries