Senior Data Engineer

Senior Data Engineer Location: Herndon, VA (Remote Work) Must have an Public Trust Clearance

KEY RESPONSIBILITIES • Provide authoritative expertise on data engineering methods and best practices, including code first development approaches and modern pipeline design patterns. • Design, implement, and maintain the data architecture that supports products and end users, with all assets managed under source control. • Design, implement, and maintain ELT and ETL pipelines for efficient processing of source data in Azure Synapse and Azure Machine Learning, using both SDK V1 and SDK V2. • Migrate source data identified by SBA OIG into Azure Data Lake Storage. • Normalize entity attributes such as addresses, phone numbers, and other common fields. • Review, maintain, and improve existing architecture and pipelines, including periodic audits addressing bottlenecks, deprecated dependencies, and architecture drift. • Establish quality controls across all pipelines and introduce error handling, logging mechanisms, and validation checks. • Incorporate source control across all pipelines and analytics codebases so code can evolve iteratively without destabilizing the architecture. • Optimize ingestion, processing, and storage across a wide variety of datasets and data types, including modern columnar formats such as Parquet. • Develop self service capabilities that let SBA OIG analysts query and export data for investigations and audits. • Author robust standard operating procedures governing the authoring, development, validation, publishing, execution, and monitoring of all data pipelines and assets in the Azure environment. • Produce detailed documentation of the data architecture, including data dictionaries, entity relationship diagrams, and pipeline process maps. • Maintain and expand the environment with additional datasets and services on request, following a defined intake and testing process before production deployment. • Stay current with emerging AI tooling relevant to data engineering and contribute to exploratory work evaluating automation and language model assisted capabilities.