Principal, Software Engineer

Position Summary...

What you'll do...

About Team Our team builds and operates a large-scale internal platform that provides real-time streaming, messaging, telemetry, and data-processing capabilities for Walmart technology teams.

The platform supports business-critical workloads across event ingestion, stream processing, messaging, observability, large-scale persisted data processing, and data-lake use cases.

We work closely with Product Management, internal customers, and dependent engineering teams to expand platform capabilities, onboard new workloads, improve reliability and performance, and operate the platform efficiently at scale.

What you’ll do · Serve as a hands-on Principal Engineer , contributing directly to architecture, detailed design, coding, code reviews, debugging, testing, deployment, and production operations. · Own complex technical initiatives end to end , from requirements clarification and solution design through implementation, rollout, production validation, and ongoing operation. · Partner with Product Management to translate product direction and customer needs into scalable technical solutions within the existing platform architecture. · Design and evolve high-throughput event-ingestion, streaming, messaging, and telemetry systems , including efficient batching, routing, serialization, compression, secure data handling, and data delivery. · Build and improve large-scale real-time and persisted data-processing solutions using technologies such as Apache Spark, Spark Structured Streaming, Apache Flink, Apache Kafka , and comparable distributed technologies. · Identify and resolve significant issues involving latency, data-processing delays, scalability, reliability, performance, data integrity, and resource utilization , driving important problems from investigation through permanent production improvement. · Engineer systems for resiliency and failure recovery, including retries, backpressure, fault isolation, failover, fallback strategies, disaster recovery, and prevention of silent data loss. · Design for efficient use of compute, memory, network, and storage resources , and drive measurable cost improvements while maintaining or improving system performance and reliability. · Build and improve large-scale data pipelines and data-lake capabilities, including ingestion, transformation, ETL, partitioning, storage, querying, and data-quality diagnostics. · Drive operational excellence through observability, metrics, distributed telemetry, dashboards, alerting, profiling, capacity planning, peak-traffic readiness, incident response, Root Cause Analysis, and disaster-recovery exercises . · Improve engineering productivity and production reliability through CI/CD automation, Kubernetes-based deployments, release engineering, configuration management, and secure credential and secret management . · Collaborate with infrastructure, messaging, platform, data, application, and other dependent teams to resolve technical dependencies and drive cross-team initiatives to completion. · Mentor engineers and raise the technical bar through design reviews, code reviews, debugging, benchmarking, technical documentation, and hands-on engineering guidance. · Develop a deep understanding of the existing platform, deliver meaningful improvements within its architecture, and use production experience and data to guide thoughtful architectural evolution over time. · Participate in the team’s production on-call rotation , including weekend coverage, and provide technical leadership during complex production incidents.

What you’ll bring · Deep experience designing, building, and operating large-scale distributed systems with demanding throughput, latency, scalability, availability, and reliability requirements. · Strong hands-on software engineering skills with deep proficiency in Java and at least one additional systems or backend programming language.

Hands-on experience with Go is strongly preferred. · Strong experience with Apache Kafka or comparable messaging technologies, including event-driven architectures, partitioning, producer behavior, delivery semantics, retries, failure handling, and secure connectivity. · Strong experience with Apache Spark / Structured Streaming, Apache Flink , or comparable large-scale streaming and data-processing technologies. · Experience building high-volume event-ingestion, telemetry, or data-processing systems , with an understanding of batching, serialization, compression, encryption, routing, and efficient data movement. · Strong big-data engineering experience including data lakes, ETL, ingestion, partitioning, large-scale storage, batch processing, and querying large datasets . · Demonstrated ability to diagnose and resolve complex performance, latency, data-delay, scalability, reliability, and data-loss problems in production. · Strong understanding of distributed-system resiliency, including retries, backpressure, fault isolation, failover, fallback strategies, capacity planning, and disaster recovery. · Demonstrated experience improving the performance and cost efficiency of large-scale systems through better architecture, workload design, resource utilization, and data-processing strategies. · Strong production observability experience using technologies such as OpenTelemetry, Prometheus, Grafana , profiling tools, metrics, dashboards, and alerting. · Experience with Kubernetes, CI/CD, deployment automation, configuration management, and production release/change-management practices . · Experience building and operating production systems in public cloud environments.

Google Cloud Platform (GCP) experience is preferred; strong AWS or Azure experience is also relevant. · Practical understanding of production security, including TLS, credentials, secrets, workload identity, access controls, and secure configuration management . · Experience with SQL and distributed query technologies such as Trino or comparable systems is valuable. · Stron