Senior Security Data Engineer

Added
8 days ago
Type
Full time
Salary
Salary not provided

Related skills

sql python kafka spark data lake

๐Ÿ“‹ Description

  • Maintain high-throughput streaming pipelines to ingest logs from various sources (Firewalls, Cloud
  • Write parsers to convert raw, messy logs into standard schemas (e.g., OCSF or ECS) for consistent
  • Implement routing logic to send "high-value" data to the SIEM and "bulk" data to low-cost Object
  • Clean and structure data to enable AI/ML detection models and advanced analytics.

๐ŸŽฏ Requirements

  • Proficiency in Python (for ETL) and SQL (for complex querying).
  • Experience with Message Queues (e.g., Kafka, Pub/Sub) and stream processing concepts.
  • Mastery of Regex and log parsing strategies for standard formats (Syslog, CEF, JSON).
  • Understanding of Data Lake principles (Parquet/Avro formats) vs. Data Warehouses.

๐ŸŽ Benefits

  • Experience with Vector Databases for storing embeddings.
  • Knowledge of Log Observability/Routing tools (middleware that routes logs).
  • Familiarity with Big Data frameworks (e.g., Spark, Flink).
Get alerts for new remote data jobs
A weekly email with new matches. Unsubscribe anytime.
Share job

Meet JobCopilot: Your Personal AI Job Hunter

Automatically Apply to Data Jobs. Just set your preferences and Job Copilot will do the rest โ€” finding, filtering, and applying while you focus on what matters.

Related Data Jobs

See more Data jobs โ†’