Cloud Observability & Performance Engineer Job Halcyon

Cloud Observability and Performance Engineer Job

The Role:

We are looking for a Cloud Observability and Performance Engineer to join our Chaos Cloud Engineering team. In this role, you will design and implement observability, monitoring, and performance strategies for cloud-hosted microservices that manage and orchestrate endpoint security agents at scale. This position is critical to ensuring the reliability, visibility, and performance optimization of our backend systems that power cloud-based security operations for millions of endpoints worldwide.

Responsibilities:

• Design, build, and maintain end-to-end observability for distributed cloud services (telemetry, logging, tracing, alerting).
• Develop and optimize metrics pipelines and dashboards (e.g., Prometheus, Grafana, OpenTelemetry, Datadog).
• Ensure high performance, availability, and scalability of agent management systems in production.
• Collaborate with development, SRE, and security teams to troubleshoot production issues using observability tooling.
• Define and implement SLOs, SLIs, and performance benchmarks for cloud components and services.
• Instrument code and services to expose business-relevant metrics and latency bottlenecks.
• Automate performance regression testing and anomaly detection.
• Support proactive incident detection and real-time monitoring strategies across multi-cloud environments.
• Design, implement, and own a performance testing framework to validate system throughput, latency, and scalability under load.
• Define baseline performance thresholds and use observability tooling to monitor and validate results.
• Provide root cause analysis and performance tuning recommendations.

Skills and Qualifications:

• 5+ years of professional work experience in observability, site reliability, or cloud performance roles.
• Strong experience with monitoring and observability stacks (e.g., Prometheus, Grafana, ELK, OpenTelemetry, Datadog, AWS CloudWatch).
• Proficiency in cloud platforms (e.g., AWS, GCP, Azure) and cloud-native services (e.g., ECS, EKS, Lambda).
• Experience with containerization and orchestration tools (e.g., Docker, Kubernetes).
• Hands-on experience designing and implementing performance or load testing frameworks for distributed systems.
• Ability to define and validate throughput baselines, latency thresholds, and system limits under real-world traffic scenarios.
• Solid knowledge of distributed systems, microservices, and performance debugging.
• Proficiency in Python, Scala, or other language(s) for tooling and automation.
• Familiarity with CI/CD pipelines, infrastructure as code (e.g., Terraform), and version control (Git).
• Ability to participate in an on-call rotation to support observability infrastructure and assist with incident investigations.

Bonus Skills and Qualifications:

• Experience with endpoint security platforms or agent-based systems.
• Familiarity with SIEM, security analytics, or cloud threat detection pipelines.
• Background in networking performance, TLS handshake optimization, or load balancing.
• Experience with SLA/SLO-driven operational excellence in high-scale environments.
• Knowledge of additional languages, such as Go.
• Experience integrating performance tests into CI/CD pipelines and visualizing results using tools like Grafana, Datadog, or similar.
• Familiarity with tools such as k6, Locust, JMeter, Gatling, or custom-built performance testing solutions.

How to Apply

Click Here to Apply