Grafana Labs develops open-source observability software for understanding the performance and health of infrastructure, applications, and frontend systems. Its platform brings together metrics, logs, traces, and profiles through products including Grafana, Loki, Tempo, and Mimir, helping organizations visualize data, monitor services, investigate anomalies, analyze root causes, and manage service level objectives. Grafana Labs offers cloud and self-managed deployment options, with integrations for technologies such as Prometheus and OpenTelemetry. The company operates in the enterprise SaaS and artificial intelligence sectors and builds tools for teams responsible for infrastructure monitoring and modern observability.
Senior Backend Engineer, Tempo Databases — Grafana Labs (Canada Remote)
Shape the technical direction and operation of Tempo, Grafana Labs’ open-source tracing backend. Improve its distributed data systems, APIs, scalability, performance, and reliability for Grafana observability products.
Canada
United States$164,490 – $197,389 / year
RemoteFull-timeSeniorEnglish RequiredBackend EngineerSoftware EngineerDescription
- Set technical direction for challenging Tempo roadmap work.
- Lead complex, multi-month projects from initial problem definition through rollout.
- Own components spanning ingestion, storage, querying, and metrics generation.
- Create and present design documents, and take part in design reviews.
- Assess performance, cost, and complexity trade-offs, and record the decisions.
- Develop structured, predictable, discoverable APIs for people, agents, products, and integrators.
- Deliver operational results against measures such as P99 write latency, repeat incidents, and total cost per ingested GB.
- Advance automation, configurable rollouts, useful alerts, and Zero Ops practices.
- Work with Product and the App Observability, Asserts, Drilldown, and Grafana Assistant teams.
- Guide junior and mid-level engineers through reviews, design feedback, and pairing.
- Take part in on-call rotations, incident response, and post-incident reviews.
- Support the open-source community, review outside contributions, and contribute to Tempo.
- Improve Tempo’s performance, autoscaling, query ranges, trace aggregation, customer limits, and multi-cell operations.
Requirements
- Demonstrated success delivering complex projects over several months, from design through implementation and operation.
- Substantial practical experience building and running distributed data systems in production.
- Experience with ingestion pipelines, storage engines, query execution, or related systems.
- Strong software craftsmanship and a record of producing clean, robust, high-performance software.
- Strong Go skills or the ability to ramp up in Go; Rust, C, or C++ experience is relevant.
- Experience owning production services and participating in an on-call rotation.
- Experience reducing operational toil and treating service level objectives as a product feature.
- Clear communication skills for fully remote, asynchronous work.
- Experience with tracing, OpenTelemetry, or large-scale observability systems is a plus.
- Experience designing query languages, SQL- or TraceQL-like engines, or APIs used programmatically is a plus.
- Experience with Parquet or other columnar storage formats, or purpose-built analytical storage formats, is a plus.
- Experience running large-scale, multi-tenant, multi-cell SaaS infrastructure on Kubernetes is a plus.
- Experience building software for AI or large language model users is a plus.
- Open-source contributions or project maintainership are a plus.
- Experience using Grafana, Prometheus, Loki, or Tempo while on call is a plus.
- Experience working on a fully remote, globally distributed team is a plus.
Benefits
- Restricted stock units (RSUs).
- A company-funded budget for modern AI coding assistants.
- Access to current frontier models from OpenAI, Anthropic, and Google.
- A fully remote, globally distributed culture.
- Career development pathways.
- In-person onboarding.
- 30 days of annual leave each year.
- Three annual leave days set aside for Grafana Shutdown Days.