Skip to content

2022-11-23: PromCon EU, KubeCon NA, Kubernetes 1.26 changes, Cilium training, eBPF HTTP session monitoring, Rust CLI tools

Thanks for reading the web version, you can subscribe to the Ops In Dev newsletter to receive it in your mail inbox.

๐Ÿ‘‹ Hey, lovely to see you again

Busy times since we last read - KubeCon NA week, PromCon EU, and Continuous Lifecycle events. And a life-changing moment after landing in Nuremberg from Detroit - a new house owner demanded my rented flat within 3 months, chaos engineering in real life somehow. Lucky me found and signed a new flat contract just one mile away, though I am super busy at the moment. Working remotely async is very helpful. Now getting back to all the good things I learned and want to share them with you :-)

The newsletter starts with an event recap in the hot topics section; you can also navigate down to "๐ŸŽฏ Release speed-run" and dive into event ideas and learnings later.

โ˜• Hot Topics

PromCon EU

It was my first in-person PromCon after changing jobs in 2020, and PromCon was amazing, as described by friends. Run by community members for the community, the talks were full of learning sessions and technical insights, while the social event allowed long-time-no-see chats - and finally meeting my teammates Andrew and Bob :-)

My personal favorite was a sneak peek into Perses, which is a newly developed Observability UI under the Apache 2.0 license, supported by Chronosphere and Amadeus. It focuses on dashboards as code and GitOps and is part of the Coredash community by the Linux Foundation. I had been talking about Perses in this newsletter and my talks and had a chat with Augustin as a Perses maintainer at the PromCon EU social event. The next day, we got a spontaneous lightning talk about Perses with a live demo, pictures in this tweet.

Perses demo in the PromCon EU lightning talk

My teammate Andrew Newdigate shared insights into capacity planning and forecasting on GitLab.com SaaS, using Open Source and building TAMland. Recommend watching the recording and reviewing the slides and TAMland project.

KubeCon NA

After meeting Marino and Julia for dinner on Saturday before KubeCon NA, Marino suggested to join Cloud-Native Rejekts, an event where you can submit your KubeCon talks that were not accepted. Amazing organization and hosts, and engaging sessions to dive into eBPF and WebAssembly uses cases. My favorites: Inspektor Gadget, a collection of eBPF-based gadgets to debug and inspect Kubernetes apps and resources. SpiderLightning, a set of abstract distributed application capabilities - going beyond WASI. State management, pub/sub, event-driven programming, running in Azure. We ended up organizing dinner in a Twitter DM group that expanded from 10 to ~50 folks, shoutout to Lian for organizing the whole thing!

On Monday, I attended the Open Observability day with a talk about "Confidence with Chaos for your Kubernetes Observability" (recording), as a helpful resource to refine alerts and detect unknown behaviour in a Kubernetes cluster for better Observability. It sparked great ideas for chaos engineering practices in MRs and with continuous delivery in production - stay tuned for more blog posts.

The OpenTelemetry Unplugged event on Tuesday was really energizing, meeting maintainers and end-users and later brainstorming together in breakout sessions about OpAmp for scaling OpenTelemetry collector deployments, and how eBPF can help OpenTelemetry. I took notes in this comment, the morning sessions were streamed on YouTube.

Three days zero day events, and three more days meeting community members, and missing all the talks - on purpose, I think of KubeCon as a way to meet, chat and connect with the community. For example, I was able to introduce my mentee Julia to many folks, companies and organisations, and Julia tweeted, vlogged and blogged about her experience. After starting with mentoring last year, I really see how much I love enabling others for their success and help foster a diverse and inclusive community.

Back to learning: My bookmarked talks helped me identify to watch the recordings after they are published by CNCF on the YouTube playlist for KubeCon NA. Here are a few recommendations to watch:

Project updates:

Community and belonging:

Kunal interviewed many folks for Civo TV, including myself on CI/CD - thanks mate for this amazing playlist!

See you all at KubeCon EU in Amsterdam!

๐ŸŽฏ Release speed-run

Prometheus v2.40.0 was released live at PromCon EU, adding experimental support for native histograms. To instrument application with native histograms, use client_golang v1.14.0 or later and set the NativeHistogramBucketFactor in your HistogramOpts (1.1 is a good starting point). Pyrra v0.5.0 improves SLO management with adding more details and duration histograms. Keep your fingers crossed for a new Prometheus Alert Manager release - it will bring support for Discord as transport.

Cilium brings new bugfix releases: v1.12.4, v1.11.11, v1.10.7. GitLab 15.6 adds support for special characters in CI/CD variables, dynamic jobs that run when a path exists, specified via environment variable, CI/CD insights for admins on job queued and duration times, and much more.

Rust 1.65.0 brings support for let-else-statements, generic associated types and more stable interfaces. Python 3.11 is 10-60% faster than 3.10, error messages that highlight the location of the problem, notes field to extend exceptions, with more insights in this Twitter thread.

๐Ÿ›ก๏ธ The Sec in Ops in Dev

Hashicorp shared their vision of the state of Vault and Kubernetes at Hashiconf, with the intention to additionally create a Kubernetes Operator to enhance vault-k8s: "This operator could be used to periodically sync a subset of Vault secrets to Kubernetes secrets for applications to consume without directly interacting with Vault.".

Chainguard Academy provides learning resources for software supply chain security with Sigstore, Wolfi, and Chainguard Enforce. The Sigstore training is provided by the Linux Foundation.

Trivy can now scan unpacked binary files, made possible by using Rekor from the Sigstore project to look up the hash of a binary file. When a relevant SBOM is found, Trivy uses the details to discover security issues.

Renovate now supports merge confidence as an additional indicator whether automated dependency updates are good to merge. Thanks Niclas.

โ›… Cloud Native

Liz Rice shared an interesting thought into future Kubernetes 1.26 changes and kube-proxy in userspace use on LinkedIn: "#Kubernetes 1.26 is removing the userspace mode of kube-proxy, but rather than changing to an iptables or ipvs implementation, for better performace at scale you could go to Ciliumโ€™s #eBPF implementation instead ๐Ÿ". Well :-)

The 4 Rs of Chaos Engineering - Reliability, Robustness, Rebound, Resilience, explained in this great post by Yury Niรฑo Roa.

๐Ÿ‘๏ธ Observability

Datadog published a great read on capturing production traffic with eBPF. It starts with explaining the problems of HTTP session monitoring, for example, with tcpdump, and how an eBPF-based traffic capturer can be built, such as a simple Golang web server. A few Kernel hooks with accepts, read, write, and close later, we learn about the user mode agent, and how to test the newly created tracer.

Jaeger Tracing provides different storage plugins next to the default with Elasticsearch; at GitLab, we started using the Clickhouse plugin, for example. Timescale wrote a blog post on how to use PostgreSQL as storage for Jaeger Tracing, abstracted with their Promscale tool stack. The article also highlights the Jaeger Tracing compliance certification that ensures reliable and tested backend types.

The OpenTelemetry collector got support for receiving SNMP in this PR. Parsing SNMP traps and their attributes within OIDs can be challenging; I'm curious to see how this receiver is used in production and which collector pipeline plugins for additional ingestion will be needed.

๐Ÿ” The inner Dev

Learning Rust by example, and getting inspired by the open source community: Command line tools can help. This article collects a great list: Alacritty, Starship, bat, LSD and exa, rip, xcp, zoxide, dust, ripgrep, fd, sd, procs, bottom, Topgrade, Broot, Tokei, etc. Even when not learning Rust, these tools can help make you more efficient. I usually type a wild combiation of du -sh --max-depth=1 and grep/tree/sortwhere dust could help faster.

WebAssembly is still a hot topic that is hard to learn and onboard. Embed HTTP servers in WASM with Rust and CSharp takes the approach of explaining a use-case where WebAssembly moves out of the browser scope and dives into practical programming to learn. The article also touches on the Docker announcement to natively support WASM, at KubeCon NA.

The article about OpenTelemetry auto-instrumentation for Python developers shares quick examples to get started with an engineering strategy, and helps to understand the value of metrics and traces with Observability.

๐Ÿ“ˆ Your next project could be ...

๐Ÿ“š Tools and tips for your daily use

  • Hurl is a command line tool that runs #HTTP requests defined in a simple plain text format. It can be used for fetching data, testing HTTP sessions and testing XML / JSON APIs.
  • Autograf, dynamically generate Grafana dashboard based on Prometheus metrics, demo photo from PromCon EU
  • dive, a tool for exploring each layer in a docker image
  • skopeo is a CLI utility to work with remote images registries and retrieve information, images, signing content
  • hiSHtory: Better Shell History. It stores your shell history in context , and is stored locally and end-to-end encrypted for syncing between computers and servers.
  • DBeaver: Free multi-platform database tool for developers, database administrators, analysts and all people who need to work with databases. Supports all popular databases. Thanks Brendan O'Leary for sharing while debugging Code Challenge
  • fubectl, a fancy CLI with aliases for kubectl
  • xlskubectl, a spreadsheet to control your Kubernetes cluster
  • kubectx, a faster way to switch between clusters and namespaces in kubectl
  • Allow canceling a running job with a new commit and pipeline, using the interruptable keyword in GitLab CI/CD. Shared here.

๐Ÿ”– Book'mark

๐ŸŽฅ Events and CFPs

2023 outlook:

๐Ÿ‘‹ CFPs due soon - 2023

Looking for more CfPs? Try CFP Land.

๐ŸŽค Shoutouts

David Schmitt for sharing a fancy Git history CLI cleaner command: git filter-branch -f --tree-filter "find * -type f -exec sed -i -e 's/accidental-slur/safe-replacement/' {} \;" HEAD

Daniel Stori for the technical debt illustration :D

Everyone at KubeCon NA and PromCon EU who made us feel welcome!

๐ŸŒ

Thanks for reading! If you are viewing the website archive, make sure to subscribe to stay in the loop!

See you next month - let me know what you think on LinkedIn.

Cheers,

Michael

PS: If you want to share items for the next newsletter, please check out the contributing guide - tag me in comments or send me a DM. Thanks!