Skip to content

2022-10-15: Events (KCD Munich, DevOps Camp, eBPF Summit, KubeCon NA), Kubernetes Observability KubeSimplify workshop, coroot, wazero, JSON Crack, Reliable Machine Learning book

Thanks for reading the web version, you can subscribe to the Ops In Dev newsletter to receive it in your mail inbox.

👋 Hey, lovely to see you again

Refreshing to see events slowly coming back - I enjoyed joining three community events since we last read, DevOps Camp Nuremberg, Open Source Automation Days, and Kubernetes Community Days Munich, sharing my knowledge about Observability and learning fresh ideas and new technology. I have also been diving into 2 days of the eBPF Summit.

This newsletter focuses on sharing some of these learnings from talks and conversations - it has gotten a bit long, suggest scrolling down to the "🎯 Release speed-run" if you are not immediately interested in the event insights :)

☕ Hot Topics

🌱 Kubernetes Community Days Munich

A lovely event in the cloud-native community in Munich, Germany. Believe it or not, the first time - ~200 attendees, 2 tracks with interesting hands-on talks and workshops, a great venue, amazing food, and wonderful folks excited to meet, learn and chat. Thanks to all organizers, sponsors, speakers, and attendees!

KCD Munich 2022

Here are a few insights to dive in: Alois Reitbauer provided a deep-dive into Open Feature, a new framework and specification for feature flags (Twitter thread with pictures). Next to the first steps with the SDK (Python/Rust experts needed!), Open Feature integrates very well with OpenTelemetry tracing: and metrics coming soon.

In the keynote on the 2nd day at KCD Munich, Michael Scholinka shared their journey about getting to development environments in Kubernetes - from shell scripts for dev environment provisioning to challenges with docker-compose, Helm charts, etc. to ending up writing their tools: GetDeck is a CLI that creates reproducible Kubernetes environments for development and testing. It is getting integrated into Unikube.

Hijacking Kubernetes in the first session, and later preventing the Kubernetes cluster being hacked in the follow-up session was a great interactive learning experience as always by Nico Meisenzahl.

Cannot wait for KCD Munich 2023 :-) Follow them on LinkedIn and Twitter and reach out to the organizers for sponsoring and more, Max Körbacher and team.

🏗️ DevOps Camp Nuremberg

This community event is organized as a bar camp, with no agenda, but each day starts with session pitches, and then the crowd builds the agenda together. I had a few ideas: Chaos Engineering, Pipeline Efficiency, OpenTelemetry, eBPF and decided to do something unusual - just a few slides from different talks and workshops to kick off a session to try the OpenTelemetry demo in Kubernetes together. Oh, this was fun to stress test the web shop with 1000 users and see that traces in Jaeger were broken or showed fancy errors.

The second pitch was a topic I wanted to learn more about and see if we could research together by reading documentation and trying things - eBPF. 3 slides, 3 EveryoneCanContribute cafe meetup recordings, 1 learning platform, and lots of interest and insights. For example, I learned that an eBPF program can be written in any language, and the Kernel module sandbox takes care of loading the binary assembly. When making a programming mistake, the sandbox crashes but not the kernel. Or that tcpdump and capturing network traffic have a lot in common with eBPF. With Cilium, Tetragon, Pixie, Falco, Parca, Tracee, Bumblebee, we also have great open-source projects to use eBPF as a first step to see what is possible. Someone mentioned redbpf in the discussion, which allows to write and build eBPF modules in Rust.

Csaba Karsai did a great session on Chaos Mesh and Chaos Engineering already, so no more topics for day 2 for me? Loving the bar camp style, I made it a GitLab AMA. It was the first time meeting many folks in Nuremberg after joining GitLab in March 2020, and sparked great conversations beyond the AMA. My session notes and ideas are documented in this organisation issue.

🐝 eBPF Summit

Amazing event - 10 hours of free learning content, from getting started, its use cases, future development and the overall idea that you don't need to be a kernel developer or C expert to dive in. The eBPF spectrum includes Networking & Security, DDoS mitigation, Application Performance Monitoring and Troubleshooting, Network load-balancing, Runtime Security, Application Tracing, and Network Observability.

I figured that this newsletter issue would need all the great talks and Youtube timestamps to allow you to choose your favorite topics. Boring solution: Commented with the timestamps table-of-content on YouTube - please navigate into day 1 and day 2 to see everything. Here are a few recommendations to watch:

I especially love the session about Building and oomkill-exporter by eBPF since it provides insights into how to combine Prometheus metrics and eBPF - a question that I ask at the end of my Observability workshops and talks currently. The Future of eBPF in the Linux Kernel highlights Rust and BPF having the same goal: Safe kernel programming, portable programs with stable interfaces, extending the C language e.g. with assertions, and sleepable BPF programs as fibers in the Kernel. Super interesting!

Last but not least - the Capture-the-flag exercises are amazing, I got side-tracked during the event diving deeper in a Vagrant or Lima VM. Recommend starting them async to learn eBPF and maybe write a blog post sharing your experience :-)

🎯 Release speed-run

HashiConf brought many updates for Terraform, Consul, HCP Boundary is GA, Waypoint 0.10, Nomad 1.4 and the full launch of the HashiCorp Developer portal. PostgresSQL 15 finally brings MERGE — conditionally insert, update, or delete rows of a table 🚀 GitLab 15.4 brings better VS Code CI/CD experience, adding assignees to tasks, etc. Jaeger UI v1.27 supports visualizing traces as flame graphs 🔥 Prometheus 2.39.0 optimizes memory usage, by half, shown here; 2.39.1 brings a bugfix for alert labelling. k6 v0.40.0 supports experimental modules, starting with Redis, WebSockets and timers. trivy v0.32.0 supports Conan (C/C++) and Gradle (Java) for dependency scanning.

🛡️ The Sec in Ops in Dev

Ian Coldwater asked on Twitter: "Tell us your wildest incident response stories (without violating your NDAs)".

⛅ Cloud Native

KubeCon NA is coming soon, Oct 24-28. I have selected a few sessions in my sched profile. On Monday, I'll join the Open Observability day; on Tuesday, it will be a CNCF project day: Cilium project contributions and later OpenTelemetry project meeting. Prometheus: The Documentary (official trailer) - full version will be released at KubeCon on Oct 25. Very much looking forward to insights and a great history that inspired metrics monitoring as an industry default and made cloud-native monitoring/observability much more convenient for everyone.

Getting started with cloud-native and Kubernetes can be overwhelming - fortunately, I found the amazing learning videos from IBM Technology some years ago. They use an on-screen drawing that is easy to follow and ensures a great learning curve. Highly recommend the Kubernetes Essentials YouTube playlist and the most recent Kubernetes Operators Explained video.

👁️ Observability

I've met Peter Zaitsev, Percona founder, at Open Source Automation Days, and we talked a bit about Observability and eBPF too. He shared coroot, a monitoring and troubleshooting tool for microservice architectures. It uses eBPF to build a service map, to follow tracing calls to other services and databases, for example, PostgreSQL, without code instrumentation. The architecture is interesting - Coroot agents act as Prometheus exporters, Prometheus collects the metrics, and the Coroot frontend uses Prometheus as a data source similar to Grafana. The installation docs provide Kubernetes and Docker container options - suggest trying it!

The friendly community at Kube Simplify creates amazing free learning resources as live workshops to learn Kubernetes. I was happy to provide a workshop on "Kubernetes Observability" ... and exceeded time and content by diving into OpenTelemetry and Chaos Engineering in 3.5 hours. Check out the slides with exercises and project with solutions. If you want to learn more about Chaos Mesh, Flant wrote a great getting started article.

Kube Simplify: Kubernetes Observability workshop

Diving into the Golden Signals for Kubernetes and answering Observability questions - interesting article from Isovalent with also tackling the problem of monitoring agent sidecar performance vs. eBPF. I'd love to see a "generic" way of collecting Observability data using eBPF and have the OpenTelemetry collector process, filter, correlate, and store the data - something that has been started in OpenTelemetry eBPF working group. Speaking of data, Yuri Shkuro, creator of Jaeger, wrote an insightful blog post about six pillars of Observability data: Metrics, the original pillar. Logs, the ancient pillar. Traces, the “new cool kid on the block” pillar. Events, the misunderstood pillar. Profiles, the geek pillar. Exceptions, the forgotten pillar. Now, wait until someone proposes netflow data types, eBPF, and more - the OpenTelemetry specification and framework allow for these proposals in the CNCF community, fortunately.

Last but not least, recommend these Container Days recordings as a learning resource:

🔍 The inner Dev

wazero was mentioned in a tweet, claiming that is has zero dependencies as a WebAssembly runtime for Go developers, allowing to safely run code compiled in other languages by loading a wasm binary. The examples integrate well into your Go build chain and also provide WASI (WebAssembly System Interface) exercises.

📈 Your next project could be ...

  • Add pets to your VS Code using this extension. Use cmd+shift+p on macOS to open the console, search for pet coding to start a new session, and repeat to e.g., throw a ball.

📚 Tools and tips for your daily use

🔖 Book'mark

🎥 Events and CFPs

👋 CFPs due soon - 2023

Looking for more CfPs? Try CFP Land.

🎤 Shoutouts

Thanks, Forrest Brazeal, for a laugh on "The Re-Org Rag (I'm My Own VP)" - must watch!

🌐

Thanks for reading! If you are viewing the website archive, make sure to subscribe to stay in the loop!

See you next month - let me know what you think on Twitter or LinkedIn.

Cheers,

Michael

PS: If you want to share items for the next newsletter, please check out the contributing guide - tag me in tweet replies or send me a DM. Thanks!