2022-10-15: Events (KCD Munich, DevOps Camp, eBPF Summit, KubeCon NA), Kubernetes Observability KubeSimplify workshop, coroot, wazero, JSON Crack, Reliable Machine Learning book¶
Thanks for reading the web version, you can subscribe to the Ops In Dev newsletter to receive it in your mail inbox.
👋 Hey, lovely to see you again¶
Refreshing to see events slowly coming back - I enjoyed joining three community events since we last read, DevOps Camp Nuremberg, Open Source Automation Days, and Kubernetes Community Days Munich, sharing my knowledge about Observability and learning fresh ideas and new technology. I have also been diving into 2 days of the eBPF Summit.
This newsletter focuses on sharing some of these learnings from talks and conversations - it has gotten a bit long, suggest scrolling down to the "🎯 Release speed-run" if you are not immediately interested in the event insights :)
☕ Hot Topics¶
🌱 Kubernetes Community Days Munich¶
A lovely event in the cloud-native community in Munich, Germany. Believe it or not, the first time - ~200 attendees, 2 tracks with interesting hands-on talks and workshops, a great venue, amazing food, and wonderful folks excited to meet, learn and chat. Thanks to all organizers, sponsors, speakers, and attendees!
Here are a few insights to dive in: Alois Reitbauer provided a deep-dive into Open Feature, a new framework and specification for feature flags (Twitter thread with pictures). Next to the first steps with the SDK (Python/Rust experts needed!), Open Feature integrates very well with OpenTelemetry tracing: and metrics coming soon.
In the keynote on the 2nd day at KCD Munich, Michael Scholinka shared their journey about getting to development environments in Kubernetes - from shell scripts for dev environment provisioning to challenges with docker-compose, Helm charts, etc. to ending up writing their tools: GetDeck is a CLI that creates reproducible Kubernetes environments for development and testing. It is getting integrated into Unikube.
🏗️ DevOps Camp Nuremberg¶
This community event is organized as a bar camp, with no agenda, but each day starts with session pitches, and then the crowd builds the agenda together. I had a few ideas: Chaos Engineering, Pipeline Efficiency, OpenTelemetry, eBPF and decided to do something unusual - just a few slides from different talks and workshops to kick off a session to try the OpenTelemetry demo in Kubernetes together. Oh, this was fun to stress test the web shop with 1000 users and see that traces in Jaeger were broken or showed fancy errors.
The second pitch was a topic I wanted to learn more about and see if we could research together by reading documentation and trying things - eBPF. 3 slides, 3 EveryoneCanContribute cafe meetup recordings, 1 learning platform, and lots of interest and insights. For example, I learned that an eBPF program can be written in any language, and the Kernel module sandbox takes care of loading the binary assembly. When making a programming mistake, the sandbox crashes but not the kernel. Or that tcpdump and capturing network traffic have a lot in common with eBPF. With Cilium, Tetragon, Pixie, Falco, Parca, Tracee, Bumblebee, we also have great open-source projects to use eBPF as a first step to see what is possible. Someone mentioned redbpf in the discussion, which allows to write and build eBPF modules in Rust.
Csaba Karsai did a great session on Chaos Mesh and Chaos Engineering already, so no more topics for day 2 for me? Loving the bar camp style, I made it a GitLab AMA. It was the first time meeting many folks in Nuremberg after joining GitLab in March 2020, and sparked great conversations beyond the AMA. My session notes and ideas are documented in this organisation issue.
🐝 eBPF Summit¶
Amazing event - 10 hours of free learning content, from getting started, its use cases, future development and the overall idea that you don't need to be a kernel developer or C expert to dive in. The eBPF spectrum includes Networking & Security, DDoS mitigation, Application Performance Monitoring and Troubleshooting, Network load-balancing, Runtime Security, Application Tracing, and Network Observability.
I figured that this newsletter issue would need all the great talks and Youtube timestamps to allow you to choose your favorite topics. Boring solution: Commented with the timestamps table-of-content on YouTube - please navigate into day 1 and day 2 to see everything. Here are a few recommendations to watch:
- Kickoff, eBPF themes and use cases
- The Future of eBPF in the Linux Kernel
- Troubleshooting and healing networks with eBPF
- eBPF: Innovations in cloud native
- eBPF and Cilium at Google
- All Your Queues Are Belong to Us: Debugging and Mitigating a Kernel Bug with eBPF
- Securing systems with eBPF Linux Security Module
- Analysis of offensive capabilities of eBPF and implementation of a rootkit
- Metalens: Live Programming and Visualizing eBPF
- Building an oomkill-exporter powered by eBPF
- BTFGen: one Step Closer to Truly Portable eBPF Programs
I especially love the session about Building and oomkill-exporter by eBPF since it provides insights into how to combine Prometheus metrics and eBPF - a question that I ask at the end of my Observability workshops and talks currently. The Future of eBPF in the Linux Kernel highlights Rust and BPF having the same goal: Safe kernel programming, portable programs with stable interfaces, extending the C language e.g. with assertions, and sleepable BPF programs as fibers in the Kernel. Super interesting!
Last but not least - the Capture-the-flag exercises are amazing, I got side-tracked during the event diving deeper in a Vagrant or Lima VM. Recommend starting them async to learn eBPF and maybe write a blog post sharing your experience :-)
🎯 Release speed-run¶
HashiConf brought many updates for Terraform, Consul, HCP Boundary is GA, Waypoint 0.10, Nomad 1.4 and the full launch of the HashiCorp Developer portal. PostgresSQL 15 finally brings MERGE — conditionally insert, update, or delete rows of a table 🚀 GitLab 15.4 brings better VS Code CI/CD experience, adding assignees to tasks, etc. Jaeger UI v1.27 supports visualizing traces as flame graphs 🔥 Prometheus 2.39.0 optimizes memory usage, by half, shown here; 2.39.1 brings a bugfix for alert labelling. k6 v0.40.0 supports experimental modules, starting with Redis, WebSockets and timers. trivy v0.32.0 supports Conan (C/C++) and Gradle (Java) for dependency scanning.
🛡️ The Sec in Ops in Dev¶
Ian Coldwater asked on Twitter: "Tell us your wildest incident response stories (without violating your NDAs)".
⛅ Cloud Native¶
KubeCon NA is coming soon, Oct 24-28. I have selected a few sessions in my sched profile. On Monday, I'll join the Open Observability day; on Tuesday, it will be a CNCF project day: Cilium project contributions and later OpenTelemetry project meeting. Prometheus: The Documentary (official trailer) - full version will be released at KubeCon on Oct 25. Very much looking forward to insights and a great history that inspired metrics monitoring as an industry default and made cloud-native monitoring/observability much more convenient for everyone.
Getting started with cloud-native and Kubernetes can be overwhelming - fortunately, I found the amazing learning videos from IBM Technology some years ago. They use an on-screen drawing that is easy to follow and ensures a great learning curve. Highly recommend the Kubernetes Essentials YouTube playlist and the most recent Kubernetes Operators Explained video.
I've met Peter Zaitsev, Percona founder, at Open Source Automation Days, and we talked a bit about Observability and eBPF too. He shared coroot, a monitoring and troubleshooting tool for microservice architectures. It uses eBPF to build a service map, to follow tracing calls to other services and databases, for example, PostgreSQL, without code instrumentation. The architecture is interesting - Coroot agents act as Prometheus exporters, Prometheus collects the metrics, and the Coroot frontend uses Prometheus as a data source similar to Grafana. The installation docs provide Kubernetes and Docker container options - suggest trying it!
The friendly community at Kube Simplify creates amazing free learning resources as live workshops to learn Kubernetes. I was happy to provide a workshop on "Kubernetes Observability" ... and exceeded time and content by diving into OpenTelemetry and Chaos Engineering in 3.5 hours. Check out the slides with exercises and project with solutions. If you want to learn more about Chaos Mesh, Flant wrote a great getting started article.
Diving into the Golden Signals for Kubernetes and answering Observability questions - interesting article from Isovalent with also tackling the problem of monitoring agent sidecar performance vs. eBPF. I'd love to see a "generic" way of collecting Observability data using eBPF and have the OpenTelemetry collector process, filter, correlate, and store the data - something that has been started in OpenTelemetry eBPF working group. Speaking of data, Yuri Shkuro, creator of Jaeger, wrote an insightful blog post about six pillars of Observability data: Metrics, the original pillar. Logs, the ancient pillar. Traces, the “new cool kid on the block” pillar. Events, the misunderstood pillar. Profiles, the geek pillar. Exceptions, the forgotten pillar. Now, wait until someone proposes netflow data types, eBPF, and more - the OpenTelemetry specification and framework allow for these proposals in the CNCF community, fortunately.
Last but not least, recommend these Container Days recordings as a learning resource:
- OpenTelemetry: The Vision, the Reality and How To Get Started - Dotan Horovits
- Cracking Microservices Performance Issues with Distributed Tracing - Dotan Horovits
- Distributed applications and Kubernetes - Matthias Haeussler
- Kubernetes made simpler - Using and developing Kubernetes Operators - Corvin Schapöhler
🔍 The inner Dev¶
wazero was mentioned in a tweet, claiming that is has zero dependencies as a WebAssembly runtime for Go developers, allowing to safely run code compiled in other languages by loading a wasm binary. The examples integrate well into your Go build chain and also provide WASI (WebAssembly System Interface) exercises.
📈 Your next project could be ...¶
- Add pets to your VS Code using this extension. Use
cmd+shift+pon macOS to open the console, search for
pet codingto start a new session, and repeat to e.g., throw a ball.
📚 Tools and tips for your daily use¶
- JSON Crack seamlessly visualizes JSON data into graphs. Available online or in a Docker container.
- Oh Shit, Git!?! helps solve tricky situations with Git, for when you want to use a time machine to revert.
- Remove Image Background using Python
- Covenant, a tool to generate SBOM (Software Bill of Material) from source code artifacts. Supported are .NET 5/6/Core, NPM.
- Pg-agent – a Postgres exporter for Prometheus focusing on query performance statistics
- GitLab CI/CD Recipes: Updating your Docker image automatically
- Eraser, cleaning up images from Kubernetes nodes
- Understanding Kubernetes in a visual way by Aurélie Vache.
- Reliable Machine Learning: Applying SRE Principles to ML in Production, new book by Cathy Chen, Niall Richard Murphy, Kranti Parisa, D. Sculley, Todd Underwood. Found via LinkedIn post from Niall Murphy.
- Best tech talks (Twitter thread July 2020) is a treasure trove for learning technology, as well as inspiration for own talks.
🎥 Events and CFPs¶
- Oct 24+25: KubeCon NA co-located events. Tip: Open Observability day, and SecurityCon/GitOpsCon where my teammate Dov Hershkovitch talks about JWT security in CI/CD.
- Oct 26-28: KubeCon NA in Detroit, Michigan.
- Nov 08-09: PromCon EU in Munich, Germany
- Nov 10-11: All Day DevOps, virtual
- Nov 16-17: Continuous Lifecycle / Container Conf in Mannheim, Germany
- Nov 28 - Dec 2: AWS re:Invent.
👋 CFPs due soon - 2023
- Feb 7-8: Civo Navigate in Tampa Bay, Florida. CFP closes Oct 18.
- Apr 17-21: KubeCon EU 2023 in Amsterdam. CFP closes Nov 18.
- June 26-28: Monitorama 2023 in Portland, OR. CFP closes Feb 3.
Looking for more CfPs? Try CFP Land.
Thanks, Forrest Brazeal, for a laugh on "The Re-Org Rag (I'm My Own VP)" - must watch!
Thanks for reading! If you are viewing the website archive, make sure to subscribe to stay in the loop!
PS: If you want to share items for the next newsletter, please check out the contributing guide - tag me in tweet replies or send me a DM. Thanks!