2023-02-21: The Inner Dev Learning AI/ML and eBPF, FluxCD, Overmind, Resource Efficiency, Container Days, Kubernetes Community Days, QCon London¶
Thanks for reading the web version, you can subscribe to the Ops In Dev newsletter to receive it in your mail inbox.
👋 Hey, lovely to see you again¶
Thanks for reading the Ops in Dev newsletter for 1 year now! I appreciate your submissions (feel free to reply to this email, or use the new submission form) and feedback on categories, for example, that the curated CFP & Events category really helps, or that readers get inspired by the tips & tools section. The interval will stay monthly: I thought of doing weekly or bi-weekly newsletters, but honestly, so many things going on, and I want to share the why and how with my own thoughts and take time for a thoughtful newsletter. You can also read it online on opsindev.news and use it as a searchable handbook.
My personal goal is to understand new technologies and explain them to beginners and experienced users so that everyone can contribute. eBPF brings me back to my time as OSS monitoring tool developer, yet I cannot fully explain eBPF in 5 minutes. Writing this newsletter helps :) AI/ML somehow inspires me to memories and "what if I had this 10 years ago?", and I want to explore ways of learning AI by turning back time (more soon). This newsletter iterates with a focus area on "🌱 The Inner Dev learning ..." with two main areas: eBPF and AI/ML.
Enjoy reading, and see you in March!
PS: If you are reading this newsletter in your browser, and you recently subscribed to the newsletter without getting an activation email - apologies, it was DNS that caused emails from Buttondown not to be sent. I have resent the activation reminder to everyone affected.
🌱 The Inner Dev learning ...¶
I believe that everyone has their inner developer, whether your role follows development, Ops, DevOps, SRE, etc. Understanding technology, creating visibility, explaining complex topics, asking critical questions, and solving problems is a team task. Let's dive into eBPF and AI/ML together!
🐝 The Inner Dev learning eBPF¶
Learning eBPF can be challenging and needs good resources to get started. ebpf.io does an amazing job, and was revamped to help with onboarding tutorials and use cases. It helped me find my learning path, too, amongst 100+ open tabs with interesting findings for my new talk about eBPF Chaos :-)
FOSDEM, Config Management Camp, and GitLab Contributor Days were amazing, lovely to meet friends old and new. I have created a new talk for Config Management Camp 2023, telling my eBPF learning story, how to debug production using eBPF, and verify reliability using chaos engineering (slides). It was the first time I installed a Linux rootkit in a VM to verify syscall hooking detection with tracee (bit scary ;)), and looked into tools such as Coroot or Inspektor Gadget. I also asked myself how to ensure security and code quality for eBPF programs in DevSecOps workflows:
- Harnessing the eBPF verifier is a great read on the challenges testing eBPF programs in CI/CD.
- Bypassing eBPF-based Security Enforcement Tools illustrates the cat-and-mouse challenges with security observability and attackers.
- Measuring CPU usage of eBPF programs with Inspektor Gadget aims to provide answers on the performance/resource add for running eBPF programs additionally.
With many companies and projects adopting eBPF, security and code quality in DevSecOps workflows will be crucial - to prevent performance regressions, and potential software supply chain attacks. All resources are collected on o11y.love, and in future blog posts.
Later in March, I'm excited to speak at QCon London in person, and share new ways to debug in production using eBPF and many practical tips in my new talk "From Monitoring to Observability: eBPF Chaos". You can join me using this special discount code:
MichaelFQUK2350 - early bird runs until March 6th too.
🤖 The Inner Dev learning AI/ML¶
ChatGPT can do many things, and there are plenty of results (one my co-workers got confirmation that OpenAI uses GitLab :D). A more advanced idea is to run a virtual machine inside ChatGPT - mindblowing.
"Fundamentally, a GPT generates text given a prompt." -- GTP in 60 lines of NumPy is a great read - even if you are not deeply familiar with Python, NumPy, and neural network training. It helps to see terms and links in the use case context, and take notes to research.
A practical example for Observability alerts was created by Robusta, asking ChatGPT how to solve alerts raised in Kubernetes. AIOps is getting real again :-)
Google research introduced MusicLM, a model generating high-fidelity music from text descriptions. With Bing AI recently announced (and now limited to 5 responses to prevent weird responses), we will see more AI/ML content everywhere.
Let's close with using AI/ChatGPT to learn something new, like eBPF. Someone shared GTPTrace on Reddit, asking ChatGPT to write eBPF programs based on existing trained resources. Got something to play with :-)
🛡️ The Sec in Ops in Dev¶
Identifying running web services on domains can be challenging, and yet more difficult with security strategies. Monitor your web attack surface with GitLab CI/CD and GitLab Pages is a great tutorial about discovering these services in CI/CD pipelines, creating screenshots using Chromium headless, and uploading them to a secured GitLab Pages deployment to verify the results.
⛅ Cloud Native¶
GitLab will integrate FluxCD for GitOps with the agent for Kubernetes. I'm excited about this integration plan and look forward to adopting FluxCD and collaborating with the cloud-native community.
2023 will be the year of efficiency: Cloud-native cost efficiency, capacity planning, and forecasting helped with machine learning. Though, there are questions we can solve ourselves: What kind of Observability data do you need? Hubspot wrote a great blog post: "Saving Millions on Logging: Finding Relevant Savings", detailing their analysis: Starting with discovery and categorizing costs, correlating applications to AWS resources, by intercepting samples of application network calls and providing cost exploration data. The second step was attributing costs to functionality to identify heavy log types - request logs were three times the size of application logs for example. Storage costs have been identified as crucial to fix. The blog post dives deeper into raw log files, JSON or compressed logs and potential solutions, and teases with a future blog post.
Quickwit is a new open-source and cloud-native log management platform. It aims to be more cost-efficient, is written in Rust and provides Elasticsearch API compatibility. The architecture is built for Kubernetes, allowing to add or remove nodes quickly. I'm excited about the high-availability promises: "Highly reliable distributed search: each node can answer any query ... All your nodes are stateless, no more cluster babysitting". Glad to see more projects that solve the long-standing log management questions for cloud-native environments :-)
Clickhouse started a great learning series, highlighting how their OLAP database can be used for storing Observability data, specifically logs. The architecture involves the OpenTelemetry Collector with a Clickhouse exporter, alternative agents explored are Fluentd, Fluentbit and Vector. The article focusses on storage schemas and efficient queries, and how to visualize collected logs.
Tracee got cgroup based pcap capture support. One of the maintainers also started a great use case thread: Track all DNS requests/responses, capture HTTP packets, trace all TCP/UDP, ICMP/ICMPv6 packets, trace all network events/packets from containers. Great to see more debugging tools using eBPF!
Unknown unknowns and how to know them by Dylan Ratcliffe also shows a demo of their new product Overmind. It follows an often overlooked aspect - infrastructure inventory and auto-discovery, resource relations and more. Looking forward to see what Dylan, David and the team are building.
Caretta is a lightweight, standalone tool to create a visual service map in a Kubernetes cluster. It uses eBPF to map service network interactions (similar to Coroot) and integrates with Grafana as a visualization layer.
📚 Tools and tips for your daily use¶
- fd is a simpler alternative to
find. Extremly fast with parallel search, understands
- beetrace allows you to trace a Python process line by line or the functions' entries and returns. It uses USDT(User Statically-Defined Tracing) probes with bpftrace.
- Odigos is an Observability Control Plane that allows to generate distributed traces instantly for any application without code changes.
- trace-vis is an online OpenTelemetry trace visualization tool.
- Sloth shows all open files, directories, sockets, pipes and devices used by all running processes on macOS, as GUI for lsof. Thanks for the tip, Andreas Lehr!
- HiDock is a macOS app to set different Dock settings on different displays.
- git rebase ... and merge conflicts. I tried Git-Sim to simulate git operations, great help!
- Awesome eBPF, a curated list of awesome projects related to eBPF.
- Interesting talks on Observability from FOSDEM 2023. Thanks for sharing Prathamesh Sonpatki!
- 5 great books about containers for busy architects on the Red Hat blog by Markus Eisele.
- Understanding Docker in a visual way by Aurélie Vache
- A practical guide for implementing SLO. Thanks for sharing Prathamesh Sonpatki!
🎯 Release speed-run¶
Homebrew 4.0.0 brings significantly faster tap updates by using JSON files downloaded from formulae.brew.sh. The auto-update interval has been reducated to 25h instead of 5 minutes. The analytics feature will migrate away from Google Analytics and use a self-hosted InfluxDB instance hosted in the EU. Consider donating to Homebrew if it helped you become more efficient :-) Zed v0.73.3 supports LUA and YAML syntax highlighting and intelli-sense suggestions. Rust 1.67.0 and Go 1.20 provides performance and maintainability improvements.
GitLab 15.8, with GitLab 15.9 just around the corner on 2023-02-22, bringing more improvements to the new tasks in issues - you can convert Markdown checklist items to a task for example. Perses 0.23.0 adds support for creating/deleting dashboards using the UI. OpenSearch 2.5 improves the index management UI, adds support for analyzing Jaeger trace data in OpenSearch Dashboards, provides Debian/Ubuntu packages and more. Cilium v1.13.0 provides a fully conformant Gateway API implementation in Kubernetes clusters, L7 load balancing for Kubernetes services (to help solve gRPC load balancing), BIG TCP to support high throughput environments, and much more. And: All Cilium & Tetragon container images are signed using cosign. cilium/ebpf-go v0.10.0 supports BTF Marshaling.
🎥 Events and CFPs¶
- Feb 23-24: Kubernetes Community Days Amsterdam, Amsterdam, The Netherlands.
- Mar 8: Container Day Security
- Mar 15-16: Chaos Carnival, virtual, see you there!
- Mar 27-29: QCon London, London, UK. See you there!
- Apr 15-16: Cloud-Native Rejekts EU 2023, Amsterdam, The Netherlands.
- Apr 17-21: KubeCon EU, Amsterdam, The Netherlands. See you there!
- May 10-12: Open Source Summit NA 2023, Vancouver, Canada.
- May 15-18: SLOConf 2023, virtual.
- Jun 15: Kubernetes Community Days Zurich in Zurich, Switzerland.
- Jun 20-23: CloudLand 2023, at Phantasialand, Germany.
- Jun 26-28: Monitorama 2023 in Portland, OR.
- Jul 17-18: Kubernetes Community Days Munich, in Munich, Germany
- Sep 11-13: Container Days EU 2023 in Hamburg, Germany.
- Aug 9-10: DevOpsDays Chicago 2023 in Chicago, IL.
- Oct 2-6: DEVOXX Belgium, Antwerp, Belgium
- Nov 6-9: KubeCon NA 2023, Chicago, IL
👋 CFPs due soon
- Jun 15: Kubernetes Community Days Zurich, Zurich, Switzerland. CFP closes Mar 31.
- Jul 17-18: Kubernetes Community Days Munich, in Munich, Germany. CFP closes Apr 30.
- Aug 9-10: DevOpsDays Chicago 2023 in Chicago, IL. CFP closes Mar 17.
- Sep 11-13 : Container Days EU 2023 in Hamburg, Germany, CFP closes Mar 31.
Looking for more CfPs?
Thanks for reading! If you are viewing the website archive, make sure to subscribe to stay in the loop!