Skip to content

2023-02-21: The Inner Dev Learning AI/ML and eBPF, FluxCD, Overmind, Resource Efficiency, Container Days, Kubernetes Community Days, QCon London

Thanks for reading the web version, you can subscribe to the Ops In Dev newsletter to receive it in your mail inbox.

👋 Hey, lovely to see you again

Thanks for reading the Ops in Dev newsletter for 1 year now! I appreciate your submissions (feel free to reply to this email, or use the new submission form) and feedback on categories, for example, that the curated CFP & Events category really helps, or that readers get inspired by the tips & tools section. The interval will stay monthly: I thought of doing weekly or bi-weekly newsletters, but honestly, so many things going on, and I want to share the why and how with my own thoughts and take time for a thoughtful newsletter. You can also read it online on opsindev.news and use it as a searchable handbook.

My personal goal is to understand new technologies and explain them to beginners and experienced users so that everyone can contribute. eBPF brings me back to my time as OSS monitoring tool developer, yet I cannot fully explain eBPF in 5 minutes. Writing this newsletter helps :) AI/ML somehow inspires me to memories and "what if I had this 10 years ago?", and I want to explore ways of learning AI by turning back time (more soon). This newsletter iterates with a focus area on "🌱 The Inner Dev learning ..." with two main areas: eBPF and AI/ML.

Enjoy reading, and see you in March!

PS: If you are reading this newsletter in your browser, and you recently subscribed to the newsletter without getting an activation email - apologies, it was DNS that caused emails from Buttondown not to be sent. I have resent the activation reminder to everyone affected.

🌱 The Inner Dev learning ...

I believe that everyone has their inner developer, whether your role follows development, Ops, DevOps, SRE, etc. Understanding technology, creating visibility, explaining complex topics, asking critical questions, and solving problems is a team task. Let's dive into eBPF and AI/ML together!

🐝 The Inner Dev learning eBPF

Learning eBPF can be challenging and needs good resources to get started. ebpf.io does an amazing job, and was revamped to help with onboarding tutorials and use cases. It helped me find my learning path, too, amongst 100+ open tabs with interesting findings for my new talk about eBPF Chaos :-)

FOSDEM, Config Management Camp, and GitLab Contributor Days were amazing, lovely to meet friends old and new. I have created a new talk for Config Management Camp 2023, telling my eBPF learning story, how to debug production using eBPF, and verify reliability using chaos engineering (slides). It was the first time I installed a Linux rootkit in a VM to verify syscall hooking detection with tracee (bit scary ;)), and looked into tools such as Coroot or Inspektor Gadget. I also asked myself how to ensure security and code quality for eBPF programs in DevSecOps workflows:

With many companies and projects adopting eBPF, security and code quality in DevSecOps workflows will be crucial - to prevent performance regressions, and potential software supply chain attacks. All resources are collected on o11y.love, and in future blog posts.

Later in March, I'm excited to speak at QCon London in person, and share new ways to debug in production using eBPF and many practical tips in my new talk "From Monitoring to Observability: eBPF Chaos". You can join me using this special discount code: MichaelFQUK2350 - early bird runs until March 6th too.

🤖 The Inner Dev learning AI/ML

ChatGPT can do many things, and there are plenty of results (one my co-workers got confirmation that OpenAI uses GitLab :D). A more advanced idea is to run a virtual machine inside ChatGPT - mindblowing.

"Fundamentally, a GPT generates text given a prompt." -- GTP in 60 lines of NumPy is a great read - even if you are not deeply familiar with Python, NumPy, and neural network training. It helps to see terms and links in the use case context, and take notes to research.

A practical example for Observability alerts was created by Robusta, asking ChatGPT how to solve alerts raised in Kubernetes. AIOps is getting real again :-)

Google research introduced MusicLM, a model generating high-fidelity music from text descriptions. With Bing AI recently announced (and now limited to 5 responses to prevent weird responses), we will see more AI/ML content everywhere.

Let's close with using AI/ChatGPT to learn something new, like eBPF. Someone shared GTPTrace on Reddit, asking ChatGPT to write eBPF programs based on existing trained resources. Got something to play with :-)

🛡️ The Sec in Ops in Dev

Identifying running web services on domains can be challenging, and yet more difficult with security strategies. Monitor your web attack surface with GitLab CI/CD and GitLab Pages is a great tutorial about discovering these services in CI/CD pipelines, creating screenshots using Chromium headless, and uploading them to a secured GitLab Pages deployment to verify the results.

⛅ Cloud Native

GitLab will integrate FluxCD for GitOps with the agent for Kubernetes. I'm excited about this integration plan and look forward to adopting FluxCD and collaborating with the cloud-native community.

2023 will be the year of efficiency: Cloud-native cost efficiency, capacity planning, and forecasting helped with machine learning. Though, there are questions we can solve ourselves: What kind of Observability data do you need? Hubspot wrote a great blog post: "Saving Millions on Logging: Finding Relevant Savings", detailing their analysis: Starting with discovery and categorizing costs, correlating applications to AWS resources, by intercepting samples of application network calls and providing cost exploration data. The second step was attributing costs to functionality to identify heavy log types - request logs were three times the size of application logs for example. Storage costs have been identified as crucial to fix. The blog post dives deeper into raw log files, JSON or compressed logs and potential solutions, and teases with a future blog post.

Quickwit is a new open-source and cloud-native log management platform. It aims to be more cost-efficient, is written in Rust and provides Elasticsearch API compatibility. The architecture is built for Kubernetes, allowing to add or remove nodes quickly. I'm excited about the high-availability promises: "Highly reliable distributed search: each node can answer any query ... All your nodes are stateless, no more cluster babysitting". Glad to see more projects that solve the long-standing log management questions for cloud-native environments :-)

Clickhouse started a great learning series, highlighting how their OLAP database can be used for storing Observability data, specifically logs. The architecture involves the OpenTelemetry Collector with a Clickhouse exporter, alternative agents explored are Fluentd, Fluentbit and Vector. The article focusses on storage schemas and efficient queries, and how to visualize collected logs.

👁️ Observability

Tracee got cgroup based pcap capture support. One of the maintainers also started a great use case thread: Track all DNS requests/responses, capture HTTP packets, trace all TCP/UDP, ICMP/ICMPv6 packets, trace all network events/packets from containers. Great to see more debugging tools using eBPF!

Unknown unknowns and how to know them by Dylan Ratcliffe also shows a demo of their new product Overmind. It follows an often overlooked aspect - infrastructure inventory and auto-discovery, resource relations and more. Looking forward to see what Dylan, David and the team are building.

Caretta is a lightweight, standalone tool to create a visual service map in a Kubernetes cluster. It uses eBPF to map service network interactions (similar to Coroot) and integrates with Grafana as a visualization layer.

📚 Tools and tips for your daily use

  • fd is a simpler alternative to find. Extremly fast with parallel search, understands .gitignore files, etc.
  • beetrace allows you to trace a Python process line by line or the functions' entries and returns. It uses USDT(User Statically-Defined Tracing) probes with bpftrace.
  • Odigos is an Observability Control Plane that allows to generate distributed traces instantly for any application without code changes.
  • trace-vis is an online OpenTelemetry trace visualization tool.
  • Sloth shows all open files, directories, sockets, pipes and devices used by all running processes on macOS, as GUI for lsof. Thanks for the tip, Andreas Lehr!
  • HiDock is a macOS app to set different Dock settings on different displays.
  • git rebase ... and merge conflicts. I tried Git-Sim to simulate git operations, great help!

🔖 Book'mark

🎯 Release speed-run

Homebrew 4.0.0 brings significantly faster tap updates by using JSON files downloaded from formulae.brew.sh. The auto-update interval has been reducated to 25h instead of 5 minutes. The analytics feature will migrate away from Google Analytics and use a self-hosted InfluxDB instance hosted in the EU. Consider donating to Homebrew if it helped you become more efficient :-) Zed v0.73.3 supports LUA and YAML syntax highlighting and intelli-sense suggestions. Rust 1.67.0 and Go 1.20 provides performance and maintainability improvements.

GitLab 15.8, with GitLab 15.9 just around the corner on 2023-02-22, bringing more improvements to the new tasks in issues - you can convert Markdown checklist items to a task for example. Perses 0.23.0 adds support for creating/deleting dashboards using the UI. OpenSearch 2.5 improves the index management UI, adds support for analyzing Jaeger trace data in OpenSearch Dashboards, provides Debian/Ubuntu packages and more. Cilium v1.13.0 provides a fully conformant Gateway API implementation in Kubernetes clusters, L7 load balancing for Kubernetes services (to help solve gRPC load balancing), BIG TCP to support high throughput environments, and much more. And: All Cilium & Tetragon container images are signed using cosign. cilium/ebpf-go v0.10.0 supports BTF Marshaling.

🎥 Events and CFPs

👋 CFPs due soon

Looking for more CfPs?

🎤 Shoutouts

Aurélie Vache and Horacio Gonzalez for showing me how to take notes and create sketches on my new iPad Pro using the Apple pencil. Experimenting since we met at Config Management Camp :)

Ops In Dev newsletter sketch for learning eBPF, AI and more

🌐

Thanks for reading! If you are viewing the website archive, make sure to subscribe to stay in the loop!

See you next month - let me know what you think on LinkedIn, Twitter, Mastodon.

Cheers,

Michael

PS: If you want to share items for the next newsletter, please check out the contributing guide - tag me in the comments, send me a DM or submit this form. Thanks!