Skip to content

2023-05-08: KubeCon EU, eBPF 201, Cilium, k8sgpt, Honeycomb query assistant, Security Chaos engineering, Kepler, ARM Assembly Internals

Thanks for reading the web version, you can subscribe to the Ops In Dev newsletter to receive it in your mail inbox.

πŸ‘‹ Hey, lovely to see you again

It's been a while, and with holidays, busy work, and KubeCon EU excitement, I decided to merge the late April and early May issues into one newsletter. It won't double the length, since many topics are now much more refined after great conversations at KubeCon EU, and re-reading some articles.

🌱 The Inner Dev learning ...

🌀️ KubeCon EU 2023

KubeCon EU 2023 was a great event to dive into conversations about eBPF for better Observability, and AI for more efficiency. Specifically, CI/CD and cloud resources were also a theme - I gave a lightning talk at the GitLab booth about "Efficient DevSecOps Pipelines in a Cloud Native World".

If you have collected your KubeCon EU talk schedule on sched.com (my schedule), the event organizers have embedded the YouTube videos directly for better viewing. Otherwise, you can visit the main event playlist and search for your favorite topics and speakers. My highlights are:

More insightful talks are added in the following newsletter sections. The zero-day events sessions are uploaded too, for example the Observability Day EU 2023. More KubeCon coverage from Chris Chinchilla, white duck, Mauricio Salatino, Rich Burroughs, Daniel Bryant, Twitter thread by BartΕ‚omiej PΕ‚otka, Optimize All The Things podcast episode.

🐝 The Inner Dev learning eBPF

Dive into the latest updates from KubeCon EU:

Testing eBPF programs on different Linux kernel architectures can ba a challenge, similar to CI/CD pipeline testing. This article shows how to use Buildroot VMs, explains the disadvantages with Vagrant and how the project Architest came to life.

Brendan Gregg said that "eBPF Observability Tools are not Security Tools" in a new blog post. The main point is that observability tools strive to reduce the overhead with eBPF, and how security tooling can benefit from this idea, for example Cilium Tetragon.

Cilium was all over KubeCon, Bill Mulligan wrote about its history in this article. The following KubeCon EU highlights are worth watching if you want to dive deeper into Cilium, too.

πŸ€– The Inner Dev learning AI/ML

Everyone talks about AI, and the problems to solve, finding use cases every day. Are they all valid, and wouldn't it be more efficient to do "manual" operations instead? This Twitter discussion about a call schedule AI assistance is a good reminder. Kelsey Hightower said:

If you're using generative AI tools to write infrastructure code, or YAML files, that's a hint you're working at the wrong level of abstraction. Instead of guessing, consider using a template to abstract away the details, and only expose the decisions a human needs to make.

In a similar manner, this r/ProgrammerHumor comic thinks about developers changing their debugging strategies, and time investments.

Days before and after AI - credits to r/ProgrammerHumor

A great way to recap the rapid AI evolution is found in 2023 State of AI in 14 Charts. Before we dive into the benefits of AI examples, let's stop for a moment and look into where attackers are at with AI - prompt injection for large language models.

Chart-GPT builds beautiful charts based on text input. The project is open source and can be helpful to learn the interaction with the OpenAI API. Turbopilot is a self-hosted copilot clone as a proof of concept that uses the Salesforce CodeGen models. Raycast introduced Quick AI, accessing AI integrations while already working fast with launching apps, searching emojis, doing translations, etc. (macOS Spotlight replacement). GitLab started ML experiments to integrate AI use cases into DevSecOps workflows. This goes beyond code suggestions - one of my personal highlights is the ability to explain security vulnerabilities using AI. As a developer, reading through CVE lists and often unknown code, understanding and also long-term fixing security vulnerabilities can be challenging.

Honeycomb released their query assistant, using generative AI to build a natural language querying experience into their product.

k8sgpt is a tool for scanning your Kubernetes clusters, diagnosing and triaging issues with the help of AI explaining the problems. If you are approaching learning AI from the DevOps/SRE perspective, the implementation parts with AI can help create your sandbox tests with AI interfaces. Introducing a problem in a Kubernetes cluster can be done using security chaos engineering (read more in the next section).

πŸ›‘οΈ The Sec in Ops in Dev

The Netflix engineering team published an interesting read about "Migrating Critical Traffic at Scale with No Downtime - Part 1". They use replay traffic testing to clone/fork production traffic, allowing to run sandboxed testing at scale, including operational insights from a simulated production environment.

Security chaos engineering is getting more attention - I recently touched on it in my eBPF Chaos talks too. How can you verify that all security policies and safeguards really work? AnaΓ―s Urlichs started a great Twitter thread, showing a misconfigured Kubernetes controller that security tooling should be able to detect. She shares a new book on "Security Chaos Engineering" to learn more, and then shows how the TrivyOperator is able to detect the misconfigured controller. Another way can be to attack Kubernetes and see how the security tooling behaves. The ultimate way is described in "69 Ways to F*** Up Your Deploy" - 8. Former employees can still deploy. sounds familiar from previous jobs ;)

Weaveworks created Flamingo - the Flux subsystem for Argo CD. Interesting approach to combine the best of both projects into a better GitOps experience for Kubernetes clusters. Keycloak joins CNCF as an incubating project. It is an Identity and Access Management (IAM) solution providing centralized authentication and authorization to applications and APIs.

There were a lot of interesting KubeCon EU talks, here's a mix of security and ops focussed highlights:

πŸ‘οΈ Observability

Akita is now in open beta, and offers observability for APIs. It comes with support for container platforms and Kubernetes, and adds metrics and error dashboards, API auto-discovery, and Chrome/Docker Desktop extensions. eBPF powers it and does not require code changes. Saiyam Pathak created a new learning tutorial for Groundcover, showing Kubernetes observability. It provides 5 scenarios to troubleshoot and show the feature set. Worth watching, or bookmarking for later. Groundcover also released their eBPF-powered agent, Flora, comparing the benchmarks to Datadog. Observable Frontends: the State of OpenTelemetry in the Browser asks the important questions - OpenTelemetry works well with web server instrumentation, but what about the clients? With the example of a React web client, the article walks through different approaches and potential solutions with OpenTelemetry.

I highly recommend watching the latest KubeCon EU project updates ...

... and practical insights and tips for more day-2-ops:

If you understand German, I've joined the programmier.bar podcast for a 1+ hour episode about Observability (German).

πŸ“š Tools and tips for your daily use

πŸ”– Book'mark

🎯 Release speed-run

In Rust 1.69.0, Cargo now suggests to automatically fix some warnings. At KubeCon EU, the OpenSSF released the spec for SLSA v1.0 which now includes the requirement to prove provenance, not only how to create it. The Prometheus remote-write spec 1.0 was published. Prometheus Operator v0.64.0 supports the Prometheus agent mode. Pyrra v0.6.0 supports visualization of burn rates in graphs. Moby 23.0 brings experimental support for CSI drivers, introduces alternative OCI runtime support, uses BuildKit by default, and significantly improves health checks for containers. GitLab 15.11 brings Code Suggestions in Beta, Google Play Store integration for release pipelines, define inputs for included CI/CD config, Kubernetes 1.26 support.

πŸŽ₯ Events and CFPs

πŸ“£ Developer Conferences Agenda by AurΓ©lie Vache got a nice visual date table update.

πŸ‘‹ CFPs due soon

Looking for more CfPs?

🎀 Shoutouts

xkcd 1838 - Machine Learning.

xkcd 1838 - Machine Learning

🌐

Thanks for reading! If you are viewing the website archive, make sure to subscribe to stay in the loop!

See you next month - let me know what you think on LinkedIn, Twitter, Mastodon.

Cheers,

Michael

PS: If you want to share items for the next newsletter, please check out the contributing guide - tag me in the comments, send me a DM or submit this form. Thanks!