2023-05-08: KubeCon EU, eBPF 201, Cilium, k8sgpt, Honeycomb query assistant, Security Chaos engineering, Kepler, ARM Assembly Internals¶
Thanks for reading the web version, you can subscribe to the Ops In Dev newsletter to receive it in your mail inbox.
👋 Hey, lovely to see you again¶
It's been a while, and with holidays, busy work, and KubeCon EU excitement, I decided to merge the late April and early May issues into one newsletter. It won't double the length, since many topics are now much more refined after great conversations at KubeCon EU, and re-reading some articles.
🌱 The Inner Dev learning ...¶
🌤️ KubeCon EU 2023¶
KubeCon EU 2023 was a great event to dive into conversations about eBPF for better Observability, and AI for more efficiency. Specifically, CI/CD and cloud resources were also a theme - I gave a lightning talk at the GitLab booth about "Efficient DevSecOps Pipelines in a Cloud Native World".
If you have collected your KubeCon EU talk schedule on sched.com (my schedule), the event organizers have embedded the YouTube videos directly for better viewing. Otherwise, you can visit the main event playlist and search for your favorite topics and speakers. My highlights are:
- Choose Your Own Adventure: The Treacherous Trek to Development
- Back to the Future: Next-Generation Cloud Native Security
- Love, Death and Robots - with Wasm & K8s on Boston Dynamics Spot
- The Day We Delete(d) Production (at CERN)
More insightful talks are added in the following newsletter sections. The zero-day events sessions are uploaded too, for example the Observability Day EU 2023. More KubeCon coverage from Chris Chinchilla, white duck, Mauricio Salatino, Rich Burroughs, Daniel Bryant, Twitter thread by Bartłomiej Płotka, Optimize All The Things podcast episode.
🐝 The Inner Dev learning eBPF¶
Dive into the latest updates from KubeCon EU:
- Past, Present, & Future of eBPF in Cloud Native Observability
- eBPF for Observability: The Good, the Bad, and the Ugly
- eBPF 201: Supercharging Your eBPF Dev Process for Cloud Native Apps
Testing eBPF programs on different Linux kernel architectures can ba a challenge, similar to CI/CD pipeline testing. This article shows how to use Buildroot VMs, explains the disadvantages with Vagrant and how the project Architest came to life.
Brendan Gregg said that "eBPF Observability Tools are not Security Tools" in a new blog post. The main point is that observability tools strive to reduce the overhead with eBPF, and how security tooling can benefit from this idea, for example Cilium Tetragon.
Cilium was all over KubeCon, Bill Mulligan wrote about its history in this article. The following KubeCon EU highlights are worth watching if you want to dive deeper into Cilium, too.
- Cilium Updates, News, Roadmap, and in the Wild
- Effortless Open Source Observability with Cilium, Prometheus and Grafana - LGTM!
- Keeping It Simple: Cilium Networking for Multicloud Kubernetes
🤖 The Inner Dev learning AI/ML¶
Everyone talks about AI, and the problems to solve, finding use cases every day. Are they all valid, and wouldn't it be more efficient to do "manual" operations instead? This Twitter discussion about a call schedule AI assistance is a good reminder. Kelsey Hightower said:
If you're using generative AI tools to write infrastructure code, or YAML files, that's a hint you're working at the wrong level of abstraction. Instead of guessing, consider using a template to abstract away the details, and only expose the decisions a human needs to make.
In a similar manner, this r/ProgrammerHumor comic thinks about developers changing their debugging strategies, and time investments.
A great way to recap the rapid AI evolution is found in 2023 State of AI in 14 Charts. Before we dive into the benefits of AI examples, let's stop for a moment and look into where attackers are at with AI - prompt injection for large language models.
Chart-GPT builds beautiful charts based on text input. The project is open source and can be helpful to learn the interaction with the OpenAI API. Turbopilot is a self-hosted copilot clone as a proof of concept that uses the Salesforce CodeGen models. Raycast introduced Quick AI, accessing AI integrations while already working fast with launching apps, searching emojis, doing translations, etc. (macOS Spotlight replacement). GitLab started ML experiments to integrate AI use cases into DevSecOps workflows. This goes beyond code suggestions - one of my personal highlights is the ability to explain security vulnerabilities using AI. As a developer, reading through CVE lists and often unknown code, understanding and also long-term fixing security vulnerabilities can be challenging.
Honeycomb released their query assistant, using generative AI to build a natural language querying experience into their product.
k8sgpt is a tool for scanning your Kubernetes clusters, diagnosing and triaging issues with the help of AI explaining the problems. If you are approaching learning AI from the DevOps/SRE perspective, the implementation parts with AI can help create your sandbox tests with AI interfaces. Introducing a problem in a Kubernetes cluster can be done using security chaos engineering (read more in the next section).
🛡️ The Sec in Ops in Dev¶
The Netflix engineering team published an interesting read about "Migrating Critical Traffic at Scale with No Downtime - Part 1". They use replay traffic testing to clone/fork production traffic, allowing to run sandboxed testing at scale, including operational insights from a simulated production environment.
Security chaos engineering is getting more attention - I recently touched on it in my eBPF Chaos talks too. How can you verify that all security policies and safeguards really work? Anaïs Urlichs started a great Twitter thread, showing a misconfigured Kubernetes controller that security tooling should be able to detect. She shares a new book on "Security Chaos Engineering" to learn more, and then shows how the TrivyOperator is able to detect the misconfigured controller. Another way can be to attack Kubernetes and see how the security tooling behaves. The ultimate way is described in "69 Ways to F*** Up Your Deploy" -
8. Former employees can still deploy. sounds familiar from previous jobs ;)
Weaveworks created Flamingo - the Flux subsystem for Argo CD. Interesting approach to combine the best of both projects into a better GitOps experience for Kubernetes clusters. Keycloak joins CNCF as an incubating project. It is an Identity and Access Management (IAM) solution providing centralized authentication and authorization to applications and APIs.
There were a lot of interesting KubeCon EU talks, here's a mix of security and ops focussed highlights:
- From SBOMs to IBOMs - Know What's Happening in Your Clusters
- Malicious Compliance: Reflections on Trusting Container Scanners
- Anatomy of a Cloud Security Breach - 7 Deadly Sins
- Hacking & Defending Kubernetes Clusters: We'll Do It LIVE!!
- No Fear, Falco Is Looking After Us!
- Automating Configuration and Permissions Testing for GitOps with OPA Conftest
- Kyverno Introduction and Deep Dive
- Kubernetes Defensive Monitoring with Prometheus
- Chaos Engineering 2023 with Chaos Mesh
- Breakpoints in Your Pod: Interactively Debugging Kubernetes Applications
- How to Make Kubernetes Rhyme with Prod-Readiness
Akita is now in open beta, and offers observability for APIs. It comes with support for container platforms and Kubernetes, and adds metrics and error dashboards, API auto-discovery, and Chrome/Docker Desktop extensions. eBPF powers it and does not require code changes. Saiyam Pathak created a new learning tutorial for Groundcover, showing Kubernetes observability. It provides 5 scenarios to troubleshoot and show the feature set. Worth watching, or bookmarking for later. Groundcover also released their eBPF-powered agent, Flora, comparing the benchmarks to Datadog. Observable Frontends: the State of OpenTelemetry in the Browser asks the important questions - OpenTelemetry works well with web server instrumentation, but what about the clients? With the example of a React web client, the article walks through different approaches and potential solutions with OpenTelemetry.
I highly recommend watching the latest KubeCon EU project updates ...
- Defining A Common Observability Query Language and Other Observability TAG Updates
- Prometheus Updates and Deep Dive
- OpenTelemetry: Using Unified Semantics to Drive Insights + Project Update
- OTel Me About Metrics: A Metrics 101 Crash Course
- Jaeger: The Future with OpenTelemetry and Metrics
- SIG Instrumentation Introduction and Deep Dive - structured logging, tracing, etc. in Kubernetes and CNCF projects
... and practical insights and tips for more day-2-ops:
- Tutorial: Exploring the Power of OpenTelemetry on Kubernetes
- Tutorial: Building an Open Source Observability Stack
- Making Sense of Your Vital Signals: The Future of Pod and Containers Monitoring
- What Could Go Wrong with a GraphQL Query and Can OpenTelemetry Help?
- Smarter Golden Signals!
- Surviving Day 2 - How to Troubleshoot Kubernetes Networking
If you understand German, I've joined the programmier.bar podcast for a 1+ hour episode about Observability (German).
📚 Tools and tips for your daily use¶
- Atuin replaces your existing shell history with a SQLite database, and records additional context for your commands. Additionally, it provides optional and fully encrypted synchronisation of your history between machines, via an Atuin server.
- Chrome Dev Tools now allow to override response headers including CORS, to experiment locally with HTTP response headers.
- Parallelizing Cucumber tests on GitLab runners, How we took our build time from 1 hour to 25 minutes in an afternoon
- q helps surface Linux networking metrics with eBPF.
- Kepler (Kubernetes-based Efficient Power Level Exporter) uses eBPF to probe energy related system stats and exports as Prometheus metrics.
- Run a command for a specific time in Linux: timeout or timelimit.
- Out of a thread about Git CLI command typos,
git config --global help.autocorrect 20was one of the tips. Alternatively, the NSFW edition for a more broad range of commands.
- Security Chaos Engineering by Kelly Shortridge and Aaron Rinehart
- Blue Fox: Arm Assembly Internals and Reverse Engineering by Maria Markstedter
- Intro to Kubernetes - containers at scale by Kaslin Fields
🎯 Release speed-run¶
In Rust 1.69.0, Cargo now suggests to automatically fix some warnings. At KubeCon EU, the OpenSSF released the spec for SLSA v1.0 which now includes the requirement to prove provenance, not only how to create it. The Prometheus remote-write spec 1.0 was published. Prometheus Operator v0.64.0 supports the Prometheus agent mode. Pyrra v0.6.0 supports visualization of burn rates in graphs. Moby 23.0 brings experimental support for CSI drivers, introduces alternative OCI runtime support, uses BuildKit by default, and significantly improves health checks for containers. GitLab 15.11 brings Code Suggestions in Beta, Google Play Store integration for release pipelines, define inputs for included CI/CD config, Kubernetes 1.26 support.
🎥 Events and CFPs¶
- May 10-12: Open Source Summit NA 2023, Vancouver, Canada.
- May 15-18: SLOConf 2023, virtual.
- Jun 15: Kubernetes Community Days Zurich in Zurich, Switzerland.
- Jun 20-23: CloudLand 2023, at Phantasialand, Germany. See you there, 2 talks accepted!
- Jun 26-28: Monitorama 2023 in Portland, OR.
- Jul 17-18: Kubernetes Community Days Munich, in Munich, Germany.
- Aug 9-10: DevOpsDays Chicago 2023 in Chicago, IL.
- Sep 11-13: Container Days EU 2023 in Hamburg, Germany. See you there!
- Oct 2-6: DEVOXX Belgium, Antwerp, Belgium.
- Oct 6-7: DevOps Camp Nuremberg, Nuremberg, Germany. See you there!
- Oct 10-12: SRECON EMEA in Dublin, Ireland.
- Oct 17-18: Kubernetes Community Days UK in London, UK.
- Nov 6-9: KubeCon NA 2023, Chicago, IL. Planning to be there.
- Nov 16-17: Continuous Lifecycle / Container Conf in Mannheim, Germany.
👋 CFPs due soon
- Oct 10-12: SRECON EMEA in Dublin, Ireland. CFP closes May 23.
- Oct 17-18: Kubernetes Community Days UK in London, UK. CFP closes July 2.
- Nov 6-9: KubeCon NA 2023, Chicago, IL. CFP closes June 18.
Looking for more CfPs?
Thanks for reading! If you are viewing the website archive, make sure to subscribe to stay in the loop!