Skip to content

2022-09-13: Container Days, eBPF and OpenTelemetry everywhere, Kubernetes Observability, SBOM Operator, Bomber, Podman as GitLab executor, WebAssembly WASI

Thanks for reading the web version, you can subscribe to the Ops In Dev newsletter to receive it in your mail inbox.

👋 Hey, lovely to see you again

We've made it to 100+ subscribers. Thanks, everyone, for reading and sharing valuable content and conversations! I was at Container Days in Hamburg, Germany, last week with great talks, ideas, and impressions - read on in the hot topics section this time. The learning curve in this newsletter follows a path - from security to container builds to eBPF as the hottest topic, to more OpenTelemetry and observability-driven workflows - to some tools and tips that I have found very useful, or got a tip from friends. I've tried to capture complex topics such as WebAssembly too. Choose your own pace to read, learn and iterate :-)

☕ Hot Topics

📦 Container Days Event Recap

Container Days felt like a smaller KubeCon event, with many engaged folks in cloud-native, Kubernetes, security, and observability topics, coming together in Northern Germany. Hamburg also provides real big container ships, thanks, Nico, for the impressions.

Real containers in Hamburg, Germany

The event was packed with four parallel tracks to choose from and left room for many conversations in-between. Day 1 ended with a relaxed after-party, reserving energy for a fully packed day 2. I remember talking a lot, especially with Dotan Horovits after his amazing Distributed Tracing talk, losing my voice in my talk, watching fascinating demos from Nico Meisenzahl hijacking Kubernetes and Liz Rice using eBPF for security observability, and of course meeting new folks and interesting projects :-)

Stormforge follows a similar approach to kubecost/infracost, combined with performance testing, and aiming to reduce Kubernetes infrastructure costs. Interesting combination and product; reach out to Johanna Luetgebrune to learn more. I've also taken note to try the Kubermatic Kubernetes Platform, announcing release 2.21. New friends from Novatec brought me into the world of event-driven architecture by Mirna Alaisami, Azure, Java, Camunda, Kafka, and new ways to look at Observability in Kubernetes by Matthias Haeussler - thanks for the great conversations :)

Post-event, the Hamburg cloud-native meetup organized a session where Nico Meisenzahl and Philip Welz shared how to prevent your Kubernetes cluster from being hacked. The slides are a wealth of knowledge and ideas. Philip reproduced the Cilium Tetragon demo from Liz Rice's Container Days talk by defining a policy that triggers when someone malicious edits /etc/passwd, which is very impressive.

The Container Days talk recordings will be available on their YouTube channel soon, I recommend subscribing. Meanwhile, check out a few folks to follow on Twitter. See you next year in Hamburg!

🎯 Release speed-run

Podman v4.2.0 brings support for GitLab Runner as Docker executor, v4.2.1 is the latest bugfix release. Prometheus v2.38.0 provides support for pretty-formatting PromQL expressions in the UI and API. The first bugfix release for the LTS version is available too: v2.37.1. Chaos Mesh v2.3.0 adds support for BlockChaos filesystem experiments, v2.3.1 is the latest bugfix release.

Kubernetes 1.25 graduates PodSecurity Admission to stable. The release is followed by k3s v1.25.0. Make sure to follow the upgrade notes. kube-state-metrics v2.6.0 brings metrics for RBAC resources, and the custom resource state metrics enhanced its features.

Trivy v0.31 supports scanning for AWS security issues. sigstore cosign v1.11.1 adds many fixes. GitLab 15.3 moves the pull-based GitOps feature without impersonation to the free tier.

🛡️ The Sec in Ops in Dev

Devs don't want to do ops? Do Devs need to do ops in the era of platforms, automated workflows, and abstracted APIs for deployments? The article quotes, "DevOps is dead," yet a platform helps bring DevOps platforms forward and together again. It will be interesting to see how vendors and users move along in 2023 when themes like Observability-driven development and MLOps reach global audiences.

Traditional Packaging is not Suitable for Modern Applications is an interesting read. While I agree on moving on to using containers, tools like flatpak for Linux may not entirely solve the problem, looking beyond operating systems e.g. into securing supply chains with common specs on image signing (sigstore/cosign). That's already one step further, and will IMHO become more dominant in the future, also "outside" of cloud-native environments. Microsoft announced built-in container support for the .NET SDK and abstracts settings like FROM for base images away into MSBuild properties when needed. You can run dotnet publish in CI/CD, log into the registry, and push the created image to later deploy in production.

You have received a JSON file as SBOM (Software Bill of Materials) from your vendor ... what's next? Scan for security vulnerabilities with Bomber, for example. It supports SDPX, CycloneDX, and Syft formats and uses OSV or the Sonatype OSS index as a vulnerability information provider. Another cool project is the SBOM Operator, providing a catalog of all images of a Kubernetes cluster to multiple targets with Syft. It supports storing the SBOM in Git, Dependency Track, OCI-Registry, and ConfigMaps for further (security) analysis.

⛅ Cloud Native

Why you should not use CPU limits in your Kubernetes cluster with multi-threaded microservices - great learning curve and read on how CPU throttling can affect performance negatively.

You may have heard about Keptn as a quality gate with SLOs and metrics in CI/CD already. This quickstart guide sheds light on many features: Keptn as a control plane for services, reusable pipelines, asynchronous event-driven workflows, validation gates, and remediation after deployments.

Creating and maintaining container images is hard, and with a variety of security scanners, many items are detected. But what if ML could help with detection and automated fixes in merge requests? That's a focus area in GitLab's Incubation Engineering team, resulting in Dokter.

👁️ Observability

Clearly, eBPF is a hot topic, and RedMonk follows closely. Liz Rice provided a deep dive into eBPF for better security observability at Container Days, and the possibility to define Cilium Tetragon tracing policies to follow/prevent syscalls is fascinating at best. Want to know what is possible with eBPF? Anomaly detection using unsupervised-learning encodes and eBPF explains the principles and follows with monitoring the Spotify process to train the ML model. "Connecting to Facebook" as action later leads to an anomaly detected as a syscall. Wow. Another great example is monitoring gRPC-C with eBPF, shining into the problem of gRPC connections applying stateful compression, making observability harder. The article describes challenges with traces, and how the solution can trace data, headers and stream close. Last but not least, I want to close the loop towards Kubernetes Observability and eBPF by pointing you to a great read that combines eBPF probes with Prometheus metrics conversion, enriched with metadata from the Kubernetes API. That's a great way to learn the basics without tool overhead, with the next article on AIOps and anomaly detection coming soon.

OpenTelemetry roadmap and latest updates was written in June, but it is never too late to catch up after Dotan provides a talk ;-) The latest OpenTelemetry updates include adding Profiling as a new type, which was discussed at KubeCon EU in May. Please help review the PR to refine the specification.

Inspired by reading about how to instrument Nginx with OpenTelemetry, I've started playing with an OpenTelemetry and Jaeger deployment in Kubernetes, and created a dedicated nginx-opentelemetry project. It includes a pre-defined configuration and a container build for Kubernetes deployments to show-case the demo with chaos engineering for my Container days talk.

Log management and analytics are challenging when it comes to scaling and data retention (which logs in my Kubernetes cluster do I need - great read). Cloudflare shared the technical implementation of their logging pipeline, Elasticsearch bottlenecks, and how they adopted Clickhouse as a columnar database with inserts, batch sizes, data modeling, partitioning, and storing different resolutions/sample intervals (ABRs).

🔍 The inner Dev

Speed up your monorepo workflow in Git is a great read that provides nifty tricks with sparse checkouts, partial clones, and shallow clones. If you are developing API workflows with Postman, the new integration with GitLab enables developers to think about API elements as the API itself, instead of treating code, API definitions, documentation, collections, tests, monitors, etc. as independent entities. All of these constitute the API.

There is some controversy going on with WebAssembly and the WASI interface, making AssemblyScript remote from its WASI support. WASI provides a modular system interface required by WebAssembly projects to interact with operating system calls, for example. It seems that portability between architectures is not hard, with more concerns shared in this issue. We may see sub-APIs for WebAssembly solving special purpose requests such as time, filesystems, sockets, etc. - until then, developer experience and getting started guides need improvements. (it's even hard to explain in this newsletter - let me know if you want to chat more and learn together)

Erica Brescia asked a thoughtful question: "Hey OSS friends - weekend q for you: Can you think of a very widely adopted/successful OSS project that was adopted almost exclusively by non-tech companies (where high tech companies eschewed it)?" Looking into the responses, we came across Shibboleth and Moodle used at academic institutions and OBS Studio for streaming. What's your take? Reply directly on Twitter :-)

📈 Your next project could be ...

📚 Tools and tips for your daily use

  • glances, a top/htop alternative for GNU/Linux, BSD, Mac OS, and Windows operating systems.
  • devbox is a command-line tool that lets you easily create isolated shells and containers. You start by defining the list of packages required by your development environment, and devbox uses that definition to create an isolated environment just for your application.
  • Debug containers with a temporary pod, for example, curl to test HTTP connectivity issues in your Kubernetes cluster.
  • k8spacket helps to understand TCP packets traffic in your Kubernetes cluster
  • k9s, a terminal-based UI to interact with your Kubernetes clusters.
  • C++ tip: Use \n for new lines in iostreams, and only flush when desired. std::endl always flushes, which can affect performance.
  • Config grep tips: Remove newline and hashes: grep -v -e '^$' -e'^ *#' $argv or egrep -v '^#|^$' $argv as shell aliases.
  • Automatically Convert Grafana Dashboards from InfluxQL to PromQL. Based on the work from Aiven.io, Logz.io released this new OSS tool to help migrations.

🔖 Book'mark

🎥 Events and CFPs

👋 CFPs due soon

Looking for more CfPs? Try CFP Land.

🎤 Shoutouts

Kelsey Hightower and Werner Vogels for sharing the analogy between open source conferences such as KubeCon, and vendor-specific like VMware Explore. Full interview.

🌐

Thanks for reading! If you are viewing the website archive, make sure to subscribe to stay in the loop!

See you next month - let me know what you think on Twitter or LinkedIn.

Cheers,

Michael

PS: If you want to share items for the next newsletter, please check out the contributing guide - tag me in tweet replies or send me a DM. Thanks!