Skip to content

2023-11-14: Rejekts & KubeCon NA: AI, DevEx, Platform Engineering, Kubernetes maturity, OpenTelemetry GA, Cilium Tetragon 1.0

Thanks for reading the web version, you can subscribe to the Ops In Dev newsletter to receive it in your mail inbox.

👋 Hey, lovely to see you again

The November newsletter comes with a delay - I had ambitious plans to finish the newsletter during the KubeCon NA week and travel (Narrator: And then Michael learned about the international terminal in ORD in Chicago). After relaxing with jetlag and reflecting on the inspiring conversations at Rejects and KubeCon NA, this newsletter covers many event insights and more things to learn in DevSecOps, AI, cloud-native, and Observability. I recommend skipping sections or opening interesting articles in separate tabs to read later this week.

Before we dive in, KubeCon NA was all about adopting AI and LLMs, improved developer experience, and platform engineering. Kubernetes is maturing, too.

🌟 Cloud-native Rejekts

Rejekts is a community-driven event where folks can submit their rejected KubeCon talks. After getting invited in 2022 and feeling welcome, I wanted to join the event again in 2023. They are such friendly humans and have many exciting topics to learn. Most talks are captured as pictures in LinkedIn or Twitter/X posts. The Rejekts YouTube livestreams also provide the full event experience to rewatch.

  • Choose Your Own Adventure: The Perilous Passage to Production by Whitney Lee and Viktor Farcic, pictures.
  • Distributed tracing with LLMs by Daniel Kim, pictures. The talk also touches on OpenLLMetry, check this related discussion and live demo
  • Helpful OpenTelemetry tools for a better learning experience by Reese Lee and Alex Boten, pictures
  • The road to zero vulnerable container images by Adrian Mouat, pictures
  • Kubernetes productivity tips by Tiffany Jernigan, pictures. Bookmark the talk slides.
  • Let's talk community by Kim McMahon, pictures
  • Wasm-optimized Linux: what, how, and, most importantly, why? by Andrew Randall and Ralph Squillace, pictures, spoiler: Flatcar Linux as CoreOS fork as WebAssembly OS.
  • Building bridges for better collaboration with Kubernetes policies and DevOps workflows by Cortney Nickerson, pictures

🍯 Co-located events: CiliumCon and more

The CNCF access-all-areas pass is a great idea and it allowed me to switch between CiliumCon, Observability Day, and BackstageCon. AppDeveloperCon was interesting, too. At CiliumCon, I learned about Cilium Tetragon 1.0, certifications, and Cilium graduation (recording, pictures). Adobe shared their story to provide an internal platform, with one constant: Kubernetes and Cilium (recording, pictures). Marino Wijay provided an impressive deep dive into Cilium and BGP networking (recording, pictures). In the afternoon, I attended a session with Dynatrace friends, and how they adopted Backstage.

CNCF YouTube playlists:

🗺️ KubeCon NA 2023

The keynote at KubeCon NA was all about adopting AI (pictures on LinkedIn), and how open-source helps enable these workloads running on cloud-native technology. Kubernetes is the de facto standard and might have its "Linux moment" in its adoption journey. Priyanka Sharma showed Ollama in a live demo on a Macbook, only using open-source components. Although the demo took a while to load, and we watched the backup recording together, in the end, the live demo worked. You can try it yourself, using kind.

KubeCon NA 2023 keynote, live demo with local kind cluster running Ollama open-source LLMs

After the keynotes, I finally met Katie Gamanji, and we started an insightful discussion about Observability and how sustainability with Kepler using eBPF can help with CI/CD Observability. After walking from the keynotes into the solutions showcase, we spotted the Ollama project and its cute logo. Open-source, aiming to provide to LLMs for everyone, used in the KubeCon NA keynote before.

Tim Hockins provided an insightful overview of the past 10 years with Kubernetes and where it will be heading with AI workloads and feature completion ("say no more often") to maintain stability (tweet).

Watch the CNCF YouTube channel for video updates. Usually, they are released within 2 weeks of the event. We will discuss more talk insights in the next newsletter issue -- my KubeCon NA schedule track interesting talks, knowing that I cannot attend all activities. More options to learn about KubeCon NA include:

  • Engin Diri's Twitter/X recap thread, citing "Mastery of Kubernetes isn't just beneficial; it's essential. The innovations built on Kubernetes I've seen are nothing short of impressive. Basic knowledge won't make you stand out."
  • Jimmy Zelinskie's thread about the biggest KubeCon takeaways, with Lachlan Evenson replying, "For me l, WASM is finding its stride,, Kubernetes maturity via LTS, AI integrations and tooling growth, and supply chain security tooling. There’s still a lot happening in the space."
  • Daniel Bryant's blog post, citing "The (slow) rise of AI, the domination of platform engineering, and the refocus on developer experience at KubeCon NA 2023"
  • Kacey Gam's blog post, with a SRE's point of view, citing "A lot of vendors this year gave out books as swag. I always appreciate stickers and shirts, but the books are valuable."
  • Hope Oluwalolope summary, citing "Exploring the project pavilion and solution showcase is definitely a must. ... It also gives an idea of the focus and current priorities in the cloud-native space."
  • Mauricio Salatino's blog post, citing "The big trends were LLM models on Kubernetes, Platform Engineering followed by Developer Experience and Security (Secure Supply Chain, SBOMs, etc.)."

Still need more?

  • Marino Wijay is hosting a Twitter/X space to recap KubeCon NA on Nov 15, 7:30 AM PT.
  • Dotan Horovits hosts the OpenObservability podcast with Lin Sun from solo.io on Nov 15 at 11 AM PT.
  • KubeCon NA 2024 happens in Salt Lake City, Utah. Check the updated section with Events and CFPs in this newsletter.

🌱 The Inner Dev learning ...

🐝 The Inner Dev learning eBPF

The eBPF documentary was released during KubeCon NA, and Thomas Graf shared more memories and insights on the Isovalent blog.

The article "Don’t Rely on eBPF Alone for Kubernetes" is must read if you are planning to adopt eBPF into your workflows, and platforms.

Odigos published an article, saying that eBPF-based auto-instrumentation [of source code] outperforms manual instrumentation. The related Hacker News discussion provides more insights, for example, with sampling for tracing. One downside of auto-instrumentation is whether it can provide enough context to help debug the application. This needs to be proven on a case-by-case basis, and can still require manual instrumentation with the OpenTelemetry SDK.

Running bpftrace requires root permissions on the Kernel. bpftime is a userspace eBPF runtime for fast uprobe and syscall hooking. For example, you can run bpftrace in userspace to trace syscalls.

Quick notes:

🤖 The Inner Dev learning AI/ML

Getting started with language models as developers can feel overwhelming. This 17 minutes video explains how to get started running open-source LLMs, and how to use them in applications. In related news, are you struggling with MLOps buzzwords? Raphaël Hoogvliets compiled a helpful cheat sheet for MLOps. Looking for a more fun way to learn? Training an unbeatable AI in Trackmania, a fast racing game with many obstacles - the video explains neural networks, and reinforcement learning to progressively advance by getting faster.

"Techniques For LLMs to Verify Themselves And Reduce Mistakes" (tweet) sounds interesting to let AI heal itself magically. The thread explains error rates and starts with the first method: "Holdout Validation" where the data is split into val and test sets. Iteration and prompt engineering help to verify the model's accuracy. The next method is Self Verification, a step in a "chain of thought (CoT)" in a large language model (LLM). When mistakes happen, the conclusion of a CoT is used as a condition to create a new sample, and ask the LLM to re-predict the original conditions. Based on the accuracy, a verification score is calculated. The "Chain-of-Verification (CoVe)" method aims to tackle hallucinations (plausible but incorrect factual information). It requires fource steps: 1) Draft a response 2) Add verification questions to fact-check the draft 3) Independent answers of the questions to avoid overlap and bias 4) Generate a verified response, following the process. Related paper: Chain-of-Verification Reduces Hallucination in Large Language Models, v2 last revised 2023-09-25. The article about training, validation and test data sets can help the learning curve, too.

My GitLab blog series "Learning a programming language with a little help from AI" continues with Python this month. As an experienced Python developer, I still learned new tricks and better algorithms and trained my brain for future tasks.

👁️ Observability

OpenTelemetry is GA: Logs are now stable alongside metrics and tracing, and the OpenTelemetry Protocol (OTLP) reaches 1.0. Tip: Checkout the OpenTelemetry Transform Processor to ingest, modify and enrich observability data. In other news, Prometheus aims to become the default metrics backend in OpenTelemetry. There is work underway for 3.0 to achieve this goal.

At Rejekts, I asked Nele Uhlemann about Fiberplane - it helps debugging infrastructure incidents in collaborative notebooks. Later at KubeCon, I learned that Fiberplane created Autometrics for enhanced application observability (video introduction at KubeCon NA).

🛡️ DevSecOps

DevSecOps efficiency continues to be a hot theme in 2023 and 2024. At KubeCon NA, it was great to see more cost monitoring awareness. My lightning talks at the GitLab booth centered the efficiency theme, too. The slides provide learning resources, and can be helpful for future adoption of Observability, AI and reusable CI/CD workflows.

At Rejekts, I learned about different methods for zero vulnerable container images created by Chainguard. Chainguard container images are powered by the Wolfi OS, and provide a suite of distroless images. The main idea behind Wolfi OS is to provide zero CVE images, with the benefit of optimized and minimal container images too.

One of the mentioned methods includes so-called "Distroless Docker Images", which are images stripped down to the bare essentials to run your application. This method provides enhanced security, optimized size and better maintenance for updates and patches. One disadvantage: Debugging becomes harder. This guide on Medium (members only) shows an example with CI/CD pipelines using Python 3.11, AWS CLI and AWS CDK.

🌤️ Cloud Native

Karpenter is a Kubernetes node lifecycle manager, created by AWS. It is now being donated to the CNCF as part of the Kubernetes Autoscaling Special Interest Group (SIG), including graduation to Beta to provide stable APIs (KubeCon keynote announcement),

For German-speaking readers, I recommend checking out the iX Developer 2023 special "Cloud Native", with practical insights into GitOps, Platform Engineering, DevOps, and Observability. I have contributed two articles about Observability for CI/CD workflows, and Debugging in production with eBPF and chaos engineering (screenshots on LinkedIn, heise.de announcement).

Quick notes:

📚 Tools and tips for your daily use

  • systeroid, a more powerful alternative to sysctl(8) with a terminal interface.
  • OtelBin helps visualizing and improving OpenTelemetry collector configurations. Learn more on the dash0 blog.
  • isotope scans AWS services, and provides suggestions on how to improve them using AWS Bedrock (AI).
  • K8GB is a Kubernetes Global Balancer, and CNCF sandbox project.
  • werf is an open-source solution for efficient and consistent software delivery to Kubernetes.
  • Keptn is a cloud-native application lifecycle orchestration.
  • Vanguard is a powerful library for Go net/http servers that enables seamless transcoding between REST and RPC protocols.
  • gittuf, a security layer for Git repositories.
  • Hardware: Elgato Prompter to help connect to your audience with natural eye contact.

🔖 Book'mark

🎯 Release speed-run

Cilium Tetragon 1.0 improves performance, only adds 2% overhead for process execution tracking and provides scalable file monitoring at minimal cost. Default Observability policies help with getting started. Coroot 0.21.0 provides application health summary to quickly identify problematic services. 0.20.0 added log monitoring support. tracee v0.19.0 adds support for managing tracee policies as CRDs in a Kubernetes Operator. The hooked_syscall event provides more details of system-level interactions and potential security problems - rootkits and other malicious actors can hook into syscalls.

k8sgpt v0.3.21 adds log analysis for Kubernetes. pprof.me now supports social previews, and user history.

PostgreSQL 16 improves performance for query parallelism, bulk data loading and logical replication. The release also adds more metrics for I/O monitoring.

🎥 Events and CFPs

2024

👋 CFPs due soon

Looking for more CfPs?

🎤 Shoutouts

The 10-hour course on Freecodecamp "Arduino for Everybody" is fantastic to learn about hardware components and software programming.

🌐

Thanks for reading! If you are viewing the website archive, make sure to subscribe to stay in the loop! See you next month 🤗

Cheers, Michael

PS: If you want to share items for the next newsletter, just reply to this newsletter, send a merge request, or let me know through LinkedIn, Twitter/X, Mastodon, Blue Sky. Thanks!