Skip to content

2023-10-10: Polar Signals Cloud GA, AI in the DevSecOps lifecycle, Observability-driven development for LLMs, KubeHound, Timoni

Thanks for reading the web version, you can subscribe to the Ops In Dev newsletter to receive it in your mail inbox.

👋 Hey, lovely to see you again

Time is flying fast, feels like yesterday writing the September newsletter :)

Hot off the press: Polar Signals Cloud is GA. It's been a pleasure to follow the team from inception, getting the first peek into continuous profiling in 2021 and learning how they build in the open with Parca using eBPF. True open-source inspiration and an innovative product!

The feature screenshot tells me I should sign up for the trial and see how my AI-assisted Rust applications are performing ...

Polar Signals in action Copyright Polar Signals, source Up

🌱 The Inner Dev learning ...

DevOps Camp Nuremberg 2023 happened last weekend, a fantastic community event with barcamp sessions. My session pitches highlighted the two learning series in this newsletter :-)

🐝 The Inner Dev learning eBPF

Following up on DevOps Camp Nuremberg 2022, we started a new eBPF learning session with a short deep-dive into the basics. We used the Learning eBPF workshop slides from Cloudland 2023, my recent Container Days talk, and discussed different use cases for observability, security, and SRE/DevOps. The group highlighted the importance of how eBPF programs are deployed (avoid sudo pipe bash install patterns) and mitigate potential performance impacts and other security risks. Cilium Tetragon got a shoutout for the TracingPolicy configuration format, following the Kubernetes spec.

When looking into the programming libraries for eBPF, someone highlighted that redbpf for Rust has been archived (thanks!), and aya-rs remains as viable alternative for writing eBPF programs in Rust. We also touched on topics of how eBPF programs (Kernel space) interact with user space programs through maps, how the verifier ensures safeguards, and "Compile Once Run Everywhere" functionality. Auto-generating traces for Observability using eBPF was also an idea to avoid shifting left everything on developers.

Other aspects we touched on: 1) Security risks: Reading decrypted TLS traffic with eBPF. peetch provides tools that can help bypass TLS protocol protections with these experiments. 2) Deploying eBPF programs also remains challenging. bpfd aims to provide a gRPC API interface for loading, unloading, modifying, and monitoring eBPF programs. The bpfd agent runs in a container in the bpfd DaemonSet, and ensures that all eBPF programs are in their desired state. I recommend following the bpfd tutorial to learn and practice.

Last month, I forgot to highlight prwu from Cilium, Packet, where are you? An eBPF-based Linux kernel networking debugger. Thanks for the Learned at Container Days poke, Rachid Zarouali.

🤖 The Inner Dev learning AI/ML

Brooke Jamieson asked on Twitter/X: "How many times have you started learning AI/ML?". To be honest, just once here in this newsletter, but it feels hard to keep up with everything. I have experimented with learning strategies, finding use cases for my daily workflows, for example, using GitLab Duo Code Suggestions to learn new programming languages, such as Rust. Events with hallway tracks also help discuss and verify the knowledge.

At DevOps Camp Nuremberg 2023, I pitched a session about "AI in the DevOps lifecycle - beyond code suggestions." We had a great discussion looking at the DevOps lifecycle stages and steps and thought about Issue/Merge Request summaries, CI/CD efficiency with root cause analysis, and guided pipeline creation, as well as diving deeper into the Sec in DevSecOps with explaining security vulnerabilities, and later auto-remediation. The one-hour discussion continued throughout the event with fresh new thoughts, for example, self-hosted LLMs that do not require internet access.

Software that uses LLMs becomes more "generally available," and complexity and requirements are growing. Wise thoughts from Charity Majors: "Debugging software is hard; the bug could be in the algorithm or the implementation. Debugging ML is exponentially harder bc it could be in the algorithm, implementation, model OR data. "Our debugging process goes from a 2D grid to a 4D hypercube"" -- based on Why machine learning is hard. The article visually explains the 2D grid with 1) Implementation bugs and 2) Algorithm correctness. AI/ML adds 3) Model Issues into the 3D picture and moves to 4D with either "Enough correct data," "not enough data," or "weak labels."

Increased complexity and hard-to-debug LLMs require new-and-old approaches: LLMs Demand Observability-Driven Development. Great read!

Patrick Debois shared free learning resources for LLMs and generative AI concepts, with sections on ops, sec, and dev (project, video). It also introduced a new project: LangChain for developing applications powered by language models. The documentation provides a quickstart tutorial for Python and other languages.

Quick notes:

👁️ Observability

Polar Signals continues their amazing journey to use eBPF for profiling Python and Ruby, released as open-source in Parca Agent v0.26.0. Other languages are on the roadmap or already supported in newer versions, for example, PHP 8 with JIT compiler enabled. In related news, Python support was also added to the Linux perf profiler.

Railway shared their journey of adopting Clickhouse as storage for 100 billion logs. They moved from a local filesystem and Google Cloud storage to add support for filtering logs, aggregate metrics queries (count, group by) and solve the problem to scale horizontally. After the first proof-of-concept, Vector, a log ingestion tool was configured to write to Clickhouse, and the UI queries were updated to match the new SQL format. Interesting read, especially if you plan to build your Observability storage backend. More insights in this Twitter/X thread.

Quick notes:

🛡️ DevSecOps

KubeHound creates a graph of attack paths in a Kubernetes cluster, allowing you to identify direct and multi-hop routes an attacker can take, visually or through complex graph queries. KubeHound connects to a Kubernetes cluster and reads the API resources, then computes the attack paths, and writes the results to a local graph database (JanusGraph). Under the hood, Redis and MongoDB are used for caching and persisting data. The getting started guide requires Docker 19.03+ and Docker Compose 2.0+.

Oracle announced their Java platform extension for VS Code, based on Netbeans and javac. It supports all Java language features and integrates with build tools, testing frameworks. "One IDE for many languages, rather than a language-specific IDE" -- unfortunately, not available in the VS Code marketplace yet.

Docker, BastionZero and the Linux Foundation announced OpenPubKey, Dan Lorenc wrote about insights on the differences between Sigstore and OpenPubKey. One of the TL;DRs is the removal of central, server-side components of Sigstore, the transparency log and the certificate authority components. One disadvantage explained in the article refers to client-side verification and how it is impossible to handle historical verification. Another issue seems to be using the raw JWT OIDC token as certificate, thus publishing it to the internet, instead of creating a new x509 certificate with public/private key pairs. Recommend to be mindful about these raised concerns, and follow the public discussions.

jq has new maintainers, a new release in five years, and updated build, CI/CD, security tooling. Read the full story on LinkedIn, thanks for sharing, Dan Lorenc.

🌤️ Cloud Native

The OpenTelemetry governance committee election was announced, and Juraci encouraged direct representation by OpenTelemetry end-users. Are you using OpenTelemetry at scale, and want to contribute to the direction? Apply now! If you are looking for more contribution possibilities for OpenTelemetry, dive into the How Can I Contribute to OpenTelemetry? article.

Quick notes:

📚 Tools and tips for your daily use

  • bindiff helps to quickly find differences and similarities in disassembled code
  • tartufo searches through git repositories for high entropy strings and secrets, digging deep into commit history.
  • copacetic is a CLI tool for directly patching container images using reports from vulnerability scanners
  • jless is a command-line JSON viewer designed for reading, exploring, and searching through JSON data. Syntax highlighting, expand/collapse objects, regex search.
  • SQLFlow can visualize SQL queries.
  • k0smotron, an open-source control plane manager for unified Kubernetes cluster management.
  • VED (Vault Exploit Defense)-eBPF leverages eBPF (extended Berkeley Packet Filter) to implement runtime kernel security monitoring and exploit detection for Linux systems.
  • Alaz is an open-source Ddosify eBPF agent that can inspect and collect Kubernetes service traffic without the need for code instrumentation, sidecars, or service restarts.
  • Crafting container images without Dockerfiles

🔖 Book'mark

🎯 Release speed-run

Falco 0.36.0 brings a more robust rule framework, file path detection, symlink resolution, and ancestors detection. Signatures are now automatically verified in falcoctl for plugins and rules. Falco also supports many Kernel versions, and created an Ansible-based Kernel version testing framework. Flagger v1.34 completes the Kubernetes Gateway API implementation with sticky sessions and mirroring support for canaries. Helm v3.13.0 brings support for a dry-run feature for templates that need to connect to a running Kubernetes server.

Pyrra v0.7.0 improves the list page, adds support for native histograms, config validation, and documents a fully encrypted OpenShift example. Coroot 0.19.0 adds new cluster-wide deployment reports, allowing to track the service rollouts with a searchable view. After announcing the new Terraform fork supported by the Linux Foundation, OpenTofu started releasing the first alpha versions: v1.6.0-alpha1 and v1.6.0-alpha2.

Rust 1.73.0 provides clearer panic messages and thread local initialization.

🎥 Events and CFPs

2024

👋 CFPs due soon

Looking for more CfPs?

🎤 Shoutouts

I stumbled over Low Byte Productions, explaining operating system concepts by practical example. For example, using xv6 as a reimplementation of Unix Version 6 for a modern RISC-V multiprocessor using ANSI C, showing how Virtual Console and printf() works. Might be worth a shot to understand concepts, in combination with learning eBPF low level programming.

Thanks for the endless watch loop: I Built a Lego Water Park Rapids Ride!.

🌐

Thanks for reading! If you are viewing the website archive, make sure to subscribe to stay in the loop! See you next month 🤗

Cheers, Michael

PS: If you want to share items for the next newsletter, just reply to this newsletter, send a merge request, or let me know through LinkedIn, Twitter/X, Mastodon, Blue Sky. Thanks!