Skip to content

2023-08-12: Learning Rust with AI, eBPF learning map, We Hack Purple+Semgrep, 90DaysOfDevOps, eBPF Summit 2023, Coroot AI Observability, Trace-based testing with OpenTelemetry

Thanks for reading the web version, you can subscribe to the Ops In Dev newsletter to receive it in your mail inbox.

👋 Hey, lovely to see you again

The AI landscape is moving fast, and so is Observability and eBPF. The latter will have a virtual eBPF Summit in September 2023 for everyone to join, inspiring new tools that help debug production workloads. Learning AI on the other hand also means finding use cases, for example learning a new programming language and writing a blog post about the experience. I also loved reading about AI Observability and Shadow AI this month. Aside from continued OSS license changes and forks, a happy community moment happened with We Hack Purple joining forces with Semgrep.

🌱 The Inner Dev learning ...

Recommended read: Developer-Led Landscape: Software Supply Chain Security by Tyler Jewell.

🐝 The Inner Dev learning eBPF

eBPF Summit 2023 happens on September 13, 2023. I was kindly invited to the CFP review and found many great ideas through the submissions. You know some of them already in this newsletter, others tackle topics in the ever-growing AI and DevSecOps landscape. Stay tuned, and register for the event.

Isovalent created an interactive learning map for eBPF, and different user persona journeys: Cloud network engineer, Platform engineer, Platform Ops, Cloud Architect, and Security Professionals. Fantastic learning resource. Bookmark this! Thanks Raphaël Pinson for sharing.

Isovalent cloud network map for eBPF

Coroot Agent 1.9.0 supports eBPF-based TLS connection tracing for Golang applications. That way the Coroot agent can capture requests before they are encrypted. The implementation PR provides insights and can be an inspiration for own learning exercises.

To get better insights into running eBPF programs, ebpfmon can help. It builds on top of bpftool and provides a terminal UI.

🤖 The Inner Dev learning AI/ML

This month, I focussed on sharing my learning experience for a new programming language and how AI can help. It turned out to be a fun challenge to "instrument" GitLab Duo Code Suggestions in a way that I got better suggestions. Learn more in the blog post Learning Rust with a little help from AI. Stay tuned for the second part on Rust with more playful advanced learning and more programming languages.

Gergely Orosz shared how to block OpenAI and ChatGPT from scraping their blog and newsletter. The reasons are insightful: Training the models gives no attribution to original content authors. The OpenAI LLMs are not publicly available and do not benefit the community. The discussion on LinkedIn led me to a great resource collection to learn LLMs and more by Nadia Privalikhina.

Elastic wrote about Avoid Shadow AI — Embrace generative AI in the SOC, raising awareness for the question: Should we allow the use of generative AI within our organizations, even within cybersecurity? And if yes, how to maintain control of the data to avoid leaking sensitive information into a public LLM. For the Elastic AI Assistant, they embraced full transparency showing all data exchanged and added functions to anonymize data. The article concludes with reminding to embrace and empower teams, to avoid the creation of Shadow AI (after Shadow IT).

The talk AI Observability at Meta Scale dives into how Meta observes and optimizes resource usage for AI workloads. Four different layers of observability are discussed: Fleet level resource usage (Aggregation and regression tracking), Meta performance (profiling/analysis platform), application tracing and instrumentation (Pytorch profiler / Kineto, BPF tracing) and bare metal telemetry and monitoring (Dynolog).

👁️ Observability

To diagnose network issues in Kubernetes, I found KubeSkoop from Alibaba which uses eBPF for network analysis. It can draw a network communication map, export metrics to Prometheus, and sends network anomaly events to Grafana Loki.

bpftune from Oracle helps to tune Linux systems automatically by observing their behavior. Supported tuners are congestion, neighbor table, route table, sysctl, TCP buffer, net buffer, netns. The architecture allows loading tuners as plugins and is described as a lightweight daemon without polling too many events.

If you ever wanted to try trace-based testing and did not know how, there is a new guide: Trace-based Testing the OpenTelemetry demo. The integration tests are AVA tests, while the frontend end-to-end tests use Cypress. The demo environment uses a shop to order products and does the checkout procedure for a better learning curve.

Coroot shared their research on adding AI-powered root cause analysis for their observability platform. Interesting read from finding the right telemetry data, adding system topology, and concluding with an interesting statement:

We want AI to assist us with anomaly analysis rather than anomaly detection

🛡️ DevSecOps

Daniel Bodky shared a great read about The Good, the Better, and the Ugly - Signing Git Commits. The good is called gitsign, the best is SSH Commit signing, and the ugly is -- you guessed it - GPG. If you have tried convincing the GPG agent to sign Git commits with TTY, you probably know the pain. I switched to SSH key commit signing some months ago, see the configuration in dotfiles repository: .gitconfig and allowed_signers

The We Hack Purple community, founded by Tanya Janca, and Semgrep are joining forces. This is exciting for many reasons: More security education for everyone, community building at Semgrep, and better developer experience with SAST scanning. Semgrep is already great - I recently looked into the Rust support in Beta. Tip: Order it now if you have not read Tanya Janca's book Alice and Bob Learn Application Security. Fantastic read.

The Hurl maintainers revamped their tutorial pages with a playful web demo which makes it easier to get started with the first command line tests. Hurl allows you to test websites for specific requests and responses and can also be helpful in CI/CD test automation. I wrote a blog post in late 2022 about Hurl and GitLab CI/CD.

🌤️ Cloud Native

Recommended watch: GitOpsConf 2023: GitLab + Flux!

After the RHEL changes to upstream source code access (more in the July newsletter issue), SuSE announced a fork of RHEL which now turned into a collaboration with Oracle and CIQ under the Open Enterprise Linux Association (OpenELA). CIQ is the company creating RockyLinux, Oracle maintains Oracle Linux, and SuSE provides RHEL support already (more context on Hacker News), making this a natural next step to foster open source collaboration.

More license changes: HashiCorp changed the license of their products to the source-available Business Software License (BuSL), which forbids commercial use in competitive products. Affected are Terraform, Packer, Vault, Boundary, Consul, Nomad, Waypoint, and Vagrant while HashiCorp APIs, SDKs, and almost all other libraries are noted to remain MPL 2.0. The full implications remain unclear, and many companies and users are consulting with lawyers now. Community members started OpenTerraform, a Terraform fork based on the last commit with the Mozilla Public License 2.0 (MPL).

Another fork happened in container land: Incus is a fork of LXD, and now part of the Linux Containers project. This happened after Canonical's decision to remove LXD from the Linux Containers project. More insights in the Hacker News discussion.

📚 Tools and tips for your daily use

🔖 Book'mark

🎯 Release speed-run

Cilium 1.14.0 brings Effortless Mutual Authentication, Service Mesh, Networking Beyond Kubernetes, High-Scale Multi-Cluster, and more (announcement blog post). Tracee v0.17.0 brings a new policy format compatible with Kubernetes CRDs, new flags for enhanced event filtering and simplified event sets. OpenSearch 2.9.0 brings search pipelines GA, production-ready neural search, ML framework GA, and monitors and detectors in OpenSearch Dashboards.

jq 1.7rc1 brings the first release after 5 years. GitLab 16.2 comes with a new Rich Text editor, improving the editing experience in issues, comments, wiki. It also supports keyless signing with Cosign, a new command palette for efficiency, triggering a Flux synchronization without any configuration and more.

Prometheus v2.46.0, OpenTelemetry Collector v0.82.0 and OpenTelemetry Collector Contribut v0.82.0, Perses 0.39.0, Keptn v1.4.1, Parca agent v0.23.3, Flux v2.0.1, Kyverno v1.10.2, Open Policy Agent v0.55.0

🎥 Events and CFPs

2024

👋 CFPs due soon

2024

Looking for more CfPs?

🎤 Shoutouts

"I'm pretty surethe application is somewhere around here" is a great comic meme for Kubernetes. Bill Mulligan went one step further, saying "Fixed it ..." (more on Twitter/X).

🌐

Thanks for reading! If you are viewing the website archive, make sure to subscribe to stay in the loop!

See you next month - let me know what you think on LinkedIn, Twitter/X, Mastodon, Blue Sky 🤗

Cheers,

Michael

PS: If you want to share items for the next newsletter, please check out the contributing guide - tag me in the comments, send me a DM or submit this form. Thanks!