2023-08-12: Learning Rust with AI, eBPF learning map, We Hack Purple+Semgrep, 90DaysOfDevOps, eBPF Summit 2023, Coroot AI Observability, Trace-based testing with OpenTelemetry¶
Thanks for reading the web version, you can subscribe to the Ops In Dev newsletter to receive it in your mail inbox.
👋 Hey, lovely to see you again¶
The AI landscape is moving fast, and so is Observability and eBPF. The latter will have a virtual eBPF Summit in September 2023 for everyone to join, inspiring new tools that help debug production workloads. Learning AI on the other hand also means finding use cases, for example learning a new programming language and writing a blog post about the experience. I also loved reading about AI Observability and Shadow AI this month. Aside from continued OSS license changes and forks, a happy community moment happened with We Hack Purple joining forces with Semgrep.
🌱 The Inner Dev learning ...¶
Recommended read: Developer-Led Landscape: Software Supply Chain Security by Tyler Jewell.
🐝 The Inner Dev learning eBPF¶
eBPF Summit 2023 happens on September 13, 2023. I was kindly invited to the CFP review and found many great ideas through the submissions. You know some of them already in this newsletter, others tackle topics in the ever-growing AI and DevSecOps landscape. Stay tuned, and register for the event.
Isovalent created an interactive learning map for eBPF, and different user persona journeys: Cloud network engineer, Platform engineer, Platform Ops, Cloud Architect, and Security Professionals. Fantastic learning resource. Bookmark this! Thanks Raphaël Pinson for sharing.
Coroot Agent 1.9.0 supports eBPF-based TLS connection tracing for Golang applications. That way the Coroot agent can capture requests before they are encrypted. The implementation PR provides insights and can be an inspiration for own learning exercises.
To get better insights into running eBPF programs, ebpfmon can help. It builds on top of bpftool and provides a terminal UI.
🤖 The Inner Dev learning AI/ML¶
This month, I focussed on sharing my learning experience for a new programming language and how AI can help. It turned out to be a fun challenge to "instrument" GitLab Duo Code Suggestions in a way that I got better suggestions. Learn more in the blog post Learning Rust with a little help from AI. Stay tuned for the second part on Rust with more playful advanced learning and more programming languages.
Gergely Orosz shared how to block OpenAI and ChatGPT from scraping their blog and newsletter. The reasons are insightful: Training the models gives no attribution to original content authors. The OpenAI LLMs are not publicly available and do not benefit the community. The discussion on LinkedIn led me to a great resource collection to learn LLMs and more by Nadia Privalikhina.
Elastic wrote about Avoid Shadow AI — Embrace generative AI in the SOC, raising awareness for the question: Should we allow the use of generative AI within our organizations, even within cybersecurity? And if yes, how to maintain control of the data to avoid leaking sensitive information into a public LLM. For the Elastic AI Assistant, they embraced full transparency showing all data exchanged and added functions to anonymize data. The article concludes with reminding to embrace and empower teams, to avoid the creation of Shadow AI (after Shadow IT).
The talk AI Observability at Meta Scale dives into how Meta observes and optimizes resource usage for AI workloads. Four different layers of observability are discussed: Fleet level resource usage (Aggregation and regression tracking), Meta performance (profiling/analysis platform), application tracing and instrumentation (Pytorch profiler / Kineto, BPF tracing) and bare metal telemetry and monitoring (Dynolog).
To diagnose network issues in Kubernetes, I found KubeSkoop from Alibaba which uses eBPF for network analysis. It can draw a network communication map, export metrics to Prometheus, and sends network anomaly events to Grafana Loki.
bpftune from Oracle helps to tune Linux systems automatically by observing their behavior. Supported tuners are congestion, neighbor table, route table, sysctl, TCP buffer, net buffer, netns. The architecture allows loading tuners as plugins and is described as a lightweight daemon without polling too many events.
If you ever wanted to try trace-based testing and did not know how, there is a new guide: Trace-based Testing the OpenTelemetry demo. The integration tests are AVA tests, while the frontend end-to-end tests use Cypress. The demo environment uses a shop to order products and does the checkout procedure for a better learning curve.
Coroot shared their research on adding AI-powered root cause analysis for their observability platform. Interesting read from finding the right telemetry data, adding system topology, and concluding with an interesting statement:
We want AI to assist us with anomaly analysis rather than anomaly detection
Daniel Bodky shared a great read about The Good, the Better, and the Ugly - Signing Git Commits. The good is called
gitsign, the best is
SSH Commit signing, and the ugly is -- you guessed it - GPG. If you have tried convincing the GPG agent to sign Git commits with TTY, you probably know the pain. I switched to SSH key commit signing some months ago, see the configuration in dotfiles repository: .gitconfig and allowed_signers
The We Hack Purple community, founded by Tanya Janca, and Semgrep are joining forces. This is exciting for many reasons: More security education for everyone, community building at Semgrep, and better developer experience with SAST scanning. Semgrep is already great - I recently looked into the Rust support in Beta. Tip: Order it now if you have not read Tanya Janca's book Alice and Bob Learn Application Security. Fantastic read.
The Hurl maintainers revamped their tutorial pages with a playful web demo which makes it easier to get started with the first command line tests. Hurl allows you to test websites for specific requests and responses and can also be helpful in CI/CD test automation. I wrote a blog post in late 2022 about Hurl and GitLab CI/CD.
🌤️ Cloud Native¶
Recommended watch: GitOpsConf 2023: GitLab + Flux!
After the RHEL changes to upstream source code access (more in the July newsletter issue), SuSE announced a fork of RHEL which now turned into a collaboration with Oracle and CIQ under the Open Enterprise Linux Association (OpenELA). CIQ is the company creating RockyLinux, Oracle maintains Oracle Linux, and SuSE provides RHEL support already (more context on Hacker News), making this a natural next step to foster open source collaboration.
More license changes: HashiCorp changed the license of their products to the source-available Business Software License (BuSL), which forbids commercial use in competitive products. Affected are Terraform, Packer, Vault, Boundary, Consul, Nomad, Waypoint, and Vagrant while HashiCorp APIs, SDKs, and almost all other libraries are noted to remain MPL 2.0. The full implications remain unclear, and many companies and users are consulting with lawyers now. Community members started OpenTerraform, a Terraform fork based on the last commit with the Mozilla Public License 2.0 (MPL).
Another fork happened in container land: Incus is a fork of LXD, and now part of the Linux Containers project. This happened after Canonical's decision to remove LXD from the Linux Containers project. More insights in the Hacker News discussion.
📚 Tools and tips for your daily use¶
- jless is a command-line JSON viewer designed for reading, exploring, and searching through JSON data.
- Use GitLab and MITRE ATT&CK Navigator to visualize adversary techniques
- Kor is a tool to discover unused Kubernetes resources. It can identify ConfigMaps, Secrets, Services, ServiceAccounts, Deployments, StatefulSets, Roles.
- Slim Toolkit to inspect, optimize and debug your containers
- Kamaji turns any Kubernetes cluster into an “admin cluster” to orchestrate other Kubernetes clusters called “tenant clusters”. Kamaji is special because the Control Plane components are running in a single pod instead of dedicated machines. This solution makes running multiple Control Planes cheaper and easier to deploy and operate.
- For better and more strict error handling in shell scripts, set
set -Eeuo pipefail. More tips in Ten Things I Wish I’d Known About bash.
- Set up your infrastructure for on-demand, cloud-based development environments in GitLab
- Optimizing dependency management with Renovate for GitLab with 500+ repositories
- Terramate helps implement and maintain highly scalable Terraform projects by adding powerful capabilities such as code generation, stacks, orchestration, change detection, data sharing and more.
- Top 15 Kubectl plugins for security engineers by Sysdig.
- 90DaysOfDevOps by Michael Cade
- Alice and Bob Learn Application Security by Tanya Janca
🎯 Release speed-run¶
Cilium 1.14.0 brings Effortless Mutual Authentication, Service Mesh, Networking Beyond Kubernetes, High-Scale Multi-Cluster, and more (announcement blog post). Tracee v0.17.0 brings a new policy format compatible with Kubernetes CRDs, new flags for enhanced event filtering and simplified event sets. OpenSearch 2.9.0 brings search pipelines GA, production-ready neural search, ML framework GA, and monitors and detectors in OpenSearch Dashboards.
jq 1.7rc1 brings the first release after 5 years. GitLab 16.2 comes with a new Rich Text editor, improving the editing experience in issues, comments, wiki. It also supports keyless signing with Cosign, a new command palette for efficiency, triggering a Flux synchronization without any configuration and more.
Prometheus v2.46.0, OpenTelemetry Collector v0.82.0 and OpenTelemetry Collector Contribut v0.82.0, Perses 0.39.0, Keptn v1.4.1, Parca agent v0.23.3, Flux v2.0.1, Kyverno v1.10.2, Open Policy Agent v0.55.0
🎥 Events and CFPs¶
- Aug 22: Kubernetes Community Days Australia in Sydney, Australia.
- Sep 11-13: Container Days EU 2023 in Hamburg, Germany. See you there!
- Sep 13: eBPF Summit 2023, online.
- Sep 20-21: Swiss Cloud Native Day, Bern, Switzerland.
- Sep 26-27: Kubernetes Community Days Austria in Vienna, Austria.
- Sep 28-29: PromCon EU 2023 in Berlin, Germany.
- Oct 2-6: DEVOXX Belgium, Antwerp, Belgium.
- Oct 6-7: DevOps Camp Nuremberg, Nuremberg, Germany. See you there!
- Oct 10-12: SRECON EMEA in Dublin, Ireland.
- Oct 17-18: Kubernetes Community Days UK in London, UK.
- Nov 6-9: KubeCon NA 2023, Chicago, IL. Planning to be there.
- Nov 6: Observability Day at KubeCon NA 2023, Chicago, IL.
- Nov 6: CiliumCon at KubeCon NA 2023, Chicago, IL.
- Nov 6: AppDeveloperCon at KubeCon NA 2023, Chicago, IL.
- Nov 16-17: Continuous Lifecycle / Container Conf in Mannheim, Germany.
- Jan 1 - Mar 31: 90DaysOfDevOps 2024 Community Event, virtual, online.
👋 CFPs due soon
Looking for more CfPs?
- CFP Land.
- Developers Conferences Agenda by Aurélie Vache.
- Kube Events.
- GitLab Speaking Resources handbook.
"I'm pretty surethe application is somewhere around here" is a great comic meme for Kubernetes. Bill Mulligan went one step further, saying "Fixed it ..." (more on Twitter/X).
Thanks for reading! If you are viewing the website archive, make sure to subscribe to stay in the loop!