2022-12-16: Kubernetes 1.26, Keptn 1.0, Flux graduation, Prometheus turns 10, GitLab Web IDE Beta, ripgrep, xq¶
Thanks for reading the web version, you can subscribe to the Ops In Dev newsletter to receive it in your mail inbox.
👋 Hey, lovely to see you again¶
A human writes this newsletter. Don't believe me? Here's a view from my new flat - and yes, the big move is made, focussing on all things that are missing (a kitchen would be lovely. Coming in January :-)). Returning to work and community, re-capping what I learned at the great in-person events in the past months, and collecting ideas for fresh content and talks for 2023.
There are plenty of interesting topics, and I wrote a blog post about Hurl, a wonderful CLI tool for website and HTTP API testing. If you do not read everything, don't worry - you can always come back to the online archive and search all past newsletters.
Make sure to take time off over the holidays to refresh your batteries! See you on the other side :-)
☕ Hot Topics¶
- Prometheus turns 10: What's Been Its Impact on Observability?. IMHO: It changed my way of thinking about monitoring, metrics, and beyond, as a former OSS monitoring tool maintainer and admin myself.
- How we diagnosed and resolved Redis latency spikes with BPF and other tools is a thorough learning walkthrough from a problem, analysis, attempts, to final solutions.
- Why and How eBay Pivoted to OpenTelemetry is an excellent use case write-up on challenges with Observability, and how an open standard helped achieve goals and future visions.
🎯 Release speed-run¶
Kubernetes 1.26 brings built-in admission control, building on the ideas of Kyverno and OPA. The userspace mode of kube-proxy is removed, inviting to use a fast eBPF solution such as Cilium instead - thanks Liz Rice. The container registry changed to
registry.k8s.io, and the release artifacts are now signed using cosign, and can be verified. More insights in this article from Datadog, and the illustrated changelog from Aurélie Vache. ⚡
Keptn 1.0 continues the path as a cloud-native SLO-driven lifecycle orchestrator, beyond quality gates where it started 2+ years ago. Cheers to the team reaching such an important milestone! 💥
Perses v0.20.0 brings more improvements for dashboards as code and better Observability. While still in development, I highly recommend subscribing to release updates. 📈
🛡️ The Sec in Ops in Dev¶
Brendan O'Leary pointed me to Chaos, a DNS dataset API, which aims to help analyze internet wide changes. Maintained by Projectdiscovery.io, who are building security tools for asset management and vulnerability scanning, for example nuclei.
Starting in April 2023, Amazon S3 will introduce two new default bucket security settings by automatically enabling S3 Block Public Access and disabling S3 access control lists (ACLs) for all new S3 buckets. With these new defaults, the few applications that need their buckets to be publicly accessible or use ACLs must deliberately configure their buckets to be public or use ACLs.
⛅ Cloud Native¶
“Flux has been an exciting project to watch as GitOps becomes mainstream,” said Chris Aniszczyk, CTO, Cloud Native Computing Foundation. “Flux users have not only adopted advanced uses of GitOps and progressive delivery, but they have done so together with key projects in the CNCF landscape like Flagger, Helm, Prometheus, and more. Flux’s graduation is a good indicator of the success and promise of cloud native technologies.”
AWS released Finch, a new open-source client for container development. macOS is the first supported OS, using open-source components including Lima, nerdctl, containerd, and BuildKit. Love to see them building in public: "Rather than iterating in private and releasing a finished project, we feel open source is most successful when diverse voices come to the party.".
The remote-write feature in Prometheus can be used in different ways: Sending metrics to long-term storage, collecting metrics from distributed (Kubernetes) clusters, or integrating with hyper cloud provider solutions ("Managed Prometheus Service"). This article explains the basic concepts, discusses root causes with increased ingestion rates, and optimization options for remote write tuning, and deeper analytics on performance scaling.
Understanding OpenTelemetry concepts can feel overwhelming. Sometimes it is great to instead peek into how to things are built in code. What is a receiver, and how to build one? The exhaustive documentation walks you through all steps, including code design patterns (factories, etc.) and adding "check your work" learning reviews. Impressive! Sascha Grunert wrote insights articles about Kubernetes container runtime observability with OpenTelemetry and finding suspicious syscalls with the seccomp notifier. Last but not least, migrating from OpenTracing can be a planning challenge too. Thanks Sonja Chevre for helping shed light!
Honeycomb.io often inspires with observability innovation, and this feature highlight is no difference: A new service map that isn't just a static dashboard - zoom in, filter, add aggregated query views or traces seamlessly. I also learned about BubbleUp, how it helps with verifying SLOs, and much more observability data analytics at scale in this article. Mindblowing.
🔍 The inner Dev¶
Last month I stumbled over Hurl to test websites and HTTP APIs, with lots of interest - and the idea to run Hurl continuously in CI/CD too. This blog post walks you through the first steps with Hurl, to response assertions, XML/JSON parsing, using the JUnit test report integration in GitLab, and running tests against deployed review apps. I had a lot of fun writing the blog post, including great discussions with the maintainers, and exceptionally good documentation. Recommend bookmarking the blog post and Hurl as a CLI tool.
The new Web IDE in GitLab will be in Beta soon (Dec 19+). It provides a familiar UI with VS Code and adds new possibilities for remote development too. As a GitLab team member, I was super excited after getting to know VS Code in the browser with Gitpod. I got early access to the feature flag on GitLab.com SaaS - this newsletter is written in the Web IDE Beta, persisted in the public project, and sent via buttondown emailing.
Shopify adopts Rust for system programming, looking into high-performance servers, Ruby extensions, compiling into WebAssembly, scaling, community, a great library ecosystem for productivity, safety with regards to more errors on compile time, and much more. Great read!
The C programming language receives a new standard: C23. This article sheds insights into what this means for C libraries, and can be a helpful resource when updating source code to the new standard in the future. Python developers, and Ops folks who run services like gunicorn, will love Python 3.11's CPU usage improvements.
📈 Your next project could be ...¶
- Explore OpenCost for real-time Kubernees cost monitoring.
- Dive into Fuzzing, learn more about AFL++ and explore the features of the Advanced Fuzzing Library, including a /metrics endpoint for Prometheus.
- Try a highly optimized, easy-to-use, auto-upgradable, HA-default & Load-Balanced, Kubernetes cluster powered by k3s-on-MicroOS and deployed for peanuts on Hetzner Cloud using this project.
📚 Tools and tips for your daily use¶
- ripgrep recursively searches directories for a regex pattern, respects .gitignore, and is faster than GNU grep. Tested it on the GitLab handbook source with 2,000+ pages.
- xq is a command-line XML and HTML beautifier and content extractor, for example, extracting the title from RSS feeds:
curl -s https://about.gitlab.com/security-releases.xml | xq -x //title.
- KubeShark, an API traffic viewer for Kubernetes, providing deep visibility into all API traffic. Familiar names? Wireshark, tcpdump GUI.
- Kaniko CI/CD template to build container images, example from the GitLab infrastructure team.
- You can copy-paste a spreadsheet into GitLab Markdown and it renders the table for you.
- DWARF-based Stack Walking Using eBPF - and follow Polar Signals, amazing learning content on their blog.
- OpenSSF Landscape, similar to the CNCF landscape, but for security.
- Why are we paying these folks - a tale of DevRel. This a great reminder that DevRel is not easy, and why I love our public handbook.
🎥 Events and CFPs¶
- Feb 4-5: FOSDEM 2023, Brussels, Belgium. See you there!
- Feb 6-8: Config Management Camp, Ghent, Belgium. See you there!
- Feb 7-8: GitLab Contributor Days 2023.1, Ghent, Belgium (orga issue, the same location as Config Management Camp). See you there!
- Feb 7-8: Civo Navigate, Tampa, Florida.
- Feb 23-24: KCD Amsterdam, Amsterdam, The Netherlands
- Apr 17-21: KubeCon EU, Amsterdam, The Netherlands. See you there!
👋 CFPs due soon
- Mar 11-12: Chemnitz Linux Days 2023, CFP closes Jan 4.
- Apr 15-16: Cloud-Native Rejekts EU 2023, CFP opens Feb 6 and closes Feb 13.
- May 15-18: SLOConf 2023, CFP closes Jan 31.
- June 15: KCD Zurich, in Switzerland. CFP opens soon.
- June/July: KCD Munich, in Germany. CFP opens soon.
- June 20-23: CloudLand 2023, at Phantasialand, Germany. CFP closes Jan 31.
- June 26-28: Monitorama 2023 in Portland, OR. CFP closes Feb 3.
Looking for more CfPs? Try CFP Land.
The creators of Alba: A Wildlife adventure published an addictive learning game with a great atmosphere to relax my brain for a few hours. Played in one go on PS5, other platforms are available too.
Thanks for reading! If you are viewing the website archive, make sure to subscribe to stay in the loop!
See you next month - let me know what you think on LinkedIn.
PS: If you want to share items for the next newsletter, please check out the contributing guide - tag me in the comments or send me a DM. Thanks!