Skip to content

2022-02-03 Welcome to opsindev.news - The Ops in Dev

Thanks for reading the web version, you can subscribe to the Ops In Dev newsletter to receive it in your mail inbox.

👋 Hey, nice to meet you

Yet another newsletter, you may wonder? Yes. Are you transitioning from Dev to Ops to DevOps, trying to learn the value in metrics and app instrumentation? Are you interested in solving engineering and Ops challenges and thinking about day-2-ops? Welcome to my story, you are invited to learn together :)

I started as a backend developer, joined an OSS monitoring project, learned how the software was used in Ops and learned to love the performance insights and tools to become a better developer. Since then, I see myself as the Ops in Dev, or a developer turned ops, with all mistakes on the way, now learning Kubernetes and cloud-native microservices and struggling with code instrumentation myself. It's been an exciting learning story in the past 20 years.

Tip: Subscribers will get an email with a newsletter subscription. opsindev.news provides an archive website including search possibilities. You can also subscribe to the RSS feed. The newsletter is organized in https://gitlab.com/dnsmichi/opsindev.news/, transparent and public so that everyone can contribute.

The newsletter is structured with headings - you can read it from top to bottom, or just jump into sections you like :)

☕ Hot Topics

👁️ Observability - Getting started is overwhelming

When you start with Observability, you might hear "app instrumentation", "adding metrics to your code", and other ops and SRE-related thoughts. It can get overwhelming quickly - try to focus and learn the why and then the how.

I've shared my horror debugging stories in my talk at Chaos Carnival last week (slides, recording); situations where I would have loved to have insights before they cause production problems for all teams involved.

Where should you start? My recommendation: Use a common standard/tool (Prometheus) with mature client libraries. For Rust, the Prometheus project released the official Rust client library, this month. The Golang client libraries have been updated too.

Learning from existing implementations can help too. I like reading the Git history and Git diffs to see the beginning and thought process and get inspired myself. PHP FPM added support for OpenMetrics for example. For writing your own exporters (a service which collects data from other services and APIs), Julius Volz suggested on Twitter to use the exporter toolkit. The thread has more ideas for getting started with writing your own Prometheus exporter :)

👀 Observability - OpenTelemetry, new ideas

Everyone talks about OpenTelemetry, providing a specification and framework for traces, and later metrics and logs. It is relatively new and can be overwhelming for beginners. Here are a few suggestions to make your journey easier.

"Bring your own backend." - OpenTelemetry provides the collector, the specification and requires you to define the backends. Jaeger for Tracing, Prometheus for metrics. Vendors can create their own OpenTelemetry distribution, enriching the code and data with more context. Keep the latter in mind when you read about the "AWS OpenTelemetry distribution" in the future.

Logz.io created a great learning series for OpenTelemetry getting started in various languages:

Instrumenting your own code can be learned, but how do you go along with someone else's code? Auto-instrumentation can help - make sure to hear about the pros and cons with this approach with OpenTelemetry, shared by Honeycomb.io.

🔥 Chaos Engineering and more

Simulating a production incident to test reliability and observability can be a challenge. Chaos engineering brings a new building block into the DevOps and Observability platform. The engineering teams at LEGO tackle their Ops challenges with chaos engineering - learn more in Happy Chaotic times at the LEGO Group.

A distilled introduction to Chaos Engineering lends a great hand to understand the principles and ideas behind Chaos Engineering. Chaos Carnival 2022 happened last week, I recommend checking out the playlists for Day 1 and Day 2 to chime into the talks and audience questions too.

There are many more resources for Observability and Chaos Engineering. I have started collecting learning resources on o11y.love, and will work on creating blog posts, workshops, and more in 2022. Michael Hausenblas creates a great newsletter at o11y.news/ to stay in the loop on the hottest topics. Highly recommend subscribing too.

Tip: Watch the space for CI/CD Observability in 2022, following the article about Overcoming observability challenges in the CI/CD pipeline. I have created a feature proposal for GitLab CI/CD to implement OpenTelementry with tracing in the first iteration.

🛡️ The Sec in Ops in Dev

You have created your Kubernetes cluster, production is deployed continuously, and you'll recognize unauthorized data access. How do you to tackle security challenges in Kubernetes?

🔐 Secrets Management

The Twitter thread from Anaïs Urlichs has many great insights: HashiCorp Vault, Mozilla SOPS, GCP KMS, GitLab CI/CD, AWS Secrets Manager. Thanks Niclas and Dennis for your answers!

Zero Trust principles with Hashicorp Vault, Consul, and Boundary are shown in Enabling Zero Trust at the Device/Machine and Human/User Layers.

☸️ Switching roles: Hacking Kubernetes

Kubernetes is a framework with many components, and it can be overwhelming trying to understand them all - both for Observability and Security. You'll find a magnitude of tools and ideas - "Open Policy Agent or Kyverno?" is just one of the questions.

Andrew Martin and Michael Hausenblas wrote a new book, switching the roles - Hacking Kubernetes is describing the problems, attack vectors and then discussing the solutions. Recommended by my friends, I've started reading the first chapters :)

🌐 Monitoring syscalls with Falco

Want to monitor syscalls with Falco? Read this great post from Federico Di Pierro and Jason Dellaluce on the Falco blog. Another example is to use syscall monitoring to detect packages doing malicious things.

🔍 The inner Dev

Did you know that several routines in Golang execute assembler instructions? There is a proposal to add a native instruction implementation for amd64 (x86_64) to improve the sha256 function.

Fantastic Symbols and Where to Find Them - Part 2 provides a great deep dive into profiling, debugging symbols, and much more.

"What happens when you make an HTTP request in Go?" asked Felix Geisendörfer on Twitter, teasing a new function call tracer for Go. It looks like there are more great tools coming for making the engineer's life easier.

Now that Generics in Go have arrived, what else is missing? Seth Vargo wrote a great blog post, suggesting a modern templating engine, pass values by references in the range clause, deterministic selects with a default branch, a unified interface for structured logging, multi-error handling, and many more great ideas. Hopefully, we'll see them all in Go 2.0!

📈 Your next project could be ...

Looking for a side project to learn something new? Here are some suggestions :)

  • New command prompt: Starship, describing itself as "The minimal, blazing-fast, and infinitely customizable prompt for any shell!".
  • Terraform for DB migrations: Atlas
  • Penpot is an OSS alternative to Figma. Thanks Michael Aigner for sharing!

📚 Tools and tips for your daily use

Tools and tips shared by our amazing community.

📦 wagoodman/dive is a tool for exploring a docker image, layer contents, and discovering ways to shrink the size of your Docker/OCI image.

🔍 Great network debugging tip with remote capture of packets using Wireshark and tcpdump over SSH, example for macOS:

ssh host  tcpdump -i any -U -Ss0 -w - 'port 31337' | /Applications/Wireshark.app/Contents/MacOS/Wireshark   -k -i -

🏗️ How to create Kubernetes manifests quickly.

kubectl create deployment foo \
  --image=nginx:1.21 \
  --dry-run=client \
  -o yaml

🤝 Brian Williams shared a GitLab productivity tip Reviewing a large MR and want to look at a specific file? Cmd + P, type in file name. - Thanks Brian!

✍️ A quick markdown cheatsheet shared from Michael Aigner's bookmarks - thanks!

🎄 Pauline Narvas and Curtis Einsmann shared a life-changing tip for VS Code trees and their indent for better visualization. 

"workbench.tree.indent": 20

🔑 "How to remove an (alternative) domain from a Lets Encrypt (SAN) certificate using certbot" by Claudio Künzler might come in handy too.

🐋 If you are looking for a Docker Desktop replacement, make sure to checkout Rancher Desktop. It provides lightweight k3s and containers. Other alternatives include podman, nerdctl and more.

🌐 Julia Evans published a new zine about "Everything in a DNS packet". Bookmark it while DNS is not a problem (yet)!

🎥 Events and CfPs

Considering to speak at events, or plan with attending? Here's a list for Feb 2022.

Beta: You can follow my events on https://cfps.dev/u/dnsmichi/events. Thanks Brendan for creating cfps.dev!

🎤 Shoutouts

Shoutout and kudos to Michael Hausenblas who created the idea with o11y.news and was so kind to share MkDocs and buttondown scripts and configuration to learn from! Special thanks to Niclas Mietz, Philip Welz, and Michael Aigner for helping review the first newsletter issue! ❤️

Thanks for reading! If you are viewing the website, make sure to subscribe to stay in the loop!

See you next month - let me know what you think on Twitter or LinkedIn.

Cheers,

Michael

PS: If you want to share items for the next newsletter, please check out the contributing guide. Thanks!