Matt Ray's ramblings

Preseed Installing Debian 11 on a Headless MacBook Pro

Fri, 06 May 2022 00:00:00 +0000

My previous employer had a generous refurbished hardware program where we could purchase used laptops at a greatly discounted rate. I took advantage of this and bought a total of 5 2015 MacBook Airs and the IT folks generously threw in a 13” MacBook Pro with a broken screen, knowing my penchant for resuscitating broken hardware.

The laptop screen would power up, but never displayed anything. While there was a potentially relevant Apple recall for displays, the repair was rejected because of apparent water damage to the machine. I attached a USB C HDMI adapter and that worked with macOS and I could have used the machine as a desktop with an external monitor, but I already had plenty of working laptops and didn’t really need it for that. I decided I would add it to my piecemeal cluster of Linux servers running Kubernetes, I just needed to get a clean installation of Debian on it.

Unfortunately USB C HDMI is not well-supported by the Debian installer. In fact, nothing worked beyond the initial grub installation screen. I initially went down the path of trying to use my USB serial console cable, but gave up after running into the harsh reality that USB serial consoles are very lightly documented (a few references in some grub documentation) and blocked by the USB Braille support in standard Debian installers (I did a build taking this out, still no luck).

Debian Preseeds

Debian preseeds are a technique for automated installations where a preseed file is provided that answers the installer questions. This can be baked into the installation medium or provided through DHCP or a network location. Given that I could get the first screen of the installer, I figured it would be easiest if I used the network location to rapidly iterate over getting the right configuration in place.

I was using the Debian 11 Netboot installer flashed onto a USB thumbdrive with Balena Etcher. On a MacBook Air or MacBook Pro, if you want to boot a USB drive you hold down the Option key while booting and you will eventually see an option titled EFI Boot, which is the Debian installer. This boots into the initial Debian installer menu which is grub providing various installation techniques.

Automated Installation

The Debian Installation Guide’s Contents of the preconfiguration file provides the basics that covered almost everything needed for the installation.

To run the preseed, select Advanced options ... and then select ... Automated install. Press the “E” key to “Edit Selection”. You will be presented with a screen similar to

setparams '... Automated install'

    set background_color=black
    linux   /install.amd/vmlinux auto=true priority=critical vga=788 --- quiet
    initrd  /install.amd/initrd.gz

You need to provide the URL of the preseed file, which was served up by the simple webserver lighttpd on http://10.0.0.2/preseed.cfg. In order to set the hostname and domain of the machine, I had to enter these as well because the preseed did not set them for some reason. The 3rd line is updated to look similar to this:

    linux   /install.amd/vmlinux auto=true priority=critical vga=788 --- quiet url=http://10.0.0.2/preseed.cfg netcfg/get_hostname=farnsworth netcfg/get_domain=bottlebru.sh

After making these changes, hit F10 to run the installation with your settings. I ran through this a few times with a working MacBook Air so I could verify the preseed settings were correct and iterate until I got everything right. Once it worked, I updated the preseed for the MacBook Pro and ran the installation. About 5 minutes later, I heard the distinctive Apple reboot sound and I was able to SSH into my new Linux server.

2016 MacBook Pro Preseed

Here are the contents of my preseed.cfg, I added comments inline. Some of these may be extraneous, but it worked. A few things to note:

localised for Australia
added mattray and root users with hashed passwords
formatted the entire drive as a single partition
the MacBook Pro used /dev/nvme0n1 as the disk instead of /dev/sda on the older MacBook Airs
there is an apt-cacher-ng proxy cache running at http://10.0.0.4:3142
the additional packages curl emacs-nox htop openssh-server rsync sudo usbutils zsh are installed
Remote root SSH is enabled so I can login to the machine once it installs

#_preseed_V1

### Network/Hostname
d-i netcfg/choose_interface select auto
d-i netcfg/get_hostname string farnsworth
d-i netcfg/get_domain string bottlebru.sh
d-i netcfg/hostname string farnsworth.bottlebru.sh
# the above settings never seemed to work, but adding
# netcfg/get_hostname=farnsworth netcfg/get_domain=bottlebru.sh
# to the grub CLI worked

### Localization
# some of this is probably overkill
d-i debian-installer/country string AU
d-i debian-installer/language string en_AU:en
d-i debian-installer/locale select en_AU.UTF-8
d-i keyboard-configuration/xkb-keymap select us
d-i keymap select us
d-i time/zone string Australia/Sydney
tzdata tzdata/Areas select Australia
tzdata tzdata/Zones/Australia select Sydney
tzsetup-udeb time/zone select Australia/Sydney

### Clock/NTP
d-i clock-setup/utc boolean true
d-i clock-setup/ntp boolean true

### Users
d-i passwd/user-fullname string Matt Ray
d-i passwd/username string mattray
# encrypted using a crypt(3) hash from whois package `mkpasswd -m sha-512`
d-i passwd/root-password-crypted password $6$w.JmcUilXD1bk/$39EE73ghigxMpHaip03dNKq0ZuzDYYD/gq2F2GAEQJxbGRlL0zidd3u5RrTBsokQ8Kid9x89bOLdzjcMfbck3/
d-i passwd/user-password-crypted password $6$Z3rruwk5Et$zwi8MxWYY5E4/l3JCBEG60VJHNKHTUhcbTAFPZHNQS7VR.HAJFlmMLxXx.Ld0SpM8HoXzY.T9XlC5R6nbmyAY1

### Partitioning
d-i partman-auto/choose_recipe select atomic
d-i partman-auto/confirm boolean true
d-i partman-auto/disk string /dev/nvme0n1
d-i partman-auto/method string regular
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true

### Package Sources
choose-mirror-bin mirror/http/countries select AU
choose-mirror-bin mirror/http/proxy string http://10.0.0.4:3142
d-i mirror/https/directory string /debian/
d-i mirror/https/hostname string deb.debian.org
d-i mirror/https/mirror select deb.debian.org

### System installation
d-i base-installer/install-recommends boolean false
d-i base-installer/kernel/override-image string linux-server
d-i debian-installer/splash boolean false
d-i pkgsel/include string curl emacs-nox htop openssh-server rsync sudo usbutils zsh
d-i pkgsel/install-language-support boolean false
d-i pkgsel/update-policy select none
d-i pkgsel/upgrade select full-upgrade
popularity-contest popularity-contest/participate boolean false
tasksel tasksel/first multiselect ssh-server
# the openssh settings never seemed to take
d-i preseed/late_command string \
 in-target sh -c 'sed -i "s/^#PermitRootLogin.*\$/PermitRootLogin yes/g" /etc/ssh/sshd_config';

### Finish and reboot
d-i grub-installer/only_debian boolean true
d-i grub-installer/bootdev  string /dev/nvme0n1
d-i finish-install/keep-consoles boolean false
d-i finish-install/reboot_in_progress note

USB C Ethernet Adapters

I have a USB C gigabit ethernet adapter that utilizes the Realtek RTL8153 chipset, but that isn’t supported in the Linux 5.10 kernel shipped with Debian Bullseye/Stable. For my installation, I used a known-working 100 megabit ethernet adapter with a USB A-C adapter. After the installation, I had to use the Debian Backports linux-image-amd64.

apt install linux-image-amd64/bullseye-backports

Farnsworth

With Debian Stable installed and the Backport kernel, I now have a new fastest machine in my cluster: (“farnsworth”).

Using Thunderbolt Networking under Linux on a Macbook Air

Fri, 11 Mar 2022 00:00:00 +0000

I have several 2015 13” MacBook Airs in various states of disrepair that I run Debian on as part of my home testing cluster. Given they don’t have ethernet ports, I’ve usually used USB network adapters because they are generally well-supported. One of my machines however, has a broken USB interface and I was unable to use this. I’d put it on a shelf to use for parts, but I found an Apple Thunderbolt to Gigabit Ethernet Adapter and figured I’d give that a whirl. Not surprisingly, Linux worked pretty seamlessly on 7+ year old technology.

After plugging it in, nothing happened. A bit of quick googling found that I needed the thunderbolt and tg3 kernel modules, so I added them with:

modprobe tg3 thunderbolt

The device now showed up under dmesg and lspci --nnk

0a:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM57762 Gigabit Ethernet PCIe [14e4:1682]
	Subsystem: Apple Inc. Thunderbolt to Gigabit Ethernet Adapter [106b:00f6]
	Kernel driver in use: tg3
	Kernel modules: tg3

But the network interface was not enabled according to ip address. I added the following to /etc/network/interfaces

# Thunderbolt
allow-hotplug ens9
iface ens9 inet dhcp

restarted my network and I was now connected via DHCP.

2: ens9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 68:5b:35:9d:a5:25 brd ff:ff:ff:ff:ff:ff
    altname enp10s0
    inet 10.0.0.22/24 brd 10.0.0.255 scope global dynamic ens9
       valid_lft 84917sec preferred_lft 84917sec
    inet6 fe80::6a5b:35ff:fe9d:a525/64 scope link
       valid_lft forever preferred_lft forever

It works after rebooting and I now have another machine for my cluster (“yancy”).

Enabling a USB Serial Console on Linux

Sun, 27 Feb 2022 00:00:00 +0000

Serial consoles have been used to communicate with headless networking and datacenter gear for decades. It works without requiring a screen, keyboard or networking which makes it helpful for debugging embedded devices and clusters of broken laptops and headless Arm boxes (which is my home lab). Most devices stopped shipping with serial ports years ago and USB to RS-232 serial adapters became common, but I picked up a USB to USB crossover serial adapter eliminating the need for separate cables. I purchased mine from Alibaba.com, which was cheaper than the name-brand alternatives (about $25 with shipping to Australia).

Configuring ttyUSB0

On Linux the serial console is usually available as ttyS0, and if it’s enabled you can connect to it with tools like screen or minicom. Physical and virtual consoles are connected to serial ports and the terminal with the application getty (“get tty”), presenting the user with a login prompt. A USB serial console would be available as ttyUSB0, so you need to connect the Linux service getty to ttyUSB0. For most recent Linux versions, this is managed by systemd so enabling and starting the service is as simple as:

systemctl enable getty@ttyUSB0.service
systemctl start getty@ttyUSB0.service

This creates a symlink from /etc/systemd/system/getty.target.wants/getty@ttyUSB0.service to /lib/systemd/system/getty@.service and starts listening on the USB serial console. I ran this on Raspbian 10, Armbian 11, and Debian 11 x86 and Arm systems and it worked across the board. I added it to my Chef recipe to ensure it stayed enabled across my current and any future machines.

Accessing the USB Serial Console with CoolTerm

I’m currently using macOS, so I initially tried out Serial which was very impressive but seemed like overkill for my limited needs. Roger Meier’s CoolTerm serial port terminal application was simple and worked like a charm.

To connect with CoolTerm follow these steps:

Start the CoolTerm application
Connect the USB serial crossover cable. I’m using a USB-C to USB-A adapter with my M1 Macbook Air, it works just fine.
Go to Options and select the Port similar to usbserial-AU03XLD7. If it is not listed, unplug the USB cable, replug it and click Re-Scan Serial Ports without the cable plugged into the remote machine yet.
The Baudrate should probably be 9600, this appears to be the default.
Data Bits should be 8, Parity None and Stop Bits 1, these are all defaults.
Under Terminal you may want to set Filter ASCII Escape Sequences if you use any special characters in your PS1 or see odd characters in your terminal.
I didn’t change any of the other defaults, it worked without having to mess with these.
Exit the Options and select Connect. At this point you should see a blank terminal and a timer with the number of bytes transmitted.
Plug the other end of the USB cable into your ttyUSB0 enabled device and a terminal should come right up. It works exactly like a virtual console or SSH session.

Unplugging it and plugging it into another machine behaves exactly like detaching a monitor. A USB console left open will stay open and switching to other machines will bring up their prompt accordingly. Have fun!

Installing Armbian 11 on a Pine64 ROCKPro64

Sun, 09 Jan 2022 00:00:00 +0000

Getting Started

The Pine64 ROCKPro64 is a fairly powerful 64-bit single board computer with a Rockchip RK3399 Hexa-Core (dual ARM Cortex A72 and quad ARM Cortex A53) processor and 4GB of RAM. I added it to my Arm devices homelab because it’s a more powerful CPU than the Raspberry Pi 4 but still priced at only $80 USD. I run Armbian on it because I had some trouble getting the official Debian 11 working, but they’re really similar in the end.

Installing the Base Image

I downloaded Armbian’s ROCKPro64 image and flashed it onto a 32 gig microSD card with Balena Etcher for macOS.

With my microSD card ready I plugged in an HDMI cable to my monitor and a USB keyboard and booted into Armbian. From the console, I updated the root password with passwd, created a mattray user and set my timezone. I’m not using wifi with this box, so the network worked out of the box.

Managing with Cinc

I installed the aarch64 build , which is a supported platform from Cinc. I got the packages from http://downloads.cinc.sh/files/stable/cinc/17.9.52/debian/11/ and installed them

apt install ./cinc_17.9.52-1_arm64.deb

I bootstrapped the boxes with my a home boxes cookbook and policyfile from my home-repo:

knife bootstrap mom -N mom --policy-group home --policy-name base

This standardizes the machine with the same packages, users, and other settings. I tagged the machine with

knife tag create mom cluster

To ensure that it picked up the cluster recipe which ensures all the members of the Kubernetes cluster have the right /etc/hosts entries and keep updated with the same version of Cinc.

Post Install

The ROCKPro64 is typically the K3s Server node when I’ve been testing out Kubernetes deployments. It’s the fastest of my Arm machines and just as easy to manage as any of the Raspberry Pi devices.

Installing Debian 11 on a Raspberry Pi Zero/3/4

Mon, 30 Aug 2021 00:00:00 +0000

Getting Started

In my home lab I have a several Raspberry Pi 4, Raspberry Pi 3 B+, and Raspberry Pi Zero W devices. I primarily ran Rasberry Pi OS Lite, but with the release of Debian 11 there are official Raspberry Pi images available with for a minimal installation. Setting up and configuring these is pretty straightforward, but I thought I’d add my notes for easier reference. The instructions are fairly similar for each, so I’m going to focus on the Raspberry Pi 4.

Installing the Base Image

I started with the Tested Images and downloaded the proper xz-compressed image for each machine type.

I flashed the image onto a 32 gig microSD card with Balena Etcher for OSX.

With my microSD card ready I plugged in my micro-HDMI adapter and HDMI cable and a USB keyboard and booted into Debian.

Configuring Debian

There are no non-root users, so you login as root with no password, which you’ll want to immediately change with passwd.

So I could work on the machine from my laptop, I enabled remote root SSH access (disable this later).

echo "PermitRootLogin yes" >> /etc/ssh/sshd_config

To set the hostname I ran:

hostnamectl set-hostname larry.bottlebru.sh

The images come with wireless support, so it was simply a matter of adding the following to /etc/network/interfaces.d/wlan0

allow-hotplug wlan0
iface wlan0 inet dhcp
    wpa-ssid MYNETWORK
    wpa-psk MYPASSWORD

Unlike with Raspberry Pi OS, the images don’t come with swap enabled, so no need to disable it. The partitions are also resized automatically, so there are no manual steps related to that either.

A quick reboot and I was able to SSH in as root remotely and administer the machine remotely.

Managing with Cinc

The Raspberry Pi 3 & 4 are running aarch64, which is a supported platform from Cinc. I got the packages from http://downloads.cinc.sh/files/stable/cinc/17.4.38/debian/10/ and installed them

apt install ./cinc-17.4.48-rpi-armv6l_armel.deb

I bootstrapped the boxes with my a home boxes cookbook and policyfile from my home-repo:

knife bootstrap larry -N larry --policy-group home --policy-name base

Now it was ready to use.

The Raspberry Pi Zero identified as armel instead of the armhf that was used for the Raspberry Pi OS builds I’d built previously, so I had to do a fresh build for the machines. Because I still have Raspberry Pi OS 10 applications running, I haven’t upgraded these nodes yet.

Switching To Zsh

Wed, 26 May 2021 00:00:00 +0000

Switching to zsh (updated for Emacs M-x shell)

I may be late to the party, but I’ve finally switched from bash to zsh. It wasn’t due to any failings of bash, it’s because I’ve been using it inside of Emacs M-x shell and never felt the need the fancy features of Oh My Zsh or any of the other new and improved shells. The Starship shell prompt is a vast improvement on the PS1 settings and scripts available in bash as well. What prompted the change was macOS and Arch Linux both defaulting to it on installation.

Making the Switch

There were a pair of useful articles I read beforehand:

[Switching to ZSH](http://zpalexander.com/switching-to-zsh/)
[Moving to zsh](https://scriptingosx.com/2019/06/moving-to-zsh/)

After reading through those, I figured I’d just make the switch and suffer the fallout.

.zshenv

The .zshenv file is used for setting user’s environment variables. I didn’t really have too many to set, but it is convenient to keep them in 1 place.

HISTSIZE=200000

GOPATH=$HOME/go
PATH=$PATH:$HOME/bin:$GOPATH/bin

EDITOR=emacsclient
GIT_EDITOR=$EDITOR

This HISTSIZE is used for maintaining the maximum size of the .zsh_history file. The $PATH settings are adding my home and local Go bin directories. EDITOR is set for anything that needs to access my preferred editor of Emacs and GIT_EDITOR is specific to Git.

.zshrc

The .zshrc is used for setting user’s interactive shell configuration and executing commands, and is read when starting an interactive shell. The advice was to forgo using an additional .zsh_profile and just use a .zshrc.

There are several setopt commands for setting .zsh_history settings (documented in the file) and the various aliases I prefer. The block

# emacs M-x shell
if [[ "dumb" == $TERM ]] ; then
  alias l='cat'
  alias less='cat'
  alias m='cat'
  alias more='cat'
  export PAGER=cat
  export TERM=xterm-256color
else
  alias l='less'
  alias m='more'
fi

explicitly sets behavior so M-x shell and iTerm behaved as expected. The GIT_EDITOR was not picked up when commiting files from the CLI within M-x shell, so I had to export GIT_EDITOR=$EDITOR to get the desired behavior of not opening another Emacs window when commiting. The export TERM=xterm-256color is to get around this Starship bug.

My PS1 is managed by my Starship config file.

Emacs

To ensure that M-x shell used zsh instead of bash I set the custom variables

 '(explicit-shell-file-name "/bin/zsh")
 '(explicit-zsh-args '("--interactive" "--login"))

and to off echoed commands from zsh added

 '(comint-process-echoes 0)

to my .emacs.d/init.el.

I did not use AUTO_CD settings because Emacs tended to lose track of the CWD and I don’t care enough to figure out how to use Dirtrack.

I added the following to my .zshrc to prevent Tramp from hanging on remote ssh zsh connections.

[[ $TERM == "dumb" ]] && unsetopt zle && PS1='$ '

Building An External Monitor From An Ipad

Tue, 02 Mar 2021 00:00:00 +0000

Building an External 9.7” Monitor from an iPad

I’ve always had a soft spot for repurposing old hardware so when I stumbled across the video Build a DIY screen out of recycled parts for cheap I knew I had to see what I could build. Sure there is lots of software for the iPad that will allow you to use it as an external monitor, but mine was an original 2010 iPad model that could no longer connect to the Apple store to install new packages.

Lenovo Thinkpad Yoga 12

After watching the linked video I originally disassembled a broken Lenovo ThinkPad Yoga 12 and scavenged the reusable parts out of it: Samsung SSD, Intel AC 7265 wireless card, the USB camera, and the screen. The Samsung SSD went into my kids’ gaming machine as an extra 192GB drive. The Intel wireless card could be used in my AMD workstation, I just haven’t hooked it up because I already have an ethernet connection. I found the model of the Lenovo screen from the Hardware Maintenance Manual and ordered the proper HDMI LCD controller board for $13.50 USD with shipping from AliExpress. It showed up, I plugged it in, and it worked at 1920x1080 resolution! Unfortunately I broke the screen trying to remove it from the laptop lid, so I had to start over.

2010 iPad

After nearly succeeding with my first attempt at building an external monitor, I knew I wanted to try again. My other “broken” laptops are still in use as Linux servers, but I had an original iPad that I had barely used since it stopped receiving iOS updates several years ago. I followed iFixit’s iPad Wi-Fi Display Frame Replacement to disassemble it, but stopped short of removing it from the glass front (lesson learned). I then went back to AliExpress and found what looked like the right model HDMI VGA LCD controller for about $19 with shipping.

A few weeks later it finally showed up and didn’t work. Did I have a bad unit, had I broken my iPad screen, or had I done something wrong?

Try, Try Again

Undeterred, I decided to try AliExpress again. You’d think with a name like Yqwsyxl LCD parts Store you’d get quality, but I decided since I was already down $19 I’d go with CZHAO Accessories Store and pay the $21 premium (with international shipping). AliExpress prices and shipping times seem to vary wildly in response to US, Australian, and Chinese currency fluctuations and this took nearly 6 weeks to show up.

Success!

Once it finally arrived, I connected the HDMI LCD controller to the display. I connected an HDMI cable, a 12v power adapter and turned it on. It powered up, the OSD display came on and reported “Looking for Source”. I plugged it into my laptop and it appeared as a 1024x768 VGA monitor.

Usage

Once I had it working, I ordered a gooseneck tablet stand and set it up next to my main 32” monitor, above my laptop. I was using it primarily for terminals, my calendar, or Slack but my son convinced me “you have too many monitors and I only have my laptop”, so now he uses it for Discord. For about $35, you should be able to convert an old laptop or tablet to a working external monitor with a stand.

Chef Infra and Cinc Client Linux Builds for 32-bit ARM EL Platforms

Tue, 09 Feb 2021 00:00:00 +0000

Graham Weldon has taken the existing instructions and created new armel builds for running on switches and other similar hardware. The build scripts do not require additional patches, but Graham has documented how to create the builds. He has graciously allowed me to host his builds, so I’ll be including them on the 32-bit Arm Chef Infra and Cinc Client Builds for Linux page going forward.

Chef Infra and Cinc Client Linux Builds for 32-bit ARM

Mon, 29 Jun 2020 00:00:00 +0000

Rather than continue to add new blog posts for each and every Chef Infra and Cinc client build, I’ve added a new 32-bit Arm Chef Infra and Cinc Client Builds for Linux page. I’ll continue publish builds and publicizing them, but only updating the landing page.

Uploading Data Bags to Chef Infra Server with knife, knife raw, and curl

Fri, 26 Jun 2020 00:00:00 +0000

The Chef Infra Server API is fairly well-documented and includes many examples. Data bags are a way to store JSON on the Chef Infra Server. The customer I’m working with needed an example of uploading data bags via the API because they wanted to upload JSON into them without using the CLI knife tool from ServiceNow. While there are supported Ruby and Go Chef Server APIs available, they wanted a curl example. This post covers locking down a set of credential and progressing from the knife CLI to using curl.

Limited Privilege Client Preparation

In order to interact with the API, we will need a client key with permissions limited to working with data bags. These commands must be run on the Chef Server. First, we’ll create the organization for testing:

chef-server-ctl org-create test1 'Test Organization 1' -f test1-validator.pem

Because I’m using a self-signed certificate and not every knife subcommand takes --node-ssl-verify-mode, I have this additional no_ssl.rb configuration file. You may not need this step if you have a proper certificate.

ssl_verify_mode :verify_none

Create the data bag manager client

Next we need to create a dbm data bag manager client that manage data bags.

knife client create dbm -f dbm.pem -k /etc/opscode/pivotal.pem -u pivotal -s "https://localhost/organizations/test1" -c no_ssl.rb --disable-editing

Grant data bag permissions

knife acl add client dbm containers data create,read,update,delete -k /etc/opscode/pivotal.pem -u pivotal -s "https://localhost/organizations/test1" -c no_ssl.rb

We’re now done with this organization on the Chef Infra Server. Copy the the dbm.pem off the Chef Infra Server to the destination.

Reusing the dbm client across multiple organizations

Having a dbm client for managing data bags within a particular organization is useful, but if you had a large number of organizations you’d probably prefer to reuse that client across them. knife client create will allow you to provide your own public key, so the following can be used to create the public key from the generated private key, then reuse it for additional organizations (in our case test2).

chef-server-ctl list-client-keys test1 dbm -v | tail -10 > dbm.pub
knife client create dbm --public-key dbm.pub -k /etc/opscode/pivotal.pem -u pivotal -s "https://localhost/organizations/test2" -c no_ssl.rb --disable-editing

From here out, the rest of the commands with the dbm client would be the same, replacing test1 with test2 or similar for your other organizations.

Using knife

If we have access to a CLI, the easiest way to manage data bags would be to use knife data bag. We could create the data bag srs

knife data bag create srs -k dbm.pem -u dbm -s "https://ndnd/organizations/test1" -c no_ssl.rb

Upload an item (defined in a data_bags/srs/h.json) file.

$ cat data_bags/srs/h.json
{
    "id": "hyperchicken",
    "payload": {
        "one": "1"
    }
}
$ knife data bag from file srs h.json -k dbm.pem -u dbm -s "https://ndnd/organizations/test1" -c no_ssl.rb
Updated data_bag_item[srs::hyperchicken]

We can also list, show and delete the items.

$ knife data bag list -k dbm.pem -u dbm -s "https://ndnd/organizations/test1" -c no_ssl.rb
srs
$ knife data bag show srs hyperchicken -k dbm.pem -u dbm -s "https://ndnd/organizations/test1" -c no_ssl.rb
id:      hyperchicken
payload:
  one:   1
$ knife data bag delete srs -k dbm.pem -u dbm -s "https://ndnd/organizations/test1" -c no_ssl.rb -y
Deleted data_bag[srs]

knife raw

The knife raw command allows us to pass straight JSON to API endpoints with the knife CLI handling authentication. Here are examples of how to create the srs data bag

$ echo '{ "name":"srs" }' > srs.json
$ knife raw -m POST /data -i srs.json -k dbm.pem -u dbm -s "https://ndnd/organizations/test1" -c no_ssl.rb
{
  "uri": "https://ndnd/organizations/test1/data/srs"
}
$ knife raw -m GET /data -k dbm.pem -u dbm -s "https://ndnd/organizations/test1" -c no_ssl.rb
{
  "srs": "https://ndnd/organizations/test1/data/srs"
}

Create and view the hyperchicken item

$ knife raw -m POST /data/srs -i data_bags/srs/h.json -k dbm.pem -u dbm -s "https://ndnd/organizations/test1" -c no_ssl.rb
{
  "id": "hyperchicken",
  "payload": {
    "one": "1"
  },
  "chef_type": "data_bag_item",
  "data_bag": "srs"
}
$ knife raw -m GET /data/srs/hyperchicken -k dbm.pem -u dbm -s "https://ndnd/organizations/test1" -c no_ssl.rb
{
  "id": "hyperchicken",
  "payload": {
    "one": "1"
  }
}

and we can delete the data bag item and entire data bag

$ knife raw -m DELETE /data/srs/hyperchicken -k dbm.pem -u dbm -s "https://ndnd/organizations/test1" -c no_ssl.rb
{
  "name": "data_bag_item_srs_hyperchicken",
  "json_class": "Chef::DataBagItem",
  "chef_type": "data_bag_item",
  "data_bag": "srs",
  "raw_data": {
    "id": "hyperchicken",
    "payload": {
      "one": "1"
    }
  }
}
$ knife raw -m GET /data/srs -k dbm.pem -u dbm -s "https://ndnd/organizations/test1" -c no_ssl.rb
{

}
$ knife raw -m DELETE /data/srs -k dbm.pem -u dbm -s "https://ndnd/organizations/test1" -c no_ssl.rb
{
  "name": "srs",
  "json_class": "Chef::DataBag",
  "chef_type": "data_bag"
}
$ knife data bag list -k dbm.pem -u dbm -s "https://ndnd/organizations/test1" -c no_ssl.rb

curl

Interacting with the Chef Infra Server API requires authenticating with the client’s private key and encoding the messages via openssl. While knife handles this under the covers, there is a detailed curl example in the documentation. I’ve pared it down a bit for brevity, this currently only handles GETs:

#!/usr/bin/env bash

_chomp () {
  # helper function to remove newlines
  awk '{printf "%s", $0}'
}

chef_api_request() {
  # This is the meat-and-potatoes, or rice-and-vegetables, your preference really.

  local method path body timestamp chef_server_url client_name hashed_body hashed_path
  local canonical_request headers auth_headers

  chef_server_url="https://ndnd/organizations/test1"
  if echo $chef_server_url | grep -q "/organizations/" ; then
    endpoint=/organizations/${chef_server_url#*/organizations/}${2%%\?*}
  else
    endpoint=${2%%\?*}
  fi
  path=${chef_server_url}$2
  client_name="dbm"
  method=$1
  body=$3

  hashed_path=$(echo -n "$endpoint" | openssl dgst -sha1 -binary | openssl enc -base64)
  hashed_body=$(echo -n "$body" | openssl dgst -sha1 -binary | openssl enc -base64)
  timestamp=$(date -u "+%Y-%m-%dT%H:%M:%SZ")

  canonical_request="Method:$method\nHashed Path:$hashed_path\nX-Ops-Content-Hash:$hashed_body\nX-Ops-Timestamp:$timestamp\nX-Ops-UserId:$client_name"
  headers="-H X-Ops-Timestamp:$timestamp \
    -H X-Ops-Userid:$client_name \
    -H X-Chef-Version:0.10.4 \
    -H Accept:application/json \
    -H X-Ops-Content-Hash:$hashed_body \
    -H X-Ops-Sign:version=1.0"

  auth_headers=$(printf "$canonical_request" | openssl rsautl -sign -inkey "dbm.pem" | openssl enc -base64 | _chomp |  awk '{ll=int(length/60);i=0; \
    while (i<=ll) {printf " -H X-Ops-Authorization-%s:%s", i+1, substr($0,i*60+1,60);i=i+1}}')

  case $method in
    GET)
      curl_command="curl -k $headers $auth_headers $path"
      echo $curl_command
      $curl_command
      ;;
    *)
      echo "Unknown Method. I only know: GET" >&2
      return 1
      ;;
    esac
  }

 chef_api_request "$@"

Note that the values of the chef_server_url and dbm user and key are hard-coded rather than take them from a config file. Output will be similar to

$ bash chef_curl.sh GET /data/srs/hyperchicken
curl -k -H X-Ops-Timestamp:2020-06-26T06:20:34Z -H X-Ops-Userid:dbm -H X-Chef-Version:0.10.4 -H Accept:application/json -H X-Ops-Content-Hash:2jmj7l5rSw0yVb/vlWAYkK/YBwk= -H X-Ops-Sign:version=1.0 -H X-Ops-Authorization-1:gVUUo9JoIXdZhFxYLgsvEOwKsmMFJLC6M+bMFu1fX6PRVtfyTfgMx2UzBcd5 -H X-Ops-Authorization-2:K0wFnDO8+fIHjMTyu3nQf2LBVCTn/S4SjKMSTMxKF0uN9BW7DU1CBYUmuOXO -H X-Ops-Authorization-3:C3ADAvmvC5N7lIJnSSffrTojSBuhWUeD16wCBqHtfh/AC4WVzPhe5t+gPpoE -H X-Ops-Authorization-4:moYCh2dyx6dnZ+NsfvU9UsfmiJJdNST7ur5kHnCh3bBgARvch8oilzBLlnQk -H X-Ops-Authorization-5:w+a5rVk3NwIDG/WNH95cYNxMtLt+WLyJVIlbMLWXlf/iZylqkcdEbckGtJjJ -H X-Ops-Authorization-6:9GMEhijpP9I+skDAjvw3pLLAHIfXH5jZSAAawxSCdw== https://ndnd/organizations/test1/data/srs/hyperchicken
{"id":"hyperchicken","payload":{"one":"1"}}

You can download it directly here.