A System-Witch's Package Manager Murder Mystery
It was a calm day on the puppy-linux thinkpad. The CPU was cool, the RAM was operating at double data rate, and a gentle breeze flowed through the chassis as the fan whirred away. A girl wanted to log onto Discord to chat with her friends, but much to her dismay, Discord needed a package update! No matter, such things happen from time to time, but she’d need to uninstall the old version first. She threw on her systems-witch hat and set to work.
The girl walked over to urxvt and activated her package manager. “Hello package manager, could you pkg u discord”? The package manager began to reply, “Uninstall the package discord:”.
There was a pause. Package managers need to think things over sometimes. Suddenly, she heard a shriek over her terminal:
/usr/sbin/pkg: line 5912: grep: Argument list too long
/usr/sbin/pkg: line 5912: uniq: Argument list too long
/usr/sbin/pkg: line 5913: mv: Argument list too long
ESC[32mUninstalled:ESC[0m discord
/usr/sbin/pkg: line 268: grep: Argument list too long
/usr/sbin/pkg: line 5933: which: Argument list too long
/usr/sbin/pkg: line 1: wc: Argument list too long
ash: -le: argument expected
/usr/sbin/pkg: line 1: wc: Argument list too long
ash: -le: argument expected
/usr/sbin/pkg: line 1: wc: Argument list too long
ash: -le: argument expected
/usr/sbin/pkg: line 1: wc: Argument list too long
ash: -le: argument expected
The last two lines repeated over and over as the script clawed desperately at the air, its mind spinning in circles. The blood drained from our little witch’s face. She hung up the terminal and the package manager slumped down. Her thoughts were swirling, but one question stood out among all the others: Why?
With not a moment to waste, she hurried to the scene of the crime.
$ sed -n '5910,5914 p' < /usr/sbin/pkg
# clean up user-installed-packages (remove duplicates and empty lines)
grep -v "^\$" ${REPO_DIR}/user-installed-packages | uniq > ${REPO_DIR}/user-installed-packages_clean
mv ${REPO_DIR}/user-installed-packages_clean ${REPO_DIR}/user-installed-packages
Odd. Just a typical grep and and uniq command. The uniq doesn’t even have any arguments! How can the argument list be too long when there’s no arguments? Surely, there must be an explanation. Someone or something had killed her package manager, and she was going to figure out what.
The girl retreated into her mental archives. Was this a shell problem? A Linux problem? Argument list too long, argument list too long… Linux certainly has a maximum argument length for programs. Was the something setting the limit to zero somehow? She paged through her memories searching for something, anything, that mentioned argument lists. Finally she found something. A memory, not about arguments, but environment variables.
You see, when the Kernel executes a program, it provides the current set of environment variables directly adjacent to the command line arguments in memory. Could it be that the argument length limit applied to the environment variables too? A query online said yes, but the stack exchange is wily and not to be trusted without verification. Our protagonist dived into the linux source code.
In fs/exec.c she found a function named bprm_stack_limits, which had this to say on the matter:
limit = max_t(unsigned long, limit, ARG_MAX);
/*
* We must account for the size of all the argv and envp pointers to
* the argv and envp strings, since they will also take up space in
* the stack. They aren't stored until much later when we can't
* signal to the parent that the child has run out of stack space.
* Instead, calculate it here so it's possible to fail gracefully.
*
* In the case of argc = 0, make sure there is space for adding a
* empty string (which will bump argc to 1), to ensure confused
* userspace programs don't start processing from argv[1], thinking
* argc can never be 0, to keep them from walking envp by accident.
* See do_execveat_common().
*/
ptr_size = (max(bprm->argc, 1) + bprm->envc) * sizeof(void *);
if (limit <= ptr_size)
return -E2BIG;
She chuckled, That second half of the block comment was an echo of a recent attack on polkit. But there at the bottom was the answer to the question at hand: max(bprm->argc, 1) + bprm->envc. The kernel source agreed, environment variables take space away from the argument list. If an environment variable was too big, it could stop the shell from running programs at all! But what environment variable could have gotten that large?
The intrepid system administrator returned to the package manager’s corpse, this time peering above the injury.
$ sed -n '5906,5910 p' < /usr/sbin/pkg
# remove $PKGNAME from user-installed-packages
NEWUSERPKGS="$(grep -v "^${PKGNAME}" ${REPO_DIR}/user-installed-packages)"
[ "$NEWUSERPKGS" != "" ] && echo "$NEWUSERPKGS" > ${REPO_DIR}/user-installed-packages
Hmm… so the entirety of user-installed-packages was loaded into NEWUSERPKGS. How big was that file?
$ wc -c /var/packages/user-installed-packages
172474 /var/packages/user-installed-packages
Well, that certainly seemed large enough to overflow a reasonable argument list length limit. It seemed to our witch that she finally had a suspect. But, how could she be sure? There weren’t any commands exporting that variable, and it’s unbecoming to levy such an accusation against a line of code without reasonable proof. Perhaps if there were some way to print the exported environment? She tried adding an env to the script, but it was no use.
/usr/sbin/pkg: line 5910: env: Argument list too long
Of course, env was a separate program, and the shell couldn’t launch those. But maybe it could run a builtin command. If exports were the problem, maybe they could be the solution too!
$ export --help
export: export [-fn] [name[=value] ...] or export -p
Options:
-f refer to shell functions
-n remove the export property from each NAME
-p display a list of all exported variables and functions
There, -p! That’s what she needed. She sprinkled an export -p into the code.
export EDITOR='vim'
export HOSTNAME='puppypc1400'
export KICAD_PATH='/usr/share/kicad'
export LS_COLORS='bd=33:cd=33'
export NEWUSERPKGS='tmux_3.0a-2|tmux|3.0a-2||Utility;shell|750K|pool/main/t/tmux|tmux_3.0a-2_amd64.deb|+libc6&ge2.27,+libevent-2.1-7&ge2.1.8-stable,+libtinfo6&ge6,+libutempter0&ge1.1.5|terminal multiplexer|ubuntu|focal|
vim-common_8.1.2269|vim-common|8.1.2269|1ubuntu5|Filesystem;filemanager|375K|pool/main/v/vim|vim-common_8.1.2269-1ubuntu5_all.deb|+xxd|Vi IMproved - Common files|ubuntu|focal|
vim-runtime_8.1.2269|vim-runtime|8.1.2269|1ubuntu5|Filesystem;filemanager|30765K|pool/main/v/vim|vim-runtime_8.1.2269-1ubuntu5_all.deb||Vi IMproved - Runtime files|ubuntu|focal|
[... snip ...]
The witch grinned. There it was, NEWUSERPKGS printing out as far as the eye could see. With a culrpit identified, she had what she needed to resurrect her package manager from its untimely death. She added an export -n NEWUSERPKGS above the wound, re-aligned her runes, and sent a jolt of energy into the package manager. Its eyes lit up.
$ pkg u discord
Uninstall the package discord:
Uninstalled: discord
$
She’d done it! Her package manager was back once again, alive and well.
But there was a loose thread dangling. There were no export commands around NEWUSERPKGS, so why was it exported to the child processes? Could a shell be instructed to export all of its variables automatically? She consulted the stack exchange once more.
“What do you mean export all at once? you can use semi colons to define in one line” said a voice in the stack exchange. “Your question is unclear” chimed another. But finally, a moment of clarity: “set -a: When this option is on, the export attribute shall be set for each variable to which an assignment is performed”. She took once more to the package manager, sifting gently through its code with her regular expressions:
$ grep -C2 'set -a' /usr/sbin/pkg
#==================== main functions ======================#
set -a
# utility funcs
There it was. The trouble maker that had set this all in motion. A bit of a silly choice for a shell script, but so things were. She offered to remove it from the package manager, but it expressed misgivings that it might start malfunctioning. She nodded. The package manager was back on its feet, so better to let it be, for now.
With her Discord client updated, the UNIX witch gave her package manager a gentle pat, and she bid it back to its slumbers. She’d wake it again, when the time arose. For now, it deserved some rest, and so did she.