Simple Executable Love2D Files, or, You Can Shove Random Data At The Start of a Zip File and it's Basically Fine
LÖVE (which I will write as love because my keyboard doesn’t have an Ö) is a neat program that’s mostly intended for writing games with lua. We’ve been using it to write an image viewer. There’s a lot of ways to package a love project up for distribution, and some of them ship a copy of love with the project and some don’t. Since my distribution provides the version of love I need, I can create a .love file with all my source code and assets in it, and then I can run it with love path/to/myprogram.love. A .love file is just a .zip with a different file extension, so that’s pretty easy to do.
I didn’t want to have to specifically execute my program by typing out love path/to/myprogram.love though. I wanted to be able to throw it into a directory on our PATH so I can just run like myprogram path/to/image.png and have it execute.
I could do this pretty easily by putting my .love file somewhere on disk, and then putting a shell script on my PATH that executes that file with love… but, what if the zip file and the script were actually the same file? I didn’t actually know much about the zip format yet, but I had to try. So I gave it a shot with one of our old Kaleidoscope generator programs:
cd kaleidoscope
zip -r ../kaleidoscope.zip .
adding: main.lua (deflated 70%)
cd ../
echo '#!/usr/bin/env love' > kaleidoscope.love
cat kaleidoscope.zip >> kaleidoscope.love
chmod +x kaleidoscope.love
./kaleidoscope.love
Huh. There it is. Well alright, let’s go a bit further. We use wayland a fair bit these days, and right now this love program is running under Xwayland. I happen to know that love uses SDL under the hood. The version on my system doesn’t enable their native wayland backend by default yet, but it seems to work fine for me, so I figured I’d set the environment variable to turn it on.
Unfortunately, it seems like you cannot actually set environment variables with a #!/usr/bin/env shebang. The program just hangs forever. This isn’t related to love or the zip file stuff at all, it always happens. But, if we could shove a shebang at the front of the zip, why not a whole shell script?
echo '#!/bin/sh
if [ -n "$WAYLAND_DISPLAY" ]; then
export SDL_VIDEODRIVER=wayland
fi
exec love "$0" "$@"
' > kaleidoscope.love
cat kaleidoscope.zip >> kaleidoscope.love
./kaleidoscope.love
Cool! I could stop here, it clearly works. But, I decided to learn a bit more about zips, because I wanted to know: is this still a valid zip file? And if not, how can I make it one?
Let’s just try to unzip it somewhere:
mkdir /tmp/whatever
cp kaleidoscope.love /tmp/whatever
cd /tmp/whatever
unzip kaleidoscope.love
Archive: kaleidoscope.love
warning [kaleidoscope.love]: 102 extra bytes at beginning or within zipfile
(attempting to process anyway)
inflating: main.lua
Interesting, so we are violating the spec, but the unzipper libraries are just able to figure it out anyway.
We looked into it a bit further and it turns out that the main thing making this work at all is that the zip file directory is stored at the end of the file, not the start. So it’s easy for software to see that it is in fact a zip file. The thing is, the directory specifies the locations of files relative to the start of the file. So we’ve shoved 102 bytes at the start of the zip file, and now all the offsets are 102-bytes away from where they should be. This is detectable, clearly, but not ideal. But, this is the only problem, actually. If we rewrote all the offsets, adding 102 to each of them, then our zip file would be completely 100% valid!
Rather than write a program to do that, I instead wrote a script that generates zip files from scratch, writing the offsets correctly as it goes. I didn’t bother actually making it compress anything, since I don’t really care about that right now. But, if you’re curious, here it is! Use at your own risk.
You need zlib installed (I use it for crc32 despite the lack of compression), though you almost certainly already do. You need cffi-lua to load it. You need luaposix. And you need lua5.3 for string.pack().
#!/usr/bin/env lua
-- Change this to whatever you want to put at the front of the zip
local love_file_loader = [[#!/bin/sh
# This is a love2d zip file! You can extract it with any unzipper tool to see
# the source code.
if [ -n "$WAYLAND_DISPLAY" ]; then
export SDL_VIDEODRIVER=wayland
fi
exec love "$0" "$@"
]]
local input_dir, output_love = ...
if input_dir == nil then
print([[
Usage: ' .. arg[0] .. ' <love project input_dir> [output.love]
Basically this zips up the input_dir and creates a file with
#!/usr/bin/env love
and then the zip file appended. which works, somehow! If you don't say what
output.love to use it will add `.love` to the project input_dir path and use
that.
]])
os.exit(1)
end
-- default .love extension. trims trailing slashes first
output_love = output_love or (input_dir:match('^(.-)/*$') .. '.love')
--[[
I feel like zip files are kinda frustrating to deal with on linux. Rather than
try to wrange various command line interfaces, let's do it ourselves.
I considered using libzip but I don't really like its interface. You have to
give it a file path and let *it* open the file if you want to write data. Meh.
It won't work for what we're trying to do.
But we probably shouldn't try to do DEFLATE in lua right now, so let's just
write uncompressed for now. After all, we're just putting lua files and pngs in
a box.
Despite this, we still pull in libz for now, because it has a crc32 function
and we need crc32. Maybe later we can add compression with it too.
https://www.zlib.net/manual.html
]]
local cffi = require('cffi')
local libz = cffi.load('z')
cffi.cdef([[
extern unsigned long crc32(
unsigned long crc,
const unsigned char *buf,
unsigned int len
);
]])
-- We'll use luaposix to traverse the directory and pack files in
local posix = require('posix')
local posix_stat = require('posix.sys.stat')
--[[
wrapper around libz crc32, needed by zip creation. A crc is just a 32-bit
number, so we take that number in and return a new one rather than updating an
object.
https://github.com/q66/cffi-lua/blob/master/docs/introduction.md#caching
]]
local libz_crc32 = libz.crc32
local libz_buf = cffi.typeof('const unsigned char*')
local function crc32_new()
return libz_crc32(0, nil, 0)
end
local function crc32_update(crc, bytes)
local buf = cffi.cast(libz_buf, bytes)
return libz_crc32(crc, buf, #bytes)
end
local function crc32_finalize(crc)
return cffi.tonumber(crc)
end
--[[
zip file creation!
https://en.wikipedia.org/wiki/Zip_(file_format)
We take a base directory path, a list of file paths relative to the base dir, an
output file handle, and a flag for whether to include the directory name in the
output. That is should create_zip('blah', ...) create files 'blah/whatever/' or
just 'whatever/'
The main reason for this is so that all the offsets in the zip file are relative
to the start of the zip with stuff shoved in front of it (such as a shebang).
The structure of a zip file is
- List of files, each with
- Local header
- data
- Central directory of file entries
- End Of Central Directory
These data structure descriptions are copy-pasted from the wikipedia article.
=== Local Header ===
0 4 Local file header signature = 0x04034b50 (PK♥♦ or "PK\3\4") (lil end
4 2 Version needed to extract (minimum)
6 2 General purpose bit flag
8 2 Compression method; e.g. none = 0, DEFLATE = 8 (or "\0x08\0x00")
10 2 File last modification time
12 2 File last modification date
14 4 CRC-32 of uncompressed data
18 4 Compressed size (or 0xffffffff for ZIP64)
22 4 Uncompressed size (or 0xffffffff for ZIP64)
26 2 File name length (n)
28 2 Extra field length (m)
30 n File name
30+n m Extra field
Note that to fill in the Compressed size without pre-compressing a file in RAM,
we can write the header, write the file data, then seek backwards.
=== Central Directory Entry ===
0 4 Central directory file header signature = 0x02014b50 (little endian)
4 2 Version made by
6 2 Version needed to extract (minimum)
8 2 General purpose bit flag
10 2 Compression method
12 2 File last modification time
14 2 File last modification date
16 4 CRC-32 of uncompressed data
20 4 Compressed size (or 0xffffffff for ZIP64)
24 4 Uncompressed size (or 0xffffffff for ZIP64)
28 2 File name length (n)
30 2 Extra field length (m)
32 2 File comment length (k)
34 2 Disk number where file starts (or 0xffff for ZIP64)
36 2 Internal file attributes
38 4 External file attributes
42 4 Relative offset of local file header (or 0xffffffff for ZIP64). This is the number of bytes between the start of the first disk on which the file occurs, and the start of the local file header. This allows software reading the central directory to locate the position of the file inside the ZIP file.
46 n File name
46+n m Extra field
46+n+m k File comment
=== End of central directory ===
0 4 End of central directory signature = 0x06054b50
4 2 Number of this disk (or 0xffff for ZIP64)
6 2 Disk where central directory starts (or 0xffff for ZIP64)
8 2 Number of central directory records on this disk (or 0xffff for ZIP64)
10 2 Total number of central directory records (or 0xffff for ZIP64)
12 4 Size of central directory (bytes) (or 0xffffffff for ZIP64)
16 4 Offset of start of central directory, relative to start of archive (or 0xffffffff for ZIP64)
20 2 Comment length (n)
22 n Comment
]]
local function create_zip(basedir, files, outf)
local filemeta = {}
for _, name in ipairs(files) do
local path = basedir .. '/' .. name
local meta = {
offset = outf:seek()
}
filemeta[name] = meta
-- stat file
local stat = posix_stat.stat(path)
-- We can only handle regular files
assert(posix_stat.S_ISREG(stat.st_mode) ~= 0)
-- Save metadata
meta.compression_method = '\0\0' -- none compression left beef
--[[
write the local header. To start off with,
- checksum is 0
- file sizes are 0
We will seek back and fill those in later.
]]
local header = string.format(
'PK\x03\x04\0\0\0\0%s\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0%s\0\0%s',
meta.compression_method,
string.pack('<I2', #name),
name
)
outf:write(header)
--[[
Read file and copy it in. Calculate crc32 as we do. Keep track of byte
count
]]
local inf = assert(io.open(path, 'r'))
local size = 0
local crc = crc32_new()
repeat
local bytes = inf:read(65536)
if bytes then
size = size + #bytes
crc = crc32_update(crc, bytes)
outf:write(bytes)
end
until not bytes
inf:close()
crc = crc32_finalize(crc)
meta.crc = crc
meta.size = size
meta.compressed_size = size
--[[
Seek backwards to fill in fields
- file
- file name
- length of file name (2 bytes)
- length of extra (2 bytes)
- crc, sizes (12 bytes)
]]
outf:seek('cur', -(meta.compressed_size + #name + 16))
local crc_and_sizes = string.pack(
'<I4I4I4', crc, size, size
)
outf:write(crc_and_sizes)
--[[
Seek forwards past the file data again
- file
- file name
- length of file name (2 bytes)
- length of extra (2 bytes)
]]
outf:seek('cur', size + #name + 4)
end
--[[
Ok at this point we've written all the files, so now we need to build the
directory entry. This is a bit simpler since we already have the crc and
sizes calculated. No need for seek shenanigans.
]]
local central_directory_size = 0
local central_directory_offset = outf:seek()
for _, name in ipairs(files) do
local meta = filemeta[name]
-- idk what else to call this
local data_fields = string.pack(
'<I4I4I4I2',
meta.crc,
meta.size,
meta.compressed_size,
#name
)
--[[
we need to calculate the relative offset from the "start of the first
disk", which I interpret to be the start of the file. I'm not going to
use the *actual* start of the file, just the start of our zip entries,
since we are doing hax to append a zip to a shebang
]]
local entry = string.format(
'\x50\x4b\x01\x02\0\0\0\0\0\0%s\0\0\0\0%s\0\0\0\0\0\0\0\0\0\0\0\0%s%s',
meta.compression_method,
data_fields,
string.pack('<I4', meta.offset),
name
)
central_directory_size = central_directory_size + #entry
outf:write(entry)
end
-- Close out the zip file with the end of directory marker
local entry_data = string.pack(
'<I2I2I4I4',
#files, -- number of records on this disk
#files, -- number of records across all disks
central_directory_size,
central_directory_offset
)
local closing_entry = string.format(
'\x50\x4b\x05\x06\0\0\0\0%s\0\0', entry_data
)
outf:write(closing_entry)
end
-- Get all the files we want to put in then zip
local file_list = {}
local function traverse(d)
for _, basename in ipairs(posix.dirent.dir(input_dir .. '/' .. d)) do
-- don't self-recurse
if basename == '.' or basename == '..' then
goto continue
end
local relpath = d .. '/' .. basename
local path = input_dir .. '/' .. relpath
local stat = posix.sys.stat.stat(path)
if posix_stat.S_ISDIR(stat.st_mode) ~= 0 then
traverse(relpath)
elseif posix_stat.S_ISREG(stat.st_mode) ~= 0 then
table.insert(file_list, relpath:match('^/(.+)'))
end
::continue::
end
end
for _, v in ipairs(file_list) do
print(v)
end
traverse('')
-- Output .love file
local f = assert(io.open(output_love, 'w'))
--[[
Start the .love file with a shebang
this works:
f:write('#!/usr/bin/env love\n')
but doesnt let you specify environment variables. to do *that* we need to make
this be a bash script. that's fine though.
]]
f:write(love_file_loader)
create_zip(input_dir, file_list, f)
f:close()
-- chmod +x it
posix.chmod(output_love, 'ugo+x')