new post: oelf - Mach-O support for sqlelf
This commit is contained in:
parent
899e806e53
commit
13be151e6c
162
_posts/2024-01-02-oelf.md
Normal file
162
_posts/2024-01-02-oelf.md
Normal file
|
|
@ -0,0 +1,162 @@
|
|||
---
|
||||
permalink: "/{{ year }}/{{ month }}/{{ day }}/oelf"
|
||||
title: "oelf - Mach-O support for sqlelf"
|
||||
published_date: "2024-01-02 13:00:00 +0200"
|
||||
layout: post.liquid
|
||||
data:
|
||||
route: blog
|
||||
tags:
|
||||
- rust
|
||||
---
|
||||
|
||||
[sqlelf] lets you explore ELF objects through the power of SQL.
|
||||
It turns any executable into a queryable database.
|
||||
Any? No, just those in ELF format, the standard binary file format on most Unix and Linux systems.
|
||||
But not on macOS.
|
||||
macOS relies on the Mach object file format, short Mach-O and sqlelf doesn't support that.
|
||||
The library it uses for parsing in theory does, but that failed on my machine.
|
||||
It depends on a heavy C++ library and I didn't want to bother figuring out how to build and change that.
|
||||
|
||||
I still wanted [sqlelf for mach-o binaries][macho-support].
|
||||
Luckily the hardest part was settled early on: [naming].
|
||||
|
||||
> <@fnordfish@mastodon.social>
|
||||
> @jer sollte auf jeden Fall „Ölf“ heißen!
|
||||
> (_translation:_ it should definitely be called „Ölf“)
|
||||
|
||||
So [oelf][oelf-py] exists now.
|
||||
The [source code is on GitHub][oelf]
|
||||
and [my fork of sqlelf adds it to sqlelf][macho-support] for easy use.
|
||||
|
||||
### Install
|
||||
|
||||
I have released pre-built versions of oelf,
|
||||
but nothing is upstreamed to sqlelf yet.
|
||||
You can install it from git:
|
||||
|
||||
```shell
|
||||
pip install git+https://github.com/badboy/sqlelf@with-macho-support#egg=sqlelf
|
||||
```
|
||||
|
||||
On my M1 MacBook sqlelf doesn't work out of the box.
|
||||
sqlelf depends on [Capstone](https://www.capstone-engine.org/)
|
||||
and the installed library coming with the Python wrapper is `x86_64` only,
|
||||
so it won't load
|
||||
|
||||
That's fixable.
|
||||
Assuming you installed into a Python venv `.venv`:
|
||||
Install capstone from Homebrew, remove the bundled library and link to the global one instead:
|
||||
|
||||
```shell
|
||||
brew install capstone
|
||||
rm .venv/lib/python3.11/site-packages/capstone/lib/libcapstone.dylib
|
||||
ln -s $(brew --cellar capstone)/5.0.1/lib/libcapstone.5.dylib .venv/lib/python3.11/site-packages/capstone/lib/libcapstone.dylib
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
Invoke `sqlelf` and pass any number of Mach-O binaries as arguments.
|
||||
This gives you an SQLite REPL, or you specify SQL commands with `--sql`.
|
||||
|
||||
For example sqlelf knows about libraries references by the binary:
|
||||
```
|
||||
$ sqlelf /usr/bin/grep --sql 'select * from macho_libs'
|
||||
┌───────────────┬────────────────────────────┐
|
||||
│ path │ lib │
|
||||
│ /usr/bin/grep │ self │
|
||||
│ /usr/bin/grep │ /usr/lib/libbz2.1.0.dylib │
|
||||
│ /usr/bin/grep │ /usr/lib/liblzma.5.dylib │
|
||||
│ /usr/bin/grep │ /usr/lib/libz.1.dylib │
|
||||
│ /usr/bin/grep │ /usr/lib/libSystem.B.dylib │
|
||||
└───────────────┴────────────────────────────┘
|
||||
```
|
||||
|
||||
None of those `/usr/lib/*.dylib` actually exists in the filesystem though,
|
||||
because Apple now ships those as a big bundled cache file instead.
|
||||
|
||||
I have not yet documented the schema nor is it anywhere near complete.
|
||||
Use `.schema` to get an overview.
|
||||
|
||||
```
|
||||
$ sqlelf /usr/bin/grep --sql '.schema macho_headers'
|
||||
CREATE TABLE macho_headers(
|
||||
path,
|
||||
magic,
|
||||
cputype,
|
||||
cpusubtype,
|
||||
filetype,
|
||||
ncmds,
|
||||
sizeofcmds,
|
||||
flags,
|
||||
reserved
|
||||
);
|
||||
```
|
||||
|
||||
Tables are persisted views over the data, so everything is in memory.
|
||||
Most values are the raw values read from the file,
|
||||
so you will have to look up what those values mean.
|
||||
|
||||
For example the headers include all sorts of magic numbers and file types as integers:
|
||||
|
||||
```
|
||||
$ sqlelf /usr/bin/grep --sql 'select * from macho_headers'
|
||||
┌───────────────┬────────────┬──────────┬────────────┬──────────┬───────┬────────────┬─────────┬──────────┐
|
||||
│ path │ magic │ cputype │ cpusubtype │ filetype │ ncmds │ sizeofcmds │ flags │ reserved │
|
||||
│ /usr/bin/grep │ 4277009103 │ 16777223 │ 3 │ 2 │ 21 │ 1688 │ 2097285 │ 0 │
|
||||
└───────────────┴────────────┴──────────┴────────────┴──────────┴───────┴────────────┴─────────┴──────────┘
|
||||
```
|
||||
|
||||
You can slice and dice the data as you wish[^1].
|
||||
|
||||
```
|
||||
$ sqlelf /usr/bin/grep --sql "select name, type, global, n_value from macho_symbols where path = '/usr/bin/grep' limit 3"
|
||||
┌─────────────────────┬────────┬────────┬────────────┐
|
||||
│ name │ type │ global │ n_value │
|
||||
│ radr://5614542 │ N_PBUD │ 0 │ 90260802 │
|
||||
│ __mh_execute_header │ N_SECT │ 1 │ 4294967296 │
|
||||
│ _BZ2_bzRead │ N_UNDF │ 1 │ 0 │
|
||||
└─────────────────────┴────────┴────────┴────────────┘
|
||||
```
|
||||
|
||||
### Status
|
||||
|
||||
I hacked together `oelf` in a matter of days.
|
||||
I'm using the excellent [pyo3] to wrap [goblin]'s functionality into a Python package, built with [maturin].
|
||||
It works reliably (yey for great tooling written in and for Rust!),
|
||||
but so far I haven't documented much.
|
||||
`oelf` itself is a bit inconsistent in how it exposes different data.
|
||||
The `sqlelf` integration is really simple,
|
||||
thanks to a nice extensible code structure of the project.
|
||||
Now every newly exposed functionality in `oelf` needs only defining the schema of a table
|
||||
and mapping the retrieved data to its columns.
|
||||
|
||||
I have yet to actually _use_ sqlelf myself more to explore binaries and all the data in there.
|
||||
I also have only a bare understanding of the ELF format and much much less of the Mach-O format,
|
||||
I'm just barely good at plugging together existing things.
|
||||
|
||||
Some things that might be good to do:
|
||||
|
||||
* Documentation (of course!)
|
||||
* Add more sections and tables and "translate" magic values
|
||||
* e.g. load commands (`macho_load_commands`) pretty-print the Rust object right now, this should be proper data in the column, maybe just JSON to begin with?
|
||||
* Can we extract and parse system libraries from the shared dyld cache?
|
||||
* Others [have built stuff](https://github.com/keith/dyld-shared-cache-extractor)
|
||||
* Can we more lazily fetch data instead of copying into a persisted table once?
|
||||
* Upstream changes or fork it so `sqlelf` actually works out of the box on any non-Linux machine
|
||||
|
||||
---
|
||||
|
||||
_Footnotes_:
|
||||
|
||||
[^1]: `adr://5614542`: [this is a workaround for a workaround because of a bug in the old classic static linker.][radr]
|
||||
|
||||
[sqlelf]: https://fzakaria.com/2023/03/19/sqlelf-and-20-years-of-nix.html
|
||||
[oelf]: https://github.com/badboy/oelf
|
||||
[oelf-py]: https://pypi.org/project/oelf/
|
||||
[my-toot]: https://hachyderm.io/@jer/111470860656151925
|
||||
[naming]: https://hachyderm.io/@fnordfish@mastodon.social/111476474716125707
|
||||
[macho-support]: https://github.com/badboy/sqlelf/tree/with-macho-support
|
||||
[pyo3]: https://pyo3.rs/
|
||||
[radr]: https://github.com/PureDarwin/PureDarwin/blob/a9f762d321016242bb95542301a91ecb4eb9bfd3/tools/cctools/misc/strip.c#L3789-L3817
|
||||
[goblin]: https://crates.io/crates/goblin
|
||||
[maturin]: https://maturin.rs/
|
||||
Loading…
Reference in a new issue